[{"data":1,"prerenderedAt":398},["ShallowReactive",2],{"project-20449610":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":35,"aspects":36,"strip":42,"post_text":43,"what":44,"said":67,"maker_in_thread":179,"shape":188,"grid":193,"attempts":361,"same_week":362,"maker_shows":308,"maker_hidden":390,"reader_version":391,"card":392},"20449610","OctoSQL","Query and join multiple databases and files, written in Go","title","Show HN: OctoSQL – Query and join multiple databases and files, written in Go","https:\u002F\u002Fgithub.com\u002Fcube2222\u002Foctosql","github.com","cube2222",1563282275,2019,310,70,94,2195,5,"legend","dev_tools","command line query tool","cli",[24,25,26,27,28],"sql","join csv","query json","multiple databases","command line","curious",0.033,{"positive":32,"neutral":33,"mixed":34,"critical":32},3,10,0,16,{"idea":37,"execution":39,"performance":41},{"positive":38,"critical":38},2,{"positive":34,"critical":40},1,{"positive":34,"critical":34},[34,32,34,34,34,34,34,34,34,34,34,40,32,34,34,32,34,34,18,32,34,34,34,34,34,34,40,32,34,32,32,32,34,34,32,32,34,32,18,34],null,{"problem":45,"audience":43,"pricing":43,"motivation":52,"time_to_build":43,"stack":56,"openness":65,"stage":66,"team_size":66},{"value":46,"source":47,"evidence":48},"Joining data from different sources without setting up heavy systems like Presto or Spark","maker",[49],{"id":50,"by":11,"text":51},"20449703","Hey, one of the authors here. The motivation behind this project is that I always wanted a simple commandline tool allowing me to join data from different places, without needing…",{"value":53,"source":47,"evidence":54},"Wanted a simple command-line tool to join data from different places and query CSV and JSON with SQL",[55],{"id":50,"by":11,"text":51},[57],{"value":58,"source":47,"evidence":59},"Go",[60,62],{"id":61},"post",{"id":63,"by":11,"text":64},"20450296","Make sure to let us know if you have any feedback after testing it out! The Go standard library really helped handling all the different json file formats you might expect. (Like…","open_source","unknown",{"praise":68,"objections":81,"requests":102,"questions":123,"alternatives":138},[69],{"value":70,"source":71,"evidence":72},"Simple interface and useful idea for joining data sources","commenter",[73,77],{"id":74,"by":75,"text":76},"20453375","sixdimensional","A great motivation and keeping it simple from an interface perspective is a cool intention. It's a great start with tons of directions you can take it, and many interesting…",{"id":78,"by":79,"text":80},"20450246","srameshc","Thanks for sharing. This is cool, just what I wanted. I like the fact I can create config files and query not just he database but also query JSON.",[82,91],{"value":83,"source":47,"evidence":84},"Only supports reads, and lookup joins scan whole files each time",[85,88],{"id":86,"by":11,"text":87},"20452477","Thanks! We only support reads, and for now we're not planning to add write support. You can use the csv output format though and later import that somewhere.",{"id":89,"by":11,"text":90},"20474368","Hey, sorry for the late response, currently totally inefficient - scan whole file each time. We plan on adding an in memory or on disk index for such cases.",{"value":92,"source":71,"evidence":93},"Apache Drill already does this",[94,98],{"id":95,"by":96,"text":97},"20450606","minxomat","> commandline tool allowing me to join data from different places, without needing to set up stuff like presto or spark. On another hand, I never encountered any tool which…",{"id":99,"by":100,"text":101},"20450746","dk-","Yeah an Drill has been around for a few years. Will likely continue to use Drill over octocat (no offense OP)",[103,109,116],{"value":104,"source":47,"evidence":105},"Support stdin piping",[106],{"id":107,"by":11,"text":108},"20450654","Thanks! 1. Initially we wanted to do one table per config-entry, but that'll be an easy to do UX improvement. 2. We're planning to add support for piping the csv\u002Fjson file…",{"value":110,"source":71,"evidence":111},"Link whole databases, not just tables",[112],{"id":113,"by":114,"text":115},"20450548","mjirv","This is great. I keep the following in my bash profile for querying CSVs with SQL, which works OK, but this is probably better: ``` function csv_to_sqlite() { echo \"used as…",{"value":117,"source":71,"evidence":118},"Add a server or daemon with scheduling",[119],{"id":120,"by":121,"text":122},"20450954","irq-1","Consider adding a server\u002Fdaemon with jobs and scheduling. This would let people copy and reconcile data between systems, or prep data at night, etc... It could also save indexes…",[124,131],{"value":125,"source":71,"evidence":126},"Does it support writing to data sources?",[127],{"id":128,"by":129,"text":130},"20452454","arendtio","Pretty cool stuff. Does it support writing to (single) data sources too or only reading queries?",{"value":132,"source":71,"evidence":133},"Can it do hash joins for large data?",[134],{"id":135,"by":136,"text":137},"20450717","polskibus","When joining tables from 2 different data sources, is the join performed by loading two tables in full into memory ? If yes - any plans on improving this, so that it does not OOM…",[139,147,158,165,172],{"value":140,"source":71,"evidence":141},"Apache Drill",[142,143],{"id":95,"by":96,"text":97},{"id":144,"by":145,"text":146},"20450674","cplat","Cool project. On a side note, Apache Drill (https:\u002F\u002Fdrill.apache.org\u002F) is also worth checking out. The homepage doesn't make it very apparent, but I've used it to query CSVs,…",{"value":148,"source":71,"evidence":149},"PostgreSQL foreign data wrappers",[150,154],{"id":151,"by":152,"text":153},"20450065","truth_seeker","Generally, I make use of file_fdw module of Postgres to achieve this. https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002F11\u002Ffile-fdw.html",{"id":155,"by":156,"text":157},"20450282","jon-wood","Postgres Foreign Data Wrappers are an incredibly powerful tool. I've in the past built a basic data warehouse for a distributed system by pulling in read slaves from assorted…",{"value":159,"source":71,"evidence":160},"Apache Calcite",[161],{"id":162,"by":163,"text":164},"20450152","MoOmer","Really cool, thanks for sharing. You all might want to look at Apache Calcite (http:\u002F\u002Fcalcite.apache.org\u002F) as well for inspiration, which has similar functionality as a subset of…",{"value":166,"source":71,"evidence":167},"Dremio",[168],{"id":169,"by":170,"text":171},"20450556","sails","Really nice! Something similar (with a GUI) is https:\u002F\u002Fwww.dremio.com\u002F that I think has gained some traction.",{"value":173,"source":71,"evidence":174},"Sqawk",[175],{"id":176,"by":177,"text":178},"20450633","blacksqr","\"Sqawk is an Awk-like program that uses SQL and can combine data from multiple files. It is powered by SQLite.\" https:\u002F\u002Fgithub.com\u002Fdbohdan\u002Fsqawk",{"comments":35,"answered":18,"key":180},[181,182,185],{"id":50,"text":51},{"id":183,"text":184},"20450129","Didn't know about that, thanks! Is there a way to connect this to other databases, to basically have views of your redis\u002FMySQL\u002Fother databases in postgres?",{"id":186,"text":187},"20450191","Seems like it's solving part of the problems we're having really well, I'll have to read up more on it, thanks!",{"commenters":189,"max_depth":190,"maker_replies":35,"first_comment_mins":191,"lifespan_hours":192},40,6,12,57,[194,196,198,200,203,204,206,207,210,213,215,218,220,222,225,229,231,234,236,239,241,244,248,250,252,255,259,262,263,264,267,270,272,274,277,279,282,284,287,290,291,293,296,299,302,304,305,306,307,309,311,312,313,317,319,320,324,326,329,331,333,336,339,341,343,344,346,351,355,358],[50,34,34,195,11,40],"maker\u002Fneutral\u002Fidea",[151,34,32,197,152,34],"project\u002Fneutral\u002Fidea",[183,34,34,199,11,40],"maker\u002Fneutral\u002Fnone",[201,34,34,202,152,34],"20450140","other\u002Fneutral\u002Fnone",[186,34,34,202,11,40],[205,34,34,202,152,34],"20450275",[155,34,34,202,156,34],[208,34,34,202,209,34],"20450707","brunoqc",[211,34,34,202,212,34],"20450959","michelpp",[214,34,34,202,152,34],"20451153",[216,34,34,202,217,34],"20451600","tempguy9999",[219,34,34,202,75,34],"20453358",[221,34,34,202,209,34],"20456158",[223,34,34,224,217,34],"20457752","commenter\u002Fneutral\u002Fnone",[226,34,34,227,228,34],"20451284","other\u002Fneutral\u002Fperformance","riku_iki",[230,34,34,227,152,34],"20451394",[232,34,34,202,233,34],"20451726","just_myles",[235,34,34,227,75,34],"20453166",[237,34,34,202,238,34],"20456311","greggyb",[95,34,40,240,96,34],"project\u002Fcritical\u002Fidea",[242,34,34,243,11,40],"20450794","maker\u002Fpositive\u002Fnone",[245,34,32,246,247,34],"20451818","project\u002Fneutral\u002Fnone","dlphn___xyz",[249,34,34,202,96,34],"20452037",[251,34,34,227,75,34],"20453143",[253,34,34,202,254,34],"20455160","nerdponx",[256,34,34,257,258,34],"20450910","other\u002Fneutral\u002Fidea","tirumaraiselvan",[260,34,34,202,261,34],"20452395","bryanrasmussen",[128,34,32,197,129,34],[86,34,34,195,11,40],[265,34,34,202,266,34],"20452737","adonese",[268,34,34,202,269,34],"20453036","monstrado",[74,34,18,271,75,34],"project\u002Fpositive\u002Fidea",[273,34,34,243,11,40],"20453440",[275,34,32,199,276,34],"20453825","jhoechtl",[278,34,34,195,11,40],"20453849",[280,34,34,202,281,34],"20454404","simonw",[283,34,34,197,11,40],"20457206",[285,34,34,202,286,34],"20454483","tensor",[288,34,34,202,289,34],"20469492","bmn__",[162,40,34,202,163,34],[292,40,34,243,11,40],"20450238",[294,40,34,202,295,34],"20450791","jadbox",[297,40,34,202,298,34],"20450872","GordonS",[300,40,34,202,301,34],"20451144","taftster",[303,40,34,202,163,34],"20451414",[144,38,34,202,145,34],[99,38,40,240,100,34],[176,32,34,202,177,34],[120,308,32,197,121,34],4,[310,308,34,195,11,40],"20450971",[113,18,32,197,114,34],[107,18,34,197,11,40],[314,190,40,315,316,34],"20456044","project\u002Fcritical\u002Fexecution","quickthrower2",[78,318,18,271,79,34],7,[63,318,34,199,11,40],[321,322,32,197,323,34],"20452401",8,"kfk",[325,322,34,202,96,34],"20452451",[327,322,34,202,328,34],"20453391","stevepike",[330,322,34,199,11,40],"20452569",[332,322,34,202,75,34],"20453287",[135,334,32,335,136,34],9,"project\u002Fneutral\u002Fperformance",[337,334,34,338,11,40],"20450744","project\u002Fneutral\u002Fexecution",[340,334,32,335,228,34],"20451278",[89,334,34,342,11,40],"project\u002Fcritical\u002Fperformance",[169,33,32,197,170,34],[345,33,34,202,75,34],"20453302",[347,348,18,349,350,34],"20450440",11,"project\u002Fpositive\u002Fnone","21stio",[352,191,34,353,354,34],"20450931","other\u002Fneutral\u002Fdocs","yannis",[356,191,34,357,11,40],"20450957","maker\u002Fneutral\u002Fdocs",[359,191,34,353,360,34],"20450981","misterman0",[],[363,368,373,377,381,386],{"id":364,"name":365,"points":366,"category":367},"20421243","Listmonk",343,"infrastructure",{"id":369,"name":370,"points":371,"category":372},"20426682","Stein",296,"data",{"id":374,"name":375,"points":376,"category":367},"20410552","Inlets 2.0",218,{"id":378,"name":379,"points":380,"category":20},"20455857","fzf live REPL",190,{"id":382,"name":383,"points":384,"category":385},"20464215","Little Bird",140,"hardware",{"id":387,"name":388,"points":389,"category":20},"20434045","Cloud Maker",123,false,"v2",{"id":393,"name":5,"line":6,"ai":390,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":394,"top":18,"kw":395,"maker":11,"domain":10,"tier":19,"attempts":34,"strip":396,"url":9,"mix":397,"praise":70,"objection":83,"sub":21},20449610,0.03,[24,25,26,27,28],"0300000000013003005300000013033300330350",[32,33,34,32],1794036579166]