[{"data":1,"prerenderedAt":445},["ShallowReactive",2],{"project-27871574":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":36,"aspects":37,"strip":45,"post_text":46,"what":47,"said":118,"maker_in_thread":232,"shape":239,"grid":243,"attempts":404,"same_week":409,"maker_shows":32,"maker_hidden":437,"reader_version":438,"card":439},"27871574","Superintendent","Work with CSV files using SQL. For data scientists and engineers","title","Show HN: Work with CSV files using SQL. For data scientists and engineers","https:\u002F\u002Fsuperintendent.app\u002F","superintendent.app","tanin",1626597627,2021,99,61,533,2533,22,"notable","data","csv sql query tool","desktop_app",[24,25,26,27,28],"csv","sql","sqlite","csv query","data analysis","debated",-0.031,{"positive":32,"neutral":33,"mixed":34,"critical":35},5,6,1,4,16,{"performance":38,"idea":40,"execution":42,"docs":43,"pricing":44},{"positive":39,"critical":39},0,{"positive":35,"critical":41},3,{"positive":34,"critical":39},{"positive":39,"critical":34},{"positive":39,"critical":39},[39,39,39,41,39,39,39,34,39,41,39,32,39,39,39,39,41,39,41,39,39,32,39,39,39,39,32,41,34,39,39,34,39,39,39,39,39,32,41,41],null,{"problem":48,"audience":62,"pricing":74,"motivation":81,"time_to_build":46,"stack":91,"openness":116,"stage":117,"team_size":116},{"value":49,"source":50,"evidence":51},"Querying and joining CSV files with SQL without custom scripts or command line setup","inferred",[52,54,58],{"id":53},"post",{"id":55,"by":56,"text":57},"27872757","rognjen","Seems useful but at the same time it's comparison with loading CSV into SQL seems kind of disingenuous since if you're familiar with SQL you're probably familiar with command…",{"id":59,"by":60,"text":61},"27874933","mjirv","I like that this lets you join CSVs, which a lot of tools in this space don’t.",{"value":63,"source":64,"evidence":65},"Data scientists, engineers and analysts who know SQL but not the command line","commenter",[66,70],{"id":67,"by":68,"text":69},"27878713","raihansaputra","It's the inbetweeners that's the ideal target audience. Analysts (including me a few years back) that knows SQL (through Tableau\u002FBigQuery), but don't know how to get comfortable…",{"id":71,"by":72,"text":73},"27874946","jonnycomputer","So its a GUI wrapper around sqlite. I guess I expect data scientists and engineers to be comfortable with command line and scripting, and there are GUIs that will work with…",{"value":75,"source":64,"evidence":76},"Free while in beta",[77],{"id":78,"by":79,"text":80},"27872348","toyg","I'm old enough to know I should give a pass to anything \"free while in beta\"...",{"value":82,"source":83,"evidence":84},"Maker finds GUI easier than command line for CSV work and wanted a tool like that","maker",[85,88],{"id":86,"by":11,"text":87},"27877628","I probably model it off myself too much. I know Postgresql very well... But I rarely use psql. I still use pgadmin. That said, I never load CSVs into Postgresql. I still tried…",{"id":89,"by":11,"text":90},"27874757","The difference is between GUI and command-line. I work with a lot of CSV files, and for some reason I rarely load them into Postgres or use Python script. I still try very hard…",[92,101,106,112],{"value":93,"source":83,"evidence":94},"SQLite",[95,98],{"id":96,"by":11,"text":97},"27874815","Hi! I'm the OP here. The performance gain is mainly from reading\u002Fwriting files in C and\u002For by Sqlite (which is also C). Any other tool that use dynamic languages like python will…",{"id":99,"by":11,"text":100},"27877604","OP here. Yes! we use Sqlite under the hood with modified C extensions, so it can be as fast as it can be. (Just try not to touch JS code for this) However, I have recently…",{"value":102,"source":83,"evidence":103},"C",[104,105],{"id":96,"by":11,"text":97},{"id":99,"by":11,"text":100},{"value":107,"source":83,"evidence":108},"Electron",[109],{"id":110,"by":11,"text":111},"27878542","My use case is a bit more complex. I have to compile it with an electron app and etc. I filed a few GitHub issues with them as well. This has great potential. The dialect is much…",{"value":113,"source":83,"evidence":114},"JavaScript",[115],{"id":99,"by":11,"text":100},"unknown","beta",{"praise":119,"objections":142,"requests":163,"questions":175,"alternatives":190},[120,131,138],{"value":121,"source":64,"evidence":122},"Fast on large files",[123,127],{"id":124,"by":125,"text":126},"27872384","cube2222","Great to see SQL being used more for such use cases! (I really like SQL) This looks nice, especially with the charting! It's good to know though that there are a few similar open…",{"id":128,"by":129,"text":130},"27876392","qorrect","I use the command line all day, it definitely has its place. But trying to manipulate large CSV files has never felt like a good fit for the CLI. I'm already using and enjoying…",{"value":132,"source":64,"evidence":133},"Useful GUI for SQL on CSV for analysts",[134,135],{"id":67,"by":68,"text":69},{"id":136,"by":68,"text":137},"27893013","Yeah, I don't know whether this is new or not, but there's a lot of positions that are supported by their data teams to be able to do SQL queries for their business needs but not…",{"value":139,"source":64,"evidence":140},"Join support for CSVs",[141],{"id":59,"by":60,"text":61},[143,155,159],{"value":144,"source":64,"evidence":145},"Command line tools already do this with SQLite",[146,147,151],{"id":55,"by":56,"text":57},{"id":148,"by":149,"text":150},"27872466","EMM_386","This open source tool on Github allows you to quickly import a CSV and write SQL queries against it. The CSV is imported into a SQLite database. It also does a lot more, highly…",{"id":152,"by":153,"text":154},"27874379","gigatexal","I do this already with sqlite3",{"value":156,"source":64,"evidence":157},"Comparison with loading CSV into SQL is misleading",[158],{"id":55,"by":56,"text":57},{"value":160,"source":64,"evidence":161},"Target audience unclear",[162],{"id":71,"by":72,"text":73},[164,171],{"value":165,"source":64,"evidence":166},"Open source the project",[167],{"id":168,"by":169,"text":170},"27878508","dantetheinferno","Why isn't this opensourced?",{"value":172,"source":83,"evidence":173},"Support DuckDB-style dialect features like WITH RECURSIVE",[174],{"id":96,"by":11,"text":97},[176,180],{"value":177,"source":64,"evidence":178},"Why is it not open source",[179],{"id":168,"by":169,"text":170},{"value":181,"source":64,"evidence":182},"Whether it is faster than Python or pandas on large files",[183,187],{"id":184,"by":185,"text":186},"27875656","banana_giraffe","> A python script loading a 1GB file is likely to take more than 1 minutes on any machine. Since when? I just tested it, and I can process a 1gb CSV file on my machine using the…",{"id":188,"by":11,"text":189},"27875977","This is OP, but that is amazing. I tried reading 1GB CSV and add all the rows into Sqlite, and it definitely takes minutes. This happens to both JS and python. Constructing a…",[191,201,209,217,221,228],{"value":192,"source":64,"evidence":193},"DuckDB",[194,198],{"id":195,"by":196,"text":197},"27876370","RobinL","Checkout the R bindings for DuckDB[0]. You should find that it does the same thing (i.e. run SQL against a dataframe\u002Ffile on disk) much faster for many SQL operations. [0]…",{"id":199,"by":196,"text":200},"27876353","DuckDB is a newer but much faster alternative that lets you run SQL on csv files (and also directories of parquet files etc.) It's so fast\u002Fgood that I've actually been using it…",{"value":202,"source":64,"evidence":203},"q",[204,205],{"id":124,"by":125,"text":126},{"id":206,"by":207,"text":208},"27872359","_1tan","Is this similar to https:\u002F\u002Fgithub.com\u002Fharelba\u002Fq ?",{"value":210,"source":64,"evidence":211},"Datasette",[212,213],{"id":124,"by":125,"text":126},{"id":214,"by":215,"text":216},"27872938","hprotagonist","The datasette author offers this tool for conversion: https:\u002F\u002Fgithub.com\u002Fsimonw\u002Fcsvs-to-sqlite",{"value":218,"source":64,"evidence":219},"sqlitebrowser",[220],{"id":148,"by":149,"text":150},{"value":222,"source":64,"evidence":223},"pandasql",[224],{"id":225,"by":226,"text":227},"27876176","beckingz","Cool tool! Panda SQL is great for doing SQL on pandas dataframes. https:\u002F\u002Fpypi.org\u002Fproject\u002Fpandasql\u002F",{"value":229,"source":64,"evidence":230},"TextQL",[231],{"id":124,"by":125,"text":126},{"comments":233,"answered":234,"key":235},10,2,[236,237,238],{"id":96,"text":97},{"id":110,"text":111},{"id":86,"text":87},{"commenters":240,"max_depth":32,"maker_replies":233,"first_comment_mins":241,"lifespan_hours":242},42,214,52,[244,248,249,252,255,257,260,263,266,268,270,272,274,276,277,279,282,283,284,286,289,291,292,295,297,302,305,306,309,312,314,316,317,319,322,324,326,329,331,332,335,339,343,345,350,353,357,361,365,368,372,376,378,382,385,387,390,392,395,398,401],[245,39,39,246,247,39],"27874857","other\u002Fneutral\u002Fnone","gaha",[195,39,39,246,196,39],[250,39,39,246,251,39],"27878401","legg0myegg0",[253,39,39,246,254,39],"27876483","nojito",[124,34,41,256,125,39],"project\u002Fmixed\u002Fperformance",[258,34,39,246,259,39],"27872498","simonw",[261,34,39,246,262,39],"27874457","jolmg",[264,34,39,265,153,39],"27879158","other\u002Fpositive\u002Fdocs",[96,34,39,267,11,34],"maker\u002Fneutral\u002Fperformance",[269,34,39,246,251,39],"27878396",[110,34,39,271,11,34],"maker\u002Fneutral\u002Fexecution",[55,234,34,273,56,39],"project\u002Fcritical\u002Fidea",[86,234,39,275,11,34],"maker\u002Fneutral\u002Fidea",[71,41,34,273,72,39],[67,41,32,278,68,39],"project\u002Fpositive\u002Fidea",[280,41,39,281,72,39],"27887247","commenter\u002Fneutral\u002Fnone",[136,41,32,278,68,39],[148,35,39,246,149,39],[285,35,39,246,56,39],"27872772",[287,32,39,246,288,39],"27872556","draegtun",[184,33,39,290,185,39],"commenter\u002Fneutral\u002Fperformance",[188,33,39,267,11,34],[293,33,39,290,294,39],"27876248","nimrody",[296,33,39,290,11,34],"27877576",[298,299,41,300,301,39],"27872566",7,"project\u002Fneutral\u002Fidea","sgt",[303,299,39,281,304,39],"27872684","simlan",[99,299,39,267,11,34],[307,299,41,300,308,39],"27872748","SKILNER",[310,299,39,311,72,39],"27874876","commenter\u002Fcritical\u002Fidea",[313,299,39,281,308,39],"27878162",[315,299,39,281,72,39],"27887270",[89,299,39,275,11,34],[128,299,32,318,129,39],"project\u002Fpositive\u002Fexecution",[320,299,39,321,11,34],"27877590","maker\u002Fneutral\u002Fnone",[214,323,39,246,215,39],8,[325,323,39,246,259,39],"27873338",[327,323,39,246,328,39],"27874521","kingosticks",[225,330,39,246,226,39],9,[199,330,39,246,196,39],[333,233,32,278,334,39],"27879518","marto1",[336,337,39,246,338,39],"27873640",11,"hodgesrm",[340,337,41,341,342,39],"27875938","project\u002Fneutral\u002Fexecution","btown",[344,337,39,271,11,34],"27877687",[346,347,34,348,349,39],"27872279",12,"maker\u002Fcritical\u002Fdocs","aphrax",[351,347,39,352,11,34],"27874719","maker\u002Fneutral\u002Fdocs",[354,355,39,246,356,39],"27872891",13,"eatonphil",[358,359,39,246,360,39],"27874028",14,"jdnier",[362,363,34,273,364,39],"27875287",15,"kmerroll",[366,36,39,246,367,39],"27873522","cube00",[369,370,39,246,371,39],"27877143",17,"robbiejs",[373,374,39,246,375,39],"27874427",18,"adam_gyroscope",[152,377,39,246,153,39],19,[379,380,39,246,381,39],"27874663",20,"refactor_master",[383,380,39,246,384,39],"27874766","zo1",[206,386,39,246,207,39],21,[388,18,39,246,389,39],"27873941","chrisweekly",[59,391,32,278,60,39],23,[393,391,39,246,394,39],"27876431","hermitcrab",[168,396,41,397,169,39],24,"project\u002Fneutral\u002Fnone",[399,396,41,397,400,39],"27880396","code_scrapping",[78,402,41,403,79,39],25,"project\u002Fneutral\u002Fpricing",[405,408],{"id":406,"t":407,"points":355,"comments":34},"27575352",1624238189,{"id":4,"t":12,"points":14,"comments":15},[410,415,419,422,427,432],{"id":411,"name":412,"points":413,"category":414},"27871790","CheckYourList",282,"productivity",{"id":416,"name":417,"points":418,"category":414},"27893303","Sioyek",272,{"id":420,"name":421,"points":241,"category":414},"27853855","Deprocrastination",{"id":423,"name":424,"points":425,"category":426},"27866289","Gorse",212,"ai",{"id":428,"name":429,"points":430,"category":431},"27882355","Should I Get a House?",203,"finance_crypto",{"id":433,"name":434,"points":435,"category":436},"27894903","WrittenRealms",200,"games",false,"v2",{"id":440,"name":5,"line":6,"ai":437,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":441,"top":18,"kw":442,"maker":11,"domain":10,"tier":19,"attempts":234,"strip":443,"url":9,"mix":444,"praise":121,"objection":144,"sub":21},27871574,-0.03,[24,25,26,27,28],"0003000103050000303005000053100100000533",[32,33,34,35],1793991030733]