[{"data":1,"prerenderedAt":286},["ShallowReactive",2],{"project-21682331":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":41,"post_text":43,"what":44,"said":91,"maker_in_thread":173,"shape":182,"grid":186,"attempts":249,"same_week":250,"maker_shows":175,"maker_hidden":278,"reader_version":279,"card":280},"21682331","Vaex","How to analyse 100 GB of data on your laptop with Python","title","Show HN: How to analyse 100 GB of data on your laptop with Python","https:\u002F\u002Ftowardsdatascience.com\u002Fhow-to-analyse-100s-of-gbs-of-data-on-your-laptop-with-python-f83363dda94","towardsdatascience.com","maartenbreddels",1575295423,2019,264,26,126,2195,6,"legend","data","dataframe library","library",[24,25,26,27],"out of memory dataframe","python big data","vaex","lazy evaluation dataframe","curious",0.129,{"positive":31,"neutral":32,"mixed":33,"critical":34},5,11,0,2,18,{"idea":37,"performance":39,"execution":40},{"positive":33,"critical":38},1,{"positive":38,"critical":38},{"positive":34,"critical":33},[38,42,42,33,42,33,31,31,38,42,42,33,42,42,33,31,33,42,33,31,42,33,42,33,31,42],3,null,{"problem":45,"audience":52,"pricing":43,"motivation":43,"time_to_build":56,"stack":63,"openness":90,"stage":90,"team_size":90},{"value":46,"source":47,"evidence":48},"Analysing datasets too large to fit in laptop memory with Python.","inferred",[49],{"id":50,"by":11,"text":51},"21683118","It is not similar to Dask, but similar to dask.dataframe. Dask.dataframe is built on top of Pandas, but that also means it inherits its issue, like memory usage, and performance.…",{"value":53,"source":47,"evidence":54},"Data scientists working with large datasets in Python.",[55],{"id":50,"by":11,"text":51},{"value":57,"source":58,"evidence":59},"Development started January 2014; maker says it took 50-80% of their time until about 2018.","maker",[60],{"id":61,"by":11,"text":62},"21685345","Thanks :) The first commit was Jan 2014, when it between 50-80% of my time until 2018 I think where I mostly developed it myself. After that, it is more difficult to say how much…",[64,70,76,80,86],{"value":65,"source":58,"evidence":66},"Python",[67],{"id":68,"by":11,"text":69},"21683356","No Python looping happening in Vaex :), otherwise, we wouldn't get this performance. We are also working on GraphQL support, with a Hasura-like API:…",{"value":71,"source":58,"evidence":72},"Apache Arrow",[73],{"id":74,"by":11,"text":75},"21684448","Indeed, numpy for numerical calculations. For strings, we have our own data structure based on Apache Arrow, but we plan\u002Fhope to move to Apache Arrow (in combination with numpy),…",{"value":77,"source":58,"evidence":78},"numpy",[79],{"id":74,"by":11,"text":75},{"value":81,"source":58,"evidence":82},"numba",[83],{"id":84,"by":11,"text":85},"21685270","Indeed, numba, Pythran or cupy can be really useful. In the example of the article, it is 'simple' vectorized math. But in general any function can be added in Vaex. Those that…",{"value":87,"source":58,"evidence":88},"Pythran",[89],{"id":84,"by":11,"text":85},"unknown",{"praise":92,"objections":114,"requests":122,"questions":138,"alternatives":147},[93,106],{"value":94,"source":95,"evidence":96},"Fast performance on large data compared with pandas-style workflows","commenter",[97,98,102],{"id":68,"by":11,"text":69},{"id":99,"by":100,"text":101},"21684747","mooneater","Ive been switching an reporting system that did analysis in postgres, to analysis in pandas (mostly business-stats type summaries). It feels like growing wings and a jetpack.…",{"id":103,"by":104,"text":105},"21683508","drblah","Ah, of cause. It makes sense you don't loop in Python. This all seems pretty interesting. I will give it a go.",{"value":107,"source":95,"evidence":108},"Pandas-like API that is easy to use",[109,110],{"id":50,"by":11,"text":51},{"id":111,"by":112,"text":113},"21683828","floki999","Dask’ s compatibility with Pandas makes it ideal. Very easy to use.",[115],{"value":116,"source":95,"evidence":117},"Analysis of the example mixes in an unrealistic interpretation of cash tips",[118],{"id":119,"by":120,"text":121},"21683820","darkstar999","> 99.97% of the passengers that pay by cash do not leave tips. Correction - 99.97% of cash tips don't go recorded. Why pay tax if they never saw it happen? Compare this to: >…",[123,134],{"value":124,"source":95,"evidence":125},"Benchmarks against Postgres and Spark",[126,130],{"id":127,"by":128,"text":129},"21682824","isoprophlex","How would this compare to ingesting this in a locally running rdbms, like postgres, and applying some judicious indexing? Or maybe a local spark cluster? I'm tempted to benchmark…",{"id":131,"by":132,"text":133},"21688250","kthejoker2","Think you'd want to compare with a columnar database since this is an analytic workload, not a transactional one. And a 4 node Spark cluster with Parquet or Arrow files and a…",{"value":135,"source":58,"evidence":136},"Support for full Dask integration",[137],{"id":50,"by":11,"text":51},[139],{"value":140,"source":95,"evidence":141},"How does it compare with Postgres, Spark, Dask or Xarray?",[142,143],{"id":127,"by":128,"text":129},{"id":144,"by":145,"text":146},"21682910","jononor","Vaex seems to be very similar to Dask and Xarray. Which one to choose?",[148,153,160,165,169],{"value":149,"source":95,"evidence":150},"Dask dataframe",[151,152],{"id":144,"by":145,"text":146},{"id":111,"by":112,"text":113},{"value":154,"source":95,"evidence":155},"Postgres",[156,157],{"id":127,"by":128,"text":129},{"id":158,"by":104,"text":159},"21683259","I am currently doing an internship as part of my masters degree where I am analyzing ~30 GB of data. I'm using Postgres + Python and it is working quite well, even on my 2014…",{"value":161,"source":95,"evidence":162},"Spark",[163,164],{"id":127,"by":128,"text":129},{"id":131,"by":132,"text":133},{"value":166,"source":95,"evidence":167},"Xarray",[168],{"id":144,"by":145,"text":146},{"value":170,"source":58,"evidence":171},"Pandas",[172],{"id":50,"by":11,"text":51},{"comments":174,"answered":175,"key":176},7,4,[177,180,181],{"id":178,"text":179},"21683163","I've never benchmarked against postgres, but would be interested about the results. I once tried monetdb, and it was orders of magnitude slower for simple calculations, so I…",{"id":68,"text":69},{"id":74,"text":75},{"commenters":183,"max_depth":31,"maker_replies":174,"first_comment_mins":184,"lifespan_hours":185},19,53,200,[187,189,193,195,196,198,200,202,203,205,208,211,213,216,219,220,224,228,230,231,233,236,238,241,242,246],[119,33,38,188,120,33],"project\u002Fcritical\u002Fidea",[190,33,42,191,192,33],"21684436","maker\u002Fneutral\u002Fnone","jovan31",[127,38,42,194,128,33],"project\u002Fneutral\u002Fperformance",[178,38,33,194,11,38],[158,38,42,197,104,33],"project\u002Fneutral\u002Fnone",[68,38,33,199,11,38],"project\u002Fpositive\u002Fperformance",[103,38,31,201,104,33],"project\u002Fpositive\u002Fnone",[99,38,31,199,100,33],[131,38,38,204,132,33],"project\u002Fcritical\u002Fperformance",[206,34,42,194,207,33],"21683866","Aardwolf",[209,34,42,194,210,33],"21683919","jofer",[74,34,33,212,11,38],"project\u002Fneutral\u002Fexecution",[214,34,42,212,215,33],"21684822","uoaei",[217,34,42,194,218,33],"21685199","Akababa",[84,34,33,194,11,38],[221,42,31,222,223,33],"21736218","maker\u002Fpositive\u002Fnone","Jackie1122",[225,175,33,226,227,33],"21709344","other\u002Fneutral\u002Fnone","deepakkhealani1",[144,31,42,229,145,33],"project\u002Fneutral\u002Fidea",[50,31,33,229,11,38],[111,18,31,232,112,33],"project\u002Fpositive\u002Fexecution",[234,174,42,191,235,33],"21684366","slowenough",[237,174,33,191,11,38],"21684435",[239,174,42,191,240,33],"21684527","make3",[61,174,33,191,11,38],[243,174,31,244,245,33],"21757213","maker\u002Fpositive\u002Fexecution","Breza",[247,174,42,191,248,33],"21686029","Rotten194",[],[251,256,261,265,270,274],{"id":252,"name":253,"points":254,"category":255},"21676933","ChessRoots",295,"games",{"id":257,"name":258,"points":259,"category":260},"21707285","Parrot.vc",263,"ai",{"id":262,"name":263,"points":264,"category":255},"21696194","Adventurekeep",182,{"id":266,"name":267,"points":268,"category":269},"21670239","Dot-to-ASCII",152,"dev_tools",{"id":271,"name":272,"points":273,"category":255},"21701124","googol",147,{"id":275,"name":276,"points":277,"category":269},"21656864","Zecoda",140,false,"v2",{"id":281,"name":5,"line":6,"ai":278,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":282,"top":18,"kw":283,"maker":11,"domain":10,"tier":19,"attempts":33,"strip":284,"url":9,"mix":285,"praise":94,"objection":116,"sub":21},21682331,0.13,[24,25,26,27],"13303055133033050305303053",[31,32,33,34],1794036583583]