[{"data":1,"prerenderedAt":309},["ShallowReactive",2],{"project-18670786":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":44,"post_text":45,"what":46,"said":92,"maker_in_thread":188,"shape":198,"grid":202,"attempts":271,"same_week":272,"maker_shows":34,"maker_hidden":301,"reader_version":302,"card":303},"18670786","Vaex","Out of Core Dataframes for Python and Fast Visualization","title","Show HN: Vaex - Out of Core Dataframes for Python and Fast Visualization","https:\u002F\u002Fmedium.com\u002Fvaex\u002Fvaex-out-of-core-dataframes-for-python-and-fast-visualization-12c102db044a","medium.com","maartenbreddels",1544690130,2018,126,32,333,2155,16,"notable","data","dataframe library","library",[24,25,26,27],"out of core dataframe","python dataframe","large dataset visualization","memory mapping","mixed",0.155,{"positive":31,"neutral":32,"mixed":33,"critical":34},5,8,0,4,17,{"idea":37,"execution":39,"design":41,"performance":42},{"positive":38,"critical":33},2,{"positive":40,"critical":33},1,{"positive":40,"critical":40},{"positive":33,"critical":43},3,[43,33,31,33,31,33,33,43,33,43,43,43,33,43,33,31,40,33,33,33,31,33,40,40,33,43,40,33,31,33,43,33],null,{"problem":47,"audience":57,"pricing":45,"motivation":45,"time_to_build":45,"stack":65,"openness":91,"stage":91,"team_size":91},{"value":48,"source":49,"evidence":50},"Exploring and computing statistics on datasets too large for memory with pandas-like tools","maker",[51,54],{"id":52,"by":11,"text":53},"18679416","Dask and vaex are not 'competing', they are orthogonal. Vaex could use dask to do the computations, but when this part of vaex was built, dask didn't exist. I recently tried…",{"id":55,"by":11,"text":56},"18682704","That is exactly the sweet spot for vaex, and with a familiar DataFrame API (read pandas like) the transition does not hurt so much. It may sound cool to set up a cluster, but in…",{"value":58,"source":49,"evidence":59},"Data scientists working with large single-machine datasets",[60,61],{"id":55,"by":11,"text":56},{"id":62,"by":63,"text":64},"18682004","wenc","Nice work. This looks like it could add a lot of value to a DS's toolbox. Exploratory data analysis of large (but not huge) datasets have always been a slow and frustrating…",[66,74,84],{"value":67,"source":68,"evidence":69},"Python","commenter",[70],{"id":71,"by":72,"text":73},"18679912","colobas","Does it have python3 support? Tried installing it on a python3.7 environment and it failed EDIT: I then tried a python3.6 environment and it worked. I guess it answers my question",{"value":75,"source":49,"evidence":76},"Apache Arrow",[77,81],{"id":78,"by":79,"text":80},"18680961","ah-","Great to see that you're supporting Apache Arrow! That makes it so much easier to gradually switch over.",{"id":82,"by":11,"text":83},"18682763","That is not correct, I just refer to the buffers\u002Fmemory, 0 copying going on. Vaex is not really opinionated about the memory model actually. The only exception is the bitmasks…",{"value":85,"source":68,"evidence":86},"HDF5",[87],{"id":88,"by":89,"text":90},"18683060","sevensor","Uses HDF5, which itself is a great file format, well suited for big tables of numbers. Good for similar reasons as SQLite3, but for different applications. Not a relational…","unknown",{"praise":93,"objections":110,"requests":132,"questions":157,"alternatives":166},[94,102,106],{"value":95,"source":68,"evidence":96},"Fast performance on billion-row datasets",[97,101],{"id":98,"by":99,"text":100},"18682827","JPKab","Such phenomenal work. BTW, for anyone on a Windows machine, getting this to work is very trivial. There is a unix only library for locking files (fcntl) which prevents it from…",{"id":52,"by":11,"text":53},{"value":103,"source":49,"evidence":104},"Pandas-like API eases transition",[105],{"id":55,"by":11,"text":56},{"value":107,"source":68,"evidence":108},"Useful for exploratory analysis of large datasets",[109],{"id":62,"by":63,"text":64},[111,122],{"value":112,"source":68,"evidence":113},"Benchmark claims depend on data being in RAM, I\u002FO not counted",[114,118],{"id":115,"by":116,"text":117},"18681279","fulafel","Desktop PCs have about 35 GB\u002Fs of memory bandwidth and can do compute at ~200 Gflops, so this is just ~10% of peak bw and leaves you a budget of 200 flops computation per float…",{"id":119,"by":120,"text":121},"18691068","aw3c2","My thought was on first access or out-of-memory sizes. This would always be bound by I\u002FO which means it is kind of a meaningless statistic. Don't get me wrong, this seems like a…",{"value":123,"source":68,"evidence":124},"Arrow conversion is not zero-copy as claimed",[125,129],{"id":126,"by":127,"text":128},"18682036","wesm","Note: Vaex has its own memory model. If you input Arrow, it converts to the Vaex data representation. Details here:…",{"id":130,"by":127,"text":131},"18684065","I'm looking at the code I linked, and you are serializing in the general case, it is not zero copy. Unpacking a bitmap is not free.",[133,143,147],{"value":134,"source":68,"evidence":135},"Comparison with Dask and pandas performance",[136,140],{"id":137,"by":138,"text":139},"18679213","themmes","First of all, great to see more powertools to choose from for my ds workflow! However, I am suprised to see no mention of Dask in the article. How do these libraries compare?",{"id":141,"by":120,"text":142},"18691072","Is this cold data? Or already in RAM? What about a billion rows that are not in RAM yet? How does it compare to plain numpy or pandas?",{"value":144,"source":68,"evidence":145},"Windows support for file locking",[146],{"id":98,"by":99,"text":100},{"value":148,"source":68,"evidence":149},"Support for Zarr or indexed Parquet",[150,154],{"id":151,"by":152,"text":153},"18680327","rax","It looks quite nice, and I will have to explore the performance comparisons with Dask more. I have recently started using Xarray for some projects, and really appreciate the…",{"id":155,"by":79,"text":156},"18681028","I'll need to have a closer look later, but would vaex fit in with somewhat indexed mapped files? E.g. parquet supports column indexes now:…",[158,162],{"value":159,"source":68,"evidence":160},"How does it perform on cold data not in RAM",[161],{"id":141,"by":120,"text":142},{"value":163,"source":68,"evidence":164},"How it compares to Dask",[165],{"id":137,"by":138,"text":139},[167,171,177,184],{"value":168,"source":68,"evidence":169},"Dask",[170],{"id":137,"by":138,"text":139},{"value":172,"source":49,"evidence":173},"Datashader",[174],{"id":175,"by":11,"text":176},"18679584","I have not looked into it, maybe datashader can do this, which is a package purely focussing on viz, while vaex is more allround (although there is overlap). If you think vaex…",{"value":178,"source":68,"evidence":179},"Gephi",[180],{"id":181,"by":182,"text":183},"18682241","blattimwind","Gephi?",{"value":185,"source":68,"evidence":186},"Pandas",[187],{"id":141,"by":120,"text":142},{"comments":189,"answered":31,"key":190},13,[191,192,195],{"id":52,"text":53},{"id":193,"text":194},"18684360","thanks!",{"id":196,"text":197},"18683025","There is an issue open for this: https:\u002F\u002Fgithub.com\u002Fvaexio\u002Fvaex\u002Fissues\u002F93 It should have been fixed, some more detailed report (version numbers installed) would be good to know.",{"commenters":199,"max_depth":31,"maker_replies":189,"first_comment_mins":200,"lifespan_hours":201},15,1396,96,[203,205,207,210,212,214,216,218,219,222,223,225,228,229,232,234,236,238,240,242,245,247,248,252,253,255,256,257,260,263,266,268],[137,33,43,204,138,33],"project\u002Fneutral\u002Fidea",[52,33,33,206,11,40],"project\u002Fpositive\u002Fidea",[208,33,31,209,138,33],"18683096","maker\u002Fpositive\u002Fnone",[193,33,33,211,11,40],"maker\u002Fneutral\u002Fnone",[98,40,31,213,99,33],"project\u002Fpositive\u002Fexecution",[196,40,33,215,11,40],"project\u002Fneutral\u002Fexecution",[217,40,33,209,11,40],"18683027",[151,38,43,204,152,33],[220,38,33,221,11,40],"18680521","project\u002Fneutral\u002Fperformance",[155,38,43,204,79,33],[88,43,43,224,89,33],"project\u002Fneutral\u002Fdesign",[226,34,43,204,227,33],"18679441","angelmass",[175,34,33,204,11,40],[230,34,43,204,231,33],"18680867","bayesian_horse",[181,34,33,233,182,33],"other\u002Fneutral\u002Fnone",[78,31,31,235,79,33],"project\u002Fpositive\u002Fdesign",[126,31,40,237,127,33],"maker\u002Fcritical\u002Fdesign",[82,31,33,239,11,40],"maker\u002Fneutral\u002Fdesign",[130,31,33,241,127,33],"commenter\u002Fcritical\u002Fdesign",[243,31,33,244,11,40],"18687651","commenter\u002Fneutral\u002Fnone",[62,246,31,206,63,33],6,[55,246,33,206,11,40],[249,250,40,251,120,33],"18681249",7,"project\u002Fcritical\u002Fperformance",[115,250,40,251,116,33],[254,250,33,221,11,40],"18681975",[141,250,43,221,120,33],[119,250,40,251,120,33],[258,250,33,259,11,40],"18697666","project\u002Fpositive\u002Fperformance",[261,32,31,206,262,33],"18680482","stestagg",[264,32,33,265,11,40],"18680529","maker\u002Fpositive\u002Fidea",[71,267,43,215,72,33],9,[269,267,33,270,11,40],"18680272","maker\u002Fneutral\u002Fexecution",[],[273,278,282,287,291,296],{"id":274,"name":275,"points":276,"category":277},"18697601","remove.bg",640,"creative",{"id":279,"name":280,"points":281,"category":277},"18701772","Fancy fonts (Beautiful Dingbats)",247,{"id":283,"name":284,"points":285,"category":286},"18673928","Minimal Google Analytics Snippet",171,"web_saas",{"id":288,"name":289,"points":290,"category":286},"18710094","DPAGE",161,{"id":292,"name":293,"points":294,"category":295},"18705661","Goldpinger",158,"infrastructure",{"id":297,"name":298,"points":299,"category":300},"18704436","Bubblin",138,"productivity",false,"v2",{"id":304,"name":5,"line":6,"ai":301,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":305,"top":18,"kw":306,"maker":11,"domain":10,"tier":19,"attempts":33,"strip":307,"url":9,"mix":308,"praise":95,"objection":112,"sub":21},18670786,0.16,[24,25,26,27],"30505003033303051000501103105030",[31,32,33,34],1793990983219]