Vaex
Data
dataframe library
SHOW HN2018
126points
32comments
#333of 2,155 in 2018
Top 16%of 2018 by points
How HN took it
MixedNo clear lean
5 positive, 8 neutral or asking, 0 mixed, 4 critical, out of 17 comments about the project. Thread score +0.15.
Performance: 0 for, 3 againstIdea: 2 for, 0 againstDesign: 1 for, 1 againstExecution: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Fast performance on billion-row datasets
From 2 comments
Such phenomenal work. BTW, for anyone on a Windows machine, getting this to work is very trivial. There is a unix only library for locking files (fcntl) which prevents it from…
JPKab on HNDask and vaex are not 'competing', they are orthogonal. Vaex could use dask to do the computations, but when this part of vaex was built, dask didn't exist. I recently tried…
maartenbreddels on HN
Pandas-like API eases transition
From 1 comment
That is exactly the sweet spot for vaex, and with a familiar DataFrame API (read pandas like) the transition does not hurt so much. It may sound cool to set up a cluster, but in…
maartenbreddels on HN
Useful for exploratory analysis of large datasets
From 1 comment
Nice work. This looks like it could add a lot of value to a DS's toolbox. Exploratory data analysis of large (but not huge) datasets have always been a slow and frustrating…
wenc on HN
Objections
Benchmark claims depend on data being in RAM, I/O not counted
From 2 comments
Desktop PCs have about 35 GB/s of memory bandwidth and can do compute at ~200 Gflops, so this is just ~10% of peak bw and leaves you a budget of 200 flops computation per float…
fulafel on HNMy thought was on first access or out-of-memory sizes. This would always be bound by I/O which means it is kind of a meaningless statistic. Don't get me wrong, this seems like a…
aw3c2 on HN
Arrow conversion is not zero-copy as claimed
From 2 comments
Note: Vaex has its own memory model. If you input Arrow, it converts to the Vaex data representation. Details here:…
wesm on HNI'm looking at the code I linked, and you are serializing in the general case, it is not zero copy. Unpacking a bitmap is not free.
wesm on HN
Requests
Comparison with Dask and pandas performance
From 2 comments
First of all, great to see more powertools to choose from for my ds workflow! However, I am suprised to see no mention of Dask in the article. How do these libraries compare?
themmes on HNIs this cold data? Or already in RAM? What about a billion rows that are not in RAM yet? How does it compare to plain numpy or pandas?
aw3c2 on HN
Windows support for file locking
From 1 comment
Such phenomenal work. BTW, for anyone on a Windows machine, getting this to work is very trivial. There is a unix only library for locking files (fcntl) which prevents it from…
JPKab on HN
Support for Zarr or indexed Parquet
From 2 comments
It looks quite nice, and I will have to explore the performance comparisons with Dask more. I have recently started using Xarray for some projects, and really appreciate the…
rax on HNI'll need to have a closer look later, but would vaex fit in with somewhat indexed mapped files? E.g. parquet supports column indexes now:…
ah- on HN
Questions
How does it perform on cold data not in RAM
From 1 comment
Is this cold data? Or already in RAM? What about a billion rows that are not in RAM yet? How does it compare to plain numpy or pandas?
aw3c2 on HN
How it compares to Dask
From 1 comment
First of all, great to see more powertools to choose from for my ds workflow! However, I am suprised to see no mention of Dask in the article. How do these libraries compare?
themmes on HN
Alternatives named
Dask
From 1 comment
First of all, great to see more powertools to choose from for my ds workflow! However, I am suprised to see no mention of Dask in the article. How do these libraries compare?
themmes on HN
Datashader
From 1 comment
I have not looked into it, maybe datashader can do this, which is a package purely focussing on viz, while vaex is more allround (although there is overlap). If you think vaex…
maartenbreddels on HN
Gephi
From 1 comment
Gephi?
blattimwind on HN
Pandas
From 1 comment
Is this cold data? Or already in RAM? What about a billion rows that are not in RAM yet? How does it compare to plain numpy or pandas?
aw3c2 on HN