warc-parquet
Data
web archive conversion
SHOW HN2022
How HN took it
5 positive, 3 neutral or asking, 2 mixed, 1 critical, out of 11 comments about the project. Thread score +0.26.
Performance: 3 for, 1 againstIdea: 1 for, 0 againstDesign: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
DuckDB is much faster than SQLite for analytical queries on columnar data
From 3 comments
Good question! As a disclaimer, I work for DuckDB Labs. There are 2 big benefits to working with Parquet files in DuckDB, and both relate to speed! DuckDB can query parquet right…
1egg0myegg0 on HNDuckDB has SQLite semantics but is natively built around columnar formats (parquet, in-memory Arrow) and strong types (including dates). It also supports very complex SQL. SQLite…
wenc on HNI've gone almost completely off Athena in favor of duckdb. Duckdb running with 12 vcpus on mid-size data (1tb) that's on ssd is routinely faster than Athena with the data on s3.…
henrydark on HN
Parquet's columnar format and compression are well suited to this data
From 2 comments
Another benefit that is often overlooked is that Parquet support various compression methods, right? SQLite's official support for compression is a commercial product, so…
pmeira on HNIt's a great question: fundamentally the Parquet format offers columnar orientation. With datasets like these, there's some research[0] indicating this is a preferable way of…
llambda on HN
Objections
DuckDB lacks distributed computation, so large datasets need a separate setup
From 2 comments
DuckDB user here. As far as I can tell, DuckDB doesn’t support distributed computation so you have to set that up yourself, whereas Athena is essentially Presto — it handles that…
wenc on HNIn the distributed computing world, the rule is you start to scale horizontally when your compute workload is too large to fit in the memory of a single machine. So it depends on…
wenc on HN
Requests
GeoParquet support in DuckDB
From 1 comment
Another benefit that is often overlooked is that Parquet support various compression methods, right? SQLite's official support for compression is a commercial product, so…
pmeira on HN
Questions
How does this compare with SQLite approaches?
From 1 comment
How does this compare with SQLite approaches shared recently?
wahnfrieden on HN
Alternatives named
SQLite with sqlite-parquet-vtable
From 1 comment
Well there's a virtual table extension to read parquet files in SQLite. I've not tried it myself. https://github.com/cldellow/sqlite-parquet-vtable
infogulch on HN
AWS Athena
From 2 comments
Do you see DuckDB as a possible replacement for AWS Athena? Where would Athena still be better than DuckDB + Parquet + Lambda?
arpinum on HNI've gone almost completely off Athena in favor of duckdb. Duckdb running with 12 vcpus on mid-size data (1tb) that's on ssd is routinely faster than Athena with the data on s3.…
henrydark on HN