sqlite-zstd
Data
sqlite compression extension
SHOW HN2022
How HN took it
10 positive, 22 neutral or asking, 0 mixed, 6 critical, out of 38 comments about the project. Thread score +0.06.
Idea: 4 for, 4 againstDesign: 3 for, 0 againstExecution: 0 for, 2 againstPerformance: 1 for, 0 againstPrivacy: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Neat approach; per-column compression and custom partitioning are useful
From 1 comment
This is really neat. I like how the compression can be applied to specific columns rather than whole rows - or whole pages. I could imagine storing JSON from an HTTP request here…
wmanley on HN
Useful for compressing GitHub statistics and JSON backups
From 2 comments
Oh, interesting. I scrape GitHub's APIs to generate user and repositories statistics data, then compress it with ZStandard to allow the databases to fit within GitHub's file size…
andrewmcwatters on HNTried this on my twitter database's `json` column (ie the JSON response from the API fetching a singular tweet) with a `dict_chooser` of '%Y-%W' to bundle weeks together and it…
zimpenfish on HN
Objections
Doing it at the filesystem level is an alternative
From 2 comments
I don't want to diss on the effort, I have an honest question: wouldn't this something that would be more appropriate to be done at the filesystem level?
avereveard on HNNo. If you did that, SQLite would have to (logically) decompress about your entire database on every query. For example, how would it read the database schema to determine…
Someone on HN
Existing compression VFS extensions are the more common approach
From 1 comment
I am happy for the author, but want to warn the credulous that this is a very strange way to approach compression with SQLite. The much more common way is to use SQLite vfs:…
d1l on HN
Title is clickbait-ish and approach may risk data loss
From 2 comments
I am happy for the author, but want to warn the credulous that this is a very strange way to approach compression with SQLite. The much more common way is to use SQLite vfs:…
d1l on HNTaking an amateurish or fancy or experimental approach to compressing a sqlite database is a great way to end up with data loss. Doesn't make it a worthless or "bad" project but…
kevingadd on HN
Requests
Support for retroactively compressing existing columns
From 1 comment
This looks great. How would I apply this retroactively? I.e. tell SQLite to compress column x in an existing database, say, of type TEXT?
leobg on HN
Django integration
From 1 comment
I love the idea. Can I use it with Django projects somehow?
hda111 on HN
Comparison with Brotli and compression VFS
From 2 comments
How does this compare with modern compression algorithms like Brotli that do context modeling, etc? I've found that they manage to aggressively compress types of data you…
kevingadd on HNI did actually previously write a VFS, though it did something else entirely: https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/ You're right that a…
phiresky on HN
Questions
What does transparent mean here
From 1 comment
What does "transparent" mean in this context?
mansoor_ on HN
Whether performance can improve from compression
From 1 comment
This mentions something I've been wondering about: >Depending on the data, this can reduce the size of the database by 80% while keeping performance mostly the same (or even…
logicallee on HN
Alternatives named
Filesystem-level compression (btrfs, ZFS)
From 2 comments
> Btrfs also supports transparent file compression. There are three algorithms available: ZLIB, LZO and ZSTD (since v4.14). Basically, compression is on a file by file basis. You…
ciupicri on HNI'm using sqlite on zfs and btrfs, with transparent compression supported by the file system, the query speed is very good and compression rate also very good. (database below…
aabbcc1241 on HN
SQLite compression VFS extensions (sqlite_zstd_vfs, ZIPVFS)
From 3 comments
I am happy for the author, but want to warn the credulous that this is a very strange way to approach compression with SQLite. The much more common way is to use SQLite vfs:…
d1l on HNI did actually previously write a VFS, though it did something else entirely: https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/ You're right that a…
phiresky on HNWhy not read the sqlite teams own explanation: https://www.sqlite.org/zipvfs/doc/trunk/www/howitworks.wiki
d1l on HN