s3sqlite

This is really neat. It looks like it's using a similar trick to https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/ from last year - querying SQLite…

This is cool. It would be interesting to see some stats like how many separate requests are needed for a single SELECT statement, since S3 charges per GET, even on a small range,…

This is cool, I'll probably end up using this (we store sqlite snapshots in S3). It'd be very handy if this was packaged as a CLI that worked like `sqlite3`, so you could get a…

This would be quite convenient when dealing with MBtiles files hosted on S3 or similar. There are efforts such as PMTiles[1] that let us use HTTP Range query to retrieve tiles…

I feel like people are pushing SQLite beyond its limits with this stuff. If you're going to do something like this, wouldn't just a normal client/server option like PostgreSQL be…

I admire people for their inventiveness, however surely attempting to square-peg-round-hole something like this into AWS S3 is only going to end up biting them in the backside ?…

If I need to query the db repeatedly (N times per second), isn't it more efficient (low latency and aws cost) to just download the db and run queries locally?

This is cool, I'll probably end up using this (we store sqlite snapshots in S3). It'd be very handy if this was packaged as a CLI that worked like `sqlite3`, so you could get a…

There have been many of these custom vfs implementations. Is there a benchmark on performance between them?

This is cool. It would be interesting to see some stats like how many separate requests are needed for a single SELECT statement, since S3 charges per GET, even on a small range,…

i wish s3 supported multiple ranges in the range header. I've done this sort of access with a custom columnar format that stored a set of columns in a single file it was a…

Sorry for the stupid question: does this work with MinIO?

Yes, I actually used MinIO to test it (see the `test.py` script)

Do you have any insight into whether the cache will alter consistency in the face of concurrent writes from multiple locations in a way that SQLite can't handle? I imagine this…

This VFS is only for reading databases stored in S3, not writing. The README gives some alternatives for other implementations which allow writing.

Is there a benefit over the existing S3 VFS sqlite3-s3vfs,which also uses APSW?

Author here! I would say the main advantage is that this just needs uploading a single file to S3. `sqlite-s3vfs` will split the database in pages and upload the pages separately…

This is nice, but before you go to the trouble of using it, consider just putting your CSV, JSON, or parquet data file in S3 and query it with S3 Select…

A comparison to S3 Select would make sense since S3 Select lets you leverage RANGE, leverage compact formats like compressed Parquet, and would presumably have a similar…

Nifty, reminds me of https://github.com/backtrace-labs/verneuil or possibly https://litestream.io/

I read of someone using AWS EFS for sqlite, which has nfs functionality, if I recall correctly. However I like the idea of using a local store with litestream to S3 and updating…

Aha! This is done in https://github.com/uktrade/mbtiles-s3-server, but using an alternative library to query SQLite on S3, https://github.com/michalc/sqlite-s3-query (Full…

> attempting to square-peg-round-hole something like this into AWS S3 I don't think it's like that. AWS already offers services like S3 Select[0] or Athena[1] that do something…