[{"data":1,"prerenderedAt":407},["ShallowReactive",2],{"project-32828799":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":43,"post_text":45,"what":46,"said":84,"maker_in_thread":209,"shape":217,"grid":221,"attempts":367,"same_week":368,"maker_shows":34,"maker_hidden":399,"reader_version":400,"card":401},"32828799","s3sqlite","Query SQLite files stored in S3","title","Show HN: Query SQLite files stored in S3","https:\u002F\u002Fgithub.com\u002Flitements\u002Fs3sqlite","github.com","polyrand",1663095788,2022,160,61,377,2570,15,"notable","dev_tools","sqlite virtual file system","library",[24,25,26,27],"sqlite","s3","virtual file system","http range requests","curious",0.103,{"positive":31,"neutral":32,"mixed":33,"critical":34},4,14,0,2,20,{"idea":37,"performance":39,"design":40,"execution":42},{"positive":38,"critical":34},3,{"positive":33,"critical":33},{"positive":41,"critical":33},1,{"positive":33,"critical":33},[44,38,33,33,33,33,33,44,33,33,33,33,38,38,33,44,41,33,44,41,33,33,33,33,38,33,38,33,33,33,33,38,38,33,33,33,38,33,38,38],5,null,{"problem":47,"audience":45,"pricing":45,"motivation":45,"time_to_build":45,"stack":58,"openness":82,"stage":83,"team_size":83},{"value":48,"source":49,"evidence":50},"Querying SQLite databases stored in S3 without downloading the whole file","inferred",[51,55],{"id":52,"by":53,"text":54},"32829495","simonw","This is really neat. It looks like it's using a similar trick to https:\u002F\u002Fphiresky.github.io\u002Fblog\u002F2021\u002Fhosting-sqlite-databases-on-github-pages\u002F from last year - querying SQLite…",{"id":56,"by":53,"text":57},"32830018","If you did that, wouldn't you have to fetch the entire SQLite database file before you could perform any queries? With this you don't need to do that - the software magically…",[59,68,74],{"value":60,"source":49,"evidence":61},"Python",[62,65],{"id":63,"by":11,"text":64},"32830355","Author here! This is something I want to look into. One of the reasons to use s3fs is that the underlying `fsspec` classes already handle caching. S3File (from s3fs) inherits…",{"id":66,"by":11,"text":67},"32829729","Author here! I would say the main advantage is that this just needs uploading a single file to S3. `sqlite-s3vfs` will split the database in pages and upload the pages separately…",{"value":69,"source":70,"evidence":71},"s3fs","maker",[72,73],{"id":63,"by":11,"text":64},{"id":66,"by":11,"text":67},{"value":75,"source":76,"evidence":77},"APSW","commenter",[78],{"id":79,"by":80,"text":81},"32832005","banana_giraffe","I implement a similar concept for Python using APSW and boto3. https:\u002F\u002Fgist.github.com\u002FQ726kbXuN\u002F56a095d0828c69625fa44c23118e5221 It's not really useful for deep dives into data,…","open_source","unknown",{"praise":85,"objections":105,"requests":124,"questions":144,"alternatives":165},[86,94],{"value":87,"source":76,"evidence":88},"Neat trick that fetches only needed pages via HTTP range requests",[89,90],{"id":52,"by":53,"text":54},{"id":91,"by":92,"text":93},"32829639","ks2048","This is cool. It would be interesting to see some stats like how many separate requests are needed for a single SELECT statement, since S3 charges per GET, even on a small range,…",{"value":95,"source":76,"evidence":96},"Useful for distributing read-only SQLite data",[97,101],{"id":98,"by":99,"text":100},"32830631","gkaemmer","This is cool, I'll probably end up using this (we store sqlite snapshots in S3). It'd be very handy if this was packaged as a CLI that worked like `sqlite3`, so you could get a…",{"id":102,"by":103,"text":104},"32832138","sntran","This would be quite convenient when dealing with MBtiles files hosted on S3 or similar. There are efforts such as PMTiles[1] that let us use HTTP Range query to retrieve tiles…",[106,117],{"value":107,"source":76,"evidence":108},"Using S3 for a database is risky and pushes SQLite beyond its limits",[109,113],{"id":110,"by":111,"text":112},"32830655","svnpenn","I feel like people are pushing SQLite beyond its limits with this stuff. If you're going to do something like this, wouldn't just a normal client\u002Fserver option like PostgreSQL be…",{"id":114,"by":115,"text":116},"32834440","traceroute66","I admire people for their inventiveness, however surely attempting to square-peg-round-hole something like this into AWS S3 is only going to end up biting them in the backside ?…",{"value":118,"source":76,"evidence":119},"Repeated queries may be cheaper with a local download",[120],{"id":121,"by":122,"text":123},"32833299","wanghq","If I need to query the db repeatedly (N times per second), isn't it more efficient (low latency and aws cost) to just download the db and run queries locally?",[125,129,137],{"value":126,"source":76,"evidence":127},"Add a CLI like sqlite3 for querying S3 files directly",[128],{"id":98,"by":99,"text":100},{"value":130,"source":76,"evidence":131},"Benchmarks comparing caching and performance against other VFS implementations",[132,136],{"id":133,"by":134,"text":135},"32830168","endisneigh","There have been many of these custom vfs implementations. Is there a benchmark on performance between them?",{"id":91,"by":92,"text":93},{"value":138,"source":76,"evidence":139},"Support for multiple ranges in a single S3 request",[140],{"id":141,"by":142,"text":143},"32831725","chrsig","i wish s3 supported multiple ranges in the range header. I've done this sort of access with a custom columnar format that stored a set of columns in a single file it was a…",[145,155],{"value":146,"source":76,"evidence":147},"Does it work with MinIO or other S3-compatible storage",[148,152],{"id":149,"by":150,"text":151},"32831551","redleader55","Sorry for the stupid question: does this work with MinIO?",{"id":153,"by":11,"text":154},"32831581","Yes, I actually used MinIO to test it (see the `test.py` script)",{"value":156,"source":76,"evidence":157},"Does caching break consistency with concurrent writes",[158,162],{"id":159,"by":160,"text":161},"32831561","vlovich123","Do you have any insight into whether the cache will alter consistency in the face of concurrent writes from multiple locations in a way that SQLite can't handle? I imagine this…",{"id":163,"by":11,"text":164},"32831608","This VFS is only for reading databases stored in S3, not writing. The README gives some alternatives for other implementations which allow writing.",[166,174,185,196,203],{"value":167,"source":76,"evidence":168},"sqlite-s3vfs",[169,173],{"id":170,"by":171,"text":172},"32829586","remram","Is there a benefit over the existing S3 VFS sqlite3-s3vfs,which also uses APSW?",{"id":66,"by":11,"text":67},{"value":175,"source":76,"evidence":176},"S3 Select",[177,181],{"id":178,"by":179,"text":180},"32832075","civilized","This is nice, but before you go to the trouble of using it, consider just putting your CSV, JSON, or parquet data file in S3 and query it with S3 Select…",{"id":182,"by":183,"text":184},"32833579","staticassertion","A comparison to S3 Select would make sense since S3 Select lets you leverage RANGE, leverage compact formats like compressed Parquet, and would presumably have a similar…",{"value":186,"source":76,"evidence":187},"Litestream",[188,192],{"id":189,"by":190,"text":191},"32829456","aleph-","Nifty, reminds me of https:\u002F\u002Fgithub.com\u002Fbacktrace-labs\u002Fverneuil or possibly https:\u002F\u002Flitestream.io\u002F",{"id":193,"by":194,"text":195},"32832186","JaggerFoo","I read of someone using AWS EFS for sqlite, which has nfs functionality, if I recall correctly. However I like the idea of using a local store with litestream to S3 and updating…",{"value":197,"source":76,"evidence":198},"sqlite-s3-query and mbtiles-s3-server",[199],{"id":200,"by":201,"text":202},"32834408","michalc","Aha! This is done in https:\u002F\u002Fgithub.com\u002Fuktrade\u002Fmbtiles-s3-server, but using an alternative library to query SQLite on S3, https:\u002F\u002Fgithub.com\u002Fmichalc\u002Fsqlite-s3-query (Full…",{"value":204,"source":70,"evidence":205},"Athena",[206],{"id":207,"by":11,"text":208},"32834859","> attempting to square-peg-round-hole something like this into AWS S3 I don't think it's like that. AWS already offers services like S3 Select[0] or Athena[1] that do something…",{"comments":210,"answered":31,"key":211},6,[212,215,216],{"id":213,"text":214},"32829766","That DynamoDB VFS looks cool! I agree that the VFS api makes one think about plenty of crazy ideas. Someone is working on a VFS based on Foundation DB[0] that looks very…",{"id":63,"text":64},{"id":163,"text":164},{"commenters":218,"max_depth":210,"maker_replies":210,"first_comment_mins":219,"lifespan_hours":220},41,55,37,[222,224,226,230,234,235,237,240,242,245,248,251,254,256,259,262,265,267,270,271,273,274,276,278,281,283,286,289,290,291,293,296,298,301,303,305,308,310,312,315,317,321,324,326,328,330,332,334,335,336,340,341,345,347,349,351,353,357,359,362,364],[52,33,44,223,53,33],"project\u002Fpositive\u002Fidea",[141,33,38,225,142,33],"project\u002Fneutral\u002Fperformance",[227,33,33,228,229,33],"32831849","other\u002Fneutral\u002Fnone","mbo",[231,41,33,232,233,33],"32829664","other\u002Fneutral\u002Fidea","psanford",[213,41,33,228,11,41],[236,41,33,228,134,33],"32830307",[238,41,33,228,239,33],"32830630","victor106",[241,41,33,228,134,33],"32830865",[243,41,33,244,239,33],"32831862","commenter\u002Fneutral\u002Fnone",[246,41,33,232,247,33],"32831230","jgraettinger1",[249,34,44,223,250,33],"32832332","Rapzid",[252,34,33,244,253,33],"32832348","encryptluks2",[255,34,33,244,250,33],"32832366",[257,34,33,228,258,33],"32832588","VHRanger",[260,34,33,232,261,33],"32832899","bob1029",[263,34,33,228,264,33],"32833478","thristian",[266,34,33,228,261,33],"32835651",[268,34,33,228,269,33],"32833722","chousuke",[91,38,38,225,92,33],[63,38,33,272,11,41],"maker\u002Fneutral\u002Fperformance",[159,38,38,225,160,33],[163,38,33,275,11,41],"project\u002Fneutral\u002Fnone",[98,31,44,277,99,33],"project\u002Fpositive\u002Fdesign",[279,31,38,280,233,33],"32831927","project\u002Fneutral\u002Fdesign",[110,44,41,282,111,33],"project\u002Fcritical\u002Fidea",[284,44,33,244,285,33],"32830688","samatman",[287,44,33,244,288,33],"32830878","freeqaz",[102,210,44,223,103,33],[200,210,38,275,201,33],[114,292,41,282,115,33],7,[294,292,33,244,295,33],"32834613","rad_gruchalski",[297,292,33,244,134,33],"32834819",[299,292,33,244,300,33],"32835008","Brystephor",[302,292,33,244,134,33],"32836616",[304,292,33,244,295,33],"32837290",[306,292,33,228,307,33],"32848685","quickthrower2",[207,292,33,309,11,41],"project\u002Fneutral\u002Fidea",[178,311,38,309,179,33],8,[313,311,33,228,314,33],"32832328","grangerg",[121,316,38,225,122,33],9,[318,319,33,232,320,33],"32831406",10,"rgovostes",[322,319,33,228,323,33],"32831531","teraflop",[325,319,33,228,320,33],"32831649",[327,319,33,232,160,33],"32831535",[193,329,33,228,194,33],11,[79,331,33,244,80,33],12,[189,333,33,228,190,33],13,[133,32,38,225,134,33],[182,18,38,225,183,33],[337,338,38,309,339,33],"32829931",16,"Datagenerator",[56,338,33,244,53,33],[342,338,33,343,344,33],"32830051","commenter\u002Fneutral\u002Fperformance","spullara",[346,338,33,244,201,33],"32830136",[348,338,33,228,285,33],"32830708",[149,350,38,275,150,33],17,[153,350,33,352,11,41],"maker\u002Fneutral\u002Fnone",[354,355,33,228,356,33],"32832097",18,"lootsauce",[170,358,38,309,171,33],19,[360,358,38,275,361,33],"32829692","8organicbits",[66,358,33,363,11,41],"maker\u002Fneutral\u002Fidea",[365,358,38,366,171,33],"32832826","maker\u002Fneutral\u002Fexecution",[],[369,374,379,384,389,394],{"id":370,"name":371,"points":372,"category":373},"32768590","Modern for HN",308,"productivity",{"id":375,"name":376,"points":377,"category":378},"32798891","HiFiScan",260,"creative",{"id":380,"name":381,"points":382,"category":383},"32828309","Infernal Sigil",238,"games",{"id":385,"name":386,"points":387,"category":388},"32837829","StackAid",226,"infrastructure",{"id":390,"name":391,"points":392,"category":393},"32823589","pg_netstat",158,"data",{"id":395,"name":396,"points":397,"category":398},"32833907","dato",134,"ai",false,"v2",{"id":402,"name":5,"line":6,"ai":399,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":403,"top":18,"kw":404,"maker":11,"domain":10,"tier":19,"attempts":33,"strip":405,"url":9,"mix":406,"praise":87,"objection":107,"sub":21},32828799,0.1,[24,25,26,27],"5300000500003305105100003030000330003033",[31,32,33,34],1794036624981]