[{"data":1,"prerenderedAt":339},["ShallowReactive",2],{"project-47534283":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":42,"post_text":45,"what":46,"said":85,"maker_in_thread":160,"shape":173,"grid":178,"attempts":301,"same_week":302,"maker_shows":33,"maker_hidden":331,"reader_version":332,"card":333},"47534283","Turbolite","a SQLite VFS serving sub-250ms cold JOIN queries from S3","title","Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3","https:\u002F\u002Fgithub.com\u002Frussellromney\u002Fturbolite","github.com","russellthehippo",1774551499,2026,185,43,328,4366,8,"notable","data","embedded database storage","library",[24,25,26,27],"sqlite vfs","s3 storage","cold queries","object storage database","loved",0.252,{"positive":31,"neutral":32,"mixed":33,"critical":34},7,10,2,1,20,{"idea":37,"design":38,"pricing":40,"performance":41},{"positive":34,"critical":34},{"positive":34,"critical":39},0,{"positive":39,"critical":39},{"positive":39,"critical":39},[43,39,39,39,43,44,39,44,39,34,39,39,39,39,39,39,39,39,43,39,43,39,43,43,44,39,39,43,39,43,39,39,44,43,39,39,44,44,44,43],3,5,"I built a SQLite VFS in Rust that serves cold queries directly from S3 with sub-second performance, and often much faster.\n\nIt’s called turbolite. It is experimental, buggy, and may corrupt data. I would not trust it with anything important yet.\n\nI wanted to explore whether object storage has gotten fast enough to support embedded databases over cloud storage. Filesystems reward tiny random reads and in-place mutation. S3 rewards fewer requests, bigger transfers, immutable objects, and aggressively parallel operations where bandwidth is often the real constraint. This was explicitly inspired b",{"problem":47,"audience":53,"pricing":57,"motivation":58,"time_to_build":57,"stack":62,"openness":83,"stage":84,"team_size":83},{"value":48,"source":49,"evidence":50},"Querying mostly-cold SQLite databases without keeping attached volumes","maker",[51],{"id":52},"post",{"value":54,"source":49,"evidence":55},"Developers with many mostly-cold databases, such as database-per-tenant architectures",[56],{"id":52},null,{"value":59,"source":49,"evidence":60},"Explore whether object storage is fast enough to back embedded databases",[61],{"id":52},[63,67,71,75],{"value":64,"source":49,"evidence":65},"Rust",[66],{"id":52},{"value":68,"source":49,"evidence":69},"SQLite",[70],{"id":52},{"value":72,"source":49,"evidence":73},"S3",[74],{"id":52},{"value":76,"source":77,"evidence":78},"zstd","commenter",[79],{"id":80,"by":81,"text":82},"47534846","carlsverre","You might be interested in taking a look at Graft (https:\u002F\u002Fgraft.rs\u002F). I have been iterating in this space for the last year, and have learned a lot about it. Graft has a…","unknown","prototype",{"praise":86,"objections":95,"requests":104,"questions":126,"alternatives":137},[87],{"value":88,"source":77,"evidence":89},"Interest in B-tree aware grouping and query-plan frontrunning ideas",[90,94],{"id":91,"by":92,"text":93},"47539580","wgjordan","Nice set of experiments! I appreciate that you're running benchmarks on real object storage setups to validate rapid design variations. (Meta-note: I love how agents have…",{"id":80,"by":81,"text":82},[96,100],{"value":97,"source":77,"evidence":98},"Write amplification from re-uploading whole page groups on dirty pages",[99],{"id":91,"by":92,"text":93},{"value":101,"source":49,"evidence":102},"Single-writer limitation",[103],{"id":52},[105,112,119],{"value":106,"source":77,"evidence":107},"Support for conditional PUT multi-writer",[108],{"id":109,"by":110,"text":111},"47535505","bob1029","Have you considered using techniques like conditional PUT to enable multiple writers? https:\u002F\u002Faws.amazon.com\u002Fabout-aws\u002Fwhats-new\u002F2024\u002F08\u002Famazon-s3-conditional-writes\u002F…",{"value":113,"source":77,"evidence":114},"Eviction policies",[115],{"id":116,"by":117,"text":118},"47534811","alex_hirner","What are your thoughts on eviction, re how easy to add some basic policy?",{"value":120,"source":77,"evidence":121},"WASM and browser support",[122],{"id":123,"by":124,"text":125},"47538976","pdyc","this is a great project, does it support wasm? i want to use it in browser with sqlite wasm.",[127,133],{"value":128,"source":49,"evidence":129},"How credentials would be handled for browser use",[130],{"id":131,"by":11,"text":132},"47548334","Also I'm curious how you would handle credentials - would you be proxying through your backend? To me turbolite is definitely a backend tool.",{"value":134,"source":49,"evidence":135},"Whether B-tree grouping direction is right",[136],{"id":52},[138,144,148,154],{"value":139,"source":77,"evidence":140},"Litestream",[141],{"id":142,"by":11,"text":143},"47536751","Yeah I mostly agree, and another comment brought up this idea kind of. If you can keep SQLite local, that’s usually the better answer. Local reads are insanely hard to beat, and…",{"value":145,"source":77,"evidence":146},"Graft",[147],{"id":80,"by":81,"text":82},{"value":149,"source":49,"evidence":150},"rqlite",[151],{"id":152,"by":11,"text":153},"47536884","rqlite does this well but it’s still server based",{"value":155,"source":49,"evidence":156},"Neon",[157],{"id":158,"by":11,"text":159},"47548255","Thanks! yeah it's definitely slow, but compared to other options it's quite fast: - Neon startup is 500ms+ - a Fly Machine with a SQLite database on a volume takes 150ms+ to…",{"comments":161,"answered":162,"key":163},22,9,[164,167,170],{"id":165,"text":166},"47535056","Very cool, thanks. I hadn’t seen Graft before, but that sounds pretty adjacent in a lot of interesting ways. I looked at the repo and see what I can apply. I've tried out all…",{"id":168,"text":169},"47536781","Thanks! I think the hole punching is the key one, as it is important for \"this delete has happened but we need to free the space without a vacuum\".",{"id":171,"text":172},"47548043","Fantastic comment, thanks for jumping in. I have well-tuned replies here lol given we're working on exactly the same problems! First: your prefetch solution is precisely what I…",{"commenters":174,"max_depth":175,"maker_replies":161,"first_comment_mins":176,"lifespan_hours":177},21,6,11,118,[179,181,183,186,188,190,192,195,197,201,203,206,210,212,215,217,220,222,224,226,227,231,232,234,235,238,240,244,247,249,252,255,257,261,263,266,267,269,270,273,274,276,279,281,286,290,294,298],[80,39,43,180,81,39],"maker\u002Fneutral\u002Fidea",[165,39,39,182,11,34],"project\u002Fneutral\u002Fdesign",[184,39,39,185,81,39],"47535817","commenter\u002Fneutral\u002Fnone",[168,39,39,187,11,34],"project\u002Fneutral\u002Fidea",[91,34,43,189,92,39],"project\u002Fmixed\u002Fdesign",[171,34,39,191,11,34],"project\u002Fpositive\u002Fperformance",[193,34,44,194,92,39],"47590620","project\u002Fpositive\u002Fidea",[196,33,39,187,11,34],"47534387",[198,33,44,199,200,39],"47535054","maker\u002Fpositive\u002Fnone","hgo",[202,33,39,185,11,34],"47535218",[204,43,43,187,205,39],"47535455","agosta",[207,43,34,208,209,39],"47536176","project\u002Fcritical\u002Fidea","hrmtst93837",[211,43,39,185,11,34],"47536718",[213,43,39,185,214,39],"47538819","AbanoubRodolf",[216,43,39,185,11,34],"47539092",[218,43,39,185,219,39],"47539413","AlexeyBelov",[221,43,39,185,11,34],"47539733",[223,43,39,185,209,39],"47552409",[225,43,39,185,11,34],"47563869",[142,43,39,187,11,34],[228,229,39,230,11,34],"47534400",4,"other\u002Fneutral\u002Fnone",[109,44,43,187,110,39],[233,44,39,187,11,34],"47536272",[152,44,39,230,11,34],[236,175,43,182,237,39],"47538973","swaminarayan",[239,175,39,182,11,34],"47539840",[241,31,43,242,243,39],"47535552","project\u002Fneutral\u002Fpricing","jijji",[245,31,43,242,246,39],"47535721","nthypes",[248,31,39,242,11,34],"47536540",[250,18,44,199,251,39],"47539029","ovaistariq",[253,18,39,254,11,34],"47540171","commenter\u002Fpositive\u002Fnone",[256,18,39,254,251,39],"47550186",[258,162,43,259,260,39],"47540377","project\u002Fmixed\u002Fperformance","mememememememo",[158,162,39,262,11,34],"project\u002Fneutral\u002Fperformance",[264,162,39,185,265,39],"47556321","tuhgdetzhh",[123,32,43,187,124,39],[268,32,39,182,11,34],"47540216",[131,32,39,182,11,34],[271,32,44,272,124,39],"47561215","project\u002Fpositive\u002Fdesign",[116,176,43,187,117,39],[275,176,39,182,11,34],"47534960",[277,176,39,278,11,34],"47563563","project\u002Fpositive\u002Fexecution",[280,176,39,182,11,34],"47535372",[282,283,44,284,285,39],"47534780",12,"project\u002Fpositive\u002Fnone","michaeljelly",[287,288,44,284,289,39],"47535254",13,"inferense",[291,292,44,284,293,39],"47536243",14,"ryanjso",[295,296,43,187,297,39],"47550087",15,"edinetdb",[299,300,43,182,297,39],"47538112",16,[],[303,308,312,317,322,327],{"id":304,"name":305,"points":306,"category":307},"47593422","1-Bit Bonsai",430,"ai",{"id":309,"name":310,"points":311,"category":307},"47536761","nullclaw",340,{"id":313,"name":314,"points":315,"category":316},"47557921","Git bayesect",337,"dev_tools",{"id":318,"name":319,"points":320,"category":321},"47580350","30u30.fyi",256,"other",{"id":323,"name":324,"points":325,"category":326},"47571183","korb",205,"consumer_social",{"id":328,"name":329,"points":330,"category":20},"47589856","pg_textsearch",203,false,"v2",{"id":334,"name":5,"line":6,"ai":331,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":335,"top":18,"kw":336,"maker":11,"domain":10,"tier":19,"attempts":39,"strip":337,"url":9,"mix":338,"praise":88,"objection":97,"sub":21},47534283,0.25,[24,25,26,27],"3000350501000000003030335003030053005553",[31,32,33,34],1794036697061]