[{"data":1,"prerenderedAt":466},["ShallowReactive",2],{"project-32303762":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":46,"post_text":48,"what":49,"said":78,"maker_in_thread":184,"shape":191,"grid":196,"attempts":427,"same_week":428,"maker_shows":38,"maker_hidden":458,"reader_version":459,"card":460},"32303762","sqlite-zstd","Reduce SQLite database size by up to 80% with transparent compression","title","Show HN: Reduce SQLite database size by up to 80% with transparent compression","https:\u002F\u002Fgithub.com\u002Fphiresky\u002Fsqlite-zstd","github.com","phiresky",1659346576,2022,420,90,80,2570,4,"legend","data","sqlite compression extension","library",[24,25,26,27],"sqlite","zstd","database compression","transparent compression","curious",0.061,{"positive":31,"neutral":32,"mixed":33,"critical":34},10,22,0,6,38,{"design":37,"idea":39,"docs":40,"performance":41,"privacy":43,"execution":44},{"positive":38,"critical":33},3,{"positive":18,"critical":18},{"positive":33,"critical":33},{"positive":42,"critical":33},1,{"positive":42,"critical":33},{"positive":33,"critical":45},2,[47,33,33,38,33,38,42,38,42,38,38,47,38,47,38,38,33,47,47,33,47,33,33,47,38,38,38,18,38,33,33,42,42,38,33,33,33,47,42,47],5,null,{"problem":50,"audience":48,"pricing":48,"motivation":48,"time_to_build":48,"stack":56,"openness":76,"stage":77,"team_size":77},{"value":51,"source":52,"evidence":53},"SQLite databases take up too much disk space","inferred",[54],{"id":55},"post",[57,65,70],{"value":58,"source":59,"evidence":60},"Rust","commenter",[61],{"id":62,"by":63,"text":64},"32305405","Joel_Mckay","sudo apt-get install archivemount archivemount mydbimage.tgz \u002Fmnt\u002Fmydbimage -o big_writes \u002F\u002F -o readonly disable write support \u002F\u002F -o nosave do not save changes upon unmount.…",{"value":25,"source":52,"evidence":66},[67],{"id":68,"by":11,"text":69},"32306496","> You can potentially get better results this way than a generic solution like the article because you might have some application knowledge about what rows are likely to be…",{"value":71,"source":52,"evidence":72},"SQLite",[73],{"id":74,"by":11,"text":75},"32306653","You can use it from python as follows: ``` db = sqlite3.connect(...) db.enable_load_extension() db.load_extension(\"libsqlite_zstd.so\") ``` You probably still have to figure out…","open_source","unknown",{"praise":79,"objections":98,"requests":125,"questions":149,"alternatives":164},[80,87],{"value":81,"source":59,"evidence":82},"Neat approach; per-column compression and custom partitioning are useful",[83],{"id":84,"by":85,"text":86},"32305161","wmanley","This is really neat. I like how the compression can be applied to specific columns rather than whole rows - or whole pages. I could imagine storing JSON from an HTTP request here…",{"value":88,"source":59,"evidence":89},"Useful for compressing GitHub statistics and JSON backups",[90,94],{"id":91,"by":92,"text":93},"32313194","andrewmcwatters","Oh, interesting. I scrape GitHub's APIs to generate user and repositories statistics data, then compress it with ZStandard to allow the databases to fit within GitHub's file size…",{"id":95,"by":96,"text":97},"32306206","zimpenfish","Tried this on my twitter database's `json` column (ie the JSON response from the API fetching a singular tweet) with a `dict_chooser` of '%Y-%W' to bundle weeks together and it…",[99,110,117],{"value":100,"source":59,"evidence":101},"Doing it at the filesystem level is an alternative",[102,106],{"id":103,"by":104,"text":105},"32304053","avereveard","I don't want to diss on the effort, I have an honest question: wouldn't this something that would be more appropriate to be done at the filesystem level?",{"id":107,"by":108,"text":109},"32304102","Someone","No. If you did that, SQLite would have to (logically) decompress about your entire database on every query. For example, how would it read the database schema to determine…",{"value":111,"source":59,"evidence":112},"Existing compression VFS extensions are the more common approach",[113],{"id":114,"by":115,"text":116},"32304154","d1l","I am happy for the author, but want to warn the credulous that this is a very strange way to approach compression with SQLite. The much more common way is to use SQLite vfs:…",{"value":118,"source":59,"evidence":119},"Title is clickbait-ish and approach may risk data loss",[120,121],{"id":114,"by":115,"text":116},{"id":122,"by":123,"text":124},"32304561","kevingadd","Taking an amateurish or fancy or experimental approach to compressing a sqlite database is a great way to end up with data loss. Doesn't make it a worthless or \"bad\" project but…",[126,133,140],{"value":127,"source":59,"evidence":128},"Support for retroactively compressing existing columns",[129],{"id":130,"by":131,"text":132},"32310263","leobg","This looks great. How would I apply this retroactively? I.e. tell SQLite to compress column x in an existing database, say, of type TEXT?",{"value":134,"source":59,"evidence":135},"Django integration",[136],{"id":137,"by":138,"text":139},"32308245","hda111","I love the idea. Can I use it with Django projects somehow?",{"value":141,"source":59,"evidence":142},"Comparison with Brotli and compression VFS",[143,146],{"id":144,"by":123,"text":145},"32304582","How does this compare with modern compression algorithms like Brotli that do context modeling, etc? I've found that they manage to aggressively compress types of data you…",{"id":147,"by":11,"text":148},"32304272","I did actually previously write a VFS, though it did something else entirely: https:\u002F\u002Fphiresky.github.io\u002Fblog\u002F2021\u002Fhosting-sqlite-databases-on-github-pages\u002F You're right that a…",[150,157],{"value":151,"source":59,"evidence":152},"What does transparent mean here",[153],{"id":154,"by":155,"text":156},"32304520","mansoor_","What does \"transparent\" mean in this context?",{"value":158,"source":59,"evidence":159},"Whether performance can improve from compression",[160],{"id":161,"by":162,"text":163},"32308551","logicallee","This mentions something I've been wondering about: >Depending on the data, this can reduce the size of the database by 80% while keeping performance mostly the same (or even…",[165,176],{"value":166,"source":59,"evidence":167},"Filesystem-level compression (btrfs, ZFS)",[168,172],{"id":169,"by":170,"text":171},"32304671","ciupicri","> Btrfs also supports transparent file compression. There are three algorithms available: ZLIB, LZO and ZSTD (since v4.14). Basically, compression is on a file by file basis. You…",{"id":173,"by":174,"text":175},"32305966","aabbcc1241","I'm using sqlite on zfs and btrfs, with transparent compression supported by the file system, the query speed is very good and compression rate also very good. (database below…",{"value":177,"source":59,"evidence":178},"SQLite compression VFS extensions (sqlite_zstd_vfs, ZIPVFS)",[179,180,181],{"id":114,"by":115,"text":116},{"id":147,"by":11,"text":148},{"id":182,"by":115,"text":183},"32304258","Why not read the sqlite teams own explanation: https:\u002F\u002Fwww.sqlite.org\u002Fzipvfs\u002Fdoc\u002Ftrunk\u002Fwww\u002Fhowitworks.wiki",{"comments":34,"answered":45,"key":185},[186,189,190],{"id":187,"text":188},"32304533","Transparent to the application - basically the SQL queries (select, insert, update) in your code base don't have to change at all. This would be in contrast to e.g. compressing…",{"id":68,"text":69},{"id":74,"text":75},{"commenters":192,"max_depth":193,"maker_replies":34,"first_comment_mins":194,"lifespan_hours":195},57,7,50,93,[197,199,203,206,209,213,217,220,222,225,228,230,232,235,237,241,244,246,249,252,254,256,258,261,264,266,269,270,272,273,275,276,279,282,284,287,288,289,292,294,297,300,303,307,310,312,314,317,319,322,325,328,332,335,338,341,345,348,351,353,355,359,363,367,369,371,373,375,378,380,382,386,387,389,390,391,394,396,398,400,403,406,408,411,414,416,418,420,422,425],[84,33,47,198,85,33],"project\u002Fpositive\u002Fdesign",[200,33,33,201,202,33],"32305198","commenter\u002Fneutral\u002Fnone","klysm",[204,33,33,201,205,33],"32305648","KingOfCoders",[207,33,33,201,208,33],"32306345","allan_s",[210,33,33,211,212,33],"32311882","commenter\u002Fneutral\u002Fperformance","OnlyMortal",[214,33,33,215,216,33],"32310512","other\u002Fneutral\u002Fnone","rovr138",[218,33,33,201,219,33],"32308902","masklinn",[221,33,33,201,202,33],"32312080",[223,33,38,224,85,33],"32305951","project\u002Fneutral\u002Fidea",[226,33,33,215,227,33],"32306019","gcr",[229,33,33,215,174,33],"32307017",[231,33,33,215,85,33],"32307376",[233,33,38,234,227,33],"32306004","project\u002Fneutral\u002Fdesign",[236,33,38,234,85,33],"32306454",[238,42,42,239,240,33],"32312557","project\u002Fcritical\u002Fidea","cryptonector",[242,42,33,201,243,33],"32320690","dataangel",[245,42,33,201,240,33],"32333874",[247,45,38,224,248,33],"32305312","yababa_y",[250,45,42,239,251,33],"32310336","alecco",[253,45,33,201,248,33],"32312104",[154,38,38,255,155,33],"project\u002Fneutral\u002Fdocs",[187,38,33,257,11,42],"maker\u002Fneutral\u002Fdocs",[259,18,38,224,260,33],"32306071","quietbritishjim",[262,18,33,215,263,33],"32306261","goldenkey",[68,18,33,265,11,42],"maker\u002Fneutral\u002Fdesign",[267,18,47,268,260,33],"32306629","project\u002Fpositive\u002Fnone",[74,18,33,255,11,42],[271,18,33,201,260,33],"32306700",[130,47,38,255,131,33],[91,34,47,274,92,33],"project\u002Fpositive\u002Fidea",[103,193,38,224,104,33],[277,193,38,224,278,33],"32304067","xdfgh1112",[280,193,38,234,281,33],"32304125","Gehinnn",[283,193,38,255,115,33],"32304170",[285,193,38,224,286,33],"32304076","fnord123",[169,193,33,215,170,33],[107,193,33,201,108,33],[290,193,33,201,291,33],"32304551","formerly_proven",[293,193,33,201,108,33],"32305443",[295,193,47,198,296,33],"32304124","antman",[298,193,47,274,299,33],"32304137","hexmiles",[301,193,33,302,11,42],"32304148","maker\u002Fneutral\u002Fidea",[304,193,33,305,306,33],"32304581","other\u002Fneutral\u002Fperformance","marginalia_nu",[308,193,33,215,309,33],"32305417","JulianWasTaken",[311,193,33,215,306,33],"32305548",[95,193,47,313,96,33],"project\u002Fpositive\u002Fperformance",[315,193,33,215,316,33],"32306562","cmurf",[318,193,33,201,11,42],"32306614",[320,193,33,215,321,33],"32310492","magicalhippo",[323,193,33,215,324,33],"32311230","Dagger2",[326,193,33,215,327,33],"32318425","telendram",[329,193,47,330,331,33],"32309241","project\u002Fpositive\u002Fprivacy","ajconway",[333,193,33,215,334,33],"32310597","gred",[161,336,38,337,162,33],8,"project\u002Fneutral\u002Fperformance",[339,336,33,305,340,33],"32311785","outworlder",[342,343,38,224,344,33],"32311756",9,"josephernest",[346,343,33,215,347,33],"32311838","hnarn",[349,343,38,234,350,33],"32311859","Scaevolus",[352,343,38,255,344,33],"32312324",[354,343,38,255,350,33],"32312404",[356,31,38,357,358,33],"32313865","project\u002Fneutral\u002Fexecution","malkia",[360,361,38,337,362,33],"32310157",11,"AtNightWeCode",[364,365,47,274,366,33],"32313784",12,"password4321",[137,368,38,357,138,33],13,[173,370,33,215,174,33],14,[372,370,33,201,316,33],"32306486",[374,370,33,211,174,33],"32306915",[376,370,33,215,377,33],"32309310","usgroup",[379,370,33,215,174,33],"32352373",[114,381,42,239,115,33],15,[383,381,33,384,385,33],"32304185","commenter\u002Fneutral\u002Fidea","Dowwie",[182,381,33,201,115,33],[122,381,42,388,123,33],"project\u002Fcritical\u002Fexecution",[147,381,33,302,11,42],[144,381,38,337,123,33],[392,381,33,211,393,33],"32305270","vlovich123",[395,381,33,201,123,33],"32315778",[397,381,33,211,393,33],"32337860",[399,381,33,201,123,33],"32338013",[401,381,33,201,402,33],"32304535","skrebbel",[404,381,33,201,405,33],"32304559","867-5309",[407,381,33,201,402,33],"32304593",[409,381,33,201,410,33],"32305525","aidenn0",[412,381,47,198,413,33],"32304583","skerit",[415,381,38,224,96,33],"32305049",[62,417,42,239,63,33],16,[419,417,42,388,306,33],"32305750",[421,417,33,201,63,33],"32305840",[423,417,47,274,424,33],"32305889","xyzzy_plugh",[426,417,33,201,63,33],"32306060",[],[429,434,439,444,449,453],{"id":430,"name":431,"points":432,"category":433},"32266086","Parsnip",536,"learning_science",{"id":435,"name":436,"points":437,"category":438},"32293178","PlayCode",397,"dev_tools",{"id":440,"name":441,"points":442,"category":443},"32331359","Adadot",325,"web_saas",{"id":445,"name":446,"points":447,"category":448},"32312413","Emery",261,"productivity",{"id":450,"name":451,"points":452,"category":20},"32269287","mvsqlite",150,{"id":454,"name":455,"points":456,"category":457},"32269722","TensorDock Core GPU Cloud",147,"infrastructure",false,"v2",{"id":461,"name":5,"line":6,"ai":458,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":462,"top":18,"kw":463,"maker":11,"domain":10,"tier":19,"attempts":33,"strip":464,"url":9,"mix":465,"praise":81,"objection":100,"sub":21},32303762,0.06,[24,25,26,27],"5003031313353533055050053334300113000515",[31,32,33,34],1794036623188]