[{"data":1,"prerenderedAt":317},["ShallowReactive",2],{"project-44284871":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":36,"aspects":37,"strip":42,"post_text":44,"what":45,"said":67,"maker_in_thread":130,"shape":141,"grid":145,"attempts":279,"same_week":280,"maker_shows":35,"maker_hidden":309,"reader_version":310,"card":311},"44284871","Zeekstd","Rust Implementation of the ZSTD Seekable Format","title","Show HN: Zeekstd – Rust Implementation of the ZSTD Seekable Format","https:\u002F\u002Fgithub.com\u002Frorosen\u002Fzeekstd","github.com","rorosen",1750020558,2025,214,46,287,3837,8,"notable","languages","compression library","library",[24,25,26,27,28],"zstd","seekable format","rust","compression","random access","curious",0.195,{"positive":32,"neutral":33,"mixed":34,"critical":35},5,19,0,1,25,{"idea":38,"performance":40,"design":41},{"positive":39,"critical":34},4,{"positive":34,"critical":34},{"positive":34,"critical":35},[34,34,32,34,34,43,43,34,43,43,43,34,43,43,43,34,43,35,34,43,43,43,34,32,34,34,32,43,43,34,34,32,43,32,34,34,34,34,43,43],3,"Hello,\n\nI would like to share a Rust implementation of the Zstandard seekable format I've been working on.\n\nRegular zstd compressed files consist of a single frame, meaning you have to start decompression at the beginning. The seekable format splits compressed data into a series of independent frames, each compressed individually, so that decompression of a section in the middle of an archive only requires zstd to decompress at most a frame's worth of extra data, instead of the entire archive.\n\nI started working with the seekable format because I wanted to resume downloads of big zstd compress",{"problem":46,"audience":52,"pricing":52,"motivation":53,"time_to_build":52,"stack":57,"openness":66,"stage":66,"team_size":66},{"value":47,"source":48,"evidence":49},"Random access decompression of zstd files without decompressing from the start","maker",[50],{"id":51},"post",null,{"value":54,"source":48,"evidence":55},"Resume downloads of big zstd files decompressed to disk on the fly; existing C seekable bindings were limited",[56],{"id":51},[58,62],{"value":59,"source":48,"evidence":60},"Rust",[61],{"id":51},{"value":63,"source":48,"evidence":64},"zstd advanced compression API bindings",[65],{"id":51},"unknown",{"praise":68,"objections":81,"requests":88,"questions":103,"alternatives":117},[69],{"value":70,"source":71,"evidence":72},"Useful for seekable compressed data and random access","commenter",[73,77],{"id":74,"by":75,"text":76},"44288747","simeonmiteff","This is very cool. Nice work! At my day job, I have been using a Go library[1] to build tools that require seekable zstd, but felt a bit uncomfortable with the lack of broader…",{"id":78,"by":79,"text":80},"44292780","conradev","This is really cool! It strikes me as being useful for genomic data, which is always stored in compressed chunks. That was the first time I really understood the hard trade-off…",[82],{"value":83,"source":48,"evidence":84},"Seekable format is not standardized",[85],{"id":86,"by":11,"text":87},"44294190","It's not standardized as far as I know.",[89,96],{"value":90,"source":71,"evidence":91},"Support for seeking to an offset without computing frames",[92],{"id":93,"by":94,"text":95},"44289103","tyilo","Correct me if I'm wrong, but it doesn't seem like you provide the equivalent of Seekable::decompress in zstd_seekable which decompresses at a specific offset, without having to…",{"value":97,"source":71,"evidence":98},"Support for piping seekable files via stdin",[99],{"id":100,"by":101,"text":102},"44294133","DesiLurker","great I can use it to pipe large logfiles and store for later retrival. is there something like zcat also?",[104,110],{"value":105,"source":71,"evidence":106},"How does it compare to zstd_seekable crate",[107],{"id":108,"by":94,"text":109},"44289063","I already use zstd_seekable (https:\u002F\u002Fdocs.rs\u002Fzstd-seekable\u002F) in a project. Could you compare the API's of this crate and yours?",{"value":111,"source":71,"evidence":112},"Support for custom dictionaries",[113],{"id":114,"by":115,"text":116},"44288831","stu2010","This is cool, I'd say that the most common tool in this space is bgzip[1]. Have you thought about training a dictionary on the first few chunks of each file and embedding the…",[118,122,126],{"value":119,"source":71,"evidence":120},"zstd_seekable crate",[121],{"id":108,"by":94,"text":109},{"value":123,"source":71,"evidence":124},"bgzip",[125],{"id":114,"by":115,"text":116},{"value":127,"source":71,"evidence":128},"zstd-seekable-format-go",[129],{"id":74,"by":75,"text":76},{"comments":131,"answered":132,"key":133},9,7,[134,135,138],{"id":86,"text":87},{"id":136,"text":137},"44289179","Thanks! I was also surprised that there are very few tools to work with the seekable format. I could imagine that at least some people have a use-case for it. Yes, the name is a…",{"id":139,"text":140},"44294208","Yes, dictionaries should be totally possible. However, I've never tried them to be honest because I usually only compress big files. They can be set on the (de)compression…",{"commenters":142,"max_depth":32,"maker_replies":131,"first_comment_mins":143,"lifespan_hours":144},31,911,58,[146,150,152,154,156,160,163,166,169,170,173,176,178,180,183,186,190,193,196,198,200,203,207,210,213,216,219,222,225,228,231,232,236,238,240,244,248,252,254,258,262,264,268,271,273,276],[147,34,34,148,149,34],"44289058","commenter\u002Fneutral\u002Fidea","rwmj",[86,34,34,151,11,35],"maker\u002Fneutral\u002Fidea",[74,35,32,153,75,34],"project\u002Fpositive\u002Fidea",[136,35,34,155,11,35],"maker\u002Fneutral\u002Fnone",[157,35,34,158,159,34],"44289413","other\u002Fneutral\u002Fnone","etyp",[114,161,43,162,115,34],2,"project\u002Fneutral\u002Fidea",[164,161,43,162,165,34],"44288965","jeroenhd",[167,161,43,162,168,34],"44292360","ikawe",[139,161,34,151,11,35],[171,43,43,162,172,34],"44291621","mbreese",[174,43,43,162,175,34],"44291811","teraflop",[177,43,43,162,172,34],"44292740",[179,43,34,162,11,35],"44294240",[181,43,43,162,182,34],"44294245","adrianmonk",[184,43,34,185,172,34],"44295231","other\u002Fneutral\u002Fidea",[187,39,43,188,189,34],"44290762","project\u002Fneutral\u002Fperformance","threeducks",[191,39,43,188,192,34],"44290972","mcraiha",[194,39,34,195,11,35],"44296197","maker\u002Fneutral\u002Fperformance",[108,32,43,197,94,34],"project\u002Fneutral\u002Fnone",[93,32,35,199,94,34],"project\u002Fcritical\u002Fdesign",[201,32,34,202,11,35],"44294319","maker\u002Fneutral\u002Fdesign",[204,205,43,162,206,34],"44289299",6,"ncruces",[208,205,43,197,209,34],"44294200","wolfgang42",[211,132,43,162,212,34],"44294976","wyager",[214,132,43,162,215,34],"44295992","wilted-iris",[217,132,34,218,11,35],"44296235","maker\u002Fpositive\u002Fperformance",[220,132,32,221,212,34],"44303652","maker\u002Fpositive\u002Fnone",[223,18,34,185,224,34],"44289259","throebrifnr",[226,18,34,185,227,34],"44289907","Scaevolus",[229,18,34,185,230,34],"44290228","andrewaylett",[78,131,32,153,79,34],[233,234,43,162,235,34],"44291724",10,"mgraczyk",[237,234,43,162,175,34],"44291826",[239,234,34,185,149,34],"44291843",[241,242,34,185,243,34],"44294794",11,"Dwedit",[245,246,32,153,247,34],"44292273",12,"Imustaskforhelp",[249,250,43,162,251,34],"44289762",13,"b0a04gl",[253,250,34,151,11,35],"44294411",[255,256,32,153,257,34],"44307377",14,"sitkack",[259,260,34,185,261,34],"44289274",15,"77pt77",[263,260,34,185,149,34],"44290419",[265,260,34,266,267,34],"44290464","other\u002Fpositive\u002Fidea","dafelst",[269,260,34,158,270,34],"44291519","cogman10",[100,272,43,197,101,34],16,[274,272,34,275,11,35],"44294385","maker\u002Fneutral\u002Fdocs",[277,272,43,197,278,34],"44296536","tarasglek",[],[281,286,291,295,300,305],{"id":282,"name":283,"points":284,"category":285},"44309320","Workout.cool",827,"consumer_social",{"id":287,"name":288,"points":289,"category":290},"44314085","Unregistry",726,"dev_tools",{"id":292,"name":293,"points":294,"category":290},"44293260","Chawan",387,{"id":296,"name":297,"points":298,"category":299},"44292103","Canine",320,"infrastructure",{"id":301,"name":302,"points":303,"category":304},"44256765","Tritium",312,"productivity",{"id":306,"name":307,"points":308,"category":290},"44306041","Lstr",227,false,"v2",{"id":312,"name":5,"line":6,"ai":309,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":313,"top":18,"kw":314,"maker":11,"domain":10,"tier":19,"attempts":34,"strip":315,"url":9,"mix":316,"praise":70,"objection":83,"sub":21},44284871,0.2,[24,25,26,27,28],"0050033033303330310333050053300535000033",[32,33,34,35],1794036676749]