[{"data":1,"prerenderedAt":299},["ShallowReactive",2],{"project-8314749":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":35,"aspects":36,"strip":44,"post_text":46,"what":47,"said":82,"maker_in_thread":187,"shape":194,"grid":198,"attempts":262,"same_week":263,"maker_shows":197,"maker_hidden":291,"reader_version":292,"card":293},"8314749","Homogenius","Packer\u002Funpacker to reduce the size of JSON","title","Show HN: Homogenius – Packer\u002Funpacker to reduce the size of JSON","http:\u002F\u002Fhomogenius.github.io\u002Fdocs\u002F","homogenius.github.io","afshinmeh",1410680381,2014,58,28,552,2272,25,"notable","languages","data serialization format","library",[24,25,26,27,28],"json compression","json size","msgpack alternative","json packer","data interchange","mixed",-0.143,{"positive":32,"neutral":32,"mixed":33,"critical":34},4,2,7,17,{"performance":37,"idea":39,"design":41,"docs":43},{"positive":33,"critical":38},1,{"positive":33,"critical":40},5,{"positive":42,"critical":42},0,{"positive":42,"critical":38},[40,40,42,38,42,42,45,42,45,38,42,42,45,42,42,42,38,42,45,38,38,45,38,40,40,38,45,42],3,null,{"problem":48,"audience":46,"pricing":46,"motivation":59,"time_to_build":46,"stack":64,"openness":79,"stage":80,"team_size":81},{"value":49,"source":50,"evidence":51},"Large JSON payloads with many repeated keys and values waste bandwidth","inferred",[52,56],{"id":53,"by":54,"text":55},"8314964","codehero","I see two great things about this tool, and not exactly for what it was designed: 1) Fix bandwidth problems caused by API designer laziness. 2) Detect API designer laziness and…",{"id":57,"by":11,"text":58},"8315039","Actually I'm trying to create a new datatype for homogenous data types, which is prevents repeating keys that happened in JSON datatype.",{"value":60,"source":61,"evidence":62},"Maker is defining a new datatype for homogeneous data that avoids repeating JSON keys","maker",[63],{"id":57,"by":11,"text":58},[65,71],{"value":66,"source":50,"evidence":67},"JavaScript",[68],{"id":69,"by":11,"text":70},"8315144","Here you go https:\u002F\u002Fwww.npmjs.org\u002Fpackage\u002Fhomogenius",{"value":72,"source":73,"evidence":74},"Go","commenter",[75],{"id":76,"by":77,"text":78},"8315018","bussiere","You loose in human readability but you win in size. An interesting project and i think about a lot of uses for it (save bandwith, avoid timeout etc ...) thks to make it and i…","open_source","launched","unknown",{"praise":83,"objections":99,"requests":141,"questions":156,"alternatives":168},[84,95],{"value":85,"source":73,"evidence":86},"Size reduction versus raw JSON, especially when combined with gzip",[87,91],{"id":88,"by":89,"text":90},"8314880","beefsack","Out of interest, do you have any size comparisons compared to gzipping the raw homogenous JSON text? Edit: I did a quick test by myself, using the first example but repeated for…",{"id":92,"by":93,"text":94},"8314903","walrus","In most real-world cases, it'll look like this: ``` gzip+Homogenius \u003C gzip \u003C Homogenius \u003C original JSON ``` gzip+Homogenius will typically be better than just gzip because…",{"value":96,"source":73,"evidence":97},"Useful for detecting lazy API payloads with repeated subtrees",[98],{"id":53,"by":54,"text":55},[100,119,127,134],{"value":101,"source":73,"evidence":102},"Binary formats like MsgPack or protobuf are smaller and simpler",[103,107,111,115],{"id":104,"by":105,"text":106},"8315388","imaginenore","Why not just use MessagePack (aka MsgPack) http:\u002F\u002Fmsgpack.org\u002F",{"id":108,"by":109,"text":110},"8315130","eli","If size is a concern, wouldn't a binary message format make more sense?",{"id":112,"by":113,"text":114},"8315396","placeybordeaux","I really don't see too many use cases here. Both parties have to buy into it, then at that point you might as well use a binary protocol.",{"id":116,"by":117,"text":118},"8315464","grimlck","Except that if you don't care about human readability, you might as well use a binary format and decrease size even more",{"value":120,"source":73,"evidence":121},"Loss of human readability is not worth the gain when gzip is available",[122,123],{"id":76,"by":77,"text":78},{"id":124,"by":125,"text":126},"8315358","sauere","Yep. Depending on the scenario, i don't think this the loss of readability is worth it. Using gzip on the webserver already cuts enough.",{"value":128,"source":73,"evidence":129},"Bzip2 results showed it can worsen compression ratio",[130],{"id":131,"by":132,"text":133},"8315323","_delirium","I get the same ordering with gzip, but it seems sensitive to compression algorithm. With bzip2 (default settings, v1.0.6), homogenius seems to actually worsen the compression…",{"value":135,"source":73,"evidence":136},"No benchmarks against gzip",[137],{"id":138,"by":139,"text":140},"8315276","stefan_kendall3","No benchmarking or comparisons with gzip.",[142,149],{"value":143,"source":73,"evidence":144},"Add a marker to indicate compressed form",[145],{"id":146,"by":147,"text":148},"8314840","akx","I think this needs some sort of marker for the compressed form to make it clear it's not to be digested as is. Maybe `$HMGNS$` or something as the first key of the array?",{"value":150,"source":73,"evidence":151},"Publish to npm",[152],{"id":153,"by":154,"text":155},"8315119","shocks","Any plans to add this to npm?",[157,161],{"value":158,"source":73,"evidence":159},"How does size compare to gzipping raw JSON?",[160],{"id":88,"by":89,"text":90},{"value":162,"source":73,"evidence":163},"Would a single shared value mapping work better than one per key?",[164],{"id":165,"by":166,"text":167},"8314943","ah-","So it's a schema plus interned values? Do you think it might be better on real-world data to use just one mapping of value id to value instead of one per column\u002Fkey?",[169,176,183],{"value":170,"source":73,"evidence":171},"MessagePack",[172,173],{"id":104,"by":105,"text":106},{"id":174,"by":105,"text":175},"8316231","So can MsgPack https:\u002F\u002Fgithub.com\u002Fcreationix\u002Fmsgpack-js https:\u002F\u002Fgithub.com\u002Fmsgpack\u002Fmsgpack-javascript",{"value":177,"source":73,"evidence":178},"Transit format by Cognitect",[179],{"id":180,"by":181,"text":182},"8315025","_ducky","I think [Transit][1] by Cognitect addresses this issue as well. [1]: https:\u002F\u002Fgithub.com\u002Fcognitect\u002Ftransit-format#caching",{"value":184,"source":73,"evidence":185},"Binary serialization formats",[186],{"id":108,"by":109,"text":110},{"comments":45,"answered":33,"key":188},[189,190,193],{"id":57,"text":58},{"id":191,"text":192},"8315134","Binary format will help as well in homogenius format. Please note that I'm defining a new datatype, not only a compressor.",{"id":69,"text":70},{"commenters":195,"max_depth":32,"maker_replies":45,"first_comment_mins":196,"lifespan_hours":197},23,80,13,[199,201,202,205,207,211,213,215,217,219,221,225,226,228,231,234,237,238,239,241,242,243,244,246,250,254,257,260],[88,42,40,200,89,42],"project\u002Fpositive\u002Fperformance",[92,42,40,200,93,42],[203,42,42,204,89,42],"8314907","commenter\u002Fneutral\u002Fperformance",[131,42,38,206,132,42],"project\u002Fcritical\u002Fperformance",[208,42,42,209,210,42],"8315815","other\u002Fneutral\u002Fperformance","Scaevolus",[212,42,42,209,125,42],"8315349",[180,38,45,214,181,42],"project\u002Fneutral\u002Fidea",[57,38,42,216,11,38],"maker\u002Fneutral\u002Fidea",[146,33,45,218,147,42],"project\u002Fneutral\u002Fdesign",[104,45,38,220,105,42],"project\u002Fcritical\u002Fidea",[222,45,42,223,224,42],"8315560","commenter\u002Fneutral\u002Fidea","dennyabraham",[174,45,42,223,105,42],[53,32,45,227,54,42],"project\u002Fmixed\u002Fidea",[229,32,42,204,230,42],"8315071","Ixiaus",[232,32,42,223,233,42],"8315241","Guillaume86",[235,32,42,223,236,42],"8315810","AYBABTME",[108,40,38,220,109,42],[191,40,42,216,11,38],[76,240,45,227,77,42],6,[124,240,38,220,125,42],[116,240,38,220,117,42],[165,34,45,218,166,42],[112,245,38,220,113,42],8,[247,245,40,248,249,42],"8315613","project\u002Fpositive\u002Fidea","jarin",[251,252,40,248,253,42],"8315245",9,"stinky613",[138,255,38,256,139,42],10,"project\u002Fcritical\u002Fdocs",[153,258,45,259,154,42],11,"project\u002Fneutral\u002Fnone",[69,258,42,261,11,38],"maker\u002Fneutral\u002Fnone",[],[264,269,274,278,283,287],{"id":265,"name":266,"points":267,"category":268},"8331233","CleverDeck",305,"learning_science",{"id":270,"name":271,"points":272,"category":273},"8315616","Picnic.css",247,"dev_tools",{"id":275,"name":276,"points":277,"category":273},"8315224","ContentBuilder.js",189,{"id":279,"name":280,"points":281,"category":282},"8323462","Berlin Hacker News",145,"productivity",{"id":284,"name":285,"points":286,"category":273},"8310137","Upmin",140,{"id":288,"name":289,"points":290,"category":273},"8312249","Pup",126,false,"v2",{"id":294,"name":5,"line":6,"ai":291,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":295,"top":18,"kw":296,"maker":11,"domain":10,"tier":19,"attempts":42,"strip":297,"url":9,"mix":298,"praise":85,"objection":101,"sub":21},8314749,-0.14,[24,25,26,27,28],"5501003031003000103113155130",[32,32,33,34],1794036522686]