[{"data":1,"prerenderedAt":295},["ShallowReactive",2],{"project-9477032":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":35,"aspects":36,"strip":40,"post_text":43,"what":44,"said":75,"maker_in_thread":161,"shape":166,"grid":171,"attempts":256,"same_week":257,"maker_shows":32,"maker_hidden":287,"reader_version":288,"card":289},"9477032","zindex","Efficiently searching compressed text files","title","Show HN: Efficiently searching compressed text files","https:\u002F\u002Fgithub.com\u002Fmattgodbolt\u002Fzindex","github.com","mattgodbolt",1430574792,2015,125,35,185,2565,8,"notable","dev_tools","command line search tool","cli",[24,25,26,27,28],"gzip","compressed files","log search","random access","indexing","curious",0.111,{"positive":32,"neutral":33,"mixed":34,"critical":34},1,6,0,7,{"idea":37,"design":38,"performance":39},{"positive":32,"critical":34},{"positive":34,"critical":34},{"positive":34,"critical":34},[41,42,34,42,34,34,42,34,34,34,34,34,34,42,34,42,34,42,34,34,34,34,34,34,34,34,34,34,34,34,34,34,34],5,3,null,{"problem":45,"audience":55,"pricing":43,"motivation":62,"time_to_build":43,"stack":67,"openness":72,"stage":73,"team_size":74},{"value":46,"source":47,"evidence":48},"Searching large gzipped text files without decompressing them entirely","maker",[49,52],{"id":50,"by":11,"text":51},"9477409","Fantastic! I had hoped others would have the same problem as me. I didn't have the luxury of being able to change my many input gzip files (unlike BGZF). The trick is to store…",{"id":53,"by":11,"text":54},"9477390","Thanks! For my logfiles it runs through them at around 3MB\u002Fsec (that's 3MB of compressed file per second). Most of it is probably I\u002FO, my giant logs are on NFS. I run the indexer…",{"value":56,"source":47,"evidence":57},"Engineers who work with large multi-gigabyte gzipped log files",[58,61],{"id":59,"by":11,"text":60},"9477361","Interesting point, thanks for the link! For my use case I wanted to put stuff together without writing too much novel code; zindex uses a SQLite file to store indices and…",{"id":53,"by":11,"text":54},{"value":63,"source":47,"evidence":64},"Maker needed to search their own large gzipped log files on NFS without changing the input files",[65,66],{"id":50,"by":11,"text":51},{"id":59,"by":11,"text":60},[68],{"value":69,"source":47,"evidence":70},"SQLite (index storage)",[71],{"id":59,"by":11,"text":60},"open_source","launched","unknown",{"praise":76,"objections":89,"requests":98,"questions":118,"alternatives":131},[77],{"value":78,"source":79,"evidence":80},"Useful idea for searching compressed logs; practical checkpoint approach","commenter",[81,85],{"id":82,"by":83,"text":84},"9478409","jules","The article's method is nice, it allows you to decompress portions of the file. I thought this would be about the FM index. The FM index allows you to search what is essentially…",{"id":86,"by":87,"text":88},"9477389","xaa","I have been wanting this for a long time, although my primary use case would be indexing a particular field. It is nice that you were able to make it work with ordinary gzip…",[90],{"value":91,"source":79,"evidence":92},"Indexing speed on large files is slow because it must scan sequentially",[93,97],{"id":94,"by":95,"text":96},"9477332","nattaylor","About how long does it take to create the index, maybe compared to a zgrep? Could it be parallelized with something like `parallel --pipe`? I haven't tried yet, but this seems…",{"id":53,"by":11,"text":54},[99,108],{"value":100,"source":79,"evidence":101},"Support indexing by column name or field in delimited files",[102,105],{"id":103,"by":87,"text":104},"9477798","Smart. I will see for myself in a minute how much space such an approach takes; I would imagine the compressor size gets nontrivially large on big files. Well, my exact use case,…",{"id":106,"by":11,"text":107},"9489860","As of just now, zindex and zq now support field-based indexing. Hope this helps your use case!",{"value":109,"source":79,"evidence":110},"Fuzzy matching such as wildcard search for MAC addresses",[111,115],{"id":112,"by":113,"text":114},"9478209","hackmiester","I only wish that the matching could be 'fuzzy' in easily defined ways. My use case is MAC addresses. So, if I search for 'dead.beef.cafe', this would be perfect if it could know…",{"id":116,"by":11,"text":117},"9478582","Great idea. zq uses the SQLite index built by zindex; so in theory I can modify it to use a 'WHERE index LIKE xxx' instead of 'WHERE index=xxx' -- then SQLish wildcards would be…",[119,124],{"value":120,"source":79,"evidence":121},"How the tool finds the right block and offset in arbitrary-size gzip blocks",[122,123],{"id":86,"by":87,"text":88},{"id":50,"by":11,"text":51},{"value":125,"source":79,"evidence":126},"Whether multiple indexes can exist on one gzip file",[127,128],{"id":103,"by":87,"text":104},{"id":129,"by":11,"text":130},"9477926","Short reply as I'm on a phone, but thanks! I can add an option to pick the nth field, and to skip the first X lines of the input; I think this'd cover your use case! The code and…",[132,143,147,154],{"value":133,"source":79,"evidence":134},"FM-index and succinct data structures",[135,139],{"id":136,"by":137,"text":138},"9477338","dpkendal","Why not store the log files in an FM-index? (https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FFM-index)",{"id":140,"by":141,"text":142},"9478205","dangirsh","See \"succinct data structures\" and \"wavelet trees\" for a similar \u002F more general idea. news.ycombinator.com\u002Fitem?id=7079427 I also must thank Edward Kmett for introducing me to…",{"value":144,"source":79,"evidence":145},"BGZF blocked gzip format",[146],{"id":86,"by":87,"text":88},{"value":148,"source":79,"evidence":149},"dictzip",[150],{"id":151,"by":152,"text":153},"9478174","101914","Reminds me of dictzip. http:\u002F\u002Fsourceforge.net\u002Fprojects\u002Fdict\u002Ffiles\u002Flatest\u002Fdownload",{"value":155,"source":79,"evidence":156},"Strigi and JStream",[157],{"id":158,"by":159,"text":160},"9480370","frik","Strigi desktop search (2006) implementation was very powerful (http:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FStrigi). It was possible to search through an compressed archive inside an compressed…",{"comments":33,"answered":41,"key":162},[163,164,165],{"id":50,"text":51},{"id":129,"text":130},{"id":106,"text":107},{"commenters":167,"max_depth":168,"maker_replies":33,"first_comment_mins":169,"lifespan_hours":170},24,4,44,57,[172,174,176,178,180,182,183,187,191,195,198,201,204,205,206,208,209,210,212,214,216,220,224,227,231,234,236,238,241,245,249,252,254],[82,34,41,173,83,34],"project\u002Fpositive\u002Fidea",[86,32,42,175,87,34],"project\u002Fneutral\u002Fidea",[50,32,34,177,11,32],"maker\u002Fneutral\u002Fexecution",[103,32,42,179,87,34],"project\u002Fneutral\u002Fdesign",[129,32,34,181,11,32],"maker\u002Fpositive\u002Fdesign",[106,32,34,181,11,32],[184,185,42,175,186,34],"9478208",2,"cjbprime",[188,42,34,189,190,34],"9477432","other\u002Fneutral\u002Fnone","PaulHoule",[192,42,34,193,194,34],"9478413","other\u002Fneutral\u002Fidea","gojomo",[196,42,34,189,197,34],"9478888","mappu",[199,42,34,189,200,34],"9479276","voltagex_",[202,168,34,189,203,34],"9478338","covi",[151,41,34,189,152,34],[136,33,42,175,137,34],[59,33,34,207,11,32],"maker\u002Fneutral\u002Fdesign",[112,35,42,179,113,34],[116,35,34,181,11,32],[94,18,42,211,95,34],"project\u002Fneutral\u002Fperformance",[53,18,34,213,11,32],"maker\u002Fpositive\u002Fperformance",[140,215,34,189,141,34],9,[217,218,34,193,219,34],"9477211",10,"est",[221,218,34,222,223,34],"9477296","commenter\u002Fneutral\u002Fidea","jimktrains2",[225,218,34,222,226,34],"9477426","xfs",[228,229,34,189,230,34],"9477822",11,"listic",[232,229,34,189,233,34],"9477907","sp332",[235,229,34,189,230,34],"9478154",[237,229,34,189,233,34],"9480290",[239,229,34,189,240,34],"9478290","_delirium",[242,243,34,193,244,34],"9478570",12,"mianos",[246,247,34,189,248,34],"9477585",13,"sultanofsaltin",[250,247,34,189,251,34],"9479390","olso",[253,247,34,189,159,34],"9480470",[158,255,34,189,159,34],14,[],[258,263,268,273,278,283],{"id":259,"name":260,"points":261,"category":262},"9498874","kiUi",136,"languages",{"id":264,"name":265,"points":266,"category":267},"9485714","nvParse",123,"data",{"id":269,"name":270,"points":271,"category":272},"9491978","My Hacker News",78,"consumer_social",{"id":274,"name":275,"points":276,"category":277},"9476603","PID",73,"learning_science",{"id":279,"name":280,"points":281,"category":282},"9502015","Estimote Indoor Location SDK",71,"infrastructure",{"id":284,"name":285,"points":286,"category":20},"9500823","Onboarded",54,false,"v2",{"id":290,"name":5,"line":6,"ai":287,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":291,"top":18,"kw":292,"maker":11,"domain":10,"tier":19,"attempts":34,"strip":293,"url":9,"mix":294,"praise":78,"objection":91,"sub":21},9477032,0.11,[24,25,26,27,28],"530300300000030303000000000000000",[32,33,34,34],1793990923960]