zindex
Dev tools
command line search tool
SHOW HN2015
How HN took it
1 positive, 6 neutral or asking, 0 mixed, 0 critical, out of 7 comments about the project. Thread score +0.11.
Idea: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Useful idea for searching compressed logs; practical checkpoint approach
From 2 comments
The article's method is nice, it allows you to decompress portions of the file. I thought this would be about the FM index. The FM index allows you to search what is essentially…
jules on HNI have been wanting this for a long time, although my primary use case would be indexing a particular field. It is nice that you were able to make it work with ordinary gzip…
xaa on HN
Objections
Indexing speed on large files is slow because it must scan sequentially
From 2 comments
About how long does it take to create the index, maybe compared to a zgrep? Could it be parallelized with something like `parallel --pipe`? I haven't tried yet, but this seems…
nattaylor on HNThanks! For my logfiles it runs through them at around 3MB/sec (that's 3MB of compressed file per second). Most of it is probably I/O, my giant logs are on NFS. I run the indexer…
mattgodbolt on HN
Requests
Support indexing by column name or field in delimited files
From 2 comments
Smart. I will see for myself in a minute how much space such an approach takes; I would imagine the compressor size gets nontrivially large on big files. Well, my exact use case,…
xaa on HNAs of just now, zindex and zq now support field-based indexing. Hope this helps your use case!
mattgodbolt on HN
Fuzzy matching such as wildcard search for MAC addresses
From 2 comments
I only wish that the matching could be 'fuzzy' in easily defined ways. My use case is MAC addresses. So, if I search for 'dead.beef.cafe', this would be perfect if it could know…
hackmiester on HNGreat idea. zq uses the SQLite index built by zindex; so in theory I can modify it to use a 'WHERE index LIKE xxx' instead of 'WHERE index=xxx' -- then SQLish wildcards would be…
mattgodbolt on HN
Questions
How the tool finds the right block and offset in arbitrary-size gzip blocks
From 2 comments
I have been wanting this for a long time, although my primary use case would be indexing a particular field. It is nice that you were able to make it work with ordinary gzip…
xaa on HNFantastic! I had hoped others would have the same problem as me. I didn't have the luxury of being able to change my many input gzip files (unlike BGZF). The trick is to store…
mattgodbolt on HN
Whether multiple indexes can exist on one gzip file
From 2 comments
Smart. I will see for myself in a minute how much space such an approach takes; I would imagine the compressor size gets nontrivially large on big files. Well, my exact use case,…
xaa on HNShort reply as I'm on a phone, but thanks! I can add an option to pick the nth field, and to skip the first X lines of the input; I think this'd cover your use case! The code and…
mattgodbolt on HN
Alternatives named
FM-index and succinct data structures
From 2 comments
Why not store the log files in an FM-index? (https://en.wikipedia.org/wiki/FM-index)
dpkendal on HNSee "succinct data structures" and "wavelet trees" for a similar / more general idea. news.ycombinator.com/item?id=7079427 I also must thank Edward Kmett for introducing me to…
dangirsh on HN
BGZF blocked gzip format
From 1 comment
I have been wanting this for a long time, although my primary use case would be indexing a particular field. It is nice that you were able to make it work with ordinary gzip…
xaa on HN
dictzip
From 1 comment
Reminds me of dictzip. http://sourceforge.net/projects/dict/files/latest/download
101914 on HN
Strigi and JStream
From 1 comment
Strigi desktop search (2006) implementation was very powerful (http://en.wikipedia.org/wiki/Strigi). It was possible to search through an compressed archive inside an compressed…
frik on HN