Gutensearch
Learning and science
full text search for literature corpus
SHOW HN2021
202points
57comments
#229of 2,533 in 2021
Top 10%of 2021 by points
How HN took it
MixedNo clear lean
5 positive, 9 neutral or asking, 1 mixed, 9 critical, out of 24 comments about the project. Thread score -0.15.
Execution: 1 for, 6 againstIdea: 4 for, 1 againstPerformance: 0 for, 1 againstDesign: 0 for, 1 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Useful for searching passages across the corpus.
From 3 comments
This is a very useful tool. I’ve known people doing research on the evolution of grammar, vocabulary, literary style, etc. who use only small subsets of the Project Gutenberg…
tkgally on HNI love this. So often when I search for passages I get bombarded with links to quote websites. This is much closer to what I’m looking for. EDIT: and it’s going to be open…
loughnane on HNI enjoy being able to grep the 200,000 books I have downloaded as part of the Pile that was on HN a while back. It allowed me to do things like "show me all the paragraphs where…
Ninjinka on HN
Would be good as a benchmark for search engines.
From 1 comment
I'd love to see this dataset used as a performance and relevance benchmark for different search engines!
tornato7 on HN
Objections
Slow ranking on common queries causes timeouts.
From 2 comments
Thanks for noticing! To be more specific, "what is a" are stop words and "cynic" is very common, so a lot of rows are returned (see my other comment). ts_rank takes too long to…
gutensearch on HNThat was definitely part of the original plan! I spotted two other attempts [1] [2] here using BERT and ElasticSearch respectively. The main performance issue with the Postgres…
gutensearch on HN
Language selection dropdown breaks on some browsers.
From 3 comments
I cannot choose another language, some weird css/html issue happens that hides all other fields. Looks like an overflow:hidden hides the possible selections.
gerardnll on HNHm. I cannot do language selection on iOS Safari. Even not with the mentioned workaround of first typing the language name prefix.
DerWOK on HNThis is indeed very useful. But I think there are some bugs like when I click on dropdown to choose language it breaks (malfunction)
Kaknut on HN
Interface is cluttered.
From 1 comment
Nice project! You might want to ask a designer to give it a look. UI is a little cluttered.
JensRantil on HN
Requests
Support for wildcards, phrases and Boolean search syntax.
From 1 comment
This is a very useful tool. I’ve known people doing research on the evolution of grammar, vocabulary, literary style, etc. who use only small subsets of the Project Gutenberg…
tkgally on HN
Add a GET query URL parameter and an API.
From 2 comments
What useful thing! It would be much better if it was possible to pass search query using GET requests. e.g. https://gutensearch.com/?q=avicenna
gebt on HNAn API is already on my roadmap! I couldn't quite figure the state of passing query parameters in the URL with Dash, though.
gutensearch on HN
Add Chinese language support.
From 1 comment
Thank you for the suggestion! The demo looks great and I'm curious to see how they automatically handle language. It would be nice to add support for Chinese which is the great…
gutensearch on HN
Consider Manticore or MeiliSearch as a search backend.
From 2 comments
Hi. Nice project! I don't know why there's no BM25 (or at least SOME TF-IDF implementation) in Postgres FTS, but if you decide you need it (and more languages support and…
snikolaev on HNI'm curious if you ever tried MeiliSearch for this, I tried it recently for something unrelated and had a very good experience with it. Since you have the corpus already, it…
StavrosK on HN
Questions
How is the search parsing done and why do some queries fail?
From 1 comment
"What is a cynic" the famous line from Lady Windmere's Fan comes up empty. You can find it here: https://www.gutenberg.org/files/790/790-h/790-h.htm The next line I can find…
kristopolous on HN
What hardware hosts the Postgres instance?
From 1 comment
What kind of hardware are you using to host the Postgres instance?
karterk on HN
Alternatives named
grep over a local copy of the corpus
From 2 comments
Surely someone doing serious research would download the entire corpus and use grep (or equivalent)? It wouldn't have the same nice interface and searches may take several…
Jabbles on HNI enjoy being able to grep the 200,000 books I have downloaded as part of the Pile that was on HN a while back. It allowed me to do things like "show me all the paragraphs where…
Ninjinka on HN
Manticore Search
From 1 comment
Hi. Nice project! I don't know why there's no BM25 (or at least SOME TF-IDF implementation) in Postgres FTS, but if you decide you need it (and more languages support and…
snikolaev on HN
Elasticsearch
From 1 comment
I am sure the Postgres full text search is pretty good by now. But I always expect projects like these to eventually reluctantly switch over to Elasticsearch in order to handle…
ilaksh on HN
recoll
From 1 comment
This is really cool. Something like this should exist. It seems like you could do it more easily, include all recent additions, and have faster search responses: 1. Mirror the…
rahimnathwani on HN