[{"data":1,"prerenderedAt":394},["ShallowReactive",2],{"project-25889420":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":34,"aspects":35,"strip":43,"post_text":45,"what":46,"said":96,"maker_in_thread":219,"shape":228,"grid":232,"attempts":355,"same_week":356,"maker_shows":33,"maker_hidden":386,"reader_version":387,"card":388},"25889420","Gutensearch","Full text search Project Gutenberg (60m paragraphs)","title","Show HN: Full text search Project Gutenberg (60m paragraphs)","https:\u002F\u002Fgutensearch.com\u002F","gutensearch.com","gutensearch",1611464298,2021,202,57,229,2533,10,"notable","learning_science","full text search for literature corpus","web_app",[24,25,26,27],"full text search","project gutenberg","postgres full text search","literary corpus","mixed",-0.147,{"positive":31,"neutral":32,"mixed":33,"critical":32},5,9,1,24,{"execution":36,"idea":38,"performance":40,"design":42},{"positive":33,"critical":37},6,{"positive":39,"critical":33},4,{"positive":41,"critical":33},0,{"positive":41,"critical":33},[31,41,44,31,41,44,41,33,41,44,41,33,41,41,41,41,44,41,41,41,41,41,41,31,41,31,44,44,33,33,33,44,44,31,44,33,41,33,41,33],3,null,{"problem":47,"audience":58,"pricing":45,"motivation":68,"time_to_build":45,"stack":74,"openness":93,"stage":94,"team_size":95},{"value":48,"source":49,"evidence":50},"Finding exact passages across Project Gutenberg books, where web quote sites dominate results.","commenter",[51,55],{"id":52,"by":53,"text":54},"25893237","loughnane","I love this. So often when I search for passages I get bombarded with links to quote websites. This is much closer to what I’m looking for. EDIT: and it’s going to be open…",{"id":56,"by":11,"text":57},"25910252","Thanks! I had the exact same problem and eventually it got me to do something about it. It is particularly bad with writers from antiquity or with a lot of popular appeal. I've…",{"value":59,"source":49,"evidence":60},"Literature scholars and corpus researchers.",[61,65],{"id":62,"by":63,"text":64},"25890522","tkgally","This is a very useful tool. I’ve known people doing research on the evolution of grammar, vocabulary, literary style, etc. who use only small subsets of the Project Gutenberg…",{"id":66,"by":63,"text":67},"25901822","I’m afraid it’s been a few years since I heard their research presentations. The only specific tool I remember is AntConc [0], as I happen to know the developer. (I teach at a…",{"value":69,"source":70,"evidence":71},"Maker wanted a searchable Gutenberg corpus; a commenter said they wanted it open sourced, and the maker has begun publishing the code.","maker",[72,73],{"id":52,"by":53,"text":54},{"id":56,"by":11,"text":57},[75,84],{"value":76,"source":70,"evidence":77},"PostgreSQL",[78,81],{"id":79,"by":11,"text":80},"25890660","Thank you for taking the time to lay out feature requests in such details! I really appreciate it. The current search box is a wrapper around Postgres phraseto_tsquery [1] whilst…",{"id":82,"by":11,"text":83},"25890383","Thanks for noticing! To be more specific, \"what is a\" are stop words and \"cynic\" is very common, so a lot of rows are returned (see my other comment). ts_rank takes too long to…",{"value":85,"source":70,"evidence":86},"Dash",[87,90],{"id":88,"by":11,"text":89},"25891237","Thanks for the suggestions! I will try them next. > Looking at your where clause, did you index language and textsearchable_index_col together ? If not, you are giving the…",{"id":91,"by":11,"text":92},"25890480","Thanks for the bug report! It has to do (I think) with Dash's columnar layout which unfolds the menu over the next few columns at least in Chrome. The quick workaround I found…","open_source","launched","unknown",{"praise":97,"objections":114,"requests":145,"questions":177,"alternatives":192},[98,107],{"value":99,"source":49,"evidence":100},"Useful for searching passages across the corpus.",[101,102,103],{"id":62,"by":63,"text":64},{"id":52,"by":53,"text":54},{"id":104,"by":105,"text":106},"25893959","Ninjinka","I enjoy being able to grep the 200,000 books I have downloaded as part of the Pile that was on HN a while back. It allowed me to do things like \"show me all the paragraphs where…",{"value":108,"source":49,"evidence":109},"Would be good as a benchmark for search engines.",[110],{"id":111,"by":112,"text":113},"25890186","tornato7","I'd love to see this dataset used as a performance and relevance benchmark for different search engines!",[115,123,138],{"value":116,"source":70,"evidence":117},"Slow ranking on common queries causes timeouts.",[118,120],{"id":119,"by":11,"text":83},"25890368",{"id":121,"by":11,"text":122},"25890290","That was definitely part of the original plan! I spotted two other attempts [1] [2] here using BERT and ElasticSearch respectively. The main performance issue with the Postgres…",{"value":124,"source":49,"evidence":125},"Language selection dropdown breaks on some browsers.",[126,130,134],{"id":127,"by":128,"text":129},"25890458","gerardnll","I cannot choose another language, some weird css\u002Fhtml issue happens that hides all other fields. Looks like an overflow:hidden hides the possible selections.",{"id":131,"by":132,"text":133},"25893931","DerWOK","Hm. I cannot do language selection on iOS Safari. Even not with the mentioned workaround of first typing the language name prefix.",{"id":135,"by":136,"text":137},"25890776","Kaknut","This is indeed very useful. But I think there are some bugs like when I click on dropdown to choose language it breaks (malfunction)",{"value":139,"source":49,"evidence":140},"Interface is cluttered.",[141],{"id":142,"by":143,"text":144},"25891060","JensRantil","Nice project! You might want to ask a designer to give it a look. UI is a little cluttered.",[146,150,160,166],{"value":147,"source":49,"evidence":148},"Support for wildcards, phrases and Boolean search syntax.",[149],{"id":62,"by":63,"text":64},{"value":151,"source":49,"evidence":152},"Add a GET query URL parameter and an API.",[153,157],{"id":154,"by":155,"text":156},"25891366","gebt","What useful thing! It would be much better if it was possible to pass search query using GET requests. e.g. https:\u002F\u002Fgutensearch.com\u002F?q=avicenna",{"id":158,"by":11,"text":159},"25891683","An API is already on my roadmap! I couldn't quite figure the state of passing query parameters in the URL with Dash, though.",{"value":161,"source":70,"evidence":162},"Add Chinese language support.",[163],{"id":164,"by":11,"text":165},"25891052","Thank you for the suggestion! The demo looks great and I'm curious to see how they automatically handle language. It would be nice to add support for Chinese which is the great…",{"value":167,"source":49,"evidence":168},"Consider Manticore or MeiliSearch as a search backend.",[169,173],{"id":170,"by":171,"text":172},"25890828","snikolaev","Hi. Nice project! I don't know why there's no BM25 (or at least SOME TF-IDF implementation) in Postgres FTS, but if you decide you need it (and more languages support and…",{"id":174,"by":175,"text":176},"25891000","StavrosK","I'm curious if you ever tried MeiliSearch for this, I tried it recently for something unrelated and had a very good experience with it. Since you have the corpus already, it…",[178,185],{"value":179,"source":49,"evidence":180},"How is the search parsing done and why do some queries fail?",[181],{"id":182,"by":183,"text":184},"25890324","kristopolous","\"What is a cynic\" the famous line from Lady Windmere's Fan comes up empty. You can find it here: https:\u002F\u002Fwww.gutenberg.org\u002Ffiles\u002F790\u002F790-h\u002F790-h.htm The next line I can find…",{"value":186,"source":49,"evidence":187},"What hardware hosts the Postgres instance?",[188],{"id":189,"by":190,"text":191},"25890723","karterk","What kind of hardware are you using to host the Postgres instance?",[193,201,205,212],{"value":194,"source":49,"evidence":195},"grep over a local copy of the corpus",[196,200],{"id":197,"by":198,"text":199},"25891022","Jabbles","Surely someone doing serious research would download the entire corpus and use grep (or equivalent)? It wouldn't have the same nice interface and searches may take several…",{"id":104,"by":105,"text":106},{"value":202,"source":49,"evidence":203},"Manticore Search",[204],{"id":170,"by":171,"text":172},{"value":206,"source":49,"evidence":207},"Elasticsearch",[208],{"id":209,"by":210,"text":211},"25895431","ilaksh","I am sure the Postgres full text search is pretty good by now. But I always expect projects like these to eventually reluctantly switch over to Elasticsearch in order to handle…",{"value":213,"source":49,"evidence":214},"recoll",[215],{"id":216,"by":217,"text":218},"25895275","rahimnathwani","This is really cool. Something like this should exist. It seems like you could do it more easily, include all recent additions, and have faster search responses: 1. Mirror the…",{"comments":220,"answered":221,"key":222},17,7,[223,226,227],{"id":224,"text":225},"25891068","Thank you for your generous offer of help! I look forward to taking it up (may take a while as I'm about to move countries and quarantine). In particular I love that one of the…",{"id":79,"text":80},{"id":164,"text":165},{"commenters":229,"max_depth":31,"maker_replies":220,"first_comment_mins":230,"lifespan_hours":231},30,166,148,[233,235,237,239,243,245,246,249,251,252,254,256,259,261,263,264,266,268,269,270,272,275,277,280,283,285,288,290,293,296,299,301,304,306,307,308,309,310,313,314,316,317,320,323,325,328,330,332,336,339,341,343,344,346,348,350,353],[170,41,31,234,171,41],"project\u002Fpositive\u002Fexecution",[224,41,41,236,11,33],"commenter\u002Fneutral\u002Fnone",[238,41,41,236,171,41],"25891293",[240,41,44,241,242,41],"25891594","project\u002Fneutral\u002Fexecution","yorwba",[62,33,31,244,63,41],"project\u002Fpositive\u002Fidea",[79,33,41,236,11,33],[247,33,41,236,248,41],"25890765","iiv",[174,33,44,250,175,41],"project\u002Fneutral\u002Fperformance",[164,33,41,236,11,33],[253,33,41,236,175,41],"25891064",[197,33,33,255,198,41],"project\u002Fcritical\u002Fidea",[257,33,41,236,258,41],"25893411","andai",[66,33,41,260,63,41],"other\u002Fneutral\u002Fnone",[154,262,44,241,155,41],2,[158,262,41,241,11,33],[265,262,44,241,155,41],"25891799",[182,44,33,267,183,41],"project\u002Fcritical\u002Fexecution",[119,44,41,250,11,33],[82,44,41,250,11,33],[271,44,41,260,183,41],"25890685",[273,44,41,274,11,33],"25890959","commenter\u002Fpositive\u002Fnone",[276,44,41,236,242,41],"25891021",[278,44,41,260,279,41],"25892720","thaumaturgy",[281,44,44,250,282,41],"25891090","talolard",[88,44,41,284,11,33],"commenter\u002Fneutral\u002Fperformance",[286,39,41,260,287,41],"25891485","_Microft",[289,39,41,260,11,33],"25891531",[291,39,41,260,292,41],"25891631","SahAssar",[294,39,41,260,295,41],"25891664","IndySun",[297,39,41,260,298,41],"25892359","robin_reala",[300,39,41,260,287,41],"25894404",[302,39,41,303,295,41],"25967565","other\u002Fpositive\u002Fnone",[305,39,41,260,295,41],"25967560",[52,31,31,244,53,41],[56,31,41,274,11,33],[111,37,31,244,112,41],[121,37,41,250,11,33],[311,37,44,250,312,41],"25890612","ngrilly",[189,37,44,250,190,41],[315,37,41,250,11,33],"25890796",[216,221,33,267,217,41],[209,318,33,319,210,41],8,"project\u002Fcritical\u002Fperformance",[321,32,33,267,322,41],"25893804","abhayhegde",[324,32,41,236,11,33],"25910267",[326,32,44,327,322,41],"25911838","project\u002Fmixed\u002Fexecution",[329,32,41,236,11,33],"25912255",[104,18,44,331,105,41],"project\u002Fneutral\u002Fidea",[333,334,31,244,335,41],"25892834",11,"nestorD",[337,334,44,331,338,41],"25892941","toomuchtodo",[340,334,41,236,335,41],"25894610",[127,342,33,267,128,41],12,[91,342,41,236,11,33],[131,345,33,267,132,41],13,[135,347,33,267,136,41],14,[349,347,41,236,11,33],"25890825",[142,351,33,352,143,41],15,"project\u002Fcritical\u002Fdesign",[354,351,41,274,11,33],"25891096",[],[357,362,366,371,376,381],{"id":358,"name":359,"points":360,"category":361},"25872922","Iconduck",528,"creative",{"id":363,"name":364,"points":365,"category":361},"25885760","Filmulator",282,{"id":367,"name":368,"points":369,"category":370},"25902165","I made an alarm clock",271,"hardware",{"id":372,"name":373,"points":374,"category":375},"25879238","Rysolv",250,"web_saas",{"id":377,"name":378,"points":379,"category":380},"25870482","Timeflake",205,"dev_tools",{"id":382,"name":383,"points":384,"category":385},"25934270","vnav.io",198,"games",false,"v2",{"id":389,"name":5,"line":6,"ai":386,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":390,"top":18,"kw":391,"maker":11,"domain":10,"tier":19,"attempts":41,"strip":392,"url":9,"mix":393,"praise":99,"objection":116,"sub":21},25889420,-0.15,[24,25,26,27],"5035030103010000300000050533111335310101",[31,32,33,32],1794036601921]