[{"data":1,"prerenderedAt":426},["ShallowReactive",2],{"project-38717114":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":36,"aspects":37,"strip":45,"post_text":47,"what":48,"said":87,"maker_in_thread":202,"shape":210,"grid":215,"attempts":389,"same_week":390,"maker_shows":34,"maker_hidden":418,"reader_version":419,"card":420},"38717114","Talk2Arxiv","Talk to any ArXiv paper just by changing the URL","title","Show HN: Talk to any ArXiv paper just by changing the URL","https:\u002F\u002Fgithub.com\u002Fevanhu1\u002Ftalk2arxiv","github.com","evanhu_",1703134100,2023,194,73,288,3109,10,"notable","learning_science","research paper chat","web_app",[24,25,26,27,28],"arxiv","rag","pdf chat","research papers","llm","curious",-0.051,{"positive":32,"neutral":33,"mixed":34,"critical":35},4,18,1,6,29,{"execution":38,"performance":40,"idea":43,"privacy":44},{"positive":39,"critical":34},2,{"positive":41,"critical":42},0,3,{"positive":41,"critical":34},{"positive":41,"critical":41},[41,41,41,41,41,41,41,41,46,42,46,41,42,42,42,41,41,41,42,41,34,42,34,46,34,42,34,46,41,42,42,42,42,42,41,41,41,42,34,42],5,"Hello HN, Talk2Arxiv is a small open-source RAG application I've been building for a few weeks. To use it just prepend any arxiv.org link with 'talk2' to load the paper into a responsive RAG chat application (e.g. www.arxiv.org\u002Fabs\u002F1706.03762 -> www.talk2arxiv.org\u002Fabs\u002F1706.03762).\n\nAll implementation details are in the GitHub. Currently, because I've opted to extract text from the PDF of the paper rather than reading the LaTeX source code (since I wanted to build a more generic PDF RAG in the process), it struggles with symbolic text \u002F mathematics, and sometimes fails to retrieve the correct c",{"problem":49,"audience":55,"pricing":55,"motivation":56,"time_to_build":55,"stack":60,"openness":84,"stage":85,"team_size":86},{"value":50,"source":51,"evidence":52},"Reading and asking questions about arXiv papers","maker",[53],{"id":54},"post",null,{"value":57,"source":51,"evidence":58},"Build a more generic PDF RAG; make arXiv papers chattable",[59],{"id":54},[61,70,76],{"value":62,"source":51,"evidence":63},"GROBID for PDF parsing",[64,67],{"id":65,"by":11,"text":66},"38717284","I spent forever looking at various PDF parsing solutions like Unstructured, and eventually stumbled across GROBID, which was an absolute perfect fit since it's entirely made for…",{"id":68,"by":11,"text":69},"38717482","Definitely I'll move to the LaTeX source code instead of a PDF backend since that allows better support for non textual data that gets poorly scraped by GROBID. That is a really…",{"value":71,"source":51,"evidence":72},"Cohere Reranker",[73],{"id":74,"by":11,"text":75},"38717445","Thank you :). I updated the README to have some more explanation of the steps. The chunking algorithm chunks by logical section (intro, abstract, authors, etc.) and also utilizes…",{"value":77,"source":78,"evidence":79},"Pinecone vector database","commenter",[80],{"id":81,"by":82,"text":83},"38767324","soliax","Awesome project. What metric and dimension size did you set on Pinecone?","open_source","beta","unknown",{"praise":88,"objections":111,"requests":134,"questions":164,"alternatives":176},[89,100],{"value":90,"source":78,"evidence":91},"The URL-prefix trick is clever and useful",[92,96],{"id":93,"by":94,"text":95},"38717371","katella","Idk where this changing the url thing started but I really like it.",{"id":97,"by":98,"text":99},"38717778","osteele","It works well with bookmarklets. This swizzles between arxiv.org \u003C-> www.talk2arxiv.org. I've now added it to my Favorites bar, next to arxiv.org\u002Fabs \u003C-> arxiv.org\u002Fpdf and…",{"value":101,"source":78,"evidence":102},"Works well and is a cool solution",[103,107],{"id":104,"by":105,"text":106},"38717390","gorkish","Very nice; appears to work well. Just an FYI that I did get a couple errors where the max context length was exceeded, one using the demo summarization task as the first query. I…",{"id":108,"by":109,"text":110},"38717177","skeptrune","This is the first time I have seen someone use GROBID. It seems like an incredibly cool solution",[112,116,123],{"value":113,"source":51,"evidence":114},"Struggles with math and symbolic text from PDF extraction",[115],{"id":54},{"value":117,"source":78,"evidence":118},"Context length errors on long papers",[119,120],{"id":104,"by":105,"text":106},{"id":121,"by":105,"text":122},"38737718","This is the paper that would reliably trigger context overflows. https:\u002F\u002Farxiv.org\u002Fabs\u002F1811.03116 It otherwise did an admirable job on this brainbender.",{"value":124,"source":78,"evidence":125},"Requests for OpenAI API key entry raises privacy concerns",[126,130],{"id":127,"by":128,"text":129},"38717475","willvarfar","Will it become normal to paste your OpenAI keys into a website and will it become the new curl-sudo-bash or checking cloud credentials into github?",{"id":131,"by":132,"text":133},"38717876","imjonse","All while calling the solution open source.",[135,146,157],{"value":136,"source":78,"evidence":137},"Move to LaTeX source or HTML instead of PDF parsing",[138,142],{"id":139,"by":140,"text":141},"38717464","pushfoo","You might be able to drop the PDF backend since they're close to getting HTML running well: https:\u002F\u002Fnews.ycombinator.com\u002Fitem?id=38713215 Using that might be easier than a…",{"id":143,"by":144,"text":145},"38717811","Mandelmus","Yes, using the LaTeX source code (or HTML, once that becomes reliable and widely used) should be much more robust than PDF parsing.",{"value":147,"source":78,"evidence":148},"Support for bioRxiv",[149,153],{"id":150,"by":151,"text":152},"38717286","aendruk","Any plans for bioRxiv?",{"id":154,"by":155,"text":156},"38719503","gtsnexp","This is great! We could also point it to https:\u002F\u002Fbiorxiv.org\u002F Awesome work!",{"value":158,"source":78,"evidence":159},"Local LLM support such as Ollama",[160],{"id":161,"by":162,"text":163},"38717665","killingtime74","What's the easiest way to adapt this to local LLMs like Ollama or Lama.cpp",[165,172],{"value":166,"source":78,"evidence":167},"Can it answer questions about references?",[168],{"id":169,"by":170,"text":171},"38725518","friend_and_foe","Is it able to answer questions about references?",{"value":173,"source":78,"evidence":174},"Which vector dimension and metric were used in Pinecone?",[175],{"id":81,"by":82,"text":83},[177,188,195],{"value":178,"source":78,"evidence":179},"Marker and Nougat for PDF to markdown conversion",[180,184],{"id":181,"by":182,"text":183},"38718269","Koaisu","Maybe you could use: https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fnougat\u002Ftree\u002Fmain or https:\u002F\u002Fgithub.com\u002FVikParuchuri\u002Fmarker Both are tools to convert pdfs into Latex or Markup with latex…",{"id":185,"by":186,"text":187},"38717394","pugio","I've spent the last couple weeks diving into various PDF parsing solutions for scientific documents. GROBID is pretty cool, but it made some mistakes when trying to parse (I…",{"value":189,"source":78,"evidence":190},"ar5iv HTML versions of arXiv papers",[191],{"id":192,"by":193,"text":194},"38717554","arbitrandomuser","I wonder if they knew that they could get html versions of the paper by just changing the link from ...arxiv.. to ar5iv..",{"value":196,"source":78,"evidence":197},"Cheshire Cat and Ollama web UI for chatting with PDFs",[198],{"id":199,"by":200,"text":201},"38721294","lolinder","There are a bunch of little projects that do this. It's on the roadmap for ollama-webui (but not implemented yet), and Ollama published a guest blog post with a simple…",{"comments":203,"answered":39,"key":204},9,[205,206,209],{"id":68,"text":69},{"id":207,"text":208},"38717421","Thank you! Thanks for pointing that out, since the underlying RAG is rather naive (simple embedding cosine similarity lookup, as opposed to knowledge graph \u002F advanced…",{"id":65,"text":66},{"commenters":211,"max_depth":212,"maker_replies":203,"first_comment_mins":213,"lifespan_hours":214},45,8,15,159,[216,218,221,224,226,229,231,234,236,239,241,244,247,250,253,255,258,261,263,265,267,269,270,273,275,278,280,283,285,286,289,292,294,297,299,301,304,306,310,312,314,315,316,317,319,321,323,324,326,329,331,333,336,340,342,343,345,347,349,351,353,355,358,361,364,367,370,372,375,378,380,384,386],[93,41,41,217,94,41],"other\u002Fneutral\u002Fnone",[219,41,41,217,220,41],"38717402","mulmen",[222,41,41,217,223,41],"38718708","quickthrower2",[225,41,41,217,220,41],"38719050",[227,41,41,228,223,41],"38719247","commenter\u002Fneutral\u002Fnone",[230,41,41,228,220,41],"38720339",[232,41,41,228,233,41],"38726701","tempaccount420",[235,41,41,228,220,41],"38758145",[237,41,41,217,238,41],"38717403","ReactiveJelly",[240,41,41,217,94,41],"38785756",[242,41,41,217,243,41],"38717406","Aachen",[245,41,41,217,246,41],"38717693","theolivenbaum",[248,41,41,217,249,41],"38718864","aembleton",[251,41,41,217,252,41],"38721910","KomoD",[97,41,46,254,98,41],"project\u002Fpositive\u002Fnone",[256,41,41,217,257,41],"38720755","tinix",[259,41,41,217,260,41],"38721899","kordlessagain",[139,34,42,262,140,41],"project\u002Fneutral\u002Fexecution",[68,34,41,264,11,34],"maker\u002Fneutral\u002Fexecution",[143,34,46,266,144,41],"project\u002Fpositive\u002Fexecution",[268,34,41,217,109,41],"38717813",[181,34,42,262,182,41],[271,34,41,217,272,41],"38718427","acqq",[161,39,42,274,162,41],"project\u002Fneutral\u002Fnone",[276,39,42,274,277,41],"38717706","chpatrick",[279,39,41,228,162,41],"38717786",[281,39,42,274,282,41],"38719265","addandsubtract",[284,39,41,217,277,41],"38719284",[199,39,41,217,200,41],[287,42,41,217,288,41],"38717967","magicalhippo",[290,42,41,228,291,41],"38718217","cristeigabriel",[293,42,41,228,288,41],"38718613",[295,42,41,228,296,41],"38728781","bruce343434",[298,42,42,274,288,41],"38731476",[300,42,41,217,296,41],"38718493",[302,42,41,217,303,41],"38718701","xboxnolifes",[305,42,41,217,282,41],"38719394",[307,42,34,308,309,41],"38720647","project\u002Fcritical\u002Fperformance","3abiton",[104,32,42,311,105,41],"project\u002Fmixed\u002Fperformance",[207,32,41,313,11,34],"maker\u002Fneutral\u002Fperformance",[121,32,34,308,105,41],[108,46,46,266,109,41],[65,46,41,264,11,34],[185,46,34,318,186,41],"project\u002Fcritical\u002Fexecution",[320,46,42,274,109,41],"38717679",[322,46,42,274,260,41],"38721881",[192,46,42,262,193,41],[325,46,41,264,11,34],"38717827",[327,35,34,308,328,41],"38717755","schneehertz",[330,35,41,313,11,34],"38717773",[154,332,46,254,155,41],7,[334,332,41,335,11,34],"38724345","maker\u002Fneutral\u002Fnone",[337,212,42,338,339,41],"38717370","project\u002Fneutral\u002Fidea","zzleeper",[74,212,41,341,11,34],"maker\u002Fneutral\u002Fidea",[81,203,42,274,82,41],[344,203,42,274,82,41],"38767506",[346,18,42,338,223,41],"38718720",[348,18,41,217,260,41],"38721982",[169,350,42,274,170,41],11,[150,352,42,274,151,41],12,[354,352,41,341,11,34],"38717299",[127,356,42,357,128,41],13,"project\u002Fneutral\u002Fprivacy",[359,356,41,217,360,41],"38717563","kn100",[362,356,41,217,363,41],"38717602","reidjs",[365,356,41,217,366,41],"38717606","TaylorAlexander",[368,356,41,217,369,41],"38717691","Tiberium",[371,356,41,217,282,41],"38719437",[373,356,41,217,374,41],"38720503","__s",[376,356,42,357,377,41],"38717825","tehlike",[131,356,34,379,132,41],"project\u002Fcritical\u002Fnone",[381,382,34,383,243,41],"38717454",14,"project\u002Fcritical\u002Fidea",[385,382,41,335,11,34],"38717466",[387,213,42,338,388,41],"38719112","hxypqr",[],[391,396,400,404,409,414],{"id":392,"name":393,"points":394,"category":395},"38733968","Heynote",1063,"productivity",{"id":397,"name":398,"points":399,"category":395},"38787892","Rem",555,{"id":401,"name":402,"points":403,"category":395},"38747710","Storyteller",338,{"id":405,"name":406,"points":407,"category":408},"38772382","Whataaabout.com",235,"consumer_social",{"id":410,"name":411,"points":412,"category":413},"38730143","Emu2",153,"ai",{"id":415,"name":416,"points":417,"category":395},"38753911","Weeks of Your",145,false,"v2",{"id":421,"name":5,"line":6,"ai":418,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":422,"top":18,"kw":423,"maker":11,"domain":10,"tier":19,"attempts":41,"strip":424,"url":9,"mix":425,"praise":90,"objection":113,"sub":21},38717114,-0.05,[24,25,26,27,28],"0000000053503330003013151315033333000313",[32,33,34,35],1794036646501]