[{"data":1,"prerenderedAt":333},["ShallowReactive",2],{"project-42093511":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":36,"aspects":37,"strip":41,"post_text":43,"what":44,"said":60,"maker_in_thread":157,"shape":169,"grid":173,"attempts":294,"same_week":295,"maker_shows":35,"maker_hidden":325,"reader_version":326,"card":327},"42093511","HTML-to-Markdown","convert entire websites to Markdown with Golang\u002FCLI","title","Show HN: HTML-to-Markdown – convert entire websites to Markdown with Golang\u002FCLI","https:\u002F\u002Fgithub.com\u002FJohannesKaufmann\u002Fhtml-to-markdown","github.com","JohannesKauf",1731145688,2024,360,47,138,3515,4,"legend","dev_tools","html to markdown converter","library",[24,25,26,27,28],"html to markdown","golang","markdown converter","web scraping","html conversion","loved",0.325,{"positive":32,"neutral":33,"mixed":34,"critical":35},7,13,0,1,21,{"execution":38,"idea":39},{"positive":34,"critical":35},{"positive":40,"critical":34},3,[34,34,35,34,40,40,34,34,34,42,34,40,34,42,40,40,42,40,34,40,34,34,34,34,34,34,40,34,42,42,40,40,40,34,34,40,34,34,42,42],5,"Hey HN!\n\nI originally built “html-to-markdown” back in 2018 (while still in high school) to handle complex HTML conversions where other libraries struggled.\n\nNow, I’ve released v2 — a complete rewrite designed to handle even more edge cases. It supports entire websites with a high accuracy.\n\nExample use: I’ve used it in my RSS reader to strip HTML down to clean Markdown, similar to the \"Reader Mode\" in your Browser.\n\nIt can be used as a Golang package or as an CLI.\n\nGive it a try & tell me what edge cases you encounter!",{"problem":45,"audience":45,"pricing":45,"motivation":46,"time_to_build":45,"stack":52,"openness":57,"stage":58,"team_size":59},null,{"value":47,"source":48,"evidence":49},"Originally built in 2018 to handle complex HTML conversions where other libraries struggled.","maker",[50],{"id":51},"post",[53],{"value":54,"source":48,"evidence":55},"Go",[56],{"id":51},"open_source","launched","unknown",{"praise":61,"objections":84,"requests":91,"questions":109,"alternatives":124},[62,74],{"value":63,"source":64,"evidence":65},"Useful tool for feeding LLMs and building AI apps.","commenter",[66,70],{"id":67,"by":68,"text":69},"42093645","cpursley","This is really nice, especially for feeding LLMs web page data (they generally understand markdown well). I built something similar for the Elixir world but it’s much more…",{"id":71,"by":72,"text":73},"42093769","paradite","I have been using these two: https:\u002F\u002Ffarnots.github.io\u002FRedditToMarkdown\u002F https:\u002F\u002Furltomarkdown.com\u002F Incredibly useful for leveraging LLMs and building AI apps.",{"value":75,"source":64,"evidence":76},"Better than pandoc for HTML to Markdown in some cases.",[77,80],{"id":78,"by":11,"text":79},"42094037","Pandoc is amazing. Especially because they support so many formats. And their html to markdown converter is (in my opinion) still the best right now. But html-to-markdown is…",{"id":81,"by":82,"text":83},"42096373","lollobomb","This is nice, I tried a plugin for pandoc in the past but didn't really work well.",[85],{"value":86,"source":48,"evidence":87},"Does not handle JavaScript-driven sites; needs a headless browser.",[88],{"id":89,"by":11,"text":90},"42093925","That is unfortunately out of scope. I like the philosophy of doing one thing really well. But nowadays—with Playwright and Puppeteer—there are great choices for Browser automation.",[92,102],{"value":93,"source":64,"evidence":94},"Built-in extraction of main article content like Readability.",[95,98],{"id":96,"by":11,"text":97},"42094494","There are two parts to it: 1) convert html to markdown This is what my library specifically addresses, and I believe it handles this task robustly. There was a lot of testing…",{"id":99,"by":100,"text":101},"42094012","throwup238","This is probably out of scope for your tool but it’d be nice to have built in n-gram deduplication where the tool strips any identical content from the header and footer, like…",{"value":103,"source":64,"evidence":104},"Support for syntax-highlighted code blocks.",[105],{"id":106,"by":107,"text":108},"42095426","juliuskiesian","One of the pain points of using this kind of tools is handling syntax highlighted code blocks. How does html-to-markdown perform in such scenarios?",[110,117],{"value":111,"source":64,"evidence":112},"Does it handle JS-driven sites?",[113],{"id":114,"by":115,"text":116},"42093778","oezi","Does it also include logic to download JS-driven sites properly or is this out of scope?",{"value":118,"source":64,"evidence":119},"Is there a Kotlin or Spring library for this?",[120],{"id":121,"by":122,"text":123},"42094642","ssousa666","I have been looking for a similar lib to use in a Kotlin\u002FSpring app - any recommendations? My specific use-case does not need to support sanitizing during the HTML -> MD…",[125,136,143,150],{"value":126,"source":64,"evidence":127},"Pandoc",[128,132],{"id":129,"by":130,"text":131},"42094474","NotACracker","Pandoc http:\u002F\u002Fwww.cantoni.org\u002F2019\u002F01\u002F27\u002Fconverting-html-markdown-using-pandoc",{"id":133,"by":134,"text":135},"42093672","Savageman","I remember a long time ago I used Pandoc for this. Fresh tools and more choice is very welcome, thanks for your work!",{"value":137,"source":64,"evidence":138},"Jina Reader",[139],{"id":140,"by":141,"text":142},"42094064","miki123211","If you need this sort of thing in any other language, there's a free, no-auth, no-api-key-required, no-strings-attached API that can do this at https:\u002F\u002Fjina.ai\u002Freader\u002F You just…",{"value":144,"source":64,"evidence":145},"Turndown",[146],{"id":147,"by":148,"text":149},"42093920","yayoohooyahoo","Turndown works quite well too: https:\u002F\u002Fgithub.com\u002Fmixmark-io\u002Fturndown",{"value":151,"source":64,"evidence":152},"Mozilla Readability",[153],{"id":154,"by":155,"text":156},"42094883","bni","I used https:\u002F\u002Fgithub.com\u002Fmozilla\u002Freadability for this",{"comments":158,"answered":40,"key":159},10,[160,163,166],{"id":161,"text":162},"42094296","Thanks, Jina actually looks quite nice for use in LLMs. I also provide a REST API [1] that you can use for free (within limits). However you have get an API Key by registering…",{"id":164,"text":165},"42095731","The problem was: Doing rate limiting on the application level was not enough. Once the request hit my backend the incoming bandwidth was already consumed — and I was charged for…",{"id":167,"text":168},"42096666","Sorry, I mixed up a few topics here: - Moved everything to a VPS - way better value for money. Extra TB of traffic only costs €1-10 with Hetzner\u002FDigitalOcean compared to 400€…",{"commenters":170,"max_depth":32,"maker_replies":158,"first_comment_mins":171,"lifespan_hours":172},33,43,27,[174,176,178,182,184,186,187,189,193,194,197,202,205,209,211,214,216,219,222,225,228,231,234,235,237,241,242,244,247,251,254,256,258,260,261,263,266,268,270,271,273,274,278,280,282,286,290],[140,34,34,175,141,34],"other\u002Fneutral\u002Fnone",[161,34,34,177,11,35],"maker\u002Fneutral\u002Fnone",[179,34,35,180,181,34],"42095574","project\u002Fcritical\u002Fexecution","emptiestplace",[164,34,34,183,11,35],"maker\u002Fneutral\u002Fexecution",[185,34,40,183,181,34],"42095765",[167,34,34,183,11,35],[188,34,40,183,181,34],"42097159",[190,34,34,191,192,34],"42095606","other\u002Fneutral\u002Fpricing","petercooper",[129,35,34,175,130,34],[195,35,34,175,196,34],"42095402","bbor",[198,199,42,200,201,34],"42093845",2,"project\u002Fpositive\u002Fnone","plaidwombat",[203,199,34,204,11,35],"42093958","maker\u002Fpositive\u002Fidea",[206,40,40,207,208,34],"42093964","project\u002Fneutral\u002Fidea","rty32",[96,40,34,210,11,35],"project\u002Fneutral\u002Fnone",[212,40,34,175,213,34],"42095678","dleeftink",[67,18,42,215,68,34],"project\u002Fpositive\u002Fidea",[217,18,40,207,218,34],"42093820","jaggirs",[220,18,40,207,221,34],"42093852","kgeist",[223,18,42,215,224,34],"42093909","simonw",[226,18,40,207,227,34],"42093929","zexodus",[229,18,40,207,230,34],"42094283","dtjohnnyb",[232,18,34,233,11,35],"42093906","commenter\u002Fpositive\u002Fnone",[99,42,40,207,100,34],[236,42,34,177,11,35],"42094564",[238,239,34,175,240,34],"42094626",6,"jot",[71,32,34,175,72,34],[121,243,34,175,122,34],8,[245,243,34,175,246,34],"42099419","ktosobcy",[248,249,34,175,250,34],"42093644",9,"sureglymop",[252,249,34,175,253,34],"42093968","stevekemp",[106,158,40,255,107,34],"project\u002Fneutral\u002Fexecution",[257,158,34,183,11,35],"42095562",[133,259,42,215,134,34],11,[78,259,34,200,11,35],[81,262,42,200,82,34],12,[264,33,40,207,265,34],"42099299","vergessenmir",[114,267,40,207,115,34],14,[269,267,40,210,224,34],"42093924",[89,267,34,207,11,35],[272,267,34,175,240,34],"42094649",[154,267,34,175,155,34],[275,276,40,210,277,34],"42094740",15,"inhumantsar",[279,276,34,175,277,34],"42094756",[147,281,34,175,148,34],16,[283,284,42,200,285,34],"42097246",17,"hello_computer",[287,288,42,200,289,34],"42099181",18,"linhns",[291,292,34,175,293,34],"42100041",19,"lakomen",[],[296,301,306,310,315,320],{"id":297,"name":298,"points":299,"category":300},"42082430","Jamcorder",590,"hardware",{"id":302,"name":303,"points":304,"category":305},"42118039","Stretch My Time Off",358,"productivity",{"id":307,"name":308,"points":309,"category":305},"42119042","Jelly",338,{"id":311,"name":312,"points":313,"category":314},"42085036","Asterogue",331,"games",{"id":316,"name":317,"points":318,"category":319},"42095879","Jaws",290,"languages",{"id":321,"name":322,"points":323,"category":324},"42104224","Krita RGBA Tech",271,"creative",false,"v2",{"id":328,"name":5,"line":6,"ai":325,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":329,"top":18,"kw":330,"maker":11,"domain":10,"tier":19,"attempts":34,"strip":331,"url":9,"mix":332,"praise":63,"objection":86,"sub":21},42093511,0.33,[24,25,26,27,28],"0010330005030533530300000030553330030055",[32,33,34,35],1794036664043]