[{"data":1,"prerenderedAt":445},["ShallowReactive",2],{"project-14588333":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":36,"aspects":37,"strip":44,"post_text":46,"what":47,"said":99,"maker_in_thread":219,"shape":227,"grid":232,"attempts":406,"same_week":407,"maker_shows":281,"maker_hidden":437,"reader_version":438,"card":439},"14588333","html5-parser","Fast C-based HTML 5 parsing for Python","title","Show HN: Fast C-based HTML 5 parsing for Python","https:\u002F\u002Fgithub.com\u002Fkovidgoyal\u002Fhtml5-parser","github.com","aumerle",1497890865,2017,169,71,215,2242,10,"notable","languages","html parsing library","library",[24,25,26,27,28],"html5 parser","python html parsing","gumbo","lxml","html5lib","curious",0.093,{"positive":32,"neutral":33,"mixed":34,"critical":35},4,15,1,3,23,{"execution":38,"performance":39,"design":41,"idea":42,"privacy":43},{"positive":34,"critical":34},{"positive":40,"critical":40},0,{"positive":40,"critical":40},{"positive":34,"critical":34},{"positive":40,"critical":34},[40,40,40,40,40,40,40,45,40,45,40,35,35,35,35,40,40,40,35,35,40,35,34,35,35,40,45,35,34,40,40,40,35,35,34,40,35,35,40,40],5,null,{"problem":48,"audience":58,"pricing":46,"motivation":67,"time_to_build":46,"stack":73,"openness":96,"stage":97,"team_size":98},{"value":49,"source":50,"evidence":51},"Parsing HTML quickly and matching browser results on malformed HTML","maker",[52,55],{"id":53,"by":11,"text":54},"14588883","I have not tested it, but it is probably slower. The advantage of this is that is follows the HTML 5 parsing spec, so it gives you the same results as a browser when parsing…",{"id":56,"by":11,"text":57},"14589016","https:\u002F\u002Fhtml5-parser.readthedocs.io\u002Fen\u002Flatest\u002F#benchmarks",{"value":59,"source":50,"evidence":60},"Python developers processing HTML, especially e-book handling in calibre",[61,64],{"id":62,"by":11,"text":63},"14589197","Yes, since I wrote html5-parser for use in calibre (i.e. to parse the HTML in e-books), sigil-gumbo is a more appropriate upstream for me. I have no objectsion to merging in…",{"id":65,"by":11,"text":66},"14588660","Because it uses a modified version of gumbo to support parsing not-well formed XHTML as well",{"value":68,"source":50,"evidence":69},"Replace html5lib in calibre with a faster parser",[70],{"id":71,"by":11,"text":72},"14588933","Indeed, I developed html5-parser to eventually replace html5lib in calibre",[74,81,85,91],{"value":75,"source":50,"evidence":76},"C",[77,80],{"id":78,"by":11,"text":79},"14592197","I use this pattern a lot. For example, in https:\u002F\u002Fgithub.com\u002Fkovidgoyal\u002Fkitty where the UI layer is in python and the performance critical code is in C. I find this pattern gives…",{"id":65,"by":11,"text":66},{"value":82,"source":50,"evidence":83},"Python",[84],{"id":71,"by":11,"text":72},{"value":86,"source":50,"evidence":87},"libxml2",[88],{"id":89,"by":11,"text":90},"14592234","This has a hard dependency on libxml2. If your language of choice has a libxml2 based tree, it can be ported to that. Basically your language needs an equivalent of lxml",{"value":92,"source":50,"evidence":93},"gumbo (bundled fork)",[94,95],{"id":65,"by":11,"text":66},{"id":62,"by":11,"text":63},"open_source","launched","solo",{"praise":100,"objections":120,"requests":153,"questions":179,"alternatives":191},[101,105,113],{"value":102,"source":50,"evidence":103},"Speed claimed about 30x faster, with benchmarks linked",[104],{"id":56,"by":11,"text":57},{"value":106,"source":107,"evidence":108},"Good documentation on safety and correctness","commenter",[109],{"id":110,"by":111,"text":112},"14591321","zer01","My favorite part about this documentation is actually the \"Safety and correctness\" section: https:\u002F\u002Fhtml5-parser.readthedocs.io\u002Fen\u002Flatest\u002F#safety-and-correctness It shows that a…",{"value":114,"source":50,"evidence":115},"Follows HTML5 spec, matching browser output on malformed HTML",[116,117],{"id":53,"by":11,"text":54},{"id":118,"by":11,"text":119},"14588633","As permissive as html5lib, which is as permissive as a modern browser, since both are base don the HTML 5 parsing spec.",[121,131,142],{"value":122,"source":107,"evidence":123},"Bundled C code raises memory safety and buffer overflow concerns",[124,128],{"id":125,"by":126,"text":127},"14589034","Animats","No, no. I don't want some complicated C blob in the middle of my Python system. I don't have time to chase buffer overflows and pointer bugs. Has anyone benchmarked this against…",{"id":129,"by":126,"text":130},"14589354","If just the tokenizer was in Rust, that might help. Much of the HTML5 error handling involves dealing with malformed tokens. Comments that begin \"\u003C-\" instead of \"\u003C--\". All that…",{"value":132,"source":107,"evidence":133},"Performance claims lack independent tests and comparisons with lxml",[134,138],{"id":135,"by":136,"text":137},"14589003","metalliqaz","Where are the test that show it's 30x faster? Otherwise, pretty cool. I only do occasional HTML parsing in scripts, so whatever is builtin and BeautifulSoup is good enough for…",{"id":139,"by":140,"text":141},"14588855","dalf","How fast is compare to lxml with the HTML parser ? I know that lxml won't parse as many cases as this module, but in many cases lxml can do the job.",{"value":143,"source":107,"evidence":144},"Bundled gumbo fork complicates maintenance",[145,149],{"id":146,"by":147,"text":148},"14588652","pwdisswordfish","Why does it bundle a copy of the gumbo library instead of just linking to it?",{"id":150,"by":151,"text":152},"14589164","sebcat","If I understand things correctly, gumbo in html5-parser was forked from https:\u002F\u002Fgithub.com\u002FSigil-Ebook\u002Fsigil-gumbo\u002F at 0830e1145fe08758d6ef24f77dfcbeac4633676f, which in turn was…",[154,162,168],{"value":155,"source":107,"evidence":156},"Benchmarks against lxml and other parsers",[157,158],{"id":139,"by":140,"text":141},{"id":159,"by":160,"text":161},"14591231","ernsheong","Would also be nice to see a performance comparison with other libraries, e.g. Go's html.Parse",{"value":163,"source":107,"evidence":164},"Bindings for other languages such as Go, Ruby or Java",[165],{"id":166,"by":160,"text":167},"14591225","Would it be possible to interface this with other languages? e.g. Go, Ruby, Java, or is there a hard dependency on Python?",{"value":169,"source":107,"evidence":170},"Integration with BeautifulSoup",[171,175],{"id":172,"by":173,"text":174},"14588507","jszymborski","This would replace html5lib in BeautifulSoup nicely :)",{"id":176,"by":177,"text":178},"14594284","wooptoo","Any luck using this with BeautifulSoup?",[180,187],{"value":181,"source":107,"evidence":182},"How does it stay fast with two tree transformations?",[183],{"id":184,"by":185,"text":186},"14590075","samblr","30x is quick! But still it doesn't appear to be sequential parser. html -> gumbo parse tree -> lxml tree What makes it efficient even with two tree transformations ? Is one of…",{"value":188,"source":107,"evidence":189},"Why bundle gumbo instead of linking to it?",[190],{"id":146,"by":147,"text":148},[192,196,202,213],{"value":28,"source":107,"evidence":193},[194,195],{"id":125,"by":126,"text":127},{"id":172,"by":173,"text":174},{"value":27,"source":107,"evidence":197},[198,199],{"id":139,"by":140,"text":141},{"id":200,"by":11,"text":201},"14592571","You do know that by default bs4 uses lxml.html to do its parsing, which is also a big blob of C, right? For example, running the following: BeautifulSoup('\u003Cp>a')…",{"value":203,"source":107,"evidence":204},"html5ever (Rust)",[205,209],{"id":206,"by":207,"text":208},"14589245","jgraham","[I was the original author of much of html5lib] Yes, http:\u002F\u002Fspeed.pypy.org\u002Fcomparison\u002F has html5lib parsing (an old, static copy of) the HTML5 spec as a benchmark. It's about 3x…",{"id":210,"by":211,"text":212},"14592403","littlestymaar","> My ideal solution here would be bindings to a Rust parser e.g. html5ever Funny how somebody submitted exactly this on HN a few hours ago :…",{"value":214,"source":107,"evidence":215},"gumbo-parser",[216],{"id":217,"by":151,"text":218},"14588729","I made a pull request[1] for gumbo-parser a while ago. While it is a wonderful project, it seems to be in need of a new maintainer, or a maintained fork. [1]…",{"comments":220,"answered":18,"key":221},16,[222,223,224],{"id":78,"text":79},{"id":71,"text":72},{"id":225,"text":226},"14592251","-Werror is used only when building on the continuous integration servers, not in the released code. That way, you get the best of both worlds. Your code wont break on new…",{"commenters":228,"max_depth":229,"maker_replies":220,"first_comment_mins":230,"lifespan_hours":231},39,6,20,45,[233,237,240,243,245,248,251,254,256,260,263,266,269,272,276,278,280,283,287,289,291,293,294,297,299,300,302,305,308,310,311,313,315,316,318,320,323,325,327,328,331,334,336,338,340,343,346,349,351,352,354,358,361,364,366,368,369,371,373,374,375,380,383,386,388,391,395,396,399,401,404],[234,40,40,235,236,40],"14590502","other\u002Fneutral\u002Fnone","jacquesm",[238,40,40,235,239,40],"14590735","penpapersw",[241,40,40,235,242,40],"14590899","mpweiher",[244,40,40,235,111,40],"14591300",[246,40,40,235,247,40],"14603649","skocznymroczny",[249,40,40,235,250,40],"14591796","girvo",[252,40,40,235,253,40],"14591850","fiddlerwoaroof",[78,40,40,255,11,34],"maker\u002Fneutral\u002Fdesign",[257,40,40,258,259,40],"14592491","commenter\u002Fneutral\u002Fnone","khanan",[261,40,40,235,262,40],"14592214","mkj",[264,40,40,235,265,40],"14592285","IgorPartola",[267,40,40,235,268,40],"14592932","sametmax",[270,40,40,258,271,40],"14594401","pg314",[273,34,45,274,275,40],"14588860","maker\u002Fpositive\u002Fnone","zaszrespawned",[71,34,40,277,11,34],"maker\u002Fneutral\u002Fnone",[279,34,45,274,185,40],"14590131",[110,281,45,282,111,40],2,"maker\u002Fpositive\u002Fexecution",[284,281,40,285,286,40],"14591380","commenter\u002Fneutral\u002Fexecution","mrout",[225,281,40,288,11,34],"maker\u002Fneutral\u002Fexecution",[135,35,35,290,136,40],"project\u002Fneutral\u002Fperformance",[56,35,40,292,11,34],"project\u002Fpositive\u002Fperformance",[184,35,35,290,185,40],[295,35,35,290,296,40],"14590928","nostrademons",[298,35,35,290,185,40],"14592804",[139,32,35,290,140,40],[53,32,40,301,11,34],"maker\u002Fneutral\u002Fperformance",[303,32,40,235,304,40],"14589185","Paul-ish",[306,32,40,235,307,40],"14589520","Buttons840",[309,32,40,235,304,40],"14590145",[217,45,40,235,151,40],[312,45,40,282,11,34],"14588782",[150,45,35,314,151,40],"project\u002Fneutral\u002Fexecution",[62,45,40,288,11,34],[317,45,35,290,296,40],"14589565",[319,45,40,235,296,40],"14588801",[321,45,40,322,11,34],"14588918","other\u002Fpositive\u002Fnone",[324,45,40,235,236,40],"14590508",[146,229,35,326,147,40],"project\u002Fneutral\u002Fdesign",[65,229,40,326,11,34],[329,229,34,330,147,40],"14588808","project\u002Fcritical\u002Fidea",[332,229,40,333,11,34],"14588825","project\u002Fneutral\u002Fidea",[335,229,35,333,147,40],"14588878",[337,229,40,333,11,34],"14588911",[339,229,35,333,296,40],"14589019",[341,229,40,235,342,40],"14592605","deathanatos",[344,229,40,345,11,34],"14592742","maker\u002Fneutral\u002Fidea",[172,347,45,348,173,40],7,"project\u002Fpositive\u002Fidea",[125,347,35,350,126,40],"project\u002Fmixed\u002Fperformance",[206,347,35,290,207,40],[129,347,34,353,126,40],"project\u002Fcritical\u002Fprivacy",[355,347,40,356,357,40],"14589469","other\u002Fneutral\u002Fperformance","gsnedders",[359,347,40,356,360,40],"14590533","olau",[362,347,40,356,363,40],"14591667","daniel_rh",[365,347,40,356,296,40],"14591724",[367,347,40,356,357,40],"14593852",[210,347,40,235,211,40],[200,347,40,370,11,34],"commenter\u002Fcritical\u002Fperformance",[166,372,35,326,160,40],8,[159,372,35,290,160,40],[89,372,40,255,11,34],[376,377,34,378,379,40],"14588812",9,"project\u002Fcritical\u002Fexecution","droithomme",[381,377,40,258,382,40],"14589316","loeg",[384,377,40,258,385,40],"14589557","dom0",[176,18,35,387,177,40],"project\u002Fneutral\u002Fnone",[389,18,40,390,11,34],"14594795","maker\u002Fneutral\u002Fdocs",[392,393,35,290,394,40],"14588600",11,"RUTHLESS_RUFUS",[118,393,40,314,11,34],[397,393,40,235,398,40],"14589155","exabrial",[400,393,40,235,147,40],"14589372",[402,393,40,235,403,40],"14594127","kbutler",[405,393,40,235,394,40],"14596275",[],[408,413,417,422,427,432],{"id":409,"name":410,"points":411,"category":412},"14558582","Snips",372,"ai",{"id":414,"name":415,"points":416,"category":20},"14583882","Decaffeinate",361,{"id":418,"name":419,"points":420,"category":421},"14587669","rust-algorithms",320,"learning_science",{"id":423,"name":424,"points":425,"category":426},"14593906","Copyfish",292,"productivity",{"id":428,"name":429,"points":430,"category":431},"14597485","NBox",286,"security_privacy",{"id":433,"name":434,"points":435,"category":436},"14597544","Upstart.me",168,"web_saas",false,"v2",{"id":440,"name":5,"line":6,"ai":437,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":441,"top":18,"kw":442,"maker":11,"domain":10,"tier":19,"attempts":40,"strip":443,"url":9,"mix":444,"praise":102,"objection":122,"sub":21},14588333,0.09,[24,25,26,27,28],"0000000505033330003303133053100033103300",[32,33,34,35],1794036556444]