[{"data":1,"prerenderedAt":293},["ShallowReactive",2],{"project-8012103":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":34,"aspects":35,"strip":42,"post_text":43,"what":44,"said":86,"maker_in_thread":174,"shape":182,"grid":186,"attempts":256,"same_week":257,"maker_shows":32,"maker_hidden":285,"reader_version":286,"card":287},"8012103","MechanicalSoup","MechanicalSoup, Python library for automating interaction with websites","title","Show HN: MechanicalSoup, Python library for automating interaction with websites","https:\u002F\u002Fgithub.com\u002Fhickford\u002FMechanicalSoup","github.com","mattme",1404938153,2014,180,27,122,2272,6,"notable","dev_tools","web automation library","library",[24,25,26,27],"python web scraping","browser automation","requests beautifulsoup","mechanize alternative","mixed",0.247,{"positive":31,"neutral":31,"mixed":32,"critical":33},5,2,1,13,{"execution":36,"idea":38,"performance":40,"docs":41},{"positive":37,"critical":33},0,{"positive":39,"critical":37},3,{"positive":37,"critical":37},{"positive":33,"critical":37},[37,37,37,39,39,37,37,37,37,31,37,31,37,37,37,37,37,39,39,37,39,33,39,31,31,31,39],null,{"problem":45,"audience":57,"pricing":43,"motivation":62,"time_to_build":43,"stack":72,"openness":85,"stage":85,"team_size":85},{"value":46,"source":47,"evidence":48},"Automating interaction with websites without a full browser","commenter",[49,53],{"id":50,"by":51,"text":52},"8013869","auvrw","they're for different use cases. mechanicalsoup is more like zombie.js ( http:\u002F\u002Fzombie.labnotes.org\u002F ) than selenium, chromedriver, or some other webdriver (…",{"id":54,"by":55,"text":56},"8012645","wc-","Selenium seems very heavy-weight to me (granted I have only used selenium server). If you don't need to interpret javascript after a page loads then you might be able to use…",{"value":58,"source":59,"evidence":60},"Python developers scraping or testing websites","inferred",[61],{"id":50,"by":51,"text":52},{"value":63,"source":47,"evidence":64},"Python mechanize was stagnant and poorly maintained",[65,69],{"id":66,"by":67,"text":68},"8013440","Deusdies","This is fantastic. I've used python mechanize in some very large projects and it was very frustrating - their lack of documentation and, well, the fact that it's complete…",{"id":70,"by":55,"text":71},"8012254","Mechanize is a core part in quite a few of my projects lately and the fact that it hasn't been modified in over 2 years has been very worrisome. There are lots of edge cases out…",[73,77,81],{"value":74,"source":59,"evidence":75},"Python",[76],{"id":50,"by":51,"text":52},{"value":78,"source":47,"evidence":79},"Requests",[80],{"id":50,"by":51,"text":52},{"value":82,"source":47,"evidence":83},"BeautifulSoup",[84],{"id":50,"by":51,"text":52},"unknown",{"praise":87,"objections":102,"requests":122,"questions":131,"alternatives":150},[88,94],{"value":89,"source":47,"evidence":90},"Fills a gap left by stagnant Python mechanize",[91,92,93],{"id":70,"by":55,"text":71},{"id":66,"by":67,"text":68},{"id":50,"by":51,"text":52},{"value":95,"source":47,"evidence":96},"Simple API and lightweight compared to Selenium",[97,98],{"id":54,"by":55,"text":56},{"id":99,"by":100,"text":101},"8015453","lazerwalker","I personally find Capybara[0] to be the happy medium for web scraping, if Python isn't a hard requirement. It has a simple API, like MechanicalSoup, but it can also easily be…",[103,111],{"value":104,"source":47,"evidence":105},"Does not execute JavaScript",[106,107],{"id":50,"by":51,"text":52},{"id":108,"by":109,"text":110},"8012828","goorpyguy","Does it have a javascript engine? Because we had to abandon BeautifulSoup\u002FMechanize over this a couple years ago and switch to HTMLUnit (Java).",{"value":112,"source":47,"evidence":113},"Very little code so far, unclear roadmap",[114,118],{"id":115,"by":116,"text":117},"8012686","jdnier","There's not a lot to it so far (a single class, three tests). I wonder if the author has a road map for the project.",{"id":119,"by":120,"text":121},"8012716","rhgraysonii","Have any documentation on a roadmap for things as they go forward? Would love to send some PR's your way :)",[123,127],{"value":124,"source":47,"evidence":125},"Publish a roadmap and documentation",[126],{"id":119,"by":120,"text":121},{"value":128,"source":47,"evidence":129},"Add JavaScript engine support",[130],{"id":108,"by":109,"text":110},[132,143],{"value":133,"source":47,"evidence":134},"How does it compare to Scrapy or CasperJS",[135,139],{"id":136,"by":137,"text":138},"8022690","webmaven","How does MechanicalSoup (or RoboBrowse, for that matter, this is the first I've heard of either) compare to Scrapy?: http:\u002F\u002Fscrapy.org\u002F",{"id":140,"by":141,"text":142},"8015312","volent","Would that be an equivalent to Casper.js ?",{"value":144,"source":47,"evidence":145},"Why use it over Selenium",[146],{"id":147,"by":148,"text":149},"8012493","diminoten","Sell me MechanicalSoup over Selenium.",[151,158,162,166,170],{"value":152,"source":47,"evidence":153},"RoboBrowser",[154],{"id":155,"by":156,"text":157},"8012729","flexd","An alternative to this is RoboBrowse [1] which is also based on requests + BeautifulSoup4 and seems a lot more mature. [1] http:\u002F\u002Frobobrowser.readthedocs.org\u002Fen\u002Flatest\u002Freadme.html",{"value":159,"source":47,"evidence":160},"Python mechanize",[161],{"id":70,"by":55,"text":71},{"value":163,"source":47,"evidence":164},"Selenium",[165],{"id":147,"by":148,"text":149},{"value":167,"source":47,"evidence":168},"Capybara",[169],{"id":99,"by":100,"text":101},{"value":171,"source":47,"evidence":172},"Scrapy",[173],{"id":136,"by":137,"text":138},{"comments":32,"answered":37,"key":175},[176,179],{"id":177,"text":178},"8018070","Haha! If I'd known, I would have used that myself.",{"id":180,"text":181},"8024168","Still, it proves the idea is a good one!",{"commenters":183,"max_depth":31,"maker_replies":32,"first_comment_mins":184,"lifespan_hours":185},24,35,110,[187,189,191,193,197,199,202,204,205,208,209,211,212,215,218,221,224,227,231,233,236,238,240,242,245,249,254],[155,37,37,188,156,37],"other\u002Fneutral\u002Fnone",[177,37,37,190,11,33],"maker\u002Fneutral\u002Fidea",[180,37,37,192,11,33],"project\u002Fpositive\u002Fidea",[194,37,39,195,196,37],"8030245","project\u002Fneutral\u002Fexecution","rahimnathwani",[70,33,39,198,55,37],"project\u002Fmixed\u002Fexecution",[200,33,37,188,201,37],"8012299","danso",[203,33,37,188,55,37],"8012303",[99,32,37,188,100,37],[206,32,37,188,207,37],"8015924","actionscripted",[66,39,31,192,67,37],[147,210,37,188,148,37],4,[54,210,31,192,55,37],[213,210,37,188,214,37],"8013085","dkhar",[216,210,37,188,217,37],"8013190","mhluongo",[219,210,37,188,220,37],"8013476","mherrmann",[222,210,37,188,223,37],"8014009","MrMeker",[225,210,37,188,226,37],"8013856","hugs",[228,210,39,229,230,37],"8012902","project\u002Fneutral\u002Fidea","smellf",[50,210,39,232,51,37],"project\u002Fmixed\u002Fidea",[234,210,37,188,235,37],"8015730","seanp2k2",[108,31,39,237,109,37],"project\u002Fneutral\u002Fperformance",[115,18,33,239,116,37],"project\u002Fcritical\u002Fexecution",[136,241,39,229,137,37],7,[119,243,31,244,120,37],8,"maker\u002Fpositive\u002Fdocs",[246,247,31,192,248,37],"8015388",9,"supsep",[250,251,31,252,253,37],"8014623",10,"maker\u002Fpositive\u002Fnone","jpd750",[140,255,39,229,141,37],11,[],[258,263,268,272,276,280],{"id":259,"name":260,"points":261,"category":262},"8011081","Sysadmin Casts",443,"learning_science",{"id":264,"name":265,"points":266,"category":267},"8004839","Hacker News Redesign",380,"productivity",{"id":269,"name":270,"points":271,"category":262},"7985692","Full Stack Python",327,{"id":273,"name":274,"points":275,"category":20},"7985162","Prelang",229,{"id":277,"name":278,"points":279,"category":262},"7985305","A Rogue State Along Two Rivers",212,{"id":281,"name":282,"points":283,"category":284},"7984989","Freeciv web client",197,"games",false,"v2",{"id":288,"name":5,"line":6,"ai":285,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":289,"top":18,"kw":290,"maker":11,"domain":10,"tier":19,"attempts":37,"strip":291,"url":9,"mix":292,"praise":89,"objection":104,"sub":21},8012103,0.25,[24,25,26,27],"000330000505000003303135553",[31,31,32,33],1793990908002]