[{"data":1,"prerenderedAt":305},["ShallowReactive",2],{"project-14765742":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":36,"aspects":37,"strip":42,"post_text":44,"what":45,"said":67,"maker_in_thread":174,"shape":185,"grid":189,"attempts":261,"same_week":266,"maker_shows":296,"maker_hidden":297,"reader_version":298,"card":299},"14765742","Sukhoi","A flexible and extensible Webcrawler in Python","title","Show HN: Sukhoi – A flexible and extensible Webcrawler in Python","https:\u002F\u002Fgithub.com\u002Fiogf\u002Fsukhoi","github.com","iogf",1499983781,2017,131,34,314,2242,15,"notable","dev_tools","web crawler","library",[24,25,26,27,28],"webcrawler","python scraper","web scraping","lxml","untwisted","curious",0.158,{"positive":32,"neutral":33,"mixed":34,"critical":35},3,6,0,1,10,{"idea":38,"design":39,"docs":40,"execution":41},{"positive":35,"critical":35},{"positive":35,"critical":34},{"positive":34,"critical":34},{"positive":34,"critical":34},[34,34,43,34,34,34,34,34,34,34,34,34,34,34,34,43,34,32,34,32,34,32,34,34,34,32,32,34,43,32,34,34,35,34],5,null,{"problem":44,"audience":44,"pricing":44,"motivation":44,"time_to_build":44,"stack":46,"openness":66,"stage":66,"team_size":66},[47,54,61],{"value":48,"source":49,"evidence":50},"Python 2","maker",[51],{"id":52,"by":11,"text":53},"14766724","It is py2 now, however, i'm gonna port it to py3 soon. I'm planning to write some better docs for it tomorrow.",{"value":27,"source":55,"evidence":56},"commenter",[57],{"id":58,"by":59,"text":60},"14768567","danpalmer","> It looks more enjoyable to use than BeautifulSoup. I don't believe BS is a full scraping solution, it's only the HTML parsing\u002Fquerying isn't it? In that case, this project…",{"value":28,"source":49,"evidence":62},[63],{"id":64,"by":11,"text":65},"14768054","Untwised is meant to solve all problems twisted solves but it does it in quite a different way. They are two different tools that would solve the same problems using different…","unknown",{"praise":68,"objections":83,"requests":99,"questions":119,"alternatives":140},[69,76],{"value":70,"source":55,"evidence":71},"Looks more enjoyable to use than BeautifulSoup",[72],{"id":73,"by":74,"text":75},"14766657","tedmiston","Pretty cool project. It looks more enjoyable to use than BeautifulSoup. How does it approach throttling or rate limiting? I didn't see this mentioned in the readme examples.…",{"value":77,"source":55,"evidence":78},"Interesting project that focuses on combining information from multiple sources",[79],{"id":80,"by":81,"text":82},"14766241","dguo","Interesting timing! I just started using Scrapy today for a project, and I'm trying to figure out how to elegantly piece together information from different sources. I'm glad to…",[84,92],{"value":85,"source":49,"evidence":86},"Python 2 only, no Python 3 support yet",[87,91],{"id":88,"by":89,"text":90},"14766710","pryelluw","Is this Python 3 compatible? Searched but the wiki is empty and the readme has examples in Python 2.",{"id":52,"by":11,"text":53},{"value":93,"source":55,"evidence":94},"Not clear how it compares to Scrapy",[95],{"id":96,"by":97,"text":98},"14765977","monksy","How does this differ from Scrapy?",[100,104,108,112],{"value":101,"source":55,"evidence":102},"Add throttling or rate limiting",[103],{"id":73,"by":74,"text":75},{"value":105,"source":55,"evidence":106},"Add caching and ETag support",[107],{"id":73,"by":74,"text":75},{"value":109,"source":55,"evidence":110},"Port to Python 3",[111],{"id":88,"by":89,"text":90},{"value":113,"source":55,"evidence":114},"Consider libev or uvloop instead of custom event loop",[115],{"id":116,"by":117,"text":118},"14768716","sandGorgon","This is very interesting. did you consider using libev\u002Fuvloop - which are generally consider battle tested async frameworks ? Is there anything missing that prompted you to…",[120,124,128,134],{"value":121,"source":55,"evidence":122},"How does it differ from Scrapy?",[123],{"id":96,"by":97,"text":98},{"value":125,"source":55,"evidence":126},"How does it handle throttling?",[127],{"id":73,"by":74,"text":75},{"value":129,"source":55,"evidence":130},"Is it pure Python and what are its dependencies?",[131],{"id":132,"by":117,"text":133},"14768272","I'm not able to figure out dependencies.. is this pure python ? Or are you using one of gevent, libev, uvloop, etc. Since it is py2, i suppose asyncio is out of the picture",{"value":135,"source":55,"evidence":136},"How does it relate to Twisted?",[137],{"id":138,"by":74,"text":139},"14766748","Awesome. It looks like you're reusing your own dependencies which is cool. Can you explain how untwisted relates to twisted a little more? I read the repo readme, but not sure…",[141,149,156,167],{"value":142,"source":55,"evidence":143},"Scrapy",[144,145],{"id":96,"by":97,"text":98},{"id":146,"by":147,"text":148},"14766505","bbernoulli","FWIW I don't believe this would be overly convoluted in scrapy. I'd probably scrape the tags and quotes in one pass... Also, generator expressions would make the examples more…",{"value":150,"source":55,"evidence":151},"BeautifulSoup",[152,153],{"id":73,"by":74,"text":75},{"id":154,"by":74,"text":155},"14789441","Yes, BeautifulSoup is focused on parsing not crawling (BS supports the lxml parser out of the box). Scrapy is more of an opinionated scraping framework whereas BS is a parsing…",{"value":157,"source":55,"evidence":158},"Selenium with headless Chrome",[159,163],{"id":160,"by":161,"text":162},"14766105","vosper","How useful are scrapers that don't execute Javascript these days? I find Selenium + PhantomJS (now Chrome Headless I guess) is pretty easy to drive from Python, and it works…",{"id":164,"by":165,"text":166},"14766146","dhruvkar","Phantom still gets blocked, as it reveals itself in the header. I've had success with a headless Chrome instance in a virtual display (xvfb) driven with Selenium, backed by…",{"value":168,"source":55,"evidence":169},"Common Search",[170],{"id":171,"by":172,"text":173},"14768871","ldng","Depending on your needs, sometimes it might be more interesting starting for there : https:\u002F\u002Fabout.commonsearch.org\u002F and then scrap whatever is missing or not fresh enough. The…",{"comments":175,"answered":176,"key":177},9,8,[178,181,184],{"id":179,"text":180},"14766159","These ones are slower in most cases, it seems for some situations the ones that dont execute js would better do the job.",{"id":182,"text":183},"14766688","The throtting can be set directly from untwisted reactor(planning to implement soon once i get untwisted on py3). I think the support for caching is really good too, i plan to…",{"id":64,"text":65},{"commenters":186,"max_depth":176,"maker_replies":175,"first_comment_mins":187,"lifespan_hours":188},18,39,89,[190,194,197,199,201,202,205,208,211,213,215,217,220,223,226,228,230,232,234,236,237,239,240,242,243,244,246,247,248,251,252,254,256,258],[191,34,34,192,193,34],"14767986","other\u002Fneutral\u002Fnone","gear54rus",[195,34,34,192,196,34],"14768335","doubleplusgood",[80,35,43,198,81,34],"project\u002Fpositive\u002Fidea",[160,200,34,192,161,34],2,[164,200,34,192,165,34],[203,200,34,192,204,34],"14766606","rectangletangle",[206,200,34,207,165,34],"14766685","commenter\u002Fneutral\u002Fnone",[209,200,34,192,210,34],"14768362","eriknstr",[179,200,34,212,11,35],"other\u002Fneutral\u002Fperformance",[214,200,34,192,204,34],"14766613",[216,200,34,192,59,34],"14768551",[218,200,34,192,219,34],"14768630","ConfucianNardin",[221,200,34,192,222,34],"14768343","Mic92",[224,200,34,192,225,34],"14770821","penetrarthur",[227,200,34,192,59,34],"14768578",[73,32,43,229,74,34],"project\u002Fpositive\u002Fdesign",[182,32,34,231,11,35],"project\u002Fneutral\u002Fexecution",[138,32,32,233,74,34],"project\u002Fneutral\u002Fdocs",[64,32,34,235,11,35],"project\u002Fneutral\u002Fidea",[132,32,32,231,117,34],[238,32,34,231,11,35],"14768643",[116,32,32,231,117,34],[241,32,34,231,11,35],"14771335",[58,32,34,192,59,34],[154,32,34,192,74,34],[171,245,32,235,172,34],4,[88,43,32,231,89,34],[52,43,34,231,11,35],[249,43,43,250,89,34],"14767938","project\u002Fpositive\u002Fnone",[96,33,32,235,97,34],[253,33,34,229,11,35],"14765990",[255,33,34,229,11,35],"14766000",[146,33,35,257,147,34],"project\u002Fcritical\u002Fidea",[259,33,34,260,11,35],"14766518","project\u002Fneutral\u002Fdesign",[262,265],{"id":263,"t":264,"points":33,"comments":43},"14707885",1499310427,{"id":4,"t":12,"points":14,"comments":15},[267,272,277,282,287,292],{"id":268,"name":269,"points":270,"category":271},"14805265","gamblers-dice",523,"languages",{"id":273,"name":274,"points":275,"category":276},"14771406","Quilt",367,"data",{"id":278,"name":279,"points":280,"category":281},"14799111","Cinc",363,"productivity",{"id":283,"name":284,"points":285,"category":286},"14794949","choppingboard.recipes",208,"web_saas",{"id":288,"name":289,"points":290,"category":291},"14769317","Python books",150,"learning_science",{"id":293,"name":294,"points":295,"category":291},"14798336","Learn Anything",145,7,false,"v2",{"id":300,"name":5,"line":6,"ai":297,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":301,"top":18,"kw":302,"maker":11,"domain":10,"tier":19,"attempts":200,"strip":303,"url":9,"mix":304,"praise":70,"objection":85,"sub":21},14765742,0.16,[24,25,26,27,28],"0050000000000005030303000330530010",[32,33,34,35],1794036557250]