[{"data":1,"prerenderedAt":286},["ShallowReactive",2],{"project-3628473":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":43,"post_text":45,"what":46,"said":80,"maker_in_thread":170,"shape":179,"grid":183,"attempts":250,"same_week":251,"maker_shows":39,"maker_hidden":279,"reader_version":280,"card":281},"3628473","hncrawl","A tiny Python-based HN crawler with scrapy","title","Show HN: I wrote a tiny Python-based HN crawler with scrapy","https:\u002F\u002Fgithub.com\u002Fmvanveen\u002Fhncrawl","github.com","mvanveen",1330072474,2012,53,28,447,1677,27,"notable","dev_tools","web scraper","cli",[24,25,26,27],"hacker news crawler","scrapy","python scraper","hn scraper","roasted",-0.2,{"positive":31,"neutral":32,"mixed":33,"critical":34},2,4,0,6,12,{"idea":37,"performance":38,"execution":40,"docs":42},{"positive":31,"critical":31},{"positive":33,"critical":39},1,{"positive":33,"critical":41},3,{"positive":33,"critical":33},[44,33,39,33,39,33,33,41,33,39,39,41,41,41,33,33,33,39,33,33,33,39,44,33,33,33,33,33],5,null,{"problem":47,"audience":45,"pricing":45,"motivation":56,"time_to_build":45,"stack":61,"openness":78,"stage":79,"team_size":79},{"value":48,"source":49,"evidence":50},"Pulling Hacker News front page stories and their external links programmatically.","inferred",[51,53],{"id":52},"post",{"id":54,"by":11,"text":55},"3628655","I wanted to use it! Originally I was targeting this API, but when I was getting a huge number of 500 errors when I was developing. The comments to the right of the page all…",{"value":57,"source":58,"evidence":59},"Hacker News API was unstable and returning many 500 errors during development.","maker",[60],{"id":54,"by":11,"text":55},[62,66,70],{"value":63,"source":49,"evidence":64},"Python",[65],{"id":52},{"value":67,"source":58,"evidence":68},"Scrapy",[69],{"id":52},{"value":71,"source":72,"evidence":73},"BeautifulSoup","commenter",[74],{"id":75,"by":76,"text":77},"3628943","wrath","I'm curious why you used BeautifulSoup when scrapy has its own built-in HTML parser (HtmlXPathSelector). Is there an advantage to BeautifulSoup or is it just the tool that you're…","open_source","unknown",{"praise":81,"objections":96,"requests":122,"questions":137,"alternatives":152},[82,89],{"value":83,"source":72,"evidence":84},"Useful for people who want to scrape HN",[85],{"id":86,"by":87,"text":88},"3628557","kuromitsu","I think many people will find this useful. I certainly do!",{"value":90,"source":72,"evidence":91},"Scrapy is a good framework",[92],{"id":93,"by":94,"text":95},"3628732","awsedwsq","Scrapy, what a fantastic framework it is! One thing bothers me with your code tough, the following could be replace with built in Scrapy-tools. Edit: The code got wrongly…",[97,107,114],{"value":98,"source":72,"evidence":99},"Crawling HN directly adds load; should use the API or a cache",[100,104],{"id":101,"by":102,"text":103},"3628644","jacquesm","Please use this: http:\u002F\u002Fapi.ihackernews.com\u002F HN is slow enough as it is ;)",{"id":105,"by":102,"text":106},"3628674","I'm sure your intentions are good. Did you take up the errors issue with the makers of the api? HN is built using a homebrew stack, it's a miracle it performs as well as it does.…",{"value":108,"source":72,"evidence":109},"Ignores HN's Crawl-delay in robots.txt",[110],{"id":111,"by":112,"text":113},"3628565","DanielRibeiro","Don't forget the Crawl-delay on HN's robots.txt: http:\u002F\u002Fnews.ycombinator.com\u002Frobots.txt",{"value":115,"source":72,"evidence":116},"Code could use built-in Scrapy tools and is unnecessarily complex",[117,118],{"id":93,"by":94,"text":95},{"id":119,"by":120,"text":121},"3629986","the_cat_kittles","Why noy just use requests and lxml, you could do the whole thing in about 10 lines in one file",[123,130],{"value":124,"source":72,"evidence":125},"Use the RSS feed instead of scraping",[126],{"id":127,"by":128,"text":129},"3628835","jimmy2times","I haven't run the crawler so I'm not sure what else it does, but if it only parses the home page and fetches the external links, why not read http:\u002F\u002Fnews.ycombinator.com\u002Frss (you…",{"value":131,"source":72,"evidence":132},"Remove hardcoded local path",[133],{"id":134,"by":135,"text":136},"3628653","zalew","yeah, I modified a bit his script. I chose his script because it logs you in so you can grab your content. btw you have a hardcoded path in there \u002FUsers\u002Fmvanveen\u002Froot\u002Fdev\u002Fnews\u002Fout\u002F",[138,148],{"value":139,"source":72,"evidence":140},"Legal considerations before crawling a website",[141,145],{"id":142,"by":143,"text":144},"3628771","zerop","On the legal front, what all should I consider before crawling a website, apart from honoring robots.txt?",{"id":146,"by":143,"text":147},"3628974","Thanks...On the legal front, what all should I consider before crawling a website, apart from honoring robots.txt?",{"value":149,"source":72,"evidence":150},"Why BeautifulSoup was used instead of Scrapy's selector",[151],{"id":75,"by":76,"text":77},[153,157,164],{"value":154,"source":72,"evidence":155},"ihackernews API",[156],{"id":101,"by":102,"text":103},{"value":158,"source":72,"evidence":159},"hacker-news-confidence project",[160],{"id":161,"by":162,"text":163},"3628873","shadowsun7","Here's mine, written on a lazy Sunday evening in January: https:\u002F\u002Fgithub.com\u002Fshadowsun7\u002Fhacker-news-confidence It runs on cron, scrapes HN once every 30 minutes, and sorts it…",{"value":165,"source":72,"evidence":166},"Zalew's Python HN grabber script",[167],{"id":168,"by":135,"text":169},"3628559","relevant http:\u002F\u002Fzalew.net\u002F2011\u002F12\u002F08\u002Fgrab-your-hackernews-stories-and-comments-python\u002F ;)",{"comments":44,"answered":33,"key":171},[172,173,176],{"id":54,"text":55},{"id":174,"text":175},"3628574","It should only scrape the front page once per run as an initial condition to unleash the other requests. But thanks for the heads up! I'll be sure to mention this in the docs.…",{"id":177,"text":178},"3628609","Cool link, thanks! It looks like he's using methods from an existing Python CLI interface project to scrape the site. Definitely a creative way to go, although I personally think…",{"commenters":180,"max_depth":44,"maker_replies":44,"first_comment_mins":181,"lifespan_hours":182},18,50,9,[184,186,190,192,194,196,199,201,205,208,210,212,216,218,220,222,223,225,227,229,231,234,236,237,240,243,246,249],[161,33,44,185,162,33],"project\u002Fpositive\u002Fidea",[187,33,33,188,189,33],"3628883","commenter\u002Fneutral\u002Fnone","willvarfar",[101,39,39,191,102,33],"project\u002Fcritical\u002Fperformance",[54,39,33,193,11,39],"maker\u002Fneutral\u002Fnone",[105,39,39,195,102,33],"maker\u002Fcritical\u002Fidea",[197,39,33,198,189,33],"3628886","other\u002Fneutral\u002Fnone",[200,39,33,188,102,33],"3628952",[202,39,41,203,204,33],"3629257","project\u002Fneutral\u002Fnone","joering2",[206,39,33,188,207,33],"3629767","muyuu",[127,31,39,209,128,33],"project\u002Fcritical\u002Fidea",[93,41,39,211,94,33],"project\u002Fcritical\u002Fexecution",[213,41,41,214,215,33],"3628838","maker\u002Fneutral\u002Fdocs","d0mine",[75,32,41,217,76,33],"maker\u002Fneutral\u002Fexecution",[111,44,41,219,112,33],"project\u002Fneutral\u002Fperformance",[174,44,33,221,11,39],"maker\u002Fpositive\u002Fexecution",[168,34,33,198,135,33],[177,34,33,224,11,39],"maker\u002Fpositive\u002Fidea",[134,34,39,226,135,33],"maker\u002Fcritical\u002Fexecution",[228,34,33,221,11,39],"3628667",[142,230,33,198,143,33],7,[232,230,33,198,233,33],"3629051","klapinat0r",[119,235,39,211,120,33],8,[86,182,44,185,87,33],[238,182,33,239,11,39],"3628629","maker\u002Fpositive\u002Fnone",[241,242,33,198,143,33],"3628563",10,[244,242,33,198,245,33],"3628626","Luyt",[247,242,33,198,248,33],"3628701","etcet",[146,242,33,198,143,33],[],[252,257,261,266,271,275],{"id":253,"name":254,"points":255,"category":256},"3637132","HackerNews News Recommender",97,"productivity",{"id":258,"name":259,"points":260,"category":20},"3626481","HTTP Status",58,{"id":262,"name":263,"points":264,"category":265},"3635279","Kana by Hand \u002F Kanji by Hand",55,"languages",{"id":267,"name":268,"points":269,"category":270},"3641396","Polymath",49,"learning_science",{"id":272,"name":273,"points":274,"category":265},"3649827","Unofficial-Google-Music-API",48,{"id":276,"name":277,"points":278,"category":20},"3644160","CircuitLab",46,false,"v2",{"id":282,"name":5,"line":6,"ai":279,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":29,"top":18,"kw":283,"maker":11,"domain":10,"tier":19,"attempts":33,"strip":284,"url":9,"mix":285,"praise":83,"objection":98,"sub":21},3628473,[24,25,26,27],"5010100301133300010001500000",[31,32,33,34],1793990879180]