[{"data":1,"prerenderedAt":211},["ShallowReactive",2],{"project-2501495":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":27,"sentiment":28,"stance_mix":29,"scored_comments":33,"aspects":34,"strip":38,"post_text":40,"what":41,"said":70,"maker_in_thread":137,"shape":147,"grid":151,"attempts":172,"same_week":173,"maker_shows":36,"maker_hidden":203,"reader_version":204,"card":205},"2501495","BrowserCrawler","crawl a website and store it in S3 from your browser","title","Show HN: crawl a website and store it in S3 from your browser","https:\u002F\u002Fgithub.com\u002Fspullara\u002Fbrowsercrawler","github.com","spullara",1304195184,2011,43,12,286,1097,27,"shown","infrastructure","web crawler","web_app",[21,24,25,26],"browser","amazon s3","website archive","debated",0.063,{"positive":30,"neutral":30,"mixed":31,"critical":32},2,0,3,7,{"performance":35,"idea":37},{"positive":31,"critical":36},1,{"positive":30,"critical":30},[36,31,36,36,31,32,32,31,31,39,31,39],5,null,{"problem":42,"audience":40,"pricing":40,"motivation":51,"time_to_build":40,"stack":59,"openness":68,"stage":69,"team_size":69},{"value":43,"source":44,"evidence":45},"Saving a website's pages to S3 from the browser.","inferred",[46,48],{"id":47},"post",{"id":49,"by":11,"text":50},"2502075","As it only does one URL at a time and uploads to S3 between requests, it shouldn't unduly load any reasonable system. I'll add an additional \"BrowserCrawler\" string to the…",{"value":52,"source":53,"evidence":54},"Send someone a link or access pages from anywhere; plans larger crawls and analysis.","maker",[55,58],{"id":56,"by":11,"text":57},"2501982","I have it going to S3 because it gives me the ability to instantly send someone a link or access from wherever I am. Also, I planned on doing larger crawls + analysis. Should be…",{"id":49,"by":11,"text":50},[60,64],{"value":61,"source":53,"evidence":62},"jQuery",[63],{"id":49,"by":11,"text":50},{"value":65,"source":53,"evidence":66},"Amazon S3",[67],{"id":49,"by":11,"text":50},"open_source","unknown",{"praise":71,"objections":84,"requests":105,"questions":117,"alternatives":122},[72],{"value":73,"source":74,"evidence":75},"Cool project","commenter",[76,80],{"id":77,"by":78,"text":79},"2501772","petervandijck","That is very cool.",{"id":81,"by":82,"text":83},"2501819","neuromancer2600","Great! I had the same idea only with GAE integration and option to download locally. Any plans on expanding that interface?",[85,92,98],{"value":86,"source":74,"evidence":87},"Crawling can overload servers and needs rate limiting",[88],{"id":89,"by":90,"text":91},"2502052","ryan-allen","Is there a header set to identify this crawler so I can limit it? For what it's worth, when people crawl sites I am responsible for, and it impacts our products performance, I…",{"value":93,"source":74,"evidence":94},"Copyright and terms of service risk from mirroring content in public S3 buckets",[95],{"id":96,"by":90,"text":97},"2502298","Ah, I'm familiar with S3 latencies. That'd work. Also, if you're uploading other people's content to public S3 buckets, you're very likely to be in violation of mosts site's…",{"value":99,"source":74,"evidence":100},"Should honour robots.txt",[101],{"id":102,"by":103,"text":104},"2502544","JoachimSchipper","What's this X-User-Agent stuff? The entire internet isn't going to special-case your hack. Just honour robots.txt.",[106,113],{"value":107,"source":74,"evidence":108},"Store pages locally on the computer",[109],{"id":110,"by":111,"text":112},"2501792","raptrex","How hard would it be to modify it to store it locally on your computer?",{"value":114,"source":74,"evidence":115},"Google App Engine integration and local download option",[116],{"id":81,"by":82,"text":83},[118],{"value":119,"source":74,"evidence":120},"Can the crawler be identified by a header to limit it?",[121],{"id":89,"by":90,"text":91},[123,130],{"value":124,"source":74,"evidence":125},"Scrapbook Firefox extension",[126],{"id":127,"by":128,"text":129},"2501960","Harkins","There is a nice Firefox plugin called Scrapbook that does this.",{"value":131,"source":74,"evidence":132},"wget",[133],{"id":134,"by":135,"text":136},"2502139","dotBen","``` wget --spider ``` is probably your friend",{"comments":138,"answered":138,"key":139},4,[140,141,144],{"id":49,"text":50},{"id":142,"text":143},"2504514","I don't that a individual storing web pages has any requirement to obey robots.txt. You should be able to click \"Save Page\" on anything you have access to. Obviously…",{"id":145,"text":146},"2501977","Probably wouldn't be that hard to modify. I haven't looked at the local file API available from Safari plugins but should be easy.",{"commenters":148,"max_depth":138,"maker_replies":138,"first_comment_mins":149,"lifespan_hours":150},8,136,28,[152,154,156,158,159,161,163,164,166,168,170,171],[89,31,36,153,90,31],"project\u002Fcritical\u002Fperformance",[49,31,31,155,11,36],"maker\u002Fneutral\u002Fperformance",[96,31,36,157,90,31],"project\u002Fcritical\u002Fidea",[102,31,36,157,103,31],[142,31,31,160,11,36],"commenter\u002Fneutral\u002Fidea",[110,36,32,162,111,31],"project\u002Fneutral\u002Fidea",[127,36,32,162,128,31],[145,36,31,165,11,36],"project\u002Fneutral\u002Fexecution",[134,36,31,167,135,31],"other\u002Fneutral\u002Fnone",[81,30,39,169,82,31],"project\u002Fpositive\u002Fidea",[56,30,31,162,11,36],[77,32,39,169,78,31],[],[174,179,184,189,194,198],{"id":175,"name":176,"points":177,"category":178},"2515612","Machine learning cheat sheet",166,"learning_science",{"id":180,"name":181,"points":182,"category":183},"2502173","Minimal Lisp in ~70 lines of Haskell",76,"languages",{"id":185,"name":186,"points":187,"category":188},"2495859","Timeslot",71,"productivity",{"id":190,"name":191,"points":192,"category":193},"2506627","Ambur",51,"web_saas",{"id":195,"name":196,"points":197,"category":188},"2503193","YC Interview Demo",50,{"id":199,"name":200,"points":201,"category":202},"2502150","my Hackathon Project",42,"consumer_social",false,"v2",{"id":206,"name":5,"line":6,"ai":203,"cat":20,"year":13,"points":14,"comments":15,"rec":27,"s":207,"top":18,"kw":208,"maker":11,"domain":10,"tier":19,"attempts":31,"strip":209,"url":9,"mix":210,"praise":73,"objection":86,"sub":21},2501495,0.06,[21,24,25,26],"101103300505",[30,30,31,32],1793990872554]