BrowserCrawler
Infrastructure
web crawler
SHOW HN2011
43points
12comments
#286of 1,097 in 2011
Top 27%of 2011 by points
How HN took it
DebatedReal support and real criticism
2 positive, 2 neutral or asking, 0 mixed, 3 critical, out of 7 comments about the project. Thread score +0.06.
Idea: 2 for, 2 againstPerformance: 0 for, 1 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Cool project
From 2 comments
That is very cool.
petervandijck on HNGreat! I had the same idea only with GAE integration and option to download locally. Any plans on expanding that interface?
neuromancer2600 on HN
Objections
Crawling can overload servers and needs rate limiting
From 1 comment
Is there a header set to identify this crawler so I can limit it? For what it's worth, when people crawl sites I am responsible for, and it impacts our products performance, I…
ryan-allen on HN
Copyright and terms of service risk from mirroring content in public S3 buckets
From 1 comment
Ah, I'm familiar with S3 latencies. That'd work. Also, if you're uploading other people's content to public S3 buckets, you're very likely to be in violation of mosts site's…
ryan-allen on HN
Should honour robots.txt
From 1 comment
What's this X-User-Agent stuff? The entire internet isn't going to special-case your hack. Just honour robots.txt.
JoachimSchipper on HN
Requests
Store pages locally on the computer
From 1 comment
How hard would it be to modify it to store it locally on your computer?
raptrex on HN
Google App Engine integration and local download option
From 1 comment
Great! I had the same idea only with GAE integration and option to download locally. Any plans on expanding that interface?
neuromancer2600 on HN
Questions
Can the crawler be identified by a header to limit it?
From 1 comment
Is there a header set to identify this crawler so I can limit it? For what it's worth, when people crawl sites I am responsible for, and it impacts our products performance, I…
ryan-allen on HN
Alternatives named
Scrapbook Firefox extension
From 1 comment
There is a nice Firefox plugin called Scrapbook that does this.
Harkins on HN
wget
From 1 comment
``` wget --spider ``` is probably your friend
dotBen on HN