Kimono
Dev tools
web scraping
SHOW HN2014
717points
230comments
#6of 2,272 in 2014
Top 1%of 2014 by points
How HN took it
MixedNo clear lean
32 positive, 51 neutral or asking, 12 mixed, 25 critical, out of 120 comments about the project. Thread score +0.10.
Execution: 5 for, 14 againstDesign: 12 for, 2 againstIdea: 8 for, 4 againstPrivacy: 0 for, 2 againstPerformance: 1 for, 1 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Useful for simplifying common scraping tasks
From 2 comments
This is excellent. Even it if doesn't work for scraping all sites, it simplifies the average use case so much that it's not even funny. Feature proposal: deal with pagination.
hcarvalhoalves on HNI like how you've thought through the end to end use case: not just generating an API, but actually making it usable. I've done my fair share of web scraping and it's not an easy…
jlees on HN
Objections
Breaks on many sites and buggy saves/selection
From 2 comments
The presentation is beautiful and the website is great, but the tech broke so I have no idea how or if this even works. This is a wonderful concept and one I've talked about…
randomdrake on HNConstructive Tone: I figured that it might be nifty to scrape cedar pollen count information from a calendar and then shoot myself an email when it was higher than 100 gr/m3.…
fsckin on HN
Legality and IP bans from scraping at scale
From 3 comments
I've written more web scraping code than I care to admit. A lot of the apps that ran on chumby devices used scraping to get their data (usually(!) with the consent of the website…
georgemcbay on HNI'm curious how you plan to avoid/circumvent the inevitable hard IP ban that the largest (and most sought after targets) will place on you and your services once you begin to…
GigabyteCoin on HNThe UX is great and a journalists everywhere will thank you. But outside of government websites I don't see how a lot of this is even legal, per se?
phillmv on HN
Requests
Authentication / login support
From 3 comments
Yes, it's one of the most popular feature requests. We don't support auth yet, but it's on our shortlist and we hope to have it ready soon.
pranade on HNWe're working on auth... it's the most requested feature at the moment. And we're still beta at the moent, so all usage is free
pranade on HNShow me it working with authentication and you will have a customer. Scraping is always something you need to write because the shit you want to get is only shown when you are…
DanBlake on HN
Pagination and crawling
From 2 comments
thanks. glad you like it. pagination, dynamic tabs (and crawling in general) is a big feature we really want to add soon. a lot of people are asking for it. the challenge will be…
pranade on HNThis is excellent. Even it if doesn't work for scraping all sites, it simplifies the average use case so much that it's not even funny. Feature proposal: deal with pagination.
hcarvalhoalves on HN
POST support and multiple targets from a list
From 2 comments
No, we don't have POST support quite yet. We're working on a solution.
pranade on HNGreat work so far. The tool was very intuitive and easy to use. My suggestion: once I've defined an API, let me apply it to multiple targets that I supply to you programatically.…
thinkzig on HN
Image extraction
From 2 comments
Thank you for building a tool I been wanting so I don't have to! Can't wait to play around with this tonight. Suggestion. Allow one to select images.
ThomPete on HNIt's a great suggestion, thanks! ... image extraction would be cool, and it's on our shortlist of features to build next
pranade on HN
Questions
How it handles robots.txt and legality
From 2 comments
This looks really slick. What happens if a website you're scraping changes its design? Do you respect robots.txt?
ForHackernews on HNThe UX is great and a journalists everywhere will thank you. But outside of government websites I don't see how a lot of this is even legal, per se?
phillmv on HN
How it adapts when source sites change
From 2 comments
> Web scraping. It's something we all love to hate. You wish the data you needed to power your app, model or visualization was available via API. But, most of the time it's not.…
bambax on HNThis looks really slick. What happens if a website you're scraping changes its design? Do you respect robots.txt?
ForHackernews on HN
Alternatives named
Simile/MIT web augmentation research
From 1 comment
The Simile group at MIT did something similar back around 2006. Automatic identification of collections in web pages (repeated structures), detection of fields by doing tree…
dunham on HN
ScraperWiki
From 1 comment
Are you familiar with ScraperWiki? I'm wondering how your work fits in with it. Edit: looks like they've moved away from that space, but have an old version available at:…
rlpb on HN
SelectorGadget
From 1 comment
Just use http://selectorgadget.com
tectonic on HN
iMacros, Beautiful Soup, Nokogiri
From 3 comments
I'm curious how you plan to avoid/circumvent the inevitable hard IP ban that the largest (and most sought after targets) will place on you and your services once you begin to…
GigabyteCoin on HN> Web scraping. It's something we all love to hate. You wish the data you needed to power your app, model or visualization was available via API. But, most of the time it's not.…
bambax on HNReally sleek interface, and looks like it could be extremely useful (I just spent a few hours cranking out Nokogiri this morning). Oh, typo: "Notice that toolbar at the toop of…
jmcgough on HN