Apifier
Infrastructure
hosted web scraping
SHOW HN2015
How HN took it
3 positive, 12 neutral or asking, 0 mixed, 5 critical, out of 20 comments about the project. Thread score -0.01.
Idea: 2 for, 3 againstPerformance: 0 for, 1 againstPrivacy: 0 for, 1 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Useful demos and examples, usable without registering
From 2 comments
I love the demos, and that you can use them without registering. One thing I couldn't find without making an account was what happens after you've used a Gigabyte. That would be…
bentpins on HNAs a website owner, is it easy to block a rude crawler by contacting you ? (How would I identify in the first place that the crawler is operated by you ? Would my server logfile…
benjmn on HN
Would use it for home search or scraping
From 2 comments
Exactly what I was looking for in order to efficiently improve my searches for a new home. Thanks!
Raphmedia on HNI like the idea - would probably use it in the future - can you talk a little bit about what technologies you are using?
misiti3780 on HN
Objections
Using a full browser like PhantomJS is slow and expensive when plain HTTP requests suffice
From 2 comments
I don't quite understand why you would use a full-blown browser like phantomjs for crawling (I've seen a lot of projects recently taking this approach, so this critique is not…
thecodemonkey on HNIf you limit to just that, then there's no benefit over 10 lines of Ruby and Nokogiri.
pkulak on HN
Pricing unclear and risky without knowing cost at scale
From 2 comments
Looks really cool! Pricing is the big stickler for me. I've been burned too many times to build any critical piece of my app with it without knowing how much it'll cost if it…
rgbrgb on HNI love the demos, and that you can use them without registering. One thing I couldn't find without making an account was what happens after you've used a Gigabyte. That would be…
bentpins on HN
Requests
Publish pricing
From 2 comments
Looks really cool! Pricing is the big stickler for me. I've been burned too many times to build any critical piece of my app with it without knowing how much it'll cost if it…
rgbrgb on HNI love the demos, and that you can use them without registering. One thing I couldn't find without making an account was what happens after you've used a Gigabyte. That would be…
bentpins on HN
Programmatic API access to results
From 1 comment
How do you access the latest crawling results programmatically? I hope you are not expecting me to click results link for a developer's tool.
necrodome on HN
Caching and change detection on a schedule
From 1 comment
I'm hoping this could save me some work. A few questions, if founders are still around: -Can you cache pages / download entire sites? -If caching, can you detect changes on a…
danielharan on HN
Questions
Does it respect robots.txt and publish IP ranges?
From 2 comments
Do you respect robots.txt? Do you publish your IP ranges?
jacquesm on HNYes, by default we respect robots.txt. There is a switch to disable it - on your own responsibility. We don't fully respect Crawl-delay, but minimum delay between requests for…
jakubbalada on HN
How malicious user code is prevented
From 2 comments
Cool! How do you stop users from trying to run malicious code?
aakilfernandes on HNThank you! The user JavaScript code runs in the context of the web pages in the same restricted environment as normal web page's JavaScript. Also, the crawling processes are…
jancurn on HN
How to block the crawler and identify its traffic
From 2 comments
As a website owner, is it easy to block a rude crawler by contacting you ? (How would I identify in the first place that the crawler is operated by you ? Would my server logfile…
benjmn on HNCurrently, there is no way to distinguish traffic from our crawlers, but of course, let us know at support@apifier.com and we will blacklist your websites from our crawlers.
jancurn on HN
Alternatives named
morph.io
From 1 comment
Also similar to https://morph.io/ which has more of a trend towards open data sets.
thomasfromcdnjs on HN
Running PhantomJS on spot instances
From 1 comment
Why should I use this instead of just firing up some spot instances with phantomjs?
Eridrus on HN