Spider Pro
Dev tools
web scraper
SHOW HN2019
How HN took it
10 positive, 25 neutral or asking, 2 mixed, 11 critical, out of 48 comments about the project. Thread score -0.03.
Pricing: 1 for, 4 againstIdea: 2 for, 2 againstExecution: 0 for, 4 againstDesign: 3 for, 0 againstPerformance: 1 for, 1 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Easy to use with clean UI
From 2 comments
This is scary good and easy to use. Clever idea with the Chrome extension too.
joshmn on HNThis has a nice UI. It reminds me of Kantu (now https://ui.vision/) which I've used with varying degrees of success. That works by recording Selenium scripts; is Spider Pro…
porker on HN
Clever browser-extension approach
From 2 comments
This is scary good and easy to use. Clever idea with the Chrome extension too.
joshmn on HNDetection is always a cat and mouse game. Using an extension in a real browser (instead of electron/headless chrome) is probably one of the hardest to detect because it requires…
penagwin on HN
Objections
Pricing unclear or not cheapest versus free tools
From 3 comments
This is lacking clarity in the payment schedule... Is it One-time? Monthly? Yearly?
visualphoenix on HNHow is it the cheapest if it is not free? There are plenty of free scrapping tools out there
chirau on HNNonsense - beautiful soup is free. Don't pay for something you can program yourself.
impatientduck on HN
Not automated enough for large-scale scraping
From 2 comments
Go watch the demo video. From what I can see it's not automated enough to do tb/week. That said I bet you could hook it into scrapy and automate it. Probably will be slow though
chrismarlow9 on HNHow is this at all sustainable as a business model? My company is in charge of scraping a few TB/week. Would we be allowed to use your service?
gravypod on HN
Contact page broken
From 2 comments
Error info: contact us at the bottom shows page not found.
wumms on HNUnfortunate that the contact us page doesn't work. I'm interested, but I wanted to ask if a FF extension is planned, since that's my primary browser.
xurias on HN
Cannot re-scrape a site without starting over
From 1 comment
Seems to work well, but it appears that you can't re-scrape a site without starting from scratch, it's a one time deal.
turtlebits on HN
Requests
Safari extension support
From 1 comment
So is a Safari extension in development?
Eric_WVGG on HN
Firefox store distribution
From 1 comment
yup, firefox is supported! Though not distributed through the webstore - I'm planning to move it to the store sometimes soon.
hyperyolo on HN
Questions
How updates work for extensions distributed via Gumroad
From 1 comment
Since the extensions are being distributed via Gumroad, how would updates work?
atarian on HN
Whether it works on sites like Zillow, Facebook, LinkedIn
From 2 comments
I wonder how Spider Pro does with Facebook, Linkedin, Whitepages and others that try their best to block scraping but still have an introductory free to view webpage...
mehhh on HNInteresting that their demo screen cast shows Zillow. Zillow is fairly aggressive in applying anti-scraping defensive measures.
uptown on HN
Alternatives named
webscraper.io
From 1 comment
webscraper.io is free and has more functionality. I've used it for a quick and easy way to scrape data off multiple pages. Probably has a slightly higher learning curve but once…
ikeboy on HN
Beautiful Soup
From 1 comment
Nonsense - beautiful soup is free. Don't pay for something you can program yourself.
impatientduck on HN
Kimono Labs
From 1 comment
Ah it reminds me of Kimono Labs. I miss that product, it was fantastic.
bvm on HN
Diffbot
From 1 comment
https://www.diffbot.com/ is still well regarded, no?
basch on HN