Headless Chrome Crawler
Dev tools
web crawler
SHOW HN2018
How HN took it
1 positive, 14 neutral or asking, 0 mixed, 4 critical, out of 19 comments about the project. Thread score -0.06.
Execution: 1 for, 4 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Pretty cool; useful for custom crawlers
From 1 comment
Pretty cool, but I recommend anyone wanting to do this kind of thing to check out the source Puppeteer library. You can do some really powerful stuff and make a custom crawler…
ptasker on HN
Objections
Headless browsers use high resources and are hard to supervise
From 2 comments
This has been possible for a long time with any browser using Selenium for example. It has APIs and client libraries for many languages. Also using a real browser brings a lot of…
codedokode on HNThere are a lot of folks reevaluating their crawling engines lately now that Chrome headless is maturing. To me there are some important considerations in terms of CPU/memory…
tegansnyder on HN
Crawlers without rate limits burden servers and need banning
From 1 comment
While as a developer I find this super interesting, as a system administrator this makes me cringe. We don't have a lot of resources for servers, and I end up spending a…
tesin on HN
Would rather extend Puppeteer than a new library
From 1 comment
Yeah I’d really rather that people made extensions to Pupeteer rather than a whole new library.
isuckatcoding on HN
Requests
Support for proxies with authentication
From 1 comment
I was stuck last time I was using headless chrome when I needed to use a proxy with an username and a paasword. Headless chrome just doesn't support it. Any changes on that?
hartator on HN
Support for scaling and distributing across machines
From 1 comment
Nice job! Can this be scaled and distributed to multiple machines?
agotterer on HN
Questions
How to access individual elements from the page body
From 1 comment
Can't see from examples, how do I get back individual elements from the body?
bryanrasmussen on HN
How it handles pages that load links via JavaScript
From 2 comments
also how does this handle pages that load with a small number of links and then uses JS to write in a bunch of DOM nodes and links?
bryanrasmussen on HNmost naive code I see is like the following ``` const page = await crawler.browser.newPage(); await page.goto(url); await page.waitForSelector("a[href]"); const hrefs = await…
bryanrasmussen on HN
Alternatives named
Puppeteer
From 1 comment
Pretty cool, but I recommend anyone wanting to do this kind of thing to check out the source Puppeteer library. You can do some really powerful stuff and make a custom crawler…
ptasker on HN
Selenium
From 1 comment
This has been possible for a long time with any browser using Selenium for example. It has APIs and client libraries for many languages. Also using a real browser brings a lot of…
codedokode on HN
Supercrawler
From 1 comment
there's this too: https://github.com/brendonboshell/supercrawler
artur_makly on HN
Browserless
From 1 comment
maybe with this? https://Browserless.io
artur_makly on HN