Flyscrape
Dev tools
web scraper
SHOW HN2023
208points
45comments
#266of 3,109 in 2023
Top 9%of 2023 by points
How HN took it
MixedNo clear lean
2 positive, 5 neutral or asking, 2 mixed, 5 critical, out of 14 comments about the project. Thread score -0.16.
Performance: 0 for, 4 againstDocs: 0 for, 1 againstIdea: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Good engineering and an easy tool for simple scraping
From 3 comments
This looks great. I wish I had this a few months ago! Giving it a try.
lucgagan on HNI will test this, great stuff
unixhero on HNI like web scraping in Go. The support for parsing HTML in x/text/html is pretty good, and libraries like github.com/PuerkitoBio/goquery go a long way to matching ergonomics in…
xyzzy_plugh on HN
Objections
Does not run JavaScript or browsers, so many real sites fail
From 3 comments
I had a look at https://github.com/philippta/flyscrape/blob/master/scrape.go. It’s just using the builtin HTTP client to fire off requests, with an identifying user agent. Which…
oefrha on HNLooks like it doesn't have the possibility of running it as a particular browser etc. Which I guess makes it fine for a lot of pages, but also a lot of scraping tasks would be…
bryanrasmussen on HNI've not looked at the source code, but if GP is correct, then absent JS rendering means there's little added value for me (a dude who scrapes a lot). Real world example, I was…
specproc on HN
Default user agent gets blocked by many sites
From 2 comments
I had a look at https://github.com/philippta/flyscrape/blob/master/scrape.go. It’s just using the builtin HTTP client to fire off requests, with an identifying user agent. Which…
oefrha on HNYeah, nobody cares that most sites will work with it. The number of sites that require it might be small, but that number tends to include significant and notable ones - the ones…
Klonoar on HN
Requests
Built-in JavaScript rendering support
From 2 comments
I've not looked at the source code, but if GP is correct, then absent JS rendering means there's little added value for me (a dude who scrapes a lot). Real world example, I was…
specproc on HNLooks interesting, and thank you for sharing this! One common issue with scraping web pages is dealing with data that is dynamically loaded. Is there a solution for this? For…
snake117 on HN
Ability to set the user agent
From 1 comment
Yeah, nobody cares that most sites will work with it. The number of sites that require it might be small, but that number tends to include significant and notable ones - the ones…
Klonoar on HN
Questions
How does it compare to colly
From 1 comment
Interesting. Can you compare it to colly? [0] Last time I looked it was the most popular choice for scraping in Go and I have some projects using it. Is it similar? Does it have…
moehm on HN
Alternatives named
colly
From 1 comment
Interesting. Can you compare it to colly? [0] Last time I looked it was the most popular choice for scraping in Go and I have some projects using it. Is it similar? Does it have…
moehm on HN
Playwright and Crawlee
From 1 comment
Its been 8+ years since i started scraping. I even wrote a popular Go web scraping framework previously: (https://github.com/geziyor/geziyor). These days, I'm not even using Go…
sunshadow on HN
Scrapy with Splash
From 1 comment
Looks interesting, and thank you for sharing this! One common issue with scraping web pages is dealing with data that is dynamically loaded. Is there a solution for this? For…
snake117 on HN