curl-impersonate
Dev tools
http client
SHOW HN2022
249points
57comments
#195of 2,570 in 2022
Top 8%of 2022 by points
How HN took it
CuriousMostly questions and neutral comments
4 positive, 8 neutral or asking, 1 mixed, 1 critical, out of 14 comments about the project. Thread score +0.18.
Idea: 3 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Useful for bypassing bot protection and learning fingerprinting tricks
From 4 comments
I run a MITM proxy for adblocking/general filtering and within the past little while I've noticed CloudFlare and other "bot protection" tends to get me blocked out of…
userbinator on HNVery cool! Thanks for sharing - it’s always nice to learn about fingerprinting tricks and workarounds, from both a privacy and a “don’t unintentionally look like a bot”…
wswope on HNVery cool. I would have used Puppeteer/Playwright in a similar scenario, but thanks for sharing the bot detection trick they employ.
oefrha on HNWould be cool if there was something like this for Python. Last time i tried to scrape something interesting i found that one of Cloudflare's enterprise options was easily…
tootahe45 on HN
Objections
Could help bots evade protections and make Firefox users get blocked more
From 3 comments
Cool, can't wait for anti-bot protection to start rejecting me because I use firefox.
vincent-toups on HNOnly a matter of time I'm afraid :( Firefox usage share is already low enough for many sites to make pages for Chrome and maybe Safari only.
npteljes on HNTor is banned (or rather, tar-pitted in endless CAPTCHAs) because the amount of legit users is massively dwarfed by the abuse If everyone running scrapers and attack probes start…
kalleboo on HN
Privacy concern that it enables profiling and approved-software checks
From 2 comments
I run a MITM proxy for adblocking/general filtering and within the past little while I've noticed CloudFlare and other "bot protection" tends to get me blocked out of…
userbinator on HN> profiling users based on other traits, and essentially determining if they are using "approved" software, is a dystopia we should fight strongly against. Stallman's Right To…
mschuster91 on HN
Requests
Impersonate Chrome instead of Firefox
From 2 comments
They should really be impersonating Chrome. If this takes off, Firefox has such a small user share that I could see sites just banning Firefox altogether, like they do with Tor
kalleboo on HNGiven the relative market shares it might make more sense to impersonate Chrome.
jandrese on HN
Use Firefox ESR rather than release version
From 1 comment
>impersonate Firefox 95 you should really be impersonating an ESR version (eg. 91). Versions from the release channel is updated every month or so, and everyone has autoupdate…
gruez on HN
Questions
Whether sites actually block non-browser clients by TLS fingerprint
From 2 comments
"Some web services therefore use the TLS handshake to fingerprint which HTTP client is accessing them. Notably, some bot protection platforms use this to identify curl and block…
1vuio0pswjnm7 on HNAre you sure they blocked you because of the handshake? Always thought it was the myriad of cookies and expiry time of said cookies that tend to make non-browser clients more…
ricardo81 on HN
Whether it could be merged back into curl
From 1 comment
Do you plan on getting this merged back into curl with an option to enable it? I can see that being useful for some people.
pabs3 on HN
Alternatives named
Puppeteer and Playwright headless browsers
From 2 comments
Very cool. I would have used Puppeteer/Playwright in a similar scenario, but thanks for sharing the bot detection trick they employ.
oefrha on HNI think most of the scraping libraries have stagnated since it's hard to scrape without a headless browser these days...too many sites with client-side rendered content.
tyingq on HN
Python scraping libraries
From 1 comment
Would be cool if there was something like this for Python. Last time i tried to scrape something interesting i found that one of Cloudflare's enterprise options was easily…
tootahe45 on HN