[{"data":1,"prerenderedAt":355},["ShallowReactive",2],{"project-30378562":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":33,"aspects":34,"strip":42,"post_text":44,"what":45,"said":73,"maker_in_thread":162,"shape":174,"grid":179,"attempts":317,"same_week":318,"maker_shows":32,"maker_hidden":347,"reader_version":348,"card":349},"30378562","curl-impersonate","Curl modified to impersonate Firefox and mimic its TLS handshake","title","Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake","https:\u002F\u002Fgithub.com\u002Flwthiker\u002Fcurl-impersonate","github.com","lwthiker",1645131457,2022,249,57,195,2570,8,"notable","dev_tools","http client","cli",[24,25,26,27,21],"curl","tls fingerprint","firefox","bot protection","curious",0.178,{"positive":31,"neutral":18,"mixed":32,"critical":32},4,1,14,{"privacy":35,"execution":37,"design":38,"idea":39,"docs":41},{"positive":36,"critical":36},0,{"positive":36,"critical":36},{"positive":36,"critical":36},{"positive":40,"critical":36},3,{"positive":36,"critical":36},[40,36,36,36,36,36,36,36,36,36,36,36,36,36,36,40,36,36,40,36,32,40,36,36,36,40,40,36,36,36,43,36,36,36,43,36,40,43,36,43],5,null,{"problem":46,"audience":44,"pricing":44,"motivation":44,"time_to_build":44,"stack":58,"openness":71,"stage":72,"team_size":71},{"value":47,"source":48,"evidence":49},"Some web services use the TLS handshake to identify and block curl and other non-browser clients","commenter",[50,54],{"id":51,"by":52,"text":53},"30383287","1vuio0pswjnm7","\"Some web services therefore use the TLS handshake to fingerprint which HTTP client is accessing them. Notably, some bot protection platforms use this to identify curl and block…",{"id":55,"by":56,"text":57},"30381440","tootahe45","Would be cool if there was something like this for Python. Last time i tried to scrape something interesting i found that one of Cloudflare's enterprise options was easily…",[59,64],{"value":24,"source":60,"evidence":61},"maker",[62],{"id":63},"post",{"value":65,"source":48,"evidence":66},"NSS",[67],{"id":68,"by":69,"text":70},"30383184","bburky","There was a conversation on their mailing list contemplating dropping NSS support. https:\u002F\u002Fcurl.se\u002Fmail\u002Flib-2022-01\u002F0120.html If you have a use case for NSS in curl, you may want…","unknown","beta",{"praise":74,"objections":91,"requests":115,"questions":133,"alternatives":149},[75],{"value":76,"source":48,"evidence":77},"Useful for bypassing bot protection and learning fingerprinting tricks",[78,82,86,90],{"id":79,"by":80,"text":81},"30381628","userbinator","I run a MITM proxy for adblocking\u002Fgeneral filtering and within the past little while I've noticed CloudFlare and other \"bot protection\" tends to get me blocked out of…",{"id":83,"by":84,"text":85},"30380040","wswope","Very cool! Thanks for sharing - it’s always nice to learn about fingerprinting tricks and workarounds, from both a privacy and a “don’t unintentionally look like a bot”…",{"id":87,"by":88,"text":89},"30381050","oefrha","Very cool. I would have used Puppeteer\u002FPlaywright in a similar scenario, but thanks for sharing the bot detection trick they employ.",{"id":55,"by":56,"text":57},[92,107],{"value":93,"source":48,"evidence":94},"Could help bots evade protections and make Firefox users get blocked more",[95,99,103],{"id":96,"by":97,"text":98},"30380852","vincent-toups","Cool, can't wait for anti-bot protection to start rejecting me because I use firefox.",{"id":100,"by":101,"text":102},"30383940","npteljes","Only a matter of time I'm afraid :( Firefox usage share is already low enough for many sites to make pages for Chrome and maybe Safari only.",{"id":104,"by":105,"text":106},"30383555","kalleboo","Tor is banned (or rather, tar-pitted in endless CAPTCHAs) because the amount of legit users is massively dwarfed by the abuse If everyone running scrapers and attack probes start…",{"value":108,"source":48,"evidence":109},"Privacy concern that it enables profiling and approved-software checks",[110,111],{"id":79,"by":80,"text":81},{"id":112,"by":113,"text":114},"30383958","mschuster91","> profiling users based on other traits, and essentially determining if they are using \"approved\" software, is a dystopia we should fight strongly against. Stallman's Right To…",[116,126],{"value":117,"source":48,"evidence":118},"Impersonate Chrome instead of Firefox",[119,122],{"id":120,"by":105,"text":121},"30381699","They should really be impersonating Chrome. If this takes off, Firefox has such a small user share that I could see sites just banning Firefox altogether, like they do with Tor",{"id":123,"by":124,"text":125},"30381720","jandrese","Given the relative market shares it might make more sense to impersonate Chrome.",{"value":127,"source":48,"evidence":128},"Use Firefox ESR rather than release version",[129],{"id":130,"by":131,"text":132},"30380589","gruez",">impersonate Firefox 95 you should really be impersonating an ESR version (eg. 91). Versions from the release channel is updated every month or so, and everyone has autoupdate…",[134,142],{"value":135,"source":48,"evidence":136},"Whether sites actually block non-browser clients by TLS fingerprint",[137,138],{"id":51,"by":52,"text":53},{"id":139,"by":140,"text":141},"30384038","ricardo81","Are you sure they blocked you because of the handshake? Always thought it was the myriad of cookies and expiry time of said cookies that tend to make non-browser clients more…",{"value":143,"source":48,"evidence":144},"Whether it could be merged back into curl",[145],{"id":146,"by":147,"text":148},"30381838","pabs3","Do you plan on getting this merged back into curl with an option to enable it? I can see that being useful for some people.",[150,158],{"value":151,"source":48,"evidence":152},"Puppeteer and Playwright headless browsers",[153,154],{"id":87,"by":88,"text":89},{"id":155,"by":156,"text":157},"30381461","tyingq","I think most of the scraping libraries have stagnated since it's hard to scrape without a headless browser these days...too many sites with client-side rendered content.",{"value":159,"source":48,"evidence":160},"Python scraping libraries",[161],{"id":55,"by":56,"text":57},{"comments":31,"answered":163,"key":164},2,[165,168,171],{"id":166,"text":167},"30382591","Thanks for the suggestion, I had no idea ESR was a thing. I've just added support for Firefox ESR 91 (it was pretty similar and required adding one cipher to the cipher list and…",{"id":169,"text":170},"30382618","I will try to impersonate Chrome next, However, I suspect this is going to be more challenging. Chrome uses BoringSSL, which curl does not support. So it means either enforcing…",{"id":172,"text":173},"30382774","I hope to do so in the future, for now the implementation is extremely hacky so I doubt it can get accepted into curl.",{"commenters":175,"max_depth":176,"maker_replies":31,"first_comment_mins":177,"lifespan_hours":178},42,6,137,121,[180,182,186,189,192,195,199,202,205,207,210,213,216,219,221,224,226,229,231,234,235,237,239,242,243,245,248,250,252,253,254,257,260,263,265,267,270,272,273,274,276,280,282,283,284,286,288,291,294,297,301,303,305,309,312],[79,36,40,181,80,36],"project\u002Fmixed\u002Fprivacy",[183,36,36,184,185,36],"30382115","commenter\u002Fneutral\u002Fnone","oh_sigh",[187,36,36,184,188,36],"30382304","llampx",[190,36,36,184,191,36],"30382395","saurik",[193,36,36,184,194,36],"30383798","bayindirh",[196,36,36,197,198,36],"30382289","other\u002Fneutral\u002Fnone","cma",[200,36,36,197,201,36],"30384378","caulk",[203,36,36,197,204,36],"30385056","raxxorrax",[206,36,36,197,80,36],"30386100",[208,36,36,197,209,36],"30382815","fake-name",[211,36,36,197,212,36],"30383110","buro9",[214,36,36,197,215,36],"30409826","kevincox",[217,36,36,197,218,36],"30383418","rtpg",[220,36,36,197,209,36],"30384463",[222,36,36,184,223,36],"30385141","lmz",[225,36,36,184,80,36],"30390778",[227,36,36,184,228,36],"30383451","charcircuit",[230,36,36,184,209,36],"30384479",[232,36,36,184,233,36],"30384534","account42",[112,36,36,197,113,36],[130,32,40,236,131,36],"project\u002Fneutral\u002Fexecution",[120,32,40,238,105,36],"project\u002Fneutral\u002Fdesign",[240,32,36,184,241,36],"30383313","jve",[104,32,36,184,105,36],[244,32,36,184,88,36],"30383685",[246,32,40,236,247,36],"30382414","LanternLight83",[166,32,36,249,11,32],"maker\u002Fpositive\u002Fexecution",[96,163,32,251,97,36],"project\u002Fcritical\u002Fnone",[100,163,36,197,101,36],[123,40,40,238,124,36],[255,40,36,184,256,36],"30381935","7v3x3n3sem9vv",[258,40,36,184,259,36],"30383199","flawi",[261,40,36,184,262,36],"30434212","Handytinge",[169,40,36,264,11,32],"maker\u002Fneutral\u002Fexecution",[51,31,40,266,52,36],"project\u002Fneutral\u002Fnone",[268,43,36,197,269,36],"30380593","0xbkt",[146,176,40,271,147,36],"project\u002Fneutral\u002Fidea",[172,176,36,264,11,32],[68,176,36,197,69,36],[275,176,36,184,147,36],"30383877",[277,278,36,197,279,36],"30388267",7,"sylware",[55,18,43,281,56,36],"project\u002Fpositive\u002Fidea",[155,18,36,197,156,36],[139,18,36,184,140,36],[285,18,36,184,56,36],"30393964",[287,18,36,184,140,36],"30395616",[83,289,43,290,84,36],9,"maker\u002Fpositive\u002Fidea",[292,289,40,293,88,36],"30381051","maker\u002Fneutral\u002Fdocs",[295,289,36,197,296,36],"30381717","jart",[298,299,40,271,300,36],"30383960",10,"octoberfranklin",[302,299,36,264,11,32],"30384210",[87,304,43,281,88,36],11,[306,307,36,197,308,36],"30381156",12,"javajosh",[310,307,36,197,311,36],"30381308","tenebrisalietum",[313,314,43,315,316,36],"30415722",13,"project\u002Fpositive\u002Fnone","kinderjaje",[],[319,324,329,334,338,342],{"id":320,"name":321,"points":322,"category":323},"30399881","Judy Records",698,"data",{"id":325,"name":326,"points":327,"category":328},"30408434","Rotator",659,"games",{"id":330,"name":331,"points":332,"category":333},"30440828","Simula One",463,"hardware",{"id":335,"name":336,"points":337,"category":323},"30417411","google-drive-to-sqlite",305,{"id":339,"name":340,"points":341,"category":333},"30446960","endless-sphere.com",304,{"id":343,"name":344,"points":345,"category":346},"30440275","Supernotes 2",251,"productivity",false,"v2",{"id":350,"name":5,"line":6,"ai":347,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":351,"top":18,"kw":352,"maker":11,"domain":10,"tier":19,"attempts":36,"strip":353,"url":9,"mix":354,"praise":76,"objection":93,"sub":21},30378562,0.18,[24,25,26,27,21],"3000000000000003003013000330005000503505",[31,18,32,32],1794036617114]