[{"data":1,"prerenderedAt":321},["ShallowReactive",2],{"project-38230476":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":27,"sentiment":28,"stance_mix":29,"scored_comments":32,"aspects":33,"strip":41,"post_text":43,"what":44,"said":70,"maker_in_thread":145,"shape":154,"grid":159,"attempts":280,"same_week":285,"maker_shows":39,"maker_hidden":313,"reader_version":314,"card":315},"38230476","Flyscrape","A standalone and scriptable web scraper in Go","title","Show HN: Flyscrape – A standalone and scriptable web scraper in Go","https:\u002F\u002Fgithub.com\u002Fphilippta\u002Fflyscrape","github.com","philippta",1699712299,2023,208,45,266,3109,9,"notable","dev_tools","web scraper","cli",[21,24,25,26],"go scraper","scriptable scraping","javascript scraping","mixed",-0.161,{"positive":30,"neutral":31,"mixed":30,"critical":31},2,5,14,{"performance":34,"design":37,"docs":38,"idea":40},{"positive":35,"critical":36},0,4,{"positive":35,"critical":35},{"positive":35,"critical":39},1,{"positive":39,"critical":35},[39,35,35,35,39,35,42,39,42,35,39,35,42,42,39,35,31,35,31,35,35,35,35,35,35,35,35,35,35,35,35,35,35,42,35,35,35,42,35,35],3,null,{"problem":43,"audience":45,"pricing":43,"motivation":43,"time_to_build":43,"stack":52,"openness":69,"stage":69,"team_size":69},{"value":46,"source":47,"evidence":48},"People who scrape websites and are comfortable with some JavaScript","maker",[49],{"id":50,"by":11,"text":51},"38236548","Colly is a great scraping library if you are a Go developer. Flyscrape on the other hand is a ready-made CLI tool that aims to be easy to use even for someone who is a little…",[53,61,65],{"value":54,"source":55,"evidence":56},"Go","commenter",[57],{"id":58,"by":59,"text":60},"38231714","xyzzy_plugh","I like web scraping in Go. The support for parsing HTML in x\u002Ftext\u002Fhtml is pretty good, and libraries like github.com\u002FPuerkitoBio\u002Fgoquery go a long way to matching ergonomics in…",{"value":62,"source":55,"evidence":63},"goja JavaScript VM",[64],{"id":58,"by":59,"text":60},{"value":66,"source":55,"evidence":67},"goquery",[68],{"id":58,"by":59,"text":60},"unknown",{"praise":71,"objections":84,"requests":108,"questions":121,"alternatives":129},[72],{"value":73,"source":55,"evidence":74},"Good engineering and an easy tool for simple scraping",[75,79,83],{"id":76,"by":77,"text":78},"38231162","lucgagan","This looks great. I wish I had this a few months ago! Giving it a try.",{"id":80,"by":81,"text":82},"38239394","unixhero","I will test this, great stuff",{"id":58,"by":59,"text":60},[85,100],{"value":86,"source":55,"evidence":87},"Does not run JavaScript or browsers, so many real sites fail",[88,92,96],{"id":89,"by":90,"text":91},"38237083","oefrha","I had a look at https:\u002F\u002Fgithub.com\u002Fphilippta\u002Fflyscrape\u002Fblob\u002Fmaster\u002Fscrape.go. It’s just using the builtin HTTP client to fire off requests, with an identifying user agent. Which…",{"id":93,"by":94,"text":95},"38231427","bryanrasmussen","Looks like it doesn't have the possibility of running it as a particular browser etc. Which I guess makes it fine for a lot of pages, but also a lot of scraping tasks would be…",{"id":97,"by":98,"text":99},"38239618","specproc","I've not looked at the source code, but if GP is correct, then absent JS rendering means there's little added value for me (a dude who scrapes a lot). Real world example, I was…",{"value":101,"source":55,"evidence":102},"Default user agent gets blocked by many sites",[103,104],{"id":89,"by":90,"text":91},{"id":105,"by":106,"text":107},"38237590","Klonoar","Yeah, nobody cares that most sites will work with it. The number of sites that require it might be small, but that number tends to include significant and notable ones - the ones…",[109,117],{"value":110,"source":55,"evidence":111},"Built-in JavaScript rendering support",[112,113],{"id":97,"by":98,"text":99},{"id":114,"by":115,"text":116},"38231550","snake117","Looks interesting, and thank you for sharing this! One common issue with scraping web pages is dealing with data that is dynamically loaded. Is there a solution for this? For…",{"value":118,"source":55,"evidence":119},"Ability to set the user agent",[120],{"id":105,"by":106,"text":107},[122],{"value":123,"source":55,"evidence":124},"How does it compare to colly",[125],{"id":126,"by":127,"text":128},"38236439","moehm","Interesting. Can you compare it to colly? [0] Last time I looked it was the most popular choice for scraping in Go and I have some projects using it. Is it similar? Does it have…",[130,134,141],{"value":131,"source":55,"evidence":132},"colly",[133],{"id":126,"by":127,"text":128},{"value":135,"source":55,"evidence":136},"Playwright and Crawlee",[137],{"id":138,"by":139,"text":140},"38232032","sunshadow","Its been 8+ years since i started scraping. I even wrote a popular Go web scraping framework previously: (https:\u002F\u002Fgithub.com\u002Fgeziyor\u002Fgeziyor). These days, I'm not even using Go…",{"value":142,"source":55,"evidence":143},"Scrapy with Splash",[144],{"id":114,"by":115,"text":116},{"comments":31,"answered":42,"key":146},[147,148,151],{"id":50,"text":51},{"id":149,"text":150},"38231523","Yes, this is correct. As of right now there is no built-in support for running as a browser. What is possible though, is to use a service like ScrapingBee (not affiliated) and…",{"id":152,"text":153},"38236565","Thanks for spotting, will get this fixed quickly.",{"commenters":155,"max_depth":156,"maker_replies":31,"first_comment_mins":157,"lifespan_hours":158},32,6,72,297,[160,162,166,167,169,170,172,176,178,180,182,184,185,187,190,194,198,202,204,207,210,213,214,218,221,224,226,229,232,234,237,239,242,245,248,251,254,257,260,263,266,268,270,272,275,278],[89,35,39,161,90,35],"project\u002Fcritical\u002Fperformance",[163,35,35,164,165,35],"38237410","commenter\u002Fneutral\u002Fnone","1vuio0pswjnm7",[105,35,35,164,106,35],[168,35,35,164,90,35],"38237595",[97,35,39,161,98,35],[171,35,35,164,165,35],"38399077",[173,35,42,174,175,35],"38241173","project\u002Fneutral\u002Fperformance","rmilejczz",[177,35,35,164,165,35],"38247037",[179,39,39,161,165,35],"38236897",[126,30,42,181,127,35],"project\u002Fneutral\u002Fnone",[50,30,35,183,11,39],"project\u002Fneutral\u002Fidea",[93,42,39,161,94,35],[149,42,35,186,11,39],"maker\u002Fneutral\u002Fperformance",[188,42,42,174,189,35],"38233707","acheong08",[191,36,42,192,193,35],"38234840","project\u002Fneutral\u002Fdesign","fyzix",[195,31,42,196,197,35],"38237229","project\u002Fmixed\u002Fdocs","krick",[199,156,39,200,201,35],"38234554","project\u002Fcritical\u002Fdocs","slig",[152,156,35,203,11,39],"maker\u002Fneutral\u002Fdocs",[76,205,31,206,77,35],7,"project\u002Fpositive\u002Fidea",[208,205,35,209,11,39],"38231712","maker\u002Fneutral\u002Fnone",[80,211,31,212,81,35],8,"project\u002Fpositive\u002Fnone",[138,18,35,164,139,35],[215,18,35,216,217,35],"38232152","other\u002Fneutral\u002Fnone","mikercampbell",[219,18,35,164,220,35],"38232425","reyostallenberg",[222,18,35,216,223,35],"38233755","mdaniel",[225,18,35,216,139,35],"38232489",[227,18,35,216,228,35],"38278748","docyes",[230,18,35,164,231,35],"38232257","hipadev23",[233,18,35,164,139,35],"38232414",[235,18,35,164,236,35],"38232567","8n4vidtmkvmk",[238,18,35,164,139,35],"38232609",[240,18,35,164,241,35],"38233442","erhaetherth",[243,18,35,164,244,35],"38233204","latchkey",[246,18,35,164,247,35],"38243033","bluecrab",[249,18,35,164,250,35],"38232617","nurettin",[252,18,35,164,253,35],"38235965","aynyc",[255,18,35,164,256,35],"38236284","gymbeaux",[58,258,42,259,59,35],10,"project\u002Fmixed\u002Fidea",[261,258,35,164,262,35],"38231760","cxr",[264,258,35,164,265,35],"38232101","cheapgeek",[267,258,35,164,59,35],"38232390",[269,258,35,216,262,35],"38255744",[114,271,42,174,115,35],11,[273,271,35,164,274,35],"38231654","figmert",[276,271,35,277,223,35],"38231922","commenter\u002Fneutral\u002Fperformance",[279,271,35,186,11,39],"38231687",[281,284],{"id":282,"t":283,"points":32,"comments":31},"37871064",1697207555,{"id":4,"t":12,"points":14,"comments":15},[286,291,295,300,304,308],{"id":287,"name":288,"points":289,"category":290},"38243949","dpoint",583,"hardware",{"id":292,"name":293,"points":294,"category":20},"38230325","Spray",393,{"id":296,"name":297,"points":298,"category":299},"38228788","DataSheetGrid",342,"languages",{"id":301,"name":302,"points":303,"category":299},"38202703","BabaBASIC",234,{"id":305,"name":306,"points":307,"category":299},"38276430","Numbat",146,{"id":309,"name":310,"points":311,"category":312},"38236198","bulk_transcribe_youtube_videos_from_playlist",125,"ai",false,"v2",{"id":316,"name":5,"line":6,"ai":313,"cat":20,"year":13,"points":14,"comments":15,"rec":27,"s":317,"top":18,"kw":318,"maker":11,"domain":10,"tier":19,"attempts":30,"strip":319,"url":9,"mix":320,"praise":73,"objection":86,"sub":21},38230476,-0.16,[21,24,25,26],"1000103130103310505000000000000003000300",[30,31,30,31],1794036644545]