[{"data":1,"prerenderedAt":315},["ShallowReactive",2],{"project-30675357":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":36,"aspects":37,"strip":45,"post_text":46,"what":47,"said":87,"maker_in_thread":179,"shape":190,"grid":194,"attempts":277,"same_week":278,"maker_shows":180,"maker_hidden":307,"reader_version":308,"card":309},"30675357","shot-scraper-template","Instantly create a GitHub repository to take screenshots of a web page","title","Show HN: Instantly create a GitHub repository to take screenshots of a web page","https:\u002F\u002Fsimonwillison.net\u002F2022\u002FMar\u002F14\u002Fshot-scraper-template\u002F","simonwillison.net","simonw",1647279273,2022,214,36,257,2570,10,"notable","dev_tools","screenshot automation template","library",[24,25,26,27,28],"github actions","screenshot","web scraping","playwright","github template","curious",0.213,{"positive":32,"neutral":33,"mixed":34,"critical":35},5,8,0,1,14,{"idea":38,"execution":40,"design":41,"pricing":42,"docs":43,"privacy":44},{"positive":39,"critical":34},3,{"positive":35,"critical":35},{"positive":34,"critical":34},{"positive":35,"critical":34},{"positive":34,"critical":34},{"positive":34,"critical":34},[34,32,34,34,39,34,34,39,34,39,34,34,39,34,39,34,32,32,39,34,34,34,34,34,34,34,34,34,34,32,35,34,32,39,34,39],"I built a GitHub repository template which automates the process of configuring a new repository to take web page screenshots using GitHub Actions.\n\nYou can try this out at https:\u002F\u002Fgithub.com\u002Fsimonw\u002Fshot-scraper-template\n\nUse the https:\u002F\u002Fgithub.com\u002Fsimonw\u002Fshot-scraper-template\u002Fgenerate interface to create a new repository using that template, and paste the URL that you want to take screenshots of in as the \"description\" field.\n\nThe new repository will then configure itself using GitHub Actions, take the screenshot and save it back to the repo!",{"problem":48,"audience":54,"pricing":55,"motivation":62,"time_to_build":54,"stack":66,"openness":84,"stage":85,"team_size":86},{"value":49,"source":50,"evidence":51},"Taking recurring screenshots of web pages without manual setup","inferred",[52],{"id":53},"post",null,{"value":56,"source":57,"evidence":58},"Free on public GitHub repos; Actions minutes cost on private repos","maker",[59],{"id":60,"by":11,"text":61},"30676228","I do. I'm happy to pay for Actions minutes on private repos, but I do worry that they'll change their policy with regards to free minutes for public repos at some point. I felt a…",{"value":63,"source":57,"evidence":64},"Make it easy to take web page screenshots using GitHub Actions",[65],{"id":53},[67,71,77],{"value":68,"source":57,"evidence":69},"GitHub Actions",[70],{"id":53},{"value":72,"source":57,"evidence":73},"Python (shot-scraper)",[74],{"id":75,"by":11,"text":76},"30676384","Related to this: you can also use my shot-scraper tool to scrape web pages from the command line using JavaScript: ``` % pip install shot-scraper % shot-scraper install %…",{"value":78,"source":79,"evidence":80},"Playwright","commenter",[81],{"id":82,"by":11,"text":83},"30679885","You could absolutely get this working with GitHub Actions with a bit of creativity. I've been playing around with my own image-diff tool for this kind of thing, but it's not yet…","open_source","launched","solo",{"praise":88,"objections":100,"requests":120,"questions":142,"alternatives":153},[89],{"value":90,"source":79,"evidence":91},"Good execution and cool idea",[92,96],{"id":93,"by":94,"text":95},"30682595","dudus","GitHub does have some very generous free tier stuff. But as these sort of things become more common it's destined to be lowered or more heavily limited. Nothing against the…",{"id":97,"by":98,"text":99},"30677182","cancan","This is really cool and I love the idea of using SVGs to even add annotations, as you mentioned on your tweets. I might be \"borrowing\" that idea soon for some our own work, and…",[101,109],{"value":102,"source":79,"evidence":103},"Free GitHub Actions minutes for public repos may be restricted later",[104,108],{"id":105,"by":106,"text":107},"30676182","0des","Love the project. Do you ever worry about Microsoft tightening the purse strings on these types of off-label uses for github?",{"id":93,"by":94,"text":95},{"value":110,"source":79,"evidence":111},"Pages with slow loading or consent popups are not captured well",[112,116],{"id":113,"by":114,"text":115},"30683381","stared","Thanks for sharing! It does not work though for pages that take time to load (at best - image previous, at worst - only background. Any ideas how to change that?",{"id":117,"by":118,"text":119},"30682962","uniqueuid","This is an awesome tool, as all of yours! Just to mention: The example of french news websites shows that popovers and consent forms are a serious problem for this kind of…",[121,128,135],{"value":122,"source":79,"evidence":123},"Visual regression testing support",[124],{"id":125,"by":126,"text":127},"30679758","lloydatkinson","This but automated visual regression testing (eg testing screenshots are the same) would be amazing. Anyone got any ideas how this might work with shot-scraper?",{"value":129,"source":79,"evidence":130},"Only commit screenshots when significantly changed",[131],{"id":132,"by":133,"text":134},"30677123","beardicus","I saw you were poking with some image diff tools recently as well, and I'm sure you've thought about this already but I'd just like to explicitly state it: it'd be great if you…",{"value":136,"source":79,"evidence":137},"Favicon extraction",[138],{"id":139,"by":140,"text":141},"30676542","marginalia_nu","That's fantastic! I'll definitely investigate using this. I implemented my own MacGyver version of this basic functionality off selenium to grab screenshots for…",[143,149],{"value":144,"source":79,"evidence":145},"What limits prevent abuse as a web archive?",[146],{"id":147,"by":94,"text":148},"30686531","What sort of limits are in place to keep people from setting up their own web archive with this?",{"value":150,"source":79,"evidence":151},"How to handle slow-loading pages?",[152],{"id":113,"by":114,"text":115},[154,158,165,172],{"value":155,"source":79,"evidence":156},"Needle (Selenium visual regression)",[157],{"id":82,"by":11,"text":83},{"value":159,"source":79,"evidence":160},"pixelmatch and puppeteer",[161],{"id":162,"by":163,"text":164},"30680106","trevinhofmann","I used pixelmatch + puppeteer at a previous job to automate this. The service would screenshot each page on the master branch and the PR branch, then comment on the PR with a…",{"value":166,"source":79,"evidence":167},"Cypress visual regression",[168],{"id":169,"by":170,"text":171},"30681242","westurner","Awesome Visual Regression Testing > lists quite a few tools and online services: https:\u002F\u002Fgithub.com\u002Fmojoaxel\u002Fawesome-regression-testing#tools-and-frameworks \"visual-regression\":…",{"value":173,"source":79,"evidence":174},"thumbnail.ws",[175],{"id":176,"by":177,"text":178},"30683747","adriangrigore","I use https:\u002F\u002Fthumbnail.ws\u002F via curl.",{"comments":180,"answered":181,"key":182},12,6,[183,184,187],{"id":75,"text":76},{"id":185,"text":186},"30676574","No, I've not thought about favicons. That's a really interesting challenge. I wonder if there's a way to detect favicons just using JavaScript that runs against a page? Not sure…",{"id":188,"text":189},"30676599","Would be kind of fun to write JavaScript that runs against the page that first looks for the meta tags, then tries to fetch(\"\u002Ffavicon.ico\") and returns either the URL or a base64…",{"commenters":191,"max_depth":32,"maker_replies":180,"first_comment_mins":192,"lifespan_hours":193},20,46,77,[195,197,198,200,202,204,208,211,213,216,221,223,226,228,230,231,233,235,236,238,239,240,243,245,247,250,252,253,256,259,263,265,268,270,273,275],[75,34,34,196,11,35],"project\u002Fpositive\u002Fidea",[139,34,32,196,140,34],[185,34,34,199,11,35],"project\u002Fneutral\u002Fnone",[188,34,34,201,11,35],"project\u002Fneutral\u002Fidea",[203,34,39,199,140,34],"30676653",[205,34,34,206,207,34],"30681715","other\u002Fneutral\u002Fnone","Freeboots",[209,35,34,210,11,35],"30676014","project\u002Fpositive\u002Fexecution",[117,35,39,212,118,34],"project\u002Fneutral\u002Fexecution",[214,35,34,215,11,35],"30684528","project\u002Fneutral\u002Fdocs",[217,218,39,219,220,34],"30681433",2,"project\u002Fneutral\u002Fdesign","Brajeshwar",[222,218,34,219,11,35],"30681605",[224,39,34,206,225,34],"30679328","moharoune",[105,227,39,201,106,34],4,[60,227,34,229,11,35],"project\u002Fneutral\u002Fpricing",[132,227,39,201,133,34],[232,227,34,212,11,35],"30677166",[93,32,32,234,94,34],"project\u002Fpositive\u002Fpricing",[97,181,32,196,98,34],[125,237,39,201,126,34],7,[82,237,34,201,11,35],[162,237,34,206,163,34],[241,237,34,242,11,35],"30680606","maker\u002Fpositive\u002Fidea",[244,237,34,206,126,34],"30682809",[246,237,34,206,163,34],"30683288",[248,237,34,206,249,34],"30680859","dewski",[251,237,34,206,11,35],"30681373",[169,237,34,206,170,34],[254,33,34,206,255,34],"30676855","endisneigh",[257,33,34,206,258,34],"30677185","crickcreek",[260,261,32,196,262,34],"30716408",9,"ediardo",[113,18,35,264,114,34],"project\u002Fcritical\u002Fexecution",[266,18,34,267,11,35],"30684536","maker\u002Fneutral\u002Fdocs",[269,18,32,210,114,34],"30685040",[271,18,39,215,272,34],"30684929","max23_",[176,274,34,206,177,34],11,[147,180,39,276,94,34],"project\u002Fneutral\u002Fprivacy",[],[279,283,288,293,298,302],{"id":280,"name":281,"points":282,"category":20},"30668137","HN Avatars",701,{"id":284,"name":285,"points":286,"category":287},"30656961","arielroffe.quest",425,"creative",{"id":289,"name":290,"points":291,"category":292},"30688691","Devzat",391,"consumer_social",{"id":294,"name":295,"points":296,"category":297},"30676595","MarkdownSite",217,"infrastructure",{"id":299,"name":300,"points":301,"category":20},"30653916","Hubfs",160,{"id":303,"name":304,"points":305,"category":306},"30639160","Shadow Calculator",120,"web_saas",false,"v2",{"id":310,"name":5,"line":6,"ai":307,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":311,"top":18,"kw":312,"maker":11,"domain":10,"tier":19,"attempts":34,"strip":313,"url":9,"mix":314,"praise":90,"objection":102,"sub":21},30675357,0.21,[24,25,26,27,28],"050030030300303055300000000005105303",[32,33,34,35],1794036618055]