[{"data":1,"prerenderedAt":390},["ShallowReactive",2],{"project-40632533":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":35,"aspects":36,"strip":42,"post_text":43,"what":44,"said":86,"maker_in_thread":175,"shape":186,"grid":189,"attempts":351,"same_week":352,"maker_shows":33,"maker_hidden":382,"reader_version":383,"card":384},"40632533","mpa-archive","Crawl a modern website to a zip, serve the website from the zip","title","Show HN: Crawl a modern website to a zip, serve the website from the zip","https:\u002F\u002Fgithub.com\u002Fpotahtml\u002Fmpa-archive","github.com","unlog",1718020020,2024,223,66,343,3515,10,"notable","dev_tools","static site generator","cli",[24,25,26,27,28],"website crawler","static site archive","headless chrome","spa archive","zip archive","mixed",0.203,{"positive":18,"neutral":32,"mixed":33,"critical":34},14,1,5,30,{"idea":37,"execution":39},{"positive":38,"critical":33},6,{"positive":40,"critical":41},0,3,[34,40,34,40,40,40,40,33,33,33,34,41,41,41,40,33,41,40,34,34,41,34,40,41,34,41,41,40,40,41,34,40,40,33,40,40,34,40,41,34],null,{"problem":45,"audience":58,"pricing":43,"motivation":66,"time_to_build":43,"stack":70,"openness":84,"stage":85,"team_size":85},{"value":46,"source":47,"evidence":48},"Client-rendered JavaScript sites are hard to index or archive as static files without server-side rendering.","maker",[49,52,55],{"id":50,"by":11,"text":51},"40632569","I'm a big fan of modern JavaScript frameworks, but I don't fancy SSR, so have been experimenting with crawling myself for uploading to hosts without having to do SSR. This is the…",{"id":53,"by":11,"text":54},"40639866","> Why would someone crawl their own website? My main use case is that the docs site https:\u002F\u002Fpota.quack.uy\u002F , Google cannot index it properly. On here…",{"id":56,"by":11,"text":57},"40641408","Modern websites execute JavaScript that render DOM nodes that are displayed on the browser. For example if you look at this site on the browser https:\u002F\u002Fpota.quack.uy\u002F and do…",{"value":59,"source":60,"evidence":61},"Developers who want static copies of JavaScript-rendered sites or offline backups.","inferred",[62,63],{"id":53,"by":11,"text":54},{"id":64,"by":11,"text":65},"40635210","That the page HTML is indexable by search engines without having to render in the server. Such unzipping to a directory served by nginx. You may also use it for archiving…",{"value":67,"source":47,"evidence":68},"Maker wanted static output for a JS docs site that Google indexed poorly, without paying for SSR.",[69],{"id":53,"by":11,"text":54},[71,77],{"value":72,"source":47,"evidence":73},"Headless Chrome (Puppeteer)",[74],{"id":75,"by":11,"text":76},"40639023","Big fan of HTTrack! reminds me of the old days and makes me sad of the current state of the web. I am not sure if HTTTrack progressed from fetching resources, long time since I…",{"value":78,"source":60,"evidence":79},"Node.js",[80],{"id":81,"by":82,"text":83},"40666398","mikeqq2024","trying to use this for mirroring a document site. disappointed at 1. it running quite slow, 2. it kept outputing error messages like \"ProtocolError: Protocol error…","open_source","unknown",{"praise":87,"objections":100,"requests":105,"questions":123,"alternatives":142},[88],{"value":89,"source":90,"evidence":91},"Saving to a simple zip is elegant and useful for archiving.","commenter",[92,96],{"id":93,"by":94,"text":95},"40645504","CGamesPlay","I like the approach here! Saving to a simple zip file is elegant. I worked on a similar idea years ago [0], but made the mistake of building it as a frontend. In retrospect, I…",{"id":97,"by":98,"text":99},"40635655","brudgers","One possible advantage I see is it creates a 1:1 correspondence between a website and a file. If what I care about is the website (and that's usually going to be the case), then…",[101],{"value":102,"source":90,"evidence":103},"Crawler ran slowly and threw protocol errors for one user.",[104],{"id":81,"by":82,"text":83},[106,113,117],{"value":107,"source":90,"evidence":108},"Add a license file.",[109],{"id":110,"by":111,"text":112},"40636343","yodon","+1 can you add a license file with an MIT or BSD or whatever your preference is? (It's very cool. I'd love to help with this project, I'm guessing others would as well)",{"value":114,"source":90,"evidence":115},"Better support for single-page apps.",[116],{"id":93,"by":94,"text":95},{"value":118,"source":90,"evidence":119},"Make the checkpoint interval a command-line option.",[120],{"id":121,"by":82,"text":122},"40645190","nice, better make it as a commandline option with default value. 250 is too many for large files and slow connection.",[124,131],{"value":125,"source":90,"evidence":126},"How does it handle single-page apps?",[127],{"id":128,"by":129,"text":130},"40644001","szhabolcs","How does it work with single page apps? If the data is loaded from the server, does it save the page contents as full, or just the source of the page?",{"value":132,"source":90,"evidence":133},"How does it differ from HTTrack or a WARC file?",[134,138],{"id":135,"by":136,"text":137},"40638402","tamimio","How is it different from HTTrack? And what about the media extension, which one is supported and which one isn’t? Sometimes when I download some sites with HTTrack, some files…",{"id":139,"by":140,"text":141},"40636252","jayemar","Is the output similar to a web archive file (warc)?",[143,147,158,168],{"value":144,"source":90,"evidence":145},"HTTrack",[146],{"id":135,"by":136,"text":137},{"value":148,"source":90,"evidence":149},"SingleFile",[150,154],{"id":151,"by":152,"text":153},"40643350","ajvs","SingleFile extension is the modern equivalent these days.",{"id":155,"by":156,"text":157},"40645275","venusenvy47","I use SingleFile on Firefox quite often for this purpose. https:\u002F\u002Faddons.mozilla.org\u002Fen-US\u002Ffirefox\u002Faddon\u002Fsingle-file\u002F",{"value":159,"source":90,"evidence":160},"wget",[161,165],{"id":162,"by":163,"text":164},"40643806","ProtoAES256","Wow! I never knew things like this existed! I always used wget (full below) but nowadays seemingly all sites are behind cloudflare so I need to pass a cookie too. Glad to see…",{"id":166,"by":82,"text":167},"40677220","not reproduce it, but 'wget -m --page-requisites --convert-links \u003Curl>' did a good job for me. never mind",{"value":169,"source":90,"evidence":170},"MHTML",[171],{"id":172,"by":173,"text":174},"40637652","jll29","Microsoft Interne Explorer (no, I'm not using it personally) had a file format called *.mht that could save a HTML page together with all the files referenced from it like inline…",{"comments":176,"answered":177,"key":178},16,9,[179,182,185],{"id":180,"text":181},"40639679","Yes! You know, I was considering this the previous couple of days, was looking around on how to construct a `mhtml` file for serving all the files at the same time. Unrelated to…",{"id":183,"text":184},"40667367","If the URL is public you may post it here or in a GitHub issue, so I can take a look to what's wrong with it.",{"id":50,"text":51},{"commenters":187,"max_depth":38,"maker_replies":176,"first_comment_mins":34,"lifespan_hours":188},40,88,[190,194,198,202,204,207,211,213,216,217,220,222,224,226,227,229,232,234,235,237,240,242,246,248,250,253,255,258,260,261,263,265,268,272,276,279,282,284,286,288,291,292,294,296,298,300,303,307,311,312,313,314,316,319,321,325,328,331,333,335,337,340,341,344,345,348],[191,40,34,192,193,40],"40638029","project\u002Fpositive\u002Fnone","kitd",[195,40,40,196,197,40],"40641129","other\u002Fpositive\u002Fnone","ilrwbwrkhv",[199,40,40,200,201,40],"40641931","other\u002Fneutral\u002Fnone","djbusby",[172,33,34,203,173,40],"project\u002Fpositive\u002Fidea",[205,33,40,200,206,40],"40638809","jdougan",[208,33,40,209,210,40],"40639427","commenter\u002Fneutral\u002Fnone","rrr_oh_man",[212,33,40,200,206,40],"40641006",[214,33,40,200,215,40],"40640967","felipefar",[155,33,40,200,156,40],[218,33,40,200,219,40],"40645471","lukan",[180,33,40,221,11,33],"project\u002Fneutral\u002Fnone",[81,33,33,223,82,40],"project\u002Fcritical\u002Fexecution",[183,33,40,225,11,33],"maker\u002Fneutral\u002Fnone",[166,33,33,223,82,40],[151,33,33,228,152,40],"project\u002Fcritical\u002Fnone",[230,33,40,200,231,40],"40646654","marban",[162,233,34,192,163,40],2,[50,41,40,221,11,33],[236,41,41,221,82,40],"40644326",[238,41,41,221,239,40],"40644721","eXpl0it3r",[121,41,41,241,82,40],"project\u002Fneutral\u002Fexecution",[243,244,41,221,245,40],"40638597",4,"renegat0x0",[247,244,40,225,11,33],"40638964",[249,244,40,200,245,40],"40643480",[251,244,33,223,252,40],"40652601","PenguinCoder",[254,244,40,209,11,33],"40652716",[256,244,40,200,257,40],"40645068","glandium",[135,34,41,259,136,40],"project\u002Fneutral\u002Fidea",[75,34,40,221,11,33],[262,34,34,192,136,40],"40639402",[264,34,40,225,11,33],"40641453",[266,38,40,200,267,40],"40653291","Per_Bothner",[269,270,34,203,271,40],"40648605",7,"sedawk",[273,274,41,259,275,40],"40639800",8,"nox101",[277,274,34,203,278,40],"40641863","jraph",[280,177,41,221,281,40],"40635752","earleybird",[283,177,40,225,11,33],"40636271",[285,177,41,225,111,40],"40636356",[287,177,40,225,11,33],"40636497",[289,177,41,225,290,40],"40636896","meiraleal",[110,177,34,192,111,40],[93,18,41,293,94,40],"project\u002Fmixed\u002Fidea",[295,18,40,225,11,33],"40646346",[139,297,41,259,140,40],11,[299,297,40,221,11,33],"40636333",[301,297,40,200,302,40],"40639626","toomuchtodo",[304,305,40,200,306,40],"40645826",12,"billpg",[308,309,41,259,310,40],"40634593",13,"ryanwaldorf",[64,309,40,259,11,33],[97,309,34,203,98,40],[128,32,41,221,129,40],[315,32,40,221,11,33],"40645163",[317,32,40,200,318,40],"40646616","zb3",[320,32,40,209,11,33],"40656542",[322,323,41,259,324,40],"40639035",15,"ivolimmen",[326,176,33,327,290,40],"40638598","project\u002Fcritical\u002Fidea",[329,176,40,209,330,40],"40638744","kej",[332,176,40,209,290,40],"40639371",[334,176,40,209,330,40],"40639498",[336,176,40,209,290,40],"40639603",[338,176,34,203,339,40],"40639886","p4bl0",[53,176,40,203,11,33],[342,176,41,225,343,40],"40641176","ibeckermayer",[56,176,40,259,11,33],[346,176,34,203,347,40],"40640932","jbaber",[349,176,40,200,350,40],"40641911","tomByrer",[],[353,358,362,367,372,377],{"id":354,"name":355,"points":356,"category":357},"40632773","at.hn",459,"consumer_social",{"id":359,"name":360,"points":361,"category":20},"40646741","Revideo",298,{"id":363,"name":364,"points":365,"category":366},"40635397","Probabilistic Tic-Tac-Toe",293,"games",{"id":368,"name":369,"points":370,"category":371},"40645743","Unforget",274,"productivity",{"id":373,"name":374,"points":375,"category":376},"40607889","sol-mate-eink",229,"hardware",{"id":378,"name":379,"points":380,"category":381},"40623883","Inspectus",197,"ai",false,"v2",{"id":385,"name":5,"line":6,"ai":382,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":386,"top":18,"kw":387,"maker":11,"domain":10,"tier":19,"attempts":40,"strip":388,"url":9,"mix":389,"praise":89,"objection":102,"sub":21},40632533,0.2,[24,25,26,27,28],"5050000111533301305535035330035001005035",[18,32,33,34],1793991093459]