[{"data":1,"prerenderedAt":266},["ShallowReactive",2],{"project-31799147":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":18,"aspects":34,"strip":38,"post_text":40,"what":41,"said":62,"maker_in_thread":143,"shape":148,"grid":153,"attempts":227,"same_week":228,"maker_shows":213,"maker_hidden":258,"reader_version":259,"card":260},"31799147","WarcDB","Web crawl data as SQLite databases","title","Show HN: WarcDB: Web crawl data as SQLite databases","https:\u002F\u002Fgithub.com\u002FFlorents-Tselai\u002FWarcDB","github.com","fforflo",1655645161,2022,171,30,352,2570,14,"notable","data","web archive database","cli",[24,25,26,27],"warc","sqlite","web archive","web crawl","curious",0.076,{"positive":31,"neutral":32,"mixed":33,"critical":33},3,9,1,{"idea":35,"execution":36},{"positive":31,"critical":33},{"positive":37,"critical":37},0,[37,31,39,37,37,37,31,37,31,31,37,39,31,37,37,37,37,37,39,31,37,31,37,37,37,31,33,37,31,31],5,null,{"problem":42,"audience":40,"pricing":40,"motivation":49,"time_to_build":40,"stack":56,"openness":61,"stage":61,"team_size":61},{"value":43,"source":44,"evidence":45},"Web crawl WARC archives are hard to query in smaller pieces","maker",[46],{"id":47,"by":11,"text":48},"31800319","I built this as a small utility within a larger project I'm working on these days. (contact me if you're curious or want to support). The WARC format is extremely simple and yet…",{"value":50,"source":44,"evidence":51},"Make WARC archives SQL-queryable in smaller pieces; maker believes SQLite may become a standard for such archives",[52,53],{"id":47,"by":11,"text":48},{"id":54,"by":11,"text":55},"31800974","> What is the advantage of moving around .sqlite-files, over just loading the (compressed) WARCs into sqlite databases when you need them? The .warc spec is ideal. I'm not saying…",[57],{"value":58,"source":44,"evidence":59},"SQLite",[60],{"id":47,"by":11,"text":48},"unknown",{"praise":63,"objections":82,"requests":98,"questions":110,"alternatives":115},[64,72],{"value":65,"source":66,"evidence":67},"SQLite offers strong well-documented guarantees and transactions for archive integrity","commenter",[68],{"id":69,"by":70,"text":71},"31800847","uniqueuid","Anecdotal evidence, but I produced a medium-size crawl in the past (~20TB compressed). I used distributed resources with off-the-shelf libraries (i.e. warcprox etc.) and managed…",{"value":73,"source":66,"evidence":74},"Idea of SQLite as a portable archive format is appealing",[75,78],{"id":76,"by":70,"text":77},"31800802","Oh hi, thanks for building this! I haven't had the chance to play with it, but my hunch is that sqlite for warc can fill a great niche and would be much more portable (and…",{"id":79,"by":80,"text":81},"31801405","mynameismon","> What is the advantage of moving around .sqlite-files, over just loading the (compressed) WARCs into sqlite databases when you need them? I suppose there could be made a case…",[83,90],{"value":84,"source":66,"evidence":85},"Loading WARCs into SQLite on demand would suffice; moving .sqlite files adds little",[86],{"id":87,"by":88,"text":89},"31800703","marginalia_nu","> PS: I do believe that at some point .sqlite will become the defacto standard for such initiatives. Sure, it's not text... but it's pretty close. What is the advantage of moving…",{"value":91,"source":66,"evidence":92},"WARC is already portable and standard; binary content does not need a database",[93,94],{"id":87,"by":88,"text":89},{"id":95,"by":96,"text":97},"31827646","ikreymer","A bit late to this thread, but I think WARC is a reasonable format for raw HTTP traffic. We should definitely have better tools to ensure WARC files produced are valid, and…",[99],{"value":100,"source":66,"evidence":101},"Suggestions for virtual table or Parquet\u002FDuckDB alternatives",[102,106],{"id":103,"by":104,"text":105},"31801491","nlohmann","Have you every played with SQLite virtual tables (https:\u002F\u002Fsqlite.org\u002Fvtab.html) - they could allow to provide an SQLite interface while keeping the same structure on disk. Though…",{"id":107,"by":108,"text":109},"31805743","m_ke","DuckDB would probably be a way better option and works amazingly well on top of parquet (https:\u002F\u002Fduckdb.org\u002Fdocs\u002Fdata\u002Fparquet)",[111],{"value":112,"source":66,"evidence":113},"What is the advantage of distributing SQLite files over loading compressed WARCs?",[114],{"id":87,"by":88,"text":89},[116,120,126,132,136],{"value":117,"source":66,"evidence":118},"DuckDB over Parquet",[119],{"id":107,"by":108,"text":109},{"value":121,"source":66,"evidence":122},"CommonCrawl Parquet files with sqlite-parquet-vtable",[123],{"id":124,"by":11,"text":125},"31801705","Good point. Actually CommonCrawl provides Parquet files for their archives too. And there's this vtable for Parquet extension. https:\u002F\u002Fgithub.com\u002Fcldellow\u002Fsqlite-parquet-vtable…",{"value":127,"source":66,"evidence":128},"ZIM format for offline website copies",[129],{"id":130,"by":88,"text":131},"31801701","End-user facing applications usually don't consume website crawls, do they? That's impractical for many reasons, the sheer size alone being perhaps the biggest obstacle. If you…",{"value":133,"source":66,"evidence":134},"WACZ format",[135],{"id":95,"by":96,"text":97},{"value":137,"source":66,"evidence":138},"WARCTools collection",[139],{"id":140,"by":141,"text":142},"31812078","TedDoesntTalk","Did you add this to https:\u002F\u002Fgithub.com\u002Fdhamaniasad\u002FWARCTools ?",{"comments":39,"answered":33,"key":144},[145,146,147],{"id":47,"text":48},{"id":54,"text":55},{"id":124,"text":125},{"commenters":149,"max_depth":150,"maker_replies":39,"first_comment_mins":151,"lifespan_hours":152},18,6,115,53,[154,156,158,159,163,165,168,170,172,174,177,179,180,182,183,185,188,190,192,194,197,200,203,205,208,211,215,218,221,224],[47,37,37,155,11,33],"project\u002Fpositive\u002Fidea",[87,37,31,157,88,37],"project\u002Fneutral\u002Fidea",[69,37,39,155,70,37],[160,37,37,161,162,37],"31803018","other\u002Fneutral\u002Fnone","rengler33",[164,37,37,161,70,37],"31803434",[166,37,37,161,167,37],"31803748","smcnally",[95,37,31,169,96,37],"project\u002Fmixed\u002Fidea",[54,37,37,171,11,33],"maker\u002Fneutral\u002Fidea",[173,37,31,157,88,37],"31801465",[175,37,31,157,176,37],"31801640","traverseda",[130,37,37,178,88,37],"commenter\u002Fneutral\u002Fidea",[79,37,39,155,80,37],[103,37,31,181,104,37],"project\u002Fneutral\u002Fexecution",[124,37,37,181,11,33],[107,37,37,184,108,37],"commenter\u002Fneutral\u002Fexecution",[186,37,37,187,11,33],"31802964","project\u002Fneutral\u002Fdesign",[189,33,37,161,70,37],"31800208",[191,33,37,157,11,33],"31800368",[76,33,39,193,70,37],"maker\u002Fpositive\u002Fidea",[195,33,31,181,196,37],"31805532","pstuart",[198,33,37,161,199,37],"31800963","ma2rten",[201,33,31,157,202,37],"31801371","lijogdfljk",[204,33,37,178,70,37],"31801470",[206,33,37,207,202,37],"31801588","commenter\u002Fneutral\u002Fnone",[209,33,37,161,210,37],"31801943","simonw",[212,213,31,157,214,37],"31803358",2,"tepitoperrito",[216,213,33,217,70,37],"31803425","project\u002Fcritical\u002Fidea",[219,213,37,161,220,37],"31805574","nikisweeting",[222,31,31,157,223,37],"31803173","jbverschoor",[140,225,31,226,141,37],4,"project\u002Fneutral\u002Fnone",[],[229,234,239,244,249,254],{"id":230,"name":231,"points":232,"category":233},"31785199","SimCB",815,"games",{"id":235,"name":236,"points":237,"category":238},"31824877","Avo",306,"dev_tools",{"id":240,"name":241,"points":242,"category":243},"31767830","Recut",268,"creative",{"id":245,"name":246,"points":247,"category":248},"31789489","Nimwave",195,"languages",{"id":250,"name":251,"points":252,"category":253},"31833415","Flat.social",166,"productivity",{"id":255,"name":256,"points":257,"category":238},"31841215","Crocodile",144,false,"v2",{"id":261,"name":5,"line":6,"ai":258,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":262,"top":18,"kw":263,"maker":11,"domain":10,"tier":19,"attempts":37,"strip":264,"url":9,"mix":265,"praise":65,"objection":84,"sub":21},31799147,0.08,[24,25,26,27],"035000303305300000530300031033",[31,32,33,33],1793991047375]