[{"data":1,"prerenderedAt":794},["ShallowReactive",2],{"project-28012829":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":35,"aspects":36,"strip":48,"post_text":50,"what":51,"said":95,"maker_in_thread":216,"shape":224,"grid":228,"attempts":755,"same_week":756,"maker_shows":18,"maker_hidden":786,"reader_version":787,"card":788},"28012829","Static.wiki","read-only Wikipedia using a 43GB SQLite file","title","Show HN: Static.wiki – read-only Wikipedia using a 43GB SQLite file","http:\u002F\u002Fstatic.wiki\u002F","static.wiki","segfall",1627681850,2021,796,189,17,2533,1,"legend","learning_science","offline wikipedia reader","web_app",[24,25,26,27,28],"wikipedia offline","sqlite","http range requests","static hosting","wikipedia mirror","curious",-0.042,{"positive":32,"neutral":33,"mixed":18,"critical":34},14,44,16,75,{"performance":37,"execution":39,"docs":42,"idea":44,"design":46,"pricing":47},{"positive":38,"critical":38},3,{"positive":40,"critical":41},2,10,{"positive":43,"critical":18},0,{"positive":45,"critical":40},5,{"positive":18,"critical":43},{"positive":43,"critical":43},[45,18,43,43,43,43,43,43,45,38,45,40,38,45,18,38,38,45,43,18,38,38,49,38,40,45,43,45,18,18,45,38,38,40,18,43,43,18,40,40],4,null,{"problem":52,"audience":50,"pricing":50,"motivation":63,"time_to_build":50,"stack":71,"openness":92,"stage":93,"team_size":94},{"value":53,"source":54,"evidence":55},"Browsing Wikipedia without relying on live servers, using a static SQLite copy","inferred",[56,60],{"id":57,"by":58,"text":59},"28013189","arianon","Shocked to find out that this is very much a static website, it's \"merely\" downloading a small portion of the 43GB SQLite file in question with HTTP Range requests, and then it…",{"id":61,"by":11,"text":62},"28013495","OP here. Happy to take questions, with the caveat that my work here is mostly glue. Some background: https:\u002F\u002Fgithub.com\u002Fsegfall\u002Fstatic-wiki The datasets I generated:…",{"value":64,"source":65,"evidence":66},"Side project; glue around existing work by phiresky","maker",[67,68],{"id":61,"by":11,"text":62},{"id":69,"by":11,"text":70},"28013610","Thank you, Phiresky. My little side project only exists because of your work.",[72,78,86],{"value":73,"source":65,"evidence":74},"SQLite",[75,77],{"id":76},"post",{"id":61,"by":11,"text":62},{"value":79,"source":80,"evidence":81},"WASM SQLite (sql.js)","commenter",[82,83],{"id":57,"by":58,"text":59},{"id":84,"by":11,"text":85},"28015881","sql.js is in memory, but the database file is only partially loaded from the static file host.",{"value":87,"source":65,"evidence":88},"Svelte",[89],{"id":90,"by":11,"text":91},"28023868","Thank you. Unfortunately, you'd have to write that stylesheet from scratch. For this demo, I just wrote them in the Svelte components.","open_source","launched","solo",{"praise":96,"objections":116,"requests":153,"questions":180,"alternatives":195},[97,105],{"value":98,"source":80,"evidence":99},"Clever use of HTTP range requests on a static host",[100,101],{"id":57,"by":58,"text":59},{"id":102,"by":103,"text":104},"28013186","nynx","This is fantastic. I knew that we’d see things like this once that hosting sqlite databases project was posted.",{"value":106,"source":80,"evidence":107},"Useful for offline reading and travel",[108,112],{"id":109,"by":110,"text":111},"28014629","NelsonMinar","I've been using static copies of Wikipedia for years; they're great to have when traveling. I mostly use the Kiwix app. Their ZIM format database for all English is 83 GB; 43 GB…",{"id":113,"by":114,"text":115},"28013167","voldemort1968","There's also https:\u002F\u002Fwww.kiwix.org\u002Fen\u002F which is great to take on long offline flights, browsing through wikipedia.",[117,127,138],{"value":118,"source":80,"evidence":119},"Hosting cost on S3 is risky at this size",[120,124],{"id":121,"by":122,"text":123},"28013266","simonw","I'm nervous on your behalf for your S3 bill, have you done napkin maths for how much this will cost to host, especially given the relative inefficiency of the autocomplete…",{"id":125,"by":122,"text":126},"28013384","This is a risk any time you host anything on the internet on any system that charges you for storage, bandwidth or CPU usage. It's fine to take that risk on as long as you're…",{"value":128,"source":80,"evidence":129},"Downloading or holding 43GB is a barrier",[130,134],{"id":131,"by":132,"text":133},"28015645","londons_explore","You still have to download it all... Which is a barrier for most...",{"id":135,"by":136,"text":137},"28014470","walrus01","How does it do for performance if you throw the whole 43GB into RAM? Plenty of very affordable workstation systems out there today gently used with 128GB in them.",{"value":139,"source":80,"evidence":140},"Search and autocomplete have bugs and are slow",[141,145,149],{"id":142,"by":143,"text":144},"28013206","max1cc","The search seems to be broken right now, but it looks like you can access whatever page you want via URL (e.g. http:\u002F\u002Fstatic.wiki\u002Fen\u002FComputer_Science)",{"id":146,"by":147,"text":148},"28013359","dleslie","Two immediate issues: 1. Doesn't work if Privacy Badger is active. 2. Nim (Programming Language) is not indexed.",{"id":150,"by":151,"text":152},"28014356","yukinon","There's a bug when the target article is titled the same as the disambiguation page. Search for \"Earthbound\", and try clicking on the article for the video game. It'll bring you…",[154,162,169],{"value":155,"source":80,"evidence":156},"Fix search and clickable links for disambiguation pages",[157,161],{"id":158,"by":159,"text":160},"28013376","dom96","Really cool! I think I see a bug: searching “Nim” shows “Nim (programming language)” but it’s not clickable.",{"id":150,"by":151,"text":152},{"value":163,"source":80,"evidence":164},"Compress article text and enlarge SQLite page size for latency",[165],{"id":166,"by":167,"text":168},"28016098","Scaevolus","You should compress the individual article texts with zlib. Right now, looking up an article text for a decent size article (\"Virus\") takes 10 round-trips-- 5 appear to be btree…",{"value":170,"source":80,"evidence":171},"Offer IPFS or torrent mirrors",[172,176],{"id":173,"by":174,"text":175},"28016572","hutrdvnj","Since it doesn't require any backend, a IPFS mirror would be great.",{"id":177,"by":178,"text":179},"28029724","xchip","Could article pages be stored as a torrent?",[181,188],{"value":182,"source":80,"evidence":183},"How are the XML dumps generated",[184],{"id":185,"by":186,"text":187},"28018987","neocodesoftware","how are the xml dumps generated?",{"value":189,"source":80,"evidence":190},"How does it compare to Kiwix",[191],{"id":192,"by":193,"text":194},"28016244","andrepd","How does this compare to https:\u002F\u002Fwww.kiwix.org\u002Fen\u002F",[196,202,209],{"value":197,"source":80,"evidence":198},"Kiwix",[199,200,201],{"id":109,"by":110,"text":111},{"id":113,"by":114,"text":115},{"id":192,"by":193,"text":194},{"value":203,"source":80,"evidence":204},"Full-text search with tantivy",[205],{"id":206,"by":207,"text":208},"28013514","phiresky","Author of the referenced blog (and library) here. This is great! The full text search engine in SQLite is sadly not really good for this - one reason is that it uses standard…",{"value":210,"source":80,"evidence":211},"sqlite_zstd_vfs",[212],{"id":213,"by":214,"text":215},"28016738","dnamlin","I plowed the 40 GiB database file into sqlite_zstd_vfs [1] which reduced it 75% to 10 GiB. This plug-in also supports HTTP access [2] in the spirit of phiresky's, however, I…",{"comments":217,"answered":38,"key":218},8,[219,220,221],{"id":69,"text":70},{"id":61,"text":62},{"id":222,"text":223},"28013746","For added fun, you can try switching between different databases: http:\u002F\u002Fstatic.wiki\u002Fen\u002FUnited_States http:\u002F\u002Fstatic.wiki\u002Fzh\u002F美国 http:\u002F\u002Fstatic.wiki\u002Ffr\u002FUnited_States…",{"commenters":225,"max_depth":32,"maker_replies":217,"first_comment_mins":226,"lifespan_hours":227},136,32,213,[229,231,233,236,240,243,246,249,251,253,257,260,263,266,269,273,277,279,282,284,287,290,292,295,298,300,302,304,307,310,312,314,316,318,320,322,324,327,330,333,335,338,340,343,346,349,352,354,358,360,362,365,368,371,373,376,379,381,384,387,389,392,395,397,400,403,406,409,412,416,420,423,426,427,430,432,434,435,438,439,442,446,448,450,453,455,457,460,462,465,468,470,473,475,478,481,484,486,489,492,494,497,500,502,506,508,511,514,516,519,521,524,527,530,534,537,540,543,545,548,549,552,555,557,560,563,565,568,571,574,578,579,583,586,589,592,595,599,602,605,607,610,613,616,618,619,623,626,630,632,635,639,641,643,647,651,655,659,661,663,666,669,672,675,678,680,684,688,691,694,696,699,701,703,707,709,711,714,718,721,723,728,730,733,736,740,744,748,752],[206,43,45,230,207,43],"project\u002Fpositive\u002Fperformance",[69,43,43,232,11,18],"maker\u002Fpositive\u002Fnone",[234,43,45,232,235,43],"28014051","ignoramous",[237,43,43,238,239,43],"28013975","other\u002Fneutral\u002Fnone","tored",[241,43,43,238,242,43],"28014168","OJFord",[244,43,43,238,245,43],"28015684","psanford",[247,43,43,248,207,43],"28015985","other\u002Fneutral\u002Fperformance",[135,43,38,250,136,43],"project\u002Fneutral\u002Fperformance",[131,43,18,252,132,43],"project\u002Fcritical\u002Fperformance",[254,43,43,255,256,43],"28015809","commenter\u002Fneutral\u002Fnone","tjoff",[258,43,43,255,259,43],"28015923","tw04",[261,43,43,255,262,43],"28016089","unknownOrigin",[264,43,43,255,265,43],"28016361","exdsq",[267,43,43,255,268,43],"28015958","shp0ngle",[270,43,43,271,272,43],"28015972","commenter\u002Fpositive\u002Fidea","unixhero",[274,43,43,275,276,43],"28016109","commenter\u002Fcritical\u002Fperformance","IshKebab",[278,43,43,255,272,43],"28016272",[280,43,43,255,281,43],"28016435","linkpuff",[283,43,43,255,272,43],"28016457",[285,43,43,255,286,43],"28016539","jbverschoor",[288,43,43,255,289,43],"28016786","Quekid5",[291,43,43,255,286,43],"28016940",[293,43,43,255,294,43],"28016971","andi999",[296,43,43,255,297,43],"28017315","Eremotherium",[299,43,43,255,294,43],"28020101",[301,43,43,255,272,43],"28020310",[303,43,43,255,276,43],"28017027",[305,43,43,255,306,43],"28017096","IgorPartola",[308,43,43,255,309,43],"28017187","gsich",[311,43,43,255,306,43],"28022248",[313,43,43,255,309,43],"28026720",[315,43,43,255,306,43],"28036791",[317,43,43,255,309,43],"28036912",[319,43,43,255,306,43],"28037601",[321,43,43,255,309,43],"28040955",[323,43,43,255,265,43],"28016352",[325,43,43,255,326,43],"28016515","azeirah",[328,43,43,255,329,43],"28016929","AussieWog93",[331,43,45,230,332,43],"28018815","jggonz",[334,43,43,255,136,43],"28020210",[336,43,43,255,337,43],"28016863","bawolff",[339,43,43,255,268,43],"28017233",[341,43,43,238,342,43],"28018112","sonograph",[344,43,43,238,345,43],"28020326","trollied",[347,43,38,250,348,43],"28015409","peterhunt",[350,43,43,238,351,43],"28015453","piyh",[353,43,38,250,348,43],"28015640",[355,43,38,356,357,43],"28015900","project\u002Fneutral\u002Fexecution","dheera",[185,43,38,359,186,43],"project\u002Fneutral\u002Fdocs",[57,18,45,361,58,43],"project\u002Fpositive\u002Fexecution",[363,18,45,364,122,43],"28013222","project\u002Fpositive\u002Fidea",[366,18,18,252,367,43],"28013285","JoeyBananas",[369,18,38,250,370,43],"28013302","CodesInChaos",[372,18,38,250,122,43],"28013304",[374,18,38,250,375,43],"28013334","floatingatoll",[377,18,38,250,378,43],"28013446","simlevesque",[380,18,38,356,207,43],"28013544",[382,18,38,250,383,43],"28017001","dmos62",[385,18,38,356,386,43],"28013575","sltkr",[388,18,38,250,375,43],"28013890",[390,18,38,356,391,43],"28016299","rzzzt",[393,18,43,255,394,43],"28013620","milankragujevic",[396,18,43,255,375,43],"28013882",[398,18,43,255,399,43],"28013493","jlg23",[401,18,45,361,402,43],"28013733","quickthrower2",[404,18,43,255,405,43],"28013603","chrisseaton",[407,18,38,250,408,43],"28014182","carlhjerpe",[410,18,43,238,411,43],"28014810","rkeene2",[413,18,43,414,415,43],"28013568","other\u002Fpositive\u002Fidea","scrame",[417,18,18,418,419,43],"28013744","project\u002Fcritical\u002Fexecution","MuffinFlavored",[421,18,38,250,422,43],"28013450","k__",[424,18,43,255,425,43],"28014064","wmf",[213,40,38,250,214,43],[428,40,38,356,429,43],"28017549","jart",[431,40,38,356,214,43],"28021862",[61,38,43,433,11,18],"maker\u002Fneutral\u002Fnone",[222,38,43,433,11,18],[436,38,38,356,437,43],"28014458","johnwheeler",[84,38,43,356,11,18],[440,38,38,356,441,43],"28016206","exikyut",[443,38,45,444,445,43],"28016091","project\u002Fpositive\u002Fdesign","ddtaylor",[90,38,43,447,11,18],"maker\u002Fneutral\u002Fdesign",[121,49,38,449,122,43],"project\u002Fneutral\u002Fpricing",[451,49,43,238,452,43],"28013347","sitkack",[454,49,43,255,122,43],"28013395",[456,49,43,238,452,43],"28014965",[458,49,43,459,159,43],"28013355","other\u002Fneutral\u002Fpricing",[125,49,43,461,122,43],"commenter\u002Fneutral\u002Fpricing",[463,49,43,461,464,43],"28014653","remram",[466,49,43,461,467,43],"28014861","RhodesianHunter",[469,49,43,461,464,43],"28014877",[471,49,43,461,472,43],"28013429","slunk",[474,49,18,252,467,43],"28014870",[476,49,38,356,477,43],"28013373","max_",[479,49,38,356,480,43],"28013386","Weryj",[482,49,38,356,483,43],"28016779","warpech",[485,49,43,255,425,43],"28013521",[487,49,43,255,488,43],"28013684","magila",[490,49,43,491,425,43],"28014098","commenter\u002Fneutral\u002Fperformance",[493,49,43,491,488,43],"28014328",[495,49,38,449,496,43],"28013894","dTal",[498,49,43,255,499,43],"28015719","rudian",[501,49,43,255,496,43],"28016334",[503,49,45,504,505,43],"28014352","project\u002Fpositive\u002Fnone","thrower123",[507,49,38,356,235,43],"28013374",[509,49,43,255,510,43],"28013981","Bayart",[512,49,43,255,513,43],"28014434","mkl",[515,49,38,250,480,43],"28013380",[517,49,43,461,518,43],"28013390","capableweb",[520,49,43,255,122,43],"28013406",[522,49,43,461,523,43],"28013462","mayli",[525,49,38,356,526,43],"28013710","Asdrubalini",[528,49,38,449,529,43],"28013825","llacb47",[531,49,38,532,533,43],"28013874","project\u002Fneutral\u002Fnone","infogulch",[535,49,38,449,536,43],"28014136","banana_giraffe",[538,49,43,539,11,18],"28015875","maker\u002Fneutral\u002Fpricing",[541,49,18,418,542,43],"28016517","jinseokim",[544,49,43,539,11,18],"28032299",[546,49,45,232,547,43],"28053920","anderspitman",[109,45,43,238,110,43],[550,45,43,255,551,43],"28014862","elcritch",[553,45,43,238,554,43],"28015439","mohiit",[113,556,43,238,114,43],6,[558,556,43,238,559,43],"28014126","rycomb",[561,556,43,238,562,43],"28015174","ce4",[564,556,43,238,441,43],"28016216",[566,556,43,238,567,43],"28016353","zelienople",[569,556,43,238,570,43],"28015952","lmm",[572,556,43,238,573,43],"28027266","41b696ef1113",[575,576,43,238,577,43],"28013150",7,"punnerud",[166,217,38,250,167,43],[580,581,45,364,582,43],"28013169",9,"mongol",[584,581,43,238,585,43],"28013236","r00fus",[587,581,43,238,588,43],"28013268","wyldfire",[590,581,43,238,591,43],"28013360","yreg",[593,581,43,238,594,43],"28013688","pronoiac",[596,581,18,597,598,43],"28013838","project\u002Fcritical\u002Fidea","phil294",[600,581,38,601,496,43],"28013901","project\u002Fneutral\u002Fidea",[603,581,43,255,604,43],"28015005","jtsiskin",[606,581,43,255,496,43],"28016320",[608,581,18,418,609,43],"28108999","ric2b",[611,581,38,532,612,43],"28014315","anigbrowl",[614,581,43,238,615,43],"28014331","phkahler",[617,581,43,238,441,43],"28016218",[102,41,45,364,103,43],[620,621,43,238,622,43],"28015557",11,"mwidell",[624,621,43,238,625,43],"28020179","spiritplumber",[627,628,43,238,629,43],"28015588",12,"harlanji",[150,631,18,418,151,43],13,[633,32,45,230,634,43],"28013370","twotwotwo",[636,637,43,238,638,43],"28013338",15,"suprbeast",[640,637,38,532,625,43],"28020173",[642,34,38,250,429,43],"28017671",[644,16,38,645,646,43],"28031421","project\u002Fmixed\u002Fexecution","dankwizard",[648,649,38,356,650,43],"28018695",18,"cafxx",[652,653,45,364,654,43],"28084685",19,"Soullessmate",[656,657,38,601,658,43],"28015772",20,"graderjs",[142,660,18,418,143,43],21,[146,662,18,418,147,43],22,[664,662,38,356,665,43],"28013398","chadcatlett",[667,662,43,255,668,43],"28013556","JadeNB",[670,662,43,255,671,43],"28015351","johnisgood",[673,662,18,418,674,43],"28013869","Retr0id",[676,662,43,255,677,43],"28016186","qwertox",[679,662,18,418,677,43],"28016180",[681,682,43,238,683,43],"28015584",23,"lanevorockz",[685,686,43,238,687,43],"28013565",24,"Camillo",[689,686,43,255,690,43],"28016031","quenix",[692,686,43,238,693,43],"28016306","foxpurple",[695,686,43,238,690,43],"28020222",[697,686,43,238,698,43],"28020524","thealig",[700,686,43,238,693,43],"28023509",[702,686,43,238,690,43],"28059062",[704,705,43,238,706,43],"28013502",25,"andymoe",[192,708,38,601,193,43],26,[158,710,18,418,159,43],27,[712,710,43,713,11,18],"28013537","maker\u002Fneutral\u002Fexecution",[715,716,43,238,717,43],"28016686",28,"ctenb",[719,716,43,255,720,43],"28018459","brokensegue",[173,722,38,601,174,43],29,[724,725,18,726,727,43],"28015094",30,"project\u002Fcritical\u002Fdocs","8note",[177,729,38,601,178,43],31,[731,226,43,238,732,43],"28013228","sgt",[734,226,43,238,735,43],"28014202","schwartzworld",[737,738,38,601,739,43],"28013969",33,"andyxor",[741,742,18,418,743,43],"28014224",34,"chovybizzass",[745,746,18,597,747,43],"28014103",35,"geenat",[749,746,43,750,751,43],"28014214","commenter\u002Fneutral\u002Fidea","dmd",[753,746,45,364,754,43],"28014941","peterkelly",[],[757,762,767,772,777,781],{"id":758,"name":759,"points":760,"category":761},"28017051","redbean",223,"infrastructure",{"id":763,"name":764,"points":765,"category":766},"28059291","Till",209,"dev_tools",{"id":768,"name":769,"points":770,"category":771},"27996348","Clockwise",162,"productivity",{"id":773,"name":774,"points":775,"category":776},"28048528","ocean",159,"languages",{"id":778,"name":779,"points":780,"category":761},"28063597","Hookdeck",126,{"id":782,"name":783,"points":784,"category":785},"28037113","Open by Design",118,"creative",false,"v2",{"id":789,"name":5,"line":6,"ai":786,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":790,"top":18,"kw":791,"maker":11,"domain":10,"tier":19,"attempts":43,"strip":792,"url":9,"mix":793,"praise":98,"objection":118,"sub":21},28012829,-0.04,[24,25,26,27,28],"5100000053523513350133432505115332100122",[32,33,18,34],1793991031236]