[{"data":1,"prerenderedAt":662},["ShallowReactive",2],{"project-33969908":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":27,"sentiment":28,"stance_mix":29,"scored_comments":34,"aspects":35,"strip":45,"post_text":48,"what":49,"said":98,"maker_in_thread":229,"shape":241,"grid":246,"attempts":625,"same_week":626,"maker_shows":42,"maker_hidden":654,"reader_version":655,"card":656},"33969908","XetHub","We scaled Git to support 1 TB repos","title","Show HN: We scaled Git to support 1 TB repos","https:\u002F\u002Fxethub.com\u002Fuser\u002Flogin","xethub.com","reverius42",1670944494,2022,279,144,156,2570,7,"legend","data","data versioning","web_app",[21,24,25,26],"git large files","dataset version control","git lfs alternative","curious",-0.189,{"positive":30,"neutral":31,"mixed":32,"critical":33},13,39,2,19,73,{"execution":36,"design":39,"idea":40,"docs":41,"pricing":43,"performance":44},{"positive":37,"critical":38},3,0,{"positive":38,"critical":32},{"positive":18,"critical":30},{"positive":38,"critical":42},1,{"positive":38,"critical":42},{"positive":42,"critical":32},[37,37,46,37,42,47,47,37,38,38,38,38,37,37,42,38,38,38,37,37,46,37,37,32,37,37,37,42,42,42,42,42,32,42,37,46,46,42,37,38],5,4,"I’ve been in the MLOps space for ~10 years, and data is still the hardest unsolved open problem. Code is versioned using Git, data is stored somewhere else, and context often lives in a 3rd location like Slack or GDocs. This is why we built XetHub, a platform that enables teams to treat data like code, using Git.\n\nUnlike Git LFS, we don’t just store the files. We use content-defined chunking and Merkle Trees to dedupe against everything in history. This allows small changes in large files to be stored compactly. Read more here: https:\u002F\u002Fxethub.com\u002Fassets\u002Fdocs\u002Fhow-xet-deduplication-works\n\nToday,",{"problem":50,"audience":60,"pricing":69,"motivation":77,"time_to_build":81,"stack":82,"openness":95,"stage":96,"team_size":97},{"value":51,"source":52,"evidence":53},"Versioning large data files alongside code in Git is hard; Git LFS stores whole files","maker",[54,56],{"id":55},"post",{"id":57,"by":58,"text":59},"33970910","sk0g","That is exactly what git-lfs is, a way to \"version control\" binary files, by storing revisions - possibly separately, while the actual repo contains text files + \"pointer\" files…",{"value":61,"source":62,"evidence":63},"ML teams and others versioning large data with Git","inferred",[64,65],{"id":55},{"id":66,"by":67,"text":68},"33970807","mardifoufs","I literally don't know anyone or any team in ML using git as a data versioning tool. It doesn't even make sense to me, and most mlops people I have talked to would agree. Is that…",{"value":70,"source":71,"evidence":72},"Free tier up to 20GB; team tier priced by contact","commenter",[73],{"id":74,"by":75,"text":76},"33970431","V1ndaar","You say you support up to 1TB repositories, but from your pricing page all I see is the free tier for up to 20GB and one for teams. The latter doesn't have a price and only a…",{"value":78,"source":52,"evidence":79},"Data is the hardest unsolved problem in MLOps; data is stored apart from code",[80],{"id":55},null,[83,87,91],{"value":84,"source":52,"evidence":85},"Rust (client, cache and storage)",[86],{"id":55},{"value":88,"source":52,"evidence":89},"Go (web application)",[90],{"id":55},{"value":92,"source":52,"evidence":93},"Vega (custom visualizations)",[94],{"id":55},"unknown","launched","company",{"praise":99,"objections":122,"requests":148,"questions":169,"alternatives":191},[100,111],{"value":101,"source":71,"evidence":102},"Block-level deduplication is a real improvement over file-level dedup",[103,107],{"id":104,"by":105,"text":106},"33970738","kspacewalk2","Sounds like `git annex` is file-level deduplication, whereas this tool is block-level, but with some intelligent, context-specific way of defining how to split up the data (i.e.…",{"id":108,"by":109,"text":110},"33970836","rajatarya","XetHub Co-founder here. Yes, one illustrative example of the difference is: Imagine you have a 500MB file (lastmonth.csv) where every day 1MB is changed. With file-based…",{"value":112,"source":71,"evidence":113},"Mount command for browsing without downloading is useful",[114,118],{"id":115,"by":116,"text":117},"33970864","Game_Ender","The tl;dr is that \"xet\" is like GitLFS (it stores pointers in Git, with the data in a remote server and uses smudge filters to make this transparent) with some additional…",{"id":119,"by":120,"text":121},"33970382","ylow","Cofounder\u002FCEO here! I think it less about \"versioning\", but the ability to modify with confidence knowing that you can go back in time anytime. (Minor clarification: we are not…",[123,137,141],{"value":124,"source":71,"evidence":125},"Using Git for large binary data is a bad idea",[126,130,133],{"id":127,"by":128,"text":129},"33970413","ltbarcly3","The lack of reasons for doing it IS the reason against. GIT isn't a magic 'good way' to store arbitrary data, it's a good way to collaborate on projects implemented using most…",{"id":131,"by":128,"text":132},"33970591","I think this is a foot-gun, it's a bad idea even if it works great, and I doubt it works very well. You should manage your build artifacts explicitly, not just jam them in git…",{"id":134,"by":135,"text":136},"33972289","subnetwork","This feels like something that is prime for abuse. I agree with @bastardoperator, treating git for file storage is going to go nowhere good.",{"value":138,"source":71,"evidence":139},"Pricing unclear for individuals with mid-sized repos",[140],{"id":74,"by":75,"text":76},{"value":142,"source":71,"evidence":143},"Login page as the link is confusing",[144],{"id":145,"by":146,"text":147},"33970242","ledauphin","The link takes me to a login page. It would be nice to see that fixed to somehow match the title.",[149,155,162],{"value":150,"source":71,"evidence":151},"Support for Perforce-style locking",[152],{"id":153,"by":58,"text":154},"33980125","The centralised model of Perforce is more of a natural fit for one thing, since by default it allows you to clone subsets, and just the latest version of files. File locking is…",{"value":156,"source":71,"evidence":157},"Better progress bar during fetch",[158],{"id":159,"by":160,"text":161},"33971109","amelius","Does this fix the problem that Git becomes unreasonably slow when you have large binary files in the repo? Also, why can't Git show me an accurate progress-bar while fetching?",{"value":163,"source":71,"evidence":164},"Support for Unreal Engine binary assets",[165],{"id":166,"by":167,"text":168},"33975031","AustinDev","Have you tested this out with Unreal Engine blueprint files? If you all can do block-based diffing on those, and other binary assets used in game development it'd be huge for…",[170,177,184],{"value":171,"source":71,"evidence":172},"How does it differ from Git LFS",[173],{"id":174,"by":175,"text":176},"33975153","angrais","How's this differ from using git LFS?",{"value":178,"source":71,"evidence":179},"How are data files chunked",[180],{"id":181,"by":182,"text":183},"33970481","amadvance","How data is split in chunks ? Just curious.",{"value":185,"source":71,"evidence":186},"Can it store Docker container images",[187],{"id":188,"by":189,"text":190},"33970694","chubot","Can it be used to store container images (Docker)? As far as I remember they are just compressed tar files. Does the compression defeat Xet's own chunking? Can you sync to…",[192,199,210,218,225],{"value":193,"source":71,"evidence":194},"git-annex",[195],{"id":196,"by":197,"text":198},"33970526","JZL003","I also have a lot of issues with versioning data. But look at git annex - it's free, self hosted and has a very easy underlying data structure [1]. So I don't even use the magic…",{"value":200,"source":71,"evidence":201},"DVC",[202,206],{"id":203,"by":204,"text":205},"33972209","bastardoperator","I actually encountered a 4TB git repo. After pulling all the binary shit out of it the repo was actually 200MB. Anything that promotes treating git like a filesystem is a bad…",{"id":207,"by":208,"text":209},"33973021","zachmu","You're suffering from a failure of imagination, maybe because you've never been able to version data usefully before. There are already lots of interesting applications, and it's…",{"value":211,"source":71,"evidence":212},"Perforce",[213,217],{"id":214,"by":215,"text":216},"33971547","Izmaki","If I had to \"version control\" a 1 TB large repo - and assuming I wouldn't quit in anger - I would use a tool which is built for this kind of need and has been used in the…",{"id":153,"by":58,"text":154},{"value":219,"source":71,"evidence":220},"Dolt",[221],{"id":222,"by":223,"text":224},"33972997","timsehn","Founder of DoltHub here. One of my team pointed me at this thread. Congrats on the launch. Great to see more folks tackling the data versioning problem. Dolt hasn't come up here…",{"value":226,"source":71,"evidence":227},"Pachyderm",[228],{"id":203,"by":204,"text":205},{"comments":230,"answered":42,"key":231},6,[232,235,238],{"id":233,"text":234},"33978100","They're not a requirement! git-xet has a --no-smudge option if you prefer to deal with an unsmudged repo.",{"id":236,"text":237},"33970287","Visit https:\u002F\u002Fxetdata.com for more info! (Sorry, can't edit the post link now.)",{"id":239,"text":240},"33971191","Mostly! (At the moment, it doesn't fully fix the slowdown associated with storing large binary files, but reduces it by 90-99%. We're working on improving to closer to 100% that…",{"commenters":242,"max_depth":243,"maker_replies":230,"first_comment_mins":244,"lifespan_hours":245},88,9,24,107,[247,251,254,257,259,261,264,266,270,273,275,279,282,285,287,291,294,296,298,300,303,306,309,310,312,314,317,319,321,324,327,330,333,336,338,340,342,345,348,351,354,357,359,362,363,365,367,370,373,375,377,380,383,386,387,390,393,396,398,400,403,406,409,412,414,416,419,421,425,428,431,433,436,439,442,446,448,451,454,459,463,466,470,472,475,477,479,482,485,487,491,492,495,497,499,501,503,507,510,513,516,518,523,525,528,532,536,539,540,542,543,545,548,550,552,554,555,557,558,560,561,563,566,569,572,574,575,578,581,584,588,591,592,596,597,599,601,604,607,609,614,617,620,622],[248,38,37,249,250,38],"33972570","project\u002Fmixed\u002Fexecution","jrockway",[252,38,38,253,120,38],"33973217","commenter\u002Fneutral\u002Fnone",[255,38,38,253,256,38],"33976502","unqueued",[233,38,38,258,11,42],"maker\u002Fneutral\u002Fnone",[260,38,38,253,120,38],"33973255",[262,38,37,263,250,38],"33973460","project\u002Fneutral\u002Fdesign",[265,38,37,263,120,38],"33973614",[267,38,37,268,269,38],"33974110","project\u002Fneutral\u002Fexecution","catiopatio",[271,38,46,272,120,38],"33974148","project\u002Fpositive\u002Fexecution",[274,38,46,272,269,38],"33976158",[276,38,37,277,278,38],"33977851","project\u002Fneutral\u002Fnone","ArchOversight",[280,38,38,281,189,38],"33973609","other\u002Fneutral\u002Fnone",[283,38,38,281,284,38],"33978329","persedes",[286,38,38,281,189,38],"33980777",[288,38,38,289,290,38],"33977474","commenter\u002Fcritical\u002Fidea","ilyt",[292,38,38,293,250,38],"33978393","commenter\u002Fneutral\u002Fidea",[196,42,42,295,197,38],"project\u002Fcritical\u002Fidea",[104,42,37,297,105,38],"project\u002Fneutral\u002Fidea",[108,42,46,299,109,38],"project\u002Fpositive\u002Fidea",[301,42,37,297,302,38],"33973664","civilized",[304,42,37,305,109,38],"33973847","project\u002Fneutral\u002Fdocs",[307,42,37,297,308,38],"33973938","prirun",[166,42,46,272,167,38],[311,42,37,277,109,38],"33976185",[313,42,37,277,167,38],"33977115",[315,42,37,277,316,38],"33979626","vvanders",[318,42,37,277,167,38],"33992736",[320,42,38,281,256,38],"33976418",[322,42,38,281,323,38],"33971524","cma",[325,42,38,281,326,38],"33971654","synergy20",[328,42,38,281,329,38],"33972001","dark-star",[331,42,38,281,332,38],"33972474","mattpallissard",[334,42,38,281,335,38],"33973029","xmodem",[337,42,38,281,332,38],"33973859",[339,42,38,281,323,38],"33973421",[341,42,38,281,329,38],"33982720",[343,42,38,281,344,38],"33982762","tatersolid",[346,42,38,281,347,38],"33973502","alchemist1e9",[349,42,38,281,350,38],"33974098","rsync",[352,42,38,281,353,38],"33974959","darau1",[355,42,38,281,356,38],"33979038","kadoban",[358,42,38,281,256,38],"33976723",[360,42,38,281,361,38],"33971797","timbotron",[222,32,38,281,223,38],[364,32,38,253,120,38],"33973506",[74,37,37,366,75,38],"project\u002Fneutral\u002Fpricing",[368,37,37,369,109,38],"33970606","maker\u002Fneutral\u002Fpricing",[371,37,37,366,372,38],"33971072","blagie",[374,37,37,366,372,38],"33982757",[376,37,37,366,75,38],"33971300",[378,47,37,297,379,38],"33975027","TacticalCoder",[381,46,38,281,382,38],"33971333","dandigangi",[384,46,38,281,385,38],"33971867","irrational",[214,230,42,295,215,38],[388,230,38,253,389,38],"33972174","tinco",[391,230,38,281,392,38],"33972218","TylerE",[394,230,38,253,395,38],"33973955","xur17",[397,230,38,281,392,38],"33974059",[399,230,38,281,395,38],"33974105",[401,230,38,281,402,38],"33977417","hinkley",[404,230,38,253,405,38],"33973347","ryneandal",[407,230,38,253,408,38],"33974838","Phelinofist",[410,230,38,281,411,38],"33974939","llanowarelves",[413,230,38,281,402,38],"33977453",[415,230,38,281,411,38],"33979006",[417,230,46,299,418,38],"33975989","mentos",[420,18,42,295,256,38],"33976558",[422,423,38,281,424,38],"33974642",8,"polemic",[426,243,37,297,427,38],"33975216","culanuchachamim",[429,243,46,299,430,38],"33975239","lazide",[432,243,42,295,427,38],"33975337",[434,243,38,435,430,38],"33975447","commenter\u002Fpositive\u002Fidea",[437,243,38,281,438,38],"33975339","layer8",[440,243,46,299,441,38],"33976547","dafelst",[443,444,37,277,445,38],"33975512",10,"iFire",[203,447,42,295,204,38],11,[449,447,37,297,450,38],"33973985","frognumber",[452,447,37,453,120,38],"33975494","maker\u002Fneutral\u002Fdesign",[455,456,46,457,458,38],"33970657",12,"project\u002Fpositive\u002Fnone","wnzl",[460,456,38,461,462,38],"33972654","other\u002Fneutral\u002Fpricing","ziml77",[464,30,42,295,465,38],"33976251","COMMENT___",[467,468,46,457,469,38],"33974708",14,"Wojtkie",[471,468,37,258,120,38],"33975624",[145,473,42,474,146,38],15,"project\u002Fcritical\u002Fdesign",[236,473,38,476,11,42],"maker\u002Fneutral\u002Fdocs",[188,478,37,297,189,38],16,[480,478,37,481,120,38],"33970802","maker\u002Fneutral\u002Fexecution",[159,483,37,484,160,38],17,"project\u002Fneutral\u002Fperformance",[239,483,38,486,11,42],"maker\u002Fneutral\u002Fperformance",[488,489,37,297,490,38],"33995748",18,"mattewong",[181,33,37,268,182,38],[493,33,37,268,494,38],"33970645","sesm",[496,33,37,481,120,38],"33970736",[498,33,37,268,494,38],"33971943",[500,33,37,268,120,38],"33973665",[502,33,37,481,120,38],"33970668",[504,505,37,297,506,38],"33971555",20,"blobbers",[508,505,37,297,509,38],"33971652","stevelacy",[511,505,38,253,512,38],"33971753","IshKebab",[514,505,42,295,515,38],"33973532","TillE",[517,505,38,293,512,38],"33974130",[519,520,42,521,522,38],"33974817",21,"project\u002Fcritical\u002Fperformance","the_arun",[524,520,37,486,120,38],"33975472",[115,526,42,527,116,38],22,"project\u002Fcritical\u002Fdocs",[529,526,38,530,531,38],"33970987","commenter\u002Fneutral\u002Fdocs","nightpool",[533,534,42,295,535,38],"33970285",23,"Aperocky",[537,534,38,293,538,38],"33970325","ch71r22",[127,534,42,295,128,38],[541,534,38,293,120,38],"33970513",[131,534,42,295,128,38],[544,534,38,293,116,38],"33970933",[546,534,38,293,547,38],"33971140","wpietri",[549,534,42,295,128,38],"33974855",[551,534,38,281,547,38],"33978708",[553,534,38,281,128,38],"33984484",[57,534,37,249,58,38],[556,534,37,486,109,38],"33971210",[153,534,42,474,58,38],[559,534,38,453,11,42],"33992296",[66,534,42,295,67,38],[562,534,38,253,128,38],"33974884",[564,534,38,253,565,38],"34034571","mickkygoel",[567,534,46,299,568,38],"33970350","guardian5x",[570,534,42,521,571,38],"33970361","substation13",[573,534,38,484,11,42],"33970415",[119,534,37,297,120,38],[576,534,38,253,577,38],"33970644","krageon",[579,534,38,253,580,38],"33970732","rafael09ed",[582,534,46,299,583,38],"33970436","treeman79",[585,534,38,586,587,38],"33970625","other\u002Fcritical\u002Fnone","bfm",[589,534,46,299,590,38],"33971113","oftenwrong",[207,534,38,253,208,38],[593,534,38,594,595,38],"33973108","other\u002Fneutral\u002Fidea","WorldMaker",[134,244,42,295,135,38],[598,244,38,263,11,42],"33978188",[174,600,37,277,175,38],25,[602,600,46,603,120,38],"33975508","project\u002Fpositive\u002Fperformance",[605,600,38,530,606,38],"33978176","ironfootnz",[608,600,38,253,120,38],"33978217",[610,611,42,612,613,38],"33971434",26,"project\u002Fcritical\u002Fpricing","web007",[615,611,38,616,512,38],"33971917","commenter\u002Fneutral\u002Fpricing",[618,611,38,616,619,38],"33972394","mdaniel",[621,611,38,616,512,38],"33973982",[623,611,38,461,624,38],"33971930","Alifatisk",[],[627,632,636,640,645,649],{"id":628,"name":629,"points":630,"category":631},"33923907","LearnGPT",352,"ai",{"id":633,"name":634,"points":635,"category":631},"33910863","Hello (sayhello.so)",318,{"id":637,"name":638,"points":639,"category":20},"33931971","Pg_CRDT",296,{"id":641,"name":642,"points":643,"category":644},"33922754","Pynecone",209,"dev_tools",{"id":646,"name":647,"points":648,"category":644},"33906520","Ezy",155,{"id":650,"name":651,"points":652,"category":653},"33985550","Juice Labs",138,"infrastructure",false,"v2",{"id":657,"name":5,"line":6,"ai":654,"cat":20,"year":13,"points":14,"comments":15,"rec":27,"s":658,"top":18,"kw":659,"maker":11,"domain":10,"tier":19,"attempts":38,"strip":660,"url":9,"mix":661,"praise":101,"objection":124,"sub":21},33969908,-0.19,[21,24,25,26],"3353144300003310003353323331111121355130",[30,31,32,33],1794036628402]