[{"data":1,"prerenderedAt":749},["ShallowReactive",2],{"project-14771406":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":47,"post_text":50,"what":51,"said":132,"maker_in_thread":252,"shape":263,"grid":267,"attempts":712,"same_week":713,"maker_shows":18,"maker_hidden":742,"reader_version":743,"card":744},"14771406","Quilt","manage data like code","title","Show HN: Quilt – manage data like code","https:\u002F\u002Fquiltdata.com\u002F","quiltdata.com","akarve",1500052577,2017,367,178,43,2242,2,"legend","data","data package manager","library",[24,21,25,26,27],"data versioning","parquet","data sharing","versioned datasets","curious",0.08,{"positive":31,"neutral":32,"mixed":33,"critical":34},19,57,1,18,95,{"design":37,"idea":39,"docs":41,"execution":42,"pricing":44,"privacy":45,"performance":46},{"positive":38,"critical":18},3,{"positive":40,"critical":38},9,{"positive":18,"critical":33},{"positive":43,"critical":43},0,{"positive":43,"critical":18},{"positive":43,"critical":38},{"positive":18,"critical":43},[33,48,38,38,43,38,38,38,18,33,33,38,38,48,38,38,33,49,43,38,38,49,38,38,48,33,33,38,49,49,49,48,49,48,33,38,38,38,49,48],4,5,null,{"problem":52,"audience":63,"pricing":71,"motivation":80,"time_to_build":50,"stack":84,"openness":129,"stage":130,"team_size":131},{"value":53,"source":54,"evidence":55},"Finding, cleaning and organizing data is slow; data lacks version control and reusable packaging like code.","maker",[56,59],{"id":57,"by":11,"text":58},"14771423","Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of…",{"id":60,"by":61,"text":62},"14772304","smilliken","It's outrageous how little tooling support there is for version control in data compared to code. Every mainstream database forgets history with updates, don't support…",{"value":64,"source":54,"evidence":65},"Analysts, data scientists and researchers sharing datasets.",[66,67],{"id":57,"by":11,"text":58},{"id":68,"by":69,"text":70},"14772192","azag0","I am a scientist who sometimes publishes data sets with academic papers and this looks super useful both as a tool and as a potential publishing best practice. Currently doing…",{"value":72,"source":54,"evidence":73},"Namespaces and public repos free; business and on-prem users charged in TB blocks and seat blocks.",[74,77],{"id":75,"by":11,"text":76},"14780648","Namespaces are free. Public repos are free for unlimited data. Should we run a charity and also make private date free? :) We understand that cleaning\u002Ffiltering\u002Fpreparation are…",{"id":78,"by":11,"text":79},"14774014","We charge business and on-prem users in TB-sized blocks. So that part is variable cost, not flat. And we sell user seats in blocks of 10. What else should we be thinking about?…",{"value":81,"source":54,"evidence":82},"Bring package management and version control practices to data.",[83],{"id":57,"by":11,"text":58},[85,94,104,110,114,123],{"value":86,"source":54,"evidence":87},"Python",[88,91],{"id":89,"by":11,"text":90},"14771812","Which languages are most interesting to you? We wrote the client with an eye towards supporting R and Scala. A PR or FR on GitHub would be ideal. We also have a Slack channel…",{"id":92,"by":11,"text":93},"14785392","Where does it break down? Quilt can package and version directly from in-memory objects so if you are working in Python (more languages planned) you can package as you go and…",{"value":95,"source":54,"evidence":96},"Parquet",[97,100],{"id":98,"by":11,"text":99},"14774135","I agree wholeheartedly with this statement: \"application layer should be responsible for storage management and performance tuning\". I would take it one step further and say that…",{"id":101,"by":102,"text":103},"14772959","sixdimensional","I think it was a really interesting (and smart) choice to convert to Parquet format. Columnar storage is so much more efficient, and working with data in Parquet is pretty fast…",{"value":105,"source":54,"evidence":106},"pandas",[107],{"id":108,"by":11,"text":109},"14773939","We support arbitrary data formats in that Quilt falls back to a raw copy if it can't parse the file. On the columnar side (things we convert to Parquet) we support XLS, CSV, TSV,…",{"value":111,"source":54,"evidence":112},"pyarrow",[113],{"id":108,"by":11,"text":109},{"value":115,"source":54,"evidence":116},"Amazon S3",[117,120],{"id":118,"by":11,"text":119},"14771839","Only the owner and designated collaborators can view it. We offer private data in the cloud (S3) and are rolling out on-premise in case you want to run Quilt on your own…",{"id":121,"by":11,"text":122},"14780667","I'd like to chat more about this. Quilt uses S3 and I think we could make the process of getting data into Databricks much simpler. Drop me a line if you'd like to discuss aneesh…",{"value":124,"source":54,"evidence":125},"Docker",[126],{"id":127,"by":11,"text":128},"14780740","We need to clarify the EULA and terms on the website. The data belongs to the users and we want to keep it that way. Quilt is not \"hosted-only\". The whole point of the on-prem…","open_source","launched","company",{"praise":133,"objections":147,"requests":170,"questions":199,"alternatives":225},[134,139],{"value":135,"source":136,"evidence":137},"Parquet conversion and columnar storage are a smart choice.","commenter",[138],{"id":101,"by":102,"text":103},{"value":140,"source":136,"evidence":141},"Useful for publishing and sharing public datasets.",[142,143],{"id":68,"by":69,"text":70},{"id":144,"by":145,"text":146},"14776208","dawiddutoit","Awesome idea. Have you thought about a marketplace perhaps? Say for instance I create a package with all the cities in the world or all species of canine, host it in a…",[148,155,163],{"value":149,"source":136,"evidence":150},"Hard to use with existing data tools; data locked into a walled garden.",[151],{"id":152,"by":153,"text":154},"14773105","stewbrew","It looks like a plain html page but requires JS to view anything except: \"Please enable JavaScript to use this site.\" What a wonderful time to live in. Anyway, do I get this…",{"value":156,"source":136,"evidence":157},"Docs require JavaScript; site not usable without JS.",[158,159],{"id":152,"by":153,"text":154},{"id":160,"by":161,"text":162},"14771557","pmoriarty","Any way your web page could be made viewable without requiring Javascript? I disable it in my web browsers because of security and privacy concerns, and it would be great if I…",{"value":164,"source":136,"evidence":165},"Pricing model questioned.",[166],{"id":167,"by":168,"text":169},"14772111","reggieband","I think you're missing a trick with the pricing. My guess is the real money will come once data is treated like a commodity. So the big, big money will be in brokerage's and…",[171,179,186,192],{"value":172,"source":136,"evidence":173},"DOI support for dataset versions.",[174,175],{"id":68,"by":69,"text":70},{"id":176,"by":177,"text":178},"14772115","torchous","Any thoughts on adding DOIs? It's a complex subject wrt versioning, in particular (new DOI per version? How to keep track?). It would help tremendously with the academic…",{"value":180,"source":136,"evidence":181},"GitHub sign-up.",[182],{"id":183,"by":184,"text":185},"14771626","forkLding","Is there an option to be able to sign up using Github? That would make life much easier for me.",{"value":187,"source":54,"evidence":188},"R language support.",[189],{"id":190,"by":11,"text":191},"14773975","Hi. R is really important to us and we want to add support (probably through SparklyR). If you email me I can add you to our Slack channel and we can talk through extending Quilt…",{"value":193,"source":136,"evidence":194},"Frictionless data spec interop.",[195],{"id":196,"by":197,"text":198},"14771637","tzm","Thanks for releasing. Looks useful and aligned with several projects I've worked on. The first thing I looked for was a canonical package \u002F resource specification in build.py.…",[200,207,214],{"value":201,"source":136,"evidence":202},"How is it different from DAT?",[203],{"id":204,"by":205,"text":206},"14771772","fredcash25","How is this different\u002Fthe same as the DAT project (https:\u002F\u002Fdatproject.org\u002F and https:\u002F\u002Fgithub.com\u002Fdatproject)?",{"value":208,"source":136,"evidence":209},"Can users run their own instance and is it open source?",[210],{"id":211,"by":212,"text":213},"14772080","ah-","How much of it is open source? Can I run my own?",{"value":215,"source":136,"evidence":216},"How does it compare to git and Pachyderm?",[217,221],{"id":218,"by":219,"text":220},"14771999","fiatjaf","What does this do that cannot be done with git or similar software + data stored in some standard format?",{"id":222,"by":223,"text":224},"14772326","gouggoug","This makes me think of http:\u002F\u002Fwww.pachyderm.io\u002F. Although Quilt seems to be more like github for data, whereas Pachyderm is more like git for data.",[226,231,235,239,246],{"value":227,"source":136,"evidence":228},"Git \u002F GitHub",[229,230],{"id":218,"by":219,"text":220},{"id":60,"by":61,"text":62},{"value":232,"source":136,"evidence":233},"Pachyderm",[234],{"id":222,"by":223,"text":224},{"value":236,"source":136,"evidence":237},"DAT",[238],{"id":204,"by":205,"text":206},{"value":240,"source":136,"evidence":241},"data.world",[242],{"id":243,"by":244,"text":245},"14772882","edraferi","This looks like a cool project -- always glad to see new tools for statistical collaboration and reproducible research. How does this compare to what data.world [1] is doing?…",{"value":247,"source":54,"evidence":248},"Synapse",[249],{"id":250,"by":11,"text":251},"14774186","I know only a bit about Synapse. It seems like they have some valuable data. I think the biggest differences are in the culture and user experience. Quilt gets its DNA from the…",{"comments":253,"answered":254,"key":255},49,32,[256,257,260],{"id":57,"text":58},{"id":258,"text":259},"14771667","Ah, that would be useful. In the meantime our docs are browsable without JS https:\u002F\u002Fdocs.quiltdata.com\u002F. The articles linked at the bottom of the docs landing page are also…",{"id":261,"text":262},"14772170","Data cleaning is so necessary. `build.yml` already supports a limited set of feature (through pandas). In addition to custom data transformations, any \"out of the box\" cleaning…",{"commenters":264,"max_depth":265,"maker_replies":253,"first_comment_mins":33,"lifespan_hours":266},79,6,76,[268,270,272,274,278,282,286,287,289,292,295,299,302,303,305,308,311,313,317,318,320,322,324,327,329,331,335,338,341,344,347,350,352,354,356,358,360,362,363,367,369,372,375,377,379,382,385,388,391,393,395,398,401,405,408,410,413,414,418,419,422,424,427,429,431,434,435,438,441,445,447,451,452,454,455,457,460,461,463,465,468,471,473,475,479,481,484,486,489,490,492,495,496,497,500,503,505,509,512,515,517,518,521,523,526,528,530,532,533,535,537,539,542,546,548,549,551,554,556,557,559,562,564,566,568,570,573,575,577,579,581,583,585,587,591,594,596,598,602,606,608,611,612,615,617,621,623,628,632,634,636,640,642,647,650,652,654,656,658,661,664,666,668,670,674,676,678,680,683,685,689,691,696,700,704,706,708,710],[57,43,43,269,11,33],"maker\u002Fneutral\u002Fidea",[160,43,38,271,161,43],"project\u002Fneutral\u002Fdesign",[258,43,43,273,11,33],"maker\u002Fneutral\u002Fdocs",[275,43,33,276,277,43],"14772215","project\u002Fcritical\u002Fdesign","jwilk",[279,43,49,280,281,43],"14771590","project\u002Fpositive\u002Fidea","zitterbewegung",[283,43,38,284,285,43],"14771738","project\u002Fneutral\u002Fidea","kevinemoore",[261,43,43,269,11,33],[288,43,38,284,281,43],"14773701",[290,43,43,291,11,33],"14777058","maker\u002Fneutral\u002Fnone",[293,43,43,294,281,43],"14777568","commenter\u002Fneutral\u002Fnone",[296,43,38,297,298,43],"14771601","project\u002Fneutral\u002Fdocs","RangerScience",[300,43,38,297,301,43],"14771734","dimaryaz",[89,43,43,269,11,33],[304,43,38,284,298,43],"14772234",[306,43,38,307,285,43],"14772840","project\u002Fneutral\u002Fexecution",[309,43,38,284,310,43],"14771610","koolba",[312,43,38,307,285,43],"14771705",[314,43,43,315,316,43],"14773946","other\u002Fneutral\u002Fnone","amygdyl",[98,43,43,315,11,33],[319,43,43,315,316,43],"14774161",[321,43,43,291,11,33],"14771763",[323,43,43,307,11,33],"14772053",[325,43,38,307,326,43],"14773634","rcthompson",[328,43,43,315,316,43],"14774363",[330,43,43,315,316,43],"14774286",[332,43,38,333,334,43],"14771617","project\u002Fneutral\u002Fpricing","purvis",[336,43,43,337,11,33],"14771878","maker\u002Fneutral\u002Fpricing",[339,43,33,340,334,43],"14771961","project\u002Fcritical\u002Fpricing",[342,43,38,271,343,43],"14771660","Zaheer",[345,43,43,346,11,33],"14771696","maker\u002Fneutral\u002Fdesign",[348,43,49,349,343,43],"14773827","project\u002Fpositive\u002Fdesign",[351,43,38,271,285,43],"14771716",[353,43,38,271,343,43],"14773823",[355,43,38,333,298,43],"14771669",[357,43,43,333,11,33],"14771933",[359,43,38,284,298,43],"14772260",[361,43,38,307,285,43],"14772471",[108,43,43,307,11,33],[364,43,33,365,366,43],"14771746","project\u002Fcritical\u002Fnone","masklinn",[368,43,43,291,11,33],"14772096",[370,43,33,365,371,43],"14772143","xyzzy_plugh",[373,43,33,365,374,43],"14773979","raimue",[376,43,33,365,366,43],"14775680",[378,43,38,291,285,43],"14772200",[380,43,33,365,381,43],"14774924","axk",[383,43,33,365,384,43],"14775632","cyphar",[386,43,33,365,387,43],"14777399","kchr",[389,43,38,390,387,43],"14777414","project\u002Fmixed\u002Fnone",[392,43,43,315,316,43],"14773733",[394,43,43,315,11,33],"14774430",[396,43,43,315,397,43],"14774580","coredog64",[399,43,43,315,400,43],"14774940","mncharity",[402,43,38,403,404,43],"14777162","project\u002Fneutral\u002Fnone","ma2rten",[406,43,38,284,407,43],"14771755","adriancooney",[409,43,38,269,285,43],"14771807",[411,43,38,284,412,43],"14773446","mbeigel",[190,43,43,269,11,33],[415,43,38,416,417,43],"14771793","project\u002Fneutral\u002Fprivacy","thenipper",[118,43,43,416,11,33],[420,43,49,280,421,43],"14771902","dhbradshaw",[423,43,43,291,11,33],"14771945",[425,43,38,291,426,43],"14772175","ivotron",[428,43,43,294,285,43],"14772597",[430,43,43,284,11,33],"14777239",[432,43,38,403,433,43],"14773344","mbreese",[250,43,43,284,11,33],[436,43,49,280,437,43],"14775289","dtech81",[439,43,49,280,440,43],"14774012","mgalka",[442,43,33,443,444,43],"14774274","project\u002Fcritical\u002Fprivacy","philipov",[446,43,43,403,11,33],"14774366",[448,43,33,449,450,43],"14780312","project\u002Fcritical\u002Fidea","Galanwe",[75,43,43,333,11,33],[453,43,33,443,450,43],"14780706",[127,43,43,416,11,33],[456,43,33,443,450,43],"14781115",[458,43,43,294,459,43],"14784140","rattray",[60,33,49,280,61,43],[462,33,43,315,371,43],"14772431",[464,33,43,315,61,43],"14772734",[466,33,43,315,467,43],"14773822","hinkley",[469,33,43,315,470,43],"14772764","Lightbody",[472,33,43,315,61,43],"14773286",[474,33,43,315,470,43],"14773835",[476,33,43,477,478,43],"14773042","other\u002Fneutral\u002Fidea","falsedan",[480,33,43,477,467,43],"14773555",[482,33,38,284,483,43],"14776126","hn_throwaway_99",[485,33,43,403,11,33],"14792143",[487,33,43,315,488,43],"14780908","justinclift",[204,18,38,284,205,43],[491,18,43,284,11,33],"14771924",[493,18,38,284,494,43],"14777607","lmeyerov",[121,18,43,291,11,33],[222,38,38,284,223,43],[498,38,43,315,499,43],"14772512","lobster_johnson",[501,38,43,315,502,43],"14773658","jaz46",[504,38,43,403,11,33],"14773998",[506,48,49,507,508,43],"14774030","project\u002Fpositive\u002Fperformance","rspeer",[510,48,43,511,11,33],"14774160","project\u002Fneutral\u002Fperformance",[513,48,38,511,514,43],"14775434","IanCal",[516,48,43,315,508,43],"14785679",[152,49,33,276,153,43],[519,49,49,280,520,43],"14773754","eternauta3k",[522,49,43,403,11,33],"14774388",[524,49,38,284,525,43],"14773976","noobermin",[527,49,43,284,11,33],"14774409",[529,49,38,416,514,43],"14775450",[531,49,43,403,11,33],"14790120",[243,265,38,284,244,43],[534,265,43,284,11,33],"14792155",[101,536,49,507,102,43],7,[538,536,38,403,285,43],"14773085",[540,536,38,403,541,43],"14774072","themgt",[543,544,43,315,545,43],"14775653",8,"geraldbauer",[547,544,43,315,244,43],"14788350",[167,40,33,340,168,43],[550,40,33,449,478,43],"14773071",[552,40,43,294,553,43],"14774938","styfle",[555,40,43,294,478,43],"14778595",[78,40,43,333,11,33],[558,40,38,333,168,43],"14774455",[560,40,43,561,11,33],"14775245","commenter\u002Fneutral\u002Fpricing",[218,563,38,284,219,43],10,[565,563,43,284,11,33],"14772036",[183,567,38,307,184,43],11,[569,567,43,307,11,33],"14771723",[571,567,49,572,184,43],"14783695","project\u002Fpositive\u002Fdocs",[574,567,43,297,11,33],"14785382",[196,576,49,280,197,43],12,[578,576,43,294,285,43],"14771780",[580,576,43,271,11,33],"14771905",[68,582,49,280,69,43],13,[584,582,38,284,285,43],"14772273",[586,582,43,315,514,43],"14781024",[588,589,38,403,590,43],"14772130",14,"pinhead",[592,589,43,315,593,43],"14774901","klodolph",[144,595,49,280,145,43],15,[597,595,43,284,11,33],"14785397",[599,600,49,349,601,43],"14771701",16,"hfourm",[603,604,49,572,605,43],"14771928",17,"dpiers",[607,604,38,297,285,43],"14772000",[609,34,33,449,610,43],"14778275","nicodjimenez",[92,34,43,307,11,33],[613,31,49,349,614,43],"14772295","temuze",[616,31,43,271,11,33],"14774083",[618,619,38,284,620,43],"14772627",20,"servilio",[622,619,43,284,11,33],"14774631",[624,625,49,626,627,43],"14776427",21,"maker\u002Fpositive\u002Fnone","tannhauser23",[629,630,38,297,631,43],"14773249",22,"wodenokoto",[633,630,38,284,285,43],"14773334",[635,630,43,297,11,33],"14774034",[637,638,38,284,639,43],"14774016",23,"jinjin2",[641,638,43,284,11,33],"14774605",[643,644,33,645,646,43],"14772985",24,"project\u002Fcritical\u002Fdocs","mdevere",[648,644,43,649,285,43],"14773093","commenter\u002Fneutral\u002Fdocs",[651,644,43,294,525,43],"14773981",[653,644,43,297,11,33],"14774046",[176,655,38,284,177,43],25,[657,655,43,284,11,33],"14772138",[659,655,43,660,177,43],"14772371","commenter\u002Fneutral\u002Fidea",[662,655,43,660,663,43],"14773182","Steeeve",[665,655,38,284,285,43],"14772145",[667,655,38,284,177,43],"14772510",[669,655,38,284,285,43],"14772955",[671,672,38,403,673,43],"14772579",26,"anon1253",[675,672,38,403,285,43],"14773164",[211,677,38,403,212,43],27,[679,677,43,403,11,33],"14772106",[681,677,38,416,682,43],"14772335","heinrichhartman",[684,677,43,333,11,33],"14772528",[686,687,43,315,688,43],"14774488",28,"nosefouratyou",[690,687,43,284,11,33],"14774598",[692,693,49,694,695,43],"14772225",29,"project\u002Fpositive\u002Fnone","synaesthesisx",[697,698,38,403,699,43],"14776791",30,"atomical",[701,702,38,333,703,43],"14772220",31,"huac",[705,702,38,333,285,43],"14772941",[707,702,49,694,703,43],"14773110",[709,702,38,403,285,43],"14773260",[711,702,43,403,11,33],"14774637",[],[714,719,724,729,734,738],{"id":715,"name":716,"points":717,"category":718},"14805265","gamblers-dice",523,"languages",{"id":720,"name":721,"points":722,"category":723},"14799111","Cinc",363,"productivity",{"id":725,"name":726,"points":727,"category":728},"14794949","choppingboard.recipes",208,"web_saas",{"id":730,"name":731,"points":732,"category":733},"14769317","Python books",150,"learning_science",{"id":735,"name":736,"points":737,"category":733},"14798336","Learn Anything",145,{"id":739,"name":740,"points":741,"category":733},"14763591","Language identification quiz",140,false,"v2",{"id":745,"name":5,"line":6,"ai":742,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":29,"top":18,"kw":746,"maker":11,"domain":10,"tier":19,"attempts":43,"strip":747,"url":9,"mix":748,"praise":135,"objection":149,"sub":21},14771406,[24,21,25,26,27],"1433033321133433150335334113555454133354",[31,32,33,34],1794036557278]