[{"data":1,"prerenderedAt":287},["ShallowReactive",2],{"project-8591193":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":27,"sentiment":28,"stance_mix":29,"scored_comments":34,"aspects":35,"strip":42,"post_text":44,"what":45,"said":87,"maker_in_thread":166,"shape":177,"grid":181,"attempts":247,"same_week":248,"maker_shows":32,"maker_hidden":279,"reader_version":280,"card":281},"8591193","Pfs","A Git-like distributed file system for the Docker ecosystem","title","Show HN: Pfs – A Git-like distributed file system for the Docker ecosystem","http:\u002F\u002Fpachyderm-io.github.io\u002Fpfs\u002F","pachyderm-io.github.io","jdoliner",1415728890,2014,74,28,412,2272,19,"notable","infrastructure","distributed file system","other",[21,24,25,26],"mapreduce","git-like versioning","docker","mixed",-0.188,{"positive":30,"neutral":31,"mixed":32,"critical":33},2,5,1,4,12,{"performance":36,"idea":38,"docs":39,"design":40,"execution":41},{"positive":37,"critical":32},0,{"positive":30,"critical":30},{"positive":37,"critical":37},{"positive":37,"critical":32},{"positive":37,"critical":37},[37,37,32,37,43,37,32,37,37,37,37,37,43,37,31,37,43,37,32,37,31,37,43,37,43,37,32,43],3,null,{"problem":46,"audience":44,"pricing":44,"motivation":59,"time_to_build":44,"stack":67,"openness":84,"stage":85,"team_size":86},{"value":47,"source":48,"evidence":49},"Provides a git-like versioned distributed file system and MapReduce over commits","maker",[50,53,56],{"id":51,"by":11,"text":52},"8592130","This is a very good question. There's a growing sentiment in the Hadoop ecosystem that MapReduce is in someway passe and I think it's somewhat unfair. A lot of the confusion…",{"id":54,"by":11,"text":55},"8591470","> Looks like you are the same jdoliner behind RethinkDB There are many great people behind RethinkDB, 2 of them are involved in pfs actually. I am however the only one named…",{"id":57,"by":11,"text":58},"8591990","> Very neat. Btrfs is a practical choice. Looks like the Go http server listens for both changes and jobs? Right now pfs is just a file system so servers only listen for changes.…",{"value":60,"source":48,"evidence":61},"Better MapReduce than Hadoop, free of JVM and with git-like history",[62,65,66],{"id":63,"by":11,"text":64},"8591305","This is a fair criticism and I think we used unclear phrasing here. It's not that we don't consider the JVM to be a modern toolchain in general. Rather we think it's not a modern…",{"id":57,"by":11,"text":58},{"id":51,"by":11,"text":52},[68,76,80],{"value":69,"source":70,"evidence":71},"Go","commenter",[72],{"id":73,"by":74,"text":75},"8591832","doublerebel","Very neat. Btrfs is a practical choice. Looks like the Go http server listens for both changes and jobs? Could you tell me about the advantages git semantics add to MapReduce?…",{"value":77,"source":48,"evidence":78},"Docker",[79],{"id":54,"by":11,"text":55},{"value":81,"source":48,"evidence":82},"btrfs",[83],{"id":54,"by":11,"text":55},"open_source","prototype","team",{"praise":88,"objections":97,"requests":120,"questions":134,"alternatives":146},[89],{"value":90,"source":70,"evidence":91},"Git-like audit log and commit-based MapReduce idea",[92,93],{"id":73,"by":74,"text":75},{"id":94,"by":95,"text":96},"8592145","coffeemug","Not the author of the project but I can think of two reasons. Firstly, you can think of map\u002Freduce as the infrastructure for higher level operations (sort of like the assembly…",[98,105,113],{"value":99,"source":70,"evidence":100},"Docker images are large compared to jars",[101],{"id":102,"by":103,"text":104},"8591339","valarauca1","Doesn't this approach make users send 1GB+ Docker images instead of small 5-10MB .jar files?",{"value":106,"source":70,"evidence":107},"Static hashing distribution and master-slave replication are not truly distributed",[108,109],{"id":57,"by":11,"text":58},{"id":110,"by":111,"text":112},"8592188","tveita","Maybe I'm misunderstanding how these pieces will fit together, but from what I can see, the file system part is a fragile implementation of master-slave replication, which isn't…",{"value":114,"source":70,"evidence":115},"Linux-specific binaries are not portable",[116],{"id":117,"by":118,"text":119},"8591818","teacup50","How is schlepping around non-portable Linux-specific binaries packaged into what is essentially a VM a \"more modern toolchain\"?",[121,128],{"value":122,"source":70,"evidence":123},"Add a license",[124],{"id":125,"by":126,"text":127},"8591472","rectang","What's the license? I don't see anything in any of the source files, nor the README, nor any files such as LICENSE\u002FCOPYING\u002Fetc at the top of the repo.",{"value":129,"source":70,"evidence":130},"Compare with ZFS and Manta",[131],{"id":132,"by":74,"text":133},"8591393","Very creative way to combine the pros of recent fault-tolerant, distributed systems. Looks like you are the same jdoliner behind RethinkDB? The link \"git-like semantics\" points…",[135,139],{"value":136,"source":70,"evidence":137},"Why btrfs rather than ZFS",[138],{"id":132,"by":74,"text":133},{"value":140,"source":70,"evidence":141},"Is there a clustered production filesystem today",[142],{"id":143,"by":144,"text":145},"8592089","ComNik","Is there a distributed, ZFS-like filesystem usable in clustered production environments today? I'm intrigued by the idea of Git-like filesystems, but am not aware of any great…",[147,155,162],{"value":148,"source":70,"evidence":149},"Hadoop HDFS and MapReduce",[150,154],{"id":151,"by":152,"text":153},"8591266","tiles","> Pachyderm will eventually be a complete replacement for Hadoop, built on top of a modern toolchain instead of the JVM. Is the JVM legitimately not considered a modern…",{"id":110,"by":111,"text":112},{"value":156,"source":70,"evidence":157},"Apache Spark and YARN",[158],{"id":159,"by":160,"text":161},"8591985","Yacoby","I am not sure I see the point in implementing MapReduce when most of the current work seems to be in generalising MapReduce - i.e. Apache Spark and YARN. Is there any reasoning…",{"value":163,"source":70,"evidence":164},"Joyent Manta",[165],{"id":132,"by":74,"text":133},{"comments":167,"answered":168,"key":169},11,9,[170,171,174],{"id":63,"text":64},{"id":172,"text":173},"8591407","Useful Docker images can actually be made quite a bit smaller than 1GB+ by using small distros like busybox and following best practices such as cleaning up temporary data…",{"id":175,"text":176},"8591852","This is a very good point. In addition Docker itself tries to address this by using btrfs's diffing. Like a lot of things in Docker it's not perfect but it's getting there.",{"commenters":178,"max_depth":31,"maker_replies":167,"first_comment_mins":179,"lifespan_hours":180},15,17,32,[182,184,186,188,190,193,194,196,199,202,205,209,211,213,215,217,219,221,223,224,226,227,230,232,235,238,240,242],[151,37,37,183,152,37],"commenter\u002Fneutral\u002Fidea",[63,37,37,185,11,32],"maker\u002Fneutral\u002Fidea",[102,37,32,187,103,37],"project\u002Fcritical\u002Fperformance",[172,37,37,189,11,32],"maker\u002Fneutral\u002Fperformance",[191,37,43,192,95,37],"8591784","project\u002Fneutral\u002Fperformance",[175,37,37,189,11,32],[117,37,32,195,118,37],"project\u002Fcritical\u002Fidea",[197,37,37,183,198,37],"8593112","shykes",[200,37,37,183,201,37],"8591405","alrs",[203,37,37,204,11,32],"8591612","maker\u002Fneutral\u002Fnone",[206,37,37,207,208,37],"8591633","commenter\u002Fneutral\u002Fnone","necubi",[210,37,37,207,118,37],"8591843",[132,32,43,212,74,37],"project\u002Fmixed\u002Fidea",[54,32,37,214,11,32],"maker\u002Fpositive\u002Fdesign",[73,32,31,216,74,37],"project\u002Fpositive\u002Fidea",[57,32,37,218,11,32],"maker\u002Fneutral\u002Fexecution",[143,30,43,220,144,37],"project\u002Fneutral\u002Fidea",[222,30,37,204,11,32],"8592137",[159,43,32,195,160,37],[51,43,37,225,11,32],"maker\u002Fpositive\u002Fidea",[94,43,31,216,95,37],[228,43,37,229,11,32],"8592268","commenter\u002Fpositive\u002Fnone",[125,33,43,231,126,37],"project\u002Fneutral\u002Fdocs",[233,33,37,234,11,32],"8591519","maker\u002Fneutral\u002Fdocs",[236,33,43,231,237,37],"8594309","Hz8NSD",[239,33,37,234,11,32],"8599489",[110,31,32,241,111,37],"project\u002Fcritical\u002Fdesign",[243,244,43,245,246,37],"8594696",6,"project\u002Fneutral\u002Fexecution","jrcii",[],[249,254,259,264,269,274],{"id":250,"name":251,"points":252,"category":253},"8592597","Let's Talk Over Coffee",427,"consumer_social",{"id":255,"name":256,"points":257,"category":258},"8568181","StackShare",218,"dev_tools",{"id":260,"name":261,"points":262,"category":263},"8566971","Colores",157,"creative",{"id":265,"name":266,"points":267,"category":268},"8595533","simple-rootkit",94,"security_privacy",{"id":270,"name":271,"points":272,"category":273},"8565977","Wall of Text",90,"productivity",{"id":275,"name":276,"points":277,"category":278},"8566982","Assembly",70,"web_saas",false,"v2",{"id":282,"name":5,"line":6,"ai":279,"cat":20,"year":13,"points":14,"comments":15,"rec":27,"s":283,"top":18,"kw":284,"maker":11,"domain":10,"tier":19,"attempts":37,"strip":285,"url":9,"mix":286,"praise":90,"objection":99,"sub":21},8591193,-0.19,[21,24,25,26],"0010301000003050301050303013",[30,31,32,33],1794036524956]