[{"data":1,"prerenderedAt":424},["ShallowReactive",2],{"project-13628952":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":34,"aspects":35,"strip":43,"post_text":44,"what":45,"said":73,"maker_in_thread":181,"shape":192,"grid":197,"attempts":386,"same_week":387,"maker_shows":308,"maker_hidden":416,"reader_version":417,"card":418},"13628952","s3-lambda","Lambda functions over S3 objects: each, map, reduce, filter","title","Show HN: s3-lambda – Lambda functions over S3 objects: each, map, reduce, filter","https:\u002F\u002Fgithub.com\u002Flittlstar\u002Fs3-lambda","github.com","wellsjohnston",1486915720,2017,177,74,199,2242,9,"notable","dev_tools","s3 utility library","library",[24,25,26,27],"s3","functional programming","node.js","batch processing","mixed",-0.298,{"positive":31,"neutral":18,"mixed":32,"critical":33},3,0,11,23,{"idea":36,"docs":38,"execution":39,"design":41,"performance":42},{"positive":31,"critical":37},5,{"positive":32,"critical":31},{"positive":32,"critical":40},1,{"positive":32,"critical":40},{"positive":32,"critical":40},[40,32,31,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,40,40,31,31,40,37,31,32,40,32,32,32,32,31,40,40,40,31,32,40,37,31],null,{"problem":46,"audience":53,"pricing":44,"motivation":61,"time_to_build":44,"stack":66,"openness":71,"stage":71,"team_size":72},{"value":47,"source":48,"evidence":49},"S3 offers no way to filter, list or process objects in bulk without writing recursive SDK code","maker",[50],{"id":51,"by":11,"text":52},"13631968","I've had the same frustration with S3, and the reasons you went over are what drove me to create this. It seems like they made S3 (10 years ago now?) and just forgot about it.…",{"value":54,"source":55,"evidence":56},"Developers processing data stored in S3 buckets","inferred",[57,60],{"id":58,"by":11,"text":59},"13629756","Thank you :) Writing this was a necessity for me, being a 1-person data team coming from a Node.js background.",{"id":51,"by":11,"text":52},{"value":62,"source":48,"evidence":63},"Frustration with S3 lacking filtering and bulk operations while working as a one-person data team",[64,65],{"id":51,"by":11,"text":52},{"id":58,"by":11,"text":59},[67],{"value":68,"source":48,"evidence":69},"Node.js",[70],{"id":58,"by":11,"text":59},"unknown","solo",{"praise":74,"objections":90,"requests":123,"questions":140,"alternatives":155},[75,83],{"value":76,"source":77,"evidence":78},"Composable map\u002Ffilter chains running in parallel over S3 objects is a strong design","commenter",[79],{"id":80,"by":81,"text":82},"13629651","simonw","First impression: this is a brilliant piece of software design. The ability to compose a map\u002Ffilter chain and execute it in parallel against every object in an S3 bucket that…",{"value":84,"source":77,"evidence":85},"Generic functional interface to S3 is useful",[86],{"id":87,"by":88,"text":89},"13630206","dhpe","Really nice to have a generic functional interface to S3. Thanks.",[91,106,116],{"value":92,"source":77,"evidence":93},"Name 'lambda' is confused with AWS Lambda; the functions run locally",[94,98,102],{"id":95,"by":96,"text":97},"13629854","dschnurr","Might make sense to rename this to avoid confusion with AWS Lambda (I immediately thought it was related). Otherwise, looks like an awesome library!",{"id":99,"by":100,"text":101},"13629979","koolba","... Except it's not! The \"lambda\" here isn't AWS Lambda. It's a locally executed function. Now if this scheduled a bunch of real Lambdas to execute the work for each bucket then…",{"id":103,"by":104,"text":105},"13630218","cocktailpeanuts","I also came here thinking this is some sort of aws lambda triggered within the context of certain s3 file. I would say anyone who's heard of AWS lambda would think that way.…",{"value":107,"source":77,"evidence":108},"Listing all keys before running is slow for millions of files",[109,113],{"id":110,"by":111,"text":112},"13630930","kvz","Getting aan index of (millions of) files on s3 is very slow for us, like, days. Is there anything you do to work around this? It seems since this is not an AWS Lambda project the…",{"id":114,"by":11,"text":115},"13631008","This does not have to do with AWS Lambda, I'm thinking about renaming it to \"functional-s3\", or something similar. To answer your question, there isn't really a workaround for…",{"value":117,"source":77,"evidence":118},"Destructive map by default is risky",[119],{"id":120,"by":121,"text":122},"13633121","wcdolphin","I thinking having the default be destructive for mapping is a strange design decision. That is going to bite someone one day soon.",[124,129,136],{"value":125,"source":77,"evidence":126},"Rename the project to avoid confusion with AWS Lambda",[127,128],{"id":95,"by":96,"text":97},{"id":103,"by":104,"text":105},{"value":130,"source":77,"evidence":131},"Run the functions on real AWS Lambda",[132,133],{"id":99,"by":100,"text":101},{"id":134,"by":81,"text":135},"13632562","Bah. My first impression was totally wrong in that case. Here's hoping someone builds a version of this that executes magically in the lambda cloud.",{"value":137,"source":48,"evidence":138},"Stream keys with a generator instead of listing all first",[139],{"id":114,"by":11,"text":115},[141,148],{"value":142,"source":77,"evidence":143},"Is it subject to S3 eventual consistency",[144],{"id":145,"by":146,"text":147},"13631688","cle","Is this susceptible to any of S3's eventual consistency constraints?",{"value":149,"source":77,"evidence":150},"Where does code run versus where is chaining done",[151],{"id":152,"by":153,"text":154},"13629856","DenisM","So... the client-side code iterates S3 objects matching a certain filter, and then schedules a lambda for each one of those objects. Is that right? Or does the iteration…",[156,163,170,177],{"value":157,"source":77,"evidence":158},"AWS Athena for querying S3",[159],{"id":160,"by":161,"text":162},"13630619","hayd","see also aws athena https:\u002F\u002Faws.amazon.com\u002Fathena\u002F ?",{"value":164,"source":77,"evidence":165},"Google BigQuery",[166],{"id":167,"by":168,"text":169},"13630095","avip","This is a nice project. For real-world use cases, we have good alternatives: 1. Migrate s3 ==> gc and use BigQuery which does support udf 2. Register to databricks (I'm not…",{"value":171,"source":77,"evidence":172},"S3 inventory reports",[173],{"id":174,"by":175,"text":176},"13637648","otterley","Have you looked into the inventory functionality? It was just added last November. http:\u002F\u002Fdocs.aws.amazon.com\u002FAmazonS3\u002Flatest\u002Fdev\u002Fstorage-inventory.html",{"value":178,"source":77,"evidence":179},"Databricks",[180],{"id":167,"by":168,"text":169},{"comments":182,"answered":183,"key":184},10,6,[185,186,189],{"id":51,"text":52},{"id":187,"text":188},"13629956","Ah yeah, just realizing this...what would you recommend?",{"id":190,"text":191},"13630785","Okay this seems like a good alternative. I just renamed the repo. Renaming it on npm...is a bit cumbersome :|",{"commenters":193,"max_depth":194,"maker_replies":182,"first_comment_mins":195,"lifespan_hours":196},43,7,130,412,[198,202,206,208,212,214,218,221,223,226,228,231,234,237,239,243,246,248,251,253,255,258,261,264,266,268,270,271,273,275,277,279,282,285,287,289,293,294,297,298,301,304,307,310,312,315,316,320,322,323,325,326,329,333,337,340,342,344,347,349,352,355,356,359,361,363,366,368,371,375,377,380,381,384],[199,32,40,200,201,32],"13631891","project\u002Fcritical\u002Fidea","hoodoof",[203,32,32,204,205,32],"13631953","commenter\u002Fneutral\u002Fidea","kot-behemoth",[207,32,32,204,201,32],"13631969",[209,32,31,210,211,32],"13632162","project\u002Fneutral\u002Fidea","maxaf",[213,32,32,204,201,32],"13632303",[215,32,32,216,217,32],"13632516","commenter\u002Fneutral\u002Fnone","LightskinKanye",[219,32,32,204,220,32],"13632388","xtracto",[222,32,32,204,201,32],"13632408",[224,32,32,204,225,32],"13633554","illumin8",[227,32,32,204,201,32],"13633929",[229,32,32,204,230,32],"13639017","posterboy",[232,32,32,233,175,32],"13637657","other\u002Fneutral\u002Fnone",[235,32,32,204,236,32],"13632679","edblarney",[51,32,32,238,11,40],"maker\u002Fneutral\u002Fidea",[240,32,32,241,242,32],"13632720","other\u002Fneutral\u002Fidea","lobster_johnson",[244,32,32,204,245,32],"13632783","jessaustin",[247,32,32,204,242,32],"13633245",[249,32,32,204,250,32],"13633532","nolite",[252,32,32,204,175,32],"13637631",[254,32,32,233,242,32],"13633251",[256,32,32,204,257,32],"13634376","philliphaydon",[259,32,32,216,260,32],"13634811","afandian",[262,32,32,216,263,32],"13635144","akx",[265,32,32,216,260,32],"13636272",[267,32,32,216,242,32],"13636356",[269,32,32,216,260,32],"13637054",[174,32,32,233,175,32],[272,32,32,233,242,32],"13637718",[274,32,32,204,225,32],"13633536",[276,32,32,204,201,32],"13633926",[278,32,32,204,257,32],"13634382",[280,32,32,233,281,32],"13636595","noobiemcfoob",[283,32,32,241,284,32],"13647646","tjholowaychuk",[95,40,40,286,96,32],"project\u002Fcritical\u002Fdocs",[187,40,32,288,11,40],"maker\u002Fneutral\u002Fdocs",[290,40,31,291,292,32],"13630005","project\u002Fneutral\u002Fdocs","andkon",[103,40,40,286,104,32],[295,40,31,291,296,32],"13630564","stevewilhelm",[190,40,32,288,11,40],[299,40,31,291,300,32],"13630997","gbrits",[302,40,31,291,303,32],"13632326","lancefisher",[305,40,40,286,306,32],"13630285","kmfrk",[80,308,37,309,81,32],2,"project\u002Fpositive\u002Fidea",[58,308,32,311,11,40],"maker\u002Fneutral\u002Fnone",[313,308,37,309,314,32],"13629802","rpedela",[99,308,40,200,100,32],[317,308,32,318,319,32],"13630051","commenter\u002Fpositive\u002Fidea","glogla",[321,308,32,233,205,32],"13631929",[134,308,40,200,81,32],[324,308,32,233,225,32],"13633563",[160,31,32,233,161,32],[327,31,32,328,11,40],"13630760","maker\u002Fpositive\u002Fnone",[330,31,32,331,332,32],"13632539","other\u002Fpositive\u002Fnone","nstj",[334,31,32,335,336,32],"13632693","other\u002Fcritical\u002Fpricing","dajohnson89",[338,31,32,339,225,32],"13633581","other\u002Fneutral\u002Fpricing",[341,31,32,335,336,32],"13681624",[343,31,32,233,225,32],"13767507",[152,345,31,346,153,32],4,"maker\u002Fneutral\u002Fexecution",[348,345,32,346,11,40],"13629906",[350,345,40,200,351,32],"13629915","daviding",[353,345,31,354,284,32],"13630820","project\u002Fneutral\u002Fdesign",[167,37,40,200,168,32],[357,183,32,233,358,32],"13632917","_Marak_",[110,194,40,360,111,32],"project\u002Fcritical\u002Fperformance",[114,194,32,362,11,40],"maker\u002Fneutral\u002Fperformance",[145,364,31,365,146,32],8,"project\u002Fneutral\u002Fexecution",[367,364,32,311,11,40],"13634157",[369,364,32,233,370,32],"13634843","primax",[372,364,40,373,374,32],"13635074","project\u002Fcritical\u002Fexecution","bpicolo",[120,18,40,376,121,32],"project\u002Fcritical\u002Fdesign",[378,18,32,379,11,40],"13634153","maker\u002Fneutral\u002Fdesign",[87,182,37,309,88,32],[382,33,31,210,383,32],"13630105","stolendog",[385,33,32,238,11,40],"13630146",[],[388,392,397,402,407,412],{"id":389,"name":390,"points":391,"category":20},"13642662","HTML Reference",728,{"id":393,"name":394,"points":395,"category":396},"13628522","Rumuki",405,"security_privacy",{"id":398,"name":399,"points":400,"category":401},"13623852","Beehive",247,"infrastructure",{"id":403,"name":404,"points":405,"category":406},"13616751","Aragon",191,"web_saas",{"id":408,"name":409,"points":410,"category":411},"13642967","illustrated-algorithms",187,"learning_science",{"id":413,"name":414,"points":415,"category":20},"13628530","DeepUI Programming Studio",186,false,"v2",{"id":419,"name":5,"line":6,"ai":416,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":420,"top":18,"kw":421,"maker":11,"domain":10,"tier":19,"attempts":32,"strip":422,"url":9,"mix":423,"praise":76,"objection":92,"sub":21},13628952,-0.3,[24,25,26,27],"1030000000000000001133153010000311130153",[31,18,32,33],1794036552553]