[{"data":1,"prerenderedAt":369},["ShallowReactive",2],{"project-22434200":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":27,"sentiment":28,"stance_mix":29,"scored_comments":33,"aspects":34,"strip":42,"post_text":45,"what":46,"said":71,"maker_in_thread":146,"shape":151,"grid":156,"attempts":332,"same_week":333,"maker_shows":32,"maker_hidden":361,"reader_version":362,"card":363},"22434200","Goodreads Data Pipeline","Thanks for the tip.","maker","Show HN: Goodreads Data Pipeline","https:\u002F\u002Fgithub.com\u002Fsan089\u002Fgoodreads_etl_pipeline","github.com","san089",1582818081,2020,213,65,252,3028,9,"notable","data","etl pipeline","other",[24,21,25,26],"goodreads","data lake","aws emr","curious",-0.038,{"positive":30,"neutral":31,"mixed":32,"critical":30},2,7,1,12,{"docs":35,"idea":37,"design":38,"execution":39,"performance":40,"pricing":41},{"positive":36,"critical":36},0,{"positive":30,"critical":36},{"positive":36,"critical":36},{"positive":36,"critical":32},{"positive":36,"critical":32},{"positive":36,"critical":36},[36,36,36,36,36,36,43,44,36,36,36,36,36,36,36,36,36,36,36,36,36,36,36,36,36,43,36,44,32,32,36,36,43,36,36,36,36,36,36,43],3,5,null,{"problem":45,"audience":45,"pricing":45,"motivation":45,"time_to_build":45,"stack":47,"openness":70,"stage":70,"team_size":70},[48,56,63],{"value":49,"source":50,"evidence":51},"AWS EMR","commenter",[52],{"id":53,"by":54,"text":55},"22439754","wefarrell","3 xlarge EMR instances sounds like overkill assuming a volume of around 11gb every 10 mins. Using postgres COPY I've loaded larger files into tables in seconds. Semi complex…",{"value":57,"source":50,"evidence":58},"Redshift",[59],{"id":60,"by":61,"text":62},"22440924","sails","Nice overview. I'd suggest to anyone interested in doing something like this to also consider the much simpler managed approach of using tools like: * Stitch [etl\u002Felt] *…",{"value":64,"source":50,"evidence":65},"Airflow",[66],{"id":67,"by":68,"text":69},"22434707","prions","Really similar to the pipelines that I engineer\u002Fmanage at my current company. Although we have our Airflow on kubernetes. One optimization though is separating your loading tasks…","unknown",{"praise":72,"objections":84,"requests":97,"questions":119,"alternatives":134},[73],{"value":74,"source":50,"evidence":75},"Interesting data pipeline to learn from",[76,80],{"id":77,"by":78,"text":79},"22434682","mrlatinos","Beyond just data replication\u002Farchival purposes, it seems you can use the this to run analysis against Goodreads entire public dataset. This is much more efficient than using…",{"id":81,"by":82,"text":83},"22434524","krmmalik","Super interesting. Surely there are some business cases of how someone could use this data for good (?) For example someone could show the disparity between a New York times…",[85,93],{"value":86,"source":50,"evidence":87},"Architecture seems overly complex for the data volume",[88,92],{"id":89,"by":90,"text":91},"22439057","swyx","the architecture also seems pretty complex - i am wondering at what level of requirements or data complexity people should consider something like this, as opposed to running a…",{"id":53,"by":54,"text":55},{"value":94,"source":50,"evidence":95},"Managed tools like dbt and Snowflake would be simpler",[96],{"id":60,"by":61,"text":62},[98,105,112],{"value":99,"source":50,"evidence":100},"Cost details of the AWS setup",[101],{"id":102,"by":103,"text":104},"22434627","kgraves","Interesting, would like some detail on the cost of this ETL setup on AWS, unfortunately I can't see anything on this from the project page.",{"value":106,"source":50,"evidence":107},"Volume and sizing metrics",[108],{"id":109,"by":110,"text":111},"22440009","TuringNYC","I read through the Readme but didn’t see any volume or velocity figures (I saw the entity count, but what does this translate into w\u002Fr\u002Ft bytes?) Anyone run this who could comment…",{"value":113,"source":50,"evidence":114},"More context on the project's purpose",[115],{"id":116,"by":117,"text":118},"22434471","adam-_-","This is potentially quite interesting to me we are having conversations on\u002Foff at work about data reporting, visualising etc., which is leading me to pay attention to related…",[120,127],{"value":121,"source":50,"evidence":122},"Does it scrape shelf tags beyond the API?",[123],{"id":124,"by":125,"text":126},"22437230","gwern","Is this limited strictly to the GoodReads API or does it pull in more interesting data like the shelf-tags? When I did https:\u002F\u002Fwww.gwern.net\u002FGoodReads the other month, I had to…",{"value":128,"source":50,"evidence":129},"How is backfilling and recovery handled?",[130],{"id":131,"by":132,"text":133},"22435903","bilater","Nice - You can use UNION ALL instead of UNION in your query at the end if you're confident the datasets don't overlap. Query is less expensive. I'm also curious what the…",[135],{"value":136,"source":50,"evidence":137},"LibraryThing",[138,142],{"id":139,"by":140,"text":141},"22435571","hombre_fatal","Have you tried looking? There's LibraryThing and a couple others. I don't think there's much value left on the table in the niche, though. Kindles have first-class Goodreads sync…",{"id":143,"by":144,"text":145},"22438847","FalconSensei","LibraryThing if you don't mind the ugly website",{"comments":32,"answered":36,"key":147},[148],{"id":149,"text":150},"22440039","Thanks for the tip. I actually thought about such separation, but it was too late to make such changes. I already laid down the architecture till that time. But you made a good…",{"commenters":152,"max_depth":153,"maker_replies":32,"first_comment_mins":154,"lifespan_hours":155},50,6,31,178,[157,161,164,168,171,174,177,179,181,184,187,189,191,193,196,199,202,205,207,210,213,215,218,222,225,227,230,231,235,237,239,242,245,248,251,253,255,259,262,265,267,269,271,273,274,275,277,279,282,285,288,290,292,295,299,302,305,308,311,314,319,320,323,326,328],[158,36,36,159,160,36],"22435136","other\u002Fneutral\u002Fnone","jonluca",[162,36,36,159,163,36],"22436735","spullara",[165,36,36,166,167,36],"22437675","commenter\u002Fneutral\u002Fnone","traverseda",[169,36,36,166,170,36],"22438536","idkris",[172,36,36,166,173,36],"22438607","jaredmosley",[175,36,36,166,176,36],"22439262","nevster",[178,36,36,166,144,36],"22438811",[180,36,36,166,167,36],"22439778",[182,36,36,166,183,36],"22439945","danielg6",[185,36,36,166,186,36],"22438251","richie5um",[116,32,43,188,117,36],"maker\u002Fmixed\u002Fdocs",[77,32,44,190,78,36],"project\u002Fpositive\u002Fidea",[89,32,43,192,90,36],"project\u002Fneutral\u002Fdesign",[194,30,36,159,195,36],"22435203","habosa",[197,30,36,159,198,36],"22435406","drusepth",[200,30,36,166,201,36],"22435540","trollied",[203,30,36,166,204,36],"22437000","dewey",[206,30,36,166,140,36],"22435592",[208,30,36,166,209,36],"22438799","maw",[211,30,36,159,212,36],"22435633","barkingcat",[214,30,36,159,163,36],"22436722",[216,30,36,159,217,36],"22436916","jjeaff",[219,30,36,220,221,36],"22440706","other\u002Fneutral\u002Fidea","johnmaguire2013",[223,30,36,159,224,36],"22442868","Kihashi",[226,30,36,220,201,36],"22435531",[228,30,36,220,229,36],"22437541","mmanfrin",[139,30,36,166,140,36],[232,30,36,233,234,36],"22436386","other\u002Fcritical\u002Fnone","dlsso",[236,30,36,166,140,36],"22437297",[238,30,36,166,144,36],"22454949",[240,30,36,241,144,36],"22438839","other\u002Fcritical\u002Fdesign",[243,30,36,220,244,36],"22435674","101008",[246,30,36,159,247,36],"22436297","spillguard",[249,30,36,166,250,36],"22437083","kirubakaran",[252,30,36,166,247,36],"22439730",[254,30,36,159,144,36],"22438825",[256,30,36,257,258,36],"22437076","other\u002Fneutral\u002Fdesign","sidthekid",[260,30,36,159,261,36],"22437780","grimgrin",[263,30,36,159,264,36],"22437969","aaron-santos",[266,30,36,166,144,36],"22438829",[131,43,43,268,132,36],"project\u002Fneutral\u002Fexecution",[67,270,43,192,68,36],4,[149,270,36,272,11,32],"maker\u002Fneutral\u002Fdesign",[81,44,44,190,82,36],[124,153,43,268,125,36],[53,31,32,276,54,36],"project\u002Fcritical\u002Fperformance",[109,31,43,278,110,36],"project\u002Fneutral\u002Fperformance",[60,280,32,281,61,36],8,"project\u002Fcritical\u002Fexecution",[283,18,36,233,284,36],"22439912","thrower123",[286,18,36,159,287,36],"22442191","slightwinder",[289,18,36,159,144,36],"22445630",[291,18,36,159,284,36],"22500345",[102,293,43,294,103,36],10,"project\u002Fneutral\u002Fpricing",[296,293,36,297,298,36],"22434948","other\u002Fneutral\u002Fpricing","jmedefind",[300,293,36,297,301,36],"22435016","dswalter",[303,293,36,166,304,36],"22435409","augmachina",[306,293,36,166,307,36],"22436002","devcpp",[309,293,36,159,310,36],"22435327","danial",[312,293,36,166,313,36],"22439601","chrisweekly",[315,316,36,317,318,36],"22435477",11,"other\u002Fcritical\u002Fprivacy","skandl",[143,316,36,159,144,36],[321,33,36,159,322,36],"22437001","ldng",[324,33,36,159,325,36],"22437193","bdibs",[327,33,36,166,322,36],"22439792",[329,330,43,331,90,36],"22439052",13,"project\u002Fneutral\u002Fidea",[],[334,339,344,349,354,358],{"id":335,"name":336,"points":337,"category":338},"22441289","Profit Hunt",528,"web_saas",{"id":340,"name":341,"points":342,"category":343},"22483436","Monitoror",295,"infrastructure",{"id":345,"name":346,"points":347,"category":348},"22451568","Edtr.io",234,"dev_tools",{"id":350,"name":351,"points":352,"category":353},"22464353","Support Examples",198,"productivity",{"id":355,"name":356,"points":357,"category":348},"22476514","InventHub",185,{"id":359,"name":360,"points":155,"category":348},"22465675","Webcodesk",false,"v2",{"id":364,"name":5,"line":6,"ai":361,"cat":20,"year":13,"points":14,"comments":15,"rec":27,"s":365,"top":18,"kw":366,"maker":11,"domain":10,"tier":19,"attempts":36,"strip":367,"url":9,"mix":368,"praise":74,"objection":86,"sub":21},22434200,-0.04,[24,21,25,26],"0000003500000000000000000305110030000003",[30,31,32,30],1793991004186]