Goodreads Data Pipeline
Data
etl pipeline
SHOW HN2020
213points
65comments
#252of 3,028 in 2020
Top 9%of 2020 by points
How HN took it
CuriousMostly questions and neutral comments
2 positive, 7 neutral or asking, 1 mixed, 2 critical, out of 12 comments about the project. Thread score -0.04.
Idea: 2 for, 0 againstExecution: 0 for, 1 againstPerformance: 0 for, 1 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Interesting data pipeline to learn from
From 2 comments
Beyond just data replication/archival purposes, it seems you can use the this to run analysis against Goodreads entire public dataset. This is much more efficient than using…
mrlatinos on HNSuper interesting. Surely there are some business cases of how someone could use this data for good (?) For example someone could show the disparity between a New York times…
krmmalik on HN
Objections
Architecture seems overly complex for the data volume
From 2 comments
the architecture also seems pretty complex - i am wondering at what level of requirements or data complexity people should consider something like this, as opposed to running a…
swyx on HN3 xlarge EMR instances sounds like overkill assuming a volume of around 11gb every 10 mins. Using postgres COPY I've loaded larger files into tables in seconds. Semi complex…
wefarrell on HN
Managed tools like dbt and Snowflake would be simpler
From 1 comment
Nice overview. I'd suggest to anyone interested in doing something like this to also consider the much simpler managed approach of using tools like: * Stitch [etl/elt] *…
sails on HN
Requests
Cost details of the AWS setup
From 1 comment
Interesting, would like some detail on the cost of this ETL setup on AWS, unfortunately I can't see anything on this from the project page.
kgraves on HN
Volume and sizing metrics
From 1 comment
I read through the Readme but didn’t see any volume or velocity figures (I saw the entity count, but what does this translate into w/r/t bytes?) Anyone run this who could comment…
TuringNYC on HN
More context on the project's purpose
From 1 comment
This is potentially quite interesting to me we are having conversations on/off at work about data reporting, visualising etc., which is leading me to pay attention to related…
adam-_- on HN
Questions
Does it scrape shelf tags beyond the API?
From 1 comment
Is this limited strictly to the GoodReads API or does it pull in more interesting data like the shelf-tags? When I did https://www.gwern.net/GoodReads the other month, I had to…
gwern on HN
How is backfilling and recovery handled?
From 1 comment
Nice - You can use UNION ALL instead of UNION in your query at the end if you're confident the datasets don't overlap. Query is less expensive. I'm also curious what the…
bilater on HN
Alternatives named
LibraryThing
From 2 comments
Have you tried looking? There's LibraryThing and a couple others. I don't think there's much value left on the table in the niche, though. Kindles have first-class Goodreads sync…
hombre_fatal on HNLibraryThing if you don't mind the ugly website
FalconSensei on HN