Arc
Data
data pipelines
SHOW HN2021
How HN took it
2 positive, 4 neutral or asking, 3 mixed, 3 critical, out of 12 comments about the project. Thread score -0.04.
Idea: 1 for, 3 againstDocs: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Good documentation
From 1 comment
I'm reading the docs thoroughly, many excellent ideas, and I'm sure I'll be borrowing some concepts. I also want to commend you for putting in the time in developing proper…
lordgroff on HN
Interesting concepts that resemble the commenter's own work
From 1 comment
I'm in the process of doing something like this internally, at a smaller scale, and it's interesting to see that many of the concepts I've been experimenting with and playing…
lordgroff on HN
Objections
Name 'Databricks alternative' is misleading; it lacks managed hosting and low-level RDD support
From 2 comments
Yeah, agreed. I was a Databricks skeptic when I first came across it, but it's value goes a LONG way beyond just managing Spark. For example, we found that Databrick's Spark (or…
Joe8Bit on HNFor example, we do a lot of of low-level RDD Operations through databricks. From skimming the Website I feel something like this is not in the scope of this project. In the end,…
0x008 on HN
SQL-first approach may polarize users
From 1 comment
I'm in the process of doing something like this internally, at a smaller scale, and it's interesting to see that many of the concepts I've been experimenting with and playing…
lordgroff on HN
Requests
Support for wildcard or auto-detected table configuration for large JDBC migrations
From 1 comment
I like it a lot, but how large scale can it be? If I want to move whole JDBC-accessible database to warehouse or lakehouse (like Postgres or Oracle to S3 with Iceberg or…
glogla on HN
Incremental loading based on timestamp fields
From 1 comment
Can you specify between complete pulls of the source vs incremental based on a timestamp field?
psing on HN
Questions
How does it compare to dbt
From 1 comment
I'm curious how this compares to www.getdbt.com which seems to target a similar audience (technical analysts wanting to do ETL) with a similar approach (SQL first).
marcinzm on HN
Does it expose the full Spark API
From 1 comment
The idea makes sense, but Databricks exposes the complete Spark API, is that true for this project as well? Spark is a lot more than Spark SQL.
0x008 on HN
Alternatives named
Databricks
From 1 comment
Yeah, agreed. I was a Databricks skeptic when I first came across it, but it's value goes a LONG way beyond just managing Spark. For example, we found that Databrick's Spark (or…
Joe8Bit on HN
dbt
From 1 comment
I'm curious how this compares to www.getdbt.com which seems to target a similar audience (technical analysts wanting to do ETL) with a similar approach (SQL first).
marcinzm on HN
Kubeflow
From 1 comment
Databricks writes some good tools, but it can get pretty expensive. Kubeflow has been evolving well and is gaining lots of traction. It's pretty neat from my experience so far.
bostonsre on HN