[{"data":1,"prerenderedAt":310},["ShallowReactive",2],{"project-26573930":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":27,"sentiment":28,"stance_mix":29,"scored_comments":33,"aspects":34,"strip":41,"post_text":43,"what":44,"said":84,"maker_in_thread":161,"shape":171,"grid":174,"attempts":267,"same_week":272,"maker_shows":30,"maker_hidden":302,"reader_version":303,"card":304},"26573930","Arc","An open-source Databricks alternative","title","Show HN: Arc, an open-source Databricks alternative","https:\u002F\u002Farc.tripl.ai\u002F","arc.tripl.ai","seddonm1",1616625643,2021,175,36,272,2533,11,"notable","data","data pipelines","library",[24,25,21,26],"etl pipelines","apache spark","jupyter notebooks","mixed",-0.042,{"positive":30,"neutral":31,"mixed":32,"critical":32},2,4,3,12,{"idea":35,"design":37,"docs":39,"execution":40},{"positive":36,"critical":32},1,{"positive":38,"critical":38},0,{"positive":36,"critical":38},{"positive":38,"critical":38},[38,38,38,38,32,38,38,38,38,32,38,42,38,36,38,36,38,32,32,38,42,38,32,38,32,38,36,38,32,38,38,38,38,38,38,38],5,null,{"problem":45,"audience":52,"pricing":43,"motivation":59,"time_to_build":43,"stack":63,"openness":82,"stage":83,"team_size":83},{"value":46,"source":47,"evidence":48},"Building reliable ETL data pipelines without much developer support","maker",[49],{"id":50,"by":11,"text":51},"26573931","After being frustrated with building 'traditional' ETL (Extract-Transform-Load) pipelines - and around the same time as the famous 'Engineers Shouldn’t Write ETL' blog post - we…",{"value":53,"source":47,"evidence":54},"Technical business analysts and data scientists",[55,56],{"id":50,"by":11,"text":51},{"id":57,"by":11,"text":58},"26575133","Cool! Most of the time we run Spark as a single-node (i.e. --master local[*]) as now we can easily utilise large nodes like 128 core, 512GB and Spark does scale vertically well…",{"value":60,"source":47,"evidence":61},"Frustration building traditional ETL pipelines",[62],{"id":50,"by":11,"text":51},[64,68,72,76],{"value":65,"source":47,"evidence":66},"Apache Spark",[67],{"id":50,"by":11,"text":51},{"value":69,"source":47,"evidence":70},"Jupyter Notebooks",[71],{"id":50,"by":11,"text":51},{"value":73,"source":47,"evidence":74},"Scala",[75],{"id":57,"by":11,"text":58},{"value":77,"source":47,"evidence":78},"Docker",[79],{"id":80,"by":11,"text":81},"26575198","We provide multiple Docker images (https:\u002F\u002Fgithub.com\u002Forgs\u002Ftripl-ai\u002Fpackages) that make the Spark deployment easy: - arc-jupyter: allows you to develop on your local machine (and…","open_source","unknown",{"praise":85,"objections":100,"requests":116,"questions":131,"alternatives":145},[86,94],{"value":87,"source":88,"evidence":89},"Good documentation","commenter",[90],{"id":91,"by":92,"text":93},"26575278","lordgroff","I'm reading the docs thoroughly, many excellent ideas, and I'm sure I'll be borrowing some concepts. I also want to commend you for putting in the time in developing proper…",{"value":95,"source":88,"evidence":96},"Interesting concepts that resemble the commenter's own work",[97],{"id":98,"by":92,"text":99},"26575044","I'm in the process of doing something like this internally, at a smaller scale, and it's interesting to see that many of the concepts I've been experimenting with and playing…",[101,112],{"value":102,"source":88,"evidence":103},"Name 'Databricks alternative' is misleading; it lacks managed hosting and low-level RDD support",[104,108],{"id":105,"by":106,"text":107},"26577939","Joe8Bit","Yeah, agreed. I was a Databricks skeptic when I first came across it, but it's value goes a LONG way beyond just managing Spark. For example, we found that Databrick's Spark (or…",{"id":109,"by":110,"text":111},"26577697","0x008","For example, we do a lot of of low-level RDD Operations through databricks. From skimming the Website I feel something like this is not in the scope of this project. In the end,…",{"value":113,"source":88,"evidence":114},"SQL-first approach may polarize users",[115],{"id":98,"by":92,"text":99},[117,124],{"value":118,"source":88,"evidence":119},"Support for wildcard or auto-detected table configuration for large JDBC migrations",[120],{"id":121,"by":122,"text":123},"26674230","glogla","I like it a lot, but how large scale can it be? If I want to move whole JDBC-accessible database to warehouse or lakehouse (like Postgres or Oracle to S3 with Iceberg or…",{"value":125,"source":88,"evidence":126},"Incremental loading based on timestamp fields",[127],{"id":128,"by":129,"text":130},"26579160","psing","Can you specify between complete pulls of the source vs incremental based on a timestamp field?",[132,139],{"value":133,"source":88,"evidence":134},"How does it compare to dbt",[135],{"id":136,"by":137,"text":138},"26575179","marcinzm","I'm curious how this compares to www.getdbt.com which seems to target a similar audience (technical analysts wanting to do ETL) with a similar approach (SQL first).",{"value":140,"source":88,"evidence":141},"Does it expose the full Spark API",[142],{"id":143,"by":110,"text":144},"26577509","The idea makes sense, but Databricks exposes the complete Spark API, is that true for this project as well? Spark is a lot more than Spark SQL.",[146,150,154],{"value":147,"source":88,"evidence":148},"Databricks",[149],{"id":105,"by":106,"text":107},{"value":151,"source":88,"evidence":152},"dbt",[153],{"id":136,"by":137,"text":138},{"value":155,"source":88,"evidence":156},"Kubeflow",[157],{"id":158,"by":159,"text":160},"26580250","bostonsre","Databricks writes some good tools, but it can get pretty expensive. Kubeflow has been evolving well and is gaining lots of traction. It's pretty neat from my experience so far.",{"comments":162,"answered":42,"key":163},14,[164,165,168],{"id":50,"text":51},{"id":166,"text":167},"26577254","Looks good. It is nice to see how much influence the 'Engineers Shouldn't Write ETL' post had! With Apache Arrow (https:\u002F\u002Farrow.apache.org\u002F) I think the future looks very bright…",{"id":169,"text":170},"26584731","The better statement would be that this facilitates the development of idempotent jobs and aims to minimise side-effects.",{"commenters":172,"max_depth":31,"maker_replies":162,"first_comment_mins":38,"lifespan_hours":173},20,212,[175,177,181,183,187,190,191,194,198,200,202,204,206,209,213,215,216,218,220,223,225,229,231,233,235,237,239,240,243,245,247,252,255,259,261,264],[50,38,38,176,11,36],"project\u002Fneutral\u002Fidea",[178,38,38,179,180,38],"26577220","commenter\u002Fneutral\u002Fidea","armanboyaci",[166,38,38,182,11,36],"commenter\u002Fpositive\u002Fperformance",[184,38,38,185,186,38],"26577456","other\u002Fneutral\u002Fnone","moreati",[188,38,32,176,189,38],"26578609","tuyguntn",[169,38,38,176,11,36],[192,38,38,185,193,38],"26579757","johnx123-up",[195,38,38,196,197,38],"26580442","commenter\u002Fneutral\u002Fnone","tyingq",[199,38,38,196,193,38],"26589033",[98,36,32,201,92,38],"project\u002Fmixed\u002Fdesign",[57,36,38,203,11,36],"project\u002Fpositive\u002Fdesign",[91,36,42,205,92,38],"maker\u002Fpositive\u002Fdocs",[207,36,38,208,11,36],"26575326","maker\u002Fneutral\u002Fdocs",[210,30,36,211,212,38],"26575067","project\u002Fcritical\u002Fidea","superyesh",[80,30,38,214,11,36],"project\u002Fneutral\u002Fexecution",[105,30,36,211,106,38],[158,30,38,217,159,38],"other\u002Fneutral\u002Fpricing",[121,32,32,219,122,38],"project\u002Fmixed\u002Fidea",[221,31,32,219,222,38],"26577189","xupybd",[224,31,38,203,11,36],"26577225",[226,42,42,227,228,38],"26574802","project\u002Fpositive\u002Fidea","crimsoneer",[230,42,38,227,11,36],"26574903",[136,232,32,176,137,38],6,[234,232,38,176,11,36],"26575234",[143,236,32,176,110,38],7,[238,236,38,214,11,36],"26577563",[109,236,36,211,110,38],[241,236,38,242,11,36],"26577773","maker\u002Fneutral\u002Fidea",[128,244,32,214,129,38],8,[246,244,38,214,11,36],"26584767",[248,249,38,250,251,38],"26575031",9,"other\u002Fpositive\u002Fnone","justosophy",[253,249,38,254,11,36],"26575247","maker\u002Fneutral\u002Fnone",[256,257,38,185,258,38],"26575304",10,"robobro",[260,257,38,254,11,36],"26575428",[262,18,38,185,263,38],"26576891","ozten",[265,18,38,185,266,38],"26578919","user-the-name",[268,271],{"id":269,"t":270,"points":232,"comments":36},"20580163",1564619863,{"id":4,"t":12,"points":14,"comments":15},[273,278,283,288,292,297],{"id":274,"name":275,"points":276,"category":277},"26568014","Practical Go Lessons",919,"languages",{"id":279,"name":280,"points":281,"category":282},"26503572","Sorted CSS Colors",559,"dev_tools",{"id":284,"name":285,"points":286,"category":287},"26537734","compiler.org",219,"learning_science",{"id":289,"name":290,"points":291,"category":282},"26554451","PyWebIO",200,{"id":293,"name":294,"points":295,"category":296},"26503715","LitiHolo",180,"hardware",{"id":298,"name":299,"points":300,"category":301},"26527268","Papercups",148,"web_saas",false,"v2",{"id":305,"name":5,"line":6,"ai":302,"cat":20,"year":13,"points":14,"comments":15,"rec":27,"s":306,"top":18,"kw":307,"maker":11,"domain":10,"tier":19,"attempts":30,"strip":308,"url":9,"mix":309,"praise":87,"objection":102,"sub":21},26573930,-0.04,[24,25,21,26],"000030000305010103305030301030000000",[30,31,32,32],1794036604889]