[{"data":1,"prerenderedAt":317},["ShallowReactive",2],{"project-36214393":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":34,"aspects":35,"strip":40,"post_text":43,"what":44,"said":86,"maker_in_thread":188,"shape":201,"grid":205,"attempts":279,"same_week":280,"maker_shows":41,"maker_hidden":309,"reader_version":310,"card":311},"36214393","Arroyo","Write SQL on streaming data","title","Show HN: Arroyo – Write SQL on streaming data","https:\u002F\u002Fgithub.com\u002FArroyoSystems\u002Farroyo","github.com","necubi",1686065488,2023,115,32,552,3109,18,"notable","data","stream processing","other",[21,24,25,26,27],"streaming sql","flink alternative","watermarks","rust stream processing","curious",0.017,{"positive":31,"neutral":32,"mixed":33,"critical":33},2,10,1,14,{"design":36,"performance":37,"docs":39},{"positive":33,"critical":33},{"positive":33,"critical":38},0,{"positive":38,"critical":38},[41,38,41,38,41,42,41,41,38,38,38,41,33,38,41,41,38,38,38,38,41,38,38,42,38,41,38,38,41,38,38,38],3,5,"Hey HN,\n\nArroyo is a modern, open-source stream processing engine, that lets anyone write complex queries on event streams just by writing SQL—windowing, aggregating, and joining events with sub-second latency.\n\nToday data processing typically happens in batch data warehouses like BigQuery and Snowflake despite the fact that most of the data is coming in as streams. Data teams have to build complex orchestration systems to handle late-arriving data and job failures while trying to  minimize latency. Stream processing offers an alternative approach, where the query is compiled into a streaming ",{"problem":45,"audience":51,"pricing":56,"motivation":57,"time_to_build":56,"stack":61,"openness":84,"stage":85,"team_size":85},{"value":46,"source":47,"evidence":48},"Building correct, reliable real-time pipelines on event streams is hard and costly to operate","maker",[49],{"id":50},"post",{"value":52,"source":53,"evidence":54},"Data teams and infra teams building streaming pipelines","inferred",[55],{"id":50},null,{"value":58,"source":47,"evidence":59},"Makers spent five years building real-time platforms at Lyft and Splunk and saw pipelines were hard to build and operate",[60],{"id":50},[62,70,74,80],{"value":63,"source":64,"evidence":65},"Rust","commenter",[66],{"id":67,"by":68,"text":69},"36224439","trevyn","No, Arroyo is written is Rust, but the web front-end is React.",{"value":71,"source":64,"evidence":72},"React (web console)",[73],{"id":67,"by":68,"text":69},{"value":75,"source":47,"evidence":76},"Kubernetes",[77],{"id":78,"by":11,"text":79},"36229223","We're big fans of Nomad! It's what we use for scheduling in our cloud platform, due to its simplicity and scheduling speed. Although we also have great support for Kubernetes as…",{"value":81,"source":47,"evidence":82},"Nomad",[83],{"id":78,"by":11,"text":79},"open_source","unknown",{"praise":87,"objections":102,"requests":117,"questions":129,"alternatives":158},[88,95],{"value":89,"source":64,"evidence":90},"Good to see non-JVM streaming projects and SQL-first API",[91],{"id":92,"by":93,"text":94},"36226521","benjaminwootton","Between Flink, Spark and KSQL, streaming is very JVM centric. It is nice to see more non JVM projects emerge. I am not sure about your premise that the operations side is…",{"value":96,"source":64,"evidence":97},"Support for Nomad is welcome",[98],{"id":99,"by":100,"text":101},"36229055","httgp","This looks great, and it’s very cool that it recommends Nomad to run it in production. I wish more products would support (or at least document how to run on) Nomad.",[103,110],{"value":104,"source":64,"evidence":105},"Late-arriving events are silently dropped by watermarks",[106],{"id":107,"by":108,"text":109},"36224906","jsty","In the watermarks documentation it mentions that events arriving after the watermark are dropped. Are there any plans to make this configurable (to disable dropping or trigger…",{"value":111,"source":64,"evidence":112},"Watermarks appear to be fixed length",[113],{"id":114,"by":115,"text":116},"36224813","thom","Unbounded streams, but with watermarks (which right now seem fixed length?): https:\u002F\u002Fdoc.arroyo.dev\u002Fconcepts#watermarks Also works based on fixed, pre-built pipelines. This is…",[118,122],{"value":119,"source":64,"evidence":120},"Configurable handling of late-arriving data and alerts",[121],{"id":107,"by":108,"text":109},{"value":123,"source":64,"evidence":124},"Nexmark or performance benchmarks against Flink and DBSP",[125],{"id":126,"by":127,"text":128},"36225649","sorenbs","As an aside - we are evaluating different streaming engines to power data projections for Prisma.io. Excited to see support for Debezium coming in v0.4.0. Would be interesting to…",[130,137,144,151],{"value":131,"source":64,"evidence":132},"How does it compare to Flink in pros and cons and resource use",[133],{"id":134,"by":135,"text":136},"36237657","gunnarmorling","Very interesting project, Arroyo has been on my watch list for a while! How would you say does Arroyo compare to Apache Flink, i.e. what are pros and cons? For instance, given…",{"value":138,"source":64,"evidence":139},"How does it compare to Materialize",[140],{"id":141,"by":142,"text":143},"36225898","dangoodmanUT","Would love to know how you look at tools list Materialize in comparison",{"value":145,"source":64,"evidence":146},"Is it an alternative to Confluent KSQL",[147],{"id":148,"by":149,"text":150},"36224507","laurensr","Would Arroyo be an alternative to Confluent KSQL?",{"value":152,"source":64,"evidence":153},"How does feature parity compare with Tinybird",[154],{"id":155,"by":156,"text":157},"36226225","KRAKRISMOTT","Very exciting, how is feature parity with tinybird? https:\u002F\u002Fwww.tinybird.co\u002F",[159,163,167,173,180,184],{"value":160,"source":64,"evidence":161},"Apache Flink",[162],{"id":134,"by":135,"text":136},{"value":164,"source":64,"evidence":165},"Confluent KSQL",[166],{"id":148,"by":149,"text":150},{"value":168,"source":64,"evidence":169},"DBSP",[170],{"id":171,"by":127,"text":172},"36225507","This is a really exciting project! I recently learned about https:\u002F\u002Fgithub.com\u002Fvmware\u002Fdatabase-stream-processor which builds on a new theoretical foundation and claims to be 9x…",{"value":174,"source":64,"evidence":175},"Benthos",[176],{"id":177,"by":178,"text":179},"36232516","yevpats","Looks cool. What is the difference between this tools and benthos (https:\u002F\u002Fwww.benthos.dev\u002F)?",{"value":181,"source":64,"evidence":182},"Materialize",[183],{"id":141,"by":142,"text":143},{"value":185,"source":64,"evidence":186},"Tinybird",[187],{"id":155,"by":156,"text":157},{"comments":189,"answered":190,"key":191},8,4,[192,195,198],{"id":193,"text":194},"36228905","In SQL we currently support specifying watermarks based on SQL expressions, so it can be a bit more expressive than just the basic fixed-delay watermark:…",{"id":196,"text":197},"36233919","There's a spectrum of streaming systems from simple, stateless processors to complex, stateful distributed processors. I'm not very familiar with Benthos, but it appears to be on…",{"id":199,"text":200},"36229235","That'd be great! We have versions of most of the nexmark queries and some internal benchmark vs Flink, we'd love to help if you're interested in benchmarking against more…",{"commenters":202,"max_depth":42,"maker_replies":189,"first_comment_mins":203,"lifespan_hours":204},20,985,164,[206,208,210,212,214,215,217,221,224,225,229,231,233,235,237,238,240,244,247,249,251,252,256,258,261,263,264,267,269,272,275,278],[114,38,41,207,115,38],"project\u002Fneutral\u002Fdesign",[193,38,38,209,11,33],"maker\u002Fneutral\u002Fdesign",[177,33,41,211,178,38],"project\u002Fneutral\u002Fnone",[196,33,38,213,11,33],"maker\u002Fneutral\u002Fnone",[171,31,41,211,127,38],[126,31,42,216,127,38],"project\u002Fpositive\u002Fperformance",[218,31,41,219,220,38],"36226030","maker\u002Fneutral\u002Fperformance","jnewhouse",[222,31,41,223,127,38],"36227521","project\u002Fneutral\u002Fperformance",[199,31,38,219,11,33],[226,31,38,227,228,38],"36229789","other\u002Fneutral\u002Fnone","ryzhyk",[230,31,38,227,127,38],"36308958",[92,41,41,232,93,38],"project\u002Fmixed\u002Fdocs",[107,190,33,234,108,38],"project\u002Fcritical\u002Fdesign",[236,190,38,209,11,33],"36228940",[134,42,41,223,135,38],[141,239,41,211,142,38],6,[241,242,38,227,243,38],"36227820",7,"fasteo",[245,242,38,227,246,38],"36228783","callmeed",[248,242,38,227,243,38],"36229085",[250,242,38,213,11,33],"36229296",[155,189,41,211,156,38],[253,189,38,254,255,38],"36226649","commenter\u002Fneutral\u002Fnone","_peregrine_",[257,189,38,209,11,33],"36229193",[99,259,42,260,100,38],9,"project\u002Fpositive\u002Fdesign",[78,259,38,262,11,33],"maker\u002Fpositive\u002Fnone",[148,32,41,211,149,38],[265,32,38,254,266,38],"36225377","kohlerm",[268,32,38,213,11,33],"36229050",[270,271,41,207,68,38],"36224136",11,[273,271,38,254,274,38],"36224177","ukuina",[276,271,38,254,277,38],"36224235","tmikaeld",[67,271,38,254,68,38],[],[281,285,290,295,299,304],{"id":282,"name":283,"points":284,"category":20},"36186185","IPFS",704,{"id":286,"name":287,"points":288,"category":289},"36148807","I made CSS Pro",505,"dev_tools",{"id":291,"name":292,"points":293,"category":294},"36154705","HN Resume to Jobs",214,"productivity",{"id":296,"name":297,"points":298,"category":289},"36164503","Mercury",131,{"id":300,"name":301,"points":302,"category":303},"36151225","StonksGPT",116,"finance_crypto",{"id":305,"name":306,"points":307,"category":308},"36232348","Ezno",113,"languages",false,"v2",{"id":312,"name":5,"line":6,"ai":309,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":313,"top":18,"kw":314,"maker":11,"domain":10,"tier":19,"attempts":38,"strip":315,"url":9,"mix":316,"praise":89,"objection":104,"sub":21},36214393,0.02,[21,24,25,26,27],"30303533000310330000300503003000",[31,32,33,33],1794036636569]