[{"data":1,"prerenderedAt":318},["ShallowReactive",2],{"project-41945458":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":36,"aspects":37,"strip":45,"post_text":47,"what":48,"said":86,"maker_in_thread":171,"shape":181,"grid":185,"attempts":279,"same_week":280,"maker_shows":172,"maker_hidden":310,"reader_version":311,"card":312},"41945458","Trench","Open-source analytics infrastructure","title","Show HN: Trench – Open-source analytics infrastructure","https:\u002F\u002Fgithub.com\u002FFrigadeHQ\u002Ftrench","github.com","pancomplex",1729865265,2024,155,38,543,3515,16,"notable","data","event analytics backend","api",[24,25,26,27,28],"event tracking","clickhouse","kafka","analytics backend","segment compatible","curious",-0.182,{"positive":32,"neutral":33,"mixed":34,"critical":35},3,11,0,7,21,{"execution":38,"pricing":40,"performance":42,"idea":43,"docs":44},{"positive":39,"critical":39},1,{"positive":34,"critical":41},2,{"positive":34,"critical":34},{"positive":39,"critical":32},{"positive":34,"critical":34},[46,34,32,32,34,32,39,39,34,34,32,34,32,34,39,32,39,34,34,46,34,32,34,46,32,34,34,39,39,34,32,34,32,34,34,32,34,39],5,"Hey HN! I want to share a new open source project I've been working on called Trench (https:\u002F\u002Ftrench.dev). It's open source analytics infrastructure for tracking events, page views, and identifying users, and it's built on top of ClickHouse and Kafka.\n\nhttps:\u002F\u002Fgithub.com\u002Ffrigadehq\u002Ftrench\n\nI built Trench because the Postgres table we used for tracking events at our startup (http:\u002F\u002Ffrigade.com\u002F) was getting expensive and becoming a performance bottleneck as we scaled to millions of end users.\n\nMany companies run into the same problem as us (e.g. Stripe, Heroku: https:\u002F\u002Fbrandur.org\u002Ffragments\u002Feven",{"problem":49,"audience":55,"pricing":63,"motivation":64,"time_to_build":63,"stack":68,"openness":83,"stage":84,"team_size":85},{"value":50,"source":51,"evidence":52},"Event tables in relational databases become expensive and slow as applications scale","maker",[53],{"id":54},"post",{"value":56,"source":57,"evidence":58},"Startups and teams with high-volume event tracking needs","inferred",[59,60],{"id":54},{"id":61,"by":11,"text":62},"41974909","I looked around, but all the open source analytics projects I could find were bloated with all kinds of UI and unnecessary code paths. They also all seemed to use row-based RDMS…",null,{"value":65,"source":51,"evidence":66},"Postgres events table at their startup became expensive and a performance bottleneck",[67],{"id":54},[69,73,77],{"value":70,"source":51,"evidence":71},"ClickHouse",[72],{"id":54},{"value":74,"source":51,"evidence":75},"Kafka",[76],{"id":54},{"value":78,"source":51,"evidence":79},"Node.js",[80],{"id":81,"by":11,"text":82},"41975168","Thank you! 1) All data is partitioned based on the \"instanceId\" of events (see `instanceId` here: https:\u002F\u002Fdocs.trench.dev\u002Fapi-reference\u002Fevents-create). Instance IDs are typically…","open_source","launched","company",{"praise":87,"objections":100,"requests":125,"questions":137,"alternatives":149},[88,96],{"value":89,"source":90,"evidence":91},"Good choice of core technologies","commenter",[92],{"id":93,"by":94,"text":95},"41974755","bosky101","1) Appreciate the single image to get started, but am particularly curious how you handle different events of a new user going to different nodes. 2) any admin interface or just…",{"value":97,"source":90,"evidence":98},"Single Docker image simplifies getting started",[99],{"id":93,"by":94,"text":95},[101,114],{"value":102,"source":90,"evidence":103},"Unclear use cases and value beyond the underlying technologies",[104,108,111],{"id":105,"by":106,"text":107},"41974751","Attummm","Looks great, but what is missing for me are use cases. Why should I use it? What are the unique selling points of your project?",{"id":109,"by":106,"text":110},"41975786","You are selling the underlying technologies(Kafka\u002FClickhouse). I'm interested in your project can do for me, my project(s), team\u002Fcompany. There is a reason that most of the…",{"id":112,"by":106,"text":113},"41978301","Maybe I wasn't clear enough but my questions have been rhetorical. They were not for me. If one starts stating technologies, it is akin to describing the individual ingredients…",{"value":115,"source":90,"evidence":116},"Kafka adds unnecessary complexity for many users",[117,121],{"id":118,"by":119,"text":120},"41977447","oulipo","What is the advantage of this rather than using a postgres plugin for clickhouse and S3 storage of the data to build a kind of data-warehouse, which wouldn't require the bloat of…",{"id":122,"by":123,"text":124},"41983896","klaussilveira","How do you guarantee ACID with Kafka being responsible for actually INSERT'ing into ClickHouse? Wouldn't it be less error prone to just use ClickHouse directly and their async…",[126,130],{"value":127,"source":90,"evidence":128},"Admin interface beyond the REST API",[129],{"id":93,"by":94,"text":95},{"value":131,"source":90,"evidence":132},"Automatic archiving of old data to cloud storage",[133],{"id":134,"by":135,"text":136},"41976133","antman","How does it scale? Can you spin up multiple containers? For upcoming features auto archiving to cloud storage old data would be great.",[138,142],{"value":139,"source":90,"evidence":140},"How events are partitioned across Kafka and nodes",[141],{"id":93,"by":94,"text":95},{"value":143,"source":90,"evidence":144},"How to visualize data, since Grafana is not bundled",[145],{"id":146,"by":147,"text":148},"41976535","codegeek","Looks good. In market for something like this and I just ran it locally. how do I visualize data ? Is Grafana not included by default. Also, minor issue in your docs. There is an…",[150,157,164],{"value":151,"source":90,"evidence":152},"PostHog",[153],{"id":154,"by":155,"text":156},"41977487","asdev","how is this different from Posthog?",{"value":158,"source":90,"evidence":159},"Jitsu",[160],{"id":161,"by":162,"text":163},"41975034","d_watt","Looks super interesting. Any positioning thoughts on this vs https:\u002F\u002Fjitsu.com ?",{"value":165,"source":90,"evidence":166},"Kinesis Firehose with S3 and Athena",[167],{"id":168,"by":169,"text":170},"41975670","hitradostava","Looks interesting, we solved this problem with Kinesis Firehose, S3 and Athena. Pricing is cheap, you can run any arbitrary SQL query and there is zero infrastructure to maintain.",{"comments":172,"answered":172,"key":173},10,[174,175,178],{"id":81,"text":82},{"id":176,"text":177},"41990810","I am thinking about setting this up as as a configuration for the type of traffic that doesn't require Kafka. That being said, Kafka has in my experience come in super handy…",{"id":179,"text":180},"41976278","Once you've outgrown a single physical server, you can continue to scale the Trench cluster by spinning up more Trench application servers and switching to dedicated Kafka and…",{"commenters":182,"max_depth":35,"maker_replies":172,"first_comment_mins":183,"lifespan_hours":184},20,4610,148,[186,188,190,192,193,195,198,200,202,205,207,209,210,212,213,215,218,219,223,225,229,233,236,238,241,242,244,249,252,255,257,262,264,265,267,270,272,274],[93,34,46,187,94,34],"maker\u002Fpositive\u002Fexecution",[81,34,34,189,11,39],"maker\u002Fneutral\u002Fexecution",[191,34,32,189,94,34],"41979096",[122,34,32,189,123,34],[176,34,34,194,11,39],"project\u002Fneutral\u002Fexecution",[196,34,32,189,197,34],"42009787","svilen_dobrev",[168,39,39,199,169,34],"project\u002Fcritical\u002Fpricing",[201,39,39,199,94,34],"41979061",[203,39,34,204,169,34],"41980305","commenter\u002Fneutral\u002Fpricing",[206,39,34,204,94,34],"42003939",[134,41,32,208,135,34],"project\u002Fneutral\u002Fperformance",[179,41,34,208,11,39],[105,32,32,211,106,34],"project\u002Fneutral\u002Fidea",[61,32,34,211,11,39],[109,32,39,214,106,34],"project\u002Fcritical\u002Fidea",[216,32,32,211,217,34],"41976796","dfltr",[112,32,39,214,106,34],[220,32,34,221,222,34],"41990565","commenter\u002Fneutral\u002Fnone","teleforce",[224,32,34,221,106,34],"41994424",[226,32,46,227,228,34],"41977886","project\u002Fpositive\u002Fidea","mind-blight",[230,32,34,231,232,34],"41983082","other\u002Fneutral\u002Fidea","Jgrubb",[146,234,32,235,147,34],4,"project\u002Fneutral\u002Fdocs",[237,234,34,235,11,39],"41976889",[239,234,46,240,147,34],"41987465","project\u002Fpositive\u002Fnone",[161,46,32,211,162,34],[243,46,34,211,11,39],"41975259",[245,246,34,247,248,34],"41982663",6,"other\u002Fneutral\u002Fnone","brody_slade_ai",[250,35,39,214,251,34],"41991679","Incipient",[118,253,39,254,119,34],8,"project\u002Fcritical\u002Fexecution",[256,253,34,208,11,39],"41977604",[258,259,32,260,261,34],"41988783",9,"project\u002Fneutral\u002Fnone","remram",[263,259,34,260,11,39],"41988900",[154,172,32,211,155,34],[266,172,34,211,11,39],"41977572",[268,172,34,247,269,34],"41979219","BohdanPetryshyn",[271,33,32,211,119,34],"41977436",[273,33,34,211,11,39],"41977494",[275,276,39,277,278,34],"41982960",12,"project\u002Fcritical\u002Fnone","biddendidden",[],[281,286,291,296,301,305],{"id":282,"name":283,"points":284,"category":285},"41934746","2048",669,"games",{"id":287,"name":288,"points":289,"category":290},"41930628","RF Hunter",594,"hardware",{"id":292,"name":293,"points":294,"category":295},"41997648","AI OmniGen",169,"ai",{"id":297,"name":298,"points":299,"category":300},"41935166","Animal Body Fat Match",142,"consumer_social",{"id":302,"name":303,"points":304,"category":300},"41980681","Ding Dong Doorbell",140,{"id":306,"name":307,"points":308,"category":309},"41952779","Mdx",127,"dev_tools",false,"v2",{"id":313,"name":5,"line":6,"ai":310,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":314,"top":18,"kw":315,"maker":11,"domain":10,"tier":19,"attempts":34,"strip":316,"url":9,"mix":317,"praise":89,"objection":102,"sub":21},41945458,-0.18,[24,25,26,27,28],"50330311003030131005030530011030300301",[32,33,34,35],1794036663453]