[{"data":1,"prerenderedAt":419},["ShallowReactive",2],{"project-38038239":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":35,"aspects":36,"strip":44,"post_text":46,"what":47,"said":104,"maker_in_thread":232,"shape":245,"grid":249,"attempts":382,"same_week":383,"maker_shows":45,"maker_hidden":412,"reader_version":413,"card":414},"38038239","ScratchDB","Open-Source Snowflake on ClickHouse","title","Show HN: ScratchDB – Open-Source Snowflake on ClickHouse","https:\u002F\u002Fgithub.com\u002Fscratchdata\u002FScratchDB","github.com","memset",1698413646,2023,261,56,187,3109,7,"legend","data","managed olap database","web_app",[24,25,26,27,28],"olap database","clickhouse","data warehouse","json ingestion","columnar storage","curious",-0.03,{"positive":32,"neutral":33,"mixed":34,"critical":18},4,21,1,33,{"idea":37,"execution":39,"performance":42,"pricing":43},{"positive":34,"critical":38},3,{"positive":40,"critical":41},0,2,{"positive":40,"critical":40},{"positive":40,"critical":41},[38,40,34,40,38,45,40,38,40,38,40,38,40,38,38,40,34,34,38,38,40,38,38,34,38,34,40,34,38,38,40,38,45,38,45,38,40,38,40,45],5,"Hello! For the past year I’ve been working on a fully-managed data warehouse built on Clickhouse. I built this because I was frustrated with how much work was required to run an OLAP database in prod: re-writing my app to do batch inserts, managing clusters and needing to look up special CREATE TABLE syntax every time I made a change. I found pricing for other warehouses confusing (what is a “credit” exactly?) and worried about getting capacity-planning wrong.\n\nI was previously building accounting software for firms with millions of transactions. I desperately needed to move from Postgres to a",{"problem":48,"audience":54,"pricing":61,"motivation":70,"time_to_build":74,"stack":75,"openness":101,"stage":102,"team_size":103},{"value":49,"source":50,"evidence":51},"Running OLAP databases like ClickHouse in production requires heavy setup, batch inserts, and schema management.","maker",[52],{"id":53},"post",{"value":55,"source":50,"evidence":56},"Developers who need analytical queries over real-time data without learning ClickHouse internals.",[57,58],{"id":53},{"id":59,"by":11,"text":60},"38042422","I haven't seen TiDB before - thanks for sharing. I honestly haven't spent tons of much time analyzing other databases (there are so many!) but I haven't seen much out there which…",{"value":62,"source":50,"evidence":63},"Hosted version billed per GB of compressed data stored plus query wall time; minimum $10 for 30 GB.",[64,67],{"id":65,"by":11,"text":66},"38042927","I don't bill separately for the compute vs storage the way bigtable does. The pricing per GB of data is inclusive of compute. The goal is for pricing to be modeled similarly to…",{"id":68,"by":11,"text":69},"38038597","You can use it for logs. I have a basic example here: https:\u002F\u002Fscratchdb.com\u002Fblog\u002Ffly-logs-to-clickhouse\u002F It's just storing data in a database so it persists as long as you want.…",{"value":71,"source":50,"evidence":72},"Author was frustrated with the operational work of running ClickHouse for an accounting app.",[73],{"id":53},null,[76,80,84,91,96],{"value":77,"source":50,"evidence":78},"Go",[79],{"id":53},{"value":81,"source":50,"evidence":82},"ClickHouse",[83],{"id":53},{"value":85,"source":50,"evidence":86},"Hetzner servers",[87,88],{"id":53},{"id":89,"by":11,"text":90},"38046592","Really interesting question(s)! Will try to answer the gist. First, the system is designed for users to be able to drop in whatever hardware they want. So your questions are…",{"value":92,"source":50,"evidence":93},"Backblaze B2",[94,95],{"id":53},{"id":89,"by":11,"text":90},{"value":97,"source":50,"evidence":98},"AWS SQS",[99,100],{"id":53},{"id":89,"by":11,"text":90},"open_source","launched","solo",{"praise":105,"objections":125,"requests":158,"questions":180,"alternatives":205},[106,114],{"value":107,"source":108,"evidence":109},"Simple JSON-in, SQL-out workflow","commenter",[110],{"id":111,"by":112,"text":113},"38045544","NortySpock","Not the parent poster, but I wouldn't race to solve the problem via supporting many more connectors that require lots of config options. You'll just spend time supporting lots of…",{"value":115,"source":108,"evidence":116},"Open-source with strong developer experience",[117,121],{"id":118,"by":119,"text":120},"38042361","tbragin","> Philosophically, I think there is a lot of room for open-source software that also has an amazing UI and developer experience. +1 to that! Welcome to the ClickHouse community!",{"id":122,"by":123,"text":124},"38042522","wkoszek","ScratchDB has save my business and it's awesome. I think if you need a columnar store, you should really try these guys",[126,137,147],{"value":127,"source":108,"evidence":128},"'Open-Source Snowflake' name implies feature parity that the project does not aim for",[129,133],{"id":130,"by":131,"text":132},"38043609","skeeter2020","I too was confused by the title. If it doesn't support feature-parity, nor approach the problem space from the same perspective as Snowflake, I don't think it's the open source…",{"id":134,"by":135,"text":136},"38044536","willseth","Calling it \"Open-Source Snowflake\" implies you are trying to be a drop-in replacement for Snowflake for at least a subset of features, SQL syntax, etc.",{"value":138,"source":108,"evidence":139},"Storage pricing appears high compared with BigQuery",[140,144],{"id":141,"by":142,"text":143},"38042460","ddorian43","Why is your storage 10X that of bigtable? How does your compute price compare to bigtable?",{"id":145,"by":142,"text":146},"38042996","I wrote wrong, I meant bigquery, sorry. It's 9x of bigquery.",{"value":148,"source":108,"evidence":149},"Nested JSON flattening has edge cases such as key collisions",[150,154],{"id":151,"by":152,"text":153},"38044513","zepolen","What happens with this? ``` { \"some_key\": \"1\" \"some\": { \"key\": \"2\" } } ```",{"id":155,"by":156,"text":157},"38044969","mosen","I tested it with the “explode” flattening mode, and it resulted in { “some_key”: “2” }",[159,166,173],{"value":160,"source":108,"evidence":161},"Support for CSV, Parquet, Avro or Protobuf ingestion",[162],{"id":163,"by":164,"text":165},"38045290","pitah1","Thanks for sharing. Looks very clean and simple to use. Do you plan on supporting non-JSON data types for insertion? For example, inserting CSV files, parquet files, Avro or…",{"value":167,"source":108,"evidence":168},"Ability to define explicit schemas and compression per column",[169],{"id":170,"by":171,"text":172},"38045020","gbrits","Congrats with the launch. This looks great. Inferring schemas on the fly is awesome to get started quickly, but are there ways to explicitly define a schema if I wanted to? For…",{"value":174,"source":108,"evidence":175},"Use a standard JSON flattening approach",[176],{"id":177,"by":178,"text":179},"38045943","lionkor","maybe use the json flattening RFC instead of inventing your own?",[181,191,198],{"value":182,"source":108,"evidence":183},"How the project differs from Tinybird and GraphJSON",[184,187],{"id":185,"by":119,"text":186},"38042000","Disclaimer: I work at ClickHouse. Thank you! Looks really interesting! I personally agree that real-time OLAP databases have potential to better serve workloads currently in…",{"id":188,"by":189,"text":190},"38043350","mike_d","> how you see your project differs from GraphJSON (https:\u002F\u002Fwww.graphjson.com\u002F) and Tinybird (https:\u002F\u002Fwww.tinybird.co\u002F)? Not OP, but neither of these appear to be open source?",{"value":192,"source":108,"evidence":193},"What the AGPL license means for hosted use",[194],{"id":195,"by":196,"text":197},"38052448","shrubble","What does the license mean, if I don't change any of the code you provide, but use it to provide a public-facing service? Like if I use it for a forum, for instance, but am using…",{"value":199,"source":108,"evidence":200},"How primary keys are chosen for auto-created tables",[201],{"id":202,"by":203,"text":204},"38043348","jed_sanders12","This looks great. I have one question. When you are automatically creating tables, how do you choose primary keys order for clickhouse table?",[206,210,214,221,225],{"value":207,"source":108,"evidence":208},"Tinybird",[209],{"id":185,"by":119,"text":186},{"value":211,"source":108,"evidence":212},"GraphJSON",[213],{"id":185,"by":119,"text":186},{"value":215,"source":108,"evidence":216},"Snowflake",[217],{"id":218,"by":219,"text":220},"38042892","CharlesW","Can you explain what \"open-source Snowflake\" means, since you don't explain it in this description, in the repo, or on the site? Is your goal explicitly to replicate all…",{"value":222,"source":108,"evidence":223},"Google BigQuery",[224],{"id":141,"by":142,"text":143},{"value":226,"source":108,"evidence":227},"TiDB with TiFlash",[228],{"id":229,"by":230,"text":231},"38041892","esafak","TiDB is an HTAP whose OLAP component (TiFlash) was based on Clickhouse: https:\u002F\u002Fnews.ycombinator.com\u002Fitem?id=23584022 If you have analyzed the competition, what are your selling…",{"comments":233,"answered":234,"key":235},17,14,[236,239,242],{"id":237,"text":238},"38043068","This is good feedback on writing a clear message - I appreciate it. The goal of this project is to build a superb developer experience around analytical databases. I think that…",{"id":240,"text":241},"38042285","Good to meet you! I hadn't seen graphjson before, I'll check that out. I'm also a fan of tinybird, and I think we have similar goals of wanting to make it easier for people to…",{"id":243,"text":244},"38043367","This is a really good question. My snarky answer is \"the way we ingest JSON doesn't require 50 pages and n configuration settings to explain.\" Here's a more factual one: - We…",{"commenters":246,"max_depth":18,"maker_replies":233,"first_comment_mins":247,"lifespan_hours":248},34,23,29,[250,252,254,256,257,261,262,264,265,268,269,273,274,277,278,281,284,286,289,292,295,296,298,302,304,307,311,314,317,318,319,321,323,325,330,331,334,336,337,339,341,343,345,347,350,352,355,359,362,366,370,371,372,373,377,380],[218,40,38,251,219,40],"project\u002Fneutral\u002Fidea",[237,40,40,253,11,34],"maker\u002Fneutral\u002Fidea",[130,40,34,255,131,40],"project\u002Fcritical\u002Fidea",[134,40,34,255,135,40],[258,40,40,259,260,40],"38047984","commenter\u002Fneutral\u002Fidea","az226",[185,34,38,251,119,40],[240,34,40,263,11,34],"maker\u002Fpositive\u002Fidea",[118,34,45,263,119,40],[266,34,40,259,267,40],"38044430","hawk_",[188,34,40,259,189,40],[270,41,38,271,272,40],"38042978","project\u002Fneutral\u002Fexecution","giovannibonetti",[243,41,40,271,11,34],[275,41,38,271,276,40],"38043875","antoniojtorres",[151,41,38,271,152,40],[279,41,40,280,11,34],"38044792","maker\u002Fneutral\u002Fexecution",[282,41,38,271,283,40],"38045145","namtab00",[285,41,40,280,11,34],"38045261",[287,41,40,288,283,40],"38045457","commenter\u002Fneutral\u002Fexecution",[290,41,38,271,291,40],"38045353","dicriseg",[293,41,38,271,294,40],"38047415","gostsamo",[155,41,38,271,156,40],[297,41,40,280,11,34],"38045268",[299,41,34,300,301,40],"38044901","project\u002Fcritical\u002Fexecution","datatrashfire",[177,41,34,303,178,40],"maker\u002Fcritical\u002Fexecution",[305,41,40,288,306,40],"38045224","zX41ZdbW",[308,38,38,309,310,40],"38042817","project\u002Fneutral\u002Fnone","tiffanyh",[312,38,38,309,313,40],"38046623","iFire",[315,32,38,271,316,40],"38038468","throwaway295729",[68,32,40,280,11,34],[163,45,38,271,164,40],[320,45,40,280,11,34],"38045314",[111,45,38,322,112,40],"project\u002Fmixed\u002Fexecution",[324,45,34,255,164,40],"38045740",[326,327,38,328,329,40],"38041920",6,"project\u002Fneutral\u002Fperformance","didip",[195,18,38,309,196,40],[141,332,34,333,142,40],8,"project\u002Fcritical\u002Fpricing",[65,332,40,335,11,34],"maker\u002Fneutral\u002Fpricing",[145,332,34,333,142,40],[338,332,40,335,11,34],"38043219",[170,340,38,271,171,40],9,[342,340,40,280,11,34],"38045219",[202,344,38,271,203,40],10,[346,344,40,280,11,34],"38043402",[348,344,38,349,203,40],"38043519","maker\u002Fneutral\u002Fperformance",[351,344,40,349,11,34],"38043627",[353,344,45,354,203,40],"38043730","maker\u002Fpositive\u002Fnone",[356,357,38,309,358,40],"38047489",11,"anon3949494",[360,357,40,361,11,34],"38049473","maker\u002Fneutral\u002Fnone",[363,364,45,354,365,40],"38051282",12,"yoav",[367,368,38,361,369,40],"38046421",13,"OmarAssadi",[89,368,40,361,11,34],[229,234,38,251,230,40],[59,234,40,253,11,34],[374,234,40,375,376,40],"38045291","commenter\u002Fneutral\u002Fnone","gregwebs",[122,378,45,379,123,40],15,"project\u002Fpositive\u002Fnone",[381,378,40,354,11,34],"38042966",[],[384,389,394,399,403,408],{"id":385,"name":386,"points":387,"category":388},"38052344","OpenSign",545,"web_saas",{"id":390,"name":391,"points":392,"category":393},"38044782","WireHole",298,"infrastructure",{"id":395,"name":396,"points":397,"category":398},"38084098","postgresql-event-sourcing",231,"dev_tools",{"id":400,"name":401,"points":402,"category":393},"38086598","MicroTCP",204,{"id":404,"name":405,"points":406,"category":407},"38093083","Halloween game (snail terminal demo)",173,"games",{"id":409,"name":410,"points":411,"category":20},"38057759","Open Australian Legal Corpus",172,false,"v2",{"id":415,"name":5,"line":6,"ai":412,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":30,"top":18,"kw":416,"maker":11,"domain":10,"tier":19,"attempts":40,"strip":417,"url":9,"mix":418,"praise":107,"objection":127,"sub":21},38038239,[24,25,26,27,28],"3010350303030330113303313101330353530305",[32,33,34,18],1793991076971]