[{"data":1,"prerenderedAt":392},["ShallowReactive",2],{"project-32084571":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":45,"post_text":48,"what":49,"said":76,"maker_in_thread":207,"shape":219,"grid":224,"attempts":352,"same_week":353,"maker_shows":221,"maker_hidden":384,"reader_version":385,"card":386},"32084571","ClickBench","A benchmark for analytical databases (Snowflake, Druid, Redshift)","title","Show HN: A benchmark for analytical databases (Snowflake, Druid, Redshift)","https:\u002F\u002Fbenchmark.clickhouse.com\u002F","benchmark.clickhouse.com","zX41ZdbW",1657728189,2022,137,57,441,2570,18,"notable","data","database benchmark","web_app",[24,21,25,26,27],"analytical database","olap","clickhouse","snowflake","debated",-0.008,{"positive":31,"neutral":32,"mixed":33,"critical":34},8,11,1,10,30,{"idea":37,"performance":39,"execution":40,"docs":41,"pricing":44},{"positive":33,"critical":38},4,{"positive":33,"critical":33},{"positive":33,"critical":38},{"positive":42,"critical":43},2,0,{"positive":43,"critical":33},[46,43,47,46,33,33,33,33,43,43,33,46,43,43,46,33,43,47,47,43,43,33,43,43,43,43,47,43,46,47,33,46,43,47,46,47,43,47,43,33],3,5,"I created a web page to compare different analytical databases (both self-managed and services, open-source and proprietary) on a realistic dataset.\nIt contains 20+ databases, each with installation and data loading scripts. And they can be compared to each other on a set of 43 queries, by data load time or by storage size.\n\nThere are switches to select different types of databases for comparison - for example, only MySQL compatible or PostgreSQL compatible.\n\nIf you play with the switches, many interesting details will be uncovered.\n\nFull description: https:\u002F\u002Fgithub.com\u002FClickHouse\u002FClickBench\u002Fb",{"problem":50,"audience":56,"pricing":67,"motivation":68,"time_to_build":67,"stack":72,"openness":73,"stage":74,"team_size":75},{"value":51,"source":52,"evidence":53},"Comparing analytical databases on performance, load time and storage size","maker",[54],{"id":55},"post",{"value":57,"source":58,"evidence":59},"People choosing analytical databases","inferred",[60,64],{"id":61,"by":62,"text":63},"32086490","derN3rd","My company recently started to invest more into analytics and we had to find a good solution on which backend\u002Fdatabase we want to decide. After some research we settled on…",{"id":65,"by":11,"text":66},"32086499","Yes, this is what I want to include - the cost metric. Although it will be tricky for systems with consumption-based pricing: BigQuery and Athena. And Snowflake has almost…",null,{"value":69,"source":52,"evidence":70},"Compare many analytical databases on a realistic dataset",[71],{"id":55},[],"open_source","launched","unknown",{"praise":77,"objections":101,"requests":138,"questions":167,"alternatives":182},[78,90],{"value":79,"source":80,"evidence":81},"Impressive effort and useful open-source benchmark","commenter",[82,86],{"id":83,"by":84,"text":85},"32088800","gianm","This is impressive work: it's time consuming to set up and benchmark so many different systems! Impressiveness of the effort notwithstanding, I also want to encourage people to…",{"id":87,"by":88,"text":89},"32089341","thomoco","Benchmarks can be quite difficult to interpret when the test parameters vary between tests. However, I think the point in providing the open-source ClickBench benchmark…",{"value":91,"source":80,"evidence":92},"Transparent comparison is useful",[93,97],{"id":94,"by":95,"text":96},"32088666","kthejoker2","Seems like a great ad for c6a.metal ! Without dollars as part of the analysis, I don't find these comparisons useful.",{"id":98,"by":99,"text":100},"32089326","Lucasoato","Yeah, also a comparison with SparkSQL on Kubernetes would be interesting! Let's get in touch, I may help with that.",[102,117,127,134],{"value":103,"source":80,"evidence":104},"Single-table queries only; no joins or complex query shapes",[105,109,113],{"id":106,"by":107,"text":108},"32087211","AdamProut","It looks like the queries are all single table queries with group-bys and aggregates over a reasonably small data set (10s of GB)? I'm sure some real workloads look like this,…",{"id":110,"by":111,"text":112},"32088839","riku_iki","> Regarding the joined table queries that are missing in the tests looks like huge issue with this benchmark.",{"id":114,"by":115,"text":116},"32087847","qoega","There are several existing benchmarks that test query optimisers with a lot of joins. It does not show performance of query engine, but more likely how good is your optimiser was…",{"value":118,"source":80,"evidence":119},"Druid results look misconfigured",[120,124],{"id":121,"by":122,"text":123},"32093000","abhishcool","In this benchmark, druid was killed and restarted after every query because druid seems to get into a degraded state otherwise. Very very likely that there is something wrong…",{"id":125,"by":84,"text":126},"32090558","It does seem odd, especially since in real world cases I'm more accustomed to seeing Druid and ClickHouse be in the same ballpark of performance. Sometimes one is somewhat faster…",{"value":128,"source":80,"evidence":129},"Redshift and ClickHouse hardware not comparable",[130],{"id":131,"by":132,"text":133},"32088361","ruw1090","Yea, Redshift looks like it was run on an ra3.xl (4 cores) and is comparing against clickhouse on a c6a.4xl (16 cores). I suspect if this were normalized Redshift would have an…",{"value":135,"source":80,"evidence":136},"Benchmarks can be misleading and should not be over-trusted",[137],{"id":83,"by":84,"text":85},[139,147,154,161],{"value":140,"source":80,"evidence":141},"Add cost or price dimension to comparisons",[142,146],{"id":143,"by":144,"text":145},"32086097","singhrac","A small suggestion: since the machines aren’t always comparable (eg Redshift) maybe it makes sense to allow the user to make a plot of ($\u002Fhr) vs query time? I realize this adds…",{"id":94,"by":95,"text":96},{"value":148,"source":80,"evidence":149},"Add BigQuery",[150],{"id":151,"by":152,"text":153},"32086833","garciasn","Can BigQuery be added? I'm interested to know how it compares in these sorts of scenarios. We use BQ because I do NOT want to have to deal w\u002Fmanagement of any of the…",{"value":155,"source":80,"evidence":156},"Add Databricks",[157],{"id":158,"by":159,"text":160},"32087916","arikfr","This is great! Will it be possible to include Databricks in the benchmark?",{"value":162,"source":80,"evidence":163},"Add clustered ClickHouse setups",[164],{"id":165,"by":132,"text":166},"32088594","Do you have any benchmark results with Clickhouse in a clustered configuration (and other db's)? Is Clickhouse expected to be run in only single node deployments?",[168,175],{"value":169,"source":80,"evidence":170},"Why ClickHouse performs so well",[171],{"id":172,"by":173,"text":174},"32086697","simonw","Anyone got a succinct explanation for why ClickHouse is doing so well in this kind of benchmark? What are the key tricks they're using to get great analytical performance, and…",{"value":176,"source":80,"evidence":177},"Why Snowflake performs so well",[178],{"id":179,"by":180,"text":181},"32088802","throwawaylala1","Wow Snowflake absolutely crushes these benchmarks. Anyone know why? I don't know much about this space.",[183,187,197,203],{"value":184,"source":80,"evidence":185},"Snowflake",[186],{"id":179,"by":180,"text":181},{"value":188,"source":80,"evidence":189},"StarRocks",[190,194],{"id":191,"by":192,"text":193},"32088719","lk888","Full disclosure - I work for StarRocks (starrocks.com) First of all, this is great. Transparent and healthy competition is always great for the customers! Regarding the joined…",{"id":195,"by":192,"text":196},"32101080","--- The replication and sharding seems pretty difficult and from reading the documentation feels like it might be pretty fragile. That's why trip.com replaced Clickhouse with…",{"value":198,"source":80,"evidence":199},"DuckDB",[200],{"id":201,"by":107,"text":202},"32089447","I don't know where you draw the line between SQL analytics and SQL data warehousing. I think your typical analytical workload definitely involves more data then this benchmark…",{"value":204,"source":80,"evidence":205},"Apache Druid",[206],{"id":83,"by":84,"text":85},{"comments":32,"answered":208,"key":209},7,[210,213,216],{"id":211,"text":212},"32087296","There are many limitations of this benchmark, indeed: https:\u002F\u002Fgithub.com\u002FClickHouse\u002FClickBench\u002F#limitations",{"id":214,"text":215},"32088264","You are absolutely right. That's why this benchmark is named \"a Benchmark For Analytical DBMS\", not \"the definitive benchmark for analytical DBMS\".",{"id":217,"text":218},"32101778","DuckDB is included in this benchmark. But there were many OOMs in this benchmark and it was not easy to make it working: https:\u002F\u002Fgithub.com\u002Fduckdb\u002Fduckdb\u002Fissues\u002F3969 The data…",{"commenters":220,"max_depth":221,"maker_replies":32,"first_comment_mins":222,"lifespan_hours":223},29,6,95,47,[225,227,230,232,235,236,238,240,242,244,246,247,249,250,252,254,258,260,261,265,268,269,271,273,275,278,280,282,286,288,291,294,297,300,302,304,306,309,313,314,317,318,320,324,326,328,330,332,335,336,338,342,344,346,347,348,350,351],[83,43,46,226,84,43],"project\u002Fmixed\u002Fnone",[228,43,43,229,192,43],"32088861","commenter\u002Fneutral\u002Fnone",[87,43,47,231,88,43],"project\u002Fpositive\u002Fidea",[233,43,46,234,107,43],"32089524","project\u002Fneutral\u002Fperformance",[125,43,46,234,84,43],[121,43,33,237,122,43],"project\u002Fcritical\u002Fexecution",[239,43,33,237,115,43],"32096851",[241,43,33,237,115,43],"32096912",[106,33,33,243,107,43],"project\u002Fcritical\u002Fidea",[211,33,43,245,11,33],"maker\u002Fneutral\u002Fdocs",[114,33,33,243,115,43],[248,33,43,229,107,43],"32088039",[214,33,43,229,11,33],[251,33,43,229,132,43],"32088117",[253,33,33,243,111,43],"32088862",[255,33,43,256,257,43],"32089102","other\u002Fneutral\u002Fidea","doliveira",[201,33,46,259,107,43],"project\u002Fneutral\u002Fidea",[217,33,43,234,11,33],[262,33,43,263,264,43],"32089704","other\u002Fneutral\u002Fnone","FridgeSeal",[266,33,43,263,267,43],"32091712","abrazensunset",[143,42,46,259,144,43],[65,42,43,270,11,33],"project\u002Fneutral\u002Fpricing",[131,42,33,272,132,43],"project\u002Fcritical\u002Fperformance",[274,42,43,234,11,33],"32088562",[276,42,47,277,132,43],"32088605","maker\u002Fpositive\u002Fdocs",[279,42,43,234,11,33],"32090262",[61,46,47,281,62,43],"project\u002Fpositive\u002Fnone",[283,46,43,284,285,43],"32087704","commenter\u002Fneutral\u002Fdocs","blinkov",[287,46,43,229,62,43],"32087811",[289,46,43,263,290,43],"32087802","gbrits",[292,46,33,237,293,43],"32088553","glogla",[295,46,43,263,296,43],"32088580","nojito",[298,46,43,263,299,43],"32088754","gaploid",[301,46,43,263,107,43],"32089464",[303,46,43,263,299,43],"32094249",[305,46,43,263,107,43],"32097287",[307,46,43,308,11,33],"32088976","project\u002Fneutral\u002Fexecution",[310,46,47,311,312,43],"32089292","project\u002Fpositive\u002Fexecution","hodgesrm",[195,46,43,263,192,43],[315,46,43,263,316,43],"32090746","ceeplusplus",[172,38,46,234,173,43],[319,38,43,311,11,33],"32086861",[321,38,47,322,323,43],"32094554","project\u002Fpositive\u002Fperformance","drchaim",[94,47,33,325,95,43],"project\u002Fcritical\u002Fpricing",[327,47,43,263,323,43],"32094562",[151,221,46,329,152,43],"project\u002Fneutral\u002Fnone",[331,221,43,329,11,33],"32086916",[333,221,47,334,152,43],"32086938","maker\u002Fpositive\u002Fnone",[179,208,46,234,180,43],[337,208,46,234,132,43],"32089177",[339,31,47,340,341,43],"32109082","project\u002Fpositive\u002Fdocs","mdcallag",[158,343,46,259,159,43],9,[345,343,43,329,11,33],"32088942",[98,343,47,281,99,43],[165,34,46,234,132,43],[349,34,43,234,11,33],"32088938",[191,32,46,259,192,43],[110,32,33,243,111,43],[],[354,359,364,369,374,379],{"id":355,"name":356,"points":357,"category":358},"32013330","PocketBase",563,"infrastructure",{"id":360,"name":361,"points":362,"category":363},"32081943","Inflation-adjusted stock charts",507,"finance_crypto",{"id":365,"name":366,"points":367,"category":368},"32037740","Aspect",374,"dev_tools",{"id":370,"name":371,"points":372,"category":373},"32043229","Ants Sandbox",367,"games",{"id":375,"name":376,"points":377,"category":378},"32034625","Vimified",344,"learning_science",{"id":380,"name":381,"points":382,"category":383},"32080469","Cleanup Photos",319,"creative",false,"v2",{"id":387,"name":5,"line":6,"ai":384,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":388,"top":18,"kw":389,"maker":11,"domain":10,"tier":19,"attempts":43,"strip":390,"url":9,"mix":391,"praise":79,"objection":103,"sub":21},32084571,-0.01,[24,21,25,26,27],"3053111100130031055001000050351305350501",[31,32,33,34],1794036622520]