[{"data":1,"prerenderedAt":184},["ShallowReactive",2],{"project-9854328":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":33,"aspects":34,"strip":38,"post_text":40,"what":41,"said":74,"maker_in_thread":119,"shape":125,"grid":128,"attempts":146,"same_week":147,"maker_shows":31,"maker_hidden":176,"reader_version":177,"card":178},"9854328","Tempest","Balancing an Elasticsearch Cluster by Shard Size","title","Show HN: Balancing an Elasticsearch Cluster by Shard Size","http:\u002F\u002Fengineering.datarank.com\u002F2015\u002F07\u002F08\u002Fbalancing-elasticsearch-cluster-by-shard-size.html","engineering.datarank.com","KennyCason",1436386808,2015,23,9,1115,2565,44,"shown","infrastructure","search cluster balancing","library",[24,25,26,27],"elasticsearch","shard balancing","cluster rebalancing","shard size","curious",0.286,{"positive":31,"neutral":31,"mixed":32,"critical":32},3,0,6,{"design":35,"idea":36,"performance":37},{"positive":32,"critical":32},{"positive":31,"critical":32},{"positive":32,"critical":32},[32,31,39,32,39,32,39,31,31],5,null,{"problem":42,"audience":54,"pricing":40,"motivation":62,"time_to_build":40,"stack":66,"openness":71,"stage":72,"team_size":73},{"value":43,"source":44,"evidence":45},"Default Elasticsearch balancer ignores shard size, so nodes end up with uneven load.","commenter",[46,50],{"id":47,"by":48,"text":49},"9857754","gt565k","You mention \"... however, that none of these balancing options are resource-aware: there is no “balance shards across nodes by the size of the shards” flag to set or knob to…",{"id":51,"by":52,"text":53},"9858783","adenverd","aewhite covered most points, but I'd also point out that a simple use case of a cluster with multiple indices of varying sizes (such as using ES as part of the ELK stack to store…",{"value":55,"source":56,"evidence":57},"Teams running large Elasticsearch clusters with skewed data.","inferred",[58],{"id":59,"by":60,"text":61},"9858036","aewhite","I'm a Sr Dev here at Datarank in charge of our ES architecture. Maybe I can shed some light on your points. \"I was under the impression that shards are kept uniform in size as ES…",{"value":63,"source":44,"evidence":64},"Clustering log-normally distributed data for fast aggregations at scale needed a size-aware balancer.",[65],{"id":59,"by":60,"text":61},[67],{"value":68,"source":44,"evidence":69},"Elasticsearch",[70],{"id":59,"by":60,"text":61},"unknown","launched","company",{"praise":75,"objections":92,"requests":101,"questions":109,"alternatives":114},[76,82],{"value":77,"source":44,"evidence":78},"Unique use case explained well",[79],{"id":80,"by":48,"text":81},"9858408","Thank you for the detailed explanation. It seems like a unique use case. I've yet much to learn about ES :). Glad you are releasing this as a plugin!",{"value":83,"source":44,"evidence":84},"Interesting alternative approach via OptaPlanner",[85,89],{"id":86,"by":87,"text":88},"9855237","divideby0","This seems quite cool. For very large clusters, I'd also consider looking into Optaplanner, which exposes a variety of probabilistic metaheuristics for balancing. There's a…",{"id":90,"by":52,"text":91},"9855653","Oh nice! That looks close to what we were trying to do with this plugin. I'm not sure it would've worked within the constraints of the Elasticsearch environment, but the…",[93,97],{"value":94,"source":44,"evidence":95},"Uneven shards may signal a poor data model or architecture",[96],{"id":47,"by":48,"text":49},{"value":98,"source":44,"evidence":99},"Balancing by shard size is not resource-aware in default ES",[100],{"id":47,"by":48,"text":49},[102],{"value":103,"source":44,"evidence":104},"Data on time to bring nodes online or offline",[105],{"id":106,"by":107,"text":108},"9857416","tobz","Do you have any data on bringing new nodes online \u002F offline? Obviously, rebalancing in general is going to be I\u002FO-intensive if you have a ton of data, but has Tempest shown any…",[110],{"value":111,"source":44,"evidence":112},"Does it speed up or slow down adding a new node?",[113],{"id":106,"by":107,"text":108},[115],{"value":116,"source":44,"evidence":117},"OptaPlanner cloud balancing",[118],{"id":86,"by":87,"text":88},{"comments":120,"answered":32,"key":121},1,[122],{"id":123,"text":124},"9855628","Nice, that looks pretty interesting. I'll make sure the team checks that out.",{"commenters":33,"max_depth":31,"maker_replies":120,"first_comment_mins":126,"lifespan_hours":127},174,20,[129,131,133,135,136,138,140,141,144],[47,32,32,130,48,32],"commenter\u002Fneutral\u002Fdesign",[59,32,31,132,60,32],"maker\u002Fneutral\u002Fdesign",[80,32,39,134,48,32],"maker\u002Fpositive\u002Fidea",[51,32,32,130,52,32],[86,120,39,137,87,32],"project\u002Fpositive\u002Fidea",[123,120,32,139,11,120],"maker\u002Fpositive\u002Fnone",[90,120,39,137,52,32],[106,142,31,143,107,32],2,"project\u002Fneutral\u002Fperformance",[145,142,31,143,52,32],"9857896",[],[148,153,158,163,168,172],{"id":149,"name":150,"points":151,"category":152},"9838955","“Who is hiring",546,"productivity",{"id":154,"name":155,"points":156,"category":157},"9839712","YouTube Filesystem",185,"dev_tools",{"id":159,"name":160,"points":161,"category":162},"9828962","Trigrad",175,"languages",{"id":164,"name":165,"points":166,"category":167},"9830531","Silicon Valley Dictionary",158,"learning_science",{"id":169,"name":170,"points":171,"category":162},"9845763","node-rdpjs",140,{"id":173,"name":174,"points":175,"category":157},"9843268","ttystudio",118,false,"v2",{"id":179,"name":5,"line":6,"ai":176,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":180,"top":18,"kw":181,"maker":11,"domain":10,"tier":19,"attempts":32,"strip":182,"url":9,"mix":183,"praise":77,"objection":94,"sub":21},9854328,0.29,[24,25,26,27],"035050533",[31,31,32,32],1793990927417]