[{"data":1,"prerenderedAt":521},["ShallowReactive",2],{"project-50118688":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":45,"post_text":46,"what":47,"said":58,"maker_in_thread":145,"shape":155,"grid":160,"attempts":483,"same_week":484,"maker_shows":220,"maker_hidden":513,"reader_version":514,"card":515},"50118688","StarSkirmish Bench","Gemini 4 Argon plays StarCraft","title","Show HN: Gemini 4 Argon plays StarCraft","https:\u002F\u002Fstarskirmish.com\u002Fbench\u002Fv0-2\u002F","starskirmish.com","__cayenne__",1792160324,2026,254,111,188,4366,5,"legend","ai","llm benchmark","other",[21,24,25,26,27],"starcraft","brood war","ai bots","coding agents","curious",-0.021,{"positive":31,"neutral":32,"mixed":33,"critical":34},6,31,1,10,48,{"idea":37,"execution":39,"performance":41,"docs":43,"design":44},{"positive":38,"critical":18},3,{"positive":40,"critical":40},0,{"positive":42,"critical":18},2,{"positive":40,"critical":40},{"positive":33,"critical":40},[18,40,33,38,40,18,40,38,38,38,38,40,40,40,40,42,33,33,33,38,18,38,40,38,38,40,18,38,33,38,33,33,38,38,38,38,18,38,33,40],"I've refreshed StarSkirmish Bench, a benchmark where LLMs play StarCraft BW by writing code, with results from the flagship models of OpenAI, Anthropic, Google, and Muse",{"problem":48,"audience":54,"pricing":54,"motivation":54,"time_to_build":54,"stack":55,"openness":56,"stage":57,"team_size":56},{"value":49,"source":50,"evidence":51},"Measures how well LLMs can write code that plays StarCraft Brood War","maker",[52],{"id":53},"post",null,[],"unknown","launched",{"praise":59,"objections":72,"requests":96,"questions":115,"alternatives":130},[60],{"value":61,"source":62,"evidence":63},"Interesting benchmark with time limits and a clean site","commenter",[64,68],{"id":65,"by":66,"text":67},"50119301","m_w_","This is an interesting benchmark - I like that time is limited, as it often isn't in other benchmarks. It would be interesting to see the reverse, how long does each model take…",{"id":69,"by":70,"text":71},"50122008","aleksiy123","The site overall looks nice too. It’s fun not to see slop for a ui for this kind of project.",[73,84],{"value":74,"source":62,"evidence":75},"Unclear what the benchmark actually measures",[76,80],{"id":77,"by":78,"text":79},"50119846","iugtmkbdfil834","I am not sure I like this benchmark, but mostly, because I am not sure what it is really evaluating. It is not an idle dismissal.. multiple factors seem to complicate what it is…",{"id":81,"by":82,"text":83},"50120363","chis","Yeah giving a 1 hour limit for code generation makes the whole thing a bit incoherent. Plus, performance is just going to depend on how closely the models have memorized the best…",{"value":85,"source":62,"evidence":86},"Results may reflect memorized bot code from training data",[87,91,92],{"id":88,"by":89,"text":90},"50130727","gritzko","StarCraft probably has hundreds of bot implementations lying on GitHub. Which makes this bench a recall-from-the-training-set competition, essentially. I tried that method on old…",{"id":81,"by":82,"text":83},{"id":93,"by":94,"text":95},"50119283","jstummbillig","How do they avoid the LLMs simply working with pre-training data? Which apparently they do, since the human written bots win. But it's a mystery, is what it is.",[97,108],{"value":98,"source":62,"evidence":99},"Comparisons of harnesses and tools",[100,104],{"id":101,"by":102,"text":103},"50121985","djoldman","> ... runs each model in its lab's 1st party CLI harness at the highest reasoning setting it offers. ``` Anthropic (Claude models): Claude Code OpenAI (GPT models): Codex Google…",{"id":105,"by":106,"text":107},"50128744","krzyk","I loved sc1 (and SC2) Do you plan on comparing harnesses also? Playing in pi, opencode Vs the original harnesses?",{"value":109,"source":62,"evidence":110},"Results against stronger bots like Pluto",[111],{"id":112,"by":113,"text":114},"50119191","BeaverGoose","They should benchmark it against Pluto. It's a artificial neutral network bot that can easily take games off the best broodwar players.",[116,123],{"value":117,"source":62,"evidence":118},"How do the agent-written bots compare to human bots and what can be learned from them",[119],{"id":120,"by":121,"text":122},"50119842","cgearhart","Is there anything interesting to learn from the agent written bots? Or are they just inferior versions of the best bots because the time limit is so short? Is there any…",{"value":124,"source":62,"evidence":125},"How do the best humans do against the LLMs",[126],{"id":127,"by":128,"text":129},"50119749","caughtinthought","How do the best humans do against the LLMs?",[131,138],{"value":132,"source":62,"evidence":133},"Pluto StarCraft bot",[134,135],{"id":112,"by":113,"text":114},{"id":136,"by":11,"text":137},"50121593","I would disagree that creating a StarCraft: BW bot doesn't require experimental feedback. The way a human author typically improves their bot is by having it play opponents and…",{"value":139,"source":62,"evidence":140},"BroodBench LLM StarCraft benchmark",[141],{"id":142,"by":143,"text":144},"50119763","MichaelNolan","So this benchmark has the LLMs writing a bit to play StarCraft. Somewhat similar there was a post here a while back that had the LLMs playing StarCraft directly -…",{"comments":146,"answered":31,"key":147},11,[148,151,152],{"id":149,"text":150},"50122563","I am very interested in benchmark correlations. Vals IOI, olympiad programming problems solved in C++, and WeirdML v3, unusual machine learning coding tasks also mirror…",{"id":136,"text":137},{"id":153,"text":154},"50121563","Dan Gant (author of Purple Wave) has written about some of this. I did a 48 hour test with GPT-6 Astra and Claude Opus 5.5 and we saw some interesting uses of Psionic Storm. I've…",{"commenters":156,"max_depth":157,"maker_replies":146,"first_comment_mins":158,"lifespan_hours":159},79,7,40,55,[161,165,169,171,174,175,178,182,185,188,190,194,195,198,200,204,206,209,212,214,215,217,219,222,223,226,229,231,234,237,240,243,245,247,250,253,255,257,260,263,266,269,271,274,277,279,282,284,285,288,292,295,297,302,304,307,308,310,314,316,320,322,325,330,333,335,339,343,346,350,354,356,360,364,367,370,372,376,379,383,385,389,391,395,398,401,403,405,408,411,415,419,423,424,427,430,435,438,442,445,448,452,455,458,461,464,466,469,473,477,479],[162,40,18,163,164,40],"50119857","project\u002Fpositive\u002Fidea","GodelNumbering",[166,40,40,167,168,40],"50122320","other\u002Fneutral\u002Fnone","slibhb",[170,40,40,167,164,40],"50122567",[172,40,40,167,173,40],"50129632","Archer6621",[149,40,40,163,11,33],[176,40,40,177,164,40],"50124305","commenter\u002Fneutral\u002Fnone",[179,40,33,180,181,40],"50124150","project\u002Fcritical\u002Fidea","daymanstep",[183,40,40,177,184,40],"50127823","scott_weber",[186,40,40,167,187,40],"50130342","sznio",[120,33,38,189,121,40],"project\u002Fneutral\u002Fidea",[191,33,38,192,193,40],"50120313","project\u002Fneutral\u002Fexecution","serf",[136,33,40,177,11,33],[196,33,40,177,197,40],"50122598","dgant",[153,33,40,199,11,33],"project\u002Fpositive\u002Fexecution",[201,33,18,202,203,40],"50122591","project\u002Fpositive\u002Fperformance","dudeinhawaii",[205,33,18,202,197,40],"50122527",[207,33,40,177,208,40],"50127368","jfreds",[210,33,40,177,211,40],"50129451","9935c101ab17a66",[213,33,40,177,121,40],"50137991",[101,42,38,192,102,40],[88,38,38,216,89,40],"project\u002Fmixed\u002Fidea",[218,38,40,189,11,33],"50131387",[142,220,38,221,143,40],4,"project\u002Fneutral\u002Fnone",[112,18,38,221,113,40],[224,18,40,225,11,33],"50119329","project\u002Fneutral\u002Fperformance",[227,18,38,192,228,40],"50128475","kzrdude",[230,18,40,192,11,33],"50135844",[232,18,38,189,233,40],"50119522","deburo",[235,18,40,167,236,40],"50120352","wredcoll",[238,18,40,167,239,40],"50122227","oblio",[241,18,40,167,242,40],"50125282","saghm",[244,18,40,167,236,40],"50128618",[246,18,40,177,242,40],"50137243",[248,18,40,167,249,40],"50122521","tarsierscripter",[251,18,40,167,252,40],"50138354","hitekker",[254,18,40,167,236,40],"50145566",[256,18,40,167,197,40],"50125779",[258,31,40,167,259,40],"50126306","andai",[261,31,40,167,262,40],"50126322","anon84873628",[264,31,40,167,265,40],"50129083","TeMPOraL",[267,157,40,167,268,40],"50121494","t1amat",[77,270,33,180,78,40],8,[272,270,38,189,273,40],"50120171","luckydata",[275,270,38,189,276,40],"50120256","NitpickLawyer",[278,270,33,180,78,40],"50120340",[280,270,40,177,281,40],"50120408","robotresearcher",[283,270,40,177,78,40],"50120452",[81,270,33,180,82,40],[286,270,38,225,287,40],"50120429","Foobar8568",[289,270,33,290,291,40],"50121519","project\u002Fcritical\u002Fperformance","boxed",[293,270,33,290,294,40],"50122389","liuliu",[296,270,40,177,197,40],"50125791",[298,299,38,300,301,40],"50120188",9,"project\u002Fneutral\u002Fdocs","explodes",[303,299,40,300,11,33],"50121625",[305,34,40,167,306,40],"50123645","Lucasoato",[65,146,18,163,66,40],[309,146,40,225,11,33],"50119371",[311,312,40,167,313,40],"50119811",12,"tweakimp",[315,312,40,167,197,40],"50125798",[317,318,38,189,319,40],"50120380",13,"aiiotnoodle",[93,321,38,192,94,40],14,[323,324,40,167,252,40],"50137691",15,[326,327,40,328,329,40],"50120527",16,"other\u002Fneutral\u002Fidea","pelagicAustral",[331,327,40,177,332,40],"50121363","fwip",[334,327,40,177,329,40],"50122167",[336,337,38,192,338,40],"50128839",17,"carychang",[340,341,38,225,342,40],"50119253",18,"qsort",[344,341,40,345,11,33],"50119421","maker\u002Fneutral\u002Fexecution",[347,348,38,189,349,40],"50120400",19,"donutdonot",[351,352,40,167,353,40],"50125391",20,"orliesaurus",[355,352,40,167,197,40],"50125765",[357,358,40,167,359,40],"50120436",21,"rdudek",[361,362,18,163,363,40],"50120307",22,"rafiyashaheen11",[365,362,40,177,366,40],"50120331","LarsDu88",[368,362,40,167,369,40],"50120361","gchamonlive",[371,362,40,177,366,40],"50124110",[373,374,38,225,375,40],"50124587",23,"TomGarden",[377,374,38,225,378,40],"50125456","oh_no",[380,381,33,290,382,40],"50123252",24,"xyst",[384,381,40,177,197,40],"50125817",[386,387,40,167,388,40],"50134136",25,"NKosmatos",[105,390,38,192,106,40],26,[392,393,38,189,394,40],"50120026",27,"brap",[396,393,33,290,397,40],"50120079","devmor",[399,393,38,225,400,40],"50120138","ghostbrainalpha",[402,393,40,167,366,40],"50120343",[404,393,33,180,394,40],"50123328",[406,393,40,407,366,40],"50123988","commenter\u002Fneutral\u002Fidea",[409,393,40,410,11,33],"50121804","maker\u002Fneutral\u002Fidea",[412,413,38,221,414,40],"50122553",28,"nozzlegear",[416,417,38,189,418,40],"50122361",29,"amishacreates",[420,421,38,189,422,40],"50119405",30,"0ddj0b",[127,32,38,225,128,40],[425,32,40,167,426,40],"50119838","thinkingtoilet",[428,32,38,192,429,40],"50129002","__m",[431,432,38,433,434,40],"50128480",32,"project\u002Fneutral\u002Fdesign","throwawayk7h",[436,432,40,167,437,40],"50128904","bob1029",[439,440,40,167,441,40],"50127503",33,"vivzkestrel",[443,444,38,189,437,40],"50128877",34,[69,446,18,447,70,40],35,"project\u002Fpositive\u002Fdesign",[449,450,38,189,451,40],"50119859",36,"brokensegue",[453,450,40,167,454,40],"50120041","MadameMinty",[456,450,40,167,457,40],"50120195","glouwbug",[459,450,38,189,460,40],"50120113","iammjm",[462,450,40,167,463,40],"50120368","cdelsolar",[465,450,40,410,11,33],"50121654",[467,450,38,433,468,40],"50123161","knicholes",[470,471,33,290,472,40],"50122539",37,"Cynddl",[474,475,40,167,476,40],"50126200",38,"ibamotanahamido",[478,475,40,167,476,40],"50126252",[480,481,40,167,482,40],"50126196",39,"delijati",[],[485,490,494,499,504,509],{"id":486,"name":487,"points":488,"category":489},"50170593","Permit Pending",941,"games",{"id":491,"name":492,"points":493,"category":20},"50127979","Llmspeed",331,{"id":495,"name":496,"points":497,"category":498},"50102247","Unfenced",319,"web_saas",{"id":500,"name":501,"points":502,"category":503},"50137950","Sphinx",253,"dev_tools",{"id":505,"name":506,"points":507,"category":508},"50140769","The Zelda timeline",231,"learning_science",{"id":510,"name":511,"points":512,"category":503},"50131189","Esp32sim",212,false,"v2",{"id":516,"name":5,"line":6,"ai":513,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":517,"top":18,"kw":518,"maker":11,"domain":10,"tier":19,"attempts":40,"strip":519,"url":9,"mix":520,"praise":61,"objection":74,"sub":21},50118688,-0.02,[21,24,25,26,27],"5013050333300002111353033053131133335310",[31,32,33,34],1794036712224]