[{"data":1,"prerenderedAt":794},["ShallowReactive",2],{"project-43331582":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":36,"aspects":37,"strip":46,"post_text":50,"what":51,"said":88,"maker_in_thread":171,"shape":184,"grid":189,"attempts":756,"same_week":757,"maker_shows":34,"maker_hidden":786,"reader_version":787,"card":788},"43331582","Factorio Learning Environment","Agents Build Factories","title","Show HN: Factorio Learning Environment – Agents Build Factories","https:\u002F\u002Fjackhopkins.github.io\u002Ffactorio-learning-environment\u002F","jackhopkins.github.io","noddybear",1741694522,2025,749,209,34,3837,1,"legend","ai","llm agent benchmark","library",[24,25,26,27,28],"factorio","llm agents","benchmark","reinforcement learning","spatial reasoning","curious",0.157,{"positive":32,"neutral":33,"mixed":34,"critical":35},18,71,2,7,98,{"idea":38,"performance":41,"design":43,"docs":44,"execution":45},{"positive":39,"critical":40},15,6,{"positive":42,"critical":42},0,{"positive":18,"critical":18},{"positive":42,"critical":42},{"positive":42,"critical":42},[47,47,48,48,48,18,18,47,49,48,48,48,48,48,48,49,47,48,49,48,48,48,48,47,48,48,47,48,48,47,48,48,49,18,48,42,42,42,48,48],5,3,4,"I'm Jack, and I'm excited to share a project that has channeled my Factorio addiction recently: the Factorio Learning Environment (FLE).\n\nFLE is an open-source framework for developing and evaluating LLM agents in Factorio. It provides a controlled environment where AI models can attempt complex automation, resource management, and optimisation tasks in a grounded world with meaningful constraints.\n\nA critical advantage of Factorio as a benchmark is its unbounded nature. Unlike many evals that are quickly saturated by newer models, Factorio's geometric complexity scaling means it won't be \"sol",{"problem":52,"audience":58,"pricing":67,"motivation":68,"time_to_build":67,"stack":72,"openness":85,"stage":86,"team_size":87},{"value":53,"source":54,"evidence":55},"Lacks a long-horizon, unbounded benchmark for testing LLM agent spatial reasoning and planning","maker",[56],{"id":57},"post",{"value":59,"source":60,"evidence":61},"AI researchers evaluating LLM agents","inferred",[62,63],{"id":57},{"id":64,"by":65,"text":66},"43331679","p10jkle","Wow, fascinating. I wonder if in a few years every in-game opponent will just be an LLM with access to a game-controlling API like the one you've created. Did you find there are…",null,{"value":69,"source":54,"evidence":70},"Creator's Factorio addiction and recognition of its potential as an AI research testbed",[71],{"id":57},[73,77,81],{"value":74,"source":54,"evidence":75},"Python",[76],{"id":57},{"value":78,"source":54,"evidence":79},"Lua",[80],{"id":57},{"value":82,"source":54,"evidence":83},"Docker",[84],{"id":57},"open_source","unknown","team",{"praise":89,"objections":98,"requests":110,"questions":134,"alternatives":157},[90],{"value":91,"source":92,"evidence":93},"Better than sparse benchmarks because the game is unbounded","commenter",[94],{"id":95,"by":96,"text":97},"43332539","spieswl","Fantastic idea. It seems like there are a lot of interesting experiments to be had here. The lab-play scenarios having a time-related component seems like a good idea, I assume…",[99,106],{"value":100,"source":92,"evidence":101},"Largest-factory goal leads to sprawling belt loops; SPM would be a better metric",[102],{"id":103,"by":104,"text":105},"43332084","WJW","Very cool and also pretty expected results tbh. Some thoughts: Factorio is a game that requires SIGNIFICANT amounts of thinking ahead, often requiring investments into things…",{"value":107,"source":92,"evidence":108},"Results were expected given Factorio's need for long-term planning",[109],{"id":103,"by":104,"text":105},[111,118,123],{"value":112,"source":92,"evidence":113},"Add tower-defence style tasks with biters",[114,115],{"id":95,"by":96,"text":97},{"id":116,"by":11,"text":117},"43332667","One thing we've been talking about is creating tasks that are a bit more 'tower defence', where biters are released every X steps \u002F seconds. The idea would be to test agents in…",{"value":119,"source":92,"evidence":120},"Layout optimisation benchmarks",[121,122],{"id":95,"by":96,"text":97},{"id":116,"by":11,"text":117},{"value":124,"source":92,"evidence":125},"Screenshot or multimodal input support",[126,130],{"id":127,"by":128,"text":129},"43333045","vessenes","OK, You’ve permanently nerd-baited me, and I wish to apply for a job at the Anthropic Factorio lab immediately. I can’t tell from the paper or these comments if you’re sending…",{"id":131,"by":132,"text":133},"43332331","kevmo314","Does it provide screenshots of the game state? I, too, would struggle to play the game pretty effectively if I could not visually see the game.",[135,143,150],{"value":136,"source":54,"evidence":137},"Are images sent back to the agents?",[138,139,140],{"id":127,"by":128,"text":129},{"id":131,"by":132,"text":133},{"id":141,"by":11,"text":142},"43332496","Agents don't have access to screenshots, as we are purely evaluating text-only models. All reasoning is conducted over object representations of the game (with positions etc). I…",{"value":144,"source":92,"evidence":145},"How does the LLM control the game loop?",[146],{"id":147,"by":148,"text":149},"43331920","jharohit","Everytime a paper like this comes out, I always have 1 question - How do they control the game using the LLMs? How does the control-feedback loop work? WHat tools, software and…",{"value":151,"source":92,"evidence":152},"Is there a human-play benchmark?",[153],{"id":154,"by":155,"text":156},"43335980","mNovak","Is there a human-play benchmark (even informally) for this style of interface? Not saying it's necessary or even relevant, I'm just curious to know what programmatic Factorio…",[158,165],{"value":159,"source":92,"evidence":160},"Pokemon Red reinforcement learning agent",[161],{"id":162,"by":163,"text":164},"43334576","scottmsul","There was a HN post here not too long ago about a team that used reinforcement learning to train an agent to beat pokemon red. They mentioned how they had to tweak the cost…",{"value":166,"source":54,"evidence":167},"Terraform Factorio provider",[168],{"id":169,"by":11,"text":170},"43333364","Someone created this Terraform provider for Factorio a few years ago: https:\u002F\u002Fregistry.terraform.io\u002Fproviders\u002Fefokschaner\u002Ffactorio\u002Flatest\u002Fdocs I think this could be a good…",{"comments":172,"answered":173,"key":174},43,16,[175,178,181],{"id":176,"text":177},"43334166","The thing is that when you create a dense ASCII representation, any gain you might make from the spatial relationships is lost by: a) the tokeniser not working on characters…",{"id":179,"text":180},"43334703","In FLE, you have access to milestones representing the first time a new entity was created, but coming up with a stratification of rewards for different degrees of automation…",{"id":182,"text":183},"43343033","This is true - there are simpler benchmarks that can saturate planning for these models. We were motivated to create a broader spectrum eval, to test multiple capabilities at…",{"commenters":185,"max_depth":186,"maker_replies":172,"first_comment_mins":187,"lifespan_hours":188},110,8,13,115,[190,192,196,198,201,204,207,211,213,217,220,223,227,230,232,234,236,238,240,243,245,247,250,252,255,256,259,262,265,267,270,272,276,279,280,283,286,289,291,294,297,298,299,302,304,306,309,313,315,317,320,324,326,328,330,333,336,338,341,344,346,348,350,351,354,357,359,362,364,366,368,370,373,376,379,381,384,388,390,394,396,400,402,405,407,410,411,414,416,419,421,422,424,426,428,431,433,435,437,440,442,444,446,448,450,454,456,458,460,462,466,468,470,472,475,478,481,484,488,491,495,498,501,504,509,513,515,518,522,526,529,533,537,541,543,547,551,554,556,560,563,565,568,570,572,576,578,582,584,586,589,593,596,598,602,605,609,613,617,621,625,627,630,633,637,642,646,649,651,653,656,658,660,662,664,666,668,671,674,677,679,682,684,687,689,691,693,695,697,699,702,705,707,709,711,714,717,720,723,725,728,732,735,738,741,743,747,750,752],[127,42,47,191,128,42],"project\u002Fpositive\u002Fidea",[193,42,48,194,195,42],"43333278","maker\u002Fneutral\u002Fperformance","martbakler",[197,42,47,191,128,42],"43333560",[199,42,47,191,200,42],"43333720","kridsdale1",[202,42,42,203,128,42],"43334227","other\u002Fneutral\u002Fnone",[205,42,47,191,206,42],"43337542","dismalpedigree",[208,42,42,209,210,42],"43340237","commenter\u002Fneutral\u002Fnone","lukan",[212,42,42,203,206,42],"43349602",[214,42,47,215,216,42],"43334032","maker\u002Fpositive\u002Fidea","grayhatter",[218,42,48,219,195,42],"43334820","project\u002Fneutral\u002Fidea",[221,42,42,203,222,42],"43338600","pyinstallwoes",[224,42,48,225,226,42],"43333554","project\u002Fneutral\u002Fdesign","jillyboel",[228,42,48,229,128,42],"43333584","project\u002Fneutral\u002Fperformance",[231,42,48,225,226,42],"43333621",[233,42,48,225,200,42],"43333738",[235,42,48,225,128,42],"43334261",[176,42,42,237,11,18],"maker\u002Fneutral\u002Fdesign",[239,42,48,194,195,42],"43334007",[241,42,48,225,242,42],"43336234","ajcp",[244,42,48,237,195,42],"43337086",[246,42,42,209,222,42],"43338628",[248,42,42,249,242,42],"43338728","commenter\u002Fneutral\u002Fdesign",[251,42,42,209,222,42],"43352839",[253,42,47,191,254,42],"43337687","groby_b",[162,18,48,219,163,42],[257,18,18,258,163,42],"43334684","project\u002Fcritical\u002Fidea",[260,18,42,209,261,42],"43337182","soulbadguy",[263,18,18,258,264,42],"43337228","csense",[179,18,42,266,11,18],"maker\u002Fneutral\u002Fidea",[268,18,42,203,269,42],"43336513","mclau156",[271,18,48,194,195,42],"43337120",[273,18,42,274,275,42],"43343134","commenter\u002Fcritical\u002Fidea","Gasp0de",[277,34,18,258,278,42],"43338731","noosphr",[182,34,42,266,11,18],[281,34,48,282,278,42],"43346931","maker\u002Fmixed\u002Fidea",[284,48,42,203,285,42],"43335404","owenpalmer",[287,48,42,209,288,42],"43338987","wordpad",[290,48,42,209,285,42],"43340161",[292,49,47,191,293,42],"43334618","Imnimo",[295,49,48,219,296,42],"43335947","fragmede",[95,47,47,191,96,42],[116,47,42,215,11,18],[300,47,48,219,301,42],"43333497","robotresearcher",[303,47,42,266,11,18],"43334105",[305,47,47,191,96,42],"43334468",[307,47,48,219,308,42],"43336302","tomrod",[310,47,48,311,312,42],"43337890","project\u002Fneutral\u002Fnone","aftbit",[314,47,42,209,11,18],"43343042",[316,47,42,203,312,42],"43374565",[318,40,48,266,319,42],"43342548","jxjnskkzxxhx",[321,40,48,322,323,42],"43342772","maker\u002Fneutral\u002Fnone","c0wb0yc0d3r",[325,40,42,322,11,18],"43343019",[327,40,42,266,11,18],"43343013",[329,40,48,266,319,42],"43349490",[331,40,48,332,195,42],"43343573","maker\u002Fneutral\u002Fdocs",[334,35,48,219,335,42],"43334233","infogulch",[337,35,42,266,11,18],"43334396",[339,35,42,340,335,42],"43334572","commenter\u002Fneutral\u002Fidea",[342,35,42,343,11,18],"43343072","commenter\u002Fpositive\u002Fidea",[345,35,42,343,335,42],"43346582",[347,35,48,219,195,42],"43334754",[349,35,48,219,335,42],"43335321",[154,186,48,311,155,42],[352,186,42,203,353,42],"43336201","sonofhans",[355,186,48,311,356,42],"43337924","goriv",[358,186,42,322,11,18],"43342280",[360,186,42,203,361,42],"43338047","janzer",[64,363,48,219,65,42],9,[365,363,42,311,11,18],"43331727",[367,363,48,311,222,42],"43331824",[369,363,42,219,11,18],"43332075",[371,363,42,203,372,42],"43331794","posterman",[374,363,42,203,375,42],"43332192","noirscape",[377,363,48,219,378,42],"43332471","PetitPrince",[380,363,42,203,296,42],"43336036",[382,363,48,219,383,42],"43339521","jkhdigital",[385,386,48,219,387,42],"43335216",10,"gglon",[389,386,42,203,383,42],"43339247",[391,392,48,311,393,42],"43332121",11,"myrmidon",[395,392,42,311,11,18],"43332317",[397,398,47,191,399,42],"43331931",12,"moconnor",[401,398,42,311,11,18],"43332060",[403,398,48,219,404,42],"43332277","uncertainrhymes",[406,398,42,311,11,18],"43332371",[408,187,47,191,409,42],"43332931","barrystaes",[169,187,42,311,11,18],[412,187,42,203,413,42],"43333429","delichon",[415,187,42,203,11,18],"43333520",[417,187,42,203,418,42],"43342599","tucnak",[131,420,48,225,132,42],14,[141,420,42,225,11,18],[423,420,48,311,128,42],"43333191",[425,420,42,311,11,18],"43343089",[427,420,48,225,195,42],"43332607",[429,420,48,225,430,42],"43332849","Hammershaft",[432,420,42,225,11,18],"43332896",[434,420,48,225,128,42],"43333241",[436,420,42,311,11,18],"43333393",[438,39,47,191,439,42],"43334194","iliketrains",[441,39,42,322,11,18],"43334413",[443,39,42,225,11,18],"43334470",[445,39,48,225,439,42],"43335458",[447,39,48,225,195,42],"43337219",[147,173,48,449,148,42],"project\u002Fneutral\u002Fexecution",[451,173,48,452,453,42],"43331985","project\u002Fneutral\u002Fdocs","collinvandyck76",[455,173,42,449,11,18],"43332031",[457,173,48,219,148,42],"43339213",[459,173,42,311,11,18],"43343157",[461,173,48,219,148,42],"43339208",[463,464,48,219,465,42],"43334839",17,"devit",[467,464,42,219,11,18],"43334956",[469,464,48,219,465,42],"43337137",[471,464,42,219,11,18],"43343146",[473,32,48,229,474,42],"43332245","mritchie712",[476,32,42,477,11,18],"43332348","project\u002Fneutral\u002Fpricing",[479,32,48,311,480,42],"43332364","0xmason",[482,32,48,229,483,42],"43338172","stared",[485,486,48,229,487,42],"43331980",19,"sturza",[489,486,42,203,490,42],"43336471","svilen_dobrev",[492,493,42,203,494,42],"43332346",20,"danso",[496,493,42,203,497,42],"43332386","HPsquared",[499,493,42,203,500,42],"43333340","pferde",[502,493,42,203,503,42],"43333869","htrp",[505,506,47,507,508,42],"43334329",21,"maker\u002Fpositive\u002Fnone","onehair",[510,511,48,311,512,42],"43331962",22,"mentalgear",[514,511,42,311,11,18],"43332048",[516,511,42,203,517,42],"43336966","throitallaway",[519,520,48,219,521,42],"43337129",23,"cmgriffing",[523,524,42,203,525,42],"43334620",24,"zoogeny",[527,524,42,203,528,42],"43334801","chompychop",[530,531,42,203,532,42],"43343680",25,"ramesh31",[534,535,48,452,536,42],"43346300",26,"nickvec",[538,539,48,452,540,42],"43335590",27,"bocklund",[542,539,42,332,11,18],"43335790",[544,545,42,203,546,42],"43331881",28,"artemonster",[548,549,47,191,550,42],"43331903",29,"keeganpoppen",[552,549,42,203,553,42],"43332074","Aardwolf",[555,549,42,203,11,18],"43332103",[557,558,42,203,559,42],"43331994",30,"alexop",[561,558,42,203,562,42],"43332020","quchen",[564,558,48,219,104,42],"43332131",[566,558,48,219,567,42],"43332240","lucianbr",[569,558,48,219,430,42],"43332889",[571,558,42,203,517,42],"43336981",[573,574,48,219,575,42],"43336510",31,"ainiriand",[577,574,48,311,517,42],"43336908",[579,580,48,219,581,42],"43332479",32,"cgannett",[583,580,48,219,581,42],"43332541",[585,580,42,219,11,18],"43332711",[587,580,42,203,588,42],"43336789","TeMPOraL",[590,591,48,225,592,42],"43332218",33,"sc68cal",[594,591,48,225,595,42],"43332527","philipwhiuk",[597,591,42,209,592,42],"43332998",[599,16,47,600,601,42],"43331782","project\u002Fpositive\u002Fdesign","leetbulb",[603,16,42,604,11,18],"43332699","maker\u002Fpositive\u002Fdesign",[606,607,42,203,608,42],"43336487",35,"idiotsecant",[610,611,42,203,612,42],"43332962",36,"Python3267",[614,615,47,191,616,42],"43331833",37,"zelias",[618,619,18,258,620,42],"43337878",38,"deadbabe",[622,623,48,229,624,42],"43332022",39,"loveparade",[626,623,48,225,562,42],"43332047",[628,623,18,629,553,42],"43332244","project\u002Fcritical\u002Fdesign",[631,623,42,203,632,42],"43332956","enragedcacti",[634,635,42,203,636,42],"43335537",40,"johnisgood",[638,639,47,640,641,42],"43335768",41,"project\u002Fpositive\u002Fnone","andbberger",[643,644,42,203,645,42],"43331894",42,"tux3",[647,644,42,203,648,42],"43331939","wegfawefgawefg",[650,644,42,203,497,42],"43332373",[103,172,48,652,104,42],"project\u002Fmixed\u002Fidea",[654,172,48,219,655,42],"43332165","accurrent",[657,172,42,266,11,18],"43332217",[659,172,47,191,655,42],"43332302",[661,172,42,322,11,18],"43332414",[663,172,42,266,11,18],"43332202",[665,172,18,258,104,42],"43332326",[667,172,42,229,11,18],"43332757",[669,172,18,258,670,42],"43332340","lupusreal",[672,172,48,219,673,42],"43332416","malfist",[675,172,48,219,676,42],"43332487","asah",[678,172,42,229,11,18],"43332569",[680,172,42,203,681,42],"43332880","aithrowawaycomm",[683,172,48,266,195,42],"43333197",[685,172,42,203,686,42],"43332409","eterm",[688,172,42,209,11,18],"43332549",[690,172,42,203,616,42],"43332651",[692,172,42,209,686,42],"43333015",[694,172,42,203,681,42],"43332833",[696,172,42,203,128,42],"43333131",[698,172,42,203,335,42],"43333490",[700,172,42,203,701,42],"43333504","sdwr",[703,172,42,203,704,42],"43333743","tavavex",[706,172,42,209,686,42],"43334282",[708,172,42,209,704,42],"43334623",[710,172,42,203,670,42],"43335367",[712,172,42,203,713,42],"43377310","deterministic",[715,172,42,203,716,42],"43332816","mrighele",[718,172,42,203,719,42],"43333240","bombcar",[721,172,42,203,722,42],"43333224","Philpax",[724,172,42,311,11,18],"43334140",[726,172,42,203,727,42],"43333259","kortilla",[729,730,42,203,731,42],"43334138",44,"Starlord2048",[733,730,48,225,734,42],"43334429","andai",[736,730,42,203,737,42],"43334643","HideousKojima",[739,730,48,225,740,42],"43335562","nostrademons",[742,730,42,191,11,18],"43334450",[744,730,42,745,746,42],"43335187","other\u002Fneutral\u002Fidea","mlsu",[748,749,48,229,595,42],"43332483",45,[751,749,42,311,11,18],"43332814",[753,754,42,203,755,42],"43333333",46,"dkural",[],[758,763,768,773,777,782],{"id":759,"name":760,"points":761,"category":762},"43328095","Seven39",735,"consumer_social",{"id":764,"name":765,"points":766,"category":767},"43347306","Time Portal",497,"games",{"id":769,"name":770,"points":771,"category":772},"43280615","CodeTracer",334,"dev_tools",{"id":774,"name":775,"points":776,"category":762},"43304785","daale.club",302,{"id":778,"name":779,"points":780,"category":781},"43333661","Pila",301,"hardware",{"id":783,"name":784,"points":785,"category":20},"43296918","rlama",295,false,"v2",{"id":789,"name":5,"line":6,"ai":786,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":790,"top":18,"kw":791,"maker":11,"domain":10,"tier":19,"attempts":42,"strip":792,"url":9,"mix":793,"praise":91,"objection":100,"sub":21},43331582,0.16,[24,25,26,27,28],"5533311543333334534333353353353341300033",[32,33,34,35],1794036671023]