[{"data":1,"prerenderedAt":479},["ShallowReactive",2],{"project-47431671":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":27,"sentiment":28,"stance_mix":29,"scored_comments":33,"aspects":34,"strip":40,"post_text":43,"what":44,"said":88,"maker_in_thread":203,"shape":211,"grid":215,"attempts":440,"same_week":441,"maker_shows":41,"maker_hidden":471,"reader_version":472,"card":473},"47431671","llm-circuit-finder","Duplicate 3 layers in a 24B LLM, logical deduction .22→.76. No training","title","Show HN: Duplicate 3 layers in a 24B LLM, logical deduction .22→.76. No training","https:\u002F\u002Fgithub.com\u002Falainnothere\u002Fllm-circuit-finder","github.com","xlayn",1773869472,2026,265,80,176,4366,5,"legend","ai","llm layer duplication","other",[21,24,25,26],"reasoning circuits","transformer layers","lm-eval benchmarks","curious",-0.116,{"positive":18,"neutral":30,"mixed":31,"critical":32},27,0,9,41,{"idea":35,"execution":37,"performance":38,"docs":39},{"positive":36,"critical":18},4,{"positive":31,"critical":31},{"positive":31,"critical":36},{"positive":31,"critical":31},[41,31,31,31,41,31,31,41,31,18,42,41,31,42,42,42,42,18,42,42,42,18,31,42,42,42,41,41,42,31,31,41,41,31,18,42,42,18,41,42],1,3,null,{"problem":45,"audience":43,"pricing":43,"motivation":54,"time_to_build":43,"stack":62,"openness":85,"stage":86,"team_size":87},{"value":46,"source":47,"evidence":48},"Improving LLM reasoning scores without retraining the model","maker",[49,51],{"id":50},"post",{"id":52,"by":11,"text":53},"47433703","I published the results for devstral... results folder of the github https:\u002F\u002Fgithub.com\u002Falainnothere\u002Fllm-circuit-finder\u002Ftree\u002Fmain\u002Fresults I'm using the following configuration…",{"value":55,"source":56,"evidence":57},"Test whether duplicating layers improves reasoning, after replicating David Ng's RYS method","commenter",[58],{"id":59,"by":60,"text":61},"47437002","simgt","> I replicated David Ng's RYS method [...] found something I didn't expect. > Transformers appear to have discrete \"reasoning circuits\" — contiguous blocks of 3-4 layers that act…",[63,69,73,79],{"value":64,"source":47,"evidence":65},"Devstral",[66],{"id":67,"by":11,"text":68},"47433853","I explored that, again with Devstral, but the execution with 4 times the same circuit lead to less score on the tests. I chat with the model to see if the thing was still working…",{"value":70,"source":47,"evidence":71},"Phi-4",[72],{"id":52,"by":11,"text":53},{"value":74,"source":47,"evidence":75},"llama.cpp",[76],{"id":77,"by":11,"text":78},"47433819","The other interesting point is that right now I'm copy pasting the layers, but a patch in llama.cpp can make the same model now behave better by a fact of simply following a…",{"value":80,"source":47,"evidence":81},"lm-evaluation-harness",[82],{"id":83,"by":11,"text":84},"47433883","You can check here the results for Devstral, speed limits me, but these are the results for the first 50 tests of the command ``` # Run lm-evaluation-harness lm_eval --model…","open_source","prototype","unknown",{"praise":89,"objections":108,"requests":144,"questions":166,"alternatives":184},[90,101],{"value":91,"source":56,"evidence":92},"Interesting, cool result on the logical deduction benchmark",[93,97],{"id":94,"by":95,"text":96},"47438524","edg5000","This is very cool",{"id":98,"by":99,"text":100},"47433716","nowittyusername","There's still a lot of low hanging fruit left IMO. Good find and rather funny to think about as you can have someone simply clone the various layers multiple times and instead of…",{"value":102,"source":56,"evidence":103},"Fits with pruning findings that middle layers carry reasoning",[104],{"id":105,"by":106,"text":107},"47433692","zhangchen","this lines up with what pruning papers have been finding, the middle layers carry most of the reasoning weight and you can often drop the outer ones without much loss. cool to…",[109,121,136,140],{"value":110,"source":56,"evidence":111},"Prior art: David Ng's RYS method and earlier layer duplication work",[112,113,117],{"id":59,"by":60,"text":61},{"id":114,"by":115,"text":116},"47433797","Karuma","Wow, every single word in the original post and on that README.md is pure LLM. How sad. In any case, this has been done at least since the very first public releases of Llama by…",{"id":118,"by":119,"text":120},"47434797","kristianp","The method used here by David Ng, was discussed a few days ago at https:\u002F\u002Fnews.ycombinator.com\u002Fitem?id=47322887",{"value":122,"source":56,"evidence":123},"Gains may be benchmark noise or come with losses on other tasks",[124,128,132],{"id":125,"by":126,"text":127},"47433647","XCSme","But if it got worse on other tests, it doesn't do much good, right?",{"id":129,"by":130,"text":131},"47434267","m3kw9","What, just randomly choose some \"layer\" and duplicate it and give some arbitrary reasoning went from 0.2 -> 0.7, i don't know man. You need to use real benchmarks.",{"id":133,"by":134,"text":135},"47433537","rao-v","I’d love to believe this is real, but I’m pretty sure you will lose performance on a “fair” mix of tasks, even after fine tuning. I know multiple teams have explored recurrent…",{"value":137,"source":47,"evidence":138},"Small test sample (50 per task) and speed-limited validation",[139],{"id":52,"by":11,"text":53},{"value":141,"source":56,"evidence":142},"Post and README appear LLM-written",[143],{"id":114,"by":115,"text":116},[145,152,159],{"value":146,"source":56,"evidence":147},"Publish per-task benchmark results",[148],{"id":149,"by":150,"text":151},"47433484","colejhudson","Would you be able to publish the individual benchmarks for Qwen2.5-Coder-32B? GSM8K specifically would be useful to look at.",{"value":153,"source":56,"evidence":154},"Test on newer Qwen models",[155],{"id":156,"by":157,"text":158},"47434173","christianqchung","Why test on Qwen 2.5 when Qwen 3 has been out for about a year, and Qwen 3.5 for a month? My problem with this is ironically entirely vibes based, that for some reason, LLMs love…",{"value":160,"source":56,"evidence":161},"Try skipping layers instead of duplicating them",[162],{"id":163,"by":164,"text":165},"47438244","4bpp","That's interesting, could you point me to some source on these findings? It seems to me that the difference between \"iterative improvement\" as you put it and \"close to the…",[167,177],{"value":168,"source":56,"evidence":169},"Why the logical deduction gain happens and whether it generalises",[170,173],{"id":171,"by":164,"text":172},"47434231","Assuming the benchmarks are sound (rather than capturing a fluke), the provided explanation still does not pass the smell test. As far as I can tell, there is nothing about the…",{"id":174,"by":175,"text":176},"47439055","the_harpia_io","the \"reasoning circuits\" framing is doing a lot of work here. tidy story, fits the result, but I'm not sure it explains anything - you duplicated layers and got better scores,…",{"value":178,"source":56,"evidence":179},"How to run these models on AMD GPUs",[180],{"id":181,"by":182,"text":183},"47438507","gukoff","How do you run these models on AMD GPUs?",[185,189,196],{"value":186,"source":56,"evidence":187},"David Ng's RYS method",[188],{"id":59,"by":60,"text":61},{"value":190,"source":56,"evidence":191},"Depth Up-Scaling (Solar 10.7B)",[192],{"id":193,"by":194,"text":195},"47433396","woadwarrior01","Reminds me of Solar 10.7B, which was a very good model for its size ~2 year ago and the \"Depth Up-Scaling\" technique behind it. Although, that involved continued training after…",{"value":197,"source":56,"evidence":198},"Layer pruning",[199],{"id":200,"by":201,"text":202},"47434357","snats","you can also have removed layers of models and keep the same score in benchmarks [1]. i feel that sometimes a lot of the layers might just be redundant and are not fully needed…",{"comments":204,"answered":42,"key":205},6,[206,209,210],{"id":207,"text":208},"47434040","Fair point on the writing style, I used Claude extensively on this project, including drafting. The experiments and ideas are mine though. On the prior art: you're right that…",{"id":67,"text":68},{"id":77,"text":78},{"commenters":212,"max_depth":204,"maker_replies":204,"first_comment_mins":213,"lifespan_hours":214},55,204,77,[216,218,222,225,226,229,232,234,238,241,245,247,250,252,254,257,258,261,264,267,269,271,274,276,278,280,284,286,289,292,294,296,299,302,305,308,310,311,313,317,319,322,325,329,333,335,336,339,343,345,347,351,353,355,358,362,364,367,370,372,373,376,379,383,385,388,390,392,395,397,400,402,406,410,413,417,419,421,425,427,431,435],[171,31,41,217,164,31],"project\u002Fcritical\u002Fidea",[219,31,31,220,221,31],"47434775","commenter\u002Fneutral\u002Fidea","kirill5pol",[223,31,31,220,224,31],"47437614","ACCount37",[163,31,31,220,164,31],[227,31,31,220,228,31],"47441476","observationist",[230,31,31,220,231,31],"47438154","zozbot234",[233,31,31,220,164,31],"47438260",[235,31,31,236,237,31],"47439191","commenter\u002Fneutral\u002Fnone","SCLeo",[239,31,41,217,240,31],"47438739","jstanley",[242,31,42,243,244,31],"47440457","project\u002Fneutral\u002Fidea","WithinReason",[246,31,31,236,240,31],"47442789",[248,31,31,236,249,31],"47439535","getnormality",[251,31,31,220,164,31],"47443687",[253,31,31,236,244,31],"47440118",[255,31,31,220,256,31],"47442550","visarga",[59,41,41,217,60,31],[259,41,31,236,260,31],"47437324","regularfry",[262,41,31,236,263,31],"47437644","yorwba",[265,41,18,266,240,31],"47438718","project\u002Fpositive\u002Fidea",[268,41,31,236,260,31],"47438909",[270,41,31,236,60,31],"47439324",[272,41,42,243,273,31],"47442740","gavinray",[275,41,31,236,260,31],"47453631",[114,277,41,217,115,31],2,[207,277,31,279,11,41],"maker\u002Fneutral\u002Fidea",[281,277,31,282,283,31],"47435727","other\u002Fneutral\u002Fnone","abhikul0",[285,277,31,282,115,31],"47439135",[287,277,42,288,283,31],"47440567","project\u002Fneutral\u002Fexecution",[290,42,42,243,291,31],"47433868","taliesinb",[293,42,42,243,291,31],"47433916",[295,42,31,282,291,31],"47433960",[297,42,42,243,298,31],"47434291","awwaiid",[300,36,42,288,301,31],"47434535","kgeist",[303,36,42,243,304,31],"47436515","gmerc",[306,36,18,307,95,31],"47438520","project\u002Fpositive\u002Fnone",[118,18,42,309,119,31],"project\u002Fneutral\u002Fnone",[193,204,42,243,194,31],[156,312,42,309,157,31],7,[314,315,42,243,316,31],"47433538",8,"SyzygyRhythm",[67,315,31,318,11,41],"maker\u002Fneutral\u002Fexecution",[320,315,42,243,321,31],"47436503","smusamashah",[323,315,42,243,324,31],"47436629","imtringued",[326,315,42,327,328,31],"47438409","project\u002Fneutral\u002Fperformance","BoredomIsFun",[330,32,31,331,332,31],"47438449","other\u002Fneutral\u002Fidea","hackpert",[98,334,18,266,99,31],10,[77,334,31,318,11,41],[337,334,31,331,338,31],"47437287","skerit",[340,341,42,243,342,31],"47437434",11,"Lerc",[344,341,31,220,60,31],"47439093",[346,341,31,220,342,31],"47442957",[348,349,42,243,350,31],"47436865",12,"Imanari",[352,349,42,243,231,31],"47438265",[200,354,42,327,201,31],13,[133,356,41,357,134,31],14,"project\u002Fcritical\u002Fperformance",[359,360,42,243,361,31],"47438460",15,"puppykito",[129,363,41,357,130,31],16,[365,363,31,282,366,31],"47434656","3eb7988a1663",[368,369,42,243,249,31],"47434657",17,[149,371,42,327,150,31],18,[52,371,31,318,11,41],[374,371,31,282,375,31],"47474040","lostmsu",[377,371,31,378,11,41],"47449413","project\u002Fpositive\u002Fperformance",[380,381,31,282,382,31],"47433807",19,"sampton",[125,384,41,357,126,31],20,[386,384,42,327,387,31],"47433664","ekianjo",[389,384,41,357,126,31],"47433681",[83,384,31,391,11,41],"maker\u002Fneutral\u002Fdocs",[393,394,31,282,328,31],"47438462",21,[181,396,31,282,182,31],22,[398,396,31,282,399,31],"47443274","DiabloD3",[94,401,18,266,95,31],23,[403,404,42,243,405,31],"47452081",24,"seertaak",[407,408,42,243,409,31],"47445076",25,"jacquesm",[411,412,42,243,328,31],"47438382",26,[414,30,31,415,416,31],"47436571","other\u002Fneutral\u002Fpricing","dhsorens30",[418,30,31,282,95,31],"47438552",[105,420,18,266,106,31],28,[422,423,31,282,424,31],"47463365",29,"aimarketintel",[174,426,41,217,175,31],30,[428,426,31,429,430,31],"47439081","commenter\u002Fcritical\u002Fnone","samlinnfer",[432,433,31,415,434,31],"47434617",31,"realaliarain74",[436,437,42,438,439,31],"47434111",32,"project\u002Fneutral\u002Fdocs","Singlaw",[],[442,447,452,457,462,466],{"id":443,"name":444,"points":445,"category":446},"47344548","s@",414,"languages",{"id":448,"name":449,"points":450,"category":451},"47378781","Hacker News archive (open-index)",408,"data",{"id":453,"name":454,"points":455,"category":456},"47427299","SVG backgrounds",403,"creative",{"id":458,"name":459,"points":460,"category":461},"47400868","Godogen",337,"dev_tools",{"id":463,"name":464,"points":465,"category":461},"47412569","zeroboot",311,{"id":467,"name":468,"points":469,"category":470},"47428650","Will my flight have Starlink?",276,"consumer_social",false,"v2",{"id":474,"name":5,"line":6,"ai":471,"cat":20,"year":13,"points":14,"comments":15,"rec":27,"s":475,"top":18,"kw":476,"maker":11,"domain":10,"tier":19,"attempts":31,"strip":477,"url":9,"mix":478,"praise":91,"objection":110,"sub":21},47431671,-0.12,[21,24,25,26],"1000100105310333353335033311300110533513",[18,30,31,32],1793991148550]