[{"data":1,"prerenderedAt":282},["ShallowReactive",2],{"project-49242475":3},{"id":4,"name":5,"line":5,"line_source":6,"title":7,"url":8,"domain":9,"maker":10,"posted_at":11,"year":12,"points":13,"comments":14,"rank_in_year":15,"shown_in_year":16,"top":17,"tier":18,"category":19,"subcategory":20,"format":21,"keywords":22,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":33,"aspects":34,"strip":41,"post_text":42,"what":43,"said":65,"maker_in_thread":150,"shape":162,"grid":165,"attempts":245,"same_week":246,"maker_shows":177,"maker_hidden":274,"reader_version":275,"card":276},"49242475","A tiny LLM running at 21,000 tok\u002Fs on a $250 FPGA (Live Demo)","title","Show HN: A tiny LLM running at 21,000 tok\u002Fs on a $250 FPGA (Live Demo)","https:\u002F\u002Fwww.mikeayles.com\u002Fblog\u002Fon-chip-llm-kv260\u002F","mikeayles.com","mikeayles",1786362727,2026,79,33,851,4366,20,"notable","ai","on-chip llm inference","hardware",[23,24,25,26,27],"fpga inference","on-chip llm","kv260","sram inference","tokens per second","loved",0.269,{"positive":31,"neutral":31,"mixed":32,"critical":32},5,1,12,{"idea":35,"execution":37,"performance":39,"design":40},{"positive":36,"critical":32},3,{"positive":38,"critical":38},0,{"positive":38,"critical":38},{"positive":32,"critical":38},[31,38,36,38,31,38,36,36,36,38,38,38,38,38,38,38,38,38,38,38,38,38,31,36,38,36,38,38,31,32,38,38,31],null,{"problem":44,"audience":42,"pricing":42,"motivation":51,"time_to_build":42,"stack":55,"openness":63,"stage":64,"team_size":63},{"value":45,"source":46,"evidence":47},"Shows LLM inference with all weights resident in on-chip SRAM, avoiding DRAM fetches","maker",[48],{"id":49,"by":10,"text":50},"49242900","I started this about 10 weeks ago when the Taalas chatjimmy demo first did the rounds, Llama 8B baked into custom silicon, 17k tok\u002Fs for a single user. Their whole thesis is that…",{"value":52,"source":46,"evidence":53},"Explore how far the on-chip weights idea (inspired by Taalas) stretches on consumer hardware",[54],{"id":49,"by":10,"text":50},[56],{"value":57,"source":46,"evidence":58},"FPGA (KV260 board)",[59,62],{"id":60,"by":10,"text":61},"49255759","Good question. short version: it's a shared wall, not a per-request bump. one KV260, one model on the fabric at a time, so the rotation is global. but the ~25s is mostly…",{"id":49,"by":10,"text":50},"unknown","launched",{"praise":66,"objections":86,"requests":116,"questions":129,"alternatives":141},[67,79],{"value":68,"source":69,"evidence":70},"Conceptually interesting and well documented","commenter",[71,75],{"id":72,"by":73,"text":74},"49253725","tomex","Crazy how little traction this kind of project gets on here. Posts about squeezing a 1+T model to seconds\u002Ftoken and you have this wave of optimism like \"It's the effort that…",{"id":76,"by":77,"text":78},"49324293","gbxk","Amazing project, I love it. How about using these Cactus models? Would it make sense for you to collab with those guys (1) for you to design a cheap but improved,…",{"value":80,"source":69,"evidence":81},"Live demo is great",[82],{"id":83,"by":84,"text":85},"49267998","anitil","This is so cool, and having a live demo is just chefs kiss",[87,98,109],{"value":88,"source":69,"evidence":89},"Model output too small to be useful; gibberish in chat",[90,94],{"id":91,"by":92,"text":93},"49248640","RetroTechie","Very nice indeed. Model weights in RAM blocks distributed all over a big FPGA: should be super helpful at minimizing RAM bandwidth bottlenecks. To say nothing of latency. But…",{"id":95,"by":96,"text":97},"49251725","serf","conceptually it's a cool idea. practically the results seem about as coherent as ``` import random; print(random.choice(list(my_dict))) ``` ..but way slower is there a practical…",{"value":99,"source":69,"evidence":100},"FPGAs are not power efficient compared to GPUs or ASICs",[101,105],{"id":102,"by":103,"text":104},"49246757","cgyvbunji","FPGAs are not power efficient at all vs GPUs and ASICs anyway, which is going to be especially true when they are fully saturated by LLM inference.",{"id":106,"by":107,"text":108},"49253510","adrian_b","Nothing can be as power efficient as an ASIC, which is designed for a specific purpose, instead of being a programmable device intended to be suitable for a large class of…",{"value":110,"source":69,"evidence":111},"FPGA design is hard to learn and niche",[112],{"id":113,"by":114,"text":115},"49246647","stevefan1999","But the problem is not that your model is fast. Sure, you can go ASIC and go even faster, but the thing around GPU is that they scale well for both training and inference, and…",[117,122],{"value":118,"source":69,"evidence":119},"Try a bigger FPGA or a larger model",[120,121],{"id":91,"by":92,"text":93},{"id":76,"by":77,"text":78},{"value":123,"source":69,"evidence":124},"Partial reconfiguration to avoid full stalls during model swaps",[125],{"id":126,"by":127,"text":128},"49249625","useiris","does the reflash actually stall every live connection, or just the ones whose request lands during that window? if the whole board goes dark for the full ~25s while any request…",[130,134],{"value":131,"source":69,"evidence":132},"Is there a practical use for a model this small?",[133],{"id":95,"by":96,"text":97},{"value":135,"source":69,"evidence":136},"What is per-user latency under many concurrent connections?",[137],{"id":138,"by":139,"text":140},"49243929","haeseong","I didn't expect the 2,000 connection sweep to stay flat, since all of them are sharing one stream. What does per user latency look like at that end of the sweep?",[142,146],{"value":143,"source":46,"evidence":144},"Taalas chip demo (Llama 8B in custom silicon)",[145],{"id":49,"by":10,"text":50},{"value":147,"source":69,"evidence":148},"Cactus compute runtime",[149],{"id":76,"by":77,"text":78},{"comments":151,"answered":36,"key":152},6,[153,156,159],{"id":154,"text":155},"49255172","Appreciated. and yeah, agreed the interesting comparison isn't \"is this scalable as-is\" (it isn't, PL fabric, cost, power), it's that the CIM \u002F analog-compute startups you…",{"id":157,"text":158},"49255033","Just a quick test to see if i'm back",{"id":160,"text":161},"49244312","You're correct, the flat line is aggregate only. the fabric is saturated from a few dozen active clients onward, so extra connections can't buy throughput, they just queue.…",{"commenters":17,"max_depth":151,"maker_replies":151,"first_comment_mins":163,"lifespan_hours":164},52,154,[166,168,170,174,175,179,183,185,188,191,193,195,197,198,200,202,204,206,208,211,215,218,220,221,223,225,226,227,230,233,236,239,241],[72,38,31,167,73,38],"project\u002Fpositive\u002Fidea",[154,38,38,169,10,32],"project\u002Fneutral\u002Fidea",[171,32,36,172,173,38],"49247272","maker\u002Fneutral\u002Fnone","dahnhiller",[157,32,38,172,10,32],[176,177,31,167,178,38],"49246939",2,"peter_d_sherman",[180,177,38,181,182,38],"49247189","commenter\u002Fneutral\u002Fnone","M4R5H4LL",[91,177,36,184,92,38],"project\u002Fmixed\u002Fexecution",[186,177,36,169,187,38],"49269831","imtringued",[189,36,36,169,190,38],"49263700","variadix",[113,192,38,181,114,38],4,[102,192,38,194,103,38],"commenter\u002Fneutral\u002Fperformance",[196,192,38,181,114,38],"49247652",[106,192,38,194,107,38],[199,192,38,194,103,38],"49253786",[201,192,38,181,107,38],"49259004",[203,192,38,181,187,38],"49269746",[205,192,38,181,103,38],"49308277",[207,192,38,194,187,38],"49269002",[209,192,38,181,210,38],"49252338","IshKebab",[212,192,38,213,214,38],"49253697","other\u002Fneutral\u002Fdocs","avmich",[216,192,38,217,84,38],"49267989","other\u002Fneutral\u002Fnone",[219,192,38,217,187,38],"49269817",[76,31,31,167,77,38],[138,151,36,222,139,38],"project\u002Fneutral\u002Fperformance",[160,151,38,224,10,32],"maker\u002Fneutral\u002Fperformance",[126,151,36,222,127,38],[60,151,38,224,10,32],[228,151,38,217,229,38],"49247367","all2",[83,231,31,232,84,38],7,"project\u002Fpositive\u002Fdesign",[95,234,32,235,96,38],8,"project\u002Fcritical\u002Fidea",[237,234,38,238,10,32],"49255328","maker\u002Fneutral\u002Fidea",[49,240,38,169,10,32],9,[242,240,31,243,244,38],"49255900","maker\u002Fpositive\u002Fnone","antonvs",[],[247,251,256,261,265,270],{"id":248,"name":249,"points":250,"category":19},"49246804","Needle2",537,{"id":252,"name":253,"points":254,"category":255},"49226623","Photosynthesis",329,"creative",{"id":257,"name":258,"points":259,"category":260},"49270040","Woxi",314,"languages",{"id":262,"name":263,"points":264,"category":21},"49197137","Altar II",306,{"id":266,"name":267,"points":268,"category":269},"49251179","everycube.alen.is",295,"games",{"id":271,"name":272,"points":273,"category":255},"49246366","Eigendrum",243,false,"v2",{"id":277,"name":5,"line":5,"ai":274,"cat":19,"year":12,"points":13,"comments":14,"rec":28,"s":278,"top":17,"kw":279,"maker":10,"domain":9,"tier":18,"attempts":38,"strip":280,"url":8,"mix":281,"praise":68,"objection":88,"sub":20},49242475,0.27,[23,24,25,26,27],"503050333000000000000053030051005",[31,31,32,32],1794036707499]