[{"data":1,"prerenderedAt":508},["ShallowReactive",2],{"project-40067677":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":35,"aspects":36,"strip":44,"post_text":46,"what":47,"said":77,"maker_in_thread":146,"shape":159,"grid":163,"attempts":469,"same_week":470,"maker_shows":218,"maker_hidden":500,"reader_version":501,"card":502},"40067677","Effort","Speeding up LLM inference 2x times (possibly)","title","Show HN: Speeding up LLM inference 2x times (possibly)","https:\u002F\u002Fasciinema.org\u002Fa\u002FpiP22yYwcaohu5cA2gyuv1W61","asciinema.org","kolinko",1713374769,2024,419,114,104,3515,3,"legend","ai","llm inference optimisation","library",[24,25,26,27,28],"llm inference","matrix multiplication","mistral","apple metal","model distillation","curious",0.182,{"positive":32,"neutral":33,"mixed":34,"critical":18},12,31,1,47,{"docs":37,"execution":40,"idea":41,"performance":43},{"positive":38,"critical":39},2,0,{"positive":38,"critical":39},{"positive":42,"critical":34},6,{"positive":34,"critical":34},[45,18,39,45,18,18,18,18,18,45,18,45,18,45,18,18,45,18,18,18,18,18,45,18,18,18,18,34,18,18,45,39,18,39,39,39,39,39,45,39],5,"Here's a project I've been working on for the last few months.\n\nIt's a new (I think) algorithm, that allows to adjust smoothly - and in real time - how many calculations you'd like to do during inference of an LLM model.\n\nIt seems that it's possible to do just 20-25% of weight multiplications instead of all of them, and still get good inference results.\n\nI implemented it to run on M1\u002FM2\u002FM3 GPU. The mmul approximation itself can be pushed to run 2x fast before the quality of output collapses.\n\nThe inference speed is just a bit faster than Llama.cpp's, because the rest of implementation could be",{"problem":48,"audience":54,"pricing":54,"motivation":55,"time_to_build":54,"stack":59,"openness":74,"stage":75,"team_size":76},{"value":49,"source":50,"evidence":51},"LLM inference is slow; it could be sped up by computing only a fraction of weight multiplications","maker",[52],{"id":53},"post",null,{"value":56,"source":50,"evidence":57},"Speed up LLM inference as a new method alongside quantization",[58],{"id":53},[60,66,70],{"value":61,"source":50,"evidence":62},"Swift",[63],{"id":64,"by":11,"text":65},"40073842","Depends on a field. The project from the thread would take me an impossible amount of time without GPT. Even the page itself would take me twice as long to generate - the charts…",{"value":67,"source":50,"evidence":68},"Python",[69],{"id":64,"by":11,"text":65},{"value":71,"source":50,"evidence":72},"Mistral",[73],{"id":53},"open_source","prototype","solo",{"praise":78,"objections":87,"requests":104,"questions":123,"alternatives":133},[79],{"value":80,"source":81,"evidence":82},"Interesting, potentially useful idea for inference speed","commenter",[83],{"id":84,"by":85,"text":86},"40070392","smcleod","Looks like the models are missing on Huggingface, I've logged an issue: https:\u002F\u002Fgithub.com\u002Fkolinko\u002Feffort\u002Fissues\u002F3",[88,97],{"value":89,"source":81,"evidence":90},"Claims are unverified; quality may collapse at Q8 or smaller models",[91,94],{"id":92,"by":11,"text":93},"40070239","Hahah thanks, although I was hoping for a flame to get adrenaline flowing to push through the night :D I also hope there will be an extra knob - or more like knobs, because…",{"id":95,"by":11,"text":96},"40083757","Yeah, more tests are needed. I got some feedback on using KL instead of the token similarity - initial tests seem to show that it is workable (compared to Q8), but not awesomely…",{"value":98,"source":81,"evidence":99},"Inference speed is still slow overall",[100],{"id":101,"by":102,"text":103},"40069768","refulgentis","It's extremely unlikely anyone will take any of this. Quick take: - it was 15x slower than llama.cpp when I used Apple's new proprietary ML framework on my MacBook - So I made it…",[105,114],{"value":106,"source":81,"evidence":107},"Test on Q8 quantization",[108,111],{"id":109,"by":11,"text":110},"40069499","Per VRAM - not much netter, because it still uses all the weights, just not all the time. I mean - it can also load less weights, but quality seems to degrade quick after…",{"id":112,"by":11,"text":113},"40073649","Below Q8 I think it will combine poorly - bucketMul needs to keep a few bits for an index. It can be offset by the fact that the numbers from each bucket are from a limited…",{"value":115,"source":81,"evidence":116},"Better documentation and clearer writeup",[117,120],{"id":118,"by":11,"text":119},"40070421","oh, thanks, I fixed it. No idea what I meant there originally. There are still a few typos on the page, I'll be fixing them tomorrow - it's midnight now, and my mental batteries…",{"id":121,"by":11,"text":122},"40068845","It’s explained in detail here: https:\u002F\u002Fkolinko.github.io\u002Feffort\u002Fequations.html Long story short - I kind of sort them and pick only the top % that would give a highest result.…",[124],{"value":125,"source":81,"evidence":126},"How does it compare to existing sparse and approximation methods",[127,130],{"id":128,"by":11,"text":129},"40068357","Similar theme, but they skip whole neurons, in my case it’s down to a level of single weights. From my experiment, skipping whole neurons (so whole rows\u002Fcolumns of matrixes)…",{"id":131,"by":11,"text":132},"40069903","The whole inference is slow, but it's matrix multiplications that count. They work reliably on all the Macbooks that I tested - at 50% effort it's the same speed as the state of…",[134,140],{"value":135,"source":81,"evidence":136},"Mixture of experts (MoE) models",[137],{"id":138,"by":11,"text":139},"40069007","I’d say that this expands on MoE really - MoE chooses dynamically which groups of weights may be needed, but it’s whole matrixes. Here it’s ingle weights. Also, this works on top…",{"value":141,"source":81,"evidence":142},"Llama.cpp",[143],{"id":144,"by":11,"text":145},"40068005","Hahah, thanks! It was a marathon to get develop this, and I'm glad it reached the front page. The name was proposed by chatgpt :) It claims it doesn't recognise this approach -…",{"comments":147,"answered":148,"key":149},46,16,[150,153,156],{"id":151,"text":152},"40073552","Thanks! The last full month was all about making sure all the research is as replicable and as trustworthy as I can do it. The original implementation was extremely inefficient,…",{"id":154,"text":155},"40074074","True, it will be easier next time because new llms allow to better searching through papers. Having said that - the tradeoff with getting into a new field is that there is 50…",{"id":157,"text":158},"40094950","I found gpt to be awesome in this though - it's good at mapping some fields into others. While working, I kept asking it about certain ideas, and if the idea was known to him, he…",{"commenters":160,"max_depth":161,"maker_replies":147,"first_comment_mins":45,"lifespan_hours":162},56,7,162,[164,168,170,174,175,179,183,186,187,191,194,197,200,204,206,208,210,214,216,220,222,224,226,230,232,235,237,241,243,246,248,251,253,256,260,261,264,265,269,271,275,276,278,282,283,286,288,291,294,296,299,301,305,306,310,312,315,317,321,323,326,330,332,333,335,340,342,346,348,352,356,358,360,362,365,369,371,376,379,382,383,386,390,392,395,397,400,402,405,407,411,414,416,417,419,420,422,425,428,430,433,435,438,439,441,444,446,449,452,455,457,461,463,466],[165,39,45,166,167,39],"40073067","project\u002Fpositive\u002Fdocs","renonce",[151,39,39,169,11,34],"maker\u002Fneutral\u002Fnone",[171,39,34,172,173,39],"40073791","maker\u002Fcritical\u002Fnone","bravura",[154,39,39,169,11,34],[176,39,45,177,178,39],"40078391","maker\u002Fpositive\u002Fexecution","llama_person",[180,39,39,181,182,39],"40107572","other\u002Fneutral\u002Fnone","fennecbutt",[184,39,39,181,185,39],"40088544","3abiton",[157,39,39,169,11,34],[188,39,18,189,190,39],"40075468","project\u002Fneutral\u002Fidea","Macuyiko",[192,39,39,181,193,39],"40085624","krageon",[195,34,45,166,196,39],"40070293","spencerchubb",[198,34,39,199,11,34],"40070709","maker\u002Fneutral\u002Fexecution",[201,38,18,202,203,39],"40068300","project\u002Fneutral\u002Fexecution","bigcat12345678",[205,38,39,169,11,34],"40068319",[207,38,18,202,203,39],"40070699",[209,38,39,202,11,34],"40070813",[211,18,18,212,213,39],"40070944","project\u002Fneutral\u002Fperformance","hatthew",[215,18,39,169,11,34],"40071052",[217,218,18,202,219,39],"40070056",4,"marmaduke",[221,218,39,202,11,34],"40070296",[223,218,18,189,219,39],"40079935",[225,218,39,189,11,34],"40083658",[227,45,18,228,229,39],"40084142","project\u002Fmixed\u002Fperformance","jadeaffenjaeger",[231,45,39,212,11,34],"40095016",[233,42,18,212,234,39],"40069516","globally_unique",[236,42,39,212,11,34],"40069807",[238,161,45,239,240,39],"40067923","project\u002Fpositive\u002Fidea","dartos",[144,161,39,242,11,34],"project\u002Fneutral\u002Fnone",[244,161,18,189,245,39],"40068134","dhruvdh",[247,161,39,189,11,34],"40068214",[249,161,39,181,250,39],"40068590","indymike",[252,161,39,181,11,34],"40068738",[254,161,45,239,255,39],"40069869","0x4139",[257,258,18,212,259,39],"40069052",8,"gsuuon",[109,258,39,212,11,34],[262,258,18,212,263,39],"40070579","AnthonyMouse",[112,258,39,212,11,34],[266,267,45,239,268,39],"40071173",9,"yousif_123123",[270,267,18,189,196,39],"40072310",[272,273,18,189,274,39],"40068275",10,"toisanji",[128,273,39,189,11,34],[277,273,18,212,274,39],"40068503",[279,280,18,189,281,39],"40068696",11,"kanwisher",[121,280,39,189,11,34],[284,280,18,189,285,39],"40069070","throwaway2562",[287,280,39,189,11,34],"40070377",[289,280,45,290,285,39],"40142927","maker\u002Fpositive\u002Fnone",[292,280,18,189,293,39],"40068983","punnerud",[295,280,39,189,11,34],"40073716",[297,32,18,202,298,39],"40069589","bick_nyers",[300,32,39,202,11,34],"40073744",[302,303,18,189,304,39],"40068916",13,"byyoung3",[138,303,39,189,11,34],[307,303,18,308,309,39],"40075285","project\u002Fneutral\u002Fdocs","anentropic",[311,303,39,308,11,34],"40083727",[313,303,18,189,314,39],"40070683","huac",[316,303,39,189,11,34],"40073779",[318,319,18,308,320,39],"40068589",14,"gcy",[118,319,39,322,11,34],"maker\u002Fneutral\u002Fdocs",[324,319,18,308,325,39],"40071799","mijoharas",[327,328,18,212,329,39],"40072617",15,"brrrrrm",[331,328,39,212,11,34],"40073598",[84,148,18,308,85,39],[334,148,39,290,11,34],"40070501",[336,337,45,338,339,39],"40068202",17,"project\u002Fpositive\u002Fperformance","avereveard",[341,337,39,239,11,34],"40068233",[343,344,18,189,345,39],"40072482",18,"wayoverthecloud",[347,344,39,189,11,34],"40073688",[349,350,39,181,351,39],"40068750",19,"saurik",[353,354,18,189,355,39],"40070292",20,"uhlo",[357,354,39,189,11,34],"40070324",[359,354,18,308,355,39],"40070369",[361,354,39,322,11,34],"40073788",[363,354,18,189,364,39],"40074619","mentos",[366,367,18,212,368,39],"40068667",21,"coolvision",[370,367,39,212,11,34],"40068772",[372,373,34,374,375,39],"40070509",22,"project\u002Fcritical\u002Fidea","a2code",[377,373,39,378,11,34],"40070590","commenter\u002Fneutral\u002Fnone",[380,373,39,381,325,39],"40071774","commenter\u002Fneutral\u002Fidea",[95,373,39,378,11,34],[384,373,39,181,385,39],"40071928","addandsubtract",[387,388,18,242,389,39],"40081714",23,"thelastparadise",[391,388,39,189,11,34],"40083739",[393,394,18,308,329,39],"40069364",24,[396,394,39,308,11,34],"40069527",[398,394,45,239,399,39],"40069633","kwikiel",[401,394,39,189,11,34],"40070475",[403,394,45,404,329,39],"40071966","project\u002Fpositive\u002Fexecution",[406,394,39,378,11,34],"40073805",[408,409,39,181,410,39],"40069478",25,"xrd",[412,409,39,181,413,39],"40069639","quadrature",[101,409,34,415,102,39],"project\u002Fcritical\u002Fperformance",[131,409,39,212,11,34],[418,409,45,239,102,39],"40069980",[92,409,39,189,11,34],[421,409,39,378,410,39],"40069986",[423,409,39,181,424,39],"40070526","observationist",[426,409,39,181,427,39],"40071798","oceanplexian",[429,409,39,181,424,39],"40076828",[431,409,39,181,432,39],"40070207","littlestymaar",[434,409,39,181,11,34],"40070695",[436,409,39,181,437,39],"40071368","iwontberude",[64,409,39,169,11,34],[440,409,39,181,432,39],"40074784",[442,409,39,181,443,39],"40072059","Hugsun",[445,409,39,239,11,34],"40069776",[447,409,39,378,448,39],"40071318","Havoc",[450,451,39,169,11,34],"40067727",26,[453,451,39,181,454,39],"40069929","dang",[456,451,39,181,11,34],"40073844",[458,459,45,239,460,39],"40068609",27,"queuebert",[462,459,39,212,11,34],"40068885",[464,459,39,181,465,39],"40069895","HPsquared",[467,459,39,181,468,39],"40069901","LorenDB",[],[471,476,481,486,491,496],{"id":472,"name":473,"points":474,"category":475},"40003710","OptiGap sensor",886,"hardware",{"id":477,"name":478,"points":479,"category":480},"40041198","gypsum",552,"learning_science",{"id":482,"name":483,"points":484,"category":485},"40017180","Mazelit",517,"games",{"id":487,"name":488,"points":489,"category":490},"40033490","markdowndown",441,"productivity",{"id":492,"name":493,"points":494,"category":495},"40028111","index advisor",417,"data",{"id":497,"name":498,"points":499,"category":475},"40014937","Recovery Kit 2",403,false,"v2",{"id":503,"name":5,"line":6,"ai":500,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":504,"top":18,"kw":505,"maker":11,"domain":10,"tier":19,"attempts":39,"strip":506,"url":9,"mix":507,"praise":80,"objection":89,"sub":21},40067677,0.18,[24,25,26,27,28],"5305333335353533533333533331335030000050",[32,33,34,18],1794036653290]