[{"data":1,"prerenderedAt":349},["ShallowReactive",2],{"project-44009321":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":36,"aspects":37,"strip":45,"post_text":47,"what":48,"said":91,"maker_in_thread":210,"shape":220,"grid":224,"attempts":313,"same_week":314,"maker_shows":32,"maker_hidden":341,"reader_version":342,"card":343},"44009321","KVSplit","Run 2-3x longer contexts on Apple Silicon","title","Show HN: KVSplit – Run 2-3x longer contexts on Apple Silicon","https:\u002F\u002Fgithub.com\u002Fdipampaul17\u002FKVSplit","github.com","dipampaul17",1747425898,2025,272,40,206,3837,6,"legend","ai","llm inference optimization","cli",[24,25,26,27,28],"kv cache quantization","llama.cpp","apple silicon","local llm","context length","curious",-0.209,{"positive":32,"neutral":33,"mixed":34,"critical":35},2,18,0,12,32,{"docs":38,"idea":39,"execution":42,"performance":43,"privacy":44},{"positive":34,"critical":34},{"positive":40,"critical":41},1,3,{"positive":34,"critical":41},{"positive":34,"critical":32},{"positive":34,"critical":40},[41,34,46,41,40,41,41,41,40,41,34,41,34,41,40,40,40,40,41,34,41,46,41,34,41,41,41,34,40,41,34,40,41,40,34,41,41,40,40,40],5,"I discovered that in LLM inference, keys and values in the KV cache have very different quantization sensitivities. Keys need higher precision than values to maintain quality.\n\nI patched llama.cpp to enable different bit-widths for keys vs. values on Apple Silicon. The results are surprising:\n\n- K8V4 (8-bit keys, 4-bit values): 59% memory reduction with only 0.86% perplexity loss - K4V8 (4-bit keys, 8-bit values): 59% memory reduction but 6.06% perplexity loss - The configurations use the same number of bits, but K8V4 is 7× better for quality\n\nThis means you can run LLMs with 2-3× longer conte",{"problem":49,"audience":55,"pricing":60,"motivation":61,"time_to_build":60,"stack":68,"openness":88,"stage":89,"team_size":90},{"value":50,"source":51,"evidence":52},"Long contexts on Apple Silicon consume too much memory for the KV cache","maker",[53],{"id":54},"post",{"value":56,"source":57,"evidence":58},"People running local LLMs on Apple Silicon Macs","inferred",[59],{"id":54},null,{"value":62,"source":51,"evidence":63},"Maker found keys and values have different quantization sensitivities and wanted longer contexts on Mac",[64,65],{"id":54},{"id":66,"by":11,"text":67},"44011484","Great question about the intuition! The difference comes from the core roles these components play in attention. Keys determine which tokens to attend to - they create the actual…",[69,76,79,83],{"value":70,"source":57,"evidence":71},"C++",[72],{"id":73,"by":74,"text":75},"44012200","Aurornis","I finally had time to read the code. The patch is unnecessary because this functionality has been in llama.cpp since 2023 if I understand this PR correctly:…",{"value":25,"source":51,"evidence":77},[78],{"id":54},{"value":80,"source":51,"evidence":81},"Metal",[82],{"id":54},{"value":84,"source":85,"evidence":86},"Shell (install script)","commenter",[87],{"id":73,"by":74,"text":75},"open_source","prototype","unknown",{"praise":92,"objections":104,"requests":146,"questions":175,"alternatives":200},[93],{"value":94,"source":85,"evidence":95},"Interesting idea about asymmetric key\u002Fvalue precision",[96,100],{"id":97,"by":98,"text":99},"44009571","nico","Great work. This seems very interesting, but I need something slightly more high level to relate to it Will it just allow me to run let’s say a model with a 2048 token context…",{"id":101,"by":102,"text":103},"44010545","3abiton","This is a brilliant idea, and initiative. Does this also apply to GPUs? And I assume should be compatible with other quantization techniques, albeit they probably require their…",[105,120,128,138],{"value":106,"source":85,"evidence":107},"Same functionality already exists via --cache-type-k and --cache-type-v flags",[108,112,116,119],{"id":109,"by":110,"text":111},"44009833","segmondy","you can do this already with -ctk and -ctv, why would anyone need this? -ctk, --cache-type-k TYPE KV cache data type for K ``` allowed values: f32, f16, bf16, q8_0, q4_0, q4_1,…",{"id":113,"by":114,"text":115},"44010621","therealsmith","Am I missing something? As far as I can see this patch does nothing except add new options that replicate the functionality of the existing --cache-type-k and --cache-type-v…",{"id":117,"by":74,"text":118},"44012275","> A note, your install script appears to still have a placeholder at the \"apply patch\" step. A suggestion, might be more user-friendly to fork llama.cpp and then include that as…",{"id":73,"by":74,"text":75},{"value":121,"source":85,"evidence":122},"Install script applies a patch that is unnecessary and looks suspicious",[123,124,125],{"id":73,"by":74,"text":75},{"id":117,"by":74,"text":118},{"id":126,"by":74,"text":127},"44014427","After looking more, this is definitely an AI driven attempt to game the system. It’s too bad the comments calling it out earlier were downvoted away. The first one was downvoted…",{"value":129,"source":85,"evidence":130},"0.86% perplexity loss is significant at short context",[131,135],{"id":132,"by":133,"text":134},"44009960","smcleod","+0.86% perplexity it's quite a bit at such a small context size though isn't it? How is it at more reasonable context sizes like 64-128k?",{"id":136,"by":133,"text":137},"44010906","Yeah I get that, that's what we yse k\u002Fv cache quantisation for now which has a lower impact on PPL than this unless I'm missing something?",{"value":139,"source":85,"evidence":140},"Maker's GitHub activity looks like LLM-generated star farming",[141,145],{"id":142,"by":143,"text":144},"44013055","imiric","Finally someone making sense. The fact this project works by applying patches instead of forking the original project and committing changes should alone be reason for concern.…",{"id":126,"by":74,"text":127},[147,157,164,168],{"value":148,"source":85,"evidence":149},"Explain how this differs from existing cache-type flags",[150,154],{"id":151,"by":152,"text":153},"44009778","ondra","Is this any different from using --cache-type-k and --cache-type-v?",{"id":155,"by":156,"text":153},"44011524","hiatus",{"value":158,"source":85,"evidence":159},"Fork llama.cpp instead of patch-apply install script",[160],{"id":161,"by":162,"text":163},"44009741","matheist","Looks interesting! Is there any intuition for why this should be the case? Did you discover it via that intuition, or just random experimentation? A note, your install script…",{"value":165,"source":85,"evidence":166},"Benchmarks at longer contexts such as 64-128K",[167],{"id":132,"by":133,"text":134},{"value":169,"source":85,"evidence":170},"Support for MLX",[171],{"id":172,"by":173,"text":174},"44009732","behnamoh","Is this patch possible to do on MLX? I'm getting better speeds on MLX. That, combined with your approach, would finally let Mac users have long conversations at usable speeds.",[176,183,193],{"value":177,"source":85,"evidence":178},"Does this work on GGUF models without conversion?",[179],{"id":180,"by":181,"text":182},"44009632","badmonster","I'm curious: is it possible to apply differentiated KV quantization (like K8V4) to models after they're already converted to .gguf format, or does this require rebuilding the…",{"value":184,"source":85,"evidence":185},"Does it speed up generation or only save memory?",[186,190],{"id":187,"by":188,"text":189},"44010883","zmmmmm","Amazing! Curious, what happens to performance? I assume you still pay the same performance price for longer context, even if you can now fit it in memory.",{"id":191,"by":98,"text":192},"44009868","Will that make the model run\u002Ffeel faster? I can run models with 30-40b parameters on my computer, but they feel a lot slower than the 1-7b ones So would this make the 30-40b…",{"value":194,"source":85,"evidence":195},"Does it require flash attention?",[196],{"id":197,"by":198,"text":199},"44011299","fennecbutt","I thought flash attention was required for quantised KV?",[201,206],{"value":202,"source":85,"evidence":203},"llama.cpp --cache-type-k and --cache-type-v flags",[204,205],{"id":109,"by":110,"text":111},{"id":113,"by":114,"text":115},{"value":207,"source":85,"evidence":208},"MLX",[209],{"id":172,"by":173,"text":174},{"comments":211,"answered":18,"key":212},7,[213,214,217],{"id":66,"text":67},{"id":215,"text":216},"44009655","Yes, that's one of the key benefits - KVSplit works with any existing .gguf model without requiring reconstruction or special conversion. The quantization happens at runtime on…",{"id":218,"text":219},"44011509","The memory savings from KVSplit scale proportionally with context length, so higher-RAM Macs (64GB\u002F128GB) benefit even more in absolute terms. On a 128GB Mac Studio, you could…",{"commenters":221,"max_depth":46,"maker_replies":211,"first_comment_mins":222,"lifespan_hours":223},24,30,21,[225,227,229,232,236,238,239,242,246,248,249,252,254,256,257,259,261,263,264,267,269,270,274,275,278,281,282,284,286,290,292,294,297,300,301,303,305,307,310,312],[161,34,41,226,162,34],"project\u002Fneutral\u002Fdocs",[66,34,34,228,11,40],"maker\u002Fpositive\u002Fidea",[230,34,46,228,231,34],"44011553","gervwyk",[233,34,41,234,235,34],"44012010","project\u002Fneutral\u002Fidea","vlovich123",[117,34,40,237,74,34],"project\u002Fcritical\u002Fexecution",[151,40,41,234,152,34],[240,40,41,234,241,34],"44010421","azinman2",[243,40,41,244,245,34],"44011977","project\u002Fneutral\u002Fperformance","landl0rd",[247,40,40,237,74,34],"44014477",[172,32,41,244,173,34],[250,32,34,251,245,34],"44011964","other\u002Fneutral\u002Fnone",[180,41,41,253,181,34],"project\u002Fneutral\u002Fexecution",[215,41,34,255,11,40],"maker\u002Fneutral\u002Fexecution",[197,41,41,253,198,34],[73,258,40,237,74,34],4,[142,258,40,260,143,34],"maker\u002Fcritical\u002Fnone",[262,258,40,260,114,34],"44013769",[126,258,40,260,74,34],[265,46,41,244,266,34],"44009707","entrepy123",[218,46,34,268,11,40],"maker\u002Fneutral\u002Fperformance",[155,46,41,234,156,34],[271,46,46,272,273,34],"44011537","maker\u002Fpositive\u002Fnone","andrewmcwatters",[97,18,41,234,98,34],[276,18,34,277,11,40],"44009665","maker\u002Fneutral\u002Fidea",[279,18,41,234,280,34],"44009726","kmacdough",[191,18,41,244,98,34],[283,18,41,244,198,34],"44011322",[285,18,34,268,11,40],"44011559",[287,211,40,288,289,34],"44015519","project\u002Fcritical\u002Fprivacy","tough",[101,291,41,234,102,34],8,[293,291,34,277,11,40],"44011539",[132,295,40,296,133,34],9,"project\u002Fcritical\u002Fperformance",[298,295,41,234,299,34],"44010430","nomel",[136,295,40,296,133,34],[302,295,34,268,11,40],"44011526",[187,304,41,244,188,34],10,[306,304,41,244,198,34],"44011311",[113,308,40,309,114,34],11,"project\u002Fcritical\u002Fidea",[311,308,40,309,74,34],"44011122",[109,35,40,309,110,34],[],[315,320,325,329,333,337],{"id":316,"name":317,"points":318,"category":319},"44022353","Voynich NLP analysis",381,"learning_science",{"id":321,"name":322,"points":323,"category":324},"44042371","90s.dev",352,"dev_tools",{"id":326,"name":327,"points":328,"category":324},"44042490","Jwno",297,{"id":330,"name":331,"points":332,"category":324},"44006231","Erlang-RED",243,{"id":334,"name":335,"points":336,"category":324},"44023144","Vaev",233,{"id":338,"name":339,"points":340,"category":324},"44020734","Hardtime.nvim",214,false,"v2",{"id":344,"name":5,"line":6,"ai":341,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":345,"top":18,"kw":346,"maker":11,"domain":10,"tier":19,"attempts":34,"strip":347,"url":9,"mix":348,"praise":94,"objection":106,"sub":21},44009321,-0.21,[24,25,26,27,28],"3053133313030311113035303330130131033111",[32,33,34,35],1794036674951]