Effort
AI
llm inference optimisation
SHOW HN2024
How HN took it
12 positive, 31 neutral or asking, 1 mixed, 3 critical, out of 47 comments about the project. Thread score +0.18.
Idea: 6 for, 1 againstDocs: 2 for, 0 againstExecution: 2 for, 0 againstPerformance: 1 for, 1 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Interesting, potentially useful idea for inference speed
From 1 comment
Looks like the models are missing on Huggingface, I've logged an issue: https://github.com/kolinko/effort/issues/3
smcleod on HN
Objections
Claims are unverified; quality may collapse at Q8 or smaller models
From 2 comments
Hahah thanks, although I was hoping for a flame to get adrenaline flowing to push through the night :D I also hope there will be an extra knob - or more like knobs, because…
kolinko on HNYeah, more tests are needed. I got some feedback on using KL instead of the token similarity - initial tests seem to show that it is workable (compared to Q8), but not awesomely…
kolinko on HN
Inference speed is still slow overall
From 1 comment
It's extremely unlikely anyone will take any of this. Quick take: - it was 15x slower than llama.cpp when I used Apple's new proprietary ML framework on my MacBook - So I made it…
refulgentis on HN
Requests
Test on Q8 quantization
From 2 comments
Per VRAM - not much netter, because it still uses all the weights, just not all the time. I mean - it can also load less weights, but quality seems to degrade quick after…
kolinko on HNBelow Q8 I think it will combine poorly - bucketMul needs to keep a few bits for an index. It can be offset by the fact that the numbers from each bucket are from a limited…
kolinko on HN
Better documentation and clearer writeup
From 2 comments
oh, thanks, I fixed it. No idea what I meant there originally. There are still a few typos on the page, I'll be fixing them tomorrow - it's midnight now, and my mental batteries…
kolinko on HNIt’s explained in detail here: https://kolinko.github.io/effort/equations.html Long story short - I kind of sort them and pick only the top % that would give a highest result.…
kolinko on HN
Questions
How does it compare to existing sparse and approximation methods
From 2 comments
Similar theme, but they skip whole neurons, in my case it’s down to a level of single weights. From my experiment, skipping whole neurons (so whole rows/columns of matrixes)…
kolinko on HNThe whole inference is slow, but it's matrix multiplications that count. They work reliably on all the Macbooks that I tested - at 50% effort it's the same speed as the state of…
kolinko on HN
Alternatives named
Mixture of experts (MoE) models
From 1 comment
I’d say that this expands on MoE really - MoE chooses dynamically which groups of weights may be needed, but it’s whole matrixes. Here it’s ingle weights. Also, this works on top…
kolinko on HN
Llama.cpp
From 1 comment
Hahah, thanks! It was a marathon to get develop this, and I'm glad it reached the front page. The name was proposed by chatgpt :) It claims it doesn't recognise this approach -…
kolinko on HN