Phind-CodeLlama-34B
AI
code generation model
SHOW HN2023
40points
6comments
#1,217of 3,109 in 2023
Top 40%of 2023 by points
How HN took it
CuriousMostly questions and neutral comments
17 positive, 48 neutral or asking, 1 mixed, 14 critical, out of 80 comments about the project. Thread score +0.07.
Performance: 6 for, 6 againstExecution: 1 for, 3 againstIdea: 3 for, 0 againstPricing: 3 for, 0 againstDocs: 0 for, 2 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Impressive that an open model beats a large closed model after fine-tuning
From 2 comments
Super impressive. Being able to beat a mega closed source model with an open source LLM and some fine tuning really calls into question the financial viability of these big…
JCM9 on HNThis is impressive, so congratulations. I also think that this is a great example of how open source can simply lead to faster and better acceleration.
lemonlym on HN
Runs locally on consumer hardware and feels usable
From 2 comments
I used the original 34B last night with 4 bit through accelerate and I was absolutely blown away. I got goosebumps because it finally felt like we have models we can run on…
syntaxing on HNIt really is good. Surprisingly it seems to answer instruct-like prompts well! I’ve been using it with Ollama (https://github.com/jmorganca/ollama) with prompts like: ``` ollama…
jmorgan on HN
Objections
HumanEval is a small benchmark and not representative of real engineering work
From 1 comment
Glad this work is happening! That said, HumanEval as the current gold standard for benchmarking models is a crime. The dataset itself is tiny (around 150) examples and all the…
bfogelman on HN
GPT-4 scores as high as 85% on HumanEval, so the comparison is questioned
From 2 comments
GPT-4 as served in the API has been getting 85% on HumanEval (compared to 69.5% claimed here) https://twitter.com/amanrsanger/status/1635751764577361921…
agnokapathetic on HNGlad this work is happening! That said, HumanEval as the current gold standard for benchmarking models is a crime. The dataset itself is tiny (around 150) examples and all the…
bfogelman on HN
Quantized GGUF versions gave poor results
From 2 comments
I actually tried the 4 bit quants (Q4_K_M) and was a bit unimpressed. Switching to Q6_K made a huge difference, but it doesn't fit on my 3090 so it was very slow. And testing on…
redox99 on HNI have a hunch something is broken with the GGUF. I had terrible results using llama cpp as well.
syntaxing on HN
Requests
Publish the training dataset
From 2 comments
Is the dataset open source?
xpl on HNGreat stuff - and training was so cheap too - it would have cost less than $200 on Runpod, and half that price on spot instances. I guess it's time languages other than Python,…
knotty66 on HN
Support for less popular languages such as Pascal and Elixir
From 2 comments
Are you planning to switch to programming-language optimized model inside Phind? So, if a user is asking for something related to Python, that the python-optimized model gets…
Fischgericht on HNQuite promising, and at last it looks like we have pretty decent models that can run locally. It is not perfect; still they fail in the first go to provide a correct answer to…
freehorse on HN
Add the prompt format to the model card
From 1 comment
Could you please add the prompt format to the model card, or at least here?
redox99 on HN
Questions
What hardware is needed to run it locally
From 2 comments
what hardware could one run this locally on? probably need 3090 etc right? finally a reason to upgrade from my m1 max
tomr75 on HNTotally out of my depth here but is there any chance that this could be added to something like Ollama and run on a 32gb or 64gb M series Mac in due course? I’m currently running…
darkteflon on HN
Whether the fine-tuning method and dataset can be reproduced
From 2 comments
Are you planning to switch to programming-language optimized model inside Phind? So, if a user is asking for something related to Python, that the python-optimized model gets…
Fischgericht on HNWhat would the hardware requirement for fine-tuning a Llama 2 ?
xmly on HN
Alternatives named
GPT-4
From 2 comments
GPT-4 as served in the API has been getting 85% on HumanEval (compared to 69.5% claimed here) https://twitter.com/amanrsanger/status/1635751764577361921…
agnokapathetic on HNIve used GPT 4 for pretty much all of my programming needs and the convenience of a 20 dollar subscription taking care of everything and letting me use a LLM without having to…
rikafurude21 on HN
Original CodeLlama
From 2 comments
Glad to see open source models becoming competitive and thanks for sharing the model to community! Lepton AI has hosted this model here free for everyone to give it a try (and…
ccding on HNI tried out the 7B one last night and wasn’t impressed. When asking to write in Python it correctly can write regex to parse a date but when you ask it the same but in TypeScript…
wdb on HN
Llama 2 70B
From 1 comment
llama2 70b is great i have tried it can easily beat gpt4 but llama2 is trained very strictly it i mean llama2 doesnot reply to many questions that gpt4 does! bonus: 100% free Ai…
yasir111 on HN