Ollama for Linux
AI
local llm runner
SHOW HN2023
49points
8comments
#1,086of 3,109 in 2023
Top 35%of 2023 by points
How HN took it
CuriousMostly questions and neutral comments
7 positive, 11 neutral or asking, 2 mixed, 2 critical, out of 22 comments about the project. Thread score +0.27.
Execution: 3 for, 0 againstIdea: 2 for, 0 againstDocs: 0 for, 1 againstDesign: 0 for, 1 againstPerformance: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Easy install and setup that works for non-experts
From 3 comments
Ollama is awesome. I am part of a team building a code AI application[1], and we want to give devs the option to run it locally instead of only supporting external LLMs from…
sqs on HNI’ve been using this on my MacBook Pro for the last couple weeks and want to say thank you! As a solutions developer not so much interested in training models but leveraging them…
biddit on HNThis is a huge deal, congrats! We've had a ton of users asking how to run their own LLMs on Linux and the unfortunate answer was always that the existing options were slightly…
sestinj on HN
Useful REST API for building apps on local models
From 2 comments
I'm actually using Ollama for it's Rest API endpoint. Llama.cpp does now have it's server implementation. Unfortunately they do have different endpoints and behave a little…
mgreg on HNOllama is awesome. I am part of a team building a code AI application[1], and we want to give devs the option to run it locally instead of only supporting external LLMs from…
sqs on HN
Modelfile lets custom model personas be created easily
From 2 comments
this is awesome, congrats on an amazingly useful release! for those that haven't used ollama, being able to specify how a model behaves via a "modelfile" is pretty darned…
jerrysievert on HNthey are just prompted differently - you can choose the model and a prompt, and ollama presents it as its own "model", so instead of `llama2:7b`, it gets presented as…
jerrysievert on HN
Objections
Too similar to llama.cpp wrappers; marketing overstates what it adds
From 2 comments
I totally agree. A lot of these programs make huge marketing claims that are really just features of Llama.cpp, not a spec from their implementation. It's fine, but the wording…
smoldesu on HNOh, this is a llama.cpp frontend. Y'all should have lead with that! I saw this on HN before, but I thought it was another from-scratch llama implementation... Which is fine, but…
brucethemoose2 on HN
Custom GGUF format adds lock-in
From 1 comment
"You guys are doing the lord's work here"? How? By forcing users into a custom model serialization format (GGUF) that is claimed to literally contain "magic"? Dark lord, maybe.
factibicongue on HN
Cannot control model cache location
From 1 comment
Ollama is awesome. I am however still waiting for the support of controlling the model cache location: https://github.com/jmorganca/ollama/issues/153 This is either for backup…
WiSaGaN on HN
Requests
Support for configuring model cache location
From 1 comment
Ollama is awesome. I am however still waiting for the support of controlling the model cache location: https://github.com/jmorganca/ollama/issues/153 This is either for backup…
WiSaGaN on HN
Ability to run multiple models simultaneously
From 1 comment
Very cool and thanks for sharing. To me a killer feature would be easily running different models simultaneously such as one for embeddings and another for completion (e.g.…
mgreg on HN
Speculative sampling support
From 1 comment
This is a huge deal, congrats! We've had a ton of users asking how to run their own LLMs on Linux and the unfortunate answer was always that the existing options were slightly…
sestinj on HN
Questions
How does performance compare to vLLM or exllama?
From 2 comments
How does this compare to vLLM or exllama? Can it run llama2 30B on one 3090 24G or 70B on two 3090 24G? https://github.com/vllm-project/vllm https://github.com/turboderp/exllama…
aftbit on HNAmazing! I use text-generation-webui to play with LLMs, but was always jealous of this much simpler interface. Somewhat related note -- does anyone know what are the performance…
kelvie on HN
How does WSL2 access the GPU?
From 1 comment
How is WSL2 able to work with GPUs?
politelemon on HN
Alternatives named
llama.cpp
From 2 comments
> run their own LLMs on Linux and the unfortunate answer was always that the existing options were slightly complicate What about https://github.com/ggerganov/llama.cpp ? It…
dorfsmay on HNOh, this is a llama.cpp frontend. Y'all should have lead with that! I saw this on HN before, but I thought it was another from-scratch llama implementation... Which is fine, but…
brucethemoose2 on HN
vLLM
From 2 comments
Congrats on the launch! I'll give it a try. I've been using vLLM on Linux so far but have wanted to be able to use a ggml backend - have you done any perf comparisons?
binarymax on HNvLLM has far more throughput with batching, but that is also a WIP feature for llama.cpp. Current standing is something like: - vLLM is the fastest overall with batching, and has…
brucethemoose2 on HN
exllama
From 2 comments
Amazing! I use text-generation-webui to play with LLMs, but was always jealous of this much simpler interface. Somewhat related note -- does anyone know what are the performance…
kelvie on HNexLLAMAv2 and GPTQ are different implementations, and currently exLLAMA is definitely the best for discrete Nvidia (and AMD?) GPUs. Its faster and uses less VRAM for the same…
brucethemoose2 on HN
text-generation-webui
From 1 comment
Amazing! I use text-generation-webui to play with LLMs, but was always jealous of this much simpler interface. Somewhat related note -- does anyone know what are the performance…
kelvie on HN
gpt4all
From 1 comment
Or https://gpt4all.io/index.html Though doesn't currently support GPU.
vorticalbox on HN