Ollama
AI
local llm runner
SHOW HN2023
284points
94comments
#164of 3,109 in 2023
Top 6%of 2023 by points
How HN took it
CuriousMostly questions and neutral comments
12 positive, 37 neutral or asking, 0 mixed, 14 critical, out of 63 comments about the project. Thread score -0.11.
Idea: 6 for, 3 againstPerformance: 2 for, 6 againstExecution: 2 for, 2 againstPrivacy: 1 for, 2 againstDesign: 0 for, 1 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Simple setup makes local LLMs approachable
From 2 comments
I've been wanting to try out local LLMs for a while but have always had the feeling that I would end up wasting my time. I imagined that I would have to spend a perfectly good…
iamdamian on HNWhoa, this is really neat. Does this mean that I can essentially try any LLM on my local machine?
willietran on HN
Modelfile idea for packaging models like Docker images
From 2 comments
Shout out to Jeff and Michael for being some of the sharpest product thinkers I've had the pleasure to work with. I've been playing with this lately and it's been loads of fun…
zenlikethat on HNOne fun use case of local models is creating "characters" or companions (à la character.ai and other tools). It's early but for those interested, this project has a way to do…
jmorgan on HN
Objections
Not clearly different from llama.cpp wrappers
From 2 comments
What does this add over llama.cpp? Is it just an "easier" way to setup llama.cpp locally? If so, I don't really get it, because setting up llama.cpp locally is quite easy and…
joebiden2 on HNI feel like I've seen a project just like this pop up almost every other day here on HN. Sorry for my ignorance, but how is this different when compared to other "LLM on your mac…
nidnogg on HN
Mac-only at launch; no Linux or Windows binaries yet
From 2 comments
While we haven't built Linux binaries yet (waiting to make sure we have _fast_ GPU support before publishing a Linux and Windows binaries) it does run on Linux. For example it's…
jmorgan on HNThe llama.cpp project is absolutely amazing. Our goal was to build with/extend the project (vs try to be an alternative). Ollama was originally inspired by the "server" example:…
jmorgan on HN
Installer asks for admin access to add a CLI symlink
From 2 comments
Looks cool. I downloaded it, moved it to the applications directory and opened it. It's asking for Administrator access to install the command line. Why?
ioman on HNIt adds a symlink for the "ollama" cli (you can see the source here: https://github.com/jmorganca/ollama/blob/main/app/src/install.ts#L18) ``` /usr/local/bin/ollama ->…
jmorgan on HN
Orca or smaller models only run on 8GB Macs
From 2 comments
I like this a lot, unfortunately I can only run Orca due to 8GB Macbook Air. This opens a lot of doors to using LLMs in art projects for me without worrying about API calls.
neontomo on HNTo add: the easiest way to get up and running is to download the Mac app: https://ollama.ai/download Then running a model (e.g. Llama 2) can be done with ``` ollama run llama2…
jmorgan on HN
Requests
Ability to chat with documents or PDFs
From 2 comments
Is there a way to use this to e.g. "Chat with a PDF?" I want to run the LLM locally so the information doesn't leak out
canadiantim on HNWhat’s the easiest way to add documents to the models, so that they could be included in responses?
aantix on HN
Save models to external drive or network path
From 2 comments
Is it possible to save the model to an external drive or network path?
jimmcslim on HNNot yet, but this totally makes sense – will look into it!
jmorgan on HN
Way to delete models properly
From 1 comment
Hey, great work. But how do I actually delete a model, if it doesn't work on my machine? the rm command doesn't actually delete the files it seems, as if I then do the run…
mxm2 on HN
GPU acceleration on Apple Silicon
From 1 comment
If I use asitop, it doesn't show any use of the GPU. Should it? Is there some flag needed to enable acceleration on MPS?
Traveler42 on HN
Questions
How does it differ from llama.cpp and other local LLM tools?
From 3 comments
What does this add over llama.cpp? Is it just an "easier" way to setup llama.cpp locally? If so, I don't really get it, because setting up llama.cpp locally is quite easy and…
joebiden2 on HNI feel like I've seen a project just like this pop up almost every other day here on HN. Sorry for my ignorance, but how is this different when compared to other "LLM on your mac…
nidnogg on HNThis looks really cool. Can you give give a quick rundowns on how your goals with this differ from Nomic’s GPT4all project? Do you intend to support fine tuning and other…
K0balt on HN
How much RAM and disk does llama2 need?
From 2 comments
It is really amazing. With this command “ollama run llama2”, I can run llama2 instantly. Do you know how much RAM and disk space needed to run this llama2 model?
magundu on HNFor disk size, you can see how big models are locally by running "ollama list" ``` NAME SIZE MODIFIED llama2:latest 3.8 GB 3 hours ago ``` Now, if you make new models that are…
jmorgan on HN
Does it run inference on CPU only?
From 2 comments
This is doing inference on the CPU, right?
littlestymaar on HNIf I use asitop, it doesn't show any use of the GPU. Should it? Is there some flag needed to enable acceleration on MPS?
Traveler42 on HN
Alternatives named
llama.cpp
From 1 comment
What does this add over llama.cpp? Is it just an "easier" way to setup llama.cpp locally? If so, I don't really get it, because setting up llama.cpp locally is quite easy and…
joebiden2 on HN
GPT4All
From 1 comment
This looks really cool. Can you give give a quick rundowns on how your goals with this differ from Nomic’s GPT4all project? Do you intend to support fine tuning and other…
K0balt on HN
PrivateGPT
From 1 comment
The llama.cpp project is absolutely amazing. Our goal was to build with/extend the project (vs try to be an alternative). Ollama was originally inspired by the "server" example:…
jmorgan on HN