[{"data":1,"prerenderedAt":528},["ShallowReactive",2],{"project-36802582":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":36,"aspects":37,"strip":47,"post_text":49,"what":50,"said":89,"maker_in_thread":224,"shape":231,"grid":236,"attempts":491,"same_week":492,"maker_shows":39,"maker_hidden":520,"reader_version":521,"card":522},"36802582","Ollama","Run LLMs on your Mac","title","Show HN: Ollama – Run LLMs on your Mac","https:\u002F\u002Fgithub.com\u002Fjmorganca\u002Follama","github.com","jmorgan",1689869204,2023,284,94,164,3109,6,"legend","ai","local llm runner","cli",[24,25,26,27,28],"local llm","llama 2","run llm locally","gguf models","llama.cpp","curious",-0.109,{"positive":32,"neutral":33,"mixed":34,"critical":35},12,37,0,14,63,{"execution":38,"performance":40,"idea":41,"design":43,"docs":45,"privacy":46},{"positive":39,"critical":39},2,{"positive":39,"critical":18},{"positive":18,"critical":42},3,{"positive":34,"critical":44},1,{"positive":34,"critical":34},{"positive":44,"critical":39},[48,34,39,42,34,42,44,42,42,44,48,42,44,44,44,42,44,44,48,48,42,48,34,44,42,42,48,44,44,42,42,42,42,42,48,42,42,42,42,42],5,null,{"problem":51,"audience":49,"pricing":49,"motivation":62,"time_to_build":49,"stack":69,"openness":86,"stage":87,"team_size":88},{"value":52,"source":53,"evidence":54},"Running large language models locally is hard to set up","commenter",[55,59],{"id":56,"by":57,"text":58},"36809319","iamdamian","I've been wanting to try out local LLMs for a while but have always had the feeling that I would end up wasting my time. I imagined that I would have to spend a perfectly good…",{"id":60,"by":11,"text":61},"36806604","The llama.cpp project is absolutely amazing. Our goal was to build with\u002Fextend the project (vs try to be an alternative). Ollama was originally inspired by the \"server\" example:…",{"value":63,"source":64,"evidence":65},"Large language models need packaging and easy running, similar to what Docker did for containers","maker",[66],{"id":67,"by":11,"text":68},"36803034","Hi HN A few folks and I have been working on this project for a couple weeks now. After previously working on the Docker project for a number of years (both on the container…",[70,77],{"value":71,"source":53,"evidence":72},"Go",[73],{"id":74,"by":75,"text":76},"36804201","mchiang","Linux support is coming, you can build it right now by running: `CGO_ENABLED=1 go build . `",{"value":78,"source":64,"evidence":79},"llama.cpp (GGML)",[80,83],{"id":81,"by":11,"text":82},"36805590","Under the hood it does a few things I'll shed some light on (at a high level!): 1. Running the model: it's built on the open-source (and amazing) llama.cpp project for running…",{"id":84,"by":11,"text":85},"36804922","This is super helpful! Currently it runs 4-bit quantization models (with names often ending in q4_0). Sorry this isn't more obvious. Will fix that and soon update the hosted…","unknown","launched","team",{"praise":90,"objections":109,"requests":148,"questions":184,"alternatives":212},[91,99],{"value":92,"source":53,"evidence":93},"Simple setup makes local LLMs approachable",[94,95],{"id":56,"by":57,"text":58},{"id":96,"by":97,"text":98},"36802952","willietran","Whoa, this is really neat. Does this mean that I can essentially try any LLM on my local machine?",{"value":100,"source":53,"evidence":101},"Modelfile idea for packaging models like Docker images",[102,106],{"id":103,"by":104,"text":105},"36802913","zenlikethat","Shout out to Jeff and Michael for being some of the sharpest product thinkers I've had the pleasure to work with. I've been playing with this lately and it's been loads of fun…",{"id":107,"by":11,"text":108},"36807830","One fun use case of local models is creating \"characters\" or companions (à la character.ai and other tools). It's early but for those interested, this project has a way to do…",[110,121,128,138],{"value":111,"source":53,"evidence":112},"Not clearly different from llama.cpp wrappers",[113,117],{"id":114,"by":115,"text":116},"36806448","joebiden2","What does this add over llama.cpp? Is it just an \"easier\" way to setup llama.cpp locally? If so, I don't really get it, because setting up llama.cpp locally is quite easy and…",{"id":118,"by":119,"text":120},"36806411","nidnogg","I feel like I've seen a project just like this pop up almost every other day here on HN. Sorry for my ignorance, but how is this different when compared to other \"LLM on your mac…",{"value":122,"source":64,"evidence":123},"Mac-only at launch; no Linux or Windows binaries yet",[124,127],{"id":125,"by":11,"text":126},"36804280","While we haven't built Linux binaries yet (waiting to make sure we have _fast_ GPU support before publishing a Linux and Windows binaries) it does run on Linux. For example it's…",{"id":60,"by":11,"text":61},{"value":129,"source":53,"evidence":130},"Installer asks for admin access to add a CLI symlink",[131,135],{"id":132,"by":133,"text":134},"36806309","ioman","Looks cool. I downloaded it, moved it to the applications directory and opened it. It's asking for Administrator access to install the command line. Why?",{"id":136,"by":11,"text":137},"36806424","It adds a symlink for the \"ollama\" cli (you can see the source here: https:\u002F\u002Fgithub.com\u002Fjmorganca\u002Follama\u002Fblob\u002Fmain\u002Fapp\u002Fsrc\u002Finstall.ts#L18) ``` \u002Fusr\u002Flocal\u002Fbin\u002Follama ->…",{"value":139,"source":53,"evidence":140},"Orca or smaller models only run on 8GB Macs",[141,145],{"id":142,"by":143,"text":144},"36832090","neontomo","I like this a lot, unfortunately I can only run Orca due to 8GB Macbook Air. This opens a lot of doors to using LLMs in art projects for me without worrying about API calls.",{"id":146,"by":11,"text":147},"36806300","To add: the easiest way to get up and running is to download the Mac app: https:\u002F\u002Follama.ai\u002Fdownload Then running a model (e.g. Llama 2) can be done with ``` ollama run llama2…",[149,160,170,177],{"value":150,"source":53,"evidence":151},"Ability to chat with documents or PDFs",[152,156],{"id":153,"by":154,"text":155},"36810415","canadiantim","Is there a way to use this to e.g. \"Chat with a PDF?\" I want to run the LLM locally so the information doesn't leak out",{"id":157,"by":158,"text":159},"36809364","aantix","What’s the easiest way to add documents to the models, so that they could be included in responses?",{"value":161,"source":53,"evidence":162},"Save models to external drive or network path",[163,167],{"id":164,"by":165,"text":166},"36809715","jimmcslim","Is it possible to save the model to an external drive or network path?",{"id":168,"by":11,"text":169},"36811169","Not yet, but this totally makes sense – will look into it!",{"value":171,"source":53,"evidence":172},"Way to delete models properly",[173],{"id":174,"by":175,"text":176},"36860520","mxm2","Hey, great work. But how do I actually delete a model, if it doesn't work on my machine? the rm command doesn't actually delete the files it seems, as if I then do the run…",{"value":178,"source":53,"evidence":179},"GPU acceleration on Apple Silicon",[180],{"id":181,"by":182,"text":183},"36832489","Traveler42","If I use asitop, it doesn't show any use of the GPU. Should it? Is there some flag needed to enable acceleration on MPS?",[185,194,204],{"value":186,"source":53,"evidence":187},"How does it differ from llama.cpp and other local LLM tools?",[188,189,190],{"id":114,"by":115,"text":116},{"id":118,"by":119,"text":120},{"id":191,"by":192,"text":193},"36812192","K0balt","This looks really cool. Can you give give a quick rundowns on how your goals with this differ from Nomic’s GPT4all project? Do you intend to support fine tuning and other…",{"value":195,"source":53,"evidence":196},"How much RAM and disk does llama2 need?",[197,201],{"id":198,"by":199,"text":200},"36804104","magundu","It is really amazing. With this command “ollama run llama2”, I can run llama2 instantly. Do you know how much RAM and disk space needed to run this llama2 model?",{"id":202,"by":11,"text":203},"36804196","For disk size, you can see how big models are locally by running \"ollama list\" ``` NAME SIZE MODIFIED llama2:latest 3.8 GB 3 hours ago ``` Now, if you make new models that are…",{"value":205,"source":53,"evidence":206},"Does it run inference on CPU only?",[207,211],{"id":208,"by":209,"text":210},"36805479","littlestymaar","This is doing inference on the CPU, right?",{"id":181,"by":182,"text":183},[213,216,220],{"value":28,"source":53,"evidence":214},[215],{"id":114,"by":115,"text":116},{"value":217,"source":53,"evidence":218},"GPT4All",[219],{"id":191,"by":192,"text":193},{"value":221,"source":64,"evidence":222},"PrivateGPT",[223],{"id":60,"by":11,"text":61},{"comments":225,"answered":226,"key":227},16,11,[228,229,230],{"id":107,"text":108},{"id":60,"text":61},{"id":146,"text":147},{"commenters":232,"max_depth":233,"maker_replies":225,"first_comment_mins":234,"lifespan_hours":235},57,4,22,115,[237,239,243,246,249,253,257,260,262,265,268,271,274,277,280,283,286,289,292,294,297,300,303,306,308,312,313,316,319,321,323,327,329,333,337,340,344,347,352,353,356,358,360,362,365,366,368,370,372,373,376,378,380,384,388,390,393,397,400,402,404,407,410,413,415,419,421,424,426,427,431,434,438,440,441,443,447,449,451,453,456,458,461,463,465,468,471,475,478,480,482,484,485,487,489],[56,34,48,238,57,34],"project\u002Fpositive\u002Fexecution",[240,34,34,241,242,34],"36809541","other\u002Fneutral\u002Fnone","poyu",[244,34,34,241,245,34],"36811577","dataexporter",[247,34,34,241,248,34],"36829960","brianjking",[250,44,44,251,252,34],"36808368","project\u002Fcritical\u002Fperformance","salutis",[254,44,42,255,256,34],"36808551","project\u002Fneutral\u002Fnone","AlphaWeaver",[258,44,42,255,259,34],"36808555","elashri",[261,44,42,255,75,34],"36808603",[263,44,34,241,264,34],"36808665","i-use-nixos-btw",[266,44,34,241,267,34],"36809538","darkteflon",[269,44,34,241,270,34],"36809849","quickthrower2",[272,44,42,255,273,34],"36810271","astrange",[275,44,34,241,276,34],"36809019","babyshake",[278,44,42,255,279,34],"36810105","Dig1t",[281,44,42,255,282,34],"36811831","kdtsh",[284,39,44,251,285,34],"36807169","mrbonner",[287,39,42,255,288,34],"36807217","lagniappe",[290,39,34,241,291,34],"36807235","anigbrowl",[293,39,34,241,285,34],"36807375",[295,39,42,255,296,34],"36807694","quadrature",[298,39,42,255,299,34],"36808759","proamdev123",[301,39,44,251,302,34],"36809770","sheepscreek",[304,39,42,305,273,34],"36810286","project\u002Fneutral\u002Fperformance",[103,42,48,307,104,34],"maker\u002Fpositive\u002Fidea",[309,42,48,310,311,34],"36806692","project\u002Fpositive\u002Fidea","danryan",[107,233,34,307,11,44],[314,233,34,241,315,34],"36809168","another-dave",[317,233,42,255,318,34],"36810572","Patrick_Devine",[114,48,44,320,115,34],"project\u002Fcritical\u002Fidea",[60,48,34,322,11,44],"maker\u002Fneutral\u002Fidea",[324,18,42,325,326,34],"36810270","project\u002Fneutral\u002Fidea","bilalq",[328,18,34,241,318,34],"36810521",[330,18,44,331,332,34],"36824413","project\u002Fcritical\u002Fdesign","paddim8",[334,335,44,320,336,34],"36813328",7,"peanutcrisis",[146,338,34,339,11,44],8,"maker\u002Fneutral\u002Fdocs",[341,338,42,342,343,34],"36808984","project\u002Fneutral\u002Fexecution","nborwankar",[345,338,42,342,346,34],"36809225","replwoacause",[348,349,42,350,351,34],"36804514",9,"project\u002Fneutral\u002Fdocs","mikeravkine",[84,349,34,339,11,44],[354,355,44,320,270,34],"36809885",10,[132,226,44,357,133,34],"project\u002Fcritical\u002Fprivacy",[359,226,44,357,133,34],"36806331",[136,226,34,361,11,44],"maker\u002Fneutral\u002Fprivacy",[363,226,48,238,364,34],"36808468","caconym_",[118,32,42,325,119,34],[367,32,48,310,104,34],"36806736",[198,369,42,305,199,34],13,[371,369,42,305,75,34],"36804176",[202,369,34,339,11,44],[374,35,48,307,375,34],"36807388","mschuster91",[377,35,34,307,11,44],"36811210",[379,35,42,325,375,34],"36811455",[381,382,34,241,383,34],"36808304",15,"sdesol",[385,382,34,386,387,34],"36821138","other\u002Fpositive\u002Fnone","d0gbread",[389,382,34,241,383,34],"36821845",[391,225,42,325,392,34],"36809394","skanga",[394,395,44,251,396,34],"36809470",17,"jdkee",[398,395,34,399,11,44],"36811326","maker\u002Fneutral\u002Fperformance",[174,401,42,350,175,34],18,[96,403,42,325,97,34],19,[405,403,42,406,318,34],"36803047","maker\u002Fneutral\u002Fnone",[408,403,48,409,97,34],"36804148","maker\u002Fpositive\u002Fnone",[411,403,34,412,392,34],"36808869","commenter\u002Fneutral\u002Fnone",[414,403,48,310,104,34],"36803055",[416,417,44,251,418,34],"36804190",20,"politelemon",[74,417,42,420,75,34],"maker\u002Fneutral\u002Fexecution",[422,417,44,423,392,34],"36808929","project\u002Fcritical\u002Fexecution",[425,417,34,409,11,44],"36809397",[125,417,34,420,11,44],[428,417,42,429,430,34],"36806147","project\u002Fneutral\u002Fdesign","williamstein",[432,417,34,433,11,44],"36806451","maker\u002Fneutral\u002Fdesign",[435,436,42,339,437,34],"36805148",21,"barefeg",[81,436,34,439,11,44],"project\u002Fpositive\u002Fdesign",[142,234,48,310,143,34],[191,442,42,322,192,34],23,[444,445,44,423,446,34],"36806203",24,"tough",[448,445,42,420,75,34],"36806232",[450,445,42,305,75,34],"36806282",[181,452,44,251,182,34],25,[454,452,48,455,182,34],"36832966","project\u002Fpositive\u002Fperformance",[457,452,48,455,182,34],"36837285",[153,459,48,460,154,34],26,"project\u002Fpositive\u002Fprivacy",[208,462,42,305,209,34],27,[464,462,42,399,75,34],"36805589",[466,462,42,305,467,34],"36815098","jpeeler",[469,462,34,470,182,34],"36832445","other\u002Fneutral\u002Fperformance",[472,473,42,429,474,34],"36811123",28,"chrispogeek",[476,473,34,477,11,44],"36811152","maker\u002Fpositive\u002Fdocs",[157,479,42,325,158,34],29,[481,479,42,322,75,34],"36809390",[164,483,42,325,165,34],30,[168,483,34,322,11,44],[67,486,34,307,11,44],31,[488,486,42,429,104,34],"36803083",[490,486,34,339,11,44],"36803154",[],[493,498,503,507,511,515],{"id":494,"name":495,"points":496,"category":497},"36860953","pycareer.io",945,"web_saas",{"id":499,"name":500,"points":501,"category":502},"36849502","ProjectionLab",766,"finance_crypto",{"id":504,"name":505,"points":506,"category":497},"36860898","Invoice Dragon",564,{"id":508,"name":509,"points":510,"category":497},"36813086","Primo",356,{"id":512,"name":513,"points":514,"category":497},"36836596","cc.dev",302,{"id":516,"name":517,"points":518,"category":519},"36882146","Continue",298,"dev_tools",false,"v2",{"id":523,"name":5,"line":6,"ai":520,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":524,"top":18,"kw":525,"maker":11,"domain":10,"tier":19,"attempts":34,"strip":526,"url":9,"mix":527,"praise":92,"objection":111,"sub":21},36802582,-0.11,[24,25,26,27,28],"5023031331531113115535013351133333533333",[32,33,34,35],1793991069790]