[{"data":1,"prerenderedAt":368},["ShallowReactive",2],{"project-37661755":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":35,"aspects":36,"strip":45,"post_text":47,"what":48,"said":88,"maker_in_thread":213,"shape":221,"grid":226,"attempts":330,"same_week":331,"maker_shows":34,"maker_hidden":360,"reader_version":361,"card":362},"37661755","Ollama for Linux","Run LLMs on Linux with GPU Acceleration","title","Show HN: Ollama for Linux – Run LLMs on Linux with GPU Acceleration","https:\u002F\u002Fgithub.com\u002Fjmorganca\u002Follama\u002Freleases\u002Ftag\u002Fv0.1.0","github.com","jmorgan",1695745796,2023,49,8,1086,3109,35,"shown","ai","local llm runner","cli",[24,25,26,27,28],"local llm","llama.cpp","gpu acceleration","linux","llama 2","curious",0.265,{"positive":32,"neutral":33,"mixed":34,"critical":34},7,11,2,22,{"idea":37,"docs":39,"execution":41,"design":43,"performance":44},{"positive":34,"critical":38},0,{"positive":38,"critical":40},1,{"positive":42,"critical":38},3,{"positive":38,"critical":40},{"positive":40,"critical":38},[42,38,38,38,42,38,42,38,40,38,38,46,46,40,42,38,46,38,46,38,38,46,42,42,42,38,42,42,38,38,38,42,42,46,38,38,38,46,42,42],5,"Hi HN,\n\nOver the last few months I've been working with some folks on a tool named Ollama (https:\u002F\u002Fgithub.com\u002Fjmorganca\u002Follama) to run open-source LLMs like Llama 2, Code Llama and Falcon locally, starting with macOS.\n\nThe biggest ask since then has been \"how can I run Ollama on Linux?\" with GPU support out of the box. Setting up and configuring CUDA and then compiling and running llama.cpp (which is a fantastic library and runs under the hood) can be quite painful on different combinations of linux distributions and Nvidia GPUs. The goal for Ollama's linux version was to automate this process",{"problem":49,"audience":55,"pricing":64,"motivation":65,"time_to_build":64,"stack":69,"openness":85,"stage":86,"team_size":87},{"value":50,"source":51,"evidence":52},"Setting up CUDA and compiling llama.cpp for GPU inference on Linux is painful","maker",[53],{"id":54},"post",{"value":56,"source":57,"evidence":58},"Developers and users wanting to run LLMs locally on Linux with Nvidia GPUs","inferred",[59,60],{"id":54},{"id":61,"by":62,"text":63},"37663809","sestinj","This is a huge deal, congrats! We've had a ton of users asking how to run their own LLMs on Linux and the unfortunate answer was always that the existing options were slightly…",null,{"value":66,"source":51,"evidence":67},"Users kept asking how to run Ollama on Linux with GPU support out of the box",[68],{"id":54},[70,73,77],{"value":25,"source":51,"evidence":71},[72],{"id":54},{"value":74,"source":51,"evidence":75},"CUDA",[76],{"id":54},{"value":78,"source":79,"evidence":80},"GGUF model format","commenter",[81],{"id":82,"by":83,"text":84},"37767674","factibicongue","\"You guys are doing the lord's work here\"? How? By forcing users into a custom model serialization format (GGUF) that is claimed to literally contain \"magic\"? Dark lord, maybe.","open_source","launched","team",{"praise":89,"objections":120,"requests":143,"questions":158,"alternatives":177},[90,102,110],{"value":91,"source":79,"evidence":92},"Easy install and setup that works for non-experts",[93,97,101],{"id":94,"by":95,"text":96},"37662915","sqs","Ollama is awesome. I am part of a team building a code AI application[1], and we want to give devs the option to run it locally instead of only supporting external LLMs from…",{"id":98,"by":99,"text":100},"37662521","biddit","I’ve been using this on my MacBook Pro for the last couple weeks and want to say thank you! As a solutions developer not so much interested in training models but leveraging them…",{"id":61,"by":62,"text":63},{"value":103,"source":79,"evidence":104},"Useful REST API for building apps on local models",[105,109],{"id":106,"by":107,"text":108},"37662685","mgreg","I'm actually using Ollama for it's Rest API endpoint. Llama.cpp does now have it's server implementation. Unfortunately they do have different endpoints and behave a little…",{"id":94,"by":95,"text":96},{"value":111,"source":79,"evidence":112},"Modelfile lets custom model personas be created easily",[113,117],{"id":114,"by":115,"text":116},"37662341","jerrysievert","this is awesome, congrats on an amazingly useful release! for those that haven't used ollama, being able to specify how a model behaves via a \"modelfile\" is pretty darned…",{"id":118,"by":115,"text":119},"37663690","they are just prompted differently - you can choose the model and a prompt, and ollama presents it as its own \"model\", so instead of `llama2:7b`, it gets presented as…",[121,132,136],{"value":122,"source":79,"evidence":123},"Too similar to llama.cpp wrappers; marketing overstates what it adds",[124,128],{"id":125,"by":126,"text":127},"37662722","smoldesu","I totally agree. A lot of these programs make huge marketing claims that are really just features of Llama.cpp, not a spec from their implementation. It's fine, but the wording…",{"id":129,"by":130,"text":131},"37662105","brucethemoose2","Oh, this is a llama.cpp frontend. Y'all should have lead with that! I saw this on HN before, but I thought it was another from-scratch llama implementation... Which is fine, but…",{"value":133,"source":79,"evidence":134},"Custom GGUF format adds lock-in",[135],{"id":82,"by":83,"text":84},{"value":137,"source":79,"evidence":138},"Cannot control model cache location",[139],{"id":140,"by":141,"text":142},"37668668","WiSaGaN","Ollama is awesome. I am however still waiting for the support of controlling the model cache location: https:\u002F\u002Fgithub.com\u002Fjmorganca\u002Follama\u002Fissues\u002F153 This is either for backup…",[144,148,154],{"value":145,"source":79,"evidence":146},"Support for configuring model cache location",[147],{"id":140,"by":141,"text":142},{"value":149,"source":79,"evidence":150},"Ability to run multiple models simultaneously",[151],{"id":152,"by":107,"text":153},"37663607","Very cool and thanks for sharing. To me a killer feature would be easily running different models simultaneously such as one for embeddings and another for completion (e.g.…",{"value":155,"source":79,"evidence":156},"Speculative sampling support",[157],{"id":61,"by":62,"text":63},[159,170],{"value":160,"source":79,"evidence":161},"How does performance compare to vLLM or exllama?",[162,166],{"id":163,"by":164,"text":165},"37663634","aftbit","How does this compare to vLLM or exllama? Can it run llama2 30B on one 3090 24G or 70B on two 3090 24G? https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm https:\u002F\u002Fgithub.com\u002Fturboderp\u002Fexllama…",{"id":167,"by":168,"text":169},"37663220","kelvie","Amazing! I use text-generation-webui to play with LLMs, but was always jealous of this much simpler interface. Somewhat related note -- does anyone know what are the performance…",{"value":171,"source":79,"evidence":172},"How does WSL2 access the GPU?",[173],{"id":174,"by":175,"text":176},"37663942","politelemon","How is WSL2 able to work with GPUs?",[178,185,195,202,206],{"value":25,"source":79,"evidence":179},[180,184],{"id":181,"by":182,"text":183},"37663899","dorfsmay","> run their own LLMs on Linux and the unfortunate answer was always that the existing options were slightly complicate What about https:\u002F\u002Fgithub.com\u002Fggerganov\u002Fllama.cpp ? It…",{"id":129,"by":130,"text":131},{"value":186,"source":79,"evidence":187},"vLLM",[188,192],{"id":189,"by":190,"text":191},"37662914","binarymax","Congrats on the launch! I'll give it a try. I've been using vLLM on Linux so far but have wanted to be able to use a ggml backend - have you done any perf comparisons?",{"id":193,"by":130,"text":194},"37663430","vLLM has far more throughput with batching, but that is also a WIP feature for llama.cpp. Current standing is something like: - vLLM is the fastest overall with batching, and has…",{"value":196,"source":79,"evidence":197},"exllama",[198,199],{"id":167,"by":168,"text":169},{"id":200,"by":130,"text":201},"37663651","exLLAMAv2 and GPTQ are different implementations, and currently exLLAMA is definitely the best for discrete Nvidia (and AMD?) GPUs. Its faster and uses less VRAM for the same…",{"value":203,"source":79,"evidence":204},"text-generation-webui",[205],{"id":167,"by":168,"text":169},{"value":207,"source":79,"evidence":208},"gpt4all",[209],{"id":210,"by":211,"text":212},"37665450","vorticalbox","Or https:\u002F\u002Fgpt4all.io\u002Findex.html Though doesn't currently support GPU.",{"comments":34,"answered":40,"key":214},[215,218],{"id":216,"text":217},"37664294","There's a ton of cool opportunity in the runtime layer. I've been keeping my eye on the compiler-based approaches. From what I've gathered many of the larger \"production\"…",{"id":219,"text":220},"37663945","Thanks Dang and sorry!!",{"commenters":222,"max_depth":223,"maker_replies":34,"first_comment_mins":224,"lifespan_hours":225},31,4,21,192,[227,229,233,236,239,241,244,246,249,251,252,255,257,260,262,265,269,270,271,273,275,276,277,279,282,284,286,289,291,293,294,298,300,301,304,306,310,311,313,316,321,324,326],[129,38,42,228,130,38],"project\u002Fmixed\u002Fidea",[230,38,38,231,232,38],"37662287","commenter\u002Fneutral\u002Fnone","UncleOxidant",[234,38,38,231,235,38],"37662417","capableweb",[237,38,38,238,130,38],"37663048","other\u002Fneutral\u002Fnone",[106,38,42,240,107,38],"project\u002Fneutral\u002Fnone",[242,38,38,238,243,38],"37663351","lhl",[152,38,42,245,107,38],"project\u002Fneutral\u002Fidea",[247,38,38,238,248,38],"37672991","d4rkp4ttern",[125,38,40,250,126,38],"project\u002Fcritical\u002Fdocs",[216,38,38,238,11,40],[253,38,38,238,254,38],"37665153","andy99",[94,40,46,256,95,38],"project\u002Fpositive\u002Fexecution",[258,34,46,256,259,38],"37663423","ForkMeOnTinder",[82,34,40,261,83,38],"project\u002Fcritical\u002Fdesign",[263,42,38,238,264,38],"37663791","dang",[266,42,42,267,268,38],"37663902","maker\u002Fneutral\u002Fnone","mchiang",[219,42,38,267,11,40],[98,223,46,256,99,38],[272,223,38,238,115,38],"37662674",[61,46,46,274,62,38],"project\u002Fpositive\u002Fidea",[181,46,38,231,182,38],[210,46,38,238,211,38],[114,278,46,274,115,38],6,[280,278,42,245,281,38],"37663034","all2",[118,278,42,283,115,38],"project\u002Fneutral\u002Fdesign",[163,32,42,285,164,38],"project\u002Fneutral\u002Fperformance",[287,32,38,238,288,38],"37664570","harph",[290,32,42,285,164,38],"37664940",[292,32,38,238,243,38],"37671692",[167,15,42,285,168,38],[295,15,38,296,297,38],"37663347","other\u002Fneutral\u002Fperformance","qeternity",[299,15,38,296,243,38],"37663468",[200,15,38,296,130,38],[140,302,42,303,141,38],9,"project\u002Fmixed\u002Fdesign",[189,305,42,285,190,38],10,[307,305,46,308,309,38],"37663255","project\u002Fpositive\u002Fperformance","Patrick_Devine",[193,305,38,296,130,38],[312,305,38,296,243,38],"37663687",[314,33,38,238,315,38],"37670936","hathym",[317,318,46,319,320,38],"37666330",12,"project\u002Fpositive\u002Fnone","aglazer",[174,322,42,323,175,38],13,"project\u002Fneutral\u002Fexecution",[325,322,38,238,309,38],"37664574",[327,328,42,323,329,38],"37663568",14,"agilob",[],[332,337,342,347,351,355],{"id":333,"name":334,"points":335,"category":336},"37613054","Paisa",662,"productivity",{"id":338,"name":339,"points":340,"category":341},"37663270","Raverie",644,"games",{"id":343,"name":344,"points":345,"category":346},"37600019","automatic_log_collector_and_analyzer",313,"infrastructure",{"id":348,"name":349,"points":350,"category":336},"37636701","chatgpt-history-export-to-md",295,{"id":352,"name":353,"points":354,"category":336},"37642671","E-Ink Day Schedule",294,{"id":356,"name":357,"points":358,"category":359},"37661767","Magentic",283,"dev_tools",false,"v2",{"id":363,"name":5,"line":6,"ai":360,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":364,"top":18,"kw":365,"maker":11,"domain":10,"tier":19,"attempts":38,"strip":366,"url":9,"mix":367,"praise":91,"objection":122,"sub":21},37661755,0.27,[24,25,26,27,28],"3000303010055130505005333033000335000533",[32,33,34,34],1793991074965]