[{"data":1,"prerenderedAt":450},["ShallowReactive",2],{"project-37148210":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":36,"aspects":37,"strip":47,"post_text":48,"what":49,"said":90,"maker_in_thread":214,"shape":222,"grid":225,"attempts":413,"same_week":414,"maker_shows":40,"maker_hidden":443,"reader_version":444,"card":445},"37148210","LlamaGPT","Self-hosted, offline, private AI chatbot, powered by Llama 2","title","Show HN: LlamaGPT – Self-hosted, offline, private AI chatbot, powered by Llama 2","https:\u002F\u002Fgithub.com\u002Fgetumbrel\u002Fllama-gpt","github.com","mayankchhabra",1692198309,2023,225,75,239,3109,8,"notable","ai","self-hosted chatbot","web_app",[24,25,26,27,28],"self-hosted llm","llama 2","offline chatbot","private ai","docker","curious",0.08,{"positive":32,"neutral":33,"mixed":34,"critical":35},7,22,0,5,34,{"idea":38,"docs":41,"execution":42,"performance":44,"privacy":45},{"positive":39,"critical":40},4,3,{"positive":34,"critical":34},{"positive":43,"critical":34},2,{"positive":34,"critical":43},{"positive":46,"critical":34},1,[40,40,34,40,40,40,34,40,34,34,34,35,40,46,40,34,34,35,40,46,35,34,34,34,34,34,35,40,40,35,46,40,40,46,34,40,35,35,46,34],null,{"problem":50,"audience":60,"pricing":48,"motivation":64,"time_to_build":48,"stack":68,"openness":87,"stage":88,"team_size":89},{"value":51,"source":52,"evidence":53},"Running Llama 2 locally is hard for non-technical people; LlamaGPT packages it as one command","maker",[54,57],{"id":55,"by":11,"text":56},"37154565","The main difference is setting everything up yourself manually, downloading the modal, optimizing the parameters for best performance, running an API server and a UI front-end -…",{"id":58,"by":11,"text":59},"37154530","It's an entire app (with a chatbot UI) that takes away the technical legwork to run the model locally. It's a simple one line `docker compose up -d` on any machine, or one click…",{"value":61,"source":52,"evidence":62},"Non-technical people wanting a private local chatbot; home server users",[63],{"id":55,"by":11,"text":56},{"value":65,"source":52,"evidence":66},"Make running a local LLM one command away for people who can't set it up manually",[67],{"id":55,"by":11,"text":56},[69,73,81],{"value":70,"source":52,"evidence":71},"Docker",[72],{"id":55,"by":11,"text":56},{"value":74,"source":75,"evidence":76},"llama.cpp server","commenter",[77],{"id":78,"by":79,"text":80},"37156498","netdur","no not llama-cpp-python, it uses llama.cpp's built in server.",{"value":82,"source":52,"evidence":83},"Nous Hermes Llama 2 7B model",[84],{"id":85,"by":11,"text":86},"37154579","It's powered by Nous Hermes Llama2 7b. From their docs: \"This model stands out for its long responses, lower hallucination rate, and absence of OpenAI censorship mechanisms.…","open_source","launched","unknown",{"praise":91,"objections":107,"requests":137,"questions":162,"alternatives":191},[92,100],{"value":93,"source":75,"evidence":94},"Simple one-command setup makes local LLMs accessible",[95,96],{"id":55,"by":11,"text":56},{"id":97,"by":98,"text":99},"37156039","Multicomp","Agreed. Gpt4all[1] offers a similar 'simple setup' but with application exe downloads, but is arguably more like open core because the gpt4all makers (nomic?) want to sell you…",{"value":101,"source":75,"evidence":102},"Private and offline operation appeals to users",[103],{"id":104,"by":105,"text":106},"37152885","QuinnyPig","I've been looking for something like this for a while. Nice!",[108,119,130],{"value":109,"source":75,"evidence":110},"GPU support is not available yet; CPU inference is slow",[111,115],{"id":112,"by":113,"text":114},"37153078","belval","Nice project! I could not find the information in the README.md, can I run this with a GPU? If so what do I need to change? Seems like it's hardcoded to 0 in the run script:…",{"id":116,"by":117,"text":118},"37154861","reneberlin","Good luck! The token\u002Fsec will be under your expectations or it will overheat. You really shouldn't play games with your data-storage. You could try it with an old laptop to see…",{"value":120,"source":75,"evidence":121},"Unclear what it adds over llama.cpp or existing installers like oobabooga",[122,126],{"id":123,"by":124,"text":125},"37154402","SubiculumCode","I didn't see any info on how this is different than installing\u002Frunning llamacpp or koboldcpp. New offerings are awesome of course, but what is it adding?",{"id":127,"by":128,"text":129},"37154967","DrPhish","Maybe I've been at this for too long and can't see the pitfalls of a normal user, but how is that easier than using an oobabooga one-click installer (an option that's been around…",{"value":131,"source":75,"evidence":132},"Name uses GPT, a possible trademark issue",[133],{"id":134,"by":135,"text":136},"37153606","caesil","So many projects still using GPT in their name. Is the thinking here that OpenAI is not going to defend that trademark? Or just kicking the can down the road on rebranding until…",[138,148,155],{"value":139,"source":52,"evidence":140},"GPU support (CUDA and Metal)",[141,144],{"id":142,"by":11,"text":143},"37154554","Ah yes, running on GPU isn't supported at the moment. But CUDA (for Nvidia GPUs) and Metal support is on the roadmap!",{"id":145,"by":146,"text":147},"37164076","dicriseg","I put up a draft PR to demo how to run it on a GPU: https:\u002F\u002Fgithub.com\u002Fgetumbrel\u002Fllama-gpt\u002Fpull\u002F11 It breaks other things like model downloading, but once I got it to a working…",{"value":149,"source":75,"evidence":150},"Ability to add private data such as emails or documents",[151],{"id":152,"by":153,"text":154},"37154007","ccozan","Ok, since is running all private, how can I add my own private data? For example I have a 20+ years of an email archive that I'd like to be ingested.",{"value":156,"source":75,"evidence":157},"Guide for non-ML users on adding private data",[158],{"id":159,"by":160,"text":161},"37158305","squarefoot","Thanks for all replies. It would probably be worth creating a HOWTO or something like that aimed at non ML experts or complete AI illiterates like myself to help putting together…",[163,169,180],{"value":164,"source":75,"evidence":165},"How to import models from Hugging Face",[166],{"id":167,"by":124,"text":168},"37154640","thanks. yeah, that IS useful. Anyone see if it contains utilities to import models from huggingface\u002Fgithub?",{"value":170,"source":75,"evidence":171},"Whether the model is censored",[172,176],{"id":173,"by":174,"text":175},"37153224","chasd00","is it a free model or is the politically-correct-only response constraints in place?",{"id":177,"by":178,"text":179},"37153416","Havoc","Llama is definitely \"censored\" though I've not found this to be an issue in practice. Guess it depends on what you want to do with it",{"value":181,"source":75,"evidence":182},"Whether it runs on a cloud server or NAS",[183,187],{"id":184,"by":185,"text":186},"37148266","albert_e","Oh I thought this was a quick guide to host it on any server (AWS \u002F other clouds) of our choosing.",{"id":188,"by":189,"text":190},"37153319","ryanSrich","Interesting. I might try to get this to work on my NAS.",[192,199,206,210],{"value":193,"source":75,"evidence":194},"Ollama",[195],{"id":196,"by":197,"text":198},"37153869","lkbm","https:\u002F\u002Fgithub.com\u002Fjmorganca\u002Follama was extremely simple to get running on my M1 and has a couple uncensored models you can just download and use.",{"value":200,"source":75,"evidence":201},"Serge",[202],{"id":203,"by":204,"text":205},"37153425","Atlas-Marbles","Very cool, this looks like a combination of chatbot-ui and llama-cpp-python? A similar project I've been using is https:\u002F\u002Fgithub.com\u002Fserge-chat\u002Fserge. Nous-Hermes-Llama2-13b is…",{"value":207,"source":75,"evidence":208},"GPT4All",[209],{"id":97,"by":98,"text":99},{"value":211,"source":75,"evidence":212},"oobabooga text-generation-webui",[213],{"id":127,"by":128,"text":129},{"comments":32,"answered":215,"key":216},6,[217,218,221],{"id":142,"text":143},{"id":219,"text":220},"37155070","It's fairly straightforward to add GPU support when running on the host, but LlamaGPT runs inside a Docker container, and that's where it gets a bit challenging.",{"id":55,"text":56},{"commenters":223,"max_depth":32,"maker_replies":32,"first_comment_mins":39,"lifespan_hours":224},53,48,[226,228,231,234,238,241,244,247,251,253,255,258,262,265,267,270,272,276,279,282,284,287,289,291,293,297,299,302,303,307,309,311,313,314,316,317,319,323,324,327,328,331,334,337,340,342,344,347,349,351,352,355,358,359,361,363,364,365,368,371,374,377,378,381,383,384,388,391,393,396,400,403,406,408,410],[152,34,40,227,153,34],"project\u002Fneutral\u002Fidea",[229,34,40,227,230,34],"37154064","rdedev",[232,34,40,227,233,34],"37154364","cromka",[235,34,34,236,237,34],"37154608","other\u002Fneutral\u002Fnone","ravishi",[239,34,34,236,240,34],"37154842","phillipcarter",[242,34,34,236,243,34],"37155151","aryamaan",[245,34,40,227,246,34],"37155040","influxmoment",[248,34,34,249,250,34],"37155289","commenter\u002Fneutral\u002Fidea","alach11",[252,34,40,227,250,34],"37155282",[254,34,40,227,160,34],"37155457",[256,34,34,236,257,34],"37155491","gzer0",[259,34,34,260,261,34],"37155502","commenter\u002Fneutral\u002Fnone","siquick",[263,34,34,236,264,34],"37155744","deet",[159,34,40,266,160,34],"project\u002Fneutral\u002Fdocs",[268,34,34,260,269,34],"37160843","infecto",[271,34,34,260,269,34],"37160810",[273,34,34,274,275,34],"37155469","other\u002Fneutral\u002Fidea","EGreg",[277,34,34,236,278,34],"37155531","bravura",[280,34,34,236,281,34],"37155667","seanthemon",[283,34,34,236,264,34],"37155909",[285,34,34,236,286,34],"37160124","FieryTransition",[203,46,35,288,204,34],"project\u002Fpositive\u002Fidea",[78,46,40,290,79,34],"project\u002Fneutral\u002Fexecution",[112,43,40,292,113,34],"project\u002Fneutral\u002Fperformance",[294,43,46,295,296,34],"37153819","project\u002Fcritical\u002Fperformance","crudgen",[142,43,34,298,11,46],"maker\u002Fneutral\u002Fexecution",[300,43,40,298,301,34],"37154690","samspenc",[219,43,34,298,11,46],[304,43,34,305,306,34],"37155556","commenter\u002Fneutral\u002Fexecution","stavros",[308,43,34,260,146,34],"37166707",[310,43,34,260,306,34],"37166884",[145,43,35,312,146,34],"maker\u002Fpositive\u002Fexecution",[123,40,40,227,124,34],[55,40,34,315,11,46],"maker\u002Fneutral\u002Fidea",[167,40,40,227,124,34],[127,40,46,318,128,34],"project\u002Fcritical\u002Fidea",[320,40,34,321,322,34],"37156121","commenter\u002Fcritical\u002Fidea","ShamelessC",[97,40,35,288,98,34],[325,39,40,227,326,34],"37156334","avivo",[134,35,34,236,135,34],[329,35,34,236,330,34],"37153814","schappim",[332,35,34,236,333,34],"37153831","khaledh",[335,35,34,236,336,34],"37153841","super256",[338,35,34,236,339,34],"37154645","raffraffraff",[341,35,34,236,306,34],"37155889",[343,35,34,236,339,34],"37158330",[345,35,34,236,346,34],"37156397","clnq",[348,35,34,236,306,34],"37155905",[350,35,34,236,336,34],"37161325",[104,215,35,288,105,34],[353,32,40,227,354,34],"37159568","stormfather",[356,18,40,227,357,34],"37154494","synaesthesisx",[58,18,34,315,11,46],[184,360,40,290,185,34],9,[362,360,34,298,11,46],"37150241",[188,360,35,288,189,34],[116,360,46,295,117,34],[366,360,34,367,11,46],"37154903","maker\u002Fneutral\u002Fperformance",[369,360,34,370,301,34],"37154716","other\u002Fpositive\u002Fnone",[173,372,40,373,174,34],10,"project\u002Fneutral\u002Fnone",[375,372,40,227,376,34],"37153408","benreesman",[177,372,46,318,178,34],[379,372,34,260,380,34],"37154376","redox99",[382,372,34,260,178,34],"37155141",[85,372,34,227,11,46],[385,386,40,373,387,34],"37153468",11,"lazzlazzlazz",[389,386,40,227,390,34],"37153851","mritchie712",[196,386,35,392,197,34],"project\u002Fpositive\u002Fexecution",[394,386,40,266,395,34],"37154168","brucemacd",[397,386,40,398,399,34],"37154479","project\u002Fneutral\u002Fprivacy","dealuromanet",[401,386,35,402,395,34],"37156004","project\u002Fpositive\u002Fprivacy",[404,386,46,318,405,34],"37155842","freedomben",[407,386,34,260,197,34],"37169477",[409,386,34,260,405,34],"37176785",[411,386,34,236,412,34],"37154220","avereveard",[],[415,419,424,429,433,438],{"id":416,"name":417,"points":418,"category":20},"37125118","Outlines",854,{"id":420,"name":421,"points":422,"category":423},"37119942","Little Rat",555,"security_privacy",{"id":425,"name":426,"points":427,"category":428},"37083309","OFRAK Tetris",472,"games",{"id":430,"name":431,"points":432,"category":20},"37128293","AI-town",429,{"id":434,"name":435,"points":436,"category":437},"37114141","learndb-py",371,"learning_science",{"id":439,"name":440,"points":441,"category":442},"37075730","Applite",280,"dev_tools",false,"v2",{"id":446,"name":5,"line":6,"ai":443,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":30,"top":18,"kw":447,"maker":11,"domain":10,"tier":19,"attempts":34,"strip":448,"url":9,"mix":449,"praise":93,"objection":109,"sub":21},37148210,[24,25,26,27,28],"3303330300053130053150000053351331035510",[32,33,34,35],1793991071918]