[{"data":1,"prerenderedAt":565},["ShallowReactive",2],{"project-35026902":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":34,"aspects":35,"strip":44,"post_text":47,"what":48,"said":59,"maker_in_thread":175,"shape":185,"grid":190,"attempts":525,"same_week":526,"maker_shows":45,"maker_hidden":557,"reader_version":558,"card":559},"35026902","Llama-dl","high-speed download of LLaMA, Facebook's 65B GPT model","title","Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model","https:\u002F\u002Fgithub.com\u002Fshawwn\u002Fllama-dl","github.com","sillysaurusx",1677990525,2023,343,130,115,3109,4,"legend","ai","model weights downloader","cli",[24,25,26,27],"llama","llm weights download","65b model","llama leak","curious",0.075,{"positive":31,"neutral":32,"mixed":33,"critical":18},9,46,0,59,{"execution":36,"pricing":38,"performance":39,"docs":40,"idea":41,"privacy":43},{"positive":18,"critical":37},2,{"positive":33,"critical":33},{"positive":18,"critical":37},{"positive":33,"critical":33},{"positive":42,"critical":33},1,{"positive":33,"critical":33},[45,46,33,33,33,33,46,33,46,33,33,46,46,46,45,45,46,45,46,46,18,46,46,46,46,42,46,46,18,45,42,33,33,33,46,46,46,46,46,33],5,3,null,{"problem":49,"audience":47,"pricing":47,"motivation":47,"time_to_build":47,"stack":55,"openness":56,"stage":57,"team_size":58},{"value":50,"source":51,"evidence":52},"Downloading large LLaMA model weights quickly.","inferred",[53],{"id":54},"post",[],"open_source","launched","unknown",{"praise":60,"objections":80,"requests":117,"questions":139,"alternatives":161},[61,69],{"value":62,"source":63,"evidence":64},"Running LLaMA on a single A100 gives output as good as davinci.","commenter",[65],{"id":66,"by":67,"text":68},"35030162","MacsHeadroom","The output is at least as good as davinci. I think some early results are using bad repetition penalty and\u002For temperature settings. I had to set both fairly high to get the best…",{"value":70,"source":63,"evidence":71},"Writeup and sharing are seen as valuable.",[72,76],{"id":73,"by":74,"text":75},"35027808","swyx","thanks for doing this, honestly your writeup seems more valuable than the model weights lol > But for what it's worth, my personal opinion is that LLaMA probably isn't…",{"id":77,"by":78,"text":79},"35031132","rnosov","Seconded. I see vast.ai listing interruptible instance with a single A100 80GB at $1\u002Fhour which is pretty reasonable. ChatGPT plus is $20\u002Fmonth which would be roughly 20 hours of…",[81,95,106],{"value":82,"source":63,"evidence":83},"Cloud GPU cost makes running the 65B model expensive.",[84,87,91],{"id":85,"by":78,"text":86},"35028950","Hmmm, the Github repo suggests that you might be able to run the 65B model on a single A100 80gb card. At the moment, the spot price on Google cloud for this card is $1.25\u002Fhour…",{"id":88,"by":89,"text":90},"35031058","nabla9","$1.25\u002Fhour is roughly a year of GPU time until it exceeds the price of A100 80GB card.",{"id":92,"by":93,"text":94},"35032315","metadat","I think OP meant that $1.25\u002Fhr makes this accessible for people try it out themselves cost effectively, without having to spend thousands or tens of thousands up front to obtain…",{"value":96,"source":63,"evidence":97},"LLaMA is a raw completion model, so comparisons to ChatGPT are unfair.",[98,102],{"id":99,"by":100,"text":101},"35033070","v64","Note that unlike ChatGPT, these models are pure text completers and have not been trained to be prompted. The llama FAQ [1] mentions this and gives tips for how to get out of the…",{"id":103,"by":104,"text":105},"35031470","minxomat","> not really competitive with ChatGPT That's impossible to judge. LLama is a foundational model. It has received neither instructional fine tuning (davinci-3) nor RLHF (ChatGPT).…",{"value":107,"source":63,"evidence":108},"Downloading weights from a leaked source raises legal and ethical concerns.",[109,113],{"id":110,"by":111,"text":112},"35028451","EMIRELADERO","I womder, could Facebook take legal action here? While some (most of) the data used to train the model is copyrighted, I don't think the model is. It's the result of a…",{"id":114,"by":115,"text":116},"35033637","ahahahahah","Are we celebrating theft from tech companies now?",[118,126,132],{"value":119,"source":63,"evidence":120},"Write up a beginner-friendly guide to running the model.",[121,122],{"id":77,"by":78,"text":79},{"id":123,"by":124,"text":125},"35030600","linearalgebra45","Would you mind publishing your notes\u002Flearnings once you gain enough understanding of this model?",{"value":127,"source":63,"evidence":128},"Share sampling settings and prompts used.",[129],{"id":130,"by":11,"text":131},"35030561","That's great to hear. Thank you very much, both for reporting this, and especially for the crucial note about temperature. In fact, sampling settings are so important and so…",{"value":133,"source":63,"evidence":134},"Support for CPU or lower-memory hardware.",[135],{"id":136,"by":137,"text":138},"35033200","davrosthedalek","I would like to support this request for AI challenged developers :) For things like these, I always wonder: How much slower would it be to run such a model on a CPU? I mean,…",[140,147,154],{"value":141,"source":63,"evidence":142},"How to run the model without ML expertise.",[143],{"id":144,"by":145,"text":146},"35031357","vanillax","How or what can someone do with this who isn't a ML expert? Is there some docker app that leverages this? To the average dev, is this useful to me? I know there's lots of \"plug…",{"value":148,"source":63,"evidence":149},"Minimum GPU needed for the smallest model.",[150],{"id":151,"by":152,"text":153},"35032931","yumraj","What’s the minimum single GPU that’ll work for the smallest model?",{"value":155,"source":63,"evidence":156},"How big is the model on disk.",[157],{"id":158,"by":159,"text":160},"35028574","arjvik","How big is this model? (i.e. disk space to store it)",[162,168],{"value":163,"source":63,"evidence":164},"llama-int8 for running LLaMA in half the memory.",[165],{"id":166,"by":100,"text":167},"35028738","If anyone is interested in running this at home, please follow the llama-int8 project [1]. LLM.int8() is a recent development allowing LLMs to run in half the memory without loss…",{"value":169,"source":63,"evidence":170},"llama-cpu for running LLaMA on CPU.",[171],{"id":172,"by":173,"text":174},"35029068","downvotetruth","Eagerly awaiting the int8 vs 4 benchmarks. Also, it can run on CPU https:\u002F\u002Fgithub.com\u002Fmarkasoftware\u002Fllama-cpu So, an 8 bit patch could allow the 65B to run on a standard 128 GB…",{"comments":31,"answered":176,"key":177},6,[178,181,184],{"id":179,"text":180},"35030106","They were trained in fp16, and researchers tend to release whatever format they trained. It’s hard enough to do a large release that it’s best not to try to have too many goals,…",{"id":182,"text":183},"35030059","Careful though — we need to evaluate llama on its own merits. It’s easy to mess up the quantization in subtle ways, then conclude that the outputs aren’t great. So if you’re…",{"id":130,"text":131},{"commenters":186,"max_depth":187,"maker_replies":31,"first_comment_mins":188,"lifespan_hours":189},74,13,217,203,[191,193,195,197,198,200,202,206,209,212,215,218,221,223,226,228,230,232,235,238,240,242,244,247,249,252,254,257,260,264,268,271,274,276,279,281,284,287,290,291,295,298,300,302,304,306,308,312,314,317,320,323,326,329,332,335,337,339,341,342,345,346,348,351,352,353,355,357,359,361,364,367,369,371,375,378,380,382,386,390,392,393,394,396,398,400,403,405,408,410,412,415,417,420,423,426,428,432,435,439,443,446,448,450,451,454,456,459,461,464,467,470,472,475,478,480,483,485,487,490,495,497,498,500,502,504,508,510,513,516,519,522],[166,33,45,192,100,33],"project\u002Fpositive\u002Fexecution",[85,33,46,194,78,33],"project\u002Fneutral\u002Fpricing",[88,33,33,196,89,33],"commenter\u002Fneutral\u002Fpricing",[92,33,33,196,93,33],[172,33,46,199,173,33],"project\u002Fneutral\u002Fperformance",[201,33,46,199,100,33],"35029092",[203,33,33,204,205,33],"35029917","other\u002Fneutral\u002Fnone","CuriouslyC",[207,33,33,204,208,33],"35030116","sp332",[210,33,33,204,211,33],"35030220","brookst",[213,33,33,204,214,33],"35031851","Scene_Cast2",[216,33,33,204,217,33],"35035673","bick_nyers",[219,33,33,204,220,33],"35030326","rfoo",[222,33,33,204,205,33],"35037181",[224,33,33,204,225,33],"35034771","nl",[227,33,33,204,217,33],"35035732",[229,33,33,204,225,33],"35036021",[231,33,33,204,217,33],"35036318",[233,33,33,234,225,33],"35038002","commenter\u002Fneutral\u002Fnone",[236,33,33,204,237,33],"35040827","lumost",[239,33,33,204,217,33],"35042904",[241,33,33,204,237,33],"35050211",[243,33,33,204,205,33],"35036328",[245,33,46,246,74,33],"35029601","project\u002Fneutral\u002Fnone",[179,33,33,248,11,42],"maker\u002Fneutral\u002Fnone",[250,33,33,204,251,33],"35030321","dspillett",[253,33,33,234,251,33],"35039562",[255,33,33,204,256,33],"35033050","charcircuit",[258,33,46,199,259,33],"35030214","nextaccountic",[261,33,33,262,263,33],"35032440","commenter\u002Fneutral\u002Fperformance","woodson",[265,33,33,266,267,33],"35030868","other\u002Fneutral\u002Fidea","causality0",[269,33,33,266,270,33],"35031124","visarga",[272,33,33,266,273,33],"35031255","SloopJon",[275,33,33,266,267,33],"35031908",[277,33,33,266,278,33],"35043708","ElFitz",[280,33,33,266,217,33],"35035818",[282,33,33,266,283,33],"35033211","bloaf",[285,33,33,266,286,33],"35034447","pixl97",[288,33,33,266,289,33],"35058462","ZunarJ5",[144,42,46,246,145,33],[292,42,42,293,294,33],"35031508","project\u002Fcritical\u002Fexecution","swalsh",[296,42,45,192,297,33],"35031848","kkielhofner",[299,37,46,246,124,33],"35028638",[301,37,46,199,100,33],"35028936",[303,37,46,194,124,33],"35029106",[305,37,33,234,100,33],"35029110",[307,37,33,204,124,33],"35029159",[309,37,33,310,311,33],"35030979","commenter\u002Fcritical\u002Fnone","mynameisvlad",[313,37,33,234,124,33],"35031273",[315,37,45,192,316,33],"35030261","smoldesu",[318,37,33,234,319,33],"35030667","jocaal",[321,37,45,322,316,33],"35030865","project\u002Fpositive\u002Fperformance",[324,37,33,325,225,33],"35034200","other\u002Fpositive\u002Fnone",[327,37,46,248,328,33],"35031376","damascus",[330,37,46,331,316,33],"35032653","project\u002Fneutral\u002Fexecution",[333,37,33,204,334,33],"35034167","fswd",[336,37,46,194,67,33],"35029766",[338,37,46,248,124,33],"35030000",[182,37,33,340,11,42],"maker\u002Fneutral\u002Fperformance",[66,37,45,322,67,33],[343,37,46,248,344,33],"35030413","akreal",[130,37,33,234,11,42],[123,37,46,347,124,33],"maker\u002Fneutral\u002Fdocs",[349,37,33,350,11,42],"35030780","project\u002Fpositive\u002Fidea",[77,37,46,347,78,33],[136,37,46,199,137,33],[354,37,46,199,225,33],"35034229",[356,37,45,322,137,33],"35035819",[358,37,46,199,225,33],"35036082",[360,37,46,199,173,33],"35034416",[362,37,33,363,11,42],"35036899","project\u002Fpositive\u002Fnone",[365,37,46,246,366,33],"35037135","KVFinn",[368,37,33,363,11,42],"35037965",[99,37,46,370,100,33],"project\u002Fneutral\u002Fdocs",[372,37,46,373,374,33],"35041807","project\u002Fneutral\u002Fidea","data_maan",[376,37,46,199,377,33],"35031427","youssefabdelm",[379,37,45,322,67,33],"35107419",[381,37,33,234,377,33],"35119867",[383,37,46,384,385,33],"35136771","maker\u002Fneutral\u002Fexecution","dangoodmanUT",[387,37,42,388,389,33],"35030027","project\u002Fcritical\u002Fperformance","ulnarkressty",[391,37,33,234,124,33],"35030082",[103,37,33,234,104,33],[73,46,46,199,74,33],[395,46,33,246,11,42],"35027861",[397,46,33,234,225,33],"35028500",[399,46,42,388,78,33],"35028412",[401,46,33,234,402,33],"35030146","yunyu",[404,46,46,199,225,33],"35028468",[406,46,46,199,407,33],"35028544","JonathanFly",[409,46,46,199,225,33],"35028663",[411,18,33,331,11,42],"35028260",[413,18,46,248,414,33],"35030631","toomuchtodo",[416,18,33,246,11,42],"35032146",[418,18,46,331,419,33],"35033063","e12e",[421,45,46,373,422,33],"35028540","bitL",[424,45,45,192,425,33],"35031006","loufe",[427,45,33,234,422,33],"35032636",[429,176,45,430,431,33],"35030955","maker\u002Fpositive\u002Fidea","version_five",[433,176,33,266,434,33],"35035575","ALittleLight",[436,437,46,373,438,33],"35030816",7,"m3kw9",[440,441,46,331,442,33],"35030114",8,"ur-whale",[444,441,42,293,445,33],"35031136","rany_",[447,441,33,234,442,33],"35033009",[449,441,33,234,256,33],"35033348",[110,31,33,266,111,33],[452,31,33,266,453,33],"35028664","kuroguro",[455,31,33,234,111,33],"35029260",[457,31,33,234,458,33],"35031269","AlDante2",[460,31,33,234,111,33],"35034085",[462,31,33,266,463,33],"35044536","cthalupa",[465,31,33,266,466,33],"35029602","jeroenhd",[468,31,33,234,469,33],"35030044","adossi",[471,31,33,234,211,33],"35030290",[473,31,33,234,474,33],"35030913","cma",[476,31,33,266,477,33],"35031189","digitallyfree",[479,31,46,373,256,33],"35033302",[481,482,46,373,237,33],"35040791",10,[151,484,46,199,152,33],11,[486,484,46,199,173,33],"35033135",[488,484,46,199,489,33],"35033258","zargon",[491,492,46,493,494,33],"35032747",12,"project\u002Fneutral\u002Fprivacy","cloudking",[496,492,46,331,159,33],"35032760",[158,187,46,246,159,33],[499,187,46,246,453,33],"35028578",[501,187,46,246,259,33],"35030238",[503,187,46,246,220,33],"35030409",[505,506,46,331,507,33],"35028745",14,"notpushkin",[114,509,33,266,115,33],15,[511,509,33,266,512,33],"35034188","anaganisk",[514,509,33,266,515,33],"35034562","antibasilisk",[517,509,33,266,518,33],"35037930","7to2",[520,509,33,266,521,33],"35038040","squokko",[523,509,33,266,524,33],"35039872","tjranagk",[],[527,532,537,542,547,552],{"id":528,"name":529,"points":530,"category":531},"35073603","Braggoscope",688,"learning_science",{"id":533,"name":534,"points":535,"category":536},"35032506","Lander",564,"games",{"id":538,"name":539,"points":540,"category":541},"35055308","NESFab",298,"languages",{"id":543,"name":544,"points":545,"category":546},"35042316","UJRPC",290,"infrastructure",{"id":548,"name":549,"points":550,"category":551},"34999774","Mathesar",281,"data",{"id":553,"name":554,"points":555,"category":556},"35071755","CodeGPT.nvim",228,"dev_tools",false,"v2",{"id":560,"name":5,"line":6,"ai":557,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":561,"top":18,"kw":562,"maker":11,"domain":10,"tier":19,"attempts":33,"strip":563,"url":9,"mix":564,"praise":62,"objection":82,"sub":21},35026902,0.08,[24,25,26,27],"5300003030033355353343333133451000333330",[31,32,33,18],1794036632036]