[{"data":1,"prerenderedAt":746},["ShallowReactive",2],{"project-37484135":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":9,"maker":10,"posted_at":11,"year":12,"points":13,"comments":14,"rank_in_year":15,"shown_in_year":16,"top":17,"tier":18,"category":19,"subcategory":20,"format":21,"keywords":22,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":46,"post_text":48,"what":49,"said":93,"maker_in_thread":183,"shape":194,"grid":198,"attempts":708,"same_week":709,"maker_shows":38,"maker_hidden":737,"reader_version":738,"card":739},"37484135","OpenPipe","Fine-tune your own Llama 2 to replace GPT-3.5\u002F4","title","Show HN: Fine-tune your own Llama 2 to replace GPT-3.5\u002F4",null,"kcorbitt",1694537631,2023,13,4,2112,3109,68,"shown","ai","fine-tuning guide","other",[23,24,25,26,27],"fine-tuning","llama 2","llm","gpt-4 alternative","self-hosting","curious",0.035,{"positive":31,"neutral":32,"mixed":33,"critical":34},12,24,1,8,45,{"performance":37,"idea":39,"execution":41,"pricing":43,"docs":44,"privacy":45},{"positive":33,"critical":38},5,{"positive":40,"critical":40},2,{"positive":33,"critical":42},0,{"positive":33,"critical":33},{"positive":14,"critical":42},{"positive":42,"critical":42},[33,42,42,14,47,14,42,33,42,33,42,42,42,42,42,42,42,42,42,42,38,42,33,14,47,14,47,42,47,47,42,38,38,33,38,47,47,40,14,47],3,"There has been a lot of interest on HN in fine-tuning open-source LLMs recently (eg. Anyscale's post at https:\u002F\u002Fnews.ycombinator.com\u002Fitem?id=37090632). I've been playing around with fine-tuning models for a couple of years, and wanted to share some insights and practical code. I’ve condensed what I’ve learned into a small set of notebooks at https:\u002F\u002Fgithub.com\u002FOpenPipe\u002FOpenPipe\u002Ftree\u002Fmain\u002Fexamples\u002Fclassify-recipes, covering labeling data, fine-tuning, running efficient inference, and evaluating costs\u002Fperformance. The 7B model we train here matches GPT-4’s labels 95% of the time on the test set,",{"problem":50,"audience":56,"pricing":61,"motivation":71,"time_to_build":9,"stack":75,"openness":90,"stage":91,"team_size":92},{"value":51,"source":52,"evidence":53},"Replacing expensive GPT-3.5\u002FGPT-4 API calls with a cheaper fine-tuned open model","maker",[54],{"id":55},"post",{"value":57,"source":58,"evidence":59},"Engineers adopting fine-tuning for LLM tasks","inferred",[60],{"id":55},{"value":62,"source":52,"evidence":63},"Notebooks are free; OpenPipe hosted fine-tuning is paid",[64,68],{"id":65,"by":66,"text":67},"37488810","derekpankaew","Can you clarify the 50x cheaper number? Is this for self-hosting, or if you're hosting on OpenPipe? The pricing on OpenPipe says it's 0.0012 to 0.0016 per 1K tokens for Llama 7b.…",{"id":69,"by":10,"text":70},"37489086","Yep, the 50x cost reduction is if you self-host a fine-tuned model using the setup demonstrated in in the linked notebooks.",{"value":72,"source":52,"evidence":73},"Share what the maker learned about fine-tuning and promote OpenPipe",[74],{"id":55},[76,80,84],{"value":77,"source":52,"evidence":78},"Llama 2",[79],{"id":55},{"value":81,"source":52,"evidence":82},"Axolotl",[83],{"id":55},{"value":85,"source":52,"evidence":86},"vLLM",[87],{"id":88,"by":10,"text":89},"37485645","Yes, if you're just using Llama 2 off the shelf (without fine-tuning) I don't think there are a lot of workloads where it makes sense as a replacement for GPT-3.5. The one…","open_source","launched","team",{"praise":94,"objections":111,"requests":141,"questions":153,"alternatives":168},[95],{"value":96,"source":97,"evidence":98},"Informative and well-organized for newcomers","commenter",[99,103,107],{"id":100,"by":101,"text":102},"37492719","davidwritesbugs","Genuinely informative reply for those (few) of us on HN who don’t know the details of LLMs, thanks",{"id":104,"by":105,"text":106},"37491558","zten","Thank you for posting this. I had to go look for your HuggingFace data sets to find the labeled variety you produced with GPT-4, but other than that, everything was easy to follow.",{"id":108,"by":109,"text":110},"37484359","accrual","This looks very helpful! I'm just starting out in the ML\u002FLLM space and have an opportunity to work on this at $dayjob, bookmarking as this looks like an excellent resource. Thank…",[112,123,134],{"value":113,"source":97,"evidence":114},"50x cost claim depends on full GPU utilization, which is unrealistic",[115,119],{"id":116,"by":117,"text":118},"37484631","minimaxir","> Fine-tuning has one huge advantage though: it is far more effective at guiding a model's behavior than prompting, so you can often get away with a much smaller model. That gets…",{"id":120,"by":121,"text":122},"37493210","szesiongteo","I think the cost calculation here does not reflect the actual scenario where most people face. In real world scenario, we don't get inputs queued up to millions and wait for the…",{"value":124,"source":97,"evidence":125},"Fine-tuned 7B models are weaker outside training data",[126,130],{"id":127,"by":128,"text":129},"37491695","atleastoptimal","Fine-tuned low parameter LLM's are superficially good but the cracks are obvious if you test them on anything that isn't very strictly tied to the training data. IMO GPT-4 is…",{"id":131,"by":132,"text":133},"37487844","avereveard","A 7b model will work for very specific cases, but it will have a hard time drawing parallels between synonims, so you'll need to be extremely careful in building your fine tuning…",{"value":135,"source":97,"evidence":136},"Claim to replace GPT-4 is inflated given its mixture-of-experts design",[137],{"id":138,"by":139,"text":140},"37489045","facu17y","\"to replace GPT-3.5\u002F4\" Very inflated statement when it comes to GPT4 since it is a MoE model with 8 separate models each an expert in one area, and you can't replace all 8 models…",[142],{"value":143,"source":97,"evidence":144},"Guidance on fine-tuning for larger models and codellama",[145,149],{"id":146,"by":147,"text":148},"37496293","braindead_in","I have been trying to figure out how to fine tune codellama. Will the llama2 examples work for codellama as well?",{"id":150,"by":151,"text":152},"37484592","indeyets","What are hardware requirements for larger models? What can I fine-tune on Nvidia A100? Will it be possible to work with 70b for example?",[154,161],{"value":155,"source":97,"evidence":156},"Fine-tuning versus RAG for private document QA",[157],{"id":158,"by":159,"text":160},"37497976","loganathanspr","I am a little bit confused whether I need fine-tuning or RAG for my use case? My use case is this: I have some private data (say 1000 word documents), I want a QA capability on…",{"value":162,"source":97,"evidence":163},"Whether input\u002Foutput pairs are required",[164],{"id":165,"by":166,"text":167},"37486319","rrherr","\"You do this by training an existing model on example input\u002Foutput pairs that demonstrate the task you want your fine-tuned model to learn.\" Are fine-tuning datasets required to…",[169,177],{"value":170,"source":97,"evidence":171},"GPT-3.5 and GPT-4 APIs",[172,176],{"id":173,"by":174,"text":175},"37485407","ronyfadel","For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v\u002Fs GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise,…",{"id":116,"by":117,"text":118},{"value":178,"source":52,"evidence":179},"DeBERTa for pure classification",[180],{"id":181,"by":10,"text":182},"37486768","Depends on your use case. If you're doing pure classification then there are smaller encoder-only models like DeBERTa that might get you better performance with a much smaller…",{"comments":184,"answered":185,"key":186},11,9,[187,188,191],{"id":88,"text":89},{"id":189,"text":190},"37484927","> My other thoughts to extend this are that you could make it seamless. To start, it'll simply pipe the user's requests to OpenAI or their existing model. So it'd be a drop in…",{"id":192,"text":193},"37484884","We're finding that when running Llama-2-7B with vLLM (https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm) on an A40 GPU we're getting consistently lower time-to-first-token and lower average…",{"commenters":195,"max_depth":34,"maker_replies":184,"first_comment_mins":196,"lifespan_hours":197},125,15,236,[199,201,205,208,211,214,217,220,223,226,228,232,235,239,243,247,250,253,257,261,263,266,269,272,276,280,282,284,287,290,293,295,298,300,304,308,311,313,316,319,322,325,328,331,333,336,338,340,342,345,347,350,353,356,359,362,365,368,371,374,376,378,381,384,386,388,391,394,397,400,403,406,409,412,415,417,420,423,425,427,430,433,435,438,440,442,445,448,452,455,458,461,464,467,470,472,475,478,481,484,487,490,491,494,496,498,502,504,507,510,512,514,516,519,521,525,527,530,534,536,538,540,544,545,547,549,552,555,559,560,563,564,567,569,572,574,578,580,582,585,588,590,593,594,598,601,605,610,612,615,617,620,622,625,629,632,634,638,640,643,645,648,650,653,657,659,663,665,669,671,673,677,679,681,685,688,690,694,697,701,704],[173,42,33,200,174,42],"project\u002Fcritical\u002Fperformance",[202,42,42,203,204,42],"37485629","commenter\u002Fneutral\u002Fnone","halflings",[206,42,42,203,207,42],"37486376","daniels11",[209,42,42,203,210,42],"37487982","og_kalu",[212,42,42,213,207,42],"37488376","other\u002Fneutral\u002Fpricing",[215,42,42,203,216,42],"37489050","all2",[218,42,42,219,207,42],"37489346","other\u002Fneutral\u002Fnone",[221,42,42,219,222,42],"37505691","two_in_one",[224,42,42,203,225,42],"37488396","achileas",[88,42,42,227,10,33],"commenter\u002Fneutral\u002Fperformance",[229,42,42,230,231,42],"37488474","commenter\u002Fneutral\u002Fpricing","bfirsh",[233,42,42,219,234,42],"37492776","yixu34",[236,42,42,237,238,42],"37499077","other\u002Fneutral\u002Fperformance","Dowwie",[240,42,47,241,242,42],"37485786","project\u002Fmixed\u002Fperformance","Arctic_fly",[244,42,47,245,246,42],"37488904","project\u002Fneutral\u002Fnone","kelseyfrog",[248,42,42,203,249,42],"37489681","FeepingCreature",[251,42,42,203,252,42],"37505519","mikewang",[254,42,38,255,256,42],"37485893","project\u002Fpositive\u002Fidea","refulgentis",[258,42,47,259,260,42],"37487851","project\u002Fneutral\u002Fexecution","homarp",[262,42,42,203,256,42],"37490324",[264,42,47,245,265,42],"37485902","MuffinFlavored",[267,42,42,230,268,42],"37486039","kuchenbecker",[270,42,42,230,271,42],"37486091","loudmax",[273,42,38,274,275,42],"37486656","project\u002Fpositive\u002Fpricing","ramesh31",[277,42,47,278,279,42],"37487543","project\u002Fneutral\u002Fdocs","thewataccount",[281,42,47,245,275,42],"37487668",[283,42,42,213,279,42],"37487879",[285,42,42,219,286,42],"37488998","mjirv",[288,42,42,219,289,42],"37490473","pdntspa",[291,42,42,227,292,42],"37596825","cheptsov",[294,42,42,219,279,42],"37486800",[296,42,42,219,297,42],"37488340","hereonout2",[299,42,42,203,279,42],"37488572",[301,42,33,302,303,42],"37489592","project\u002Fcritical\u002Fidea","axpy906",[305,42,42,306,307,42],"37486442","other\u002Fneutral\u002Fprivacy","AnonymousPlanet",[309,42,42,219,310,42],"37486780","unoti",[312,42,42,306,307,42],"37489824",[314,42,42,306,315,42],"37490594","carom",[317,42,42,306,318,42],"37494699","Aerbil313",[320,42,42,219,321,42],"37490874","fomine3",[323,42,42,306,324,42],"37493525","ozgune",[326,42,42,219,327,42],"37490773","deet",[329,42,33,200,330,42],"37486556","brucethemoose2",[332,42,42,230,275,42],"37486602",[334,42,42,227,335,42],"37486957","YetAnotherNick",[337,42,42,237,275,42],"37487046",[339,42,42,227,335,42],"37487069",[341,42,42,237,275,42],"37487140",[343,42,42,227,344,42],"37487679","pocketarc",[346,42,42,237,275,42],"37489606",[348,42,42,227,349,42],"37491113","jpdus",[351,42,42,213,352,42],"37486942","mrybczyn",[354,42,42,213,355,42],"37487175","PUSH_AX",[357,42,42,237,358,42],"37488983","why_only_15",[360,42,42,219,361,42],"37487245","read_if_gay_",[363,42,42,219,364,42],"37487337","sacred_numbers",[366,42,42,213,367,42],"37487478","hutzlibu",[369,42,42,203,370,42],"37488387","csjh",[372,42,42,219,373,42],"37487386","haxton",[375,42,42,219,174,42],"37487507",[377,42,42,219,222,42],"37487790",[379,42,42,219,380,42],"37488411","yumraj",[382,42,42,219,383,42],"37488533","sarthaksrinivas",[385,42,42,219,370,42],"37488379",[387,42,42,219,358,42],"37488963",[389,42,42,219,390,42],"37489487","jiggawatts",[392,42,42,219,393,42],"37493978","avion23",[395,42,42,219,396,42],"37507294","JackRumford",[398,42,42,230,399,42],"37487305","octacat",[401,42,42,203,402,42],"37498557","flangola7",[404,42,42,230,405,42],"37498686","blitz_skull",[407,42,42,230,408,42],"37542514","kread",[410,42,42,203,411,42],"37503498","Mystery-Machine",[413,42,42,213,414,42],"37487653","computerex",[416,42,42,213,132,42],"37487891",[418,42,42,237,419,42],"37489601","ttt3ts",[421,42,42,219,422,42],"37489754","zakki",[424,42,42,219,419,42],"37490173",[426,42,42,219,422,42],"37490412",[428,42,42,237,429,42],"37518435","efreak",[431,42,42,219,432,42],"37490613","apstls",[434,42,42,219,419,42],"37490669",[436,42,42,203,437,42],"37492100","coryrc",[439,42,42,219,419,42],"37510896",[441,42,42,219,437,42],"37511400",[443,42,42,219,444,42],"37495073","Reviving1514",[446,42,42,213,447,42],"37506290","0x008",[449,42,42,450,451,42],"37489646","other\u002Fneutral\u002Fidea","nborwankar",[453,42,42,213,454,42],"37492162","yessenzhar",[456,42,42,237,457,42],"37492388","tuckerconnelly",[459,42,42,213,460,42],"37498898","mr_o47",[462,42,42,213,463,42],"37504689","robertnishihara",[465,33,38,255,466,42],"37484521","tikkun",[468,33,42,450,469,42],"37484540","bongobingo1",[189,33,42,471,10,33],"maker\u002Fneutral\u002Fidea",[473,33,42,219,474,42],"37485592","bambax",[476,33,42,450,477,42],"37486300","NavinF",[479,33,42,450,480,42],"37490695","skybrian",[482,33,42,219,483,42],"37489628","jlm521",[485,33,42,219,486,42],"37491956","animeshjain",[488,33,42,219,489,42],"37496524","nasir",[116,40,33,200,117,42],[492,40,42,227,493,42],"37484801","moonchrome",[495,40,42,227,117,42],"37484848",[497,40,42,227,493,42],"37488459",[499,40,47,500,501,42],"37485270","project\u002Fneutral\u002Fperformance","7speter",[503,40,47,500,204,42],"37485816",[505,40,38,506,297,42],"37487796","project\u002Fpositive\u002Fperformance",[508,40,47,500,509,42],"37486702","gsuuon",[192,40,42,511,10,33],"maker\u002Fpositive\u002Fperformance",[513,40,42,230,297,42],"37485091",[515,40,42,203,117,42],"37485145",[517,47,47,259,518,42],"37485972","binarymax",[520,47,47,259,242,42],"37486241",[522,47,47,523,524,42],"37491877","project\u002Fneutral\u002Fidea","behnamoh",[526,47,47,278,242,42],"37492342",[528,47,38,529,524,42],"37496906","maker\u002Fpositive\u002Fdocs",[531,47,38,532,533,42],"37489262","project\u002Fpositive\u002Fexecution","llwj",[65,14,47,535,66,42],"project\u002Fneutral\u002Fpricing",[69,14,42,537,10,33],"maker\u002Fneutral\u002Fpricing",[120,38,33,539,121,42],"project\u002Fcritical\u002Fpricing",[541,542,42,219,543,42],"37486658",6,"divbzero",[181,542,42,219,10,33],[546,542,42,219,271,42],"37487194",[548,542,42,219,447,42],"37611539",[550,542,42,219,551,42],"37487218","jw903",[553,542,42,219,554,42],"37489483","daemonologist",[556,557,42,219,558,42],"37485651",7,"brianjking",[158,34,42,219,159,42],[561,34,42,219,562,42],"37502797","mohitgangrade",[165,185,47,245,166,42],[565,185,42,566,10,33],"37486687","maker\u002Fneutral\u002Fnone",[568,185,47,566,166,42],"37489316",[570,185,42,219,571,42],"37513652","omneity",[573,185,42,219,571,42],"37513607",[575,576,42,219,577,42],"37485885",10,"Maschinesky",[579,576,42,471,10,33],"37486105",[581,576,42,450,242,42],"37486109",[583,184,42,219,584,42],"37493174","imhoguy",[586,31,42,219,587,42],"37486841","ingridpan",[589,13,47,523,238,42],"37499793",[108,591,38,592,109,42],14,"project\u002Fpositive\u002Fdocs",[104,196,38,592,105,42],[595,596,42,219,597,42],"37488234",16,"rookie123",[599,596,42,600,10,33],"37488506","maker\u002Fneutral\u002Fdocs",[602,603,42,219,604,42],"37498698",17,"smko",[606,607,38,608,609,42],"37520483",18,"project\u002Fpositive\u002Fnone","anitakirkovska",[150,611,47,500,151,42],19,[613,611,42,614,10,33],"37484809","maker\u002Fneutral\u002Fperformance",[616,611,42,203,151,42],"37485131",[618,611,42,219,619,42],"37492780","FrostKiwi",[127,621,33,200,128,42],20,[623,621,42,219,624,42],"37492788","kytazo",[626,627,42,219,628,42],"37485897",21,"halyconWays",[630,627,42,219,631,42],"37487210","fabmilo",[100,633,38,529,101,42],22,[635,636,47,245,637,42],"37484872",23,"msp26",[639,636,42,566,10,33],"37484943",[641,636,42,219,642,42],"37485600","selfhoster11",[644,636,42,219,642,42],"37485456",[646,32,47,535,647,42],"37486025","jxf",[649,32,42,537,10,33],"37486207",[651,32,38,652,647,42],"37487469","maker\u002Fpositive\u002Fnone",[654,655,42,219,656,42],"37487440",25,"he11ow",[131,658,33,200,132,42],26,[660,661,42,219,662,42],"37486123",27,"3abiton",[664,661,42,219,330,42],"37486393",[666,667,42,219,668,42],"37489073",28,"robot",[146,670,47,245,147,42],29,[138,672,33,302,139,42],30,[674,675,47,523,676,42],"37486190",31,"idosh",[678,675,47,471,242,42],"37486499",[680,675,38,608,676,42],"37487950",[682,675,47,683,684,42],"37489890","project\u002Fneutral\u002Fprivacy","throw03172019",[686,687,42,219,315,42],"37490504",32,[689,687,42,219,330,42],"37501824",[691,692,42,219,693,42],"37486007",33,"notShabu",[695,692,42,219,696,42],"37487251","ftxbro",[698,699,47,523,700,42],"37489185",34,"OhNoNotAgain_99",[702,699,42,219,703,42],"37489588","sandkoan",[705,706,42,219,707,42],"37518825",35,"kasheef",[],[710,714,719,724,729,733],{"id":711,"name":712,"points":713,"category":19},"37427127","HackYourNews",335,{"id":715,"name":716,"points":717,"category":718},"37470611","splat",309,"languages",{"id":720,"name":721,"points":722,"category":723},"37444683","Ghidra Plays Mario",207,"dev_tools",{"id":725,"name":726,"points":727,"category":728},"37499375","Lantern",196,"data",{"id":730,"name":731,"points":732,"category":718},"37438028","cursed-typescript",186,{"id":734,"name":735,"points":736,"category":19},"37433218","Rivet",176,false,"v2",{"id":740,"name":5,"line":6,"ai":737,"cat":19,"year":12,"points":13,"comments":14,"rec":28,"s":741,"top":17,"kw":742,"maker":10,"domain":743,"tier":18,"attempts":42,"strip":744,"url":-1,"mix":745,"praise":96,"objection":113,"sub":20},37484135,0.04,[23,24,25,26,27],"","1004340101000000000050143430330551533243",[31,32,33,34],1794036641657]