[{"data":1,"prerenderedAt":364},["ShallowReactive",2],{"project-41512142":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":36,"aspects":37,"strip":45,"post_text":47,"what":48,"said":90,"maker_in_thread":178,"shape":189,"grid":194,"attempts":327,"same_week":328,"maker_shows":263,"maker_hidden":356,"reader_version":357,"card":358},"41512142","felafax","Tune LLaMa3.1 on Google Cloud TPUs","title","Show HN: Tune LLaMa3.1 on Google Cloud TPUs","https:\u002F\u002Fgithub.com\u002Ffelafax\u002Ffelafax","github.com","felarof",1726067684,2024,189,52,422,3515,13,"notable","ai","llm fine-tuning infrastructure","other",[24,25,26,27,28],"llama fine-tuning","tpu","jax","non-nvidia gpu","llm training","curious",-0.145,{"positive":32,"neutral":33,"mixed":34,"critical":35},2,16,0,7,25,{"performance":38,"idea":40,"pricing":41,"execution":43,"design":44},{"positive":34,"critical":39},3,{"positive":34,"critical":34},{"positive":42,"critical":39},1,{"positive":34,"critical":34},{"positive":34,"critical":42},[39,34,34,34,39,39,46,34,39,46,39,39,39,42,34,42,39,34,34,39,34,39,34,39,39,34,34,39,34,42,42,34,42,34,39,34,42,34,34,42],5,"Hey HN, we wanted to share our repo where we fine-tuned Llama 3.1 on Google TPUs. We’re building AI infra to fine-tune and serve LLMs on non-NVIDIA GPUs (TPUs, Trainium, AMD GPUs).\n\nThe problem: Right now, 90% of LLM workloads run on NVIDIA GPUs, but there are equally powerful and more cost-effective alternatives out there. For example, training and serving Llama 3.1 on Google TPUs is about 30% cheaper than NVIDIA GPUs.\n\nBut developer tooling for non-NVIDIA chipsets is lacking. We felt this pain ourselves. We initially tried using PyTorch XLA to train Llama 3.1 on TPUs, but it was rough: xla i",{"problem":49,"audience":55,"pricing":60,"motivation":61,"time_to_build":65,"stack":71,"openness":88,"stage":89,"team_size":88},{"value":50,"source":51,"evidence":52},"Developer tooling for fine-tuning and serving LLMs on non-NVIDIA chips is lacking","maker",[53],{"id":54},"post",{"value":56,"source":57,"evidence":58},"Developers and businesses fine-tuning LLMs on TPUs, Trainium or AMD GPUs","inferred",[59],{"id":54},null,{"value":62,"source":51,"evidence":63},"Non-NVIDIA chips are cheaper but tooling for them is lacking",[64],{"id":54},{"value":66,"source":51,"evidence":67},"Translating Torch to JAX took 2-3 weeks; PyTorch XLA setup took nearly 4 weeks",[68],{"id":69,"by":11,"text":70},"41514563","It took roughly 2-3 weeks to translate Torch to JAX, but I had past experience writing JAX from my time at Google. We spent nearly 4 weeks getting PyTorch XLA working on TPU.…",[72,76,84],{"value":73,"source":51,"evidence":74},"JAX",[75],{"id":54},{"value":77,"source":51,"evidence":78},"PyTorch XLA",[79,80],{"id":54},{"id":81,"by":82,"text":83},"41516825","hi","The codebase heavily uses PyTorch XLA libraries (torch_xla.*), which are specific to TPU. Key TPU-specific elements include XLA device initialization, SPMD execution mode,…",{"value":85,"source":51,"evidence":86},"Google Cloud TPU",[87],{"id":54},"unknown","beta",{"praise":91,"objections":101,"requests":131,"questions":141,"alternatives":160},[92],{"value":93,"source":94,"evidence":95},"Cost-effective alternative to NVIDIA GPUs","commenter",[96,100],{"id":97,"by":98,"text":99},"41518674","reissbaker","Runpod charges $3.49\u002Fhr for an H100 SXM, which is fairly cheap as far as on-demand H100s go. A v5p TPU is $4.20\u002Fhr, but has 95GB RAM instead of 80GB on the H100 — so you'll need…",{"id":69,"by":11,"text":70},[102,112,127],{"value":103,"source":94,"evidence":104},"Cost comparison against NVIDIA is not apples-to-apples and was not run on NVIDIA",[105,109],{"id":106,"by":107,"text":108},"41515364","p1esk","Then how did you compute the 30-70% cost efficiency numbers compared to Nvidia if you haven’t run this Llama finetuning task on Nvidia GPUs?",{"id":110,"by":107,"text":111},"41515597","It would be a lot more convincing if you actually ran it yourself and did a proper apples to apples comparison, especially considering that’s the whole idea behind your project.",{"value":113,"source":94,"evidence":114},"Google Cloud GPU prices are inflated compared with other providers",[115,119,123],{"id":116,"by":117,"text":118},"41516267","KaoruAoiShiho","It's also comparing prices on google cloud, which has its own markup, a lot more expensive than say runpod. Runpod is $1.64\u002Fhr for the A100 on secure cloud while the A100 on…",{"id":120,"by":121,"text":122},"41517819","Palmik","> For example, training and serving Llama 3.1 on Google TPUs is about 30% cheaper than NVIDIA GPUs When you say this, you should specify which Nvidia GPU you mean (I assume h100…",{"id":124,"by":125,"text":126},"41518601","danvdb","Wouldn't really say it's the cheapest option...there are other providers like Lambda Labs or Ori.co where you can find them way cheaper",{"value":128,"source":51,"evidence":129},"No LoRA support in the JAX path",[130],{"id":54},[132],{"value":133,"source":94,"evidence":134},"Add LoRA training support in JAX",[135,138],{"id":136,"by":98,"text":137},"41518714","Very cool! Unlocking TPU training is a big win. FWIW, if this helps prioritize: personally I'd find LoRA training for Llama 3.1 most useful (which it sounds like currently isn't…",{"id":139,"by":11,"text":140},"41521714","Thanks for the detailed feedback! Yes, supporting LoRA fine-tuning is one of the things we are already working on. btw, we have LoRA supported with Llama3 PyTorch-XLA model.…",[142,149],{"value":143,"source":94,"evidence":144},"How the 30-70% cost savings were computed",[145,148],{"id":146,"by":107,"text":147},"41515219","Did you try running this task (finetuning Llama) on Nvidia GPUs? If yes, can you provide details (which cloud instance and time)? I’m curious about your reported 30-70% speedup.",{"id":106,"by":107,"text":108},{"value":150,"source":94,"evidence":151},"Speed comparison with Unsloth",[152,156],{"id":153,"by":154,"text":155},"41515693","xrd","Anyone want to comment on this versus the fine tune speedups from llama3.1 with unsloth?",{"id":157,"by":158,"text":159},"41514231","htrp","What was the estimate for how much time you guys took to translate the torch to Jax vs how much you spent on XLA?",[161,169,173],{"value":162,"source":94,"evidence":163},"Unsloth",[164,168],{"id":165,"by":166,"text":167},"41516535","nl","I'm pretty sure anyone finetuning Lllama now on a regular basis is using https:\u002F\u002Fgithub.com\u002Funslothai\u002Funsloth so comparisons should be against that. The open source version is…",{"id":153,"by":154,"text":155},{"value":170,"source":94,"evidence":171},"Runpod",[172],{"id":97,"by":98,"text":99},{"value":174,"source":94,"evidence":175},"Lambda Labs",[176,177],{"id":124,"by":125,"text":126},{"id":97,"by":98,"text":99},{"comments":179,"answered":180,"key":181},12,9,[182,183,186],{"id":139,"text":140},{"id":184,"text":185},"41515709","Thank you! Yeah, there are a few complexities and very little documentation around JAX, plus a lot of missing libraries.",{"id":187,"text":188},"41514894","Google published this benchmark a year or so ago comparing TPU vs NVIDIA…",{"commenters":190,"max_depth":191,"maker_replies":179,"first_comment_mins":192,"lifespan_hours":193},31,10,197,24,[195,197,201,204,207,210,213,215,217,221,223,227,229,232,235,238,242,244,246,250,251,253,254,257,259,262,265,267,268,270,274,275,278,282,285,287,288,290,292,294,295,297,298,300,302,305,307,310,313,316,320,324],[165,34,39,196,166,34],"project\u002Fneutral\u002Fperformance",[198,34,34,199,200,34],"41516588","other\u002Fneutral\u002Fnone","syntaxing",[202,34,34,199,203,34],"41516662","randomcatuser",[205,34,34,199,206,34],"41520899","mistrial9",[208,34,34,199,209,34],"41521633","segmondy",[211,34,39,196,212,34],"41518105","pilooch",[136,42,39,214,98,34],"project\u002Fneutral\u002Fidea",[139,42,34,216,11,42],"maker\u002Fneutral\u002Fnone",[218,32,46,219,220,34],"41515307","maker\u002Fpositive\u002Fnone","axpy906",[184,32,34,222,11,42],"maker\u002Fneutral\u002Fdocs",[224,39,39,225,226,34],"41518299","project\u002Fneutral\u002Fpricing","fbn79",[228,39,39,225,98,34],"41518748",[230,39,46,231,226,34],"41518864","project\u002Fpositive\u002Fpricing",[233,39,39,214,234,34],"41520188","voiper1",[236,39,39,214,237,34],"41522125","staticman2",[239,240,39,196,241,34],"41514020",4,"mandoline",[187,240,34,243,11,42],"project\u002Fpositive\u002Fperformance",[120,46,42,245,121,34],"project\u002Fcritical\u002Fpricing",[247,46,34,248,249,34],"41517886","commenter\u002Fneutral\u002Fpricing","spullara",[124,46,42,245,125,34],[252,46,34,248,121,34],"41519124",[97,46,39,225,98,34],[255,46,34,256,121,34],"41519135","commenter\u002Fneutral\u002Fperformance",[258,46,34,256,98,34],"41519576",[260,46,34,261,11,42],"41521952","maker\u002Fpositive\u002Fpricing",[157,263,39,264,158,34],6,"project\u002Fneutral\u002Fexecution",[69,263,34,266,11,42],"maker\u002Fneutral\u002Fexecution",[153,35,39,196,154,34],[269,35,34,196,11,42],"41515939",[271,272,39,264,273,34],"41515829",8,"tcdent",[81,272,39,264,82,34],[276,180,39,225,277,34],"41514882","ricw",[279,180,34,280,281,34],"41514966","other\u002Fneutral\u002Fpricing","cherioo",[283,180,34,199,284,34],"41516268","m3kw9",[286,180,34,225,11,42],"41515113",[146,180,39,196,107,34],[289,180,34,196,11,42],"41515262",[106,180,42,291,107,34],"project\u002Fcritical\u002Fperformance",[293,180,34,196,11,42],"41515482",[110,180,42,291,107,34],[296,180,34,196,11,42],"41515702",[116,180,42,245,117,34],[299,180,34,248,249,34],"41517891",[301,180,34,248,117,34],"41521319",[303,191,39,214,304,34],"41514897","khimaros",[306,191,34,214,11,42],"41514947",[308,191,42,291,309,34],"41516379","Havoc",[311,191,34,199,312,34],"41516696","pogue",[314,191,34,199,315,34],"41516722","kccqzy",[317,318,34,199,319,34],"41518871",11,"faangguyindia",[321,179,42,322,323,34],"41515535","project\u002Fcritical\u002Fdesign","stroupwaffle",[325,179,34,326,11,42],"41515791","maker\u002Fneutral\u002Fdesign",[],[329,334,338,342,347,351],{"id":330,"name":331,"points":332,"category":333},"41465735","Wealthfolio",924,"finance_crypto",{"id":335,"name":336,"points":337,"category":20},"41467704","Infinity",481,{"id":339,"name":340,"points":341,"category":20},"41457633","AnythingLLM",368,{"id":343,"name":344,"points":345,"category":346},"41473518","HN Books",285,"learning_science",{"id":348,"name":349,"points":345,"category":350},"41500938","YourNextStore","web_saas",{"id":352,"name":353,"points":354,"category":355},"41456411","Hacker League",192,"games",false,"v2",{"id":359,"name":5,"line":6,"ai":356,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":360,"top":18,"kw":361,"maker":11,"domain":10,"tier":19,"attempts":34,"strip":362,"url":9,"mix":363,"praise":93,"objection":103,"sub":21},41512142,-0.14,[24,25,26,27,28],"3000335035333101300303033003011010301001",[32,33,34,35],1793991099997]