[{"data":1,"prerenderedAt":425},["ShallowReactive",2],{"project-38487199":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":36,"aspects":37,"strip":44,"post_text":45,"what":46,"said":98,"maker_in_thread":186,"shape":199,"grid":203,"attempts":379,"same_week":387,"maker_shows":32,"maker_hidden":417,"reader_version":418,"card":419},"38487199","Unsloth","80% faster, 50% less memory, 0% loss of accuracy Llama finetuning","title","Show HN: 80% faster, 50% less memory, 0% loss of accuracy Llama finetuning","https:\u002F\u002Fgithub.com\u002Funslothai\u002Funsloth","github.com","danielhanchen",1701441760,2023,385,119,87,3109,3,"legend","ai","llm finetuning","library",[24,25,26,27,28],"llama finetuning","qlora","lora training","triton kernels","gpu memory","curious",0.032,{"positive":32,"neutral":33,"mixed":34,"critical":35},4,21,0,5,30,{"pricing":38,"performance":40,"idea":41,"design":42,"execution":43},{"positive":39,"critical":18},1,{"positive":34,"critical":39},{"positive":39,"critical":34},{"positive":39,"critical":34},{"positive":34,"critical":39},[18,34,34,18,39,18,18,18,39,18,18,34,18,18,18,18,18,34,18,18,35,39,18,35,18,39,39,35,34,18,18,34,34,18,18,34,34,35,34,34],null,{"problem":47,"audience":54,"pricing":62,"motivation":72,"time_to_build":45,"stack":82,"openness":95,"stage":96,"team_size":97},{"value":48,"source":49,"evidence":50},"Finetuning Llama models is slow and memory-hungry on local GPUs.","maker",[51],{"id":52,"by":11,"text":53},"38487200","Hi HN! I'm just sharing a project I've been working on during the LLM Efficiency Challenge - you can now finetune Llama with QLoRA 5x faster than Huggingface's original…",{"value":55,"source":56,"evidence":57},"People finetuning Llama models on their own NVIDIA GPUs.","inferred",[58,59],{"id":52,"by":11,"text":53},{"id":60,"by":11,"text":61},"38495106","M1 support is probably coming in the future if there is enough support - could you make an Issue at https:\u002F\u002Fgithub.com\u002Funslothai\u002Funsloth - that would be much appreciated! I think…",{"value":63,"source":49,"evidence":64},"Open source core; paid Pro and Max tiers for multi-GPU speedups.",[65,66,69],{"id":52,"by":11,"text":53},{"id":67,"by":11,"text":68},"38496080","I wouldn't characterize it exactly as \"shareware\", since the OSS version is fully functional, trains 5x faster on a single GPU. The paid version just makes it train 30x faster on…",{"id":70,"by":11,"text":71},"38495526","Good point - we thought about this - we're still figuring out the pricing as we go along - so all suggestions are welcome! I'm all very new to this, so I'm literally making stuff…",{"value":73,"source":49,"evidence":74},"Maker says earlier package Hyperlearn was taken without credit; wants to fund living costs while staying open.",[75,78],{"id":76,"by":11,"text":77},"38496044","Good point - the main issue is we encountered this exact issue with our old package Hyperlearn (https:\u002F\u002Fgithub.com\u002Fdanielhanchen\u002Fhyperlearn). I OSSed all the code to the…",{"id":79,"by":80,"text":81},"38495915","squigz","What is the motivation behind open sourcing code at all if you're not going to open source it all? I don't mean this to be an asshole, as I truly want to understand the motivation.",[83,87,91],{"value":84,"source":49,"evidence":85},"Triton kernels",[86],{"id":52,"by":11,"text":53},{"value":88,"source":49,"evidence":89},"Xformers for Flash Attention",[90],{"id":52,"by":11,"text":53},{"value":92,"source":49,"evidence":93},"Manual autograd engine with hand-derived backprop",[94],{"id":52,"by":11,"text":53},"open_source","beta","team",{"praise":99,"objections":119,"requests":141,"questions":162,"alternatives":177},[100,112],{"value":101,"source":102,"evidence":103},"Interesting results and excitement to try it.","commenter",[104,108],{"id":105,"by":106,"text":107},"38496601","neilmovva","promising results, excited to try it out! question on the perf benchmarks: why do all the results with 2 GPUs & DDP take longer than the single GPU case? Both benchmarks do the…",{"id":109,"by":110,"text":111},"38494949","TuringNYC","wow i wish i could do this with all my M1 pro max neural cores",{"value":113,"source":102,"evidence":114},"Maker is refreshingly open about monetization.",[115],{"id":116,"by":117,"text":118},"38496380","ShamelessC","Refreshing to see, especially on this forum! Best of luck.",[120,127,134],{"value":121,"source":102,"evidence":122},"Multi-GPU benchmarks look slower than single GPU, questioning the test.",[123,126],{"id":124,"by":106,"text":125},"38497114","I agree that synchronization causes overhead, so 2x GPUs won't achieve the ideal 0.5x total runtime. But here, taking your Alpaca benchmark as an example, we are seeing 2x GPUs…",{"id":105,"by":106,"text":107},{"value":128,"source":102,"evidence":129},"Paid tier gating of the fastest version seems odd.",[130],{"id":131,"by":132,"text":133},"38495505","yunohn","This seems very interesting, but I’m very confused why you have gated the \u002Fmax\u002Fimum speedup version for enterprise-only? It would make more sense to have only the Free and Paid…",{"value":135,"source":102,"evidence":136},"Open-core model seen as shareware-like.",[137],{"id":138,"by":139,"text":140},"38495928","sbierwagen","Shareware. See the open source preview, pay a million dollars for the full version.",[142,153,158],{"value":143,"source":102,"evidence":144},"Support for older GPUs such as 1080 Ti and P100.",[145,149],{"id":146,"by":147,"text":148},"38495534","0x6c6f6c","I see this mentions 2018+ GPUs, but I'm curious why this wouldn't work on say a 1080Ti. My cursory look at the hardware specs shows this has support for CUDA8+ and this states…",{"id":150,"by":151,"text":152},"38494918","graphe","Somewhat related: is it worth it to use a P100 or P40? I was gonna get one but seems like pascal isn't being supported by more and more projects.",{"value":154,"source":102,"evidence":155},"Apple M1 support.",[156,157],{"id":109,"by":110,"text":111},{"id":60,"by":11,"text":61},{"value":159,"source":102,"evidence":160},"Share benchmark code.",[161],{"id":124,"by":106,"text":125},[163,170],{"value":164,"source":102,"evidence":165},"How does it compare to PyTorch inference optimizations?",[166],{"id":167,"by":168,"text":169},"38500845","naveen99","How does this compare to PyTorch labs optimizations for Sam and llama2 ? https:\u002F\u002Fgithub.com\u002Fpytorch-labs\u002Fsegment-anything-fast https:\u002F\u002Fgithub.com\u002Fpytorch-labs\u002Fgpt-fast",{"value":171,"source":102,"evidence":172},"What are minimum hardware requirements?",[173],{"id":174,"by":175,"text":176},"38497147","3abiton","Any source recommendations on howto find out the minimum hw requirementsto train LLM with Xbit and YB parameters?",[178,182],{"value":179,"source":49,"evidence":180},"Hugging Face's original training implementation.",[181],{"id":52,"by":11,"text":53},{"value":183,"source":102,"evidence":184},"PyTorch gpt-fast and segment-anything-fast (inference only).",[185],{"id":167,"by":168,"text":169},{"comments":187,"answered":188,"key":189},37,16,[190,193,196],{"id":191,"text":192},"38500136","Oh thanks for the cool tips! Super good points - I'll have to ask around sadly I'm just the engineering guy loll :(",{"id":194,"text":195},"38504971","Thanks a bunch a lot for the kind support!! If I have any questions I'll definitely ask - again much appreciated!!",{"id":197,"text":198},"38639080","For those interested - just released a new blog post on all our optimizations. There are also 59 fully reproducible benchmarks! https:\u002F\u002Funsloth.ai\u002Fblog\u002Fmistral-benchmark",{"commenters":200,"max_depth":201,"maker_replies":187,"first_comment_mins":34,"lifespan_hours":202},27,8,306,[204,208,210,212,213,215,218,220,222,225,227,229,230,232,234,236,238,240,242,244,247,249,251,254,256,257,259,264,266,269,272,275,277,279,281,283,285,290,292,296,299,301,302,307,309,311,312,313,316,318,320,322,323,326,328,331,333,336,338,340,342,345,348,351,353,356,358,360,364,366,370,372,377],[205,34,18,206,207,34],"38499039","maker\u002Fneutral\u002Fpricing","peytoncasper",[191,34,34,209,11,39],"commenter\u002Fneutral\u002Fnone",[211,34,34,209,207,34],"38504184",[194,34,34,209,11,39],[197,39,34,214,11,39],"project\u002Fpositive\u002Fperformance",[105,216,18,217,106,34],2,"project\u002Fneutral\u002Fperformance",[219,216,34,217,11,39],"38496734",[124,216,39,221,106,34],"project\u002Fcritical\u002Fperformance",[223,216,34,224,11,39],"38497250","project\u002Fneutral\u002Fdocs",[167,18,18,226,168,34],"project\u002Fneutral\u002Fnone",[228,18,34,226,11,39],"38504568",[150,32,18,226,151,34],[231,32,34,226,11,39],"38495254",[233,32,18,226,151,34],"38502035",[235,32,34,226,11,39],"38504977",[131,35,39,237,132,34],"project\u002Fcritical\u002Fpricing",[70,35,34,239,11,39],"project\u002Fneutral\u002Fpricing",[241,35,18,239,132,34],"38495582",[243,35,34,239,11,39],"38495939",[245,35,18,239,246,34],"38497897","hirako2000",[248,35,34,239,11,39],"38498103",[146,250,18,226,147,34],6,[252,250,18,226,253,34],"38495770","zargon",[255,250,34,226,11,39],"38495936",[174,250,18,226,175,34],[258,250,34,224,11,39],"38497257",[260,261,18,262,263,34],"38495386",7,"project\u002Fneutral\u002Fidea","gentleman11",[265,261,34,262,11,39],"38495421",[267,261,18,262,268,34],"38496066","selfhoster11",[270,261,34,271,11,39],"38496107","other\u002Fneutral\u002Fnone",[273,201,18,262,274,34],"38494555","foota",[276,201,34,262,11,39],"38495128",[278,201,18,262,274,34],"38496049",[280,201,34,262,11,39],"38496151",[282,201,18,262,274,34],"38496903",[284,201,34,209,11,39],"38497042",[286,287,35,288,289,34],"38495049",9,"project\u002Fpositive\u002Fdesign","xrd",[291,287,34,288,11,39],"38495100",[293,287,39,294,295,34],"38495259","project\u002Fcritical\u002Fexecution","juunpp",[297,287,34,298,11,39],"38495300","project\u002Fneutral\u002Fexecution",[109,300,18,217,110,34],10,[60,300,34,226,11,39],[303,304,35,305,306,34],"38496663",11,"project\u002Fpositive\u002Fidea","aerioux",[308,304,34,262,11,39],"38496742",[79,310,18,206,80,34],12,[138,310,39,237,139,34],[67,310,34,239,11,39],[314,310,39,315,117,34],"38495961","maker\u002Fcritical\u002Fpricing",[317,310,34,206,11,39],"38496060",[116,310,35,319,117,34],"maker\u002Fpositive\u002Fnone",[321,310,34,209,11,39],"38496432",[76,310,34,206,11,39],[324,310,18,239,325,34],"38496057","kristopolous",[327,310,34,239,11,39],"38496072",[329,310,18,239,330,34],"38496178","Jayakumark",[332,310,34,239,11,39],"38496285",[334,310,34,271,335,34],"38496188","Zuiii",[337,310,34,271,11,39],"38496290",[339,310,34,271,335,34],"38496330",[341,310,34,209,11,39],"38496431",[343,310,18,344,80,34],"38496636","maker\u002Fneutral\u002Fidea",[346,310,34,347,11,39],"38496785","maker\u002Fneutral\u002Fnone",[349,310,18,347,350,34],"38496933","nikolayasdf123",[352,310,34,347,11,39],"38497051",[354,310,34,271,355,34],"38498079","MyFirstSass",[357,310,34,347,11,39],"38498183",[52,359,34,214,11,39],13,[361,362,35,363,350,34],"38496944",14,"project\u002Fpositive\u002Fpricing",[365,362,34,239,11,39],"38497033",[367,362,34,368,369,34],"38498013","other\u002Fneutral\u002Fpricing","rightbyte",[371,362,34,368,11,39],"38498624",[373,374,34,375,376,34],"38496167",15,"other\u002Fcritical\u002Fnone","29athrowaway",[378,374,34,347,11,39],"38496305",[380,383,384],{"id":381,"t":382,"points":18,"comments":34},"38475955",1701364106,{"id":4,"t":12,"points":14,"comments":15},{"id":385,"t":386,"points":201,"comments":216},"47414372",1773762658,[388,393,398,403,408,413],{"id":389,"name":390,"points":391,"category":392},"38531759","Beeper Mini",1521,"consumer_social",{"id":394,"name":395,"points":396,"category":397},"38531604","How did I get here",659,"dev_tools",{"id":399,"name":400,"points":401,"category":402},"38506166","Plantarium",439,"creative",{"id":404,"name":405,"points":406,"category":407},"38473743","PowerSync",356,"infrastructure",{"id":409,"name":410,"points":411,"category":412},"38472333","Paper Tactics",303,"games",{"id":414,"name":415,"points":416,"category":402},"38509173","MAIM",229,false,"v2",{"id":420,"name":5,"line":6,"ai":417,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":421,"top":18,"kw":422,"maker":11,"domain":10,"tier":19,"attempts":18,"strip":423,"url":9,"mix":424,"praise":101,"objection":121,"sub":21},38487199,0.03,[24,25,26,27,28],"3003133313303333303351353115033003300500",[32,33,34,35],1794036645466]