[{"data":1,"prerenderedAt":219},["ShallowReactive",2],{"project-38304148":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":32,"aspects":33,"strip":36,"post_text":38,"what":39,"said":75,"maker_in_thread":122,"shape":135,"grid":139,"attempts":175,"same_week":183,"maker_shows":171,"maker_hidden":212,"reader_version":213,"card":214},"38304148","tensorli","less than 650 LOC trainable GPT only using NumPy","title","Show HN: less than 650 LOC trainable GPT only using NumPy","https:\u002F\u002Fgithub.com\u002Fjoennlae\u002Ftensorli","github.com","joennlae",1700231674,2023,90,18,714,3109,23,"notable","learning_science","educational machine learning implementation","library",[24,25,26,27,28],"gpt","numpy","transformer","neural network","from scratch","curious",0,{"positive":30,"neutral":32,"mixed":30,"critical":30},5,{"docs":34,"performance":35},{"positive":30,"critical":30},{"positive":30,"critical":30},[30,30,37,30,30,30,30,30,30,30,30,30,37,37,30,37,37,30],3,null,{"problem":40,"audience":38,"pricing":38,"motivation":47,"time_to_build":38,"stack":51,"openness":73,"stage":74,"team_size":74},{"value":41,"source":42,"evidence":43},"Shows how a GPT-style transformer works by implementing training in NumPy","maker",[44],{"id":45,"by":11,"text":46},"38312145","This would be interesting to consider. But at the moment, nothing is optimized, so many things must be tackled first (especially in the backwards path, for example, buffering) to…",{"value":48,"source":42,"evidence":49},"Educational exercise for the maker",[50],{"id":45,"by":11,"text":46},[52,60,65],{"value":53,"source":54,"evidence":55},"Python","inferred",[56],{"id":57,"by":58,"text":59},"38308513","p1esk","I wonder how easy it would be to port this library from numpy to cupy.",{"value":61,"source":42,"evidence":62},"NumPy",[63],{"id":64},"post",{"value":66,"source":67,"evidence":68},"PyTorch (used in equivalence tests)","commenter",[69],{"id":70,"by":71,"text":72},"38308734","gfaure","Nice! The README mentions `LayerNorm` is implemented here, but while it's in the equivalence tests with PyTorch, I don't see it in the implementation.","open_source","unknown",{"praise":76,"objections":84,"requests":104,"questions":113,"alternatives":121},[77],{"value":78,"source":67,"evidence":79},"Interesting demo of how small a GPT can be",[80],{"id":81,"by":82,"text":83},"38314428","Paul-Craft","Mo Gawdat has famously said that GPT-4 was something like \"4300 lines of code,\" and that he could have written that when he was a kid. He's clearly a smart man, so, I think we…",[85,96],{"value":86,"source":67,"evidence":87},"Title uses the branded name GPT for generic transformer architecture",[88,92],{"id":89,"by":90,"text":91},"38307533","cuuupid","I think people are forgetting that transformer architectures are a wider field from GPT and predate GPT3 by 3+ years. Referring to transformer architectures using a branded…",{"id":93,"by":94,"text":95},"38307668","tverbeure","FWIW: the GitHub project description says “GPT-like”. It’s the title here that dropped the “like”.",{"value":97,"source":67,"evidence":98},"LayerNorm is listed in README but not visible as a separate implementation",[99,100],{"id":70,"by":71,"text":72},{"id":101,"by":102,"text":103},"38309767","dauertewigkeit","It's part of the TensorLi definition where all the magic happens.",[105],{"value":106,"source":67,"evidence":107},"Port to CuPy or cuNumeric for GPU speed",[108,109],{"id":57,"by":58,"text":59},{"id":110,"by":111,"text":112},"38310283","eslaught","Or cuNumeric: https:\u002F\u002Fdeveloper.nvidia.com\u002Fcunumeric",[114],{"value":115,"source":67,"evidence":116},"Whether GPT is subject to trademark",[117],{"id":118,"by":119,"text":120},"38312030","quickthrower2","Is GPT subject to trademark. It stands for Generative Pre-training Transformer?",[],{"comments":123,"answered":124,"key":125},4,1,[126,129,132],{"id":127,"text":128},"38312085","They are still applying: https:\u002F\u002Ftmsearch.uspto.gov\u002Fbin\u002Fshowfield?f=doc&state=4805:wl5r8q.4.6",{"id":130,"text":131},"38312050","The author here: I absolutely agree with you. I went for a bit more catchy title.",{"id":133,"text":134},"38312194","That is true. I went for a simple implementation of the layer norm and included it in the tensorli definition. But it would have been better to define it as a moduli for clarity.",{"commenters":136,"max_depth":123,"maker_replies":123,"first_comment_mins":137,"lifespan_hours":138},13,218,12,[140,142,145,147,150,153,155,158,159,160,162,164,165,166,168,170,173,174],[89,30,30,141,90,30],"other\u002Fneutral\u002Fnone",[143,30,30,141,144,30],"38307565","PartiallyTyped",[93,30,37,146,94,30],"project\u002Fneutral\u002Fdocs",[148,30,30,141,149,30],"38307736","jimmyl02",[151,30,30,141,152,30],"38308489","cchance",[154,30,30,141,82,30],"38314480",[156,30,30,141,157,30],"38307907","__loam",[118,30,30,141,119,30],[127,30,30,141,11,124],[161,30,30,141,82,30],"38314473",[130,30,30,163,11,124],"maker\u002Fneutral\u002Fnone",[81,30,30,141,82,30],[70,124,37,146,71,30],[101,124,37,167,102,30],"project\u002Fneutral\u002Fnone",[133,124,30,169,11,124],"maker\u002Fneutral\u002Fdesign",[57,171,37,172,58,30],2,"project\u002Fneutral\u002Fperformance",[110,171,37,172,111,30],[45,171,30,172,11,124],[176,179,182],{"id":177,"t":178,"points":37,"comments":30},"38250479",1699885824,{"id":180,"t":181,"points":171,"comments":30},"38291766",1700152911,{"id":4,"t":12,"points":14,"comments":15},[184,189,193,197,202,207],{"id":185,"name":186,"points":187,"category":188},"38327017","Ad Accelerator",713,"consumer_social",{"id":190,"name":191,"points":192,"category":20},"38316936","UneeBee",459,{"id":194,"name":195,"points":196,"category":20},"38365638","Pickcode",420,{"id":198,"name":199,"points":200,"category":201},"38285228","screenshot-to-code",358,"dev_tools",{"id":203,"name":204,"points":205,"category":206},"38368570","Neum AI",155,"ai",{"id":208,"name":209,"points":210,"category":211},"38360776","Stella Nera",154,"hardware",false,"v2",{"id":215,"name":5,"line":6,"ai":212,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":30,"top":18,"kw":216,"maker":11,"domain":10,"tier":19,"attempts":37,"strip":217,"url":9,"mix":218,"praise":78,"objection":86,"sub":21},38304148,[24,25,26,27,28],"003000000000330330",[30,32,30,30],1794036644861]