[{"data":1,"prerenderedAt":694},["ShallowReactive",2],{"project-47655408":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":34,"aspects":35,"strip":45,"post_text":48,"what":49,"said":89,"maker_in_thread":237,"shape":249,"grid":254,"attempts":655,"same_week":656,"maker_shows":41,"maker_hidden":686,"reader_version":687,"card":688},"47655408","GuppyLM","A tiny LLM to demystify how language models work","title","Show HN: I built a tiny LLM to demystify how language models work","https:\u002F\u002Fgithub.com\u002Farman-bd\u002Fguppylm","github.com","armanified",1775434812,2026,915,134,14,4366,1,"legend","learning_science","educational llm implementation","library",[24,25,26,27],"small language model","transformer from scratch","pytorch tutorial","llm education","mixed",0.463,{"positive":31,"neutral":32,"mixed":18,"critical":33},33,25,6,65,{"docs":36,"idea":39,"execution":42,"performance":43,"design":44},{"positive":37,"critical":38},4,0,{"positive":40,"critical":41},18,2,{"positive":18,"critical":37},{"positive":38,"critical":38},{"positive":18,"critical":38},[46,38,38,37,38,46,38,38,47,47,38,46,38,38,47,46,46,37,37,47,46,37,47,46,46,47,47,47,46,18,47,46,47,38,38,38,46,47,47,47],3,5,"Built a ~9M param LLM from scratch to understand how they actually work. Vanilla transformer, 60K synthetic conversations, ~130 lines of PyTorch. Trains in 5 min on a free Colab T4. The fish thinks the meaning of life is food.\n\nFork it and swap the personality for your own character.",{"problem":50,"audience":56,"pricing":72,"motivation":73,"time_to_build":72,"stack":77,"openness":86,"stage":87,"team_size":88},{"value":51,"source":52,"evidence":53},"Makes how language models work less mysterious by building a tiny one from scratch.","maker",[54],{"id":55},"post",{"value":57,"source":58,"evidence":59},"Developers and learners who want to understand transformers by reading and training a small model.","commenter",[60,64,68],{"id":61,"by":62,"text":63},"47660830","thomasfl","Is there some documentation for this? The code is probably the simplest (Not So) Large Language Model implementation possible, but it is not straight forward to understand for…",{"id":65,"by":66,"text":67},"47662856","jbethune","Forked. Very cool. I appreciate the simplicity and documentation.",{"id":69,"by":70,"text":71},"47661611","rpdaiml","This is a nice idea. A tiny implementation can be way more useful for learning than yet another wrapper around a big model, especially if it keeps the training loop and inference…",null,{"value":74,"source":52,"evidence":75},"To understand how language models actually work by building one.",[76],{"id":55},[78,82],{"value":79,"source":52,"evidence":80},"PyTorch",[81],{"id":55},{"value":83,"source":52,"evidence":84},"Google Colab T4 GPU",[85],{"id":55},"open_source","launched","unknown",{"praise":90,"objections":125,"requests":164,"questions":194,"alternatives":221},[91,103,114],{"value":92,"source":58,"evidence":93},"Great educational tool for learning how LLMs work",[94,98,99],{"id":95,"by":96,"text":97},"47656472","ordinarily","It's genuinely a great introduction to LLMs. I built my own awhile ago based off Milton's Paradise Lost: https:\u002F\u002Fwww.wvrk.org\u002Fworks\u002Fmilton",{"id":69,"by":70,"text":71},{"id":100,"by":101,"text":102},"47663129","EmilioOldenziel","Building it yourself is always the best test if you really understand how it works.",{"value":104,"source":58,"evidence":105},"Fun personality and humor in the fish responses",[106,110],{"id":107,"by":108,"text":109},"47656073","AndrewKemendo","I love these kinds of educational implementations. I want to really praise the (unintentional?) nod to Nagel, by limiting capabilities to representation of a fish, the user is…",{"id":111,"by":112,"text":113},"47656684","gnarlouse","I... wow, you made an LLM that can actually tell jokes?",{"value":115,"source":58,"evidence":116},"Simple, readable code without abstractions",[117,121],{"id":118,"by":119,"text":120},"47657756","jiusanzhou","The decision to strip out GQA\u002FRoPE\u002FSwiGLU and go vanilla transformer is the right call here — at 9M params those additions add complexity without meaningful gains, and keeping…",{"id":122,"by":123,"text":124},"47659993","areys","The constraint-driven approach here is what makes it actually useful as a learning tool. When you're working with ~130 lines of PyTorch, you can't hide behind abstractions —…",[126,136,143,150],{"value":127,"source":58,"evidence":128},"Examples appear copied from training data and the model handles unknown queries poorly",[129,133],{"id":130,"by":131,"text":132},"47657381","zwaps","I like the idea, just that the examples are reproduced from the training data set. How does it handle unknown queries?",{"id":134,"by":11,"text":135},"47659630","It mostly doesn't, at 9M it has very limited capacity. The whole idea of this project is to demonstrate how Language Models work.",{"value":137,"source":58,"evidence":138},"Title oversells it; the project does not really demystify anything",[139],{"id":140,"by":141,"text":142},"47658521","Propelloni","Great work! I still think that [1] does a better job of helping us understand how GPT and LLM work, but yours is funnier. Then, some criticism. I probably don't get it, but I…",{"value":144,"source":58,"evidence":145},"Seems like low-effort AI-generated project",[146],{"id":147,"by":148,"text":149},"47660754","rahen","I don't mean to be 'that guy', but after a quick review, this really feels like low-effort AI slop to me. There is nothing wrong using AI tools to write code, but nothing here…",{"value":151,"source":58,"evidence":152},"Lowercase-only training data means uppercase input is ignored",[153,157,161],{"id":154,"by":155,"text":156},"47657431","mudkipdev","This is probably a consequence of the training data being fully lowercase: You> hello Guppy> hi. did you bring micro pellets. You> HELLO Guppy> i don't know what it means but…",{"id":158,"by":159,"text":160},"47659214","Duplicake","I love this! Seems like it can't understand uppercase letters though",{"id":162,"by":11,"text":163},"47659711","Uppercase letters were intentionally ignored.",[165,180,187],{"value":166,"source":58,"evidence":167},"Documentation on how the synthetic data was generated",[168,172,176],{"id":169,"by":170,"text":171},"47661663","CaseFlatline","I am trying to find how the synthetic data was created (looking through the repo) and didn't find it. Maybe I am missing it - Would love to see the prompts and process on that…",{"id":173,"by":174,"text":175},"47659103","fawabc","how did you generate the synthetic data?",{"id":177,"by":178,"text":179},"47658488","gdzie-jest-sol","* How creating dataset? I download it but it is commpresed in binary format. * How training. In cloud or in my own dev * How creating a gguf",{"value":181,"source":58,"evidence":182},"Comparison with microgpt and minGPT",[183],{"id":184,"by":185,"text":186},"47658988","fg137","How does this compare to Andrej Karpathy's microgpt (https:\u002F\u002Fkarpathy.github.io\u002F2026\u002F02\u002F12\u002Fmicrogpt\u002F) or minGPT (https:\u002F\u002Fgithub.com\u002Fkarpathy\u002FminGPT)?",{"value":188,"source":58,"evidence":189},"Write-up on how it was built and what was referenced",[190],{"id":191,"by":192,"text":193},"47657270","kaipereira","This is so cool! I'd love to see a write-up on how made it, and what you referenced because designing neural networks always feel like a maze ;)",[195,203,214],{"value":196,"source":58,"evidence":197},"How does it handle longer context and unknown queries?",[198,202],{"id":199,"by":200,"text":201},"47657319","kubrador","how's it handle longer context or does it start hallucinating after like 2 sentences? curious what the ceiling is before the 9M params",{"id":130,"by":131,"text":132},{"value":204,"source":58,"evidence":205},"How small can the training data and model get before output stops being coherent?",[206,210],{"id":207,"by":208,"text":209},"47656687","martmulx","How much training data did you end up needing for the fish personality to feel coherent? Curious what the minimum viable dataset looks like for something like this.",{"id":211,"by":212,"text":213},"47656815","LeonTing1010","This is a great way to learn. Most people treat LLMs as black boxes, but building one from scratch — even tiny — forces you to understand attention, tokenization, and loss at a…",{"value":215,"source":58,"evidence":216},"Can it be trained through conversation alone?",[217],{"id":218,"by":219,"text":220},"47658470","bblb","Could it be possible to train LLM only through the chat messages without any other data or input? If Guppy doesn't know regular expressions yet, could I teach it to it just by…",[222,226,233],{"value":223,"source":58,"evidence":224},"Karpathy's microgpt and minGPT",[225],{"id":184,"by":185,"text":186},{"value":227,"source":58,"evidence":228},"bbycroft.net LLM 3D visualizer",[229],{"id":230,"by":231,"text":232},"47658540","totetsu","https:\u002F\u002Fbbycroft.net\u002Fllm has 3d Visualization of tiny example LLM layers that do a very good job at showing what is going on (https:\u002F\u002Fnews.ycombinator.com\u002Fitem?id=38505211)",{"value":234,"source":58,"evidence":235},"rasbt LLMs-from-scratch",[236],{"id":140,"by":141,"text":142},{"comments":238,"answered":47,"key":239},9,[240,243,246],{"id":241,"text":242},"47659601","I haven't compared it with anything yet. Thanks for the suggestion; I'll look into these.",{"id":244,"text":245},"47659724","Pretty neat! I'll definitely take a deeper look into this.",{"id":247,"text":248},"47659783","OMG! Why didn't I thought fo this first :P",{"commenters":250,"max_depth":251,"maker_replies":238,"first_comment_mins":252,"lifespan_hours":253},107,7,93,159,[255,257,259,263,267,269,271,274,276,279,282,285,287,291,294,298,301,304,306,310,313,315,318,320,323,324,328,329,332,335,337,339,343,347,350,352,355,357,359,362,366,368,370,371,374,377,380,383,386,389,392,394,397,399,402,405,408,410,412,415,418,423,427,429,431,432,436,441,443,446,450,454,457,458,462,464,467,469,472,476,479,481,484,488,491,494,496,500,502,504,506,510,514,517,519,523,526,530,534,538,540,542,544,547,549,553,557,561,565,569,573,575,578,582,586,588,590,592,595,597,600,602,605,607,610,612,615,618,622,624,626,629,631,634,637,640,643,647,650,653],[184,38,46,256,185,38],"project\u002Fneutral\u002Fnone",[241,38,38,258,11,18],"maker\u002Fneutral\u002Fnone",[260,38,38,261,262,38],"47660479","commenter\u002Fpositive\u002Fnone","BrokenCogs",[264,38,38,265,266,38],"47660639","commenter\u002Fneutral\u002Fnone","tantalor",[268,38,38,265,262,38],"47660953",[270,38,38,265,266,38],"47661087",[272,38,38,265,273,38],"47661317","philipallstar",[275,38,38,265,185,38],"47661740",[277,38,38,265,278,38],"47660871","Kye",[280,38,38,265,281,38],"47662757","layer8",[283,38,38,265,284,38],"47664228","stronglikedan",[61,18,47,286,62,38],"project\u002Fpositive\u002Fdocs",[288,18,46,289,290,38],"47661028","project\u002Fneutral\u002Fdocs","achenatx",[292,18,46,289,293,38],"47662419","dominotw",[295,18,38,296,297,38],"47662721","commenter\u002Fneutral\u002Fdocs","bigmadshoe",[299,18,38,300,284,38],"47664184","other\u002Fneutral\u002Fdocs",[302,18,38,300,303,38],"47665243","dr_hooo",[305,18,46,289,297,38],"47678632",[307,18,38,308,309,38],"47662754","other\u002Fneutral\u002Fnone","sigmoid10",[311,18,46,289,312,38],"47662783","arcanemachiner",[314,18,38,308,284,38],"47664157",[316,18,38,308,317,38],"47665817","libria",[319,18,38,308,293,38],"47667841",[321,18,38,308,322,38],"47676127","BorisMelnik",[230,41,38,308,231,38],[325,41,38,326,327,38],"47659377","other\u002Fpositive\u002Fnone","maverickxone",[244,41,38,308,11,18],[330,41,38,326,331,38],"47660577","skramzy",[333,41,38,326,334,38],"47672681","devsteru",[95,46,47,336,96,38],"project\u002Fpositive\u002Fidea",[154,37,46,338,155,38],"project\u002Fneutral\u002Fexecution",[340,37,47,341,342,38],"47657703","project\u002Fpositive\u002Fexecution","functional_dev",[344,47,38,345,346,38],"47659731","other\u002Fneutral\u002Fidea","algoth1",[348,47,38,308,349,38],"47661623","MarkusQ",[351,47,38,308,346,38],"47663454",[353,47,38,308,354,38],"47664968","eden-u4",[356,47,38,308,349,38],"47666314",[358,47,38,308,155,38],"47670930",[360,33,38,308,361,38],"47660517","neurworlds",[363,251,46,364,365,38],"47656263","project\u002Fneutral\u002Fidea","SilentM68",[247,251,38,367,11,18],"maker\u002Fneutral\u002Fidea",[130,369,46,338,131,38],8,[134,369,38,367,11,18],[372,238,38,308,373,38],"47657440","brcmthrowaway",[375,238,38,308,376,38],"47657453","AlecSchueler",[378,238,38,308,379,38],"47657818","loveparade",[381,238,38,308,382,38],"47658502","59nadir",[384,238,38,308,385,38],"47659016","wiseowise",[387,238,38,308,388,38],"47659098","toyg",[390,238,38,308,391,38],"47659739","siva7",[107,393,47,336,108,38],10,[395,393,38,265,396,38],"47656282","dvt",[398,393,38,265,108,38],"47656367",[400,393,46,364,401,38],"47657561","Terr_",[403,393,38,265,404,38],"47657645","andoando",[406,393,38,265,407,38],"47660534","vixen99",[409,393,38,265,404,38],"47662841",[218,411,46,364,219,38],11,[413,411,38,265,414,38],"47659401","roetlich",[416,411,46,364,417,38],"47661029","tatrions",[419,420,47,421,422,38],"47656569",12,"project\u002Fpositive\u002Fnone","cbdevidal",[424,420,46,425,426,38],"47656881","project\u002Fneutral\u002Fperformance","vunderba",[169,428,46,289,170,38],13,[430,428,46,289,426,38],"47662776",[69,16,47,336,70,38],[433,434,46,364,435,38],"47661880",15,"jzer0cool",[437,438,47,439,440,38],"47664221",16,"maker\u002Fpositive\u002Fnone","BiraIgnacio",[191,442,47,286,192,38],17,[444,40,47,336,445,38],"47657236","ankitsanghi",[447,448,47,421,449,38],"47656768",19,"NyxVox",[451,452,47,336,453,38],"47661474",20,"Leomuck",[158,455,18,456,159,38],21,"project\u002Fcritical\u002Fexecution",[162,455,38,258,11,18],[459,460,47,336,461,38],"47666522",22,"ergocoder",[111,463,47,421,112,38],23,[465,463,18,456,466,38],"47657832","murkt",[199,468,46,425,200,38],24,[470,32,47,336,471,38],"47664234","bharat1010",[473,474,46,338,475,38],"47659509",26,"drincanngao",[477,474,38,478,11,18],"47659593","project\u002Fneutral\u002Fdesign",[173,480,46,289,174,38],27,[482,480,46,289,483,38],"47690039","abkolan",[485,486,38,308,487,38],"47657734",28,"rclkrtrzckr",[489,486,38,308,490,38],"47658360","search_facility",[492,486,38,308,493,38],"47658999","danparsonson",[495,486,38,308,385,38],"47659022",[497,498,46,425,499,38],"47659428",29,"amelius",[501,498,38,425,11,18],"47659698",[503,498,38,308,499,38],"47660136",[100,505,47,336,101,38],30,[507,508,47,336,509,38],"47659002",31,"ananandreas",[511,512,47,336,513,38],"47658519",32,"ben8bit",[515,31,47,336,516,38],"47661348","nobodyandproud",[65,518,47,286,66,38],34,[520,521,47,336,522,38],"47656191",35,"nullbyte808",[524,521,38,525,11,18],"47659761","maker\u002Fpositive\u002Fidea",[527,528,46,364,529,38],"47657643",36,"monksy",[531,532,47,336,533,38],"47658155",37,"cpldcpu",[535,536,47,421,537,38],"47662556",38,"winter_blue",[177,539,46,289,178,38],39,[541,539,18,456,178,38],"47658493",[543,539,46,338,178,38],"47658798",[545,539,38,308,546,38],"47658523","freetonik",[147,548,18,456,148,38],40,[550,548,18,551,552,38],"47683517","project\u002Fcritical\u002Fidea","zhainya",[554,555,47,421,556,38],"47661009",41,"Vektorceraptor",[558,559,47,421,560,38],"47676800",42,"tombelieber",[562,563,47,421,564,38],"47658856",43,"Elengal",[566,567,47,421,568,38],"47657245",44,"oyebenny",[570,571,18,551,572,38],"47660163",45,"hughw",[574,571,38,364,11,18],"47660234",[576,571,38,265,577,38],"47660326","uxcolumbo",[579,580,47,336,581,38],"47671512",46,"hahooh",[583,584,47,336,585,38],"47658213",47,"hackerman70000",[587,584,47,336,499,38],"47659440",[589,584,38,308,391,38],"47659703",[591,584,38,308,499,38],"47660067",[593,584,38,308,594,38],"47659728","zkmon",[596,584,38,308,499,38],"47659842",[598,584,38,308,599,38],"47660185","darepublic",[601,584,38,308,407,38],"47660452",[603,584,38,265,604,38],"47667395","michaelhoney",[606,584,38,308,266,38],"47660664",[608,584,38,265,609,38],"47661313","philote",[611,584,38,265,266,38],"47661497",[613,584,38,308,614,38],"47661665","hca",[616,584,38,265,617,38],"47674828","inanutshellus",[619,620,38,308,621,38],"47741102",48,"aimemobe",[207,623,46,289,208,38],49,[211,625,46,289,212,38],50,[627,625,47,286,628,38],"47656849","secabeen",[122,630,47,336,123,38],51,[632,630,38,265,633,38],"47660094","moonu",[118,635,47,636,119,38],52,"project\u002Fpositive\u002Fdesign",[638,635,38,265,639,38],"47657803","ngruhn",[641,635,38,265,642,38],"47657870","3m",[644,645,47,336,646,38],"47657185",53,"dinkumthinkum",[140,648,46,649,141,38],54,"project\u002Fmixed\u002Fidea",[651,648,38,265,652,38],"47659206","jadengeller",[654,648,38,265,141,38],"47681427",[],[657,661,666,671,676,681],{"id":658,"name":659,"points":660,"category":20},"47640728","mvidia",964,{"id":662,"name":663,"points":664,"category":665},"47673360","Brutalist Concrete Laptop Stand",786,"hardware",{"id":667,"name":668,"points":669,"category":670},"47625952","Blogosphere",783,"consumer_social",{"id":672,"name":673,"points":674,"category":675},"47624645","Apfel",743,"ai",{"id":677,"name":678,"points":679,"category":680},"47696562","Is Hormuz open yet?",484,"web_saas",{"id":682,"name":683,"points":684,"category":685},"47666024","Ghost Pepper",467,"productivity",false,"v2",{"id":689,"name":5,"line":6,"ai":686,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":690,"top":18,"kw":691,"maker":11,"domain":10,"tier":19,"attempts":38,"strip":692,"url":9,"mix":693,"praise":92,"objection":127,"sub":21},47655408,0.46,[24,25,26,27],"3004030055030053344534533555315350003555",[31,32,18,33],1794036697905]