[{"data":1,"prerenderedAt":298},["ShallowReactive",2],{"project-39726781":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":34,"aspects":35,"strip":40,"post_text":43,"what":44,"said":92,"maker_in_thread":163,"shape":166,"grid":170,"attempts":260,"same_week":261,"maker_shows":37,"maker_hidden":290,"reader_version":291,"card":292},"39726781","flash-attention-minimal","Flash Attention in ~100 lines of CUDA","title","Show HN: Flash Attention in ~100 lines of CUDA","https:\u002F\u002Fgithub.com\u002Ftspeterkim\u002Fflash-attention-minimal","github.com","tspeterkim",1710603119,2024,230,39,324,3515,10,"notable","ai","ml kernels","library",[24,25,26,27],"flash attention","cuda kernel","transformer attention","gpu kernels","debated",0.018,{"positive":31,"neutral":32,"mixed":33,"critical":31},2,4,0,8,{"execution":36,"docs":38,"idea":39},{"positive":37,"critical":37},1,{"positive":33,"critical":37},{"positive":37,"critical":33},[41,33,33,33,33,33,37,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,41,33,33,33,33,33,37,33,41,42,33,41,42,33,33],3,5,null,{"problem":45,"audience":59,"pricing":43,"motivation":70,"time_to_build":43,"stack":76,"openness":90,"stage":91,"team_size":91},{"value":46,"source":47,"evidence":48},"Shows how FlashAttention works by implementing it compactly in CUDA","inferred",[49,51,55],{"id":50},"post",{"id":52,"by":53,"text":54},"39727520","araes","For those who have no idea what's being discussed, quick background. Discussing: Transformer [1] memory issues and approximate attention [2] in machine learning training.…",{"id":56,"by":57,"text":58},"39731535","danielhanchen","Fantastic work! Extremely neat and clear implementation! Interesting note on the backward pass - what do you think are the main blockers for a backward pass?",{"value":60,"source":47,"evidence":61},"Developers learning ML kernels and GPU programming",[62,66],{"id":63,"by":64,"text":65},"39728210","zer0zzz","This is fantastic. I am just starting in the ML space (compile from compilers) and I love short kernels that I can use to understand things better with.",{"id":67,"by":68,"text":69},"39727770","saiojd","What does __syncthreads() do here exactly? I'm new to CUDA, could get the overall idea of the FlashAttention paper but not the details.",{"value":71,"source":47,"evidence":72},"Maker aimed to understand and reimplement FlashAttention minimally",[73],{"id":74,"by":11,"text":75},"39733204","Thanks Daniel. The main blocker is me not able to fully grasp the backward pass. (trying to understand Appendix B.2 in the original paper) I need to get more comfortable with…",[77,82],{"value":78,"source":79,"evidence":80},"CUDA","maker",[81],{"id":50},{"value":83,"source":84,"evidence":85},"PyTorch","commenter",[86],{"id":87,"by":88,"text":89},"39729922","einpoklum","My GPU work is not in ML (deep or otherwise); but ... 1. \"100 lines of CUDA\" + PyTorch; maybe this is useful and maybe it isn't, but counting lines of code on top of a huge…","open_source","unknown",{"praise":93,"objections":103,"requests":112,"questions":127,"alternatives":140},[94],{"value":95,"source":84,"evidence":96},"Neat, clear, minimal implementation",[97,98,102],{"id":56,"by":57,"text":58},{"id":99,"by":100,"text":101},"39727161","treesciencebot","Pretty neat implementation. In general, for these sort of exercises (and even if the intention is to go to prod with custom kernels) I lean towards Triton to write the kernels…",{"id":63,"by":64,"text":65},[104,108],{"value":105,"source":84,"evidence":106},"Line count is misleading given reliance on PyTorch codebase",[107],{"id":87,"by":88,"text":89},{"value":109,"source":84,"evidence":110},"Synchronous default-stream kernel launches are not ideal GPU use",[111],{"id":87,"by":88,"text":89},[113],{"value":114,"source":84,"evidence":115},"Explanations of ML concepts for non-ML readers",[116,120,123],{"id":117,"by":118,"text":119},"39727776","robrenaud","Regarding your comment about how fast the research and industry is moving, would HN readers be interested in relevant one or two paragraph summaries that are basically \"explain…",{"id":121,"by":53,"text":122},"39728244","That sounds at least somewhat helpful. Honestly, a gradient for some of this stuff would be nice. Explain to me like I'm: \"five\", \"a high schooler\", \"a college grad (not…",{"id":124,"by":125,"text":126},"39730930","andoando","I would love an explanation for software enginners \u002F CS majors who aren't familiar with ML. Last I studied ML was 2016 and that was stuff like decision trees, k nearest…",[128,132],{"value":129,"source":84,"evidence":130},"What __syncthreads() does in the kernel",[131],{"id":67,"by":68,"text":69},{"value":133,"source":84,"evidence":134},"Whether Triton is preferable to CUDA for custom kernels",[135,136],{"id":99,"by":100,"text":101},{"id":137,"by":138,"text":139},"39730758","queuebert","As a person who finds CUDA extremely easy to write and integrate, what does Triton have to offer?",[141,149,156],{"value":142,"source":84,"evidence":143},"Triton DSL",[144,145],{"id":99,"by":100,"text":101},{"id":146,"by":147,"text":148},"39727715","p1esk","The DSL: https:\u002F\u002Fopenai.com\u002Fresearch\u002Ftriton",{"value":150,"source":84,"evidence":151},"CUTLASS",[152],{"id":153,"by":154,"text":155},"39727718","whimsicalism","yeah even the official flashattention is moving many implementations from cutlass to triton except for the main mha backward\u002Fforward pass",{"value":157,"source":84,"evidence":158},"Halide",[159],{"id":160,"by":161,"text":162},"39732772","dcanelhas","If CPU\u002FGPU execution speed is the goal while simultaneously code golfing the source size, https:\u002F\u002Fhalide-lang.org\u002F might have come in handy.",{"comments":37,"answered":37,"key":164},[165],{"id":74,"text":75},{"commenters":167,"max_depth":42,"maker_replies":37,"first_comment_mins":168,"lifespan_hours":169},27,47,21,[171,173,177,179,180,182,183,187,190,193,196,198,200,203,204,206,207,208,211,214,216,217,220,221,224,225,228,231,234,236,239,241,244,245,247,249,252,255,258],[99,33,41,172,100,33],"project\u002Fneutral\u002Fexecution",[174,33,33,175,176,33],"39727626","commenter\u002Fneutral\u002Fnone","ixaxaar",[178,33,33,175,154,33],"39727713",[146,33,33,175,147,33],[181,33,33,175,100,33],"39728148",[153,33,33,175,154,33],[184,33,37,185,186,33],"39730033","project\u002Fcritical\u002Fdocs","jart",[188,33,33,175,189,33],"39733002","synquid",[191,33,33,175,192,33],"39733986","imtringued",[194,33,33,175,195,33],"39728029","fpgamlirfanboy",[197,33,33,175,100,33],"39728155",[199,33,33,175,195,33],"39728281",[201,33,33,202,195,33],"39728646","other\u002Fneutral\u002Fnone",[137,33,33,175,138,33],[205,33,33,175,154,33],"39731325",[52,37,33,202,53,33],[117,37,33,202,118,33],[209,37,33,202,210,33],"39727845","jprete",[212,37,33,202,213,33],"39730195","jhanoncomm",[215,37,33,202,154,33],"39728181",[121,37,33,202,53,33],[218,37,33,202,219,33],"39728526","imjonse",[124,37,33,202,125,33],[222,37,33,202,223,33],"39728159","godelski",[67,31,41,172,68,33],[226,31,33,175,227,33],"39727804","cavisne",[229,31,33,175,230,33],"39729333","xrd",[232,31,33,175,233,33],"39729842","winwang",[235,31,33,175,227,33],"39730231",[237,31,33,238,230,33],"39730435","commenter\u002Fpositive\u002Fnone",[87,41,37,240,88,33],"project\u002Fcritical\u002Fexecution",[242,41,33,175,243,33],"39731163","chillee",[160,32,41,172,161,33],[56,42,42,246,57,33],"maker\u002Fpositive\u002Fexecution",[74,42,33,248,11,37],"maker\u002Fneutral\u002Fexecution",[250,42,41,251,57,33],"39734187","maker\u002Fneutral\u002Fnone",[63,253,42,254,64,33],6,"project\u002Fpositive\u002Fidea",[256,253,33,175,257,33],"39728245","lagrange77",[259,253,33,175,64,33],"39729901",[],[262,266,271,276,281,285],{"id":263,"name":264,"points":265,"category":20},"39706004","Skyvern",422,{"id":267,"name":268,"points":269,"category":270},"39759325","jnv",373,"dev_tools",{"id":272,"name":273,"points":274,"category":275},"39756325","Godspeed",328,"productivity",{"id":277,"name":278,"points":279,"category":280},"39714053","algebraic-nnhw",310,"hardware",{"id":282,"name":283,"points":284,"category":270},"39770908","GritQL",280,{"id":286,"name":287,"points":288,"category":289},"39755858","Causal 2.0",261,"web_saas",false,"v2",{"id":293,"name":5,"line":6,"ai":290,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":294,"top":18,"kw":295,"maker":11,"domain":10,"tier":19,"attempts":33,"strip":296,"url":9,"mix":297,"praise":95,"objection":105,"sub":21},39726781,0.02,[24,25,26,27],"300000100000000000000000300000103503500",[31,32,33,31],1793991086828]