[{"data":1,"prerenderedAt":421},["ShallowReactive",2],{"project-44422480":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":35,"aspects":36,"strip":44,"post_text":45,"what":46,"said":73,"maker_in_thread":162,"shape":175,"grid":180,"attempts":383,"same_week":384,"maker_shows":39,"maker_hidden":414,"reader_version":415,"card":416},"44422480","TokenDagger","A tokenizer faster than OpenAI's Tiktoken","title","Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken","https:\u002F\u002Fgithub.com\u002FM4THYOU\u002FTokenDagger","github.com","matthewolfe",1751286838,2025,281,73,198,3837,6,"legend","ai","tokenizer library","library",[24,25,26,27,28],"tokenizer","bpe","tiktoken","c++","llm inference","curious",0.1,{"positive":18,"neutral":32,"mixed":33,"critical":34},16,0,5,27,{"performance":37,"idea":40,"docs":41,"execution":42},{"positive":38,"critical":39},4,1,{"positive":39,"critical":39},{"positive":33,"critical":33},{"positive":39,"critical":43},3,[34,33,33,33,33,33,33,33,33,33,33,33,33,33,43,34,43,33,43,33,39,34,33,34,43,39,33,33,33,43,43,39,43,34,43,34,39,43,43,39],"TokenDagger is a drop-in replacement for OpenAI’s Tiktoken (the tokenizer behind Llama 3, Mistral, GPT-3.*, etc.). It’s written in C++ 17 with thin Python bindings, keeps the exact same BPE vocab\u002Fspecial-token rules, and focuses on raw speed.\n\nI’m teaching myself LLM internals by re-implementing the stack from first principles. Profiling TikToken’s Python\u002FRust implementation showed a lot of time was spent doing regex matching. Most of my perf gains come from a) using a faster jit-compiled regex engine; and b) simplifying the algorithm to forego regex matching special tokens at all.\n\nBenchmarki",{"problem":47,"audience":53,"pricing":53,"motivation":54,"time_to_build":53,"stack":58,"openness":71,"stage":72,"team_size":72},{"value":48,"source":49,"evidence":50},"Tiktoken tokenization is slow, largely due to regex matching.","maker",[51],{"id":52},"post",null,{"value":55,"source":49,"evidence":56},"Teaching themselves LLM internals by re-implementing the stack from first principles.",[57],{"id":52},[59,63,67],{"value":60,"source":49,"evidence":61},"C++17",[62],{"id":52},{"value":64,"source":49,"evidence":65},"Python bindings",[66],{"id":52},{"value":68,"source":49,"evidence":69},"JIT-compiled regex engine",[70],{"id":52},"open_source","unknown",{"praise":74,"objections":87,"requests":109,"questions":124,"alternatives":136},[75],{"value":76,"source":77,"evidence":78},"Drop-in replacement with substantial speedup is appealing.","commenter",[79,83],{"id":80,"by":81,"text":82},"44422818","chrismustcode","There’s something beautiful about creating a drop in replacement for something that improves performance substantially. ScyllaDB comes to mind",{"id":84,"by":85,"text":86},"44427659","isjustintime","Very cool. We use Tiktoken and I'd love to see the performance impact. Pretty great decision to make it drop-in compatible.",[88,102],{"value":89,"source":77,"evidence":90},"Tokenization is rarely the bottleneck in LLM inference or training.",[91,95,99],{"id":92,"by":93,"text":94},"44426006","serjester","Tokenizing text is ridiculously small part of the overall computation that goes into serving a request. With that said if you’re doing this on petabytes of data, never hurts to…",{"id":96,"by":97,"text":98},"44426054","refibrillator","Tokenization is typically done on CPU and is rarely (if ever) a bottleneck for training or inference. GPU kernels typically dominate in terms of wall clock time, the only…",{"id":100,"by":11,"text":101},"44428487","To echo the other replies, the tokenizer is definitely not the bottleneck. It just happens to be the first step in inference, so it's what I did first.",{"value":103,"source":77,"evidence":104},"Dropping special-token regex may lose correctness checks for unspecified special tokens.",[105],{"id":106,"by":107,"text":108},"44555820","anonymoushn","Isn't this incorrect? If the user doesn't specify what to do with almost all of the special tokens, you still must detect them so you can raise an error.",[110,117],{"value":111,"source":77,"evidence":112},"Benchmark against Hugging Face tokenizers.",[113],{"id":114,"by":115,"text":116},"44425649","Tiberium","Can you also compare the performance with https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftokenizers\u002F? Would be helpful, since the benchmark in the tiktoken readme seems to be very outdated.",{"value":118,"source":77,"evidence":119},"Push speedups upstream into tiktoken.",[120],{"id":121,"by":122,"text":123},"44423953","pamelafox","Just curious whether it's possible to push any of your performance improvements to tiktoken itself?",[125],{"value":126,"source":77,"evidence":127},"Does skipping special-token regex change output?",[128,132],{"id":129,"by":130,"text":131},"44422939","konsalexee","> simplifying the algorithm to forego regex matching special tokens at all Does that mean there could be cases with less quality in terms of tokenization?",{"id":133,"by":134,"text":135},"44425100","matrix2596","is is possible for your tokenizer to give different tokenization ever then openai tokenizer? i am asking because there are multiple ways to tokenize the same string?? sry if i am…",[137,144,148,155],{"value":138,"source":77,"evidence":139},"Tiktoken",[140],{"id":141,"by":142,"text":143},"44427963","notatallshaw","It looks like TikToken is written in Rust (https:\u002F\u002Fgithub.com\u002Fopenai\u002Ftiktoken\u002Ftree\u002Fmain\u002Fsrc), are the gains here actually from porting to C++?",{"value":145,"source":77,"evidence":146},"Hugging Face tokenizers",[147],{"id":114,"by":115,"text":116},{"value":149,"source":77,"evidence":150},"bpe crate",[151],{"id":152,"by":153,"text":154},"44423369","p0","How does this compare to the BPE crate [1]? Its main selling point is support for incrementally re-tokenising text, but it's also faster than tiktoken. [1]…",{"value":156,"source":77,"evidence":157},"tokie",[158],{"id":159,"by":160,"text":161},"44424134","kevmo314","Nice work! I tried something similar a while back ago: https:\u002F\u002Fgithub.com\u002Fkevmo314\u002Ftokie The takeaway I also found was that the running cost was really dominated by…",{"comments":163,"answered":164,"key":165},15,10,[166,169,172],{"id":167,"text":168},"44424382","Agreed. A former mentor of mine told me a nice way of viewing software development: 1. Make it work. 2. Make it fast. 3. Make it pretty. Transformers & LLMs have been developed…",{"id":170,"text":171},"44428459","Fair chance I'm remembering it wrong :D",{"id":173,"text":174},"44422833","Agreed. I figured nobody would use it otherwise.",{"commenters":176,"max_depth":177,"maker_replies":163,"first_comment_mins":178,"lifespan_hours":179},55,7,32,326,[181,185,187,191,194,197,200,203,206,208,211,214,217,220,222,224,227,230,233,236,239,242,245,247,250,253,255,258,260,262,266,268,271,276,279,281,285,286,290,291,294,296,297,299,302,303,306,307,309,312,315,318,320,324,328,329,331,332,335,337,341,345,347,348,350,354,356,361,366,370,374,376,380],[182,33,34,183,184,33],"44422888","project\u002Fpositive\u002Fperformance","npalli",[167,33,33,186,11,39],"other\u002Fneutral\u002Fnone",[188,33,33,189,190,33],"44424439","commenter\u002Fneutral\u002Fnone","diggan",[192,33,33,189,193,33],"44424671","abybaddi009",[195,33,33,189,196,33],"44424691","stavros",[198,33,33,189,199,33],"44424765","darknoon",[201,33,33,189,202,33],"44424906","DSingularity",[204,33,33,189,205,33],"44425794","gabrielhidasy",[207,33,33,189,196,33],"44425846",[209,33,33,189,210,33],"44424757","robertfw",[212,33,33,189,213,33],"44424931","gopalv",[215,33,33,189,216,33],"44425051","kevindamm",[218,33,33,189,219,33],"44430314","acosmism",[221,33,33,189,205,33],"44425719",[170,33,33,223,11,39],"maker\u002Fneutral\u002Fnone",[225,33,33,189,226,33],"44428747","mindcrime",[228,33,33,186,229,33],"44425934","binarymax",[231,33,33,186,232,33],"44426074","jotux",[234,33,33,186,235,33],"44424572","saretup",[237,33,33,189,238,33],"44424608","tbalsam",[240,33,33,189,241,33],"44425185","bigyabai",[243,33,33,189,244,33],"44429751","mdaniel",[246,33,33,189,241,33],"44429894",[248,33,33,189,249,33],"44425265","janalsncm",[251,33,33,186,252,33],"44424990","ipsum2",[141,33,43,254,142,33],"project\u002Fneutral\u002Fperformance",[256,33,43,254,257,33],"44430216","fhub",[80,39,34,259,81,33],"project\u002Fpositive\u002Fidea",[173,39,33,261,11,39],"maker\u002Fneutral\u002Fidea",[263,39,43,264,265,33],"44422860","maker\u002Fneutral\u002Fdocs","parhamn",[267,39,33,264,11,39],"44422914",[269,39,33,186,270,33],"44422898","pvg",[272,273,43,274,275,33],"44423489",2,"project\u002Fneutral\u002Fexecution","pama",[277,273,33,278,11,39],"44425102","maker\u002Fneutral\u002Fexecution",[280,273,33,278,11,39],"44425489",[282,43,33,283,284,33],"44425827","other\u002Fneutral\u002Fperformance","superlopuh",[92,43,43,254,93,33],[287,43,39,288,289,33],"44426605","project\u002Fcritical\u002Fexecution","odyssey7",[96,43,43,254,97,33],[292,43,34,183,293,33],"44427256","benreesman",[100,43,33,295,11,39],"maker\u002Fneutral\u002Fperformance",[159,38,34,183,160,33],[298,38,33,223,11,39],"44424322",[300,38,33,186,301,33],"44439086","22c",[114,34,43,254,115,33],[304,34,39,305,229,33],"44425951","project\u002Fcritical\u002Fperformance",[152,18,43,254,153,33],[308,18,33,261,11,39],"44424394",[310,177,33,186,311,33],"44423436","frabcus",[313,177,33,186,314,33],"44424008","weberer",[316,177,33,186,317,33],"44424141","Deathmax",[319,177,33,261,11,39],"44424212",[321,322,43,274,323,33],"44423338",8,"fkyoureadthedoc",[325,326,33,186,327,33],"44439670",9,"justinhj",[129,164,43,274,130,33],[330,164,33,274,11,39],"44423046",[106,164,39,288,107,33],[121,333,43,334,122,33],11,"project\u002Fneutral\u002Fnone",[336,333,33,278,11,39],"44424042",[338,339,43,334,340,33],"44424452",12,"polynomial",[342,343,33,186,344,33],"44423337",13,"manishsharan",[84,346,34,183,85,33],14,[133,163,43,274,134,33],[349,163,33,274,11,39],"44425112",[351,32,34,352,353,33],"44428396","project\u002Fpositive\u002Fexecution","semiinfinitely",[355,32,33,223,11,39],"44428469",[357,358,39,359,360,33],"44430975",17,"project\u002Fcritical\u002Fidea","singularity2001",[362,363,43,364,365,33],"44423793",18,"project\u002Fneutral\u002Fidea","EGreg",[367,368,43,364,369,33],"44427754",19,"sheerun",[371,372,43,223,373,33],"44424606",20,"silentsea90",[375,372,33,264,11,39],"44424942",[377,378,33,186,379,33],"44423620",21,"janwilmake",[381,378,39,288,382,33],"44425810","_flux",[],[385,390,395,399,404,409],{"id":386,"name":387,"points":388,"category":389},"44396518","I'm an airline pilot",1539,"creative",{"id":391,"name":392,"points":393,"category":394},"44433409","Spegel",426,"dev_tools",{"id":396,"name":397,"points":398,"category":394},"44445238","glass3d",420,{"id":400,"name":401,"points":402,"category":403},"44412207","Octelium",358,"security_privacy",{"id":405,"name":406,"points":407,"category":408},"44421776","New Ensō",251,"productivity",{"id":410,"name":411,"points":412,"category":413},"44428438","OpenFile",244,"web_saas",false,"v2",{"id":417,"name":5,"line":6,"ai":414,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":30,"top":18,"kw":418,"maker":11,"domain":10,"tier":19,"attempts":33,"strip":419,"url":9,"mix":420,"praise":76,"objection":89,"sub":21},44422480,[24,25,26,27,28],"5000000000000035303015053100033135351331",[18,32,33,34],1794036677578]