⚡! Tokenization is the first thing you do in CS336 (language models from scratch). If Marcel can work his magic for the rest of the LM pipeline, the world will be a better place.

\n
@marcelroed:\nIntroducing the world's fastest tokenizer implementation, Gigatoken!
\n
Gigatoken is ~500-1000x faster than HuggingFace, and ~100x faster than OpenAI's tiktoken for most tokenizer definitions on most machines.
\n
These baselines are already multithreaded Rust implementations! 🧵 https://t.co/zxzXPFBCHv
\n

$\"\"$

","freeContentEncoded","textContent","published","strategy","unknown","locked","deletedAt","isSyncedFromOwnBlog","original","duplicatesCount","duplicateVersion","originalArticleId","errorsCount","appearance","twitter:repost","color","backgroundColor","shortSummary","summary","bullets","persons","places","words","sentiment","subject","meta","eventId","795e5b534e8c07976d609d8f64ba7442842e63e2f64def87aa4be4307b7efcef","nostrRootEvent","event",{"_19":218,"_222":223,"_224":115,"_68":225,"_235":26,"_188":236,"_237":238},"sig","2210ea2de12ad4f416bd7cb24dbc5945f5a77350ab9cae08cb78c6c83d605edd85c79b09ee24cd0aa05358067765e82d5bb4c02a8f409f378ab5e4dd8e169d6b","kind",[226,227,231,233],[215,176],[228,229,40,230],"a","30023:983dd797fe72ad62fc8e3ece7eed1f126550972885c8f835d9af5528dad655a3:019f88b5-28c7-13a4-7158-0779dcb28e45","mention",[232,40],"client",[234,26,40],"p","pubkey","⚡! Tokenization is the first thing you do in CS336 (language models from scratch). If Marcel can work his magic for the rest of the LM pipeline, the world will be a better place.\n\n\n\nnostr:naddr1qvzqqqr4gupzpxpa67tluu4dvt7gu0kw0mk37yn92ztj3pwglq6ant649rddv4drqqjrqvfevcursc3495ersceh95cnxcf595mnzdfc95crwdeev33kyv3cv56r2gpmnsw ","created_at",1784694444,"authorNPub","markdown","⚡! Tokenization is the first thing you do in CS336 (language models from scratch). If Marcel can work his magic for the rest of the LM pipeline, the world will be a better place.\n\n> @marcelroed:\n> Introducing the world's fastest tokenizer implementation, Gigatoken!\n> \n> Gigatoken is ~500-1000x faster than HuggingFace, and ~100x faster than OpenAI's tiktoken for most tokenizer definitions on most machines.\n> \n> These baselines are already multithreaded Rust implementations! 🧵 https://t.co/zxzXPFBCHv\n\n\n \n![](https://pbs.twimg.com/media/HNxfhRHbMAAOxeD.png)","freeMarkdown","kind1EventId","kind30023EventId","3d2046641cba0009102b13931cb593dde8e2fb921dac73208145ef72a4ee2f7c","kind30023",{"_19":245,"_222":248,"_224":249,"_68":250,"_235":26,"_188":262,"_237":238},"51ccd916f210927a448a4938f75aa5ac408eb9193915e90f8ce21b3f79cf691839be61195286bd242a565923180f0823638acee76f73c4e6cf421f0e10ae01e2",30023,[251,254,255,258,260,261],[252,253],"r","https://bender.layer3.press/articles/019f88b5-28c7-13a4-7158-0779dcb28e45",[31,176],[256,257],"published_at","1784694444",[259,171],"d",[232,40],[234,26,40],"⚡! Tokenization is the first thing you do in CS336 (language models from scratch). If Marcel can work his magic for the rest of the LM pipeline, the world will be a better place.\n\n> @marcelroed:\n> Introducing the world's fastest tokenizer implementation, Gigatoken!\n> \n> Gigatoken is ~500-1000x faster than HuggingFace, and ~100x faster than OpenAI's tiktoken for most tokenizer definitions on most machines.\n> \n> These baselines are already multithreaded Rust implementations! 🧵 https://t.co/zxzXPFBCHv\n\n\n \n![](https://pbs.twimg.com/media/HNxfhRHbMAAOxeD.png)\nhttps://bender.layer3.press/articles/019f88b5-28c7-13a4-7158-0779dcb28e45","publishedToNostr","htmlContentKind30023","pubDate","2026-07-22T04:27:24.000Z","modifiedAt","2026-07-22T07:23:22.208Z","storyId","likes","zaps","reposts","comments","tweets",[],"relatedArticles",[],"actionData","errors"]