Hugging Face makes BERT-style tokenization up to 50× faster by skipping ASCII work
A Hugging Face tokenizers blog post shows a pending change that speeds bert-base-uncased encoding about 50× versus v0.23 on English text, mainly by skipping ASCII normalization with SIMD and caching Unicode page behavior.
Photo: Mbzt / Wikimedia Commons, CC BY-SA 4.0. Hugging Face logo via Wikimedia Commons (public domain).
Hugging Face engineer Simon Brandeis published "Tokenmaxxing BERT: 50x faster tokenization" on the Hugging Face blog on October 15, detailing a pull-request-stage speedup for BERT-style WordPiece tokenizers in the tokenizers library as the project heads toward a v1.0.0 release.
Profiling showed that about 88% of encode time for bert-base-uncased on English text was spent in normalization, not WordPiece itself. The new path skips normalization on runs of ASCII using SIMD checks, caches whether a page of 256 Unicode code points needs normalization work, and avoids allocating when text is already normalized. Brandeis reports end-to-end English throughput rising from about 23 MB/s to about 151 MB/s per core in the measurements shown.
The post says the three most-downloaded Hub models—all-MiniLM-L6-v2, ms-marco-MiniLM-L6-v2 and bge-small-en-v1.5—all use BERT's normalizer, and that across the 10,000 most-downloaded models, WordPiece accounts for about 627 million monthly downloads. Token IDs remain identical to the prior path on every corpus tested, according to the post.
Gains are largest on English and source code; scripts with little ASCII, such as Chinese under BERT's CJK padding rules, see little or no speedup, and tiny inputs can be slightly slower. The work builds on the earlier tokenizers v1 release-candidate post that reported roughly 3–30× faster BPE encoding versus v0.23.