DMIS Lab releases UltraBERT, open encoders that lead both classification and retrieval
Korea University's DMIS Lab published UltraBERT-150M and UltraBERT-380M plus UltraBERT-Embed-150M on Hugging Face under Apache 2.0. The encoders mix 5% and 50% masking with decoupled heads and claim top scores on 9 of 10 benchmarks at both sizes.
Photo: Mannaa mohamed / Wikimedia Commons, CC BY-SA 4.0. Hugging Face logo via Wikimedia Commons (public domain).
DMIS Lab at Korea University released UltraBERT on the Hugging Face blog on October 14: English encoders at 150M and 380M parameters with a native 32K-token context, plus UltraBERT-Embed-150M, a dense embedding model. Everything is Apache 2.0 and hosted in the UltraBERT collection on Hugging Face.
The lab says prior encoders force a trade-off: low masking favors classification, high masking favors retrieval. UltraBERT mixes 5% and 50% masked sequences in each batch with a separate language-modeling head for each ratio, and distills from a 1.3B teacher only on the low-masking head. Training runs on about 2T tokens, with the last 100B at 32K context.
In the lab's comparisons, UltraBERT took the top score on 9 of 10 benchmarks at both sizes, including 91.0 on GLUE and 47.7 on BEIR for the 380M model, with 99% passkey accuracy out to 32K tokens. UltraBERT-Embed-150M scores 67.24 on MTEB(eng, v2), which the authors call the best dense embedding model under 200M parameters, trained without distillation from a larger embedder. These are the authors' reported results.
The models need transformers 5 or newer and trust_remote_code. Checkpoints include the encoder and the 5% distilled MLM head, with task classes for sequence classification, token classification and question answering. An arXiv paper, "UltraBERT: Beyond the Limit of BERT Pre-Training" (2610.15028), accompanies the release.