Repository navigation
Unusually slow training speed #17
Description
I am training a tokenizer with a vocab size of 50,368 and have added some special tokens (mask, pad, unk, s, /s)
However, the count pairs step encounters an issue, taking 20 minutes to reach 99% progress, and from then on the speed becomes slower, with an estimated 2-3 hours to complete.
Is this a problem with the Hugging Face tokenizer library, or is it an issue with SuperBPE?
Activity
Iambestfeed commentedon Oct 5, 2025
PythonNut commentedon Oct 14, 2025
As far as we can tell, this is normal behavior. Training SuperBPE tokenizers is a lot slower than normal BPE tokenizers because the lack of pretokenization prevents the library from compressing the training text into a "bag of words" (with frequencies). If the progress is very slow (multiple days) you may be running out of memory.