SuperBPE produces garbage characters for non-English languages (Sinhala) #23

When training SuperBPE on Sinhala data, both Stage 1 (BPE) and Stage 2 (SuperBPE) produce garbage characters in the vocabulary and tokenizer output.

Tokens look like:

à¶Ĵ
à·ı
රණ

instead of valid Sinhala characters.

The issue is present in:

• tokenizer.json
• vocab.json
• Tokenization output during inference

Training procedure

I followed the repository instructions exactly:

Stage 1: Learn subwords with whitespace pretokenization (regular BPE)
scripts/train_tokenizer.sh

Stage 2: Resume training without whitespace pretokenization (SuperBPE)
scripts/extend_tokenizer.sh

No modifications were made to the scripts.

Dataset

• Dataset: Helsinki-NLP/opus-100 (en-si)
• Used Sinhala side (si)
• Dataset is UTF-8
• Same issue appears in both Stage 1 and Stage 2 outputs

Observation

This does not appear to be a decoding issue at inference time, because the garbage characters are already present inside:

• tokenizer.json
• vocab.json

This suggests the corruption happens during training.

Example

Original:

ඒ කාරණාවෙන් මාව අර්ථවත් උනා , ඒකෙන් මට ශක්තිය ලැබුනා∙ .

BPE tokenized:

['à¶Ĵ', 'Ġà¶ļ', 'à·ı', 'රණ', 'à·ı', 'à·Ģ', 'à·Ļ', 'න', 'à·Ĭ', 'Ġම', 'à·ı', 'à·Ģ', 'Ġà¶ħර', 'à·Ĭ', 'ථ', 'à·Ģà¶Ń', 'à·Ĭ', 'Ġà¶ĭන', 'à·ı', 'Ġ,', 'Ġà¶Ĵà¶ļ', 'à·Ļ', 'න', 'à·Ĭ', 'Ġමට', 'Ġà·ģ', 'à¶ļ', 'à·Ĭ', 'à¶Ń', 'à·Ĵ', 'ය', 'Ġල', 'à·IJ', 'බ', 'à·Ķ', 'න', 'à·ı', 'âĪĻ', 'Ġ.']

SuperBPE tokenized:

['à¶Ĵ', 'Ġà¶ļ', 'à·ı', 'රණ', 'à·ı', 'à·Ģ', 'à·Ļ', 'න', 'à·Ĭ', 'Ġම', 'à·ı', 'à·Ģ', 'Ġà¶ħර', 'à·Ĭ', 'ථ', 'à·Ģà¶Ń', 'à·Ĭ', 'Ġà¶ĭන', 'à·ı', 'Ġ,', 'Ġà¶Ĵà¶ļ', 'à·Ļ', 'න', 'à·Ĭ', 'Ġමට', 'Ġà·ģ', 'à¶ļ', 'à·Ĭ', 'à¶Ń', 'à·Ĵ', 'ය', 'Ġල', 'à·IJ', 'බ', 'à·Ķ', 'න', 'à·ı', 'âĪĻ', 'Ġ.']

Both stages produce identical corrupted tokens.

Expected behavior

Vocabulary and tokens should contain valid Sinhala Unicode characters.

Environment

• Python 3.11
• Windows 11 / WSL Ubuntu