[Laurianti]

BertTokenizer now follows the basic tokenization of BERT and Hugging Face in two places:

• BertNormalizer turns \t, \n and \r into spaces instead of removing them, so note\nbook is note book, not notebook. It also removes format and private-use characters, as the text cleaning of BERT does, and keeps the characters that are not assigned in Unicode, which become [UNK].
• BertTokenizer uses a new internal pre-tokenizer: every punctuation character is a token, with the definition of BERT (Unicode punctuation and every ASCII character that is not a letter, a digit or whitespace), and every other run of non-whitespace characters is a word. BertNormalizer sets apart the punctuation outside the Basic Multilingual Plane, which the pre-tokenizer can't match on UTF-16 code units. So $ + < = > ^ ` | ~ are tokens, and °, © and emoji reach WordPiece, which finds them in the vocabulary or gives [UNK]. PreTokenizer.CreateWordOrPunctuation is unchanged.

The cases of both issues, and a few more, are in BertTokenizerTests, with the tokens Hugging Face tokenizers gives on the same vocabulary; they fail without the change. Compared with Hugging Face tokenizers 0.23.3, texts encoded to the same ids:

Corpus	Texts	all-MiniLM-L6-v2	bert-base-uncased	bert-base-cased
WikiText-103, test split, non-empty lines	2,891	2,054 → 2,891	2,054 → 2,891	2,054 → 2,891
XNLI, test split, premises in 15 languages	75,150	69,627 → 75,150	69,627 → 75,150	69,630 → 75,150
Source code in C#, Java, Python, C++ and TypeScript, blocks of 20 lines	5,962	282 → 5,962	282 → 5,962	282 → 5,962
Every code point from U+0000 to U+3FFFF and U+E0000 to U+E01EF, alone	260,592	57,122 → 258,679	57,122 → 258,679	58,029 → 260,507
Every punctuation and symbol code point between two words, as in one§line	8,617	584 → 8,489	584 → 8,489	582 → 8,485

Options: LowerCaseBeforeTokenization and RemoveNonSpacingMarks on for the uncased models, off for the cased one.

No text that had the same ids as Hugging Face before has different ids now, except U+1734 alone with the uncased models: .NET classifies it as a spacing mark and the Unicode tables of tokenizers as a non-spacing one, which accent stripping removes. The code points that still differ are mostly Hangul syllables and combining marks with RemoveNonSpacingMarks, and characters assigned in recent Unicode versions, as the punctuation added after Unicode 8.0, which tokenizers does not treat as punctuation; they are outside these two issues.

Fixes #7724
Fixes #7725

[Copilot]

🟡 Changes recommended

Supplementary-plane punctuation can merge adjacent known words into a single unknown token.

1 open issue

• Normalizer fails to split supplementary-plane punctuation · New

What changed in this PR

Aligns BERT basic tokenization with whitespace and symbol handling described in #7724 and #7725, without changing the public word-or-punctuation pre-tokenizer.

Changes:

• Preserves word boundaries around tabs and line breaks.
• Adds BERT-specific punctuation splitting and symbol retention.
• Adds token and offset regression tests.

File	Description
test/​Microsoft.ML.Tokenizers.Tests/​BertTokenizerTests.cs	Adds whitespace, symbol, and offset tests.
src/​Microsoft.ML.Tokenizers/​PreTokenizer/​PreTokenizer.cs	Adds an internal BERT pre-tokenizer.
src/​Microsoft.ML.Tokenizers/​Normalizer/​BertNormalizer.cs	Updates whitespace and control-character cleaning.
src/​Microsoft.ML.Tokenizers/​Model/​BertTokenizer.cs	Selects the new BERT pre-tokenizer by default.

_{Review effort: Balanced}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.79%. Comparing base (29f19e5) to head (da32d66).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7820      +/-   ##
==========================================
+ Coverage   70.77%   70.79%   +0.02%     
==========================================
  Files        1420     1421       +1     
  Lines      279877   280027     +150     
  Branches    27604    27614      +10     
==========================================
+ Hits       198077   198259     +182     
+ Misses      73620    73612       -8     
+ Partials     8180     8156      -24     

Flag	Coverage Δ
Debug	70.79% <100.00%> (+0.02%)	⬆️
production	64.23% <100.00%> (+0.01%)	⬆️
test	90.77% <100.00%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/Microsoft.ML.Tokenizers/Model/BertTokenizer.cs	64.46% <100.00%> (+1.26%)	⬆️
...crosoft.ML.Tokenizers/Normalizer/BertNormalizer.cs	79.16% <100.00%> (+16.30%)	⬆️
...crosoft.ML.Tokenizers/PreTokenizer/PreTokenizer.cs	90.32% <100.00%> (-4.33%)	⬇️
...icrosoft.ML.Tokenizers.Tests/BertTokenizerTests.cs	100.00% <100.00%> (ø)

... and 14 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.