[Laurianti]

When several pairs of symbols can merge into pieces with the same score, SentencePieceBpeModel merged the rightmost pair first, while SentencePiece merges the leftmost one, so some runs of repeated characters were encoded differently:

Text	Before	Now, as SentencePiece
____	1, 4770, 1649 (▁__, __)	1, 903, 22359 (▁_, ___)
......	1, 6317, 3045 (▁.., ....)	1, 13035, 636 (▁...., ..)
~~~~	1, 3695, 30022, 7377 (▁~, ~, ~~)	1, 3695, 7377, 30022 (▁~, ~~, ~)

SymbolPair.CompareTo now breaks ties with Left.CompareTo(other.Left), as the SymbolPair of CodeGenTokenizer does. The three cases are added to LlamaTestData, with the ids, tokens and offsets SentencePiece 0.2.2 gives; they fail without the change. Compared with SentencePiece 0.2.2 (Hugging Face tokenizers gives the same ids on all three sets):

Corpus	Texts	Encoded differently before	After	Fixed
WikiText-103, test split	2,891 paragraphs	1 (0.03%)	0	1 of 1
XNLI, test split, 15 languages	75,150 sentences	39 (0.05%)	0	39 of 39
Source code in C#, Java, Python, C++ and TypeScript from public repositories	7,560 blocks of 20 lines	1,567 (20.7%)	0	1,567 of 1,567
Total	85,601	1,607	0	1,607 of 1,607

Fixes #7751

[Copilot]

Copilot review overview

🟢 Approval recommended

The focused fix matches SentencePiece behavior and has comprehensive regression coverage.

Review effort: Balanced
Findings: None

What changed in this PR

Aligns SentencePiece BPE tie-breaking with the reference implementation.

Changes:

• Merges leftmost equal-score pairs first.
• Adds regression cases for repeated symbols.

File	Description
SentencePieceBpeModel.cs	Corrects equal-score pair ordering.
LlamaTests.cs	Verifies IDs, tokens, and offsets.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.25%. Comparing base (5a44dfd) to head (cb66549).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #7752   +/-   ##
=======================================
  Coverage   70.25%   70.25%           
=======================================
  Files        1419     1419           
  Lines      272569   272593   +24     
  Branches    27938    27938           
=======================================
+ Hits       191500   191521   +21     
- Misses      73652    73657    +5     
+ Partials     7417     7415    -2     

Flag	Coverage Δ
Debug	70.25% <100.00%> (+<0.01%)	⬆️
production	64.51% <100.00%> (-0.01%)	⬇️
test	89.79% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...osoft.ML.Tokenizers/Model/SentencePieceBpeModel.cs	77.55% <100.00%> (ø)
test/Microsoft.ML.Tokenizers.Tests/LlamaTests.cs	99.86% <100.00%> (+<0.01%)	⬆️

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.