Add a CLDR person-name corpus and an agreement ratchet against its stated fields

[derek73]

The gate passes at every ledger baseline (1.4.0 through 2.4.0: exit 0, no unexplained diffs, no contests). This branch includes master with #655, the fix for #653, which a CLDR rendering had exposed at 1.4.0.

Gives nameparser its first check against ground truth from outside the project: Unicode CLDR's person-name test data, which states every name's fields separately (title, given, given2, surname or prefix + core, surname2, generation, credentials), in native scripts.

What's added

• A radar corpus. tools/differential/build_cldr_corpus.py fetches CLDR release-48's personNameTest files and writes:

  • corpus_cldr.jsonl: 1,870 lines from 2,640 renderings in 110 locale files; each name's long, formal rendering in givenFirst, surnameFirst and sorting order, with surname-first forms tagged shape: 4;
  • cldr_fields.jsonl: each rendering's CLDR fields;
  • cldr_LICENSE: the Unicode-3.0 notice.

• An agreement ratchet. tools/differential/cldr_agreement.py parses every rendering and compares, role by role, with CLDR. The roles are title, given names, family and suffix, plus the particle/base split where CLDR states it. Disagreements are recorded in cldr_agreement.json. tests/v2/test_cldr_agreement.py fails on:

  • a regression;
  • progress not yet recorded;
  • a changed wrong answer;
  • a recorded rendering that no longer exists.

  Re-record with uv run python tools/differential/cldr_agreement.py --record.

• Agreement today (2026-10-12): 9,025 of 11,880 role verdicts. The record holds 956 disagreeing renderings, which doubles as a measured worklist of names the parser and CLDR read differently.

role	agree
title	2389 / 2640
given_names	1724 / 2640
family	1798 / 2640
suffix	2367 / 2640
family_particles	414 / 660
family_base	346 / 660

Decisions along the way

• Ledger guards don't measure CLDR names. Counted, they moved fourteen 1.4.0-ledger rules' recorded reach at once (the comma rules 461 → 771 names). _UNMEASURED_CORPORA keeps them out of _CORPUS_NAMES; the harness still classifies their diffs on the radar tier.
• Property tests don't use CLDR names yet. Four Slavic renderings exposed A decomposed (NFD) name loses an abbreviated title: Dr. inž. Janez Novak reads inž. as the given name #652 (decomposed inž. isn't a period abbreviation, so the comma reverses the name). differential_corpus() leaves the file out until that's fixed. Including it costs test_properties.py 34.5 s → 39.6 s (py3.11).
• fix(#379) ledger regex: its member di case-folded onto the Turkish dotless ı (Yıldırım). Its three ledger copies now use (?ai); on every corpus name, nothing else changes whether it matches.
• The trailing-Ph. D. [[never]] entry was narrowed to the split pair its why is about, as its own comment asked. It had refused 17 CLDR names: 16 whose only 1.4 diff is the suffix separator (M.D., Ph.D. → M.D. Ph.D.), plus an unsplit Ph.D. title move. Its recorded effect moves from 7 to 3 captures.
• CLDR names aren't adjudicated in gate contests. At 1.4.0 they raised three UNPINNED CONTESTs (two pairs of initials-per-word rules with equal fields, and the CJK order rule against the delimited-nickname rule). For two of them, an honest pin would have had to name a rule that doesn't describe what moved. compare._UNMEASURED_CORPORA now names the files whose names are classified (by file order) but not adjudicated; a name any other corpus holds still is. A test holds it equal to the guards' copy.
• Release notes: CLDR's fictional renderings now enter the classified summary wherever a rule classifies them. They add 60 / 203 / 200 / 186 / 12 / 1 radar-unclassified lines at baselines 1.4.0 through 2.4.0 (after A nickname loses its comma: Jane (Janey, Jo) Smith gives nickname Janey Jo (1.4.0: Janey, Jo) #653's fix). The README and AGENTS.md say to count them out of release figures.

Review

• Executed from a written spec and plan; every deviation is recorded in the commit messages.

• A whole-branch review found false disagreements: only the ASCII comma was stripped, and unspaced fields were joined in field order. Fixing them moved 57 entries to agreeing and unmasked one false agreement. A review of that fix commit then corrected one explanation (the 17).

• Negative controls:

  • comparison rules: 4 mutations;
  • ratchet: 3 mutations;
  • the first round of review-driven tests: 2 reverts.

  Each fails at least one test.

• Full suite on the merged branch: 15,108 passed. Ruff and mypy are clean.

Deferred (minor):

• changes() doesn't report a CLDR bump that changes only text/expected; the docstring says so.
• The build script doesn't refuse a name missing its long formal rendering (none are missing today).
• The property-test exclusion (conftest.py, until A decomposed (NFD) name loses an abbreviated title: Dr. inž. Janez Novak reads inž. as the given name #652) names the file directly rather than through _UNMEASURED_CORPORA; it's a different reason with a different lifetime.
• shape_for checks only the ASCII comma (harmless).

Related: #652; #653 (fixed by #655).

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.02%. Comparing base (e5bde5e) to head (2a8a785).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #654   +/-   ##
=======================================
  Coverage   99.02%   99.02%           
=======================================
  Files          46       46           
  Lines        4520     4520           
=======================================
  Hits         4476     4476           
  Misses         44       44           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.