Skip to content

feat: add Estonian (et) PII language pack with isikukood validator - #1592

Merged
maziyarpanahi merged 5 commits into
maziyarpanahi:masterfrom
PouyanJay:feat/estonian-isikukood-pii
Jul 15, 2026
Merged

maziyarpanahi merged 5 commits into
maziyarpanahi:masterfrom
PouyanJay:feat/estonian-isikukood-pii

Conversation

@PouyanJay

Copy link
Copy Markdown
Contributor

Summary

Implements #891: adds Estonian as a national-ID-only language pack, mirroring the structure of the Latvian (lv) pack.

  • validate_estonian_isikukood: century/sex first digit (1–8), embedded birth-date plausibility, two-pass modulo-11 checksum with weight fallback (including the fallback-to-zero case)
  • _ESTONIAN_PII_PATTERNS: isikukood national_id plus Estonian date, phone (+372), street-address, and context-gated postcode cues
  • et registered in LANGUAGE_PII_PATTERNS / NATIONAL_ID_ONLY_LANGUAGES, et -> et_EE locale, fake data, and locale format hints
  • generate_estonian_isikukood + EstonianIsikukoodProvider surrogate provider wired through the national-ID registry
  • synthetic Estonian golden fixture + unit tests (validator accept/reject incl. weight-fallback vectors, generator and Anonymizer surrogate round-trips, pattern spans, fixture offsets, offline no-leakage de-identification)

Two deviations from the issue's file list

The issue names five files; this diff touches nine. Both deviations were necessary for the feature to actually work:

  1. Golden fixture path. The issue says openmed/eval/golden/et.jsonl, but the fixture loader only globs fixtures/**/*.jsonl — a file at the issue's path would be silently ignored and never exercised. The fixture therefore lives at openmed/eval/golden/fixtures/i18n/et.jsonl, alongside the ten existing language fixtures.

  2. Four wiring files (~18 additive lines total). Registering et in NATIONAL_ID_ONLY_LANGUAGES auto-enrolls it in the locale-coherence and locale-format regression suites, which assert wiring outside the issue's five files. Without these, make test fails and surrogates can't round-trip:

    • openmed/core/anonymizer/registry.py — _LOCALE_ID_METHODS["et_EE"] = "isikukood" (surrogate dispatch)
    • openmed/core/anonymizer/providers/registry_ids.py — provider registration so Faker instances gain the isikukood method
    • openmed/core/locale_formats.py — et date-order/number-separator hints (required by test_format_hint_covers_every_wired_language)
    • openmed/core/anonymizer/providers/__init__.py — exports, matching every sibling pack

Data provenance

Every isikukood in code, tests, and the fixture is synthetic, generated algorithmically from the checksum rules (including checksum-valid-but-date-impossible rejection vectors). No real Estonian personal codes anywhere.

Verification

  • make test: 4640 passed, 40 skipped, 0 failed
  • pre-commit run --all-files: all hooks covering these files pass
  • The locale-coherence gate now auto-covers et, including a 40-seed surrogate→validator round-trip

Out of scope, as per the issue: no Estonian model training, no registry data bundled.

Closes #891

Add Estonian as a national-ID-only language:

- validate_estonian_isikukood: century/sex first digit, embedded
  birth-date plausibility, two-pass modulo-11 checksum with weight
  fallback (and fallback-to-zero)
- _ESTONIAN_PII_PATTERNS: isikukood national_id plus Estonian date,
  phone (+372), street address, and context-gated postcode cues
- register et in LANGUAGE_PII_PATTERNS, NATIONAL_ID_ONLY_LANGUAGES,
  et -> et_EE locale, fake data, and locale format hints
- generate_estonian_isikukood + EstonianIsikukoodProvider Faker
  surrogate provider wired through the national-ID registry
- synthetic Estonian golden fixture (fixtures/i18n/et.jsonl)
- unit tests: validator accept/reject incl. weight-fallback vectors,
  200-sample generator round-trip, Anonymizer surrogate round-trip,
  pattern spans, fixture offsets, offline no-leakage de-identification

All values are synthetic, generated algorithmically from the checksum
rules. Closes maziyarpanahi#891.
@maziyarpanahi maziyarpanahi added feature New capability good first issue Good for newcomers help wanted Extra attention is needed P2 Medium roadmap-v2 OpenMed V2 roadmap backlog labels Jul 15, 2026
@maziyarpanahi

Copy link
Copy Markdown
Owner

Pouyan, thank you for this comprehensive Estonian language pack. The two-pass checksum and date validation, native Faker wiring, synthetic offset fixture, and leakage coverage are especially strong.

I verified the implementation against the official Estonian personal-code structure and added a small maintainer follow-up to keep the validator fail-closed: current century and sex markers 1–6 only, canonical ASCII digits rather than silent normalization, non-string rejection, and native street diacritics. The branch is current with master; the focused suite passes (479 tests), the full suite passes (4,751 tests, 52 skipped), formatting, lint, and type gates pass, and all hosted checks are green. This is ready to merge—excellent work.

@maziyarpanahi
maziyarpanahi merged commit 5e66ceb into maziyarpanahi:master Jul 15, 2026
19 checks passed
@PouyanJay
PouyanJay deleted the feat/estonian-isikukood-pii branch August 20, 2026 05:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New capability good first issue Good for newcomers help wanted Extra attention is needed P2 Medium roadmap-v2 OpenMed V2 roadmap backlog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add an Estonian (et) PII language pack with isikukood personal-code validator

2 participants