Repository navigation
feat: add Estonian (et) PII language pack with isikukood validator - #1592
Merged
maziyarpanahi merged 5 commits intoJul 15, 2026
Merged
Conversation
Add Estonian as a national-ID-only language: - validate_estonian_isikukood: century/sex first digit, embedded birth-date plausibility, two-pass modulo-11 checksum with weight fallback (and fallback-to-zero) - _ESTONIAN_PII_PATTERNS: isikukood national_id plus Estonian date, phone (+372), street address, and context-gated postcode cues - register et in LANGUAGE_PII_PATTERNS, NATIONAL_ID_ONLY_LANGUAGES, et -> et_EE locale, fake data, and locale format hints - generate_estonian_isikukood + EstonianIsikukoodProvider Faker surrogate provider wired through the national-ID registry - synthetic Estonian golden fixture (fixtures/i18n/et.jsonl) - unit tests: validator accept/reject incl. weight-fallback vectors, 200-sample generator round-trip, Anonymizer surrogate round-trip, pattern spans, fixture offsets, offline no-leakage de-identification All values are synthetic, generated algorithmically from the checksum rules. Closes maziyarpanahi#891.
maziyarpanahi
self-requested a review
July 14, 2026 08:14
This was referenced Jul 14, 2026
Owner
|
Pouyan, thank you for this comprehensive Estonian language pack. The two-pass checksum and date validation, native Faker wiring, synthetic offset fixture, and leakage coverage are especially strong. I verified the implementation against the official Estonian personal-code structure and added a small maintainer follow-up to keep the validator fail-closed: current century and sex markers 1–6 only, canonical ASCII digits rather than silent normalization, non-string rejection, and native street diacritics. The branch is current with master; the focused suite passes (479 tests), the full suite passes (4,751 tests, 52 skipped), formatting, lint, and type gates pass, and all hosted checks are green. This is ready to merge—excellent work. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements #891: adds Estonian as a national-ID-only language pack, mirroring the structure of the Latvian (
lv) pack.validate_estonian_isikukood: century/sex first digit (1–8), embedded birth-date plausibility, two-pass modulo-11 checksum with weight fallback (including the fallback-to-zero case)_ESTONIAN_PII_PATTERNS: isikukoodnational_idplus Estonian date, phone (+372), street-address, and context-gated postcode cuesetregistered inLANGUAGE_PII_PATTERNS/NATIONAL_ID_ONLY_LANGUAGES,et -> et_EElocale, fake data, and locale format hintsgenerate_estonian_isikukood+EstonianIsikukoodProvidersurrogate provider wired through the national-ID registryAnonymizersurrogate round-trips, pattern spans, fixture offsets, offline no-leakage de-identification)Two deviations from the issue's file list
The issue names five files; this diff touches nine. Both deviations were necessary for the feature to actually work:
Golden fixture path. The issue says
openmed/eval/golden/et.jsonl, but the fixture loader only globsfixtures/**/*.jsonl— a file at the issue's path would be silently ignored and never exercised. The fixture therefore lives atopenmed/eval/golden/fixtures/i18n/et.jsonl, alongside the ten existing language fixtures.Four wiring files (~18 additive lines total). Registering
etinNATIONAL_ID_ONLY_LANGUAGESauto-enrolls it in the locale-coherence and locale-format regression suites, which assert wiring outside the issue's five files. Without these,make testfails and surrogates can't round-trip:openmed/core/anonymizer/registry.py—_LOCALE_ID_METHODS["et_EE"] = "isikukood"(surrogate dispatch)openmed/core/anonymizer/providers/registry_ids.py— provider registration so Faker instances gain theisikukoodmethodopenmed/core/locale_formats.py—etdate-order/number-separator hints (required bytest_format_hint_covers_every_wired_language)openmed/core/anonymizer/providers/__init__.py— exports, matching every sibling packData provenance
Every isikukood in code, tests, and the fixture is synthetic, generated algorithmically from the checksum rules (including checksum-valid-but-date-impossible rejection vectors). No real Estonian personal codes anywhere.
Verification
make test: 4640 passed, 40 skipped, 0 failedpre-commit run --all-files: all hooks covering these files passet, including a 40-seed surrogate→validator round-tripOut of scope, as per the issue: no Estonian model training, no registry data bundled.
Closes #891