[Krishnam24maheshwari]

Summary

This PR introduces a fully automated, end-to-end Data Commons import pipeline for the Sweden Census — Category 1: Demographic Population Statistics (Sweden_Census_Demographic) dataset, sourced from Statistics Sweden (Statistiska centralbyrån, SCB) across 4 official PxWeb v1 REST API tables under BE0101A (Population Statistics):

• National Population by Age and Sex (1860–present):
  • For (1860–2024, BE0101A/BefolkningR1860N): https://api.scb.se/OV0104/v1/doris/sv/ssd/START/BE/BE0101/BE0101A/BefolkningR1860N
  • Active CKM (2025–present, BE0101A/BefolkningR1860NCKM): https://api.scb.se/OV0104/v1/doris/sv/ssd/START/BE/BE0101/BE0101A/BefolkningR1860NCKM
• Sub-National Population by Region, Marital Status, Age and Sex (1968–present):
  • For (1968–2024, BE0101A/BefolkningNy): https://api.scb.se/OV0104/v1/doris/sv/ssd/START/BE/BE0101/BE0101A/BefolkningNy
  • Active CKM (2025–present, BE0101A/BefolkningCKM): https://api.scb.se/OV0104/v1/doris/sv/ssd/START/BE/BE0101/BE0101A/BefolkningCKM

The pipeline replaces legacy manual scripts with a resilient, zero-touch Python download, aggregation, and stat_var_processor.py workflow that ingests population counts across 322 Swedish geographies (country/SWE, 3 NUTS1 major regions, 8 NUTS2 national areas, 21 NUTS3 counties, and 289 municipalities) and 9 marginal breakdowns (across 23 CSV sub-imports):

• Historical Parity (1860–2022, 426,062 observations across 139 StatVars): Preserves 100.00% historical coverage and zero-deletion continuity (0 deleted records, 0 modified records) across all marginal groups and 322 places.
• New Refresh Coverage (2023–2025, 23,010 observations across 139 StatVars): Automatically discovered, downloaded, aggregated, and processed from the upstream SCB PxWeb REST API (7,670 observations per year × 3 years).
• Combined Dataset (1860–2025 National / 1968–2025 Sub-National): Produces 449,072 StatVarObservation nodes across 139 canonical Google3 StatisticalVariables with 0 deleted records, 0 modified records, 0 missing references, and 0 lint errors.

Key Changes & Features

1. Automated SCB PxWeb API Fetch & Multi-Level Rollup (download_demographic.py)

• Dynamic Year & Metadata Discovery: Queries the SCB PxWeb table metadata endpoints via download_util.request_url (handling UTF-8 BOM utf-8-sig payloads) to fetch 1860–2024 for national population in BefolkningR1860N and 1968–2024 for sub-national population in BefolkningNy, and dynamically discovers 2025+ releases (2025, 2026, 2027, ...) from the active CKM endpoints (BefolkningR1860NCKM and BefolkningCKM) without code changes.
• Official SCB Total-Age Queries for 100% Historical Parity:
  • Queries SCB's official total-age codes (Alder='tot' for <= 2024 and Alder='TotSA' for 2025+) directly for non-age breakdowns (population_country.csv, population_gender_country.csv, and regional marital-status/gender totals) rather than summing single-year ages, preserving historical unknown-age counts (1861–1869, 1871–1875, 1956–1959) and exact regional totals (1978 Örebro län) with 0 modified historical observations.
• Rate-Limit-Safe Chunked Fetching & Retry Logic:
  • Chunks BefolkningR1860N / BefolkningR1860NCKM national queries by 60-year windows, BefolkningNy / BefolkningCKM county single-year age queries by 5-year windows, and municipality/county total-age (Alder='tot' / 'TotSA') queries by 20-year windows to stay well below SCB PxWeb's 150,000-cell limit.
  • Respects SCB's 10 requests / 10 seconds rate limit (1.2s inter-request pacing) with built-in retry handling via download_util.
• Hierarchical Geographic & Age-Bracket Aggregation:
  • Derives the complete regional hierarchy dynamically from sweden_places_resolved.csv, aggregating 21 Swedish counties (01–25) into 3 NUTS1 (SE1–SE3), 8 NUTS2 (SE11–SE33), and 21 NUTS3 (SE110–SE332) regional hierarchies alongside 290 raw municipalities (289 resolved active municipalities).
  • Rolls up single-year ages (0..110+ and CKM open-ended 100+1) into 12 canonical national age brackets (0-4 år through 95-104 år and 105+ år for BefolkningR1860N / BefolkningR1860NCKM) and 11 canonical sub-national age brackets (0-4 år through 85-94 år and 95+ år for BefolkningNy / BefolkningCKM), while ignoring CKM multi-year summary codes (100+5, 100+10, 5-9, etc.).
  • Coerces SCB missing/suppressed tokens (.., ., -, "") to 0 to maintain 100% historical baseline continuity, writing all 23 wide-format CSVs atomically via .tmp files.

2. Unified Single-Config Marginal Schema & stat_var_processor.py Pipeline (run.sh)

• Bounded Parallel Execution (run.sh):
  • Runs stat_var_processor.py across all 23 CSVs with bounded concurrency (MAX_PARALLEL=4) and strict non-zero exit propagation.
• Unified Single-Config Architecture (demographic_pvmap.csv, demographic_metadata.csv, demographic_statvar_remap.csv):
  • Consolidates all 23 CSV sub-imports (spanning 0 to 4 dimension columns) under a single demographic_metadata.csv (mapped_columns=4), a single demographic_pvmap.csv, and a single demographic_statvar_remap.csv.
  • Uses Cell:3:1,observationAbout,dcid:country/SWE alongside region,observationAbout,{Data} so the same PVMap seamlessly handles both the 4 national files and the 19 sub-national files.
  • Uses header-scoped year keys (None:1860 through None:2035) so year headers match exclusively on Row 3 (get_last_column_header_key(col_index) == None) without colliding with numeric population counts in data rows when mapped_columns=4.
• Complete Marginal & Dimensional Coverage (139 Canonical StatVars across 23 CSV Sub-Imports):
  • Group 1A — National Population Breakdowns (39 StatVars, 4 CSVs: population_country, population_gender_country, population_age_country, population_age_gender_country):
    • Total (Count_Person), Gender (Count_Person_Male, Count_Person_Female), Age (12 brackets), and Age × Gender (24 StatVars) for country/SWE (1860–2025; 6,474 SVObs).
  • Group 1B.1 — Sub-National Population Total (1 StatVar, 4 CSVs: population_{nuts1,nuts2,nuts3,municipality}):
    • Count_Person across 321 regional places (1968–2025; 18,618 SVObs).
  • Group 1B.2 — Sub-National Population by Gender (2 StatVars, 4 CSVs: population_gender_{nuts1,nuts2,nuts3,municipality}):
    • Count_Person_Male, Count_Person_Female across 321 regional places (1968–2025; 37,236 SVObs).
  • Group 1B.3 — Sub-National Population by Marital Status (4 StatVars, 4 CSVs: population_maritalstatus_{nuts1,nuts2,nuts3,municipality}):
    • Count_Person_{Unmarried,Married,Widowed,Divorced} across 321 regional places (1968–2025; 74,472 SVObs).
  • Group 1B.4 — Sub-National Population by Marital Status × Gender (8 StatVars, 4 CSVs: population_maritalstatus_gender_{nuts1,nuts2,nuts3,municipality}):
    • Count_Person_{Male,Female}_{Unmarried,Married,Widowed,Divorced} across 321 regional places (1968–2025; 148,944 SVObs).
  • Group 1B.5 — Sub-National Population by Age × Marital Status × Gender (88 StatVars, 3 CSVs: population_age_maritalstatus_gender_{nuts1,nuts2,nuts3}):
    • Count_Person_{Age}_{Male,Female}_{Unmarried,Married,Widowed,Divorced} across 11 age brackets, 4 marital statuses, 2 genders, and 32 NUTS1–3 regions (1968–2025; 163,328 SVObs).
• Dimension & Age Bracket Mappings (13 Mapped Age Brackets):
  • Gender (Kon): män → dcs:Male, kvinnor → dcs:Female
  • Marital Status (Civilstand): ogifta → dcs:Unmarried, gifta → dcs:Married, änkor/änklingar → dcs:Widowed, skilda → dcs:Divorced
  • Shared 10-Year Age Brackets (0–94 Years):
    • 0-4 år → [0 4 Years]
    • 5-14 år → [5 14 Years] (remapped to Count_Person_Years5To14, Count_Person_Years5To14_Male, Count_Person_Years5To14_Female via demographic_statvar_remap.csv)
    • 15-24 år → [15 24 Years]
    • 25-34 år → [25 34 Years]
    • 35-44 år → [35 44 Years]
    • 45-54 år → [45 54 Years]
    • 55-64 år → [55 64 Years]
    • 65-74 år → [65 74 Years]
    • 75-84 år → [75 84 Years]
    • 85-94 år → [85 94 Years]
  • National Upper Age Brackets (BefolkningR1860N / BefolkningR1860NCKM):
    • 95-104 år → [95 104 Years]
    • 105+ år → [105 - Years]
  • Sub-National Upper Age Bracket (BefolkningNy / BefolkningCKM):
    • 95+ år → [95 - Years]
• Canonical DCID Remapping & Historical Place Set Parity:
  • Maps all 321 sub-national Swedish geographies via sweden_places_resolved.csv (nuts/SE* and wikidataId/Q*, with 2463 Åsele marked #ignore,skip in demographic_pvmap.csv for 100% historical parity) and national country/SWE.
  • Pre-populates None:1860–None:2035 observationDate keys in demographic_pvmap.csv for zero-touch future annual releases.

3. Production Orchestration & Validation (manifest.json, validation_config.json)

• manifest.json: Configured for automated weekly execution (cron_schedule: "0 05 * * 1", every Monday at 05:00 UTC) across all 23 cleaned CSVs (output/*.csv) with output/population_nuts1.tmcf.
• validation_config.json: Enforces MAX_DATE_CONSISTENT and SQL_VALIDATOR freshness (MaxDate >= '2025').

Directory Structure

statvar_imports/sweden_census/demographic/
├── test_data/
│   ├── population_country_input.csv                                    # Group 1A (National Total) sample input
│   ├── population_country_output.{csv,tmcf}                            # Group 1A expected output
│   ├── population_nuts1_input.csv                                      # Group 1B.1 (Sub-National Total) sample input
│   ├── population_nuts1_output.{csv,tmcf}                              # Group 1B.1 expected output
│   ├── population_maritalstatus_gender_nuts1_input.csv                 # Group 1B.4 (Marital×Gender) sample input
│   ├── population_maritalstatus_gender_nuts1_output.{csv,tmcf}         # Group 1B.4 expected output
│   ├── population_age_maritalstatus_gender_nuts1_input.csv             # Group 1B.5 (Age×Marital×Gender) sample input
│   └── population_age_maritalstatus_gender_nuts1_output.{csv,tmcf}     # Group 1B.5 expected output
├── download_demographic.py                                             # SCB PxWeb REST API downloader & NUTS/age aggregator (23 CSVs)
├── run.sh                                                              # Orchestrates stat_var_processor.py across all 23 CSVs
├── demographic_test.py                                                 # Hermetic unit & E2E processor test suite (9 tests)
├── sweden_places_resolved.csv                                          # 321 resolved Sweden NUTS & Municipality place DCIDs
├── demographic_pvmap.csv                                               # Unified PVMap (places, gender, marital status, ages, None:1860-2035)
├── demographic_metadata.csv                                            # Unified processor configuration (mapped_columns=4)
├── demographic_statvar_remap.csv                                       # Canonical Google3 StatVar DCID remap (Years5To14*)
├── manifest.json                                                       # Data Commons automated import manifest
├── validation_config.json                                              # Import validation rules and thresholds
└── README.md                                                           # Documentation and usage guide

[gemini-code-assist]

Code Review

This pull request introduces an automated import pipeline for Sweden Census Demographic Population Statistics (1860–2025) from Statistics Sweden (SCB), including data downloading, processing, and validation scripts, alongside comprehensive tests and configuration files. The feedback suggests improving the robustness of download_demographic.py by handling potential ValueError exceptions in _parse_int_value when parsing non-numeric or null values, and wrapping the deletion of http_cache.sqlite in a try-except block to prevent failures if the file is locked.