Reliable Visual Intelligence Under Corruption

Reproducible study of calibration and selective prediction for handwritten-digit recognition under distribution shift.

The benchmark compares logistic regression, a two-hidden-layer multilayer perceptron (MLP), a five-member MLP ensemble, and clean-validation temperature-scaled variants on the official UCI Optical Recognition of Handwritten Digits split. Test corruptions include Gaussian noise, center occlusion, and contrast reduction at three severities.

Main result

Clean-set calibration does not transfer uniformly across shift mechanisms.

• Clean calibrated ensemble: 96.77% accuracy, 1.02% ECE.
• Severe Gaussian noise: ensemble plus temperature scaling reduces ECE from 13.14% (single MLP) to 2.60%.
• Severe contrast reduction: clean-fitted temperature scaling increases single-MLP ECE from 1.92% to 32.26%, without changing accuracy.
• Severe center occlusion: every method falls below 26% accuracy.

Reproduce

Python 3.12 is recommended.

python -m pip install -r requirements.txt
python download_data.py
python run_experiments.py
python -m unittest discover -s tests -v
python validate_results.py

download_data.py is idempotent. The repository already contains the exact official data files used for the archived results; their SHA-256 hashes are recorded in results/metadata.json.

Evidence layout

reliable_digits/          Core calibration, corruption, data, metric, and plotting modules
tests/                    Regression tests for reliability logic
data/raw/                 Official UCI training and test files
results/all_metrics.csv   Every model-condition metric
results/member_metrics.csv
results/selected_predictions.npz
results/metadata.json     Seeds, versions, timing, parameters, and data hashes
figures/                  Publication-ready generated figures
run_experiments.py        End-to-end experiment entry point
validate_results.py       Artifact and report-critical metric checks

Experimental protocol

• Official training file: 3,823 samples; stratified into 3,058 fit and 765 validation samples.
• Official writer-separated test file: 1,797 samples.
• MLP architecture: 64 -> 128 -> 64 -> 10, ReLU, Adam.
• Ensemble seeds: 11, 29, 47, 83, 101.
• Metrics: accuracy, macro-F1, NLL, multiclass Brier score, 15-bin ECE, AURC, and error at 80% coverage.
• Corruptions: deterministic Gaussian noise, central occlusion, and contrast reduction.

Dataset attribution

E. Alpaydin and C. Kaynak, “Optical Recognition of Handwritten Digits,” UCI Machine Learning Repository, 1998. DOI: 10.24432/C50P49. Dataset files are distributed under CC BY 4.0; see DATA_LICENSE.md.

License

Code is released under the MIT License. Dataset files retain their CC BY 4.0 terms.