Default to_spec() files contain bare NaN tokens and are not valid JSON for strict parsers #80

Description

Specs written by to_spec(path) for a default Preprocessor are not valid JSON. The SimpleImputer steps keep missing_values=np.nan in their state. The serializer writes that value as a bare NaN token, which RFC 8259 does not allow. Python's json module accepts the token, so PreTab's own round trip works. Strict consumers reject the file: JavaScript's JSON.parse, Go's encoding/json, Rust's serde_json, Jackson with its default settings, and JSON schema validators built on such parsers. Any inf / -inf in fitted state would be written the same way, as bare Infinity / -Infinity tokens.

Only a Preprocessor without imputers (numerical_imputation=None and categorical_imputation=None) currently writes a standard JSON file.

Affected code

• pretab/compose/serialize.py:85-88: _encode passes floats through unchanged, including NaN and ±inf.
• pretab/compose/serialize.py:91-92 emits arrays with tolist(), which keeps NaN and ±inf.
• pretab/preprocessor.py:1023 writes the spec file with json.dump and the default allow_nan=True. :1083 serializes the same state with json.dumps for fingerprint_, also with allow_nan=True.

Reproduction

import json, os, tempfile
import numpy as np, pandas as pd
from pretab import Preprocessor

rng = np.random.default_rng(0)
X = pd.DataFrame({"age": rng.normal(40, 10, 100), "income": rng.normal(50, 10, 100)})
path = os.path.join(tempfile.mkdtemp(), "spec.json")
Preprocessor(random_state=0).fit(X, rng.normal(size=100)).to_spec(path)

text = open(path).read()
print("NaN tokens in spec:", text.count("NaN"))
def reject(token):
    raise ValueError(f"invalid JSON token {token}")
try:
    json.loads(text, parse_constant=reject)   # behaves like a strict (RFC 8259) parser
except ValueError as e:
    print("strict parse:", e)

Output (PreTab 1.0.0 (6bacaba), Python 3.12, numpy 2.5.3, pandas 2.3.3, scikit-learn 1.9.1, scipy 1.18.1):

NaN tokens in spec: 4
strict parse: invalid JSON token NaN

Every token is a "missing_values": NaN entry of an imputer. node -e 'JSON.parse(require("fs").readFileSync(process.argv[1]))' spec.json fails with SyntaxError: Unexpected token 'N', ..."_values": NaN, ... is not valid JSON.

Expected behavior

compose/serialize.py describes the format as "Portable, versioned JSON (de)serialization for the fitted preprocessor" and the document as "human-inspectable". docs/core_concepts/reproducibility.md calls it "a portable declarative spec". A portable JSON artifact should parse with any standard JSON parser.

Actual behavior

Every spec of a Preprocessor that imputes, which is the default, contains non-standard NaN tokens.

Impact

Specs cannot be read, validated or diffed by non-Python tooling such as model registries, CI checks, or JS / Go services, even though portability is the purpose of the format.

Root cause

Non-finite floats are not tagged by the encoder, and the JSON writer allows NaN.

Suggested fix direction

Encode non-finite floats (scalars and float-array elements) as a tagged value, e.g. {"__float__": "nan"}, decode the tag, and dump with allow_nan=False. Specs written by older versions stay loadable, since Python's reader accepts their NaN tokens. The tagged encoding changes the canonical JSON behind fingerprint_.

Related issues

• to_spec / fingerprint_ fail or write unloadable specs for object arrays of numpy scalars and builtin dtypes #66: the same array branch of _encode writes object arrays with a raw tolist(), so numpy scalars break to_spec / fingerprint_; both fixes change how _encode writes array elements.