Skip to content

fix(splines): keep family policy defaults and extrapolate out-of-range rows - #121

Open
ChrisW09 wants to merge 1 commit into
mainfrom
fix/84-policy-merge-extrapolation
Open

ChrisW09 wants to merge 1 commit into
mainfrom
fix/84-policy-merge-extrapolation

Conversation

@ChrisW09

@ChrisW09 ChrisW09 commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #84.

The B-, M-, I-, P-spline and tensor-product transformers clip out-of-range values by default, but a policy mapping that named only constant switched them to "extrapolate", and under "extrapolate" or "warn" the B-, M-, P-spline and tensor-product bases returned all-zero rows for out-of-range values. A standalone spline with such a policy silently lost every out-of-range value.

Root cause

Two independent causes:

  • BasePreTabTransformer._resolved_policy passed an explicit policy to RepresentationPolicy.resolve, which builds RepresentationPolicy(**mapping). Every axis the mapping did not name took the dataclass default (out_of_range="extrapolate"), not the family default ("clip").
  • "extrapolate" and "warn" pass out-of-range values through to bases that are zero outside the knot span: BSpline(..., extrapolate=False) followed by nan_to_num in the B- and M-spline transformers, and the Cox-de Boor recursion bspline_basis in the P-spline and tensor-product transformers.

Fix

  • BasePreTabTransformer._resolved_policy first builds the family policy from _policy, _constant_policy and _out_of_range_policy, then merges a mapping onto it with RepresentationPolicy.merge, so a mapping changes only the axes it names. A RepresentationPolicy instance is still used verbatim.
  • BSplineTransformer._design_matrix and MSplineTransformer._mspline_basis evaluate with BSpline(..., extrapolate=True). In-range values are identical either way.
  • New helper pretab.core.knots.extrapolate_bspline_rows re-evaluates the rows whose value lies outside the knot span with BSpline(..., extrapolate=True), which extends the boundary polynomial pieces. PSplineTransformer and TensorProductSplineTransformer apply it to their Cox-de Boor basis; rows inside the span are left untouched.

Behaviour changes

  • A policy mapping on any transformer that exposes policy keeps the family default for every axis it does not name. For the clipping families this keeps "clip". Natural-cubic and cubic-regression splines are unaffected, because their family default equals the dataclass default.
  • A mapping with an unknown axis (for example {"missing": "error"}) raises InvalidParamError (a ValueError) instead of a bare TypeError. A mapping value of None is ignored instead of raising.
  • Under "extrapolate" or "warn", out-of-range rows of the B-spline, M-spline, P-spline and tensor-product bases are extrapolated values instead of zeros. The B-spline, P-spline and tensor-product rows sum to 1. The I-spline still returns 0 below and 1 above the range. The default ("clip") output is unchanged.

Documentation

  • policy parameter docstrings of BaseSplineTransformer (B/M/I), PSplineTransformer and TensorProductSplineTransformer: "warn" / "extrapolate" evaluate an out-of-range value on the extended boundary polynomial pieces (the I-spline stays at 0/1), and a mapping such as {"constant": "warn"} keeps the clipping.
  • _resolved_policy docstring and the pretab/core/policy.py module docstring: a mapping overrides only the axes it names on top of the family defaults; an instance is used verbatim.
  • Docstring of the new extrapolate_bspline_rows.
  • docs/representations/choosing_a_method.md, "Out-of-range input at transform": "extrapolate" is available for the clipping families, what "extrapolate" / "warn" evaluate, and that a mapping keeps the family's other defaults.

Tests

  • test_partial_policy_mapping_keeps_the_family_out_of_range_default[BSplineTransformer|MSplineTransformer|PSplineTransformer]: policy={"constant": "warn"} gives the same out-of-range rows as the default policy. Fails on main, passes here.
  • test_partial_policy_mapping_keeps_the_family_out_of_range_default[ISplineTransformer]: the same for the I-spline, whose out-of-range rows are 0/1 under both clip and extrapolate. Guard test: passes on both by design.
  • test_tensorproduct_partial_policy_mapping_keeps_clipping: the same for the tensor product. Fails on main, passes here.
  • test_policy_mapping_with_an_unknown_axis_is_rejected: {"missing": "error"} raises InvalidParamError. Fails on main (TypeError), passes here.
  • test_extrapolated_bspline_rows_are_not_zeroed[extrapolate|warn x BSplineTransformer|PSplineTransformer]: out-of-range rows sum to 1, and an in-range row equals the clipped transformer's. Fails on main, passes here.
  • test_extrapolated_mspline_and_tensor_rows_are_not_zeroed: the M-spline out-of-range row is non-zero, and the tensor-product row with one out-of-range coordinate sums to 1. Fails on main, passes here.

Reproduction

Output of the script from #84 on this branch (the last case also emits DataWarning: BSplineTransformer received values outside the fitted range at transform time.):

policy=None                                                              -> transform(1.5) = [[0.0, 0.0, 0.0, 0.0, 0.0, 1.0]]
policy={'constant': 'warn'}                                              -> transform(1.5) = [[0.0, 0.0, 0.0, 0.0, 0.0, 1.0]]
policy=RepresentationPolicy(constant='allow', out_of_range='extrapolate') -> transform(1.5) = [[0.0, 0.0, -0.78, 9.571, -31.603, 23.812]]
policy=RepresentationPolicy(constant='allow', out_of_range='warn')       -> transform(1.5) = [[0.0, 0.0, -0.78, 9.571, -31.603, 23.812]]

Checks

  • just test: 1449 passed, 59 skipped, 7 xfailed; coverage 93.71% (gate 90%).
  • Minimum dependencies (CI job: Python 3.10, numpy 1.24.4, pandas 2.0.3, scipy 1.10.1, scikit-learn 1.6.0): 1436 passed, 72 skipped, 7 xfailed; scripts/quickstart.py passes.
  • just docs: strict build succeeds.
  • just check: all pre-commit and pre-push hooks pass (ruff format, ruff lint, prettier, file hygiene, pyright with 0 errors).
  • Import direction between subpackages is unchanged.

Merge notes

This change was previously part of #86, which is closed in favour of one pull request per issue.

…e rows

A policy mapping such as {"constant": "warn"} was expanded with the
dataclass defaults, so it reset out_of_range to "extrapolate" and dropped
the clip default of the B/M/I-spline, P-spline and tensor-product
families. Under "extrapolate" / "warn" those bases then evaluated to zero
outside the knot span (BSpline(extrapolate=False) plus nan_to_num, and
the Cox-de Boor recursion), turning every out-of-range row into an
all-zero basis row.

A mapping now overrides only the axes it names on top of the family
defaults (an unknown axis raises a typed error; a RepresentationPolicy
instance is still used verbatim), and out-of-range rows are evaluated by
extending the boundary polynomial pieces: BSpline(extrapolate=True) for
the B/M-splines, and for the out-of-range rows of the P-spline /
tensor-product basis. In-range values are unchanged. The policy
parameter docs of the clipping families and the edge-case notes in
choosing_a_method.md describe both behaviours.

Fixes #84

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A partial spline policy mapping drops the "clip" default, and "extrapolate"/"warn" give all-zero basis rows

1 participant