Repository navigation
Conversation
…e rows
A policy mapping such as {"constant": "warn"} was expanded with the
dataclass defaults, so it reset out_of_range to "extrapolate" and dropped
the clip default of the B/M/I-spline, P-spline and tensor-product
families. Under "extrapolate" / "warn" those bases then evaluated to zero
outside the knot span (BSpline(extrapolate=False) plus nan_to_num, and
the Cox-de Boor recursion), turning every out-of-range row into an
all-zero basis row.
A mapping now overrides only the axes it names on top of the family
defaults (an unknown axis raises a typed error; a RepresentationPolicy
instance is still used verbatim), and out-of-range rows are evaluated by
extending the boundary polynomial pieces: BSpline(extrapolate=True) for
the B/M-splines, and for the out-of-range rows of the P-spline /
tensor-product basis. In-range values are unchanged. The policy
parameter docs of the clipping families and the edge-case notes in
choosing_a_method.md describe both behaviours.
Fixes #84
This was referenced Oct 16, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #84.
The B-, M-, I-, P-spline and tensor-product transformers clip out-of-range values by default, but a
policymapping that named onlyconstantswitched them to"extrapolate", and under"extrapolate"or"warn"the B-, M-, P-spline and tensor-product bases returned all-zero rows for out-of-range values. A standalone spline with such a policy silently lost every out-of-range value.Root cause
Two independent causes:
BasePreTabTransformer._resolved_policypassed an explicitpolicytoRepresentationPolicy.resolve, which buildsRepresentationPolicy(**mapping). Every axis the mapping did not name took the dataclass default (out_of_range="extrapolate"), not the family default ("clip")."extrapolate"and"warn"pass out-of-range values through to bases that are zero outside the knot span:BSpline(..., extrapolate=False)followed bynan_to_numin the B- and M-spline transformers, and the Cox-de Boor recursionbspline_basisin the P-spline and tensor-product transformers.Fix
BasePreTabTransformer._resolved_policyfirst builds the family policy from_policy,_constant_policyand_out_of_range_policy, then merges a mapping onto it withRepresentationPolicy.merge, so a mapping changes only the axes it names. ARepresentationPolicyinstance is still used verbatim.BSplineTransformer._design_matrixandMSplineTransformer._mspline_basisevaluate withBSpline(..., extrapolate=True). In-range values are identical either way.pretab.core.knots.extrapolate_bspline_rowsre-evaluates the rows whose value lies outside the knot span withBSpline(..., extrapolate=True), which extends the boundary polynomial pieces.PSplineTransformerandTensorProductSplineTransformerapply it to their Cox-de Boor basis; rows inside the span are left untouched.Behaviour changes
policymapping on any transformer that exposespolicykeeps the family default for every axis it does not name. For the clipping families this keeps"clip". Natural-cubic and cubic-regression splines are unaffected, because their family default equals the dataclass default.{"missing": "error"}) raisesInvalidParamError(aValueError) instead of a bareTypeError. A mapping value ofNoneis ignored instead of raising."extrapolate"or"warn", out-of-range rows of the B-spline, M-spline, P-spline and tensor-product bases are extrapolated values instead of zeros. The B-spline, P-spline and tensor-product rows sum to 1. The I-spline still returns 0 below and 1 above the range. The default ("clip") output is unchanged.Documentation
policyparameter docstrings ofBaseSplineTransformer(B/M/I),PSplineTransformerandTensorProductSplineTransformer:"warn"/"extrapolate"evaluate an out-of-range value on the extended boundary polynomial pieces (the I-spline stays at 0/1), and a mapping such as{"constant": "warn"}keeps the clipping._resolved_policydocstring and thepretab/core/policy.pymodule docstring: a mapping overrides only the axes it names on top of the family defaults; an instance is used verbatim.extrapolate_bspline_rows.docs/representations/choosing_a_method.md, "Out-of-range input at transform":"extrapolate"is available for the clipping families, what"extrapolate"/"warn"evaluate, and that a mapping keeps the family's other defaults.Tests
test_partial_policy_mapping_keeps_the_family_out_of_range_default[BSplineTransformer|MSplineTransformer|PSplineTransformer]:policy={"constant": "warn"}gives the same out-of-range rows as the default policy. Fails onmain, passes here.test_partial_policy_mapping_keeps_the_family_out_of_range_default[ISplineTransformer]: the same for the I-spline, whose out-of-range rows are 0/1 under both clip and extrapolate. Guard test: passes on both by design.test_tensorproduct_partial_policy_mapping_keeps_clipping: the same for the tensor product. Fails onmain, passes here.test_policy_mapping_with_an_unknown_axis_is_rejected:{"missing": "error"}raisesInvalidParamError. Fails onmain(TypeError), passes here.test_extrapolated_bspline_rows_are_not_zeroed[extrapolate|warn x BSplineTransformer|PSplineTransformer]: out-of-range rows sum to 1, and an in-range row equals the clipped transformer's. Fails onmain, passes here.test_extrapolated_mspline_and_tensor_rows_are_not_zeroed: the M-spline out-of-range row is non-zero, and the tensor-product row with one out-of-range coordinate sums to 1. Fails onmain, passes here.Reproduction
Output of the script from #84 on this branch (the last case also emits
DataWarning: BSplineTransformer received values outside the fitted range at transform time.):Checks
just test: 1449 passed, 59 skipped, 7 xfailed; coverage 93.71% (gate 90%).scripts/quickstart.pypasses.just docs: strict build succeeds.just check: all pre-commit and pre-push hooks pass (ruff format, ruff lint, prettier, file hygiene, pyright with 0 errors).Merge notes
main.pretab/core/knots.py/pretab/core/base.py, and the PRs for Target-aware location selectors return too few, duplicate, or boundary locations on tied/discrete features #57 (fix(placement): fill tied-data locations with distinct interior points #102) and Preprocessor crashes on an all-missing column at fit (numerical, one-hot) or silently drops it (int) #95 (fix(preprocessor): keep a block for columns with no observed value at fit #130) also editdocs/representations/choosing_a_method.md. Each merges cleanly with this branch.MSplineTransformer._mspline_basisa few lines below theextrapolate=Truechange. ISplineTransformer: 200-point trapezoid integration produces wrong and all-zero basis columns #53 evaluates the I-spline in closed form on values clipped to the knot range, so the I-spline still returns 0/1 outside the range, as the docstrings here state. With each of B/M/I spline and two-feature thin-plate spline transformers encode a missing value as an all-zero row #87, MSplineTransformer: absolute clipping thresholds make the basis depend on feature scale #63, ISplineTransformer: 200-point trapezoid integration produces wrong and all-zero basis columns #53, Spline knots on tied data land on the range boundary or repeat, producing dead columns and all-zero rows #56 and Standalone transformers drop DataFrame feature names, so reordered columns are silently misaligned #85 merged onto this branch,tests/core/test_policy.pyandtests/expansion/splinepass.This change was previously part of #86, which is closed in favour of one pull request per issue.