Skip to content

fix(metrics): flatten column-vector targets in NegativeLogLikelihood - #509

Open
ChrisW09 wants to merge 2 commits into
mainfrom
fix/nll-metric-targets
Open

ChrisW09 wants to merge 2 commits into
mainfrom
fix/nll-metric-targets

Conversation

@ChrisW09

@ChrisW09 ChrisW09 commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #508

Problem

NegativeLogLikelihood.__call__ turned y_true into a tensor unchanged and called distribution.compute_loss(y_pred, y_true). For the (N, 1) targets that the training loop passes (and that evaluate() forwards unchanged when y_true is a column vector), log_prob broadcast the batch of N distributions against the column vector into an (N, N) grid, and the metric returned the mean of that grid. A val_metrics={"nll": NegativeLogLikelihood(family)} therefore logged a different value from val_loss for the same family. LogScore wraps the same metric and had the same problem.

Fix

NegativeLogLikelihood squeezes a trailing singleton dimension of y_true ((N, 1) → (N,)) before computing the loss. Multi-output targets (e.g. Dirichlet components) and 1-D targets are unchanged.

Tests

tests/metrics/test_metric_contracts.py::TestLikelihoodMetricsAcceptColumnTargets:

  • test_column_and_flat_targets_give_the_family_loss[NegativeLogLikelihood|LogScore]: flat and column targets both give the family's compute_loss (negated for LogScore).
  • test_logged_validation_nll_matches_validation_loss: in an MLPLSS fit, the logged val_nll equals val_loss.

All three fail on main and pass with this change.

Docs

No change needed. The metric was already documented as the family NLL.

Checks

  • just test: 3239 passed, 46 skipped, 28 xfailed; coverage 91.38%
  • just docs: build succeeded (warnings treated as errors)
  • just check: all hooks passed (ruff, prettier, pyright)

The training loop passes LSS targets as (N, 1) column vectors. In
NegativeLogLikelihood (and LogScore) they broadcast against the batch of N
distributions into an (N, N) grid of log-probabilities, so the logged
metric differed from the family loss. Squeeze a trailing singleton target
dimension before computing the loss.

Fixes #508
evaluate() forwards y_true unchanged, so it passes (N, 1) targets only
when the caller does.

Refs #508

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] NegativeLogLikelihood/LogScore broadcast (N, 1) targets into an N x N grid, so val_nll differs from val_loss

1 participant