Skip to content

feat(observability): label a task whose pods were never created - #413

Draft
yuechao-qin wants to merge 1 commit into
ycq/observability-05-pending-reasonfrom
ycq/observability-06-pods-never-admitted
Draft

yuechao-qin wants to merge 1 commit into
ycq/observability-05-pending-reasonfrom
ycq/observability-06-pods-never-admitted

Conversation

@yuechao-qin

@yuechao-qin yuechao-qin commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

What

A task whose pods are refused at admission now says so, instead of looking identical to
a task nobody has told us anything about.

The execution.pending_reason label gains a sixth value, no_pod_created, and the two
scenarios it separates are:

  • a task waiting for capacity that will eventually arrive, and
  • a task whose pods were refused outright, so nothing will ever arrive

The second is a tenant's own namespace quota, RBAC rule or pod security policy. It
should never page the on-call for Tangle capacity.

Why

The existing five values all come from reading a pod. When admission refuses every pod,
there is no pod to read, and the label falls back to none — the same value a task gets
when the launcher is not Kubernetes at all. A four-hour wait caused by an exhausted quota
is therefore indistinguishable from a four-hour wait we have no information about.

Which admission rule refused the pod is not knowable from the data we hold. Kubernetes
records that as a FailedCreate Event, and launcher_data serialises the Job and its
pods, never Events. This change reports that nothing was scheduled, not why — enough to
route the alert, and the operator reads Events to fix it.

Changes

What the new value means

no_pod_created is returned when the launcher is Kubernetes and the serialised pod map
is empty. It is last-stamp-wins, not an observation: a Job is launched with an empty pod
map, so every task is briefly in this state, and a healthy one is overwritten within
seconds by the first poll that sees a pod. It survives to be read only when no pod ever
appears.

A guard that was missing

pending_reason previously collapsed a non-Kubernetes launcher into an empty pod map.
That was harmless while an empty map meant "say nothing", and would now label every
local-process task as a refused Kubernetes pod. The launcher key is checked explicitly.

Why there are now two listeners

The orchestrator supplies the first launcher_data as a constructor argument. That does
reach the existing set listener, which computes the right reason — and __init__ then
throws it away, because extra_data is declared after launcher_data and gets assigned
its own default of None. Until now that destroyed nothing, since a Job at launch has an
empty pod map and no reason existed to lose. no_pod_created is the first reason that
can exist at construction time, so the clobber starts to matter here.

A before_insert hook stamps the row at flush, once the object is fully built, where no
declaration order can reach it. The write rides the same INSERT and costs no second
statement.

Neither listener can replace the other. The metric reads the reason at the session's
before_commit, and every mapper flush hook runs after that, so a reason stamped at
before_update would be read a commit too late — the set listener has to stay for
polls. The set listener in turn cannot survive __init__ — the insert hook has to stay
for the first write. Both docstrings say so, and the stamping body is shared rather than
duplicated.

Tests

  • a Job with no pods is not silent
  • a launcher that is not Kubernetes still says nothing
  • a pod that exists but is not Pending is still nothing
  • the reason supplied to the constructor is persisted, read back from the database
  • a later poll overwrites the reason written at the first write
  • the set listener alone would lose the constructor's reason, which fails loudly if
    extra_data is ever declared above launcher_data

661 passing. Mutation probes: dropping the no-pod branch turns 3 red, dropping the
not-Kubernetes guard turns 16 red, moving the insert hook to before_update turns 2 red,
moving the set listener to before_update turns 15 red, and declaring extra_data
above launcher_data turns the new ordering test red on its own.

@yuechao-qin
yuechao-qin added this pull request to stack #408 October 27, 2026 23:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant