[yuechao-qin]

What

A latency alert can say a transition is slow. It cannot say whether that is one
automated pipeline retrying or forty people waiting. This counter answers the second
question, so an alert can carry a link to a ranked list of who is affected.

• one submitter stuck, versus every submitter stuck
• a service account looping, versus a room full of humans blocked
• the same question during an incident and after it, from one series

Why

The owner cannot go on the latency histogram: it multiplies every bucket of a
native-histogram series by the number of submitters. A counter written only on the slow
path costs one series per owner and is paid for by the slow transitions, not by all of
them.

	Owner on the histogram	Owner on this counter
Series added	every bucket × every owner	one per owner
Written	on all ~126,000 transitions a day	only above the threshold
Database read	on all of them	only above the threshold

Changes

execution.status_transition.slow, with execution.status.from,
execution.status.to and owner.

What counts as slow

Once a task is RUNNING the clock measures work, not queueing:

Transition	Gap		Why
QUEUED → PENDING	2h	counted	waiting for a machine
PENDING → RUNNING	9h	counted	waiting for a machine
PENDING → FAILED	7h	counted	waited, then never ran
RUNNING → SUCCEEDED	4h	skipped	a job that ran for four hours
QUEUED → PENDING	4s	skipped	under the threshold

Threshold: TANGLE_SLOW_TRANSITION_SECONDS, default 6000 — a deployment sets it to
whatever its own latency alert fires on.

Where the owner is read

before_commit runs inside the flush in progress, so a SELECT there re-enters it:

  before_commit    record histogram; if slow, stash the id on session.info
  after_commit     drain, one lookup on a session of its own, count

• one union query, in the shape the orchestrator already uses: an arbitrary node
  reaches its run through the ancestor link, a root execution has no such link to
  itself, so neither half answers alone
• a failed read is swallowed and the point still counted with owner="none" — an
  alert fires on this, so a point missing one label beats no point

What is stored

One monotonic series per owner, so a raw value is a lifetime total:

Series	Lifetime	Last hour	A submitter?
…slow{from="QUEUED", to="PENDING", owner="pipeline-runner@…"}	37	18	✅
…slow{from="QUEUED", to="PENDING", owner="batch-scorer@…"}	6	6	✅
…slow{from="QUEUED", to="PENDING", owner="none"}	3	0	❌ not this hour

What is viewed

Question	Query	Answer
who is affected	sum by (owner)(increase(…[1h]))	pipeline-runner@… 18, batch-scorer@… 6
how many people	count(count by (owner)(increase(…[1h]) > 0))	2

• none counts as one submitter when it moves, so the headcount is a floor
• ⚠️ created_by is often a service account, so the honest unit is teams: a count of
  3 can mean three humans or three pipelines

Tests

• a transition above the threshold is counted, one below it is not
• a slow RUNNING → SUCCEEDED and a slow RUNNING → FAILED are not counted, and a
  slow PENDING → FAILED still is
• the owner resolved both ways — a root execution, and a node under one via the ancestor
  link; each half of the union has a test that fails without it
• a node belonging to no run is counted with none, not dropped
• a lookup that raises still counts the point
• the commit path issues no SELECT — every statement run before the owner lookup is
  captured and asserted on, which is the rule the rest of the design exists to keep