[yuechao-qin]

What

A preempted task restarting looks exactly like a slow queue, so one restart can set the
p99 that pages the on-call. Two different waits currently share one number:

• Waiting to start — how long until the task got a machine. This is Tangle's queue,
  and it is what the alert thinks it is measuring.
• Waiting to resume — how long until the task got its machine back, after the
  cluster took it away mid-run. Not Tangle's queue at all.

The task never had a machine in the first case, and had one and lost it in the second.

Why

PENDING->RUNNING means two unrelated things, and the metric cannot tell them apart:

  A NORMAL TASK
    QUEUED -> PENDING -> RUNNING          PENDING->RUNNING = 40 s
                                          "how long to get a machine"

  A PREEMPTED TASK  (Kueue suspends the job, the pod goes away, it re-queues)
    QUEUED -> PENDING -> RUNNING -> PENDING -> RUNNING -> PENDING -> RUNNING
                      |                     |                     |
                   40 s                   4 h                   2 h
                      \_____________________|_____________________/
                       ONE task, THREE samples. Two of them measure how long
                       until a tenant's cluster gave the machine BACK.

PENDING->RUNNING runs at ~300/hour, so a [5m] alert window holds ~25 samples — and
p99 of 25 samples is the single slowest one. A resumption does not have to be common to
be the number that pages someone; it has to happen once in the window.

Measured in production over 24 h: RUNNING->PENDING runs at 2.0/hour, p99 10,190 s.

Changes

One attribute, computed from the history list the session already holds. No query, no
new I/O on the commit path.

execution.run_attempt	when
first	the first time this execution reaches RUNNING
repeat	every time after that — the machine is coming back
none	every transition that does not end in RUNNING

none rather than omitting the attribute: an attribute that disappears forms a second
series that no filter on the first one covers.

"Attempt" counts times the execution held a machine, not times it was retried after an
error.

Nothing is merged and nothing is dropped — all three samples above are still recorded,
so existing queries and dashboards keep working.

Tests

• 8 parametrised cases walking the preempted history one entry at a time
• an end-to-end commit sequence asserting ["first", "repeat", "repeat"]
• the existing transition test updated for the new attribute