Measure payment phase latency, contract calls, and background chain work #36

Problem

Node already records RPC method counts/durations and payment-operation counts, but the reported payment delay cannot be diagnosed by payment phase or contract function. All contract reads appear as eth_call. Caching/batching work also needs evidence distinguishing request latency from background RPC load. See builder-core #89 and the existing metrics implementation.

go-livepeer provides the payment/watchers to incorporate in the related implementation tickets, but its existence does not establish measured latency for node. Extend node's current observability layer rather than importing a second metrics framework.

Work and acceptance

• Reuse the useful observability changes from BlueClawNetwork/livepeer-node PR #1: durations for challenge, refresh, receipt, and /generate-live-payment; fixed-allowlist contract/function counters; and shared-chain observation age, tracked entries, refresh outcomes/durations, and batch fallback metrics. Keep existing public metric names/labels compatible where possible.
• Count an HTTP JSON-RPC batch as one transport round trip and each contained contract call separately. Cover JSON-RPC errors carried over HTTP 200, partial/missing batch responses, transport failures, and batch refusal/fallback. Scrapes must read memory, not cause RPC traffic. Logs/labels must not expose RPC URLs, credentials, calldata, payer addresses, or other unbounded inputs.
• Distinguish observation age from head progress and last successful RPC contact using the shared observer's definitions. Preserve per-entry partial-failure visibility and log meaningful health transitions without repeating the same failure every refresh.
• Use the shared counting JSON-RPC fixture in normal CI to enforce zero foreground RPCs for warm requests and the expected call budget for cold requests. Keep focused correctness tests for RPC failures, stale data, canonical pinned reads and reorg/fallback cases; consolidate overlapping helpers and policy matrices.
• Expose foreground and background work separately through production metrics. Keep deterministic call-count assertions in CI; measure payer churn, population growth and request-rate/load scaling externally. Do not claim overall RPC volume is constant as the payer population grows.
• Document payment-phase metrics. End-to-end performance is being measured externally with livepeer/e2e-runner, including representative deployment/provider comparisons and load scaling. Record source revisions, provider and batch behavior with those measurements. Builder-core timings and cross-job challenge reuse stay in builder-core's tracking issue.

Coordinate instrumentation with shared block/round state #31, contract bindings #32, payer funds #33, signer observation #34, and recipient activity #35.

Reuse and attribution

For any go-livepeer code/tests incorporated here or in the associated counting harness, preserve originating author metadata and git-level provenance through suitable cherry-pick -x commits or a history-preserving import where practical. Where adaptation prevents that, retain license/copyright notices and add inline attribution identifying the source repository, file, commit, and author(s). Apply the same requirement to the observability and test code taken from PR #1; its metrics commit is 2c2ce3c.