Repository navigation
Instrument payment phases and simplify shared chain paths - #43
Conversation
j0sh
changed the title
025f316 to
eb1f8fd
Compare
eb1f8fd to
15c6057
Compare
|
Completed the requested end-to-end payment-path comparison against the combined stack, using builder issue #89 as the reference. Performance-test implementation and reproduction harness: livepeer/e2e-runner#3. Revision scope: these measurements and pass/fail outcomes are for combined commit At 110 ms additional latency per JSON-RPC HTTP exchange, the combined stack reduced median serial dispatch from 3,248.753 ms to 2.619 ms (99.9194%). Its warm latency was slightly higher than PR #1, and go-livepeer remained faster at 32 clients. The combined stack passed the shared cache-correctness cases that failed on PR #1.
The six go-livepeer warm failures were paid-request HTTP 409 The test ran matched signer/orchestrator pairs on one GCP Pinned comparison inputs:
There were 48 final cells and 27,648 attempted warm jobs: three repetitions at 0/110 ms added RPC delay and 1/32 clients, with 128/1,024 jobs per repetition respectively. Each cell used fresh worker processes, payer, and databases, sharing one already-active recipient. Eight warmup jobs preceded measured load; explicit readiness retries were retained as warmup records. Timing targeted approximately 10,000,000 wei ticket EV and one ticket per job. Actual ticket parameters were decoded and retained; go-livepeer selected a different face value/winning probability with the same approximate EV. JSON-RPC batches retained their original bodies and incurred one delay per HTTP exchange. All latency distributions and observed goodput
Quantiles are pooled across the three repetitions. Serial p99 is omitted because only 384 successes are available per condition; concurrent conditions have at least 3,067. These are finite closed-loop bursts. The cached paths finish quickly, so the goodput figures do not establish sustained saturation capacity. Ratios are observed differences, without statistical confidence bounds. Immediate first-job probes are separate from warm results. The combined stack succeeded on 10/12 (83.3%), with two signer HTTP 400 At 110 ms, median successful first-job dispatch for the serial condition was 455.163 ms combined, 230.778 ms PR #1, 228.656 ms go-livepeer, and 3,255.484 ms baseline Node. The controlled payment-read sequence used four cold exchanges combined versus two batched exchanges for PR #1; both used zero warm exchanges, versus 29 warm exchanges for baseline Node. The cold failures were retained and were not substituted with successful warm requests. The real-chain and shared controlled correctness outcomes were:
The frozen-head age requirement is asserted for cached implementations. The direct baseline performs fresh RPC reads and has no equivalent cache-age promise. PR #1 failed the same additional, test-only scenarios against the shared provider fixture:
The combined stack rejected the stale/orphaned observations and canceled provider work. Other shared cases covered funding/current-round/registry changes, payer failure isolation, pre-cancellation, concurrent cold reads, and payer populations of 0/16/128/1,024. In the 32-distinct-key cold burst, combined provider concurrency was bounded at four, with 28/32 requests rejected while warm cached reads remained available. This is an admission-control tradeoff, separate from the all-successful warm load. Recovery covered expired parameters, actual redemption and duplicate rejection, persisted pending tickets across restart, RPC interruption, delayed finality, and reorg recovery. The fixtures used genuine held finalized headers, canonical EIP-1898 reads, and actual snapshot/revert. Preparation allowed a bounded 20-second wait for funded-payer visibility before scenario faults; strict post-expiry rejection remained in place. Canonical replay was distinguished by exact transaction hash from the original already-public transaction: its original receipt still had to follow expiry, every settlement receipt had to match the canonical block hash, and canonical escrow was debited once. Superseded setup/assertion failures and subsequent reruns were retained. A separate race-detector probe against the unchanged go-livepeer manifest-ID generator, in an isolated pinned source export with 32 callers, found a data race in Quiet-head refresh costs from the controlled provider test show the scaling tradeoff. Each entry is HTTP exchanges / contained JSON-RPC calls per refresh:
These quiet-history costs do not imply constant work when every payer changes on-chain. Whole-worker idle traffic also remains nonzero: under this harness at 110 ms after known-payer load, combined emitted approximately 19.18 contained RPC calls/s, versus 17.40 PR #1, 17.93 baseline Node, and 1.57 go-livepeer, counting both workers. Load-window RPC totals include background polling/redemption traffic and are not exact foreground counts. Worker resource statistics at 110 ms and 32 clients
CPU comes from worker process counters and excludes setup, the load generator, and Anvil. RSS was sampled every 100 ms; the table takes the maximum sampled peak across repetitions. Interpretation limits: the final reviewed combined revision was measured after the frozen baseline matrix on the same VM, so small differences between cached implementations can include ordering effects. The local chain mined at one-second intervals; production execution-block cadence, large history gaps, event-heavy histories, and Internet latency can alter results. This tests application behavior rather than production consensus safety. Test-only overlays did not alter the product checkouts. The raw JSONL requests/RPC logs, per-cell metadata, source/binary hashes, fixture overlays, and reproduction harness are retained with the run artifacts. The temporary VM and boot disk were deleted and independently verified absent. |
15c6057 to
ddd14e4
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The regression test counts background observer requests, so ticker timing can inflate cold or warm request budgets and cause CI failures even when the foreground path is correct.
1 open issue
What changed in this PR
This PR extends payment observability across the signer, orchestrator, and shared Ethereum client, while simplifying chain interfaces and consolidating regression fixtures.
Changes:
- Adds payment-phase durations, allowlisted contract-call counts, and foreground/background RPC accounting.
- Exposes memory-only observer health, freshness, progress, and partial-failure metrics.
- Removes redundant adapters and reuses fixtures for payment budgets, startup, and canonical-state checks.
| File | Description |
|---|---|
| signer/service.go | Records payment-generation duration. |
| signer/service_test.go | Checks duration metrics and invalidated signing. |
| signer/payment_test.go | Shares paid-request setup and adds cached RPC budgets. |
| signer/app.go | Registers observer metrics and labels background work. |
| signer/app_test.go | Reuses the shared readiness fixture. |
| pm/ethchain.go | Removes the redundant payment-chain adapter. |
| orchestrator/payment_recipient_test.go | Checks phase durations and reuses observer setup. |
| orchestrator/payment_engine.go | Times challenge, refresh, and receipt phases. |
| orchestrator/metrics.go | Adds the payment-duration helper. |
| orchestrator/metrics_test.go | Shares observed-chain test setup. |
| orchestrator/app.go | Uses the concrete chain and instruments background work. |
| orchestrator/app_test.go | Consolidates startup RPC fixtures. |
| internal/test/chainrpc.go | Extends shared RPC simulation and argument checks. |
| internal/observability/metrics.go | Adds contract, work, and observer metrics. |
| internal/observability/metrics_test.go | Checks local scrapes, transitions, and metric compatibility. |
| eth/transactionmanager.go | Labels transaction monitoring as background work. |
| eth/transactionmanager_test.go | Verifies background context propagation. |
| eth/senderwatcher.go | Preserves work labels for shared cold reads. |
| eth/senderwatcher_test.go | Checks failure visibility and consolidates recovery cases. |
| eth/rpc.go | Adds structured observations and contract-label allowlisting. |
| eth/rpc_test.go | Updates transport checks for structured observations. |
| eth/pricefeed_test.go | Reuses the RPC test helper. |
| eth/payment.go | Implements the payment interface directly and simplifies address resolution. |
| eth/payment_test.go | Reuses canonical-read and receipt fixtures. |
| eth/payment_observer.go | Adds refresh hooks and simplifies lifecycle state. |
| eth/payment_observer_test.go | Consolidates header and observation-availability checks. |
| eth/payment_contracts.go | Removes redundant address helpers. |
| eth/payment_contracts_test.go | Uses the renamed payer-funds method. |
| docs/metrics.md | Documents payment durations and RPC accounting. |
Review effort: Balanced
| before := p.rpc.Stats().RoundTrips | ||
| p.request(t, t.Context(), "cold") | ||
| require.Equal(t, 4, p.rpc.Stats().RoundTrips-before, "one cold pair read in each process") | ||
| require.EqualValues(t, 4, test.MetricSum(t, p.metrics.Registry, "rpc_work_total", map[string]string{"work": "foreground", "kind": "transport"})+test.MetricSum(t, p.service.metrics.Registry, "rpc_work_total", map[string]string{"work": "foreground", "kind": "transport"})) | ||
| before = p.rpc.Stats().RoundTrips | ||
| p.request(t, t.Context(), "warm") | ||
| require.Equal(t, before, p.rpc.Stats().RoundTrips, "new manifests for a known payer read memory") |
Instrument payment phases and authoritative RPC work. Consolidate shared chain paths and RPC fixtures, retain CI regressions, and remove the local payment benchmarks and historical comparison harness.
ddd14e4 to
0b13772
Compare
Concurrent payment receipts queue behind the global payment-engine mutex. Verify request-local tickets and encode challenges before that lock, reuse prepared admission SELECTs, combine watermark and exposure into one read, and record accepted counters after unlocking. Stacked directly on #43 (`15c6057`), without the main rebase. Session identity, freshness, expiry, capacity, replay, collateral, liability, randomness exposure, durable winning-ticket storage, and credit publication retain their checks and transaction boundary. No admission results are cached. Counters finish before `Receive` returns and only emit for accepted batches. The 32-caller replay regression requires exactly one accepted two-ticket batch, durable winners, exact credit, and matching counters. The existing SQL rollback test now also requires zero accepted counters on failure. The local follow-up at `9cff0b6` targets the remaining contention: CPU/mutex/block profiles show receipts holding the engine mutex across SQLite admission and WAL work, which also queues challenge and charge operations. The follow-up combines only the watermark/exposure read in the same transaction and moves immutable challenge encoding before the mutex; the liability query is unchanged. Eight counterbalanced local 32-client trials (8,192 paid HTTP jobs each, real signatures and SQLite, constant chain) improved the median of trial p50s from **2.010 to 1.882 ms (6.4%)**, with throughput **13,406 to 14,105 jobs/s (5.2%)**. Every paired p50 improved. Serial latency was effectively unchanged (0.297/0.298 ms, five trials). Direct receipts with 16 and 256 outstanding winners showed no regression. A larger one-statement prototype regressed with populated liability tables and was discarded. These follow-up results are local; the cloud table below applies to `2c11ea6`. The previous frozen-source [e2e comparison](livepeer/e2e-runner#3) measured `15c6057` against this change (`2c11ea6`) and clean upstream go-livepeer `2a4f7c4`. Three repetitions rotated implementation order on one GCP `n2-standard-8`, using Go 1.27.3, native Anvil, protocol `3c01f3a`, and 0/110 ms added latency per JSON-RPC HTTP exchange. Each repetition used 128 serial or 1,024 concurrent warm jobs after eight warmups. | Added RPC delay | Clients | #43 p50, ms | Measured `2c11ea6` p50, ms | go-livepeer p50, ms | |---|---:|---:|---:|---:| | 0 ms | 1 | 1.541 | 1.411 | 1.626 | | 0 ms | 32 | 13.341 | 7.960 | 7.316 | | 110 ms | 1 | 1.547 | 1.412 | 1.651 | | 110 ms | 32 | 13.178 | 7.865 | 7.407 | On that cloud run, at 32 clients median dispatch fell about 40% and observed goodput rose about 52% versus #43. Median dispatch improved in every paired repetition and condition. At 110 ms / 32 clients, p95 fell from 20.194 to 14.261 ms. Measured `2c11ea6` medians were 6–9% higher than go-livepeer at 32 clients; `2c11ea6` was faster in the serial conditions. Both Node revisions succeeded on 6,912/6,912 warm requests; go-livepeer succeeded on 6,910/6,912, with two paid-request HTTP 409 `session already exists` responses. All three succeeded on 12/12 separate cold probes in this run. Failed requests remain in attempted counts and goodput; latency quantiles use successes. A separate unchanged go-livepeer manifest-ID diagnostic reproduced a data race and 246 duplicate IDs in 32,768 generations. Cloud validation at `15c6057` and `2c11ea6`: 38/38 real-chain protocol cases, 6/6 recovery cases, 13/13 controlled-cache scenarios, and full repository `go test -race ./...` passed. The candidate also passed the local orchestrator race suite and repeated targeted race checks. The requested Sol/Astra, Opus, Gemini, Muse, Grok, Terra documentation, and final minimization review cycle is complete with no accepted findings remaining. Dispatch measures challenge → remote signing → paid forwarding to the echo runner's recorded arrival. These finite closed-loop bursts exclude GPU inference and Internet gateway latency. The go-livepeer reference matches the PR #43 benchmark pin; it is separate from the modified local checkout. Raw requests/RPC traces, manifests, binary/source hashes, and review reports are retained. The temporary VM and boot disk were deleted. GitHub CI at `9cff0b6` passed the race/SDK process tests, vet and staticcheck. Its [vulnerability scan](https://github.com/livepeer/node/actions/runs/42151654894/job/127676071069) fails on GO-2026-6865 in `klauspost/compress v1.20.1` (fixed in `v1.20.2`). The same `govulncheck v1.8.0` scan reproduces the identical finding on unchanged parent `15c6057`; this PR leaves `go.mod` and `go.sum` unchanged. This inherited CI blocker remains at the measured revision. The `9cff0b6` follow-up passed the full orchestrator race suite in 67.670 seconds and the same full review workflow. No benchmark fixtures, dependency changes, cached admission results or weakened checks are included.
Depends on #42. Merge after that PR.
Adds payment-phase duration histograms, fixed-allowlist contract-call counts, and foreground/background RPC accounting. Existing public metric names and label sets remain compatible. Observer health, observation age, head/L1 progress and partial payer failures come from memory; scrapes make no RPC calls.
Removes redundant chain/address adapters and duplicate observer flags, and reuses shared startup and paid-request fixtures. The concrete payment chain implements the existing interfaces directly. Cancellation remains serialized with cold-read admission before shutdown joins readers.
Focused regression tests run in normal CI. They enforce four foreground RPCs for a cold cached payment and zero for warm requests, alongside the direct-read budget, RPC failures, stale observations, canonical history, reorg, cancellation and settlement checks.
Production metrics support diagnosis and external measurement. End-to-end performance, deployment/provider comparisons and load scaling are measured through livepeer/e2e-runner.
Validation: affected package race suites (
eth,internal/observability,orchestrator,signer) and diff/format checks passed. Source attribution is retained.Closes #36.