Repository navigation
Cache payer collateral with bounded canonical recovery - #42
Merged
Merged
Conversation
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 341b39a0c53c05203d4e7b01d887a12ace1ebd89 Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 6a283a135e40e6738b06b4152ee56c44b75ab32d Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 57e28870688c191b0d669d3a1776f848348ef470 Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 5b74602591c67645a422b5414d0fb052158248e4 Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
…ogic, withdrawRound instead of withdrawBlock Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 93e7256c8ac37c35f8b48414a969003d90b8e76f Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 30942f652de14e5ac1eea7a7ea7e0ff7c6cc7f7d Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 1799b33eafcb50fa48171dd5509a50d05fb8fab4 Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: ebdaed880f96e07b72288090e54f1141e8b404ab Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 70d62a01a49014ce2650f27b7c70b58af5b6f31b Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
…hat watchers for both latest rounds and latest blocks Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: b4605f705bcad5e355bb0c738d784e4503089900 Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 46f074f0cfe6eb794f73bff15476bfb3f7094aa4 Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
…onitor is enabled Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 06245d9dbb2878bee37680aef263a892f6ec8f62 Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: aa6f7d578ad93c0fa36226f105c41452b6119e2f Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
…e server. This results in a node that can run all on-chain services using the on-chain registered keys Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: c162e069f2559108641807bbcfacf3fab884e154 Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 87c6e404c7a8bd5552e66e346d3659d696e17f4f Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: f30819e43a6b0d9272301a402220d7df64d70063 Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 754c1ed203539275332363eb2cd2940a7fa6c72a Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Fix typos across project. Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 12358f5082b18a3e200dd99534ea92f91baea258 Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Signed-off-by: houpo-bob <houpocun@outlook.com> Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 9755a0dd280821a87e73d40871fc7f2775f150f5 Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
j0sh
force-pushed
the
ja-codex/payment-33-cached-collateral
branch
from
November 4, 2026 00:03
de98232 to
0f1c07b
Compare
j0sh
force-pushed
the
ja-codex/payment-33-cached-collateral
branch
from
November 4, 2026 00:21
0f1c07b to
6539bca
Compare
Cache payer collateral with bounded shared cold reads and canonical event invalidation. Preserve withdrawal and reserve accounting, held observation revocation, payer failure isolation, and durable settlement watermarks. Retain source attribution in the implementation and tests. Retain the scoped go-livepeer source history as an additional parent. Its original authors, committers, dates, and messages are preserved; Original-Commit records upstream identities. This commit owns the Node adaptation and integration. Source-Repository: https://github.com/livepeer/go-livepeer Source-Revision: 597f882cb9f4d42ad8430b0f592afd5929af4a6b Source-History: 8f0bcf7
j0sh
force-pushed
the
ja-codex/payment-33-cached-collateral
branch
from
November 6, 2026 23:14
6539bca to
e88b092
Compare
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 57e28870688c191b0d669d3a1776f848348ef470 Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 5b74602591c67645a422b5414d0fb052158248e4 Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
…ogic, withdrawRound instead of withdrawBlock Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 93e7256c8ac37c35f8b48414a969003d90b8e76f Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 30942f652de14e5ac1eea7a7ea7e0ff7c6cc7f7d Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 1799b33eafcb50fa48171dd5509a50d05fb8fab4 Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: ebdaed880f96e07b72288090e54f1141e8b404ab Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 70d62a01a49014ce2650f27b7c70b58af5b6f31b Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
…hat watchers for both latest rounds and latest blocks Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: b4605f705bcad5e355bb0c738d784e4503089900 Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 46f074f0cfe6eb794f73bff15476bfb3f7094aa4 Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
…onitor is enabled Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 06245d9dbb2878bee37680aef263a892f6ec8f62 Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: aa6f7d578ad93c0fa36226f105c41452b6119e2f Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
…e server. This results in a node that can run all on-chain services using the on-chain registered keys Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: c162e069f2559108641807bbcfacf3fab884e154 Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 87c6e404c7a8bd5552e66e346d3659d696e17f4f Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: f30819e43a6b0d9272301a402220d7df64d70063 Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 754c1ed203539275332363eb2cd2940a7fa6c72a Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Fix typos across project. Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 12358f5082b18a3e200dd99534ea92f91baea258 Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
Signed-off-by: houpo-bob <houpocun@outlook.com> Source-Repository: https://github.com/livepeer/go-livepeer Original-Commit: 9755a0dd280821a87e73d40871fc7f2775f150f5 Source-Scope: eth/watchers/senderwatcher.go; eth/watchers/senderwatcher_test.go
j0sh
force-pushed
the
ja-codex/payment-33-cached-collateral
branch
from
November 6, 2026 23:32
799025b to
30bd2b7
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
Missing definitions, dependencies, and test fixtures prevent compilation, while the watcher remains disconnected from the payment path and has unresolved critical cache-correctness issues.
17 open issues
Undefined test fixtures prevent compilation Store removed-round refresh results back in the cache Isolate removed-round reread failures per cache entry Guard missing cache entries before emitting metrics Invalidate collateral and claims before removed-log recovery Keep unknown claim totals uncached until a full lookup Send event notifications after releasing the cache lock Process block and round events in canonical order Invalidate cached state when event delivery fails Return and store copies of claimed reserve values Include claimant in cache keys and recipient-specific updates Reconcile cold reads with canonical event updates Return deep copies to protect cached sender info Missing watcher interfaces and support prevent compilation Bound cache growth with capacity and idle eviction Cold reads lack deduplication, limits, and cancellation Cache is not connected to the payment observation path
Review effort: Balanced
j0sh
added a commit
that referenced
this pull request
Nov 6, 2026
Depends on #42. Merge after that PR. Adds payment-phase duration histograms, fixed-allowlist contract-call counts, and foreground/background RPC accounting. Existing public metric names and label sets remain compatible. Observer health, observation age, head/L1 progress and partial payer failures come from memory; scrapes make no RPC calls. Removes redundant chain/address adapters and duplicate observer flags, and reuses shared startup and paid-request fixtures. The concrete payment chain implements the existing interfaces directly. Cancellation remains serialized with cold-read admission before shutdown joins readers. Focused regression tests run in normal CI. They enforce four foreground RPCs for a cold cached payment and zero for warm requests, alongside the direct-read budget, RPC failures, stale observations, canonical history, reorg, cancellation and settlement checks. Production metrics support diagnosis and external measurement. End-to-end performance, deployment/provider comparisons and load scaling are measured through [livepeer/e2e-runner](https://github.com/livepeer/e2e-runner). Validation: affected package race suites (`eth`, `internal/observability`, `orchestrator`, `signer`) and diff/format checks passed. Source attribution is retained. Closes #36.
There was a problem hiding this comment.
🟡 Changes recommended
Two critical issues can admit unfunded tickets: reverted receipts release liability without advancing the collateral watermark, and later round initialization fails to invalidate cached collateral.
19 open issues
Invalidate cached collateral when the initialized round changes · New Reverted liability releases must advance the collateral watermark · New Undefined test fixtures prevent compilation Store removed-round refresh results back in the cache Isolate removed-round reread failures per cache entry Guard missing cache entries before emitting metrics Invalidate collateral and claims before removed-log recovery Keep unknown claim totals uncached until a full lookup Send event notifications after releasing the cache lock Process block and round events in canonical order Invalidate cached state when event delivery fails Return and store copies of claimed reserve values Include claimant in cache keys and recipient-specific updates Reconcile cold reads with canonical event updates Return deep copies to protect cached sender info Missing watcher interfaces and support prevent compilation Bound cache growth with capacity and idle eviction Cold reads lack deduplication, limits, and cancellation Cache is not connected to the payment observation path
Review effort: Balanced
| func (o *PaymentObserver) payerChanges(ctx context.Context, prior, head *miniHeader, snapshot ChainSnapshot, continuous bool) (map[common.Address]bool, bool) { | ||
| o.mu.RLock() | ||
| reset := !continuous || (prior != nil && new(big.Int).Sub((*big.Int)(head.Number), (*big.Int)(prior.Number)).Cmp(big.NewInt(128)) > 0) || | ||
| o.current.CurrentRound.Cmp(snapshot.CurrentRound) != 0 || o.current.addresses != snapshot.addresses |
| return err | ||
| } | ||
| defer tx.Rollback() | ||
| if phase == "confirmed" || phase == "used" { |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Depends on #41. Merge after that PR.
Known payers use immutable cached collateral at the shared observation. A bounded hash-linked history proof keeps unchanged payers warm across canonical polling gaps; incomplete history resets entries for pinned rereads. Funding, withdrawal, reserve, round and Controller target changes invalidate the affected entries, including upgrades with unchanged public proxy addresses.
Cold reads share work by payer/recipient and have a global four-key limit with no distinct-key queue. Cancellation, eviction, failures and shutdown preserve those bounds. A durable settlement watermark is recorded with liability release and checked inside admission, preventing pre-settlement collateral from being paired with released liability.
Retains recipient-specific reserve/withdrawal semantics, replay guards, durable acknowledgement and redemption protections. Shared implementation and tests retain source attribution.
Validation: affected and full repository race suites, vet, canonicality/financial/recovery regressions and diff checks passed on the final reviewed series.
Closes #33.