Repository navigation
perf(beacon): walk the registry once in epoch steps 1-3 instead of per-index reads - #630
Closed
MegaRedHand wants to merge 3 commits into
Closed
MegaRedHand wants to merge 3 commits into
MegaRedHand wants to merge 3 commits into
Conversation
…r-index reads Justification, inactivity updates and rewards built index Vecs, then did a tree descent (state.validator(i)) per participant and a binary search per eligible validator, and the pulled-up-tip check repeats justification on every imported block. Each of those reads is a tree walk; the registry's iter() is a leaf-slice walk. Every spec function keeps its shape and signature, but now iterates validators().iter().enumerate() and indexes the flat participation and score lists by position. New helpers compute the unslashed participating balance and the total active balance (plus base reward per increment) in one pass without an index Vec; the public get_total_active_balance and get_base_reward_per_increment are untouched. The previous implementations are kept as a test-only reference, and randomized tests (altair and electra, short lists, off-grid balances, scores near the top of the range, leaking epochs) compare outputs, errors and post-step state against it.
…erf/beacon-epoch-participation-a
This was referenced Sep 30, 2026
This was referenced Oct 2, 2026
4 tasks
zexoverz
pushed a commit
to zexoverz/ethlambda
that referenced
this pull request
Oct 22, 2026
## 🗒️ Description / Motivation
Merges `beacon-chain-integration` into `main`. This adds an **Ethereum
Beacon Chain** client (phase0 through fulu) to the ethlambda binary,
alongside the Lean consensus client:
- **`ethlambda beacon`**: a beacon node. It checkpoint-syncs, follows
the chain over gossip and req/resp, validates execution payloads through
an execution client's Engine API, and serves the Beacon API, including
the validator endpoints.
- **`ethlambda validator`**: a beacon validator client (attest,
aggregate, propose) that talks to any beacon node over the standard REST
API.
- **`ethlambda node`** (lean) keeps its behavior, apart from the items
under "Behavior changes for lean" below.
Both chains run on the same storage layer, P2P actor, RPC crate and
chain actor. `BeaconState` has a `Lean` variant holding lean's `State`,
and each shared handler dispatches on it once, at the top.
Each piece was reviewed as its own PR into the integration branch
(listed under "Related Issues / PRs"). This PR merges that branch.
## What Changed
| Area | Change |
|---|---|
| `ethlambda-types` (`beacon/`) | Containers for phase0 to fulu, presets
(compile-time `preset-minimal` feature), runtime fork config, and a
`Lean` variant on `ForkName` and `BeaconState` |
| `ethlambda-state-transition` (`beacon/`) | State transition, epoch
processing, fork upgrades, fork choice, BLS (`blst`), KZG (`c-kzg`),
gossip validation rules, DAS custody math |
| `ethlambda-ssz-tree` (**new**) | Persistent Merkle `List`/`Vector` for
the validator registry and balances, so consecutive states share
unchanged subtrees |
| `ethlambda-storage` | One `Store` for both chains. Beacon states are
stored as snapshots plus VCDIFF deltas and written on a background
thread. Adds a committee cache, and `db_version`/`chain`/`preset`
metadata that resume checks |
| `ethlambda-blockchain` | Beacon path in the chain actor
(`BlockChain::spawn_beacon`): import, holding blocks until their parent
or columns arrive, applying gossip aggregates to fork choice, block
production and aggregation |
| `ethlambda-engine` (**new**) | Engine API client with JWT auth:
`newPayload`, `forkchoiceUpdated`, `getPayload`, `getBlobs` |
| `ethlambda-p2p` | Beacon wire: blocks, aggregates, subnet attestations
and data columns are validated before relay;
`beacon_blocks_by_{range,root}/2` served and requested; range sync;
column custody and sampling; attestation subnet backbone; discv5 always
on for `beacon` |
| `ethlambda-rpc` | Beacon API under `/eth/v1` and `/eth/v2`: JSON by
default, SSZ on request. The router is picked by `Store::chain()`, so
`/lean/v0` and the Beacon API are never served together |
| `ethlambda-validator` (**new**) | Validator client: duties,
attestations, aggregation selection, proposals, keymanager API. It has
**no slashing protection** (recorded in `docs/spec_deviations.md`) |
| `bin/ethlambda` | `beacon` and `validator` sub-commands; `--network`
takes a built-in name (`mainnet`, `sepolia`, `hoodi`) or a directory of
published network files; beacon checkpoint sync; `benchmark import`
replays a real block corpus offline; `run_node` is one startup path for
both chains |
| CI / Makefile | One beacon spec-test job per preset, minimal-preset
check, `make test` split into `test-consensus` and `test-node` (one
runner's disk no longer fits both), benchmark-smoke and tooling jobs,
disk-cleanup action |
| Docs | New: `beacon_stf.md`, `beacon_wire.md`, `beacon_engine.md`,
`cli.md`. Updated: `data_storage.md`, `metrics.md`, `rpc.md`,
`checkpoint_sync.md`, `discovery.md`, `benchmarking.md`,
`spec_deviations.md`, README (beacon quickstart with ethrex) |
## Correctness / Behavior Guarantees
- **Separation:** beacon behavior lives in `beacon` modules that lean
code never reads, and those modules never read lean's. The shared crates
dispatch on the chain once per handler.
- **Lean state at a beacon accessor:** this panics, naming the function
it reached (`lean_state_unreachable` / `lean_fork_unreachable`), so the
caller never gets a wrong answer back. `ForkName::Lean` is left out of
`ForkName::ALL`, so neither `parse` nor a fork upgrade can reach it.
- **Spec conformance:** every consensus-specs `v1.6.1` fixture case
passes on both presets: 5705 mainnet and 40009 minimal, including 150
mainnet `fork_choice` cases. Gossip fixtures are pinned to
`v1.7.0-beta.1`.
- **Resume safety:** a data directory written by another chain, preset
or format version is refused at startup, before any state is read.
### ⚠️ Behavior changes for lean (`ethlambda node`)
- **Existing data directories are refused on resume.** A directory
written by `main` has no `db_version` key, so `Store::from_db_state`
returns `DbVersionMismatch { found: 0, expected: 4 }`. There is no
migration: an upgraded node needs a fresh data directory (or checkpoint
sync).
- `blst` and `c-kzg` are now dependencies of the lean binary too, since
the state-transition crate holds both chains' rules. They are not
feature-gated.
- libssz is a git dependency pinned by `rev`: the beacon containers need
`DerefMut`/`IndexMut` on `SszList` (lambdaclass/libssz#33), which no
crates.io release has yet.
- The `release-fast` profile (used for tests) now sets `debug-assertions
= true`.
- `P2P::spawn` now runs before the chain actor is spawned, so gossip
arriving in that short window is dropped.
- `/lean/v0/blocks/finalized` answers JSON when asked for it by name.
SSZ stays the default.
## Tests Added / Run
- **Beacon spec tests:** `make test-beacon` (or `test-beacon-mainnet` /
`test-beacon-minimal`). Each fixture case is a separately named test,
e.g. `cargo test -p ethlambda-state-transition --test beacon_spec_tests
--features beacon-spec-tests -- electra/attester_slashing`. CI runs one
job per preset.
- **`make test`** still needs no consensus-spec download: the beacon
spec target is behind the `beacon-spec-tests` feature, and the BLS/KZG
vector tests report as ignored without it.
- **Minimal preset:** the `Test minimal preset` CI job builds and runs
the `ethlambda-types` unit tests with `preset-minimal` on.
- **Benchmark smoke:** a CI job runs the `benchmark` sub-command end to
end.
- Unit and integration tests in every new crate (`ssz-tree`, `engine`,
`validator`) and in the beacon paths of storage, p2p, rpc and the chain
actor.
## Related Issues / PRs
<details>
<summary>PRs merged into the integration branch (reviewed on
<code>lambdaclass/ethlambda_private</code>)</summary>
**Types and state transition**
- lambdaclass/ethlambda_private#1 Beacon Chain containers under a
`beacon` namespace
- lambdaclass/ethlambda_private#2 Beacon Chain state transition, phase0
through fulu
- lambdaclass/ethlambda_private#37 keep the registry and balances in
persistent Merkle trees
**Follower, sync and storage**
- lambdaclass/ethlambda_private#3 follow mainnet gossip behind an
`ethlambda beacon` sub-command
- lambdaclass/ethlambda_private#6 unify the lean and beacon storage
layer
- lambdaclass/ethlambda_private#7 checkpoint sync for the beacon chain
- lambdaclass/ethlambda_private#8 serve beacon blocks by range and by
root
- lambdaclass/ethlambda_private#9 give the follower a chain, from the
anchor to the tip
- lambdaclass/ethlambda_private#11 fulu data availability sampling,
custody through the availability gate
- lambdaclass/ethlambda_private#13 validate execution payloads over the
Engine API
- lambdaclass/ethlambda_private#15 follow a network described by files
on disk
- lambdaclass/ethlambda_private#26 add sepolia and hoodi as built-in
networks
- lambdaclass/ethlambda_private#48, lambdaclass/ethlambda_private#52,
lambdaclass/ethlambda_private#53, lambdaclass/ethlambda_private#54: held
blocks, their columns and range batches
**P2P**
- lambdaclass/ethlambda_private#18 stop outbound req/resp requests
crossing protocols
- lambdaclass/ethlambda_private#23 rank dial candidates by custody
redundancy
- lambdaclass/ethlambda_private#28 count why peers drop us
- lambdaclass/ethlambda_private#38 validate beacon gossip before
propagating it
- lambdaclass/ethlambda_private#43 run every data column check in p2p
- lambdaclass/ethlambda_private#47 validate beacon aggregates and subnet
attestations before relaying
- lambdaclass/ethlambda_private#19 apply gossip aggregates to fork
choice, backbone two attestation subnets
**API and validator**
- lambdaclass/ethlambda_private#21 serve the Beacon API from `ethlambda
beacon`
- lambdaclass/ethlambda_private#44 report the preset and constants from
`/eth/v1/config/spec`
- lambdaclass/ethlambda_private#16 beacon-chain validator client
- lambdaclass/ethlambda_private#34 serve the validator API
- lambdaclass/ethlambda_private#40 aggregate and propose through
`ethlambda beacon`
- lambdaclass/ethlambda_private#56 pack other nodes' attestations into
produced blocks
**Performance and metrics**
- lambdaclass/ethlambda_private#10, lambdaclass/ethlambda_private#20,
lambdaclass/ethlambda_private#22, lambdaclass/ethlambda_private#24,
lambdaclass/ethlambda_private#33, lambdaclass/ethlambda_private#49:
registry scans, parallel pubkey validation, committee cache, background
state writes, BLS key validation
- lambdaclass/ethlambda_private#17, lambdaclass/ethlambda_private#31:
import timing by phase
- lambdaclass/ethlambda_private#25 benchmark block import by replaying a
real corpus
**CI**
- lambdaclass/ethlambda_private#12, lambdaclass/ethlambda_private#30,
lambdaclass/ethlambda_private#45: split lint/test jobs, benchmark smoke
job, runner selection
</details>
Open PRs based on `beacon-chain-integration`, **not** included here:
lambdaclass#626, lambdaclass#627, lambdaclass#629, lambdaclass#630, lambdaclass#632, lambdaclass#634, lambdaclass#636, lambdaclass#638, lambdaclass#642, lambdaclass#644 to lambdaclass#652,
lambdaclass#656, lambdaclass#658.
## ✅ Verification Checklist
- [ ] Ran `make fmt` (clean)
- [ ] Ran `make lint` (clippy with `-D warnings`, clean)
- [ ] Ran `make test` (`test-consensus` plus `test-node`, at
`release-fast`), all passing
- [ ] Ran `make test-beacon` (both presets), all passing
---------
Co-authored-by: Pablo Deymonnaz <pdeymon@fi.uba.ar>
Co-authored-by: Klaus Lungwitz <klaus.lungwitz@lambdaclass.com>
Co-authored-by: Klaus @ LambdaClass <18153834+klaus993@users.noreply.github.com>
Co-authored-by: Chase Wright <chase.wright@ethereum.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One of three alternative implementations of the same follow-up to the persistent-tree registry (epoch participation without per-index registry reads): (a) this PR, (b) #632, (c) #633. They are meant to be compared, and at most one of (a) and (b) should merge; (c) is stacked on (b).
Motivation
The registry is now a persistent Merkle tree, so
state.validator(i)is a tree descent rather than an array index, whilevalidators().iter()walks leaf slices.Epoch steps 1-3 (justification, inactivity updates, rewards) built index
Vecs, then did one descent per participant and one binary search per eligible validator. The pulled-up-tip check in fork choice repeats justification on every imported block, so part of that cost is paid per block, not per epoch.Change: option (a), iterator passes, spec-shaped
Every spec function keeps its shape and signature. Each now walks
validators().iter().enumerate()and indexes the flat participation and inactivity-score lists by position:get_unslashed_participating_indicesis one pass, no descents.get_unslashed_participating_balance, plus crate-internal one-pass versions of the total active balance and the base reward per increment. The publicget_total_active_balanceandget_base_reward_per_incrementare untouched.Vec.get_flag_index_deltasandget_inactivity_penalty_deltasdecide eligibility and participation inline, with the leak flag computed once.process_inactivity_updatesmakes one immutable pass, then one pass over the scores.process_rewards_and_penaltiesis unchanged: it still builds eight deltaVecs and does 8Nget_mut. That is the known remaining cost of this option.Testing
u64::MAX, epochs 0, 1 and 2, and leaking epochs.-D warningsand theethlambda-state-transitionlib tests pass.epoch_processing,rewards,sanity,finality,random,fork_choiceandtransitionpass on both presets.Benchmark (all three options)
ethlambda benchmark import replayon ethlambda-5:cargo build --release, mainnet corpus of 128 blocks (slots 15279073..=15279200, 4 epoch-start blocks), legs interleavedbase a b cfor two rounds. Base against base: 0.996. Every leg imported all 128 blocks.stf, epoch-start block (mean of 4)stfp50, other blocksstfvs base (ratio of totals)get_headp50get_headrises asstffalls, although no option touches the head computation. This is probably contention with the background state writer, as seen in earlier import benchmarks; it is not verified. Block wall time still falls.Base
beacon-chain-integration@c79fabd5, merged into the branch.