Skip to content

fix(beacon): verify a block's signature before it waits for its parent or columns - #629

Draft
MegaRedHand wants to merge 7 commits into
mainfrom
fix/beacon-verify-block-signature-before-waiting
Draft

MegaRedHand wants to merge 7 commits into
mainfrom
fix/beacon-verify-block-signature-before-waiting

Conversation

@MegaRedHand

@MegaRedHand MegaRedHand commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator

A beacon block that waits, for its parent's post-state and then for its custody columns, is kept unjudged until what it waits for arrives. A held one also has its columns asked for again every slot until finality evicts it. Mainnet gossip carries altered copies of real blocks (a blob transaction re-encoded after signing, the original signature kept): nobody signed their root, and no peer has columns for them.

An earlier change moved the signature check for a parentless block into gossip, but that check has two gaps:

  • It reads the head state only through Store::cached_state, and queues the block unchecked when it finds none. After a resume nothing caches the head state before the first import.
  • Range sync and by-root fetches never pass through gossip validation at all.

What it cost

eth-5, keep-DB restart, 2026-09-25 19:41Z. Slot 15295105 had two gossip blocks from proposer 2128692, 2 ms apart: the altered copy 9b320e3a (917 KB, first) and the real b4a78feb (367 KB). The parent was held for columns, so gossip returned Queue(ParentNotReady), and with no head state cached yet the altered copy went to the chain actor unchecked. It was held for columns and logged Data column fetch failed after max retries 13 times, once per slot.

In the first ~45 s after that restart Prometheus shows 3 queue/parent_not_ready, 0 accept, and no reject/bad_signature until the first import.

Change

Every path a beacon block takes into waiting now requires a verified proposer signature first.

gossip beacon_block, parent has no post-state
  │
  ├─ head state cached?  no ─────────────────────▶ Ignore(signature_unverified)   [was: Queue]
  └─ precheck_block(Recent(head))
       ├─ Ok ────────────────────────────────────▶ Queue ─▶ chain actor
       ├─ BadSignature ──────────────────────────▶ Reject
       └─ UnknownProposer ───────────────────────▶ Ignore(signature_unverified)   [was: Queue]

chain actor (gossip, range sync, by-root)
  ├─ parent has no post-state ─▶ precheck_block(Recent(head))   ─▶ Err: refuse, drop parked subtree
  └─ columns missing          ─▶ precheck_block(Parent(parent)) ─▶ Err: refuse
Where Before After
gossip::block::queue_if_signed (was queue_unless_forged) Queue unless the signature is shown forged Queue only once it verifies; otherwise Ignore(SignatureUnverified) (new IgnoreReason)
precheck_block, Reference::Recent, proposer not in state Ok(()) Err(UnknownProposer), so Ok always means the signature verified
Validated::forward a block is forwarded on every outcome forwarded only on Accept / Queue, so an Overloaded block is dropped too
Chain actor, before parking for a parent no check precheck_before_waiting(Wait::Parent): signature against the head's post-state; on failure also discard_pending_subtree
Chain actor, before holding for columns no check precheck_before_waiting(Wait::Columns): every precheck rule against the parent's post-state; returns ImportError::Precheck
Resume (fetch_initial_beacon_state, both resuming returns) head state not cached Store::cache_head_state()

A fresh checkpoint sync already caches its anchor through insert_state, so the resume path was the only one missing it.

New metric: lean_beacon_blocks_refused_before_waiting_total{reason, wait}, where reason is the PrecheckError label and wait is parent or columns.

Docs (separate commit): beacon_wire.md and metrics.md no longer say a block is forwarded to the chain actor on every outcome or when Overloaded. They also now list the signature_unverified ignore reason and the new counter.

Trade-offs

  • A real block can now be dropped from gossip: when no head state is cached, when its proposer is newer than the head state, or when the validation pool is overloaded. It still arrives through a child's by-root fetch or range sync once its parent has imported. The code comment on spawn_stateful_checks notes that no block was measured Overloaded on the mainnet followers over the 24 hours before this change.
  • One BLS verification per waiting block in the chain actor, on top of the one in its eventual state transition. This applies only to blocks that wait, not to blocks imported straight away.

Tests

Check Result
New: an_unknown_parent_by_a_proposer_the_head_state_lacks_is_dropped passes
New: only_an_accepted_or_queued_block_is_forwarded passes
New: a_resumed_store_caches_its_head_state_on_request passes
Updated: a_block_whose_parent_was_never_seen_is_dropped_with_no_state_to_check_it (was ..._is_queued) passes
Updated: a_proposer_missing_from_the_reference_state_is_unknown_to_both_references passes
--lib: blockchain 140 passed, 2 failed (see below)
--lib --bins: state-transition, p2p, storage, ethlambda pass
make lint, cargo fmt --check clean

Not yet run on a follower.

Known failures (why this is a draft)

Test Failure Cause
releasing_once_every_custody_column_arrives_clears_the_hold panic at precheck_before_waiting: "the reference block has a post-state" The release goes through on_block, and the fixture's parent (ZERO) has no state, so the parent-wait precheck runs. The head (ZERO) has no state either, so the expect panics.
a_childs_fan_out_does_not_livelock_a_held_parent blocks_awaiting_columns.contains_key(&parent_root) is false fulu_block is unsigned and bare_state() has no validators, so the columns-wait precheck refuses the block with UnknownProposer instead of holding it.

Neither test is wrong: they hold unsigned blocks, and holding now requires a valid signature. The signing helpers (secret_key_for, with_validators_at) are #[cfg(test)] pub(crate) in ethlambda-state-transition, so the blockchain crate cannot reach them. Options:

  • A. Expose test_state behind a test-utils feature and give these tests keyed states and signed blocks.
  • B. Add a test-only switch that skips precheck_before_waiting. This is smaller, but these tests would then stop covering the precheck path.
  • C. Separately: decide whether a missing reference state should refuse the block instead of panicking.

Base

  • Based on beacon-chain-integration @ c79fabd5, merged into the branch. One conflict, in verdict.rs tests: both sides added a test at the same spot, so both are kept.

@MegaRedHand MegaRedHand added the beacon Ethereum Beacon Chain client label Oct 1, 2026
@MegaRedHand MegaRedHand mentioned this pull request Oct 5, 2026
4 tasks
Base automatically changed from beacon-chain-integration to main October 20, 2026 16:19
…t or columns

A beacon block that waits, for its parent's post-state and then for its
custody columns, is kept unjudged until what it waits for arrives, and a
held one has its columns asked for again every slot until finality evicts
it. Mainnet gossip carries altered copies of real blocks (a blob
transaction re-encoded after signing, the original signature kept): nobody
signed their root and no peer has columns for them.

Gossip's queue path checked the signature only against a cached head
state, and queued the block anyway when it found none. After a resume
nothing caches the head state before the first import, so in that window
every parentless block reached the chain actor unchecked, and an altered
one sat held for columns until finality. Range sync and by-root fetches
never pass through gossip validation at all.

- Gossip queues a parentless block only once the head state verifies its
  signature. One it cannot judge is ignored as `signature_unverified`, and
  only an accepted or queued block is forwarded, so an overloaded one no
  longer reaches the actor unjudged.
- `precheck_block` against a recent state reports an unknown proposer
  instead of passing it, so `Ok` always means the signature verified.
- The chain actor runs `precheck_block` before parking a block for its
  parent (signature, against the head state) and before holding it for
  columns (every rule, against the parent's post-state).
- Resuming from the DB caches the head's state, so gossip can judge a
  parentless block from the first message on.
The previous commit stopped forwarding a block gossip could not verify the
signature of, `Overloaded` included, so the "still forwarded" and "either
way" passages no longer hold. It also added an ignore reason and a counter
nothing documented yet.
@MegaRedHand
MegaRedHand force-pushed the fix/beacon-verify-block-signature-before-waiting branch from 90ca98d to 15475f2 Compare October 20, 2026 17:32
zexoverz pushed a commit to zexoverz/ethlambda that referenced this pull request Oct 22, 2026
## 🗒️ Description / Motivation

Merges `beacon-chain-integration` into `main`. This adds an **Ethereum
Beacon Chain** client (phase0 through fulu) to the ethlambda binary,
alongside the Lean consensus client:

- **`ethlambda beacon`**: a beacon node. It checkpoint-syncs, follows
the chain over gossip and req/resp, validates execution payloads through
an execution client's Engine API, and serves the Beacon API, including
the validator endpoints.
- **`ethlambda validator`**: a beacon validator client (attest,
aggregate, propose) that talks to any beacon node over the standard REST
API.
- **`ethlambda node`** (lean) keeps its behavior, apart from the items
under "Behavior changes for lean" below.

Both chains run on the same storage layer, P2P actor, RPC crate and
chain actor. `BeaconState` has a `Lean` variant holding lean's `State`,
and each shared handler dispatches on it once, at the top.

Each piece was reviewed as its own PR into the integration branch
(listed under "Related Issues / PRs"). This PR merges that branch.

## What Changed

| Area | Change |
|---|---|
| `ethlambda-types` (`beacon/`) | Containers for phase0 to fulu, presets
(compile-time `preset-minimal` feature), runtime fork config, and a
`Lean` variant on `ForkName` and `BeaconState` |
| `ethlambda-state-transition` (`beacon/`) | State transition, epoch
processing, fork upgrades, fork choice, BLS (`blst`), KZG (`c-kzg`),
gossip validation rules, DAS custody math |
| `ethlambda-ssz-tree` (**new**) | Persistent Merkle `List`/`Vector` for
the validator registry and balances, so consecutive states share
unchanged subtrees |
| `ethlambda-storage` | One `Store` for both chains. Beacon states are
stored as snapshots plus VCDIFF deltas and written on a background
thread. Adds a committee cache, and `db_version`/`chain`/`preset`
metadata that resume checks |
| `ethlambda-blockchain` | Beacon path in the chain actor
(`BlockChain::spawn_beacon`): import, holding blocks until their parent
or columns arrive, applying gossip aggregates to fork choice, block
production and aggregation |
| `ethlambda-engine` (**new**) | Engine API client with JWT auth:
`newPayload`, `forkchoiceUpdated`, `getPayload`, `getBlobs` |
| `ethlambda-p2p` | Beacon wire: blocks, aggregates, subnet attestations
and data columns are validated before relay;
`beacon_blocks_by_{range,root}/2` served and requested; range sync;
column custody and sampling; attestation subnet backbone; discv5 always
on for `beacon` |
| `ethlambda-rpc` | Beacon API under `/eth/v1` and `/eth/v2`: JSON by
default, SSZ on request. The router is picked by `Store::chain()`, so
`/lean/v0` and the Beacon API are never served together |
| `ethlambda-validator` (**new**) | Validator client: duties,
attestations, aggregation selection, proposals, keymanager API. It has
**no slashing protection** (recorded in `docs/spec_deviations.md`) |
| `bin/ethlambda` | `beacon` and `validator` sub-commands; `--network`
takes a built-in name (`mainnet`, `sepolia`, `hoodi`) or a directory of
published network files; beacon checkpoint sync; `benchmark import`
replays a real block corpus offline; `run_node` is one startup path for
both chains |
| CI / Makefile | One beacon spec-test job per preset, minimal-preset
check, `make test` split into `test-consensus` and `test-node` (one
runner's disk no longer fits both), benchmark-smoke and tooling jobs,
disk-cleanup action |
| Docs | New: `beacon_stf.md`, `beacon_wire.md`, `beacon_engine.md`,
`cli.md`. Updated: `data_storage.md`, `metrics.md`, `rpc.md`,
`checkpoint_sync.md`, `discovery.md`, `benchmarking.md`,
`spec_deviations.md`, README (beacon quickstart with ethrex) |

## Correctness / Behavior Guarantees

- **Separation:** beacon behavior lives in `beacon` modules that lean
code never reads, and those modules never read lean's. The shared crates
dispatch on the chain once per handler.
- **Lean state at a beacon accessor:** this panics, naming the function
it reached (`lean_state_unreachable` / `lean_fork_unreachable`), so the
caller never gets a wrong answer back. `ForkName::Lean` is left out of
`ForkName::ALL`, so neither `parse` nor a fork upgrade can reach it.
- **Spec conformance:** every consensus-specs `v1.6.1` fixture case
passes on both presets: 5705 mainnet and 40009 minimal, including 150
mainnet `fork_choice` cases. Gossip fixtures are pinned to
`v1.7.0-beta.1`.
- **Resume safety:** a data directory written by another chain, preset
or format version is refused at startup, before any state is read.

### ⚠️ Behavior changes for lean (`ethlambda node`)

- **Existing data directories are refused on resume.** A directory
written by `main` has no `db_version` key, so `Store::from_db_state`
returns `DbVersionMismatch { found: 0, expected: 4 }`. There is no
migration: an upgraded node needs a fresh data directory (or checkpoint
sync).
- `blst` and `c-kzg` are now dependencies of the lean binary too, since
the state-transition crate holds both chains' rules. They are not
feature-gated.
- libssz is a git dependency pinned by `rev`: the beacon containers need
`DerefMut`/`IndexMut` on `SszList` (lambdaclass/libssz#33), which no
crates.io release has yet.
- The `release-fast` profile (used for tests) now sets `debug-assertions
= true`.
- `P2P::spawn` now runs before the chain actor is spawned, so gossip
arriving in that short window is dropped.
- `/lean/v0/blocks/finalized` answers JSON when asked for it by name.
SSZ stays the default.

## Tests Added / Run

- **Beacon spec tests:** `make test-beacon` (or `test-beacon-mainnet` /
`test-beacon-minimal`). Each fixture case is a separately named test,
e.g. `cargo test -p ethlambda-state-transition --test beacon_spec_tests
--features beacon-spec-tests -- electra/attester_slashing`. CI runs one
job per preset.
- **`make test`** still needs no consensus-spec download: the beacon
spec target is behind the `beacon-spec-tests` feature, and the BLS/KZG
vector tests report as ignored without it.
- **Minimal preset:** the `Test minimal preset` CI job builds and runs
the `ethlambda-types` unit tests with `preset-minimal` on.
- **Benchmark smoke:** a CI job runs the `benchmark` sub-command end to
end.
- Unit and integration tests in every new crate (`ssz-tree`, `engine`,
`validator`) and in the beacon paths of storage, p2p, rpc and the chain
actor.

## Related Issues / PRs

<details>
<summary>PRs merged into the integration branch (reviewed on
<code>lambdaclass/ethlambda_private</code>)</summary>

**Types and state transition**
- lambdaclass/ethlambda_private#1 Beacon Chain containers under a
`beacon` namespace
- lambdaclass/ethlambda_private#2 Beacon Chain state transition, phase0
through fulu
- lambdaclass/ethlambda_private#37 keep the registry and balances in
persistent Merkle trees

**Follower, sync and storage**
- lambdaclass/ethlambda_private#3 follow mainnet gossip behind an
`ethlambda beacon` sub-command
- lambdaclass/ethlambda_private#6 unify the lean and beacon storage
layer
- lambdaclass/ethlambda_private#7 checkpoint sync for the beacon chain
- lambdaclass/ethlambda_private#8 serve beacon blocks by range and by
root
- lambdaclass/ethlambda_private#9 give the follower a chain, from the
anchor to the tip
- lambdaclass/ethlambda_private#11 fulu data availability sampling,
custody through the availability gate
- lambdaclass/ethlambda_private#13 validate execution payloads over the
Engine API
- lambdaclass/ethlambda_private#15 follow a network described by files
on disk
- lambdaclass/ethlambda_private#26 add sepolia and hoodi as built-in
networks
- lambdaclass/ethlambda_private#48, lambdaclass/ethlambda_private#52,
lambdaclass/ethlambda_private#53, lambdaclass/ethlambda_private#54: held
blocks, their columns and range batches

**P2P**
- lambdaclass/ethlambda_private#18 stop outbound req/resp requests
crossing protocols
- lambdaclass/ethlambda_private#23 rank dial candidates by custody
redundancy
- lambdaclass/ethlambda_private#28 count why peers drop us
- lambdaclass/ethlambda_private#38 validate beacon gossip before
propagating it
- lambdaclass/ethlambda_private#43 run every data column check in p2p
- lambdaclass/ethlambda_private#47 validate beacon aggregates and subnet
attestations before relaying
- lambdaclass/ethlambda_private#19 apply gossip aggregates to fork
choice, backbone two attestation subnets

**API and validator**
- lambdaclass/ethlambda_private#21 serve the Beacon API from `ethlambda
beacon`
- lambdaclass/ethlambda_private#44 report the preset and constants from
`/eth/v1/config/spec`
- lambdaclass/ethlambda_private#16 beacon-chain validator client
- lambdaclass/ethlambda_private#34 serve the validator API
- lambdaclass/ethlambda_private#40 aggregate and propose through
`ethlambda beacon`
- lambdaclass/ethlambda_private#56 pack other nodes' attestations into
produced blocks

**Performance and metrics**
- lambdaclass/ethlambda_private#10, lambdaclass/ethlambda_private#20,
lambdaclass/ethlambda_private#22, lambdaclass/ethlambda_private#24,
lambdaclass/ethlambda_private#33, lambdaclass/ethlambda_private#49:
registry scans, parallel pubkey validation, committee cache, background
state writes, BLS key validation
- lambdaclass/ethlambda_private#17, lambdaclass/ethlambda_private#31:
import timing by phase
- lambdaclass/ethlambda_private#25 benchmark block import by replaying a
real corpus

**CI**
- lambdaclass/ethlambda_private#12, lambdaclass/ethlambda_private#30,
lambdaclass/ethlambda_private#45: split lint/test jobs, benchmark smoke
job, runner selection

</details>

Open PRs based on `beacon-chain-integration`, **not** included here:
lambdaclass#626, lambdaclass#627, lambdaclass#629, lambdaclass#630, lambdaclass#632, lambdaclass#634, lambdaclass#636, lambdaclass#638, lambdaclass#642, lambdaclass#644 to lambdaclass#652,
lambdaclass#656, lambdaclass#658.

## ✅ Verification Checklist

- [ ] Ran `make fmt` (clean)
- [ ] Ran `make lint` (clippy with `-D warnings`, clean)
- [ ] Ran `make test` (`test-consensus` plus `test-node`, at
`release-fast`), all passing
- [ ] Ran `make test-beacon` (both presets), all passing

---------

Co-authored-by: Pablo Deymonnaz <pdeymon@fi.uba.ar>
Co-authored-by: Klaus Lungwitz <klaus.lungwitz@lambdaclass.com>
Co-authored-by: Klaus @ LambdaClass <18153834+klaus993@users.noreply.github.com>
Co-authored-by: Chase Wright <chase.wright@ethereum.org>
Main's eventstream (#626) added only_an_accepted_gossip_block_is_announced,
written when every gossip block reached the chain actor whatever its
verdict. This branch forwards a block only on Accept or Queue, the
verdicts that mean its signature verified, so the Overloaded block in that
test is now dropped rather than handed on silently.
@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

Ignore this one for now. I'll revisit it later

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

beacon Ethereum Beacon Chain client

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant