Skip to content

[BUG] Startup instance restore overwrites a newer accepted heartbeat #7763

Closed

Description

@Aias00

Problem

InstanceCheckService.fetchInstanceData() reads persisted observations and unconditionally puts each object into the live map. Heartbeats can be handled while database loading is in progress. If a fresh heartbeat arrives after the snapshot was read but before its entry is installed, the old snapshot replaces the freshly accepted observation, including its status and last-heartbeat time.

This is distinct from missing heartbeat persistence (#7723): the race also reproduces with an explicitly nonzero old stored timestamp. It is not fixed solely by refreshing metadata on repeated reports (#7692).

Verified reproduction

Baseline 66d7d954ca37d651a5ad916f988a8081157a7ab4, Java 17; actual current checker class recompiled from source.

  1. A controlled persistence service reads an old OFFLINE observation and pauses before returning it.
  2. Call the real handleBeatInfo for the same identity; it creates a fresh ONLINE observation with a later timestamp.
  3. Release the database read and let the real fetchInstanceData finish.
  4. Assert that the live map now contains the old OFFLINE object with a timestamp earlier than the already accepted heartbeat.

The latch-controlled probe passed and all fixture-owned threads were terminated. It tests method interleaving, not a deployed startup or master election.

Evidence

Acceptance criteria

  • Preserve a live observation accepted after the database snapshot when restoring the same identity; define freshness/order explicitly.
  • Add a deterministic latch/barrier regression for restore concurrent with new and existing identity heartbeats.
  • Test older/missing stored timestamps, namespace separation and master handover without manufacturing a fresh timestamp for unobserved instances.
  • Coordinate restore and publication with shutdown/lifecycle semantics; avoid changing normal metadata-refresh behavior incidentally.
  • Retain needed persisted entries that have not reported locally while keeping a documented authoritative status view.

Related: #7723, #7692, #7697 and #7718.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions