Skip to content

fix(admin): keep accepted heartbeats when restoring instances from the database (#7763) - #7780

Merged
Aias00 merged 1 commit into
apache:masterfrom
konlue:fix/instance-restore-no-clobber
Oct 25, 2026
Merged

Aias00 merged 1 commit into
apache:masterfrom
konlue:fix/instance-restore-no-clobber

Conversation

@konlue

@konlue konlue commented Oct 17, 2026

Copy link
Copy Markdown
Contributor

Fixes #7763.

Summary

InstanceCheckService#fetchInstanceData (startup / master election) unconditionally overwrites the live map with persisted observations, racing handleBeatInfo on request threads.

Because heartbeat time is not persisted, a restored observation is always older than a heartbeat accepted in the meantime, so the restore can roll a live instance back to a stale snapshot (status and last-heartbeat time included).

Modifications

  • Replace the unconditional put with putIfAbsent: a persisted observation only fills identities that have no live entry.
  • Add regression tests: a heartbeat accepted before the restore keeps its heartbeat time and ONLINE state; identities absent from the map are still restored.

Verifying this change

  • Red-green: testFetchInstanceDataKeepsLiveHeartbeatOverStaleSnapshot fails on the current code (the stale snapshot replaces the live entry) and passes with putIfAbsent; full InstanceCheckServiceTest 14/14 green with no regressions.

  • Targeted tests pass:

    ./mvnw -B -ntp -pl shenyu-admin test -Dtest=InstanceCheckServiceTest

    14/14 green.

  • Checkstyle passes:

    ./mvnw -B -ntp -pl shenyu-admin checkstyle:check

    clean.

Notes

The fix does not depend on heartbeat-time persistence (#7723): even once that lands, "do not clobber a live entry" remains the correct ordering rule. Local entries continue to be aged out by doCheck, so nothing can be frozen by skipping the overwrite.

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — the ordering rule is right, and the failure mode is worse than the PR body suggests.

Evidence I checked

  • fetchInstanceData() has exactly one caller chain: setup() (InstanceCheckService.java:86), invoked from ShenyuStandaloneService / ShenyuClusterService. It is a start-up / master-election path, not a periodic refresh, so nothing else depends on the overwrite to pick up DB changes.
  • syncDB() (:176-180) pushes the whole live map into the DB every 10s: the table is a mirror of the map, not an independent source of truth. A restored row therefore cannot be newer than a live entry, which makes putIfAbsent the correct and cheapest rule. A merge on heartbeat time would be strictly better, but the time is not persisted, so it is not available — your reasoning holds.
  • The blast radius deserves one more line in the comment: a restored InstanceInfoVO carries no heartbeat time, so doCheck() (:151-174) marks it OFFLINE and then DELETED within one cycle, and the next syncDB() persists that DELETED state. In other words the old code could tombstone a live gateway in the database, not just show a stale status.

Non-blocking

  1. If fetchInstanceData() ever becomes periodic, this rule must be revisited. A sentence in the method javadoc would prevent that trap later.
  2. #7776 adds a guard to handleBeatInfo in the same file, ~20 lines below. Both hunks are independent, so whichever lands second just needs a rebase.
  3. In the regression test the load-bearing assertion is the state (offline 0 from the DB row vs ONLINE from the live entry). The timestamp assertion can pass by coincidence on master, because buildVO() already sets lastHeartBeatTime to System.currentTimeMillis() and handleBeatInfo runs in the same millisecond. Fine as is, just don't rely on it alone.

CI note (not a blocker): pr_build is green. The two red e2e-case jobs are unrelated — the rocketmq case fails on Failed to consume RocketMQ access log (DividePluginTest) and the apache-dubbo case ends with ##[error]The operation was canceled. after repeated Failed to send request .../actuator/health. Neither touches the admin instance map; please rerun the e2e workflow when convenient.

@Aias00

Aias00 commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Ping on state: this PR is approved on the code, but the required checks are red and it cannot be merged yet.

The failing jobs are e2e / e2e-case / list-contract / native-sql, which are currently failing across the repository and are tracked in #7970. The build job in this run does not fail on compilation: it stops at the gate step that consumes the e2e result ([[ "failure" == "success" ]] || exit -1), so there is nothing in this PR to fix for that.

No code change is expected from you. Once the shared e2e problem is resolved, please rerun the failed jobs (or push a trivial rebase) so the required checks go green, and this can be merged.

@Aias00

Aias00 commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

@konlue This PR is approved, but 3 checks are red: e2e, e2e-case (e2e-logging-rocketmq-compose, shenyu-e2e-case-logging-rocketmq, :shenyu-examples-http), e2e-case (e2e-apache-dubbo-sync-compose, shenyu-e2e-case-apache-dubbo, :shenyu-examples-apache-du....

I inspected these lanes today and they fail for infrastructure reasons rather than on the diff — the integrated-test jobs die while starting their containers (Elasticsearch /_cat/health returning 000 until the job is cancelled), and the http lane also hits the known CachePluginTest.testMemoryModle / testRedisModle flake. The same jobs are red on master and on unrelated PRs right now.

Please rebase onto master and push (or ask for a re-run) so the checks execute again. If a case is still failing on a fresh run, paste the test name and I'll look properly.

@konlue
konlue force-pushed the fix/instance-restore-no-clobber branch from 0a9e0d7 to ea1e6cd Compare October 24, 2026 16:05
@konlue

konlue commented Oct 24, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto the latest master to re-trigger the checks

@Aias00
Aias00 merged commit 745df15 into apache:master Oct 25, 2026
55 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Startup instance restore overwrites a newer accepted heartbeat

2 participants