Repository navigation
fix(admin): keep accepted heartbeats when restoring instances from the database (#7763) - #7780
Conversation
Aias00
left a comment
There was a problem hiding this comment.
LGTM — the ordering rule is right, and the failure mode is worse than the PR body suggests.
Evidence I checked
fetchInstanceData()has exactly one caller chain:setup()(InstanceCheckService.java:86), invoked fromShenyuStandaloneService/ShenyuClusterService. It is a start-up / master-election path, not a periodic refresh, so nothing else depends on the overwrite to pick up DB changes.syncDB()(:176-180) pushes the whole live map into the DB every 10s: the table is a mirror of the map, not an independent source of truth. A restored row therefore cannot be newer than a live entry, which makesputIfAbsentthe correct and cheapest rule. Amergeon heartbeat time would be strictly better, but the time is not persisted, so it is not available — your reasoning holds.- The blast radius deserves one more line in the comment: a restored
InstanceInfoVOcarries no heartbeat time, sodoCheck()(:151-174) marks itOFFLINEand thenDELETEDwithin one cycle, and the nextsyncDB()persists thatDELETEDstate. In other words the old code could tombstone a live gateway in the database, not just show a stale status.
Non-blocking
- If
fetchInstanceData()ever becomes periodic, this rule must be revisited. A sentence in the method javadoc would prevent that trap later. - #7776 adds a guard to
handleBeatInfoin the same file, ~20 lines below. Both hunks are independent, so whichever lands second just needs a rebase. - In the regression test the load-bearing assertion is the state (offline
0from the DB row vsONLINEfrom the live entry). The timestamp assertion can pass by coincidence on master, becausebuildVO()already setslastHeartBeatTimetoSystem.currentTimeMillis()andhandleBeatInforuns in the same millisecond. Fine as is, just don't rely on it alone.
CI note (not a blocker): pr_build is green. The two red e2e-case jobs are unrelated — the rocketmq case fails on Failed to consume RocketMQ access log (DividePluginTest) and the apache-dubbo case ends with ##[error]The operation was canceled. after repeated Failed to send request .../actuator/health. Neither touches the admin instance map; please rerun the e2e workflow when convenient.
|
Ping on state: this PR is approved on the code, but the required checks are red and it cannot be merged yet. The failing jobs are No code change is expected from you. Once the shared e2e problem is resolved, please rerun the failed jobs (or push a trivial rebase) so the required checks go green, and this can be merged. |
|
@konlue This PR is approved, but 3 checks are red: I inspected these lanes today and they fail for infrastructure reasons rather than on the diff — the Please rebase onto |
0a9e0d7 to
ea1e6cd
Compare
|
Rebased onto the latest master to re-trigger the checks |
Fixes #7763.
Summary
InstanceCheckService#fetchInstanceData(startup / master election) unconditionally overwrites the live map with persisted observations, racinghandleBeatInfoon request threads.Because heartbeat time is not persisted, a restored observation is always older than a heartbeat accepted in the meantime, so the restore can roll a live instance back to a stale snapshot (status and last-heartbeat time included).
Modifications
putwithputIfAbsent: a persisted observation only fills identities that have no live entry.Verifying this change
Red-green:
testFetchInstanceDataKeepsLiveHeartbeatOverStaleSnapshotfails on the current code (the stale snapshot replaces the live entry) and passes withputIfAbsent; fullInstanceCheckServiceTest14/14 green with no regressions.Targeted tests pass:
./mvnw -B -ntp -pl shenyu-admin test -Dtest=InstanceCheckServiceTest14/14 green.
Checkstyle passes:
clean.
Notes
The fix does not depend on heartbeat-time persistence (#7723): even once that lands, "do not clobber a live entry" remains the correct ordering rule. Local entries continue to be aged out by
doCheck, so nothing can be frozen by skipping the overwrite.