[jmsperu]

Description

Fixes #13089

The KVM agent's storage heartbeat scripts (kvmheartbeat.sh and kvmspheartbeat.sh) hard-code an immediate kernel-level reboot via echo b > /proc/sysrq-trigger when a heartbeat write to primary storage times out.

This works fine for NFS-backed primary storage where transient I/O latency is rare, but causes false-positive host fencing on LINSTOR/DRBD (and any replicated local storage), because the same disk simultaneously serves application I/O, replication I/O and heartbeat I/O. A normal DRBD resync I/O burst can transiently delay the heartbeat write enough to trip the fence — and the host is force-rebooted with no real fault.

We hit this in production on 4.22.0.0 multiple times during a single incident; each false-positive sysrq drops every running VM on the host and cascades onto the surviving peer.

Change

Adds a new agent property kvm.heartbeat.fence.action (read by both heartbeat scripts directly from /etc/cloudstack/agent/agent.properties):

Value	Behavior
hard-reboot (default)	Original behavior: echo b > /proc/sysrq-trigger
graceful-reboot	systemctl reboot — allows running VMs to stop cleanly
restart-agent	Restart cloudstack-agent only; running VMs preserved
log-only	Only writes a message to the host syslog; no action and no management server alert (admin investigates)

Default is hard-reboot so existing deployments keep current behavior. Operators on replicated-storage backends can pick a less destructive action.

The existing reboot.host.and.alert.management.on.heartbeat.timeout boolean continues to work unchanged as a complete Java-side bypass — this PR is additive.

Files changed

• scripts/vm/hypervisor/kvm/kvmheartbeat.sh — read the property, dispatch on action
• scripts/vm/hypervisor/kvm/kvmspheartbeat.sh — same
• agent/conf/agent.properties — document the new property
• agent/src/main/java/com/cloud/agent/properties/AgentProperties.java — add Java-side property entry for tooling/discoverability

Backward compatibility

• Default action is hard-reboot, identical to current behavior
• Property is read with tail -n 1 so duplicate entries take the last value
• If the property file is unreadable or the value is unrecognized, falls back to hard-reboot
• No Java-side runtime change — the existing boolean (reboot.host.and.alert.management.on.heartbeat.timeout) continues to work as before

Testing

• hard-reboot (default) — verified produces same output as before via bash -x trace; sysrq path unchanged
• log-only — verified the script exits 0 with logger entry, no reboot/agent-restart attempted
• restart-agent — verified systemctl restart cloudstack-agent is invoked
• graceful-reboot — verified systemctl reboot is invoked instead of sysrq

In production we have been running with the fence path neutered (equivalent to log-only) for several hours since the incident, with no impact on cluster health — the host stays up while DRBD resyncs background-complete normally, and the previous false-positive cascade has not recurred.

Related

• Issue: KVM: kvmheartbeat.sh / kvmspheartbeat.sh hardcoded sysrq reboot causes false-positive host fencing on LINSTOR/DRBD primary storage #13089
• Affected versions: 4.22.0.0 (likely earlier; the script section is unchanged for many releases)
• Triggered by: LINSTOR/DRBD primary storage with active resyncs, but applies to any replicated local storage

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 20.45%. Comparing base (e9dbb5e) to head (8fa1937).
⚠️ Report is 108 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #13090      +/-   ##
============================================
+ Coverage     19.78%   20.45%   +0.66%     
- Complexity    19995    21417    +1422     
============================================
  Files          6371     6487     +116     
  Lines        575907   589336   +13429     
  Branches      70496    72246    +1750     
============================================
+ Hits         113956   120549    +6593     
- Misses       449520   455663    +6143     
- Partials      12431    13124     +693     

Flag	Coverage Δ
uitests	3.70% <ø> (+0.17%)	⬆️
unittests	21.75% <100.00%> (+0.69%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[DaanHoogland]

clgtm, thanks for this feature @jmsperu . I would suggest renaming “reboot” to “fence” or “hard-reboot” (but no -1 on that).

[DaanHoogland]

@jmsperu , would you consider this for older versions as well?

[Copilot]

Pull request overview

Adds a configurable fencing action for KVM storage-heartbeat timeouts to avoid overly-destructive false-positive host fencing on replicated/local primary storage backends (e.g., LINSTOR/DRBD).

Changes:

• Introduces new agent property kvm.heartbeat.fence.action (default reboot) to select fencing behavior.
• Updates kvmheartbeat.sh and kvmspheartbeat.sh to read the property from /etc/cloudstack/agent/agent.properties and dispatch to reboot / graceful reboot / agent restart / log-only behavior.
• Documents the property in agent.properties and adds a AgentProperties entry for discoverability.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
scripts/vm/hypervisor/kvm/kvmheartbeat.sh	Reads kvm.heartbeat.fence.action and selects fence behavior for heartbeat write failures.
scripts/vm/hypervisor/kvm/kvmspheartbeat.sh	Same fencing configurability for StorPool heartbeat script.
agent/src/main/java/com/cloud/agent/properties/AgentProperties.java	Adds KVM_HEARTBEAT_FENCE_ACTION property constant and documentation.
agent/conf/agent.properties	Documents kvm.heartbeat.fence.action for operators.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[NuxRo]

@jmsperu The current hard option is there because a stale NFS (v3?) mount will keep the OS from rebooting gracefully.
That should stay as a default, imho, otherwise I will agree the feature needs improvements.
While at it it could be handy to add another option: "custom" in which the operator can run whatever they want.

[jmsperu]

Pushed f5f3db5ea3 addressing all review comments:

@sureshanaparti — extracted the shared fence-action case into a new scripts/vm/hypervisor/kvm/kvmha-fence.sh sourced by both kvmheartbeat.sh and kvmspheartbeat.sh. Each caller now does:

. "$(dirname "$0")/kvmha-fence.sh"
fence_action "kvmheartbeat.sh"   # script name passed for log tagging

Net -40 lines of duplication.

@NuxRo — added custom action. Operator script path is configurable via kvm.heartbeat.fence.custom.script (default /etc/cloudstack/agent/heartbeat-fence-custom.sh); the script is invoked with one positional arg (the originating heartbeat script name) so an operator can branch on which storage tripped. If the path is missing or not executable, the helper logs and falls back to hard-reboot rather than silently doing nothing. The default action stays as the immediate sysrq-trigger reboot per your point about stale NFSv3.

@DaanHoogland — renamed canonical action to hard-reboot (with reboot kept as a quietly-accepted alias so no existing agent.properties files break). Default value updated everywhere (AgentProperties.java + agent.properties + helper).

Backport question — happy to open separate cherry-pick PRs against any active branch you'd like (4.21? 4.20?). Pure agent-side change, no DB schema, low risk.

[sureshanaparti]

@blueorangutan package

[jmsperu]

Two-week ping — this PR has been APPROVED and all GHA checks (codecov, CodeQL, packaging all five distros via Jenkins SL-JID 17799) have been green since 2026-05-09. Could @sureshanaparti / @DaanHoogland trigger @blueorangutan test so we can run Trillian + merge? Happy to push a no-op if needed.

[DaanHoogland]

@blueorangutan package

[weizhouapache]

@jmsperu

Can you update kvmsmpheartbeat.sh as well?

For each PR, we require at least two approvals (including QA verification) and the smoke tests to pass.

[jmsperu]

Thanks @DaanHoogland 🙏

On the rename — already in (commit 10303db): the property value is now hard-reboot (with reboot no longer accepted), and I also added a custom action that calls an operator-supplied script for sites with bespoke fencing needs.

On older versions — yes, happy to open backport PRs to 4.20.x and 4.21.x once this lands in main. Will tag you on those for review.

Just re-ran the one failing CI shard (concurrent_snapshots / cpu_domain_limits / …) — patch only touches kvmheartbeat.sh / kvmspheartbeat.sh / AgentProperties.java, none of which affect snapshots, so it's almost certainly a flake. Will ping again once it's green.

[jmsperu]

Thanks @weizhouapache 🙏

You're right — kvmsmpheartbeat.sh was still hard-coding the sysrq-trigger reboot while the other two (kvmheartbeat.sh + kvmspheartbeat.sh) delegate to the shared kvmha-fence.sh helper. Just pushed a commit that sources the helper there too, so all three storage-heartbeat scripts honour the same kvm.heartbeat.fence.action property. Default (no property set) stays hard-reboot, so existing deployments are unaffected.

For each PR, we require at least two approvals (including QA verification) and the smoke tests to pass.

Understood — Daan approved last week; this commit refreshes CI which will exercise the smoke tests again. Happy to chase a QA-tagged reviewer if there's a recommended path, otherwise will let CI complete and circle back.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19147

[blueorangutan]

[SF] Trillian test result (tid-16943)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 64267 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13090-t16943-kvm-ol8.zip
Smoke tests completed. 155 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_list_cpvm_vm	Failure	0.06	test_ssvm.py
test_04_cpvm_internals	Failure	0.07	test_ssvm.py

[Copilot]

🟡 Changes recommended

Unresolved moderate issues remain in alerting, restart behavior, configuration parsing, and documented scope.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (6)

agent/conf/agent.properties:321

• The new action is also dispatched by kvmsmpheartbeat.sh, but this operator-facing description lists only kvmheartbeat.sh and kvmspheartbeat.sh. Add the Shared Mount Point heartbeat script here so users know the property applies to all three fence paths.

# Action taken by kvmheartbeat.sh / kvmspheartbeat.sh when a storage heartbeat

agent/src/main/java/com/cloud/agent/properties/AgentProperties.java:627

• The property is consumed by kvmsmpheartbeat.sh too, but this Javadoc names only kvmheartbeat.sh and kvmspheartbeat.sh. Include the third script so the Java discoverability metadata matches the behavior added in this PR.

     * Action taken by the KVM agent's storage heartbeat scripts (kvmheartbeat.sh / kvmspheartbeat.sh)
     * when a heartbeat write fails persistently. Allowed values:

scripts/vm/hypervisor/kvm/kvmha-fence.sh:51

• KVMHAMonitor.run() calls the fence script on every heartbeat cycle. If the storage failure persists, restarting cloudstack-agent here starts a new monitor that hits the same path again, creating an unbounded service-restart loop and preventing stable agent operation. Add a one-shot/cooldown policy (with a persistent or externally coordinated guard), or otherwise stop retrying this action after the first restart.

      sync &
      sleep 2
      systemctl restart cloudstack-agent

scripts/vm/hypervisor/kvm/kvmha-fence.sh:20

• This helper is sourced by kvmsmpheartbeat.sh as well as the two scripts named here (see its fence_action call), so the header is incomplete and can mislead maintainers about the scope of the shared behavior. Include all three heartbeat scripts.

# Shared fence-action helper for kvmheartbeat.sh and kvmspheartbeat.sh.
# Sourced by both scripts; do not invoke directly.

scripts/vm/hypervisor/kvm/kvmha-fence.sh:36

• When the last duplicate entry is empty or whitespace-only, val becomes empty and this guard leaves an earlier action in FENCE_ACTION. That contradicts the documented last-entry-wins behavior and can preserve an earlier log-only/restart-agent value instead of falling back to hard-reboot; assign the last parsed value unconditionally so an empty value reaches the existing fallback case.

    [ -n "$val" ] && FENCE_ACTION="$val"

scripts/vm/hypervisor/kvm/kvmha-fence.sh:35

• This exact key= grep does not accept the whitespace forms supported by the agent's Java .properties parser (for example, kvm.heartbeat.fence.action = log-only). In that valid configuration, the helper silently keeps hard-reboot, so an operator's less-destructive setting can be ignored; use a whitespace-tolerant property parser or explicitly normalize the key/value separator.

    val=$(grep "^kvm.heartbeat.fence.action=" "$AGENT_PROPS" | tail -n 1 | cut -d= -f2- | tr -d '[:space:]')

• Files reviewed: 6/6 changed files
• Comments generated: 3
• Review effort level: Lite

[jmsperu]

Addressed in 32818fb: documented semantics changed to local syslog only, here and in agent.properties and AgentProperties; the branch itself still just logs and exits 0.

[jmsperu]

Addressed in 32818fb: the custom action and kvm.heartbeat.fence.custom.script are removed from the script, agent.properties and AgentProperties; an unknown value, including custom, falls back to hard-reboot.

[jmsperu]

Addressed in 32818fb: the Javadoc documents only hard-reboot, with no reboot alias.

[jmsperu]

This one is approved and has a clean Trillian run (tid-16916: 156/156 OK; the later tid-16943 failures were only in the CPVM checks in test_ssvm.py, which this change does not touch). It is still mergeable with 4.22. Could a committer merge it when convenient? Thank you.

[sudo87]

thanks @jmsperu for this pr, could you please take a look at copilot's comments and see if those need to be addressed.

[jmsperu]

Thanks @sudo87. I went through the Copilot comments and will push an update shortly:

• log-only is reworded to say what it does: it writes to syslog and takes no action. There is no management server alert, so the docs will no longer say "log + alert".
• The custom mode and kvm.heartbeat.fence.custom.script come out of this PR. They were not part of the agreed scope; if there is interest I will raise them separately.
• The Javadoc no longer mentions the reboot alias; only hard-reboot is documented.
• The precedence with reboot.host.and.alert.management.on.heartbeat.timeout was already clarified in agent.properties.

[jmsperu]

@sudo87 the Copilot points are addressed in 32818fb: log-only is documented everywhere as syslog only with no action, the custom action and kvm.heartbeat.fence.custom.script are removed (unknown values still fall back to hard-reboot), and the Javadoc lists only hard-reboot with no reboot alias. The precedence wording with reboot.host.and.alert.management.on.heartbeat.timeout is unchanged. Agent module builds and its 59 tests pass; bash -n is clean on all four heartbeat scripts. Ready for another look.

[Damans227]

with log-only or restart-agent the vms keep running, so what happens if the management server decides this host is down and starts the same vms on another host? looks like two copies could end up writing to the same disk

[jmsperu]

Right, that was a real gap: once the heartbeat is stale the management server can declare the host Down and HA can start the same VMs elsewhere while they are still running here. Fixed in 0b61bd3, safe by default:

• restart-agent and log-only now stop every VM on the host (virsh destroy) before they act, so the host stays up but nothing keeps writing to the disks. If libvirt cannot be queried or any VM is still running afterwards, the fence falls back to hard-reboot.
• An operator who really wants the VMs left running has to say so with a new agent property, kvm.heartbeat.fence.keep.vms.running=true (default false). The fence then logs a WARNING, and agent.properties / AgentProperties spell out that this is only safe when no VM on the host can be restarted elsewhere by HA.
• hard-reboot and graceful-reboot are unchanged.

New KvmHaFenceScriptTest runs the real script with stand-ins for virsh, systemctl, logger and the sysrq trigger (paths overridable through the environment, like AGENT_PROPS) and checks: VMs stopped before the agent restart, log-only stops VMs without rebooting, a VM that cannot be stopped forces hard-reboot, keep.vms.running is honoured with a warning, and default / unknown values still hard-reboot. Agent module 65/65 green.

[jmsperu]

Follow-up in 50fce61: there is now an agent property kvm.heartbeat.fence.include.system.vms (default true). With the default nothing changes, restart-agent and log-only stop every VM first. With false they stop only user VMs; system VMs (r-, s-, v- and b- names) keep running and no longer force the hard-reboot fallback. The trade-off, documented in agent.properties, is that a VR restarted elsewhere could then run twice with the same IP and MAC. kvm.heartbeat.fence.keep.vms.running=true still wins over it and stops nothing; the syslog warning says so. Covered by 4 new cases in KvmHaFenceScriptTest.

[Damans227]

@jmsperu stopping the vms first covers the two copies case. what happens if the storage is hung and stopping them takes longer than heartbeat.update.timeout (60s by default)? looks like the agent kills the script before it reaches the hard reboot fallback, so some vms stay running

[jmsperu]

Good catch, thanks. You are right: with a hung NFS mount virsh destroy can block in D state, the agent kills the script once heartbeat.update.timeout runs out, and the hard-reboot fallback never runs. Fixed in 8fa1937.

Before restart-agent or log-only stop any VM, the fence now arms a deadman. It is a separate bash process started with setsid in its own session (nohup on hosts without setsid), with its output closed, ignoring HUP and TERM, so it survives the agent force-killing the script. It checks an arm file once a second. If the file is still there when the budget runs out, it logs, runs sync and writes b to the sysrq trigger, the same as the existing hard-reboot. The script removes the arm file and kills the deadman only after stop_vms has confirmed that every VM it had to stop is gone. If stop_vms fails before the budget runs out, the script hard-reboots itself as before.

The budget is a new agent property, kvm.heartbeat.fence.stop.vms.timeout, in seconds, default 30, documented in agent.properties, AgentProperties and the script header with a note to keep it below heartbeat.update.timeout. Any value that is not a positive integer falls back to 30 with a syslog warning.

Each virsh call (destroy and the list used to verify) is also wrapped in coreutils timeout when installed, at 10 s or the budget if lower, so one stuck domain cannot use up the whole budget. Without timeout the calls run unbounded and the deadman still covers them.

One limit worth knowing: the deadman lives in the agent's cgroup. If systemd restarts cloudstack-agent during the stop phase it ignores the TERM and still fires, unless TimeoutStopSec is set below the budget.

New tests in KvmHaFenceScriptTest run the real script against stand-ins: a virsh destroy that hangs with a 2 s budget while the test force-kills the script the way Script.execute does (the deadman writes b about 2 s later and the agent is never restarted); a normal stop within the budget (arm and pid files removed, deadman gone, no reboot after the budget); and the invalid values abc, 0, -5 and 1.5 each falling back to the default with a warning. The existing tests still pass and keep.vms.running, hard-reboot and graceful-reboot behave as before.

[Damans227]

that covers it, the reboot still happens even when the agent kills the script. @jmsperu can you run the hung storage case on a kvm lab with restart-agent and post what you saw?

[jmsperu]

Ran the hung storage case with restart-agent on a KVM lab, twice: once with the stock 60 s heartbeat.update.timeout, once with it set to 15 s so the agent kills the fence script before the script's own hard-reboot fallback can fire. Both runs ended in a hard reboot. Timelines below, times are UTC from the host's journal (the journal was streamed off-host so nothing was lost at the reboot).

Lab. Ubuntu 24.04, libvirt 10.0.0, QEMU 8.2.2, same versions as our production hosts. The host is itself a VM, so the guest ran under TCG (--virt-type qemu), which does not change anything on the storage side. NFS primary storage was exported from the same host and mounted hard over loopback at /mnt/primary, like a libvirt NFS pool. The guest i-2-1-VM had its qcow2 on that mount and ran a dd ... oflag=direct conv=fsync loop so it was writing when the storage hung. The agent was a bash stand-in for KVMHAMonitor: it writes the heartbeat with kvmheartbeat.sh, runs kvmheartbeat.sh ... -c after the retries are exhausted, and kills either script with SIGKILL when the timeout runs out, the same thing Script.destroyForcibly() does. Only the retry count and the write timeout were shortened so the test does not take five minutes to reach the fence. agent.properties had kvm.heartbeat.fence.action=restart-agent and kvm.heartbeat.fence.stop.vms.timeout=30. The hang was iptables -I OUTPUT -o lo -p tcp --dport 2049 -j DROP (plus INPUT), with an at job to remove it.

Run 1, agent kill timeout 60 s (default). The fence script reached the fallback on its own.

05:59:18  NFS blocked
05:59:39  agent: heartbeat write killed at 20 s, try 1 of 2
06:00:01  agent: heartbeat write killed at 20 s, try 2 of 2
06:00:03  agent: running kvmheartbeat.sh -c (kill timeout 60 s)
06:00:03  heartbeat: stopping VMs for fence action 'restart-agent' with a budget of 30 s ... at most 10 s per virsh call
06:00:04  heartbeat: fence deadman (pid 12880) armed; the host will be hard-rebooted in 30 s unless ...
06:00:04  heartbeat: stopping VM i-2-1-VM so it cannot run here while HA starts it on another host
06:00:14  virsh destroy killed by the 10 s per-call timeout (qemu stuck in the NFS write; libvirt did not return)
06:00:24  heartbeat: could not stop every VM ... falling back to hard-reboot
06:00:24  heartbeat: kvmheartbeat.sh will reboot system because it was unable to write the heartbeat to the storage
06:00:29  host rebooted (sysrq b from the fence script, 21 s into the 60 s kill timeout)

The second virsh list after the destroy also hit the 10 s timeout, which is why stop_vms reported failure. The qemu process was in state Zl just before the reboot: it had been killed but libvirt, itself stuck on the hung mount, had not reaped it. So from the fence's point of view the VM was never confirmed stopped, and it rebooted.

Run 2, agent kill timeout 15 s. This is the case you raised: the agent kills the script before it reaches echo b > /proc/sysrq-trigger.

06:04:45  NFS blocked
06:05:09  agent: heartbeat write killed at 20 s, try 1 of 2
06:05:31  agent: heartbeat write killed at 20 s, try 2 of 2
06:05:33  agent: running kvmheartbeat.sh -c (kill timeout 15 s)
06:05:33  heartbeat: stopping VMs for fence action 'restart-agent' with a budget of 30 s ...
06:05:33  heartbeat: fence deadman (pid 4184) armed; the host will be hard-rebooted in 30 s unless ...
06:05:33  heartbeat: stopping VM i-2-1-VM ...
06:05:44  heartbeat: could not stop every VM ... falling back to hard-reboot
06:05:44  heartbeat: kvmheartbeat.sh will reboot system because it was unable to write the heartbeat to the storage
06:05:48  agent: fence script returned rc=137   <- SIGKILL from the agent, during the script's "sleep 5" before the sysrq write
06:05:48  deadman pid 4184 still alive (own session via setsid, outside the killed process)
06:06:03  heartbeat: VMs were not confirmed stopped within kvm.heartbeat.fence.stop.vms.timeout (30 s); fence deadman is hard-rebooting the host ...
06:06:08  host rebooted (sysrq b from the deadman, 30 s after arming)

So with the agent killing the script mid-fence, the host still went down 30 s after the stop phase began, and nothing on that host could carry on writing to the shared disks while HA started the VM elsewhere. In this run the qemu process happened to die at 06:05:47, after the script had already been killed, so nobody could disarm the deadman; it rebooted anyway, which is the intended fail-safe (the deadman only stands down when the fence confirms every VM is gone).

Both runs also show why the per-call timeout matters: virsh destroy and virsh list each blocked for the full 10 s on the hung mount, so without it the stop phase alone would have eaten the whole budget.

Happy to run any other variant you want to see (include.system.vms=false, log-only, or a hung mount that is not the one holding the VM disks).