bug: VM config preflight fails when e2fsprogs is transiently busy #4314

User Story

I use OpenShell’s CI test suite to validate changes before submission. I directly encountered an intermittent failure when a newly created e2fsprogs test executable was reported as temporarily busy during the VM host-tool preflight tests.

I need preflight to tolerate this short-lived Linux condition so that CI results reliably distinguish real e2fsprogs failures from transient executable availability.

Problem Statement

On Linux, openshell-gateway config preflight can fail when an e2fsprogs executable such as mke2fs, debugfs, or e2fsck is temporarily busy during or shortly after host provisioning.

The executable becomes available shortly afterward, but OpenShell reports the transient failure immediately and treats the host-tool check as unsuccessful.

Impact / Why This Matters

This can cause otherwise valid local VM deployments to fail automated configuration validation intermittently. Users must rerun preflight manually or add external delays to provisioning workflows.

Those workarounds make deployment automation less reliable and can make a healthy e2fsprogs installation appear broken.

It also results in occasional test failures.

Acceptance Criteria

• VM host-tool preflight tolerates a short-lived executable-busy condition.
• Preflight succeeds if the executable becomes available within a bounded interval.
• A persistently busy executable still produces a clear error.
• Unrelated execution errors continue to fail without unnecessary delay.
• Interrupting preflight during the bounded wait cancels it promptly.
• Automated coverage verifies the transient Linux behavior.

Reproduction Steps

1. Configure an OpenShell gateway to use the local VM compute driver on Linux.

2. Provision or update the required e2fsprogs executables as part of the host setup.

3. Immediately run:

   openshell-gateway config preflight -- --config /path/to/gateway.toml --compute-driver vm
   

4. If Linux reports an executable as temporarily busy, observe that OpenShell fails preflight immediately.

5. Run the same command again after the transient condition clears and observe that it succeeds without any configuration or installation changes.

OR

Repeatedly run mise run ci and observe occasional failures:

[test:rust] failures:
[test:rust]
[test:rust] ---- e2fsprogs::tests::filesystem_preflight_bounds_each_output_stream stdout ----
[test:rust]
[test:rust] thread 'e2fsprogs::tests::filesystem_preflight_bounds_each_output_stream' (821653) panicked at crates/openshell-core/src/e2fsprogs.rs:493:13:
[test:rust] assertion failed: error.contains(&format!("{stream} exceeded 8192 bytes"))
[test:rust] note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[test:rust]
[test:rust] ---- e2fsprogs::tests::filesystem_preflight_retains_exited_leader_until_held_pipe_cleanup stdout ----
[test:rust]
[test:rust] thread 'e2fsprogs::tests::filesystem_preflight_retains_exited_leader_until_held_pipe_cleanup' (821661) panicked at crates/openshell-core/src/e2fsprogs.rs:540:9:
[test:rust] run /tmp/.tmpRAn848/mke2fs -V: Text file busy (os error 26). Install e2fsprogs 1.43 or newer, or repair the selected installation and the gateway service PATH; rerun config preflight with the service account and environment
[test:rust]
[test:rust] ---- e2fsprogs::tests::filesystem_preflight_retains_missing_interpreter_error stdout ----
[test:rust]
[test:rust] thread 'e2fsprogs::tests::filesystem_preflight_retains_missing_interpreter_error' (821662) panicked at crates/openshell-core/src/e2fsprogs.rs:417:9:
[test:rust] run /tmp/.tmpOKzN5L/mke2fs -V: Text file busy (os error 26). Install e2fsprogs 1.43 or newer, or repair the selected installation and the gateway service PATH; rerun config preflight with the service account and environment
[test:rust]
[test:rust]
[test:rust] failures:
[test:rust]     e2fsprogs::tests::filesystem_preflight_bounds_each_output_stream
[test:rust]     e2fsprogs::tests::filesystem_preflight_retains_exited_leader_until_held_pipe_cleanup
[test:rust]     e2fsprogs::tests::filesystem_preflight_retains_missing_interpreter_error
[test:rust]
[test:rust] test result: FAILED. 540 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.33s

Suggested UX (if applicable)

N/A

Environment

• openshell 0.1.4-dev.3+g29c9c53c2
• fedora 43

Logs