Skip to content

Add preflight checks for -d/-b/-m paths and close install/upgrade doc gaps (#122) - #127

Open
cttyler wants to merge 22 commits into
masterfrom
cttyler-2026-10-issue122
Open

cttyler wants to merge 22 commits into
masterfrom
cttyler-2026-10-issue122

Conversation

@cttyler

@cttyler cttyler commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

Implements #122 and fixes several installer/updater bugs found while testing in a BSW Gen7 lab (a p4hms monitoring
hub + 5 p4d servers).

#122: preflight checks and docs

  • New preflight_check_dirs in p4prom_common.sh. Custom -d, -b, and (for install_p4prom.sh) -m
    paths must already exist and pass a real test write, so an unmounted volume fails with a clear error
    instead of silently filling the root disk. Paths left at their defaults are not checked.
  • Wired into install_prom_graf.sh, update_prom_graf.sh, install_p4prom.sh, and update_p4prom.sh.
  • doc/P4Prometheus_Installation.adoc (HTML/PDF regenerated):
    • p4prom_common.sh is required by all four scripts, including air-gapped installs
    • what -d does and doesn't relocate (files that stay in /etc/*)
    • services-per-role table
    • combined-role hosts: running install_node.sh on a host already set up by install_p4prom.sh replaces
      the node_exporter unit and silently stops p4_* metrics

Bug fixes found during testing (separate commits)

  • update_prom_graf.sh reported "Grafana service not found" while Grafana was running (grep -q under
    pipefail), and exited 1 on the first update after install.
  • Installer health checks now retry, so a slow Grafana start isn't reported as a failure.
  • systemd_enable_and_restart used systemctl start, so a changed unit file wasn't picked up; it now
    restarts, clears the start-rate limit first, and fails if the service isn't active afterwards. This also
    fixes repeated updates leaving p4prometheus in start-limit-hit while the script returned success.
  • update_p4prom.sh checked versions via PATH, so a custom -b caused a reinstall on every run.
  • update_p4prom.sh on a never-installed host now says to run install_p4prom.sh; with -nosdp it no
    longer needs a reachable p4d to report config problems.
  • install_node.sh / update_node.sh restart node_exporter.

Session logs

This PR adds ai_dev_support/: session logs, issue drafts, and the lab test plan/results from the
AI-assisted development of this work. They are kept for training and audit-trail value and are not part of
the shipped product (see ai_dev_support/README.md).

Testing

Lab results are in ai_dev_support/test_plan_122.md: negative tests for each script, installs and updates
with custom -d/-b/-m, a replica, a node-only host, the combined-role failure and recovery, and a 12x
repeated-update stress run. Prometheus on the hub scrapes all five hosts.

cttyler and others added 22 commits October 15, 2026 19:16
Add shared preflight_check_dirs() to p4prom_common.sh that bails with
a clear error if a non-default -d (data root) or -b (bin dir) path
does not exist or is not writable, instead of silently creating it or
failing later with a confusing error. Default paths are left alone
(assumed to always exist).

- install_prom_graf.sh / install_p4prom.sh: call preflight check right
  after the root-privilege check.
- update_prom_graf.sh / update_p4prom.sh: call preflight check after
  state-file loading completes, since data_root/bin_dir/metrics_root
  can be overridden by the saved state file. Previously these updater
  scripts validated the checks too early (before state-file loading),
  so a state-file-restored custom path was never actually validated.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…122)

- Add "Expected systemd Services by Role" table under Deployment Roles.
- Add "Combined-Role Hosts" guidance (e.g. Helix Core replica + Swarm):
  only run install_p4prom.sh, not install_node.sh, to avoid duplicate
  node_exporter management.
- Clarify p4prom_common.sh is required by all 4 install/update scripts,
  not just the two Helix Core server ones; add wget example and an
  air-gapped-installation cross-reference anchor.
- Add IMPORTANT note that -d/-b custom paths must pre-exist (no more
  silent auto-mkdir) and a table of files that always stay under
  /etc/* and /usr/share/* regardless of -d.
- Regenerate derived HTML/PDF from the updated .adoc source.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Establish naming convention: ai_dev_support/ holds AI-assistant working
files that are not part of the shipped product (session logs, issue
drafts, scratch test plans), and may or may not be versioned. Reserve
a future ai/ tree (ai/dev/, ai/ops/) for AI skills that will actually
ship as part of the product: ai/dev/ for developing/testing
P4Prometheus, ai/ops/ for operating it in live environments. This
mirrors a convention being piloted in SDP; p4prometheus is the first
Git-hosted project applying it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reverse the earlier decision to gitignore ai_dev_support/. After
consideration, the folder's session logs, issue drafts, and test plans
have ongoing training and audit-trail value, and their size is
negligible for a full clone. This is a permanent versioning decision,
distinct from the future ai/dev/ and ai/ops/ trees which will be
versioned because they are part of the shipped product itself.

Add ai_dev_support/README.md documenting the folder's purpose, the
session-log naming convention, and explicit guidance on keeping
secrets (especially GitHub PATs) out of committed session logs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add ai_dev_support/github-issue-draft-123.md: filed as issue #123,
  "Improve support for air-gapped installation".
- Add ai_dev_support/github-issue-draft-124.md: filed as issue #124,
  "Add an uninstall/reset script for P4Prometheus components".
- Update session-log-2026-09-25-preflight-checks-docs.md with the
  ai_dev_support/ versioning decision, the #123/#124 filing, and
  clarified Git branch/PR/push mechanics discussed this session.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Alertmanager is installed by default, the lock timer is p4monitor_locks,
and install_node.sh on a p4prom host silently stops P4 metrics.
Record hub test results in the test plan.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…lth check race

check_service_exists used list-unit-files | grep -q under pipefail, which can
false-negative via SIGPIPE. A missing rendered-checksum key in install.env
aborted the first update under set -e. Health checks now retry so a slow
Grafana start is not reported as a failure.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
systemctl start is a no-op on a running service, so a rewritten unit file was
not picked up until a manual restart.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…k_dirs

A custom -m path must now exist, like -d/-b. [[ -w ]] is always true for root,
so probe with a test file instead; this catches read-only mounts.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ck for -nosdp

On a never-installed host the message now says to run install_p4prom.sh
instead of complaining about /hxlogs/metrics, and -nosdp no longer needs a
reachable p4d or set P4PORT before configuration problems are reported.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… take effect

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…session log

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
systemd_enable_and_restart now clears the start-rate limit before restarting
and fails if the service is not active afterwards. update_p4prom.sh checked
component versions via PATH, so a custom -b caused a reinstall on every run.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s draft

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant