chore(ci): reduce apache2 integration runner usage

.github/actions/setup-integration/action.yml

@@ -0,0 +1,49 @@
+---
+name: Prepare integration dependencies
+description: Restore Workstation and cache downloads for an exact Policyfile dependency snapshot.
+inputs:
+  dependencies:
+    description: Resolved Git dependency revisions.
+    required: true
+  dependency-key:
+    description: Hash of the Policyfile and resolved Git dependencies.
+    required: true
+  workstation-version:
+    description: Exact Workstation version selected by the prepare job.
+    default: latest
+  workstation-installed:
+    description: Skip restoring Workstation when prepare has already installed it.
+    default: 'false'
+runs:
+  using: composite
+  steps:
+    - name: Restore Cinc Workstation
+      id: workstation
+      if: inputs.workstation-installed != 'true'
+      uses: sous-chefs/.github/.github/actions/install-workstation@codex/cache-cinc-workstation
+      with:
+        version: ${{ inputs.workstation-version }}
+    - name: Configure dependency cache and pin Git revisions
+      shell: bash
+      env:
+        DEPENDENCIES: ${{ inputs.dependencies }}
+      run: |
+        echo "CHEF_WORKSTATION_HOME=$RUNNER_TEMP/chef-workstation" >> "$GITHUB_ENV"
+        git config --global maintenance.auto false
+        git config --global gc.auto 0
+        python3 .github/ci/integration.py pin
+    - name: Restore cookbook downloads
+      id: cache
+      uses: actions/cache/restore@v5
+      with:
+        path: ${{ runner.temp }}/chef-workstation/cache
+        key: policyfile-v1-${{ runner.os }}-${{ runner.arch }}-${{ steps.workstation.outputs.version || inputs.workstation-version }}-${{ inputs.dependency-key }}
+    - name: Resolve cookbooks for this checkout
+      shell: bash
+      run: /opt/cinc-workstation/bin/chef-cli install Policyfile.rb
+    - name: Save cookbook downloads
+      if: steps.cache.outputs.cache-hit != 'true'
+      uses: actions/cache/save@v5
+      with:
+        path: ${{ runner.temp }}/chef-workstation/cache
+        key: policyfile-v1-${{ runner.os }}-${{ runner.arch }}-${{ steps.workstation.outputs.version || inputs.workstation-version }}-${{ inputs.dependency-key }}

.github/ci/.gitignore

@@ -0,0 +1 @@
+__pycache__/

.github/ci/README.md

@@ -0,0 +1,64 @@
+# Integration performance
+
+Each OS job runs its suites on one GitHub-hosted runner. Workstation, Kitchen,
+kitchen-dokken and downloaded base images are shared within that job. Each suite
+still runs `kitchen test --destroy=always` with two converges and a fresh container.
+
+The prepare job selects one Workstation version and exact cookbook Git revisions,
+then restores or populates the cookbook download cache before the matrix starts.
+Every matrix job regenerates its Policyfile lock for its own checkout. Neither
+Kitchen state nor converged containers are cached.
+
+Grouped jobs have a 30-minute deadline to prevent a stalled suite holding a
+runner for hours.
+
+CI sets `KITCHEN_REUSE_IMAGES=true`: Dokken pulls an image when it is missing,
+then uses that local image for later suites. Local runs keep Dokken's usual
+pull-on-each-create behaviour unless explicitly given that variable. Standard
+GitHub-hosted VMs are fresh for each job; this does not retain a VM across runs.
+
+## Docker image delivery benchmark
+
+[Run on 20 October 2026](https://github.com/sous-chefs/apache2/actions/runs/39509952839)
+compared three fresh runners per method and OS. Every pair included the Cinc
+image and the OS image. Both methods used identical digests and checked image IDs.
+
+| OS | Pull median (range) | Restore and load median (range) |
+| --- | ---: | ---: |
+| AlmaLinux 9 | 11.05s (10.69–12.01s) | 13.84s (13.31–19.72s) |
+| openSUSE Leap 15 | 15.96s (14.97–20.03s) | 19.89s (18.87–20.61s) |
+| Ubuntu 24.04 | 14.35s (13.56–17.61s) | 21.11s (15.77–24.54s) |
+
+Restore timing includes Actions cache download, decompression and `docker load`.
+Pull timing includes fetching the digest and tagging it for Dokken. Both exclude
+checkout, manifest download, runner setup and queue time. Cache creation is an
+additional cost reported in the seed jobs. Direct pulls won all three medians,
+so normal integration jobs do not restore Docker image archives.
+
+The separate image benchmark only runs on manual dispatch or changes to its own
+workflow/script. Its artifacts retain the individual measurements for seven days.
+The results above describe these images and runners; they are not a universal
+claim that image caching is slower.
+
+## Suite grouping comparison
+
+The [initial comparison](https://github.com/sous-chefs/apache2/actions/runs/39509953258)
+used the same commit, cookbook revisions, runner type and eight-job limit for
+both layouts. Nine complete matched OS groups covered 78 suites:
+
+| Layout | Active job time |
+| --- | ---: |
+| Individual suite jobs | 7,359s (122.7 minutes) |
+| Suites grouped by OS | 4,931s (82.2 minutes) |
+
+Grouping used 33.0% fewer active runner minutes for that subset. This excludes
+queue time and does not measure end-to-end completion time. The complete matrix
+comparison did not finish: two individual AlmaLinux jobs were still in Dokken
+after almost 30 minutes. At the recorded snapshot, 15 grouped OS jobs and 99
+individual jobs had passed. Unfinished OS comparisons are excluded from the
+figures above, so the result must not be presented as a full-matrix saving.
+
+The temporary individual baseline jobs have been removed. The normal grouped
+workflow retains all 148 original suite/platform combinations. The additional
+image-reuse and cache-priming changes were applied after this comparison; their
+benefit is not included in the measured saving.

.github/ci/docker_images.py

@@ -0,0 +1,99 @@
+"""Compare registry pulls with restored Docker archives on fresh runners."""
+import argparse
+import hashlib
+import json
+import os
+from pathlib import Path
+import statistics
+import subprocess
+import time
+
+
+def docker(*args):
+    return subprocess.check_output(["docker", *args], text=True).strip()
+
+
+def inspect(tag):
+    return json.loads(docker("image", "inspect", tag))[0]
+
+
+def validate(images):
+    for image in images:
+        if inspect(image["tag"])["Id"] != image["id"]:
+            raise ValueError("Wrong image loaded: " + image["tag"])
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("action", choices=["seed", "start", "pull", "load", "report"])
+    args = parser.parse_args()
+    root = Path(os.environ.get("RUNNER_TEMP", "/tmp"))
+    archive = root / "dokken-images.tar"
+    manifest = root / "images.json"
+    clock = root / "image-start.txt"
+    if args.action == "seed":
+        tags = ["cincproject/cinc:latest", os.environ["PLATFORM_IMAGE"] + ":latest"]
+        images = []
+        started = time.monotonic()
+        for tag in tags:
+            docker("pull", "--platform", "linux/amd64", tag)
+            row = inspect(tag)
+            images.append(dict(tag=tag, ref=row["RepoDigests"][0], id=row["Id"]))
+        pull_seconds = time.monotonic() - started
+        started = time.monotonic()
+        docker("save", "--output", str(archive), *tags)
+        manifest.write_text(json.dumps(images))
+        key = "dokken-images-v1-Linux-X64-" + hashlib.sha256(json.dumps(images, sort_keys=True).encode()).hexdigest()
+        with open(os.environ["GITHUB_OUTPUT"], "a") as output:
+            output.write("key=" + key + "\n")
+        with open(os.environ["GITHUB_STEP_SUMMARY"], "a") as output:
+            output.write(f"Seed {tags[1]}: pull {pull_seconds:.2f}s; docker save {time.monotonic()-started:.2f}s; archive {archive.stat().st_size/1024**2:.1f} MiB.\n")
+    elif args.action == "start":
+        # Called before actions/cache, so restore includes download and extraction.
+        images = json.loads(manifest.read_text())
+        for image in images:
+            result = subprocess.run(["docker", "image", "inspect", image["tag"]], capture_output=True)
+            if result.returncode == 0:
+                raise ValueError("Benchmark runner already contains " + image["tag"])
+        clock.write_text(str(time.monotonic()))
+    elif args.action in {"pull", "load"}:
+        images = json.loads(manifest.read_text())
+        if args.action == "pull":
+            for image in images:
+                docker("pull", "--platform", "linux/amd64", image["ref"])
+                docker("tag", image["ref"], image["tag"])
+        else:
+            docker("load", "--input", str(archive))
+        validate(images)
+        result = dict(os=os.environ["PLATFORM"], mode=args.action,
+                      trial=int(os.environ["TRIAL"]),
+                      seconds=time.monotonic()-float(clock.read_text()),
+                      images=images)
+        Path(f"result-{result['os']}-{result['mode']}-{result['trial']}.json").write_text(json.dumps(result))
+        with open(os.environ["GITHUB_STEP_SUMMARY"], "a") as output:
+            output.write(f"{result['os']} {result['mode']} trial {result['trial']}: **{result['seconds']:.2f}s**, both image IDs verified.\n")
+    else:
+        rows = [json.loads(p.read_text()) for p in Path("results").glob("result-*.json")]
+        if len(rows) != 18:
+            raise ValueError(f"Expected 18 measurements, got {len(rows)}")
+        lines = ["| OS | Registry pull median (range) | Cache restore + load median (range) |", "| --- | ---: | ---: |"]
+        for platform in sorted({r['os'] for r in rows}):
+            columns = []
+            snapshots = {json.dumps(r['images'], sort_keys=True) for r in rows if r['os']==platform}
+            if len(snapshots) != 1:
+                raise ValueError("Benchmark image snapshots differ")
+            for mode in ("pull", "load"):
+                values = [r['seconds'] for r in rows if r['os']==platform and r['mode']==mode]
+                if len(values) != 3:
+                    raise ValueError("Missing repeats")
+                columns.append(f"{statistics.median(values):.2f}s ({min(values):.2f}–{max(values):.2f}s)")
+            lines.append("| " + " | ".join([platform, *columns]) + " |")
+        lines.append("\nCache timing includes Actions download/decompression and docker load. Pull timing includes digest pulls and tagging. Checkout, manifest download and runner queue/setup are excluded from both. Cache creation is reported separately by seed jobs; it is not free. No containers or runner state are reused.")
+        report = "\n".join(lines) + "\n"
+        print(report)
+        with open(os.environ["GITHUB_STEP_SUMMARY"], "a") as output:
+            output.write(report)
+
+
+if __name__ == "__main__":
+    main()

.github/ci/integration.py

@@ -0,0 +1,109 @@
+"""Plan isolated Kitchen suites, classify changes and pin dependency snapshots."""
+import argparse
+import hashlib
+import json
+import os
+from pathlib import Path
+import re
+import subprocess
+import sys
+
+# Preserve the existing CI coverage during the grouping trial.
+SUITES = {"basic-site", "default", "mod-auth-cas", "php", "module-template",
+          "mod-wsgi", "ports", "ssl", "install-override"}
+DEPENDENCY = re.compile(r"cookbook '([^']+)', git: '([^']+)', branch: '([^']+)'")
+
+
+def documentation_only(paths):
+    return bool(paths) and all(
+        p.lower().endswith(".md") or p.startswith(("docs/", "documentation/"))
+        or p in {"LICENSE", "CODE_OF_CONDUCT", "CONTRIBUTING", "TESTING"}
+        for p in paths)
+
+
+def plan(instances):
+    groups = {}
+    for row in instances:
+        name = row["instance"]
+        suite = next((s for s in SUITES if name.startswith(s + "-")), None)
+        if suite:
+            platform = name[len(suite) + 1:]
+            groups.setdefault(platform, []).append(name)
+    grouped = [{"os": p, "instances": sorted(names)} for p, names in sorted(groups.items())]
+    individual = [{"os": g["os"], "instances": [n]} for g in grouped for n in g["instances"]]
+    return grouped, individual
+
+
+def pin(source, manifest):
+    def replacement(match):
+        name, url, branch = match.groups()
+        entry = manifest[name]
+        if entry["url"] != url or entry["branch"] != branch:
+            raise ValueError("Dependency manifest does not match " + name)
+        revision = entry["revision"]
+        if not re.fullmatch(r"[0-9a-f]{40}", revision):
+            raise ValueError("Invalid Git revision for " + name)
+        return f"cookbook '{name}', git: '{url}', ref: '{revision}'"
+    if len(DEPENDENCY.findall(source)) != len(manifest):
+        raise ValueError("Dependency manifest is incomplete")
+    return DEPENDENCY.sub(replacement, source)
+
+
+def run_instances(instances):
+    failures = []
+    env = dict(os.environ, MULTIPLE_CONVERGE="2")
+    for instance in instances:
+        print("::group::Kitchen " + instance, flush=True)
+        result = subprocess.run(["kitchen", "test", "^" + re.escape(instance) + "$",
+                                 "--destroy=always"], env=env, check=False)
+        print("::endgroup::", flush=True)
+        if result.returncode:
+            failures.append(instance)
+    if failures:
+        print("::error::Failed instances: " + ", ".join(failures))
+    return bool(failures)
+
+
+def output(values):
+    with open(os.environ["GITHUB_OUTPUT"], "a") as stream:
+        for key, value in values.items():
+            stream.write(key + "=" + (value if isinstance(value, str) else json.dumps(value, separators=(",", ":"))) + "\n")
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("action", choices=["changes", "prepare", "pin", "run"])
+    args = parser.parse_args()
+    if args.action == "changes":
+        base, head = os.environ.get("BASE_SHA", ""), os.environ.get("HEAD_SHA", "HEAD")
+        paths = []
+        if base and base.strip("0"):
+            result = subprocess.run(["git", "diff", "--name-only", "--no-renames", "-z", base, head],
+                                    text=True, capture_output=True, check=False)
+            if result.returncode == 0:
+                paths = [p for p in result.stdout.split("\0") if p]
+        output({"integration": "false" if documentation_only(paths) else "true"})
+    elif args.action == "prepare":
+        instances = json.loads(subprocess.check_output(["kitchen", "list", "--json"], text=True))
+        grouped, individual = plan(instances)
+        if len(individual) != 148 or len(grouped) != 17:
+            raise ValueError("Kitchen coverage changed; review the CI plan before continuing")
+        source = Path("Policyfile.rb").read_text()
+        manifest = {}
+        for name, url, branch in DEPENDENCY.findall(source):
+            revision = subprocess.check_output(["git", "ls-remote", "--exit-code", url, "refs/heads/" + branch], text=True).split()[0]
+            manifest[name] = dict(url=url, branch=branch, revision=revision)
+        digest = hashlib.sha256((source + json.dumps(manifest, sort_keys=True)).encode()).hexdigest()
+        output(dict(grouped={"include": grouped},
+                    dependencies=manifest, dependency_key=digest))
+        print(f"Planned {len(individual)} suites across {len(grouped)} fresh runners")
+    elif args.action == "pin":
+        path = Path("Policyfile.rb")
+        path.write_text(pin(path.read_text(), json.loads(os.environ["DEPENDENCIES"])))
+    else:
+        return run_instances(json.loads(os.environ["KITCHEN_INSTANCES"]))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

.github/ci/test_integration.py

@@ -0,0 +1,40 @@
+import io
+import unittest
+from unittest.mock import patch
+import integration
+
+
+class IntegrationTest(unittest.TestCase):
+    def test_docs_filter_is_conservative(self):
+        self.assertTrue(integration.documentation_only(["README.md", "documentation/example.md"]))
+        for paths in ([], ["resources/module.rb"], ["README.md", "Policyfile.rb"], [".github/workflows/ci.yml"]):
+            self.assertFalse(integration.documentation_only(paths))
+
+    def test_grouping_preserves_exact_instance_set_and_exclusions(self):
+        names = ["default-ubuntu-2404", "php-ubuntu-2404", "default-almalinux-10", "custom-template-ubuntu-2404"]
+        grouped, individual = integration.plan([{"instance": n} for n in names])
+        self.assertEqual(len(grouped), 2)
+        self.assertEqual(sorted(n for g in grouped for n in g["instances"]), sorted(r["instances"][0] for r in individual))
+        self.assertNotIn("custom-template-ubuntu-2404", [r["instances"][0] for r in individual])
+        self.assertNotIn("mod-auth-cas-almalinux-10", [r["instances"][0] for r in individual])
+
+    def test_pin_requires_matching_snapshot(self):
+        source = "cookbook 'apt', git: 'https://example/apt.git', branch: 'main'\n"
+        manifest = {"apt": dict(url="https://example/apt.git", branch="main", revision="a" * 40)}
+        self.assertIn("ref: '" + "a" * 40 + "'", integration.pin(source, manifest))
+        with self.assertRaises(ValueError):
+            integration.pin(source, {})
+
+    @patch("integration.subprocess.run")
+    @patch("sys.stdout", new_callable=io.StringIO)
+    def test_each_suite_gets_its_own_lifecycle_even_after_failure(self, stdout, run):
+        run.return_value.returncode = 1
+        self.assertTrue(integration.run_instances(["default-ubuntu-2404", "php-ubuntu-2404"]))
+        self.assertEqual(run.call_count, 2)
+        for call in run.call_args_list:
+            self.assertEqual(call.args[0][-1], "--destroy=always")
+            self.assertEqual(call.kwargs["env"]["MULTIPLE_CONVERGE"], "2")
+
+
+if __name__ == "__main__":
+    unittest.main()