Skip to content

[release-5.0] USHIFT-7587: AI model serving RPM only in the image used for AI model serving RF test - #7580

Merged
openshift-merge-bot[bot] merged 3 commits into
openshift:release-5.0from
agullon:USHIFT-7587-release-5.0
Oct 26, 2026
Merged

openshift-merge-bot[bot] merged 3 commits into
openshift:release-5.0from
agullon:USHIFT-7587-release-5.0

Conversation

@agullon

@agullon agullon commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Backport of #7570 to release-5.0 (clean cherry-picks of d79350d, 652d883 and eba4c50, -x trailers in the commits; no conflicts).

While running the 5.0.0-rc.7 release tests, only the x86_64 lrel jobs showed performance degradation (memory-pressure page-cache thrash); the arm jobs did not. A large part of it traced to the x86_64-only microshift-ai-model-serving (kserve) RPM being installed into the shared rhel{98,102}-bootc-brew-lrel-optional base image, which is the FROM base for ~39 non-AI lrel scenarios. On x86 each of them loaded kserve's CRDs/OpenAPI into the MicroShift apiserver, inflating resident heap by ~940 MB (measured) for scenarios that never exercise AI model serving.

Changes

  • Remove the MICROSHIFT_X86_64_RPMS block from both -brew-lrel-optional images (el9 + el10).
  • Install the AI RPM (x86_64-gated) in the lrel FIPS image, consumed exclusively by *@ai-model-serving-online-fips.
  • Rename -brew-lrel-fips → -brew-lrel-fips-ai (el9 + el10) and repoint start_image in both ai-model-serving-online-fips scenarios.

suites/optional/ has no kserve references, so the optional scenario is unaffected.

Same change as #7570 on main, so the release-5.0 release-qualification runs (where the rc tests actually execute) get the fix too.

RPM/ostree image (second commit)

The RPM/ostree twin test/image-blueprints/layer4-release/group3/rhel98-brew-lrel-optional.toml (and the .image-installer ISO built from it) also installed the kserve RPM on x86_64. It is the start image for 19 non-AI release scenarios; no RPM/ostree scenario runs suites/ai-model-serving.

On every cold start (cleanup --all, fresh database) the kserve webhook has no endpoints for several minutes. MicroShift applies manifests.d paths sequentially with a 10m retry per path, so 010-kserve / 050-runtimes block everything after them. This is the root cause of the chronic e2e-aws-tests-release flakes on release-5.0:

  • 080-metrics-server lands after the 5m waitForNamespace deadline, the metrics certs are never provisioned and the 600s healthcheck fails: el98-lrel@osconfig "Verify Cluster ID Change For New Database", el98-yminus1@el98-lrel@lvms-standard "Log Scan", "Verify Local Host Name Resolution".
  • Test manifests.d dirs get applied ~14m after restart, past the kustomize suite wait, and the retries exhaust the 60m robot budget: el98-lrel@iso-standard.

Every failing run has Failed to provision metrics-server certs: waiting for namespace openshift-monitoring: context deadline exceeded at the failure time. The bootc el98-lrel@iso-standard twin, whose image no longer has kserve, passed 52/52 in the same CI run.

Router test race (third commit)

Unrelated to kserve, but small and found in the same CI analysis. Deploy Web Server / Deploy Web Server Signed in test/resources/router.resource ran oc create and immediately oc wait pod --selector, which exits with "no matching resources found" if the pod is not created yet (el98-lrel@router-routes "Edge And Passthrough Routes" failed this way, 160ms before the pod was bound). Now they wait with Named Deployment Rollout Should Be Complete (oc rollout status). Verified with robocop and robot --dryrun suites/router/ (31/31).

Jira

https://redhat.atlassian.net/browse/USHIFT-7587

🤖 Generated with Claude Code

While running the 5.0.0-rc.7 release tests, the x86_64 lrel jobs showed
performance degradation (memory-pressure page-cache thrash) that the arm jobs
did not. A large part of it traced to the x86_64-only microshift-ai-model-serving
(kserve) RPM being installed into the shared rhel{98,102}-bootc-brew-lrel-optional
base image. That image is the FROM base for ~39 non-AI lrel scenarios, so on x86
every one of them loaded kserve's CRDs/OpenAPI into the MicroShift apiserver,
inflating resident heap by ~940 MB (measured) for scenarios that never exercise
AI model serving and squeezing available memory on the memory-constrained x86
metal hosts.

Scope the AI RPM to the single scenario that actually tests it:

- Remove the MICROSHIFT_X86_64_RPMS block from both -brew-lrel-optional images.
- Install the AI RPM (x86_64-gated) in the lrel FIPS image, which is consumed
  exclusively by the *@ai-model-serving-online-fips scenario.
- Rename the FIPS containerfiles -brew-lrel-fips -> -brew-lrel-fips-ai (el9 +
  el10) to reflect that they now carry the AI RPM, and repoint start_image in
  both ai-model-serving-online-fips scenarios accordingly.

suites/optional/ has no kserve references, so the optional scenario is unaffected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED

(cherry picked from commit d79350d)
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Oct 22, 2026
@openshift-ci-robot

openshift-ci-robot commented Oct 22, 2026 •

Copy link
Copy Markdown

@agullon: This pull request references USHIFT-7587 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

Backport of #7570 to release-5.0 (clean cherry-pick of d79350d, -x trailer in the commit; no conflicts).

While running the 5.0.0-rc.7 release tests, only the x86_64 lrel jobs showed performance degradation (memory-pressure page-cache thrash); the arm jobs did not. A large part of it traced to the x86_64-only microshift-ai-model-serving (kserve) RPM being installed into the shared rhel{98,102}-bootc-brew-lrel-optional base image, which is the FROM base for ~39 non-AI lrel scenarios. On x86 each of them loaded kserve's CRDs/OpenAPI into the MicroShift apiserver, inflating resident heap by ~940 MB (measured) for scenarios that never exercise AI model serving.

Changes

  • Remove the MICROSHIFT_X86_64_RPMS block from both -brew-lrel-optional images (el9 + el10).
  • Install the AI RPM (x86_64-gated) in the lrel FIPS image, consumed exclusively by *@ai-model-serving-online-fips.
  • Rename -brew-lrel-fips → -brew-lrel-fips-ai (el9 + el10) and repoint start_image in both ai-model-serving-online-fips scenarios.

suites/optional/ has no kserve references, so the optional scenario is unaffected.

Same change as #7570 on main, so the release-5.0 release-qualification runs (where the rc tests actually execute) get the fix too.

Jira

https://redhat.atlassian.net/browse/USHIFT-7587

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification

This PR uses the pipeline controller for second-stage tests. Selection and triggering follow the repository configuration.

Use /test ? to list jobs and /pipeline help for commands.

@agullon agullon changed the title USHIFT-7587: AI model serving RPM only in the bootc image used for AI model serving RF test [release-5.0] USHIFT-7587: AI model serving RPM only in the bootc image used for AI model serving RF test Oct 22, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-tests
/test e2e-aws-tests-arm
/test e2e-aws-tests-bootc-arm-el10
/test e2e-aws-tests-bootc-arm-el9
/test e2e-aws-tests-bootc-el10
/test e2e-aws-tests-bootc-el9

1 similar comment
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-tests
/test e2e-aws-tests-arm
/test e2e-aws-tests-bootc-arm-el10
/test e2e-aws-tests-bootc-arm-el9
/test e2e-aws-tests-bootc-el10
/test e2e-aws-tests-bootc-el9

@openshift-ci
openshift-ci Bot requested review from eslutsky and jerpeter1 October 22, 2026 11:30
@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 22, 2026
@agullon

agullon commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

/test e2e-aws-tests-cache
/test e2e-aws-tests-cache-arm

@agullon

agullon commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

/test e2e-aws-tests-release
/test e2e-aws-tests-release-arm
/test e2e-aws-tests-bootc-release-el9
/test e2e-aws-tests-bootc-release-el10
/test e2e-aws-tests-bootc-release-arm-el9
/test e2e-aws-tests-bootc-release-arm-el10

@agullon

agullon commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

/override ci/prow/e2e-aws-tests-bootc-release-el10
/override ci/prow/e2e-aws-tests-bootc-release-el9
/override ci/prow/e2e-aws-tests-release

  • e2e-aws-tests-bootc-release-el10: el102-lrel@ai-model-serving-online-fips "Test OpenVINO model". The kserve webhook had no endpoints because the 4GiB VM was starved: the check-payload FIPS scan took 23 min and kubelet logs lagged up to 20 min. Same failure in 2 of 3 runs on [release-5.0] Release Testing 5.0.0-rc.7 #7562. The kserve pods are running, which shows the AI RPM is installed in the new -fips-ai image.
  • e2e-aws-tests-bootc-release-el9: el98-lrel@dual-stack-configuration1 "DNS Resolution After Resource Change". The same test failed on [release-5.0] Release Testing 5.0.0-rc.7 #7562. The el9 AI scenario passed.
  • e2e-aws-tests-release (RPM/ostree, doesn't use the bootc blueprints or scenarios this PR touches): iso-standard, osconfig, lvms-standard and router-routes. iso-standard, osconfig and lvms-standard also fail on [release-5.0] Release Testing 5.0.0-rc.7 #7562. router-routes is a 503 from the router during setup.

@openshift-ci

openshift-ci Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

@agullon: Overrode contexts on behalf of agullon: ci/prow/e2e-aws-tests-bootc-release-el10, ci/prow/e2e-aws-tests-bootc-release-el9, ci/prow/e2e-aws-tests-release

Details

In response to this:

/override ci/prow/e2e-aws-tests-bootc-release-el10
/override ci/prow/e2e-aws-tests-bootc-release-el9
/override ci/prow/e2e-aws-tests-release

  • e2e-aws-tests-bootc-release-el10: el102-lrel@ai-model-serving-online-fips "Test OpenVINO model". The kserve webhook had no endpoints because the 4GiB VM was starved: the check-payload FIPS scan took 23 min and kubelet logs lagged up to 20 min. Same failure in 2 of 3 runs on [release-5.0] Release Testing 5.0.0-rc.7 #7562. The kserve pods are running, which shows the AI RPM is installed in the new -fips-ai image.
  • e2e-aws-tests-bootc-release-el9: el98-lrel@dual-stack-configuration1 "DNS Resolution After Resource Change". The same test failed on [release-5.0] Release Testing 5.0.0-rc.7 #7562. The el9 AI scenario passed.
  • e2e-aws-tests-release (RPM/ostree, doesn't use the bootc blueprints or scenarios this PR touches): iso-standard, osconfig, lvms-standard and router-routes. iso-standard, osconfig and lvms-standard also fail on [release-5.0] Release Testing 5.0.0-rc.7 #7562. router-routes is a 503 from the router during setup.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@agullon

agullon commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

/label jira/valid-bug
/label backport-risk-assessed
/verified by CI

@openshift-ci openshift-ci Bot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. labels Oct 23, 2026
@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Oct 23, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@agullon: This PR has been marked as verified by CI.

Details

In response to this:

/label jira/valid-bug
/label backport-risk-assessed
/verified by CI

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

The RPM/ostree twin of the bootc change. rhel98-brew-lrel-optional (and the
image-installer ISO built from it) still installed the x86_64-only kserve RPM,
and it is the start image for 19 non-AI release scenarios. No RPM/ostree
scenario runs suites/ai-model-serving.

On every cold start (cleanup --all, fresh database) kserve's webhook has no
endpoints for several minutes. MicroShift applies manifests.d paths
sequentially, retrying each for up to 10m, so 010-kserve and 050-runtimes
block every later path:

- 080-metrics-server lands after the 5m waitForNamespace deadline, the
  metrics certs are never provisioned and the 600s healthcheck fails
  (osconfig "Verify Cluster ID Change For New Database", lvms-standard
  "Log Scan", "Verify Local Host Name Resolution").
- Test manifests.d dirs are applied ~14m after restart, past the kustomize
  suite's 200s wait, and the retries plus sos collection exhaust the 60m
  robot budget (iso-standard).

The bootc iso-standard twin, whose image no longer carries kserve, passed
52/52 in the same CI run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED

(cherry picked from commit 652d883)
@openshift-ci-robot openshift-ci-robot removed the verified Signifies that the PR passed pre-merge verification criteria label Oct 23, 2026
@agullon agullon changed the title [release-5.0] USHIFT-7587: AI model serving RPM only in the bootc image used for AI model serving RF test [release-5.0] USHIFT-7587: AI model serving RPM only in the image used for AI model serving RF test Oct 23, 2026
…tests

Deploy Web Server ran `oc create` and immediately `oc wait pod --selector`.
When the ReplicaSet had not created the pod yet, `oc wait` exits at once with
"no matching resources found" instead of waiting, failing the test setup.
Seen in el98-lrel@router-routes ("Edge And Passthrough Routes"): oc wait gave
up 160ms before the pod was bound.

Wait on the Deployment with `oc rollout status`, which blocks until the
replicas are available even if no pod exists yet. Both web-server manifests
name the Deployment web-server-deploy. Deploy Web Server Signed had the same
pattern, usually hidden by the preceding secret poll.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED

(cherry picked from commit eba4c50)
@openshift-ci-robot

openshift-ci-robot commented Oct 23, 2026 •

Copy link
Copy Markdown

@agullon: This pull request references USHIFT-7587 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set.

Retaining the jira/valid-bug label as it was manually added.

Details

In response to this:

Summary

Backport of #7570 to release-5.0 (clean cherry-picks of d79350d, 652d883 and eba4c50, -x trailers in the commits; no conflicts).

While running the 5.0.0-rc.7 release tests, only the x86_64 lrel jobs showed performance degradation (memory-pressure page-cache thrash); the arm jobs did not. A large part of it traced to the x86_64-only microshift-ai-model-serving (kserve) RPM being installed into the shared rhel{98,102}-bootc-brew-lrel-optional base image, which is the FROM base for ~39 non-AI lrel scenarios. On x86 each of them loaded kserve's CRDs/OpenAPI into the MicroShift apiserver, inflating resident heap by ~940 MB (measured) for scenarios that never exercise AI model serving.

Changes

  • Remove the MICROSHIFT_X86_64_RPMS block from both -brew-lrel-optional images (el9 + el10).
  • Install the AI RPM (x86_64-gated) in the lrel FIPS image, consumed exclusively by *@ai-model-serving-online-fips.
  • Rename -brew-lrel-fips → -brew-lrel-fips-ai (el9 + el10) and repoint start_image in both ai-model-serving-online-fips scenarios.

suites/optional/ has no kserve references, so the optional scenario is unaffected.

Same change as #7570 on main, so the release-5.0 release-qualification runs (where the rc tests actually execute) get the fix too.

RPM/ostree image (second commit)

The RPM/ostree twin test/image-blueprints/layer4-release/group3/rhel98-brew-lrel-optional.toml (and the .image-installer ISO built from it) also installed the kserve RPM on x86_64. It is the start image for 19 non-AI release scenarios; no RPM/ostree scenario runs suites/ai-model-serving.

On every cold start (cleanup --all, fresh database) the kserve webhook has no endpoints for several minutes. MicroShift applies manifests.d paths sequentially with a 10m retry per path, so 010-kserve / 050-runtimes block everything after them. This is the root cause of the chronic e2e-aws-tests-release flakes on release-5.0:

  • 080-metrics-server lands after the 5m waitForNamespace deadline, the metrics certs are never provisioned and the 600s healthcheck fails: el98-lrel@osconfig "Verify Cluster ID Change For New Database", el98-yminus1@el98-lrel@lvms-standard "Log Scan", "Verify Local Host Name Resolution".
  • Test manifests.d dirs get applied ~14m after restart, past the kustomize suite wait, and the retries exhaust the 60m robot budget: el98-lrel@iso-standard.

Every failing run has Failed to provision metrics-server certs: waiting for namespace openshift-monitoring: context deadline exceeded at the failure time. The bootc el98-lrel@iso-standard twin, whose image no longer has kserve, passed 52/52 in the same CI run.

Router test race (third commit)

Unrelated to kserve, but small and found in the same CI analysis. Deploy Web Server / Deploy Web Server Signed in test/resources/router.resource ran oc create and immediately oc wait pod --selector, which exits with "no matching resources found" if the pod is not created yet (el98-lrel@router-routes "Edge And Passthrough Routes" failed this way, 160ms before the pod was bound). Now they wait with Named Deployment Rollout Should Be Complete (oc rollout status). Verified with robocop and robot --dryrun suites/router/ (31/31).

Jira

https://redhat.atlassian.net/browse/USHIFT-7587

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@agullon
agullon marked this pull request as draft October 23, 2026 10:03
@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 23, 2026
@agullon

agullon commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

/test e2e-aws-tests-cache
/test e2e-aws-tests-cache-arm

@agullon
agullon marked this pull request as ready for review October 23, 2026 10:09
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 23, 2026
@openshift-ci
openshift-ci Bot requested a review from pacevedom October 23, 2026 10:11
@agullon

agullon commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

/test e2e-aws-tests-release
/test e2e-aws-tests-release-arm
/test e2e-aws-tests-bootc-release-el9
/test e2e-aws-tests-bootc-release-el10
/test e2e-aws-tests-bootc-release-arm-el9
/test e2e-aws-tests-bootc-release-arm-el10

@agullon

agullon commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

/pipeline required

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-tests
/test e2e-aws-tests-arm
/test e2e-aws-tests-bootc-arm-el10
/test e2e-aws-tests-bootc-arm-el9
/test e2e-aws-tests-bootc-el10
/test e2e-aws-tests-bootc-el9

@agullon

agullon commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

/test e2e-aws-tests-bootc-release-el10

@kasturinarra

Copy link
Copy Markdown
Contributor

/test e2e-aws-tests-release-arm
/test e2e-aws-tests-bootc-release-arm-el9
/test e2e-aws-tests-bootc-release-el9

@kasturinarra

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Oct 26, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-tests
/test e2e-aws-tests-arm
/test e2e-aws-tests-bootc-arm-el10
/test e2e-aws-tests-bootc-arm-el9
/test e2e-aws-tests-bootc-el10
/test e2e-aws-tests-bootc-el9

@openshift-ci

openshift-ci Bot commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: agullon, kasturinarra

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [agullon,kasturinarra]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@agullon

agullon commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

/verified by CI

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Oct 26, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@agullon: This PR has been marked as verified by CI.

Details

In response to this:

/verified by CI

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@agullon

agullon commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

/test test-unit

@openshift-ci

openshift-ci Bot commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

@agullon: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-bot
openshift-merge-bot Bot merged commit cd73a3a into openshift:release-5.0 Oct 26, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants