Skip to content

[release-4.20] OCPBUGS-136332: Fix flaky vSphere failure domain VAP e2e tests - #1591

Open
vr4manta wants to merge 1 commit into
openshift:release-4.20from
vr4manta:OCPBUGS-136332
Open

vr4manta wants to merge 1 commit into
openshift:release-4.20from
vr4manta:OCPBUGS-136332

Conversation

@vr4manta

@vr4manta vr4manta commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Sippy's VSphereMultiVCenterDay2 (4.20) feature-gate tracking showed three of the new failure-domain VAP e2e tests failing intermittently:

  • "should block removing a failure domain referenced by a MachineSet" (12.5% pass) and "should allow an unrelated Infrastructure update..." (81.2% pass) both create a MachineSet via createVAPTestMachineSet, which always uses the same deterministic name. Two of the three call sites already wait for confirmed deletion during cleanup to avoid a same-name collision with a subsequent test; the MachineSet test's DeferCleanup was missing that wait, so a still-terminating MachineSet from it could cause a later Create to fail with 409 AlreadyExists.

  • "should block removing a failure domain referenced by a Machine" (64.3% pass) fetched Infrastructure/cluster once and called Update directly with no retry. Its resourceVersion can be bumped concurrently by other controllers reconciling status (this feature itself syncs vCenter connection details into Infrastructure.Status), so a transient 409 conflict was failing the test instead of being retried.

Add the missing wait-for-deletion to the MachineSet test's cleanup, and wrap the Machine test's Infrastructure update in retry.RetryOnConflict.

Sippy's VSphereMultiVCenterDay2 (4.20) feature-gate tracking showed three
of the new failure-domain VAP e2e tests failing intermittently:

- "should block removing a failure domain referenced by a MachineSet"
  (12.5% pass) and "should allow an unrelated Infrastructure update..."
  (81.2% pass) both create a MachineSet via createVAPTestMachineSet,
  which always uses the same deterministic name. Two of the three call
  sites already wait for confirmed deletion during cleanup to avoid a
  same-name collision with a subsequent test; the MachineSet test's
  DeferCleanup was missing that wait, so a still-terminating MachineSet
  from it could cause a later Create to fail with 409 AlreadyExists.

- "should block removing a failure domain referenced by a Machine"
  (64.3% pass) fetched Infrastructure/cluster once and called Update
  directly with no retry. Its resourceVersion can be bumped concurrently
  by other controllers reconciling status (this feature itself syncs
  vCenter connection details into Infrastructure.Status), so a transient
  409 conflict was failing the test instead of being retried.

Add the missing wait-for-deletion to the MachineSet test's cleanup, and
wrap the Machine test's Infrastructure update in retry.RetryOnConflict.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Nov 3, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification

This PR uses the pipeline controller for second-stage tests. Selection and triggering follow the repository configuration.

Use /test ? to list jobs and /pipeline help for commands.

@openshift-ci-robot openshift-ci-robot added the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Nov 3, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@vr4manta: This pull request references Jira Issue OCPBUGS-136332, which is invalid:

  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
  • expected Jira Issue OCPBUGS-136332 to depend on a bug targeting a version in 4.21.0, 4.21.z and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Sippy's VSphereMultiVCenterDay2 (4.20) feature-gate tracking showed three of the new failure-domain VAP e2e tests failing intermittently:

  • "should block removing a failure domain referenced by a MachineSet" (12.5% pass) and "should allow an unrelated Infrastructure update..." (81.2% pass) both create a MachineSet via createVAPTestMachineSet, which always uses the same deterministic name. Two of the three call sites already wait for confirmed deletion during cleanup to avoid a same-name collision with a subsequent test; the MachineSet test's DeferCleanup was missing that wait, so a still-terminating MachineSet from it could cause a later Create to fail with 409 AlreadyExists.

  • "should block removing a failure domain referenced by a Machine" (64.3% pass) fetched Infrastructure/cluster once and called Update directly with no retry. Its resourceVersion can be bumped concurrently by other controllers reconciling status (this feature itself syncs vCenter connection details into Infrastructure.Status), so a transient 409 conflict was failing the test instead of being retried.

Add the missing wait-for-deletion to the MachineSet test's cleanup, and wrap the Machine test's Infrastructure update in retry.RetryOnConflict.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Nov 3, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Central YAML; Path: .coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: 86112fe0-d91b-4bc6-a4fb-c63f2e7b230c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.

@vr4manta vr4manta changed the title OCPBUGS-136332: Fix flaky vSphere failure domain VAP e2e tests [release-4.20] OCPBUGS-136332: Fix flaky vSphere failure domain VAP e2e tests Nov 3, 2026
@openshift-ci
openshift-ci Bot requested review from JoelSpeed and nrb November 3, 2026 13:20
@openshift-ci

openshift-ci Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign radekmanak for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci

openshift-ci Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

@vr4manta: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants