Repository navigation
[release-4.18] OCPBUGS-136438: on-prem: redispatch API VIP connections away from unreachable masters - #6714
Conversation
…achable masters The API VIP haproxy keeps a powered-off master in round-robin rotation until its health check marks it DOWN, which takes 12-17s on main and 25-30s on release branches. haproxy runs in TCP mode with timeout connect 10s, retries 3 and no option redispatch, so a client connection assigned to that master is retried on the same dead master and never moved. The client gives up first: client-go fails with "net/http: TLS handshake timeout" after 10s, and roughly a third of new connections stall like this for as long as the master stays in rotation (OCPBUGS-72554). Make the masters backend move such connections to another master: - option redispatch 1: redispatch on every connection retry. haproxy skips the server that just failed and does not apply its 1s turnaround delay when it switches servers. - timeout connect 2s: abandon a connection attempt to an unreachable master after 2s instead of 10s. Two failed attempts still end well within client-go's TLS handshake timeout, and a single lost SYN (1s initial retransmission) is still tolerated. A connection that lands on a powered-off master is now delayed by at most about 2s instead of failing, and a master whose apiserver refuses connections is skipped at once. The /readyz check timing from OCPBUGS-43428 is unchanged, so an apiserver briefly answering 500 is tolerated exactly as before. The shorter connect timeout also bounds the TCP and TLS phase of the health check, so a powered-off master leaves the rotation sooner. The new test checks these properties on the rendered template for every on-prem platform. Assisted-By: Claude Opus 5.5
|
@openshift-cherrypick-robot: Detected clone of Jira Issue OCPBUGS-72554 with correct target version. Will retitle the PR to link to the clone. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
|
Pipeline controller notification This PR uses the pipeline controller for second-stage tests. Selection and triggering follow the repository configuration. Use |
openshift-ci
Bot
changed the title
openshift-ci-robot
added
jira/severity-important
|
@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-136438, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/lgtm |
|
Scheduling tests matching the |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: mkowalski The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@openshift-cherrypick-robot: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
This is an automated cherry-pick of #6681
/assign openshift-ci-robot