[jhadvig]

Backport of #16911 and #17121 to release-5.0. Clean cherry-pick, no conflicts.

---

Analysis / Root cause:

Console sessions are stored entirely in per-pod process memory. When a pod restarts (upgrades, scaling, OOM, operator reconciliation), all sessions are lost and users must re-authenticate. This affects all users but is most disruptive for clusters using external IdPs (htpasswd, LDAP) where re-authentication requires full credential entry.

Two architectural limitations cause this:

1. Cookie encryption keys are generated randomly per process — cookies from one pod can't be decrypted by another
2. The refresh token cookie stores only a reference ID pointing to an in-memory map, not the actual token

Jira: https://redhat.atlassian.net/browse/OCPBUGS-71237
Related: https://redhat.atlassian.net/browse/OCPBUGS-58468
Companion operator PR: openshift/console-operator#1204

Solution description:

Phase 1: Shared encryption keys

Accept file-based encryption keys (provided by the console-operator via session-secret Secret) instead of generating random keys per process. Falls back to random generation when no key files are provided (backward compat with older operators).

Phase 2: Cookie-based session recovery

For OIDC auth (refresh tokens available):

• Store the actual encrypted refresh token in the cookie instead of a reference ID
• Existing recovery mechanism exchanges it with the OIDC provider for new tokens
• Backward-compatible fallback for old-format cookies

For OpenShift auth (no refresh tokens — OAuth server doesn't support refresh_token grant):

• New openshift-recovery-token cookie stores encrypted access token + expiry
• recoverSession() reads the token, validates expiry locally, creates a new server-side session
• No OAuth server interaction needed — fully transparent backend recovery

Security hardening

• Refresh token value removed from error log messages
• Refresh token revoked at OAuth server on logout
• Cookie MaxLength raised to 8192 for large OIDC JWT refresh tokens
• All type assertions use comma-ok guards
• Dead refreshTokenID field removed

Also includes #17121, a follow-up fix for the e2e session-persistence tests failing due to pre-loaded storageState.

Screenshots / screen recording:

Test setup:
Deploy cluster with htpasswd IdP, login, restart console pods, verify session persists.

Test cases:

• kubeadmin session persists across pod restart (verified on live cluster)
• htpasswd IDP user session persists across pod restart (verified on live cluster)
• Logout clears session, recovery cookie, and revokes tokens
• Old-format cookies still work via fallback
• E2e test: session survives pod deletion and plugin toggle

Browser conformance:

• Chrome
• Firefox
• Safari (or Epiphany on Linux)

Additional info:

• Requires companion operator PR: OCPBUGS-71237: Generate session-secret for all auth types console-operator#1204
• Based on Jefferson Ramos' solution exploration for OCPBUGS-71237

Reviewers and assignees:

🤖 Generated with Claude Code

[openshift-merge-bot]

Pipeline controller notification

This PR uses the pipeline controller for second-stage tests. Selection and triggering follow the repository configuration.

Use /test ? to list jobs and /pipeline help for commands.

[openshift-ci-robot]

@jhadvig: This pull request references Jira Issue OCPBUGS-71237, which is invalid:

• expected the bug to target either version "5.0.0." or "openshift-5.0.0.", but it targets "5.1.0" instead
• expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is Verified instead
• release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
• expected Jira Issue OCPBUGS-71237 to depend on a bug targeting a version in 5.1.0 and in one of the following states: MODIFIED, ON_QA, VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Backport of #16911 and #17121 to release-5.0. Clean cherry-pick, no conflicts.

---

Analysis / Root cause:

Console sessions are stored entirely in per-pod process memory. When a pod restarts (upgrades, scaling, OOM, operator reconciliation), all sessions are lost and users must re-authenticate. This affects all users but is most disruptive for clusters using external IdPs (htpasswd, LDAP) where re-authentication requires full credential entry.

Two architectural limitations cause this:

1. Cookie encryption keys are generated randomly per process — cookies from one pod can't be decrypted by another
2. The refresh token cookie stores only a reference ID pointing to an in-memory map, not the actual token

Jira: https://redhat.atlassian.net/browse/OCPBUGS-71237
Related: https://redhat.atlassian.net/browse/OCPBUGS-58468
Companion operator PR: openshift/console-operator#1204

Solution description:

Phase 1: Shared encryption keys

Accept file-based encryption keys (provided by the console-operator via session-secret Secret) instead of generating random keys per process. Falls back to random generation when no key files are provided (backward compat with older operators).

Phase 2: Cookie-based session recovery

For OIDC auth (refresh tokens available):

• Store the actual encrypted refresh token in the cookie instead of a reference ID
• Existing recovery mechanism exchanges it with the OIDC provider for new tokens
• Backward-compatible fallback for old-format cookies

For OpenShift auth (no refresh tokens — OAuth server doesn't support refresh_token grant):

• New openshift-recovery-token cookie stores encrypted access token + expiry
• recoverSession() reads the token, validates expiry locally, creates a new server-side session
• No OAuth server interaction needed — fully transparent backend recovery

Security hardening

• Refresh token value removed from error log messages
• Refresh token revoked at OAuth server on logout
• Cookie MaxLength raised to 8192 for large OIDC JWT refresh tokens
• All type assertions use comma-ok guards
• Dead refreshTokenID field removed

Also includes #17121, a follow-up fix for the e2e session-persistence tests failing due to pre-loaded storageState.

Screenshots / screen recording:

Test setup:
Deploy cluster with htpasswd IdP, login, restart console pods, verify session persists.

Test cases:

• kubeadmin session persists across pod restart (verified on live cluster)
• htpasswd IDP user session persists across pod restart (verified on live cluster)
• Logout clears session, recovery cookie, and revokes tokens
• Old-format cookies still work via fallback
• E2e test: session survives pod deletion and plugin toggle

Browser conformance:

• Chrome
• Firefox
• Safari (or Epiphany on Linux)

Additional info:

• Requires companion operator PR: OCPBUGS-71237: Generate session-secret for all auth types console-operator#1204
• Based on Jefferson Ramos' solution exploration for OCPBUGS-71237

Reviewers and assignees:

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 6694e6b8-7465-47ab-ba0f-81ecbac90a2b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jhadvig

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• cmd/OWNERS [jhadvig]
• frontend/OWNERS [jhadvig]
• pkg/OWNERS [jhadvig]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@jhadvig: This pull request references Jira Issue OCPBUGS-132619, which is invalid:

• release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Backport of #16911 and #17121 to release-5.0. Clean cherry-pick, no conflicts.

---

Analysis / Root cause:

Console sessions are stored entirely in per-pod process memory. When a pod restarts (upgrades, scaling, OOM, operator reconciliation), all sessions are lost and users must re-authenticate. This affects all users but is most disruptive for clusters using external IdPs (htpasswd, LDAP) where re-authentication requires full credential entry.

Two architectural limitations cause this:

1. Cookie encryption keys are generated randomly per process — cookies from one pod can't be decrypted by another
2. The refresh token cookie stores only a reference ID pointing to an in-memory map, not the actual token

Jira: https://redhat.atlassian.net/browse/OCPBUGS-71237
Related: https://redhat.atlassian.net/browse/OCPBUGS-58468
Companion operator PR: openshift/console-operator#1204

Solution description:

Phase 1: Shared encryption keys

Accept file-based encryption keys (provided by the console-operator via session-secret Secret) instead of generating random keys per process. Falls back to random generation when no key files are provided (backward compat with older operators).

Phase 2: Cookie-based session recovery

For OIDC auth (refresh tokens available):

• Store the actual encrypted refresh token in the cookie instead of a reference ID
• Existing recovery mechanism exchanges it with the OIDC provider for new tokens
• Backward-compatible fallback for old-format cookies

For OpenShift auth (no refresh tokens — OAuth server doesn't support refresh_token grant):

• New openshift-recovery-token cookie stores encrypted access token + expiry
• recoverSession() reads the token, validates expiry locally, creates a new server-side session
• No OAuth server interaction needed — fully transparent backend recovery

Security hardening

• Refresh token value removed from error log messages
• Refresh token revoked at OAuth server on logout
• Cookie MaxLength raised to 8192 for large OIDC JWT refresh tokens
• All type assertions use comma-ok guards
• Dead refreshTokenID field removed

Also includes #17121, a follow-up fix for the e2e session-persistence tests failing due to pre-loaded storageState.

Screenshots / screen recording:

Test setup:
Deploy cluster with htpasswd IdP, login, restart console pods, verify session persists.

Test cases:

• kubeadmin session persists across pod restart (verified on live cluster)
• htpasswd IDP user session persists across pod restart (verified on live cluster)
• Logout clears session, recovery cookie, and revokes tokens
• Old-format cookies still work via fallback
• E2e test: session survives pod deletion and plugin toggle

Browser conformance:

• Chrome
• Firefox
• Safari (or Epiphany on Linux)

Additional info:

• Requires companion operator PR: OCPBUGS-71237: Generate session-secret for all auth types console-operator#1204
• Based on Jefferson Ramos' solution exploration for OCPBUGS-71237

Reviewers and assignees:

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[jhadvig]

/jira refresh

[openshift-ci-robot]

@jhadvig: This pull request references Jira Issue OCPBUGS-132619, which is valid.

7 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
• release note text is set and does not match the template
• dependent bug Jira Issue OCPBUGS-71237 is in the state Verified, which is one of the valid states (MODIFIED, ON_QA, VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
• dependent Jira Issue OCPBUGS-71237 targets the "5.1.0" version, which is one of the valid target versions: 5.1.0
• bug has dependents

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[jhadvig]

OCPBUGS-132619 Session Persistence - Verification Results

Tested on a live cluster (ci-ln-r0b2cgk-72292) built from this PR + console-operator PR openshift/console-operator#1247.

Test	Description	Result
Test 1	kubeadmin session persists through console pod restart	✅ PASSED
Test 2	htpasswd user session persists through console pod restart	✅ PASSED
Test 3	OAuth logout clears session, no ghost recovery	✅ PASSED
Test 4	Keycloak OIDC setup (realm, client, testuser)	✅ COMPLETED
Test 5	OIDC testuser session persists through console pod restart	✅ PASSED
Test 6	OIDC logout clears both console session and IdP session	✅ PASSED

Test details

• Session persistence (Tests 1, 2, 5): Logged in, verified dashboard loads, deleted all console pods via oc delete pods -n openshift-console -l component=ui, waited for rollout, reloaded the page. Dashboard loaded without re-login in all cases.
• Logout (Tests 3, 6): After clicking Log out, navigating back to the console URL redirected to the login page. No ghost session recovery. For OIDC (Test 6), logout also cleared the Keycloak IdP session (redirected to Keycloak login, not auto-logged in via SSO).
• htpasswd setup (Test 2): Configured htpasswd IDP via oauth/cluster.
• OIDC setup (Test 4): Deployed Keycloak on the cluster, configured authentication.config.openshift.io with type: OIDC, waited for kube-apiserver rollout across all 3 nodes.

All tests passed 👍

[jhadvig]

/verified by @jhadvig

[openshift-ci-robot]

@jhadvig: This PR has been marked as verified by @jhadvig.

Details

In response to this:

/verified by @jhadvig

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[jhadvig]

/pipeline required

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-gcp-console
/test e2e-gcp-console-techpreview

[openshift-ci]

@jhadvig: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[Leo6Leo]

the backport itself looks good, but it carries a logout bug from main that I reproduced on a live cluster

Bug (inherited from main): logout recreates the session it just deleted.
pkg/auth/oauth2/auth_openshift.go, logout():

1. DeleteSession deletes the server-side session and emits an expiring Set-Cookie for the session-token cookie.
2. getLoginState then runs. The server lookup is nil, OpenShift auth has no refresh token, so it falls into recoverSession.
3. recoverSession reads openshift-recovery-token from the request (still present; only the response header cleared it), calls AddSession, creates a fresh server-side session and emits a second Set-Cookie for the same session-token cookie name/path, with a value.
4. Revocation runs against the recovered login state, then 204.

Reproduced on a 5.1.0-ec.1 cluster as well (console built from a commit that includes #16911; operator providing shared keys; single pod served the whole sequence):

• POST /api/console/logout → 204, no error logged.
• The session's OAuthAccessToken was deleted (revocation ran).
• 28 ms later, a request through the console proxy still returned 200 as kube:admin. It kept returning 200 for ~2 min (API server token cache), then returned a Kubernetes Status 401 body, i.e. the console was still proxying for that browser's cookies. A correct logout produces the console's own empty 401 from the auth middleware, not a proxied one.
• Side effect observed with the UI open: once the cache expired, the frontend's 401 handling triggered a second logout and then silently re-authenticated via the still-valid OAuth server session. A kube:admin user may never notice they were logged out.

Do we wanna create a separate issue and have a new PR targeting it? If so, I can give LGTM to this PR

The reproduction report is generated with the assistance of Claude.

[jhadvig]

@Leo6Leo please create a separate issue to track the logout bug 🙏
Would like to get this one in, fix the issue on main, verify it and then backport, is that ok with you :)