Skip to content

Provider refresh fails for an hour after a gateway restart in operator workspace mode #4526

Closed

Description

@mrunalp

What happens

In Kubernetes operator workspace mode, a provider credential refresh that is due when the gateway starts fails with:

provider credential refresh errored ... strategy=oauth2_client_credentials status=configuration_required
next_refresh_at_ms=... seconds_until_refresh=3600
error=code: 'The caller does not have permission to execute the specified operation',
message: "workspace 'oce-31f14cd55f071a7' is not in the operator namespace allowlist"

The namespace carries the operator label the whole time, and the same workspace keeps working for Sandboxes and static providers after the restart. Only refreshes that run in the first sweep fail. Each one is recorded as configuration_required with refresh_configuration_invalid, so its next attempt is an hour later. When the current access token expires, Sandboxes get no token for that provider until then, or until someone calls RotateProviderCredential.

Why

On v0.1.3-pre.2, and unchanged on main as far as I can see:

  • KubernetesComputeDriver::new creates an empty OperatorNamespaceAllowlist and fills it from spawn_namespace_label_watcher in the background (crates/openshell-driver-kubernetes/src/driver.rs).
  • spawn_refresh_worker starts a tokio::time::interval, whose first tick fires immediately (crates/openshell-server/src/provider_refresh.rs, started from lib.rs).
  • The first sweep reaches the workspace check before the watcher has listed the labeled namespaces, so the check answers PermissionDenied (grpc.rs, config.rs: "is not in the operator namespace allowlist").
  • impl From<Status> for RefreshFailure maps PermissionDenied to fix_configuration("refresh_configuration_invalid"), which schedules the retry REFRESH_CONFIGURATION_RETRY_SECONDS (3600 s) later.

In the same run, a refresh-token grant and a second client-credentials grant failed the same way at 03:57:06.94–.97, about 17 ms after provider credential refresh worker started. Sources that were not due in that first sweep later refreshed normally.

Reproduce

  1. Run the gateway with server.drivers.kubernetes.workspaceMode=operator and operatorNamespaceLabel set.
  2. Configure an OAuth2 client-credentials refresh on a provider in a labeled workspace and let it mint.
  3. Scale the gateway StatefulSet to 0, wait past the provider's next_refresh_at, and scale it back to 1.
  4. GetProviderRefreshStatus reports configuration_required / refresh_configuration_invalid with next_refresh_at an hour later.

Possible fixes

  • Hold the refresh worker's first sweep, or workspace-scoped refreshes, until the operator allowlist watcher has completed its initial list.
  • Or treat a workspace missing from a not-yet-synced allowlist as retryable (Unavailable, 60 s) rather than a configuration error.

Found by the OpenClaw Enterprise integration suite, which restarts the gateway to test outages; it currently forces a rotation to recover.

Activity

  1. added a commit that references this issue on Oct 14, 2026
    8b774e2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions