You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Provider refresh fails for an hour after a gateway restart in operator workspace mode #4526
In Kubernetes operator workspace mode, a provider credential refresh that is due when the gateway starts fails with:
provider credential refresh errored ... strategy=oauth2_client_credentials status=configuration_required
next_refresh_at_ms=... seconds_until_refresh=3600
error=code: 'The caller does not have permission to execute the specified operation',
message: "workspace 'oce-31f14cd55f071a7' is not in the operator namespace allowlist"
The namespace carries the operator label the whole time, and the same workspace keeps working for Sandboxes and static providers after the restart. Only refreshes that run in the first sweep fail. Each one is recorded as configuration_required with refresh_configuration_invalid, so its next attempt is an hour later. When the current access token expires, Sandboxes get no token for that provider until then, or until someone calls RotateProviderCredential.
Why
On v0.1.3-pre.2, and unchanged on main as far as I can see:
KubernetesComputeDriver::new creates an empty OperatorNamespaceAllowlist and fills it from spawn_namespace_label_watcher in the background (crates/openshell-driver-kubernetes/src/driver.rs).
spawn_refresh_worker starts a tokio::time::interval, whose first tick fires immediately (crates/openshell-server/src/provider_refresh.rs, started from lib.rs).
The first sweep reaches the workspace check before the watcher has listed the labeled namespaces, so the check answers PermissionDenied (grpc.rs, config.rs: "is not in the operator namespace allowlist").
impl From<Status> for RefreshFailure maps PermissionDenied to fix_configuration("refresh_configuration_invalid"), which schedules the retry REFRESH_CONFIGURATION_RETRY_SECONDS (3600 s) later.
In the same run, a refresh-token grant and a second client-credentials grant failed the same way at 03:57:06.94–.97, about 17 ms after provider credential refresh worker started. Sources that were not due in that first sweep later refreshed normally.
Reproduce
Run the gateway with server.drivers.kubernetes.workspaceMode=operator and operatorNamespaceLabel set.
Configure an OAuth2 client-credentials refresh on a provider in a labeled workspace and let it mint.
Scale the gateway StatefulSet to 0, wait past the provider's next_refresh_at, and scale it back to 1.
GetProviderRefreshStatus reports configuration_required / refresh_configuration_invalid with next_refresh_at an hour later.
Possible fixes
Hold the refresh worker's first sweep, or workspace-scoped refreshes, until the operator allowlist watcher has completed its initial list.
Or treat a workspace missing from a not-yet-synced allowlist as retryable (Unavailable, 60 s) rather than a configuration error.
Found by the OpenClaw Enterprise integration suite, which restarts the gateway to test outages; it currently forces a rotation to recover.
Activity
fix(driver-kubernetes): wait for operator namespace listing before wo…
test(openshell): prove model, tool and gateway outages fail closed (#…