Skip to content

ate-setup: reconcile ate-api-server ConfigMaps and roll out on config changes - #3370

Open
Mofi Rahman (moficodes) wants to merge 1 commit into
agent-substrate:mainfrom
moficodes:fix/3271-auth-configmap-reconcile-hash
Open

Mofi Rahman (moficodes) wants to merge 1 commit into
agent-substrate:mainfrom
moficodes:fix/3271-auth-configmap-reconcile-hash

Conversation

@moficodes

@moficodes Mofi Rahman (moficodes) commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

EnsureAPIServerPrerequisites created ConfigMap/ate-api-authentication and ConfigMap/ate-api-config only when missing (unless --ateapi-log-level was passed), ignoring --ateapi-expected-jwt-issuer on re-deploy and leaving legacy or unknown 1.x rollback fields in place where ate-api-server's strict YAML/prototext loaders would CrashLoopBackOff. Additionally, Deployment/ate-api-server's ate.dev/env-hash annotation only digested environment variables rather than the mounted ate-api-authentication and ate-api-config ConfigMaps.

Fixes #3271

Summary

EnsureAPIServerPrerequisites
  ensureAPIAuthenticationConfig -> ateapiauth.ParseAuthenticationConfig
    (rewrites when missing, when --ateapi-expected-jwt-issuer is set, or when stored YAML fails strict schema validation; preserves valid custom issuers and extra jwtProviders)
  ensureAPIConfig -> apiconfig.ParseTextProto
    (rewrites when missing, when --ateapi-log-level is set, or when stored prototext fails strict schema/level validation; preserves valid customized log levels)

DeployAPIServer / UpdateAPIAuthenticationConfig / UpdateAPIConfig
  apiServerEnvHash(env, ate-api-authentication, ate-api-config) -> annotates Deployment/ate-api-server pod template (ate.dev/env-hash)

Evidence

  • Before: Re-running ate-setup deploy ate-system --ateapi-expected-jwt-issuer <issuer> skipped ConfigMap/ate-api-authentication, and upgrading across schema removals or rolling back 1.x -> 1.0.0 with unknown fields in authentication.yaml or apiconfig.textproto left ate-api-server in CrashLoopBackOff.
  • After: Valid customized ConfigMaps (custom JWT issuers, extra external jwtProviders, hand-edited log levels) survive ate-setup deploy untouched when flags are unset; explicit flags or invalid/unknown rollback fields rewrite the ConfigMaps cleanly and update ate.dev/env-hash, rolling Deployment/ate-api-server to Ready.

Merge Danger

Door: Two-way

Blast Radius: Small (internal/ateapiauth, internal/apiconfig, and cmd/ate-setup prerequisite reconciliation and ate.dev/env-hash computation)

Rationale

api-server crashes on any config field it doesn't recognize, and ate-setup never touched those config maps once they existed. the PR makes ate-setup rewrite the config maps when the server can't parse it, and keeps valid custom ones.

However this need to be in 1.0,0 otherwise if 1.0.x adds a config field, then someone rollback from 1.0.x to 1.0.0, the old server will fail the rollback as it cannot parse the yaml.

AI use: AI-assisted implementation and unit tests; reviewed and verified by Mofi Rahman (@moficodes) on Kind and GKE.

  • Tests pass
  • Docs updated

@thompsonmax

Copy link
Copy Markdown
Collaborator

I think we need to get this in for 1.0.0. As Haven Xia (@HavenXia) mentioned offline, this may break if a user does a rollback to 1.0.0 from 1.x.

Comment thread cmd/ate-setup/internal/steps/prereqs.go Resolved
Comment thread cmd/ate-setup/internal/steps/prereqs.go Resolved
@thompsonmax Max Thompson (thompsonmax) added this to the M3 milestone Oct 29, 2026
@thompsonmax Max Thompson (thompsonmax) added kind/bug Something isn't working / bugfixes prio/P0 Highest priority / required for next milestone labels Oct 29, 2026
…e-api-server on change

EnsureAPIServerPrerequisites previously created ConfigMap/ate-api-authentication
only when it was missing. On cluster upgrades with older schema configurations,
or when operators provided --ateapi-expected-jwt-issuer, existing ConfigMaps
were kept unmodified and flag overrides were ignored. Additionally, the
deployment env-hash only digested environment variables, leaving running
ate-api-server pods on stale authentication and server configurations.

Reconcile ate-api-authentication when missing, when an expected JWT issuer is
explicitly configured, or when the stored configuration fails schema
validation (such as after an upgrade changes the config schema or a rollback
leaves unsupported fields). Valid custom authentication configurations with
extra JWT providers are preserved. Similarly, ensureAPIConfig validates
stored apiconfig.textproto payloads and rewrites invalid configurations to
defaults while preserving valid customized log levels. Fold both
ate-api-authentication and ate-api-config into the deployment env-hash so
configuration changes trigger a rolling update of Deployment/ate-api-server.

Signed-off-by: Mofi Rahman <mofi@google.com>
@moficodes
Mofi Rahman (moficodes) force-pushed the fix/3271-auth-configmap-reconcile-hash branch from a88c65b to 2f4b82b Compare October 29, 2026 22:57
@moficodes Mofi Rahman (moficodes) changed the title ate-setup: reconcile ate-api-authentication ConfigMap and roll out ate-api-server on config changes ate-setup: reconcile ate-api-server ConfigMaps and roll out on config changes Oct 29, 2026
@HavenXia

Haven Xia (HavenXia) commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

[From standup] Here's a more straightforward explanation for reviewer to understand. Tim Hockin (@thockin)

api-server crashes on any config field it doesn't recognize, and ate-setup never touched those config maps once they existed. the PR makes ate-setup rewrite the config maps when the server can't parse it, and keeps valid custom ones.

However this need to be in 1.0.0, otherwise if 1.0.x adds a config field, then someone rollback from 1.0.x to 1.0.0, the old server will fail the rollback as it cannot parse the yaml.

@thockin

Copy link
Copy Markdown
Collaborator

I did not get to this one yet, but I will either later today or this weekend

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/bug Something isn't working / bugfixes prio/P0 Highest priority / required for next milestone

Projects

None yet

4 participants