celery-worker startup probe failures on aurora-prod (suspected root cause of May 14 Pre-Discovery activation outage) #429

Summary

Repeated celery-worker startup probe failures observed in the aurora namespace on the production GKE cluster (aurora-prod, us-west1, project aurora-saas-prod, Helm release aurora-oss).

Strongly suspected to be the root cause of the May 14 "Pre-Discovery dead for all 8 orgs" activation bug, which we'd previously narrowed down to a likely celery-beat → celery-worker dispatch failure but couldn't fully confirm.

How this surfaced

While dogfooding Aurora's MCP to investigate Xav & Mario's usage (after my local gcloud token expired and kubectl couldn't refresh non-interactively), the cross-account investigation repeatedly flagged celery-worker startup probe failures in the aurora namespace as a strong signal. We weren't able to fully confirm hard numbers because Aurora's chat-fanout synthesis step returned empty (tracked separately — that's an app-code bug, not infra).

Environment

• GCP project: aurora-saas-prod
• GKE cluster: aurora-prod (us-west1)
• kubectl context: gke_aurora-saas-prod_us-west1_aurora-prod
• Namespace: aurora
• Helm release: aurora-oss (chart aurora-oss-1.2.15)
• Current image tag: sha-7870a59
• Affected workload: celery-worker (1 replica)

What we need to confirm

• Exact startup probe error message and failure mode (timeout vs. failed exec vs. HTTP non-200)
• Restart count and CrashLoopBackOff state of the celery-worker pod
• Whether the worker is failing to boot (import error, broker connect failure) or just slow to become ready (probe initialDelaySeconds / failureThreshold too aggressive)
• Connectivity to Redis (Memorystore) and Postgres (Cloud SQL) from the worker pod
• Recent image / Helm values / chart changes that correlate with onset
• Whether celery-beat is dispatching tasks that the worker never picks up (queue depth in Redis)

Hypotheses to validate

1. Probe misconfiguration — startup probe too tight for current worker boot time (slow imports, DB migrations on boot, etc.)
2. Broker / DB connectivity — worker can't reach Redis or Postgres on startup, probe never goes green
3. Bad image / config in sha-7870a59 — regression in worker entrypoint or celery config
4. Resource starvation — OOMKill or CPU throttling during boot

Related

• Aurora RCA incident: 97867bdc-0b82-4552-8503-d39430be7381 (session 39bca4aa-4ec9-4b41-abf4-55e99c26d554) — full automated investigation running in background, will produce a postmortem.
• Original symptom: May 14 — Pre-Discovery dead for all 8 orgs (activation pipeline never ran).

Next steps

1. Wait for Aurora RCA postmortem to complete and link findings here.
2. Pull hard data from cluster: kubectl describe pod -n aurora -l app=celery-worker, kubectl logs --previous, restart counts.
3. Check Redis queue depth + celery-beat dispatch logs for the May 14 window.
4. Once confirmed, ship fix (probe tuning, image rollback, or config fix) via Helm upgrade per the documented prod deploy process.

/cc activation, infra