Skip to content

Re-verify issue #209 (bare-404 Host-header routing) against current staging gateway ingress #249

Open
Open
Re-verify issue #209 (bare-404 Host-header routing) against current staging gateway ingress#249

Description

@jameswnl
jameswnl
opened on Sep 1, 2026

Summary

Issue #209 (already tracked in code -- see OpenShellSpawner._wait_ready_with_host()'s _INGRESS_MISMATCH_STREAK_LIMIT mechanism and its docstring) describes a gateway ingress that doesn't support Host-header-based HTTP routing to sandbox ports on its main TLS port, causing 5 consecutive bare 404s (no body, no content-type) and an early, diagnostic RuntimeError instead of the full readiness timeout.

This was observed live twice against the staging gateway (<staging-gateway-host>:443) via lightspeed-stack's workflow-ephemeral and workflow-ephemeral-approval demo scenarios, both with the current (post-#238/#244/#246) fixed cloud_agents code:

RuntimeError: Gateway returned a bare 404 (no body, no content-type) 5 times in a row for
Host 'default--intimate-bowfin.openshell.localhost' at
https://<staging-gateway-host>:443/health.

In both observed runs, the sandbox successfully reached CreateSandbox -> exec'd the server -> ExposeService, and only then failed the Host-header health check -- i.e. this is a later-stage failure than plain provisioning flakiness (see #248 for that separate issue), and matches the exact fingerprint _wait_ready_with_host()'s bare-404 detection was built for.

What's NOT yet confirmed

A follow-up attempt to independently re-reproduce this (spawn a fresh bare sandbox, manually curl /health with the correct vs. an incorrect Host header to directly compare the ingress's behavior) did not get far enough to test it: the fresh sandbox got stuck in SANDBOX_PHASE_PROVISIONING for 60s+ before ever reaching a state where the HTTP health check could be attempted (see #248). So the current live behavior of staging's ingress for Host-header routing has not been independently re-confirmed in this pass -- only the original two occurrences (with full tracebacks matching #209's known fingerprint) are the evidence so far.

Suggested next steps

  • Once Staging gateway: sandboxes intermittently stuck in Provisioning phase for 60s+ (not an image-arch issue) #248's provisioning flakiness allows a sandbox to reliably reach READY, manually compare GET /health with the correct Host: default--<sandbox>.openshell.localhost header vs. an obviously wrong Host header vs. no Host header, on the current staging gateway, to directly confirm whether the ingress is still ignoring the Host header (vs. some other transient cause).
  • If confirmed, determine whether staging exposes a separate HTTP ingress route for sandbox traffic (distinct from the main gRPC/TLS port) that OpenShellSpawner's http_endpoint parameter could be pointed at instead -- the error message itself suggests this workaround, but it hasn't been verified whether such a route actually exists for this specific staging deployment.
  • If no such route exists, this may require staging's ingress config to be updated to support Host-header-based virtual routing on its main port, matching the working configurations already confirmed on local Kind and (per the original OpenShellSpawner: unnecessary direct LLM egress when using a custom provider URL (fixed in #220) #209 investigation) a real OCP cluster.

Activity

  1. jameswnl commented on Sep 1, 2026

    @jameswnl
    jameswnl
    on Sep 1, 2026
    CollaboratorAuthor

    Update: re-confirmed live against the current staging gateway, "not yet confirmed" section resolved

    Just reproduced this end-to-end against <staging-gateway-host>:443 (the live gateway named above), with a new diagnostic script that isolates exactly this question: #250 adds scripts/gateway-verification/diagnose_host_header_routing.py.

    What it does

    1. Creates a bare sandbox and polls readiness via the raw gRPC stub (see below for why).
    2. Starts the real lightspeed_agentic.app server inside it.
    3. Confirms the app itself is healthy with a direct in-sandbox curl 127.0.0.1:8080/health (ruling out an app-level failure).
    4. Calls ExposeService, then issues three gateway-proxied GET /health requests: correct virtual Host header, an obviously-wrong Host header, and no Host header at all.

    Result (this run)

    • In-sandbox curl 127.0.0.1:8080/health → 200 — app confirmed healthy.
    • Gateway-proxied, correct Host → bare 404 (no body, no content-type).
    • Gateway-proxied, wrong Host → identical bare 404.
    • Gateway-proxied, no Host → identical bare 404.

    All three are indistinguishable. This resolves the "What's NOT yet confirmed" section above: it's not that the ingress might be routing to some other sandbox or behaving inconsistently — Host-header-based HTTP routing to sandbox ports isn't functioning at all on this gateway's main TLS port right now, for any Host value. The original #209 fingerprint (bare 404, no body/content-type) is confirmed as the current, live behavior of this specific staging deployment, not a stale/transient observation from the earlier #238/#244 investigation.

    This also directly answers the next step this issue proposed: since Host-header multiplexing on the main port is confirmed non-functional (not just flaky), the path forward is checking whether this gateway exposes a separate HTTP ingress route for sandbox traffic that OpenShellSpawner's http_endpoint param could point at instead, or getting the ingress config fixed to support Host-header virtual routing on the main port (matching local Kind and real OCP, which both already work per the original #209 investigation).

    Bonus, relevant to #248

    While reproducing this, none of my spawns actually got stuck in Provisioning (9–18s to Ready each time) — but I found that SandboxStatus.conditions (a repeated, Kubernetes-condition-shaped {type, status, reason, message, last_transition_time}) is already populated by the gateway with real diagnostic detail during Provisioning (observed: reason=DependenciesNotReady, message="Pod exists with phase: Pending"). The Python SDK's SandboxStatusRef (openshell.sandbox._sandbox_ref()) silently drops this field, exposing only phase/current_policy_version/exit_code — confirmed still true in both the pinned SDK (0.0.111) and the unreleased ~/ws/Openshell source. That's likely why #248's repro saw "no message field populated at any point" — it's a client-visibility gap, not proof the gateway is silent. The new script bypasses this by reading conditions off the raw GetSandbox stub response directly, so a future stuck-in-Provisioning repro should show the actual reason without needing cluster-admin access.

    Script: #250

  2. jameswnl
    added 2 commits that reference this issue on Sep 1, 2026

    Add live-gateway diagnostic script for issue #249 (bare-404 Host-head…

    2e3e0bc

    Add live-gateway diagnostic script for issue #249 (bare-404 Host-head…

    76a646b

  3. jameswnl commented on Sep 2, 2026

    @jameswnl
    jameswnl
    on Sep 2, 2026
    CollaboratorAuthor

    Re-verified against the prod gateway (openshell-prod) — routing WORKS here, opposite of staging

    The prior evidence in this issue (and #250) was gathered against the staging gateway. I re-ran the same #250 diagnostic (scripts/gateway-verification/diagnose_host_header_routing.py, unchanged) against the prod gateway openshell-prod-openshell-prod.apps.rosa.df92w-mrmzx-si5.jryo.p3.openshiftapps.com:443. Result: Host-header routing is functioning — the #209/#249 bare-404 fingerprint does NOT reproduce on this deployment.

    Result

    in-sandbox curl 127.0.0.1:8080/health   -> 200
ExposeService virtual_host: default--noble-ibis.openshell.localhost
[correct Host] 200  content-type=application/json  body={"status":"ok"}
[wrong Host]   404  content-type=text/plain         body="Service endpoint not found"
[no Host]      404  content-type=None               body=""
VERDICT: Host-header routing is WORKING -- correct Host reached the app.

    Contrast with staging, where correct/wrong/no Host all returned an identical bare 404 (no body, no content-type). On prod:

    • Correct Host reaches the app (200 {"status":"ok"}).
    • Wrong Host returns a real application-level 404 (text/plain, Service endpoint not found) — proving the ingress is genuinely doing Host-based virtual routing, not blindly 404ing.
    • No-Host is the only bare 404, which is expected.

    This confirms #209/#249 is a per-deployment ingress-config issue, not universal: it reproduces on staging but not on prod (which matches the working local-Kind and real-OCP configs from the original #209 investigation).

    Two setup gaps hit while connecting (may bite others / the spawner)

    1. openshell-cli is a public OIDC client. The client-credentials grant is rejected ("Public client not allowed to retrieve service account"), so OidcClientCredentialsTokenProvider / the diagnostic's client-credentials path can't authenticate to this realm. I used the CLI's cached ~/.config/openshell/gateways/ocp-prod/oidc_token.json and minted fresh access tokens via the refresh_token grant (300s access-token TTL, so re-minted per run).
    2. Gateway serves a self-signed cert (CN=openshell-prod-external, issuer CN=openshell-selfsigned-ca). certifi can't verify it (CERTIFICATE_VERIFY_FAILED), and the openshell CLI hits the same wall (invalid peer certificate: UnknownIssuer) — i.e. the CLI currently can't talk to this gateway over TLS. I extracted the CA from the cluster secret openshell-prod/openshell-prod-server-external-tls (.data.ca\.crt), which verifies the gateway cert (return code 0), and passed it as the TLS CA.

    Side notes

  4. jameswnl commented on Sep 2, 2026

    @jameswnl
    jameswnl
    on Sep 2, 2026
    CollaboratorAuthor

    Conclusion: this is a per-deployment (ingress) issue, not a gateway/code defect

    Reconfirmed on a second, independent prod cluster (openshell-prod-openshell-prod.apps.rosa.xpit3-4ibom-49z.pddx.p3.openshiftapps.com), same #250 diagnostic, unchanged:

    • correct Host → 200 {"status":"ok"}
    • wrong Host → real app-level 404 text/plain "Service endpoint not found" (genuine Host-based virtual routing)
    • no Host → bare 404 (expected)

    Also validated the full path end-to-end on this cluster — workflow-ephemeral-approval spawned sandboxes to Ready and the gateway proxied /health correctly.

    Summary across deployments:

    Deployment Host-header routing
    staging ✗ bare 404 for all Host values (the #209/#249 fingerprint)
    prod cluster A (df92w…) ✓ works
    prod cluster B (xpit3…) ✓ works

    So the bare-404 is a property of the staging ingress, not the OpenShell gateway or lightspeed-cloud-agents. The http_endpoint= fallback the error message suggests remains the right escape hatch for any ingress that can't do Host-header routing on its TLS port. Recommend tracking the staging ingress fix separately and closing this verification issue.

  5. jameswnl commented on Sep 2, 2026

    @jameswnl
    jameswnl
    on Sep 2, 2026
    CollaboratorAuthor

    Root cause traced and filed upstream: openshift-online/hypershell#234.

    Neither of HyperShell's ingress modes (route or gateway-api) provisions any routing for OpenShell's per-sandbox ExposeService HTTP virtual-hosting — only a single fixed hostname per gateway ever gets created, pointed at the gRPC port. OpenShell's HTTP-exposure feature depends on the raw TLS connection reaching the gateway pod so it can demux internally by Host header, but a client hitting the per-sandbox hostname presents that hostname as the TLS SNI, which doesn't match the platform router's one registered Route/GRPCRoute hostname — so the platform router itself returns the bare 404 (no body/content-type) before the request ever reaches the OpenShell gateway.

    This confirms the ingress is not "flaky" or transiently misconfigured on staging — it's a structural gap in how the staging gateway is provisioned (by HyperShell), not something fixable on the cloud_agents/OpenShellSpawner side, and not an OpenShell core bug either. See hypershell#234 for the full trace (file/line citations into both reconcilePassthroughRoute and reconcileGatewayAPIResources).

    Relevant to #221's scoping decision: since the platform-side fix has unclear timeline/ownership (hypershell#234 is newly filed, and the adjacent #220 there is closed without adding wildcard-hostname support), #221's "full replacement" option (gRPC ForwardTcp tunnel as the sole transport, bypassing Host-header routing entirely) looks like the more durable path forward rather than waiting on upstream ingress changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions