work-unit: control plane — process phase, drain, and probe truth #1413

Activity

1. strawgate

   added

   enhancementNew feature or requestNew feature or request

   productionRequired for production readinessRequired for production readiness

   P2: mediumMinor bug, missing validation, metric inaccuracyMinor bug, missing validation, metric inaccuracy

   work-unitScheduling ticket for one agent runScheduling ticket for one agent run

   architectureSystemic design improvements that eliminate classes of bugsSystemic design improvements that eliminate classes of bugs

   on Apr 6, 2026
2. github-actions commented on Apr 6, 2026

   

   github-actionsBot

   on Apr 6, 2026 – with GitHub Actions

   Contributor

   More actions

   tl;dr: Outcome A — Ready for implementation. The current probe plane does not model process/pipeline phase truth yet (/ready is only !pipelines.is_empty()), so this can be implemented mechanically by adding explicit phase state and wiring existing pipeline drain transitions into diagnostics.

   Recommendation

   Implement this slice now as explicit phase-backed probes: add shared phase types in logfwd-types, wire runtime transitions from Pipeline::run_async, expose /live (and keep /health compatibility), and make /ready phase-derived (serving only).

   Findings

   • Probe routing is currently /health + /ready, no /live: crates/logfwd-io/src/diagnostics.rs:485-488
   • /health is unconditional process-ok payload: crates/logfwd-io/src/diagnostics.rs:532-541
   • /ready is only pipeline-presence: crates/logfwd-io/src/diagnostics.rs:553-555
   • Ready semantics explicitly ignore freshness but do not include drain/shutdown phase truth: crates/logfwd-io/src/diagnostics.rs:549-552
   • Diagnostics model has counters/timings but no process/pipeline lifecycle phase field: crates/logfwd-io/src/diagnostics.rs:36-91, 1001-1027
   • logfwd-types diagnostics currently only exports ComponentStats (no phase/readiness model): crates/logfwd-types/src/diagnostics.rs:10-118
   • Lifecycle typestates already exist in pipeline machine: crates/logfwd-types/src/pipeline/lifecycle.rs:19-26, transitions at 153-163, 280-310
   • Pipeline performs drain/stop transition on shutdown but does not publish it to diagnostics: crates/logfwd/src/pipeline.rs:721-759
   • Main registers diagnostics pipelines before run_async begins: crates/logfwd/src/main.rs:935-937, run starts 991-998
   • Known race documented: /ready can be 200 before receiver sockets bind: DEVELOPING.md:195-197
   • Current probe tests cover only empty/non-empty pipeline readiness: crates/logfwd-io/src/diagnostics.rs:1634-1666

   Verification

   I ran current ready-endpoint tests to confirm baseline behavior:

   $ cargo test -p logfwd-io test_ready_endpoint_no_pipelines_returns_503 -- --nocapture
   test diagnostics::tests::test_ready_endpoint_no_pipelines_returns_503 ... ok
   
   $ cargo test -p logfwd-io test_ready_endpoint_with_pipeline_returns_200 -- --nocapture
   test diagnostics::tests::test_ready_endpoint_with_pipeline_returns_200 ... ok
   

   These confirm today’s readiness contract is purely pipeline registration presence.

   Detailed Action Plan (mechanical spec)

   1. Add shared phase model in crates/logfwd-types/src/diagnostics.rs.

      • Current file content: only pub struct ComponentStats and impls (:10-118).
      • Add:
        • pub enum ProcessPhase { Startup, Serving, Draining, Shutdown, Stopped }
        • pub enum PipelinePhase { Starting, Serving, Draining, Stopped }
        • helper methods for stable wire strings + readiness/liveness truth (is_ready, is_live semantics)
      • Keep this crate as the shared source of truth for diagnostics state vocabulary.

   2. Extend diagnostics runtime state in crates/logfwd-io/src/diagnostics.rs.

      • Current DiagnosticsServer struct: pipelines, start_time, etc. (:365-380) but no phase field.
      • Add fields + APIs:
        • process phase store (atomic or lock-protected enum representation)
        • per-pipeline phase store keyed by pipeline name (or alongside PipelineMetrics if preferred)
        • mutator methods used by runtime (set_process_phase, set_pipeline_phase)

   3. Wire probe endpoints to explicit truth in crates/logfwd-io/src/diagnostics.rs.

      • Routing change: add /live in match table near :485-493.
      • Keep /health compatibility: either alias to serve_live or preserve endpoint with same liveness semantics.
      • Change serve_ready (:553-567): derive status code/body from process phase (200 only when Serving; 503 in Startup/Draining/Shutdown/Stopped).

   4. Expose phase in status JSON in crates/logfwd-io/src/diagnostics.rs.

      • Current /api/pipelines payload build: :1001-1027 has counters but no lifecycle phase.
      • Add fields:
        • process-level phase under system
        • per-pipeline phase under each pipeline object
        • optionally derived ready boolean for operator clarity

   5. Emit runtime phase transitions from pipeline execution in crates/logfwd/src/pipeline.rs.

      • Current shutdown transition exists internally: :721-759.
      • Add explicit phase publish points:
        • entering run loop (run_async start, around :569) => pipeline Serving
        • shutdown signal branch (:635-637) => pipeline Draining
        • post-drain stop/force-stop completion (:721-759) => pipeline Stopped
      • Apply to both graceful and forced stop paths.

   6. Set process-level phase from startup/shutdown orchestration in crates/logfwd/src/main.rs.

      • Current diagnostics server construction: :931-941; pipeline run launch: :991-998.
      • Add process phase updates:
        • startup before pipeline tasks begin
        • serving once pipelines are running
        • draining/shutdown on cancellation path
        • stopped on final exit path

   7. Tests to write/update

      • crates/logfwd-io/src/diagnostics.rs:
        • new /live test (200 in all non-dead process phases)
        • readiness matrix test: startup/draining/shutdown/stopped => 503; serving => 200
        • /api/pipelines includes phase fields
      • crates/logfwd/src/pipeline.rs:
        • phase transition order on shutdown path (serving → draining → stopped)
        • forced-stop path still marks stopped phase and preserves existing drain invariants

   8. Verification commands

      • cargo fmt --all
      • cargo clippy --all-targets -- -D warnings
      • cargo test -p logfwd-io diagnostics::tests::
      • cargo test -p logfwd pipeline::tests::
      • just ci

   Related Items

   Type	Link	Relevance
   Issue	#1413	This work-unit
   Issue	#1303	Probe/control-plane contract target
   Issue	#1306	Shared lifecycle/readiness substrate target
   PR	#1402	Touches crates/logfwd/src/pipeline.rs (potential merge conflict area)
   File	crates/logfwd-io/src/diagnostics.rs	Probe handlers and diagnostics payloads
   File	crates/logfwd/src/pipeline.rs	Drain/shutdown transitions to wire into diagnostics truth
   File	crates/logfwd-types/src/diagnostics.rs	Shared diagnostics phase model home
   File	crates/logfwd-types/src/pipeline/lifecycle.rs	Existing lifecycle typestate source
   Doc	DEVELOPING.md:195-197	Existing readiness race evidence

   Note

   🔒 Integrity filtering filtered 6 items

   Integrity filtering activated and filtered the following items during workflow execution.
   This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.

   • issue:work-unit: control plane — process phase, drain, and probe truth #1413 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
   • resource:search_pull_requests (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
   • issue:IO_CONTRACTS.md — living contract for I/O and control-plane semantics #1303 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
   • issue:meta: I/O control-plane substrate — lifecycle, health, readiness, and status #1306 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
   • pr:feat(pipeline): evolve Processor trait to chain with error handling, flush, and SmallVec output #1402 (pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
   • pr:DO NOT MERGE: Prototype source metadata injection #1370 (pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)

   ---

   What is this? | From workflow: Issue Triage

   Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
3. strawgate

   added

   component:diagnosticsDiagnostics endpoints, dashboard, health, probes, and trace explorerDiagnostics endpoints, dashboard, health, probes, and trace explorer

   component:pipelinePipeline orchestration, batching, runtime flow, and worker coordinationPipeline orchestration, batching, runtime flow, and worker coordination

   on Apr 6, 2026
4. strawgate

   changed the title [-]work-unit: logfwd-io/diagnostics — process phase, drain, and probe truth[/-] [+]work-unit: control plane — process phase, drain, and probe truth[/+] on Apr 6, 2026
5. strawgate commented on Apr 7, 2026

   

   strawgate

   on Apr 7, 2026

   OwnerAuthor

   More actions

   Resolved by PRs #1454, #1491, #1509. Verified: probes derived from explicit ComponentHealth phase truth.
6. strawgate

   closed this as completedon Apr 7, 2026