meta: I/O control-plane substrate — lifecycle, health, readiness, and status

Purpose

Meta tracker for the broader I/O control-plane substrate: lifecycle, readiness, status, and diagnostics architecture.

This remains open as an architecture umbrella and sequencing artifact, not as a single execution work-unit.

Active open children

Closed slices already landed

Notes

Use narrow work-unit issues for implementation batches under this umbrella.

tl;dr: Outcome B — this needs a human decision before implementation because readiness semantics and lifecycle ownership are not defined enough to implement mechanically without choosing behavior.

Recommendation

Adopt a short decision in this issue for lifecycle + readiness policy first, then implement mechanically. I recommend windowed/degraded readiness (required components + threshold/time window), with an additive metrics rollout (component_name added without removing existing labels initially).

Findings (code evidence)

Current readiness is startup-only and route is /ready (not /readyz)
- crates/logfwd-io/src/diagnostics.rs:484-488 routes /health + /ready.
- crates/logfwd-io/src/diagnostics.rs:553-556 sets ready when pipelines are non-empty.
- crates/logfwd-io/src/diagnostics.rs:545-552 explicitly documents that data-flow freshness is not a readiness gate.
No component lifecycle/health abstraction exists yet
- crates/logfwd-io/src/input.rs:43-67 (InputSource) has no start/shutdown/health.
- crates/logfwd-output/src/sink.rs:74-90 (Sink) has shutdown only; no start/health.
- crates/logfwd-io/src/otlp_receiver.rs:266-273 relies on Drop for shutdown.
Metrics exist, but proposed ComponentMetrics shape is not 1:1 with today’s model
- crates/logfwd-types/src/diagnostics.rs:10-27 has ComponentStats (lines_total, bytes_total, errors_total, plus rotations/parse errors).
- crates/logfwd-io/src/diagnostics.rs:66-73 has batch_latency_nanos_total and inflight_batches at pipeline scope, not component scope.
- crates/logfwd-io/src/diagnostics.rs:96, 157-159, 177-179 show labels pipeline, input, output (no component_name).
Changing readiness path/policy has cross-surface impact
- Probes/docs currently reference /ready:
  - deploy/daemonset.yml:58-61
  - tests/e2e/manifests/logfwd-daemonset.yaml:56-59
  - book/src/config/reference.md:445-447
  - book/src/deployment/monitoring.md:14-16
  - book/src/troubleshooting.md:197-198
Existing tests enforce current readiness semantics
- crates/logfwd-io/src/diagnostics.rs:1645-1677 (/ready 503 with no pipeline, 200 with pipeline even before data).

Verification

I ran:

cargo test -p logfwd-io test_ready_endpoint_ -- --nocapture

Output:

running 2 tests
test diagnostics::tests::test_ready_endpoint_no_pipelines_returns_503 ... ok
test diagnostics::tests::test_ready_endpoint_with_pipeline_returns_200 ... ok

test result: ok. 2 passed; 0 failed

Detailed Action Plan (post-decision mechanical steps)

Decision needed now

Pick one readiness policy:

Strict: any required component unhealthy => 503.
Startup-only: keep current gate; expose per-component health in body only.
Windowed/degraded (recommended): readiness uses required components + error/time thresholds to avoid flapping.

Mechanical implementation once policy is chosen

Define lifecycle primitives
- Add Component trait and Health enum in chosen crate (logfwd-types vs logfwd-io) and wire exports.
Bridge existing interfaces
- Add lifecycle wrappers/adapters for InputSource and Sink (or fold trait into them), including OTLP receiver replacing Drop-only shutdown path.
Add ComponentMetrics harness
- Build from current ComponentStats in crates/logfwd-types/src/diagnostics.rs.
- Add component_name attribute in PipelineMetrics::add_input/add_output paths (crates/logfwd-io/src/diagnostics.rs:149-187).
Readiness endpoint changes
- Update routing in crates/logfwd-io/src/diagnostics.rs:484-488 to support /readyz (and keep /ready alias during transition).
- Implement policy-based aggregation for component health in serve_ready.
Update tests and docs/probes
- Update diagnostics tests in crates/logfwd-io/src/diagnostics.rs:1645-1677.
- Update deployment probes/docs listed above to match /readyz policy.
Verification command
- cargo test -p logfwd-io
- cargo test -p logfwd --test it
- just ci

Related Items

Type	Link	Relevance
Issue	#1306	This triage target
Issue	#1304	Declared dependency (`IoAction`)
Issue	#1283	Parent I/O refactor research context
Issue	#1311	Receiver refactor depends on lifecycle work
Issue	#1313	File appender work depends on lifecycle work
Issue	#192	Historical failure-metrics context
File	`crates/logfwd-io/src/diagnostics.rs`	Current readiness + diagnostics behavior
File	`crates/logfwd-types/src/diagnostics.rs`	Existing component metric primitives
File	`crates/logfwd-io/src/input.rs`	Input interface lacks lifecycle/health
File	`crates/logfwd-output/src/sink.rs`	Sink interface has partial lifecycle (`shutdown`)

Label recommendation: needs-decision.

Note

🔒 Integrity filtering filtered 5 items

Integrity filtering activated and filtered the following items during workflow execution.
This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.

issue:meta: I/O control-plane substrate — lifecycle, health, readiness, and status #1306 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
resource:search_pull_requests (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
issue:P1: Add failure metrics (dropped batches, per-sink errors, buffer size, parse errors) #192 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
issue:research: I/O component refactor — library evaluation and benchmarking #1283 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
issue:Unified IoAction error type + HTTP response classifier #1304 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)

What is this? | From workflow: Issue Triage

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

Cleanup pass: #1412, #1414, #1413, #1440, #1415, and #1416 are done. Remaining open items are still substantial (#1303 #1445 #1310 #1311 #1312 #1313 #1314 #1315 #1283), so please split or rewrite this umbrella with completed sub-items removed.

Converted this from to to keep execution queues narrow. Implementation should run via scoped work-units (current examples: #1549, #1548, #1495, #1555).

Follow-up clarification: this issue is now a meta tracker (not an execution work-unit). Run implementation via scoped work-units such as #1549, #1548, #1495, and #1555.

Deep-Dive Research: I/O Control-Plane Substrate

Current State — Extensive but Fragmented

ComponentHealth: 6 states, combine() Kani-proved commutative/associative
4 separate health reducers (each Kani-proved): pipeline input, receiver, polling input, output sink
Readiness policy: Aggregates across all pipelines → ready/not-ready
HTTP endpoints: /live, /ready, /admin/v1/status, /admin/v1/traces, etc.
Per-component atomic counters (ComponentStats): lock-free on hot path, dual-write to OTel

What "Substrate" Means

No unified Component trait exists. Each component type (file input, TCP, OTLP receiver, output sink, worker pool) has ad-hoc lifecycle. No structured startup ordering, no dependency graph, no hot-reload, no component-level readiness reporting.

Recommended Approach

Lightweight IoComponent trait + ComponentRegistry:

trait IoComponent: Send + Sync {
    fn start(&mut self) -> Result<(), IoError>;
    fn health(&self) -> ComponentHealth;
    fn shutdown(&mut self) -> Pin<Box<dyn Future<Output = ()>>>;
    fn name(&self) -> &str;
}

Evolve toward actor-like substrate incrementally. Existing health reducers and ComponentStats stay as-is — they're well-designed. The gap is standardized lifecycle management.

Complexity: L

Many files touched (each component implements trait), but each change is incremental.

Resolved. I/O control-plane substrate built: PipelineMachine with typestate lifecycle, ComponentHealth enum, per-component health tracking (receiver_health.rs, polling_input_health.rs), readiness/liveness endpoints.

Navigation Menu

Repository navigation

meta: I/O control-plane substrate — lifecycle, health, readiness, and status #1306

Purpose

Active open children

Closed slices already landed

Notes

Recommendation

Decision needed now

Mechanical implementation once policy is chosen

Deep-Dive Research: I/O Control-Plane Substrate

Current State — Extensive but Fragmented

What "Substrate" Means

Recommended Approach

Complexity: L

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

meta: I/O control-plane substrate — lifecycle, health, readiness, and status #1306

Description

Purpose

Active open children

Closed slices already landed

Notes

Activity

github-actions commented on Apr 6, 2026

Recommendation

Decision needed now

Mechanical implementation once policy is chosen

strawgate commented on Apr 7, 2026

strawgate commented on Apr 7, 2026

strawgate commented on Apr 7, 2026

strawgate commented on Apr 24, 2026

Deep-Dive Research: I/O Control-Plane Substrate

Current State — Extensive but Fragmented

What "Substrate" Means

Recommended Approach

Complexity: L

strawgate commented on Apr 25, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions