File tailer refactor — split monolith, typestate, per-file correctness

Problem

tail.rs is still the clearest example of why we want the full correct-by-construction architecture. It is a large stateful subsystem whose correctness depends on file lifecycle transitions, event ordering, identity rules, bounded reads, and checkpoint semantics, yet much of that logic is still concentrated in one monolithic file.

This issue is not just about splitting a file. It is about turning the tailer into a subsystem with an explicit state model and verifiable invariants.

Target Architecture

crates/logfwd-io/src/tail/
├── mod.rs          # Public API: FileTailer and tail-facing event types
├── discovery.rs    # Glob expansion, directory watching, rescan, duplicate suppression
├── reader.rs       # Per-file IO, truncation, EOF, budgeted reads, eviction
├── identity.rs     # FileIdentity, SourceId derivation, fingerprint policy
├── state.rs        # Explicit file lifecycle state model / typestate-adjacent core
├── config.rs       # TailConfig with validated defaults and fairness controls
└── tests/          # Domain-focused regression/proptest/model tests

Desired end-state

illegal lifecycle combinations are removed by design, not merely guarded by comments and booleans
per-file lifecycle is explicit: discovery, activation, draining, eviction, reopen, truncation, deletion
fairness and bounded-read behavior are deliberate policy, not incidental iteration order
identity and checkpoint rules are documented and testable
the downstream tail->framing contract is preserved and made explicit

Sub-tasks

Modularize the tailer — split the monolith into coherent modules without losing current behavior
Introduce an explicit state model — typestate where it materially pays off, or an equally rigorous explicit runtime state machine where async/IO edges make compile-time typestate too awkward
Per-file fairness and bounded reads — make per-file budget/fairness policy explicit and configurable
Identity policy hardening — (dev, inode, fingerprint) and empty-file/reopen/eviction semantics are explicit and tested
Truncation / rotation / EOF contract hardening — event ordering is documented and regression-tested end-to-end
Proof/test coverage — Kani for pure identity/transition helpers, proptest for randomized file-operation sequences, targeted regression tests for event semantics

Design stance

The destination here is ambitious: we do want the tailer to feel "correct by construction" rather than merely "less bad than before."

That does not require pretending every async/filesystem interaction should be encoded in type parameters. The point is to make invalid states and invalid transitions difficult or impossible to represent, while keeping the implementation readable and verifiable.

Related issues fixed or absorbed

Fixes or absorbs the architectural dimension of:

Relates to:

refactor: optional logfwd-io split after Phase C and module-first cleanup (#851) #863 split logfwd-io
research: I/O component refactor — library evaluation and benchmarking #1283 research
I/O correctness refactor — correct by construction, not by patch #1315 architecture umbrella

tl;dr: Outcome A — ready for implementation. crates/logfwd-io/src/tail.rs is still a 2326-line monolith, and the remaining work is now a mechanical extraction + typestate pass with explicit per-file state transitions.

Recommendation

Proceed with a behavior-preserving refactor in logfwd-io that:

splits tail.rs into focused modules,
replaces bool-driven per-file lifecycle with typestate transitions,
keeps existing event/checkpoint semantics unchanged,
tightens the read budget to the issue target (256 KiB/file/poll).

This is implementable without new design decisions.

Findings

Monolith still present: crates/logfwd-io/src/tail.rs:1 (2326 lines in tree). Public orchestrator and internal layers are still in one file:
- FileDiscovery at tail.rs:223
- FileReader at tail.rs:412
- FileTailer::poll at tail.rs:927
Bool-driven lifecycle still present: eof_emitted in TailedFile (tail.rs:56-66) and transitions at tail.rs:501, tail.rs:616, tail.rs:638, tail.rs:659-661.
Read budget mismatch: current MAX_READ_PER_POLL is 4 MiB (tail.rs:430), while this issue calls for 256 KiB/file/poll.
Glob stack still old: glob::glob path expansion (tail.rs:201) and dependency glob = "0.3" in crates/logfwd-io/Cargo.toml:17 (no globset/walkdir yet).
Per-source remainder is already done (don’t re-implement): FramedInput.sources per source at crates/logfwd-io/src/framed.rs:54-60, SourceState at framed.rs:33-41.
Regression coverage already exists for most correctness fixes: truncation, rotation drain, eviction checkpointing, etc. (tail.rs:1966+, tail.rs:2027+, tail.rs:2110+).

Verification

I ran targeted tailer tests:

$ cargo test -p logfwd-io test_truncation_emits_truncated_event
test tail::tests::test_truncation_emits_truncated_event ... ok

$ cargo test -p logfwd-io test_read_cap_prevents_oom
test tail::tests::test_read_cap_prevents_oom ... ok

$ cargo test -p logfwd-io test_evicted_offsets_in_checkpoint_data
test tail::tests::test_evicted_offsets_in_checkpoint_data ... ok

Detailed Action Plan (mechanical)

Split module layout (move-only first, no behavior change)
- Create:
  - crates/logfwd-io/src/tail/mod.rs
  - crates/logfwd-io/src/tail/discovery.rs
  - crates/logfwd-io/src/tail/reader.rs
  - crates/logfwd-io/src/tail/identity.rs
  - crates/logfwd-io/src/tail/config.rs
- Move code from current anchors:
  - FileIdentity (tail.rs:26-52) -> identity.rs
  - TailConfig (tail.rs:131-163) -> config.rs
  - FileDiscovery (tail.rs:223-402) -> discovery.rs
  - FileReader (tail.rs:412-812) -> reader.rs
  - FileTailer + public API (tail.rs:819-1025) -> mod.rs
- Keep pub enum TailEvent shape unchanged (tail.rs:91-128).
Introduce per-file typestate (replace boolean lifecycle flags)
- In reader.rs, replace eof_emitted: bool (tail.rs:65) with a small state enum that encodes legal transitions (e.g., Active, EofReported, Draining, Evicted).
- Re-express transition points currently spread across:
  - truncation reset (tail.rs:496-506)
  - data paths (tail.rs:612-646)
  - no-data EOF gating (tail.rs:655-667)
- Preserve event ordering guarantees:
  - Truncated before Data (tail.rs:628-645)
  - single EOF per no-data streak (tail.rs:655-667)
Apply per-file read budget target
- Change FileReader::MAX_READ_PER_POLL from 4 MiB (tail.rs:430) to 256 KiB.
- Update expectation/comments in the existing regression (tail.rs:2027+) to match new cap.
Discovery stack follow-up for dependency alignment
- Replace glob::glob expansion path (tail.rs:201) with the approved globset + walkdir approach (dependency issue linkage in File tailer refactor — split monolith, typestate, per-file correctness #1310).
- Update crates/logfwd-io/Cargo.toml dependencies accordingly.
Tests to add/update
- Keep existing regression tests in tail.rs (or move to tail/tests.rs) and add:
  - typestate transition legality test (no invalid state combinations),
  - EOF gate behavior test over NoData -> NoData -> Data -> NoData,
  - read-budget assertion at 256 KiB/file/poll.
- Keep checkpoint/state-machine integration coverage in crates/logfwd-io/tests/it/checkpoint_state_machine.rs.
Verification command for implementation PR
- cargo test -p logfwd-io tail::tests::
- cargo test -p logfwd-io --test it checkpoint_state_machine
- cargo clippy -p logfwd-io --all-targets -- -D warnings
- cargo fmt --all --check

Related Items

Type	Link	Relevance
Issue	#1310	Current refactor target and acceptance criteria
Issue	#1283	Parent research/design thread for I/O refactors
Issue	#1304	Declared dependency in #1310
Issue	#1307	Declared dependency in #1310 (`globset`/`walkdir`)
PR (open)	#1268	Touches tailer behavior/perf; likely merge-conflict area
PR (open)	#1290	`logfwd-io` test/contract work in adjacent area
File	`crates/logfwd-io/src/tail.rs:223`	`FileDiscovery` split anchor
File	`crates/logfwd-io/src/tail.rs:412`	`FileReader` split anchor
File	`crates/logfwd-io/src/tail.rs:927`	`FileTailer::poll` orchestration anchor
File	`crates/logfwd-io/src/framed.rs:54`	Per-source remainder already implemented

Note

🔒 Integrity filtering filtered 8 items

Integrity filtering activated and filtered the following items during workflow execution.
This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.

issue:File tailer refactor — split monolith, typestate, per-file correctness #1310 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
resource:search_pull_requests (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
issue:research: I/O component refactor — library evaluation and benchmarking #1283 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
issue:Unified IoAction error type + HTTP response classifier #1304 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
issue:Replace glob with globset + walkdir in file tailer #1307 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
issue:bug: shared remainder buffer across files — cross-file data corruption #797 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
issue:bug: read_new_data allocates unbounded Vec — OOM on large files #800 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
issue:refactor: optional logfwd-io split after Phase C and module-first cleanup (#851) #863 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)

What is this? | From workflow: Issue Triage

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

Updated this issue to describe the full tailer architecture target: a real subsystem with an explicit lifecycle/state model, fairness/identity/checkpoint semantics, and proofs/tests, not just a file split.

Two concrete cleanup slices from the 2026-04-06 fan-in map directly under this tailer track:

bug: enforce strict FileTailer read budget and add fairness regression coverage #1441 strict FileTailer read-budget enforcement
test: add multi-source checkpoint and remainder interleaving regression suite #1442 multi-source checkpoint/remainder interleaving regression suite

Execution meta: #1450

The fan-in recommendation was to land these hardening slices before any larger seam movement under #1310.

Repo-wide cleanup pass: I split the unique discovery glob bug out of #1422 into #1453. That keeps this architecture umbrella focused on the broader tailer subsystem work while the concrete cwd-relative glob bug gets a leaf of its own.

Partial-resolution status update (April 11, 2026):

Landed toward this architecture:

Tailer code is already split into focused modules under crates/logfwd-io/src/tail (discovery, reader, identity, tailer, tests, verification), rather than a single monolith.
Additional reducer/formal coverage work landed recently (for example PR refactor(tail): extract reducer state and expand formal+property coverage #1759).

Remaining scope to keep this issue open:

Final explicit lifecycle/typestate rigor and any remaining per-file correctness invariants called out in the issue body.
Any remaining fairness/identity/checkpoint edge semantics not yet encoded as stable boundary guarantees.

Recommendation:

Treat this as partially resolved and keep as umbrella for the remaining invariants-focused tail work.

Partial-resolution refresh (April 11, 2026, post-merge wave):

Additional landed work since the prior update:

PR fix(io): WS06+WS07+WS08 tail and aux hardening cluster #1797 (fix(io): WS06+WS07+WS08) directly hardened tailer correctness seams in crates/logfwd-io/src/tail/{discovery,reader,tailer,tests}.rs.
Current main keeps the tailer modularized under crates/logfwd-io/src/tail/ with dedicated identity.rs, state.rs, and verification.rs.

Assessment:

This issue remains partially resolved.
Major structural goals (module split + substantial per-file correctness hardening) are in place.
Keep open for remaining explicit lifecycle/contract closure work and final invariant completion across the full tailer end-state described here.

docs: add architecture research fanout prompts for 5 workstreams

docs: add architecture research synthesis memo

Deep-Dive Research: File Tailer Refactor

Current State (Code-Verified)

7 modules in crates/logfwd-io/src/tail/:

Module	Responsibility
`tailer.rs`	`FileTailer`: poll loop, glob rescan, error backoff, adaptive polling
`discovery.rs`	`FileDiscovery`: notify watcher, glob expansion, change detection
`reader.rs`	`FileReader`: open files, read data, LRU eviction, fingerprint matching
`lifecycle.rs`	Typestate structs — `Active`, `EvictedClosedCached` + scaffolded but unwired: `Untracked`, `DiscoveredUnopened`, `DeletedCleanupPending`, `TerminalRemoved`
`state.rs`	Pure EOF model + backoff delay (Kani-proved)
`identity.rs`	`FileIdentity` (device, inode, fingerprint), `ByteOffset`
`glob.rs`	Glob pattern expansion

Key finding: lifecycle.rs has #![allow(dead_code)] — the typestate transitions are scaffolded but not wired. Only Active and EvictedClosedCached are used. The Session<S> wrapper exists but transitions aren't implemented.

Gaps

Typestate transitions incomplete — File state changes go through FileReader/FileDiscovery methods, not through typed state machine. Invalid transitions not caught at compile time.
Reader↔Discovery intertwined — FileDiscovery::detect_changes() mutates FileReader directly
poll() does too much — ~60 lines: drain events → backoff check → poll interval → glob rescan → detect changes → read all → cleanup → evict → update backoff
No per-file error isolation — had_error is a single boolean ORed across all operations. One file's error backoffs entire tailer.
EOF is timer-coupled — EofState uses Instant::now() directly

Recommended Implementation (All Three Complement)

Per-file errors first (S) — Replace had_error: bool with HashMap<PathBuf, ErrorState>. Backoff per file, not per tailer. Immediate correctness win.
Phase extraction (M) — Split poll() into DrainEvents → GlobRescan → DetectChanges → ReadAll → Cleanup → Evict. Each phase is a pure function (&mut TailerState, &Config) → PhaseResult.
Typestate completion (M) — Wire Session<Active> → Session<EvictedClosedCached> etc. Make illegal transitions compile errors. The scaffolding in lifecycle.rs suggests this was always the intent.

Key Files

tailer.rs, lifecycle.rs, reader.rs, discovery.rs, state.rs, tla/TailLifecycle.tla

Complexity: L (each individual option is M)

Research: File Tailer Typestate Patterns (2026-04-24)

Current State

Tailer modularized into 7 modules in `tail/`. Typestate scaffolding exists but is unused — `lifecycle.rs` defines `Session` with 6 state types, but is `#[allow(dead_code)]`. Only `Active` and `EvictedClosedCached` are used as type aliases. Verification is strong for pure reducers (TLA+, 7 Kani proofs, 2000-case proptest) but does not cover the file lifecycle state machine.

External Patterns

Vector: HashMap<Fingerprint, FileWatcher>, no typestate. Known fingerprint collision issues.

Filebeat: One goroutine per file, implicit state in lifecycle. Known stale harvester race.

Typestate consensus (Cliffle, Hoverbear, Microsoft): Best for linear protocols with few states. Breaks down with HashMap storage (needs homogeneous values) and async boundaries.

Recommended: Hybrid Typestate

Typestate for transitions (compile-time enforcement):
```rust
impl Session {
fn read(self, ...) -> Result<(Session, ReadResult), Session>;
fn evict(self) -> Session;
fn detect_truncation(self) -> Session;
}
```
Runtime enum for storage (HashMap needs homogeneous values):
```rust
enum FileState { Active(Session), Evicted(Session), ... }
```

What current code allows that shouldn't be possible

`read_new_data` on an evicted file (silently returns NoData)

`drain_file` on nonexistent path (silently returns false)

Truncation stays in Active state with offset reset — no state transition

Implementation Phases

Wire existing `Session` / `Session` with typed transition methods, wrap in FileState enum (no behavior change)

Move truncation detection to explicit transition, add Discovered state, extend TLA+

Formalize tail→framed event ordering contract in TLA+, proptest interleaving

Add Deleted/Terminal states, make drain_file consume Session

Sources: Cliffle typestate blog, Microsoft Rust Patterns, Hoverbear state machine patterns, Vector file source docs, Filebeat harvester docs.

Mechanical Implementation Spec: File Tailer Typestate Phase 1 — 2026-04-24

Summary

Wire the existing Session<S> scaffolding in lifecycle.rs (currently #[allow(dead_code)]) with typed transition methods. Use them in evict_lru and open_file_at. No behavior change — pure type safety improvement. HashMap structure unchanged.

Files Changed (2)

File	Change
`crates/logfwd-io/src/tail/lifecycle.rs`	Remove `#![allow(dead_code)]`. Add `FileState` enum, `Session<Active>::evict()`, `Session<EvictedClosedCached>::restore()`, accessor helpers. Add 3 unit tests (evict preserves fields, restore preserves fields, round-trip).
`crates/logfwd-io/src/tail/reader.rs`	Add `use FileState, Session`. Rewrite `evict_lru` to use `Session::new(tailed).evict(path, source_id).state`. Rewrite `open_file_at` evicted-match branch to use `Session::new(evicted).restore(file, offset).state`. Eliminate 4 mutable `stored_*` variables.

Key Transitions

impl Session<Active> {
    fn evict(self, path: PathBuf, source_id: SourceId) -> Session<EvictedClosedCached>;
}
impl Session<EvictedClosedCached> {
    fn restore(self, file: File, offset: u64) -> Session<Active>;
}

What is NOT in Phase 1

No FileState enum used in HashMap storage (still two separate maps)
No changes to discovery.rs, tailer.rs, or test files
No new states (Discovered, Deleted, Terminal) — those are Phase 2+
TailedFile and EvictedFile type aliases preserved

Verification

just clippy && cargo test -p logfwd-io && just ci

Full spec with exact old→new code for evict_lru and open_file_at: see research thread above.

Navigation Menu

Repository navigation

File tailer refactor — split monolith, typestate, per-file correctness #1310

Problem

Target Architecture

Desired end-state

Sub-tasks

Design stance

Related issues fixed or absorbed

Recommendation

Deep-Dive Research: File Tailer Refactor

Current State (Code-Verified)

Gaps

Recommended Implementation (All Three Complement)

Key Files

Complexity: L (each individual option is M)

Research: File Tailer Typestate Patterns (2026-04-24)

Current State

External Patterns

Recommended: Hybrid Typestate

What current code allows that shouldn't be possible

Implementation Phases

Mechanical Implementation Spec: File Tailer Typestate Phase 1 — 2026-04-24

Summary

Files Changed (2)

Key Transitions

What is NOT in Phase 1

Verification

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

File tailer refactor — split monolith, typestate, per-file correctness #1310

Description

Problem

Target Architecture

Desired end-state

Sub-tasks

Design stance

Related issues fixed or absorbed

Activity

github-actions commented on Apr 6, 2026

Recommendation

strawgate commented on Apr 6, 2026

strawgate commented on Apr 6, 2026

strawgate commented on Apr 6, 2026

strawgate commented on Apr 11, 2026

strawgate commented on Apr 11, 2026

strawgate commented on Apr 24, 2026

Deep-Dive Research: File Tailer Refactor

Current State (Code-Verified)

Gaps

Recommended Implementation (All Three Complement)

Key Files

Complexity: L (each individual option is M)

strawgate commented on Apr 25, 2026

Research: File Tailer Typestate Patterns (2026-04-24)

Current State

External Patterns

Recommended: Hybrid Typestate

What current code allows that shouldn't be possible

Implementation Phases

strawgate commented on Apr 25, 2026

Mechanical Implementation Spec: File Tailer Typestate Phase 1 — 2026-04-24

Summary

Files Changed (2)

Key Transitions

What is NOT in Phase 1

Verification

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions