bug: generator/otlp inputs accept max_open_files and ignore it silently

Summary

Non-file inputs such as generator and otlp accept max_open_files during validation even though the setting only makes sense for file-based inputs. The field is silently ignored instead of being rejected as irrelevant.

Reproduction

This config validates successfully even though generator has no open-file limit:

input:
  type: generator
  max_open_files: 7
output:
  type: null

The same passes for otlp input.

Run:

git fetch origin
git switch --detach origin/master
cargo build -p logfwd
./target/debug/logfwd --config config.yaml --validate

Actual behavior

Validation succeeds:

ready: default
config ok: 1 pipeline(s)

Expected behavior

max_open_files should be rejected for non-file inputs instead of being silently ignored.

Affected commit

Observed on origin/master at df59f6bd143bc737a2d198c8b381bd3321114a28.

tl;dr: Issue #1072 is still valid — max_open_files is accepted for generator/otlp during config validation, but only file inputs consume it at runtime, so it is silently ignored.

Recommendation

Reject max_open_files for all non-file input types in Config::validate, and add non-file regression tests (at minimum generator and otlp; ideally include tcp/udp parity since the same gap exists there).

Status

Still valid: yes.
Already done: only max_open_files: 0 lower-bound validation exists (from prior fix for file inputs).
Not done: file-only enforcement for max_open_files.

Findings

InputConfig exposes max_open_files globally, though docs say file-only:

// crates/logfwd-config/src/lib.rs:242-244
/// Maximum number of file descriptors to keep open simultaneously.
/// Applies only to `file` inputs. Defaults to 1024 when not set.
pub max_open_files: Option(usize),

Validation currently only rejects 0; it does not reject non-file usage:

// crates/logfwd-config/src/lib.rs:643-659, 668-673
match input.input_type {
    InputType::File => { ... }
    InputType::Udp | InputType::Tcp => { ... }
    InputType::Otlp | InputType::Generator | InputType::ArrowIpc => {}
}

if input.max_open_files == Some(0) {
    return Err(..."max_open_files must be at least 1"...);
}

Runtime only uses max_open_files in file-tail path:

// crates/logfwd/src/pipeline.rs:1282-1287
let tail_config = TailConfig {
    ...
    max_open_files: cfg.max_open_files.unwrap_or(1024),
    ..Default::default()
};

Generator/Otlp branches do not read this field (crates/logfwd/src/pipeline.rs:1299-1330).

Existing tests only cover file behavior (crates/logfwd-config/src/lib.rs:1695-1734), with no non-file rejection tests.

Verification

I verified current behavior with config validation and existing tests.

$ cargo run -q -p logfwd -- --config /tmp/gh-aw/agent/config-generator-max-open.yaml --validate
ready: default
config ok: 1 pipeline(s)

$ cargo run -q -p logfwd -- --config /tmp/gh-aw/agent/config-otlp-max-open.yaml --validate
ready: default
config ok: 1 pipeline(s)

$ cargo test -q -p logfwd-config max_open_files -- --nocapture
running 3 tests
...
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 54 filtered out; finished in 0.00s

This confirms the bug: non-file configs with max_open_files currently validate successfully.

Detailed Action Plan

In crates/logfwd-config/src/lib.rs (Config::validate, around lines 638-674), add a file-only guard:
- If input.max_open_files.is_some() and input.input_type != InputType::File, return validation error:
  - pipeline '{name}' input '{label}': max_open_files is only supported for file inputs
Keep existing lower-bound check for file inputs (max_open_files != 0).
Add tests near existing max_open_files tests (crates/logfwd-config/src/lib.rs:1695+):
- reject generator + max_open_files
- reject otlp + max_open_files
- (recommended parity) reject tcp + max_open_files, reject udp + max_open_files
- keep existing file accept/reject tests unchanged.
If docs mention max_open_files, ensure wording is explicit that this is file-input-only and enforced at validation time.

Key Files

File	Current relevant lines	Why it matters
`crates/logfwd-config/src/lib.rs`	`233-245`	`InputConfig` declares `max_open_files`
`crates/logfwd-config/src/lib.rs`	`638-674`	Input validation path currently misses non-file rejection
`crates/logfwd-config/src/lib.rs`	`1695-1734`	Existing tests only cover file cases
`crates/logfwd/src/pipeline.rs`	`1276-1288`	`max_open_files` consumed only for file input
`crates/logfwd/src/pipeline.rs`	`1299-1330`	Generator/OTLP path ignores `max_open_files`

Open PRs

I checked open PRs and found #1027, #1003, #1000 currently open. In this run, file-level PR diff reads were blocked by MCP integrity policy, so I could not confirm direct file overlap for the exact files above.

Related Items

Type	Link	Relevance
Issue	#1072	This issue
Issue	#1075	Same bug pattern for `tcp`/`udp` inputs
Issue	#696 (closed)	Prior file-input lower-bound fix (`max_open_files: 0`)
File	`crates/logfwd-config/src/lib.rs`	Validation and tests to update
File	`crates/logfwd/src/pipeline.rs`	Confirms runtime consumption is file-only

Note

🔒 Integrity filtering filtered 6 items

Integrity filtering activated and filtered the following items during workflow execution.
This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.

issue:bug: generator/otlp inputs accept max_open_files and ignore it silently #1072 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
issue:bug: tcp/udp inputs accept max_open_files and ignore it silently #1075 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
issue:file input: max_open_files: 0 silently disables all file reading #696 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
pr:Convert JsonLinesSink to async reqwest (migrate from ureq) #1000 (pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
pr:perf: add throughput ceiling benchmark (generator → scan → null) #1027 (pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
pr:adopt thiserror 2.0 across workspace, define per-crate error enums #1003 (pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)

What is this? | From workflow: Issue Triage

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

Validation: CONFIRMED — shared root cause with all silent-ignore config issues

Both InputConfig and OutputConfig are single flat structs (#[serde(deny_unknown_fields)]) with fields shared across all input/output types. Config::validate() checks that required fields are present for each type but does not reject extraneous optional fields that have no meaning for the selected type.

This bug is one of ~12 confirmed instances of the same pattern (see also #1057, #1058, #1060, #1064, #1065, #1066, #1070, #1071, #1072, #1073, #1075, #1077, #1078).

Root cause confirmed:

crates/logfwd-output/src/lib.rs: runtime builders use match cfg.output_type and silently ignore fields not consumed by the selected arm.
crates/logfwd-config/src/lib.rs Config::validate(): validates required fields per type but has no reject-if-present checks for inapplicable optional fields.

Unified fix — add per-type field rejection to Config::validate():

// Example for OutputConfig:
match output.output_type {
    OutputType::Stdout | OutputType::Null => {
        if output.endpoint.is_some() { warn_or_err("endpoint not used by this output type"); }
        if output.index.is_some() { warn_or_err("index not used by this output type"); }
        if output.compression.is_some() { warn_or_err("compression not used by this output type"); }
        if output.auth.is_some() { warn_or_err("auth not used by this output type"); }
        // etc.
    }
    OutputType::Elasticsearch => {
        if output.protocol.is_some() { warn_or_err("protocol not used by elasticsearch"); }
        // etc.
    }
    // ...
}

A single pass over all output types covering all optional fields fixes all 12 related issues at once.

fix(config): reject type-inappropriate fields in input/output validation

Fixed on master in PR #1123 (923a033bd8e855f4011f8ad9d5248b9e44a3546a).

I rechecked origin/master; these type-inappropriate config fields are now rejected during validation, so this issue/work-unit looks stale and safe to close.

Navigation Menu

Repository navigation

bug: generator/otlp inputs accept max_open_files and ignore it silently #1072

Summary

Reproduction

Actual behavior

Expected behavior

Affected commit

Recommendation

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

bug: generator/otlp inputs accept max_open_files and ignore it silently #1072

Description

Summary

Reproduction

Actual behavior

Expected behavior

Affected commit

Activity

github-actions commented on Apr 5, 2026

Recommendation

strawgate commented on Apr 5, 2026

strawgate commented on Apr 5, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions