Skip to content

P1: File tailer max open files + idle eviction #193

Closed

Description

@strawgate

Problem

tail.rs HashMap of open files grows unbounded. No eviction. Deleted files leak FDs forever (last_read field exists but is never checked for eviction).

K8s node with 1000 pods = potentially 1000+ open FDs with no limit.

Fix

  1. Add max_open_files: usize to TailConfig (default: 1024)
  2. After the read loop in poll(), add LRU eviction: if files.len() > max_open_files, sort by last_read, evict oldest until under limit
  3. Add cleanup for deleted files: if !path.exists() during poll, remove from files HashMap
  4. Track open_files metric

Test scenarios

  1. Create 10 files, set max_open_files: 5. Assert only 5 FDs open, 5 least-recently-read evicted.
  2. Write to an evicted file → re-opened on next glob rescan.
  3. Delete a file → FD cleaned up on next poll.
  4. open_files metric tracks count accurately.

Files to change

  • crates/logfwd-core/src/tail.rs: add config field, eviction pass after line 404, deleted file cleanup
  • crates/logfwd-config/src/lib.rs: add max_open_files to input config
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

enhancementNew feature or requestproductionRequired for production readiness

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions