Skip to content

Fix: enrichment CSV loader pads missing fields as empty string, not null #383

Closed

Description

@strawgate

Bug

read_csv_to_batch() in crates/logfwd-io/src/enrichment.rs pads short CSV rows with String::new(), making it impossible to distinguish NULL from empty string in SQL (IS NULL vs = ''). The Arrow schema already marks columns as nullable.

Location

crates/logfwd-io/src/enrichment.rs — search for String::new() in the CSV row padding logic.

Fix

Use Option<String> or build a nullable StringArray where missing fields are null, not empty strings.

Acceptance criteria

  • Short CSV rows produce null values, not empty strings
  • IS NULL works correctly in DataFusion queries on enrichment tables
  • Add a test with short rows verifying null vs empty
  • Tests pass (just test)
  • Read AGENTS.md and CONTRIBUTING.md before starting

Activity

  1. google-labs-jules commented on Mar 31, 2026

    @google-labs-jules
    Contributor

    Jules has failed to create a task. You can try again later by removing and re-adding the 'jules' label.

  2. google-labs-jules commented on Mar 31, 2026

    @google-labs-jules
    Contributor

    Jules is on it. When finished, you will see another comment and be able to review a PR.

  3. google-labs-jules commented on Mar 31, 2026

    @google-labs-jules
    Contributor

    Ready for a review! A PR has been created.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions