Repository navigation
bug: JSON null values silently dropped from output — field disappears entirely #442
Description
Summary
JSON null values are silently removed from the output. A field with value null in the input simply disappears — it is not emitted as "field":null in the output JSON. This changes the shape of the data and can break downstream consumers that depend on field presence.
Root Cause
Two layers contribute:
-
Scanner (
crates/logfwd-core/src/scanner.rs, line ~175): Theb'n'arm callsbuilder.append_null_by_idx(), which in the streaming builder is a no-op (nulls are represented as gaps — no value stored). -
Output (
crates/logfwd-output/src/lib.rs, line ~165):write_row_jsonskips null values:
if arr.is_null(row) {
continue; // ← null values completely omitted from output
}Reproduction
printf '{"a":null,"b":"present","c":42}\n' > /tmp/null.loginput:
type: file
path: /tmp/null.log
format: json
output:
type: stdoutActual output:
{"b":"present","c":42}Expected output:
{"a":null,"b":"present","c":42}Impact
- Schema change: downstream consumers expecting field
ato always exist get no field at all - Distinguishable states merged:
{"a":null}(explicitly null) and{}(field absent) become indistinguishable - Elasticsearch mapping: fields set to
nullmay not get their mapping created if the first batch has all nulls - Silent: no indication the field was present in the input
Note
This is partially by design — Arrow's columnar format represents nulls as gaps, and the output naturally skips them. However, for JSON fidelity, the output should optionally (or always) emit "field":null when the input explicitly contained null.
Fix
In write_row_json, instead of continue for null values:
if arr.is_null(row) {
if !first { out.push(b','); }
first = false;
out.push(b'"');
out.extend_from_slice(col.field_name.as_bytes());
out.extend_from_slice(b"\":null");
continue;
}Or make this configurable with a preserve_nulls option.
Activity
github-actions commentedon Mar 31, 2026
2 related issues
Note
🔒 Integrity filtering filtered 5 items
Integrity filtering activated and filtered the following items during workflow execution.
This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.
issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)What is this? | From workflow: AI: Duplicate Issue Detector
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
fix: output sinks dispatch on Arrow DataType, not column name suffix
fix: output sinks dispatch on Arrow DataType, not column name suffix