Repository navigation
Refactor: zero-copy pipeline — Bytes as universal currency, Framer/Aggregator model #303
Description
Summary
Redesign the read → parse → scan pipeline to eliminate unnecessary copies. Based on research of Vector, Fluent Bit, simdjson, and simd-json architectures.
Current status
The streaming StructuralIndex prototype is working (structural.rs on feat/structural-index-continued). Key sub-issues:
- Migrate scanner from ChunkIndex to StreamingClassifier #332 — Migrate scanner from ChunkIndex to StreamingClassifier
- Wire NewlineFramer and CRI parser to StructuralIndex bitmasks #335 — Wire NewlineFramer and CRI parser to bitmasks
- Unified SIMD StructuralIndex: extend ChunkIndex to detect all structural characters #313 — StructuralIndex architecture (parent epic)
Target data flow
FileReader reads into BytesMut → freeze() → Bytes (refcounted)
For each 64-byte block:
Stage 1: wide::u8x16 SIMD detect 10 chars → RawBlockMasks (stack)
Stage 2: StreamingClassifier → ProcessedBlock (escape-aware, string-masked)
Stage 3: Consume bitmasks immediately:
- Newline bitmask → line ranges (framer)
- Space bitmask → CRI field ranges
- Quote+comma+colon → JSON field extraction (scanner)
- Results → FieldSink → Arrow builders → RecordBatch
Cross-block state: 2 u64 values. Bitmask storage: zero heap.
Key changes from original design
- No stored bitmask vectors — simdjson/simd-json never store them
- Portable SIMD via
widecrate — no hand-rolled platform backends - Streaming per-block — not two-pass (SIMD then scanner)
- Sequential scanner — trailing_zeros iteration, not random-access ChunkIndex
References
dev-docs/research/structural-index-research.mddev-docs/research/wide-crate-evaluation.mddev-docs/DECISIONS.md(two-stage architecture, unified StructuralIndex)
Activity
docs: zero-copy pipeline design based on Vector/simdjson research
strawgate commentedon Mar 31, 2026
Architecture update: unified SIMD StructuralIndex
Benchmark results from #313 change the framing architecture. Instead of separate passes (memchr for newlines + ChunkIndex for quotes/backslashes), we'll extend ChunkIndex into a StructuralIndex that detects all structural characters in one SIMD pass.
Benchmark (NEON, ~760KB NDJSON):
Scaling is linear at ~28µs per character. The unified approach is viable.
Data flow changes:
This eliminates the separate memchr newline pass and makes CRI field extraction and future CSV parsing essentially free — the bitmasks are already computed.
refactor: pipeline bypasses format.rs — raw bytes to scanner (#303, #335
refactor: pipeline bypasses format.rs — raw bytes to scanner (#303, #335
strawgate commentedon Mar 31, 2026
Implemented in PR #399. Pipeline now bypasses format.rs — raw bytes go directly to scanner. CRI uses parse_cri_line + CriAggregator.
refactor: pipeline bypasses format.rs — raw bytes to scanner (#303, #335
refactor: pipeline bypasses format.rs — raw bytes to scanner (#303, #335
refactor: pipeline bypasses format.rs — raw bytes to scanner (#303, #335
refactor: pipeline bypasses format.rs, delete FormatParser (#303, #335)
refactor: pipeline bypasses format.rs, delete FormatParser (#303, #335)
refactor: pipeline bypasses format.rs, delete FormatParser (#303, #335)
refactor: pipeline bypasses format.rs — raw bytes to scanner (#303, #335