[Copilot]

• Explore repository structure, existing benchmarks, generators, scan configs, and pipeline architecture
• Create throughput_ceiling criterion benchmark measuring pure generator → scan → null throughput
• Register new benchmark in crates/logfwd-bench/Cargo.toml
• Add bench-ceiling and bench-ceiling-self recipes to justfile
• Add self-ceiling.yaml passthrough scenario (no filter)
• Update bench/scenarios/README.md with new scenario
• Verify benchmark compiles and all tests pass (clippy, fmt, cargo test, criterion --test)
• Capture single-core benchmark results with taskset -c 0 cargo bench -p logfwd-bench --bench throughput_ceiling -- --noplot and taskset -c 0 just bench-ceiling-self 10
• Result: max measured criterion throughput was 2.2244–2.2373 M lines/sec with mixed-schema input plus 3-field pushdown; this is the highest single-core scan→null result in the sweep
• Result: max measured full-schema narrow-input throughput was 1.9951–2.2908 M lines/sec at 221.72–223.16 MiB/sec
• Result: mixed variable-schema full extraction sustained 969.65–983.37 K lines/sec at 251.21–256.83 MiB/sec
• Result: wide 20-field full extraction sustained 404.46–409.45 K lines/sec at 234.17–236.24 MiB/sec
• Result: best mixed-schema batch size was 10K rows at 1.0039–1.0076 M lines/sec; 100K rows regressed to 892.49–903.32 K lines/sec, so larger batches do not keep improving throughput
• Result: 3-field pushdown on mixed input reached 2.2244–2.2373 M lines/sec versus 990.25–994.18 K lines/sec for extract_all (~2.25x faster); keep_raw: false made no meaningful difference versus default (985.38–991.18 K lines/sec)
• Result: scan_detached beat streaming on mixed input (990.81–994.48 K lines/sec vs 931.32–942.09 K lines/sec, about 5–6% faster)
• Result: the limiting factor is scanner throughput, not generator or null sink overhead — generator-only reached 2.1332–2.1410 M lines/sec, scan-only reached 991.35–995.52 K lines/sec, and null-sink-only was effectively free (~160,000 Gelem/sec)
• Result: end-to-end single-core passthrough pipeline just bench-ceiling-self 10 processed 10,344,000 lines in 10.0s = 1,033,057 lines/sec
• Result: compared with the README 1.7M lines/sec claim, the measured passthrough pipeline on this machine is lower (~1.03M/sec), but narrow/full-schema and pushdown-heavy criterion cases still exceed 2.2M/sec; the current bound for realistic mixed-schema full extraction is the scanner at about 1.0M lines/sec

Note

Add throughput ceiling benchmarks for generator → scan → null pipeline

• Adds a new Criterion benchmark target throughput_ceiling with five benchmark groups: by schema, by batch size, by scan config, by scan mode, and bottleneck isolation (generator/scan/sink measured separately).
• Adds just bench-ceiling to run the Criterion benchmarks and just bench-ceiling-self to run the end-to-end passthrough scenario via bench/scenarios/self-ceiling.yaml (generator → SELECT * → null).
• Includes the new benchmark in bench-full so it runs alongside existing criterion suites.

^{Macroscope summarized 8120445.}

[Copilot]

Pull request overview

Adds a new benchmark and scenario to measure the pipeline’s single-core “throughput ceiling” in an I/O-free setup, helping contextualize performance limits (generator/scanner/null sink) and compare against existing end-to-end and criterion benchmarks.

Changes:

• Added a new Criterion benchmark (throughput_ceiling) that sweeps schema shape, batch size, scan config, and scan mode, plus isolates generator/scan/sink costs.
• Added a self-contained “ceiling” scenario (self-ceiling.yaml) and a just bench-ceiling-self recipe for end-to-end passthrough (SELECT *) without network.
• Registered the new benchmark in logfwd-bench and wired it into just bench-full plus a dedicated just bench-ceiling command.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
justfile	Adds recipes to run the new ceiling benchmark (criterion) and self-contained ceiling scenario.
crates/logfwd-bench/Cargo.toml	Registers the new throughput_ceiling criterion bench target.
crates/logfwd-bench/benches/throughput_ceiling.rs	Implements the new ceiling benchmark suite (schema/batch/config/mode sweeps + isolation).
bench/scenarios/self-ceiling.yaml	Adds a no-filter passthrough self scenario (SELECT *) with null output.
bench/scenarios/README.md	Documents how to run the new scenario and adds it to the scenarios table.