RFC / Proposal: Domain-specific polymorphic pre-filtering (BCJ & Multi-Stride) on binary payloads #4857

Hi Yann and the Zstandard team,

First, thank you for developing Zstandard and FSE. Zstd is indisputably the world's most versatile, high-throughput general compressor.

Over the past weeks, i investigated the performance boundaries of flat LZ77/FSE pipelines when processing structured binary payloads (compiled machine code, floating-point geometry, continuous 16-bit imaging). Because Zstd treats streams as flat byte sequences, relative branch offsets (CALL/BL) and interleaved float mantissas introduce artificial entropy that escapes match finders.

I built an open-source C99 reference prototype called binpack (https://github.com/Santirobm/binpack) that runs sub-microsecond inspection (< 1 µs) over buffer headers to conditionally activate reversible pre-filters (BCJ x86/ARM, Delta-2, and Stride-2/4/8) operating at > 2 GB/s before feeding the stream to Zstandard.

I have a few benchmarks in my github page

Decompression times remain under 180 ms with 100% SHA-256 bit-exact verification.

I´ve a few questions:

1. Has the Zstd team considered introducing optional pre-filtering stages in the official "zstd" CLI or wrapper tooling for binary workloads?
2. How does the team view the balance between microsecond pre-filtering overhead vs. wire-transfer compression gains on binary distribution pipelines?All benchmarks, reproduction datasets, and C99/Python sources are available under MIT at: https://github.com/Santirobm/binpack

Feedback and technical critique are warmly welcome!

Santiago.