Repository navigation
Fix RLE detection for the first block of every MT job - #4851
Huang-404-Q wants to merge 1 commit into
Conversation
ZSTDMT_compressionJob calls ZSTD_compressBegin_advanced_internal for every job, which resets isFirstBlock to 1. The non-first-job branch flushes the frame header with srcSize=0, and ZSTD_compressContinue_internal returns early without emitting a block, so isFirstBlock is still 1 when the job's first real block is compressed. The RLE guards therefore skip RLE for the first block of every job, not just the frame's true first block. Clear isFirstBlock in the non-first-job branch. The frame's first block stays protected for decoders <= v1.4.3. Adds a fuzzer test: 2 MiB of zeros compressed with nbWorkers=0 and with nbWorkers=2 + jobSize=512 KB must produce the same size and roundtrip. Before the fix the MT output is 104 bytes vs 83; after, both are 83. Fixes meta#4457.
|
Hi @Huang-404-Q! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks! |
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks! |
Summary
ZSTDMT_compressionJobcallsZSTD_compressBegin_advanced_internalfor every job, which resetsisFirstBlockto 1. The non-first-job branch flushes the frame header withsrcSize=0, andZSTD_compressContinue_internalreturns early without emitting a block, soisFirstBlockis still 1 when the job's first real block is compressed. The RLE guards therefore skip RLE for the first block of every job, not just the frame's true first block.Clear
isFirstBlockin the non-first-job branch. The frame's first block stays protected for decoders <= v1.4.3.Fixes #4457.
Test
Added a fuzzer test next to the existing RLE detection test: 2 MiB of zeros compressed with
nbWorkers=0and withnbWorkers=2+ZSTD_c_jobSize=512 KBmust produce the same size and roundtrip.Before the fix the MT output is 104 bytes vs 83 for single-thread; after, both are 83. Verified locally with
cmake --build build-test --target fuzzer && ./build-test/tests/fuzzer.exe -i1 -s1 -vv.AI assistance
Tool(s) used: Claude Code
How you used it: the assistant read the code, wrote the fix and the test, and ran the fuzzer before and after. A human reviewed the diff and takes responsibility for the change.