You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Yes, this is a known issue. There is a heuristic that attempts to determine when to test a block for conversion to RLE, but it does not always trigger. We could likely improve this by relaxing the condition, thereby increasing the likelihood that the RLE test is executed.
I believed the heuristic is based on the compressed size of the block,
i.e. if the compressed size is "small enough", then it's a hint that it might be a candidate for RLE.
Blocks consisting of a single match with 0 literals should fit in this description.
I'm surprised if they are not.
These blocks already pass the RLE heuristic, so relaxing it wouldn't catch them. On the default path ZSTD_compressBlock_internal emits RLE when !zc->isFirstBlock && cSize < rleMaxLength && ZSTD_isRLE(...). For an all-zeros block cSize is 8 and ZSTD_isRLE is true, so the only thing blocking RLE is !zc->isFirstBlock.
isFirstBlock gets reset to 1 by every worker job. ZSTDMT_compressionJob calls ZSTD_compressBegin, so the first block of each ~8MB job is treated like the frame's first block and skips the RLE path. Only the frame's real first block needs that guard for the pre-v1.4.3 decoder, not every job.
--single-thread on the repro emits them correctly, leaving only the real first block and the Hello world block as non-RLE. With -T the number of extra single-sequence blocks tracks the job count, e.g. -T4 on a 33 MB all-zeros input gives 4 of them.
Thanks for looking into that @eyupcanakman! Looks like it will be an easy fix in ZSTDMT to ensure that zc->isFirstBlock = false for all but the first job. We have to keep the isFistBlock check around to work around this bug in the CLI.
Activity
Cyan4973 commentedon Aug 11, 2025
Yes, this is a known issue. There is a heuristic that attempts to determine when to test a block for conversion to RLE, but it does not always trigger. We could likely improve this by relaxing the condition, thereby increasing the likelihood that the RLE test is executed.
terrelln commentedon Aug 12, 2025
I think there could be a small improvement to the heuristic. The blocks found are a single match. with 0 literals.
Cyan4973 commentedon Aug 12, 2025
I believed the heuristic is based on the compressed size of the block,
i.e. if the compressed size is "small enough", then it's a hint that it might be a candidate for RLE.
Blocks consisting of a single match with 0 literals should fit in this description.
I'm surprised if they are not.
eyupcanakman commentedon Jul 17, 2026
These blocks already pass the RLE heuristic, so relaxing it wouldn't catch them. On the default path
ZSTD_compressBlock_internalemits RLE when!zc->isFirstBlock && cSize < rleMaxLength && ZSTD_isRLE(...). For an all-zeros blockcSizeis 8 andZSTD_isRLEis true, so the only thing blocking RLE is!zc->isFirstBlock.isFirstBlockgets reset to 1 by every worker job.ZSTDMT_compressionJobcallsZSTD_compressBegin, so the first block of each ~8MB job is treated like the frame's first block and skips the RLE path. Only the frame's real first block needs that guard for the pre-v1.4.3 decoder, not every job.--single-threadon the repro emits them correctly, leaving only the real first block and theHello worldblock as non-RLE. With-Tthe number of extra single-sequence blocks tracks the job count, e.g.-T4on a 33 MB all-zeros input gives 4 of them.terrelln commentedon Jul 17, 2026
Thanks for looking into that @eyupcanakman! Looks like it will be an easy fix in ZSTDMT to ensure that
zc->isFirstBlock = falsefor all but the first job. We have to keep theisFistBlockcheck around to work around this bug in the CLI.