Repository navigation
as_chunks_mut loop vectorizes worse than the equivalent chunks_exact_mut loop (x86-64, small constant trip count) #165050
Description
Clippy's chunks_exact_to_as_chunks lint suggests replacing chunks_exact(N) with as_chunks::<N>(). Following it in a hex encoder made a hot path slower. The two functions below write the same bytes, but on the default x86-64 target the as_chunks_mut one is about 15% slower: LLVM's loop vectorizer interleaves it by 2, which leaves the last 4 of its 20 iterations as scalar code.
I tried this code (rustc -C opt-level=3 --crate-type=lib --emit=asm):
#[inline(always)]
fn hex_digit(nibble: u8) -> u8 {
let d = nibble & 0x0F;
d + b'0' + if d > 9 { 39 } else { 0 }
}
#[unsafe(no_mangle)]
pub fn encode_chunks_exact(bytes: &[u8; 20], out: &mut [u8; 40]) {
for (pair, &b) in out.chunks_exact_mut(2).zip(bytes) {
if let [h, l] = pair {
*h = hex_digit(b >> 4);
*l = hex_digit(b);
}
}
}
#[unsafe(no_mangle)]
pub fn encode_as_chunks(bytes: &[u8; 20], out: &mut [u8; 40]) {
for ([h, l], &b) in out.as_chunks_mut::<2>().0.iter_mut().zip(bytes) {
*h = hex_digit(b >> 4);
*l = hex_digit(b);
}
}I expected to see this happen: the same machine code for both. That is what happens with -C target-cpu=x86-64-v3 or higher, and with a trivial loop body (*h = b >> 4; *l = b & 0xF;): LLVM merges the two functions.
Instead, this happened:
| instructions (SIMD) | time per call (Ryzen 9 9950X, best of 9) | |
|---|---|---|
encode_chunks_exact |
91 (90) | 3.82 ns |
encode_as_chunks |
137 (71) | 4.37 ns (+14%) |
With -C remark=all -C debuginfo=1:
encode_chunks_exact:
loop-vectorize (analysis): the cost-model indicates that interleaving is not beneficial
loop-vectorize (success): vectorized loop (vectorization width: 4, interleaved count: 1)
loop-unroll (success): completely unrolled vectorized loop with 5 iterations
encode_as_chunks:
loop-vectorize (success): vectorized loop (vectorization width: 4, interleaved count: 2)
loop-unroll (success): completely unrolled vectorized loop with 2 iterations
loop-unroll (success): completely unrolled epilogue loop with 4 iterations
Why, as far as I can tell
The IR entering LoopVectorizePass is the same for both functions except for the store address (names as in repro.ll below):
; encode_chunks_exact
%off = shl nuw nsw i64 %i, 1
%dst = getelementptr inbounds nuw i8, ptr %out, i64 %off
; encode_as_chunks
%dst = getelementptr inbounds nuw [2 x i8], ptr %out, i64 %iThe vectorizer interleaves a loop whose cost is below -small-loop-cost (default 20). Bisecting that flag (-C llvm-args=-small-loop-cost=N) puts the chunks_exact_mut loop's cost at 20 and the as_chunks_mut loop's at 19: encode_as_chunks gets IC 1 at N ≤ 19, and encode_chunks_exact gets IC 2 at N ≥ 21. So one instruction, the shl, decides it. With a constant trip count of 20, VF 4 × IC 2 = 8 does not divide it, and 4 iterations are left to scalar code. It looks like the interleave count is chosen without weighing the scalar remainder it creates for a small constant trip count.
The same choice helps when the trip count is not known. The loops over slices of run-time length (in the repro below) get VF 8 with IC 1 and IC 2 the same way, and there as_chunks_mut is faster: 3.61 vs 4.96 ns at 20 bytes, equal at 4,096 bytes. On aarch64 the as_chunks_mut loop is instead fully unrolled and SLP-vectorized (43 instructions against 34).
Standalone LLVM reproducer
The two loops as they enter the vectorizer, cleaned up (opt from rustc 1.99.1's llvm-tools, LLVM 23.1.1):
$ opt -passes=loop-vectorize -pass-remarks=loop-vectorize -pass-remarks-analysis=loop-vectorize -disable-output repro.ll
remark: <unknown>:0:0: the cost-model indicates that interleaving is not beneficial
remark: <unknown>:0:0: vectorized loop (vectorization width: 4, interleaved count: 1)
remark: <unknown>:0:0: vectorized loop (vectorization width: 4, interleaved count: 2)
$ opt -passes=loop-vectorize -pass-remarks=loop-vectorize -small-loop-cost=19 -disable-output repro.ll
remark: <unknown>:0:0: vectorized loop (vectorization width: 4, interleaved count: 1)
remark: <unknown>:0:0: vectorized loop (vectorization width: 4, interleaved count: 1)opt -passes='default<O3>' followed by llc -O3 turns it into 91 and 139 instructions.
repro.ll
; Two loops that differ only in how the store address is computed (from rustc 1.99.1's IR
; just before LoopVectorizePass). LV interleaves the second by 2 (loop cost 19 < the
; default -small-loop-cost of 20) but not the first (cost 20: the extra shl). With a
; constant trip count of 20, VF 4 x IC 2 = 8 leaves a 4-iteration scalar remainder.
;
; opt -passes=loop-vectorize -pass-remarks=loop-vectorize \
; -pass-remarks-analysis=loop-vectorize -disable-output repro.ll
target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
target triple = "x86_64-unknown-linux-gnu"
; out.chunks_exact_mut(2): the offset is an explicit shl.
define void @chunks_exact(ptr noalias readonly captures(none) dereferenceable(20) %bytes, ptr noalias writeonly captures(none) dereferenceable(40) %out) {
entry:
br label %loop
loop:
%i = phi i64 [ 0, %entry ], [ %i.next, %loop ]
%i.next = add nuw nsw i64 %i, 1
%off = shl nuw nsw i64 %i, 1
%dst = getelementptr inbounds nuw i8, ptr %out, i64 %off
%src = getelementptr inbounds nuw i8, ptr %bytes, i64 %i
%b = load i8, ptr %src, align 1
%hi = lshr i8 %b, 4
%hi.0 = or disjoint i8 %hi, 48
%hi.c = icmp ugt i8 %b, -97
%hi.adj = select i1 %hi.c, i8 39, i8 0
%hi.d = add nuw nsw i8 %hi.0, %hi.adj
store i8 %hi.d, ptr %dst, align 1
%lo = and i8 %b, 15
%lo.0 = or disjoint i8 %lo, 48
%lo.c = icmp samesign ugt i8 %lo, 9
%lo.adj = select i1 %lo.c, i8 39, i8 0
%lo.d = add nuw nsw i8 %lo.0, %lo.adj
%dst1 = getelementptr inbounds nuw i8, ptr %dst, i64 1
store i8 %lo.d, ptr %dst1, align 1
%done = icmp eq i64 %i.next, 20
br i1 %done, label %exit, label %loop
exit:
ret void
}
; out.as_chunks_mut::<2>(): the scale is in the GEP's element type.
define void @as_chunks(ptr noalias readonly captures(none) dereferenceable(20) %bytes, ptr noalias writeonly captures(none) dereferenceable(40) %out) {
entry:
br label %loop
loop:
%i = phi i64 [ 0, %entry ], [ %i.next, %loop ]
%i.next = add nuw nsw i64 %i, 1
%dst = getelementptr inbounds nuw [2 x i8], ptr %out, i64 %i
%src = getelementptr inbounds nuw i8, ptr %bytes, i64 %i
%b = load i8, ptr %src, align 1
%hi = lshr i8 %b, 4
%hi.0 = or disjoint i8 %hi, 48
%hi.c = icmp ugt i8 %b, -97
%hi.adj = select i1 %hi.c, i8 39, i8 0
%hi.d = add nuw nsw i8 %hi.0, %hi.adj
store i8 %hi.d, ptr %dst, align 1
%lo = and i8 %b, 15
%lo.0 = or disjoint i8 %lo, 48
%lo.c = icmp samesign ugt i8 %lo, 9
%lo.adj = select i1 %lo.c, i8 39, i8 0
%lo.d = add nuw nsw i8 %lo.0, %lo.adj
%dst1 = getelementptr inbounds nuw i8, ptr %dst, i64 1
store i8 %lo.d, ptr %dst1, align 1
%done = icmp eq i64 %i.next, 20
br i1 %done, label %exit, label %loop
exit:
ret void
}Assembly of both functions (x86-64, Intel syntax, directives removed)
encode_chunks_exact:
movd xmm5, dword ptr [rdi]
movdqa xmm3, xmm5
psrlw xmm3, 4
movdqa xmm2, xmmword ptr [rip + .LCPI2_0]
pand xmm3, xmm2
movdqa xmm0, xmmword ptr [rip + .LCPI2_1]
movdqa xmm6, xmm5
pminub xmm6, xmm0
pcmpeqb xmm6, xmm5
movd xmm1, dword ptr [rip + .LCPI2_2]
pandn xmm6, xmm1
paddb xmm6, xmm3
movdqa xmm3, xmmword ptr [rip + .LCPI2_3]
pand xmm5, xmm3
movdqa xmm4, xmmword ptr [rip + .LCPI2_4]
movdqa xmm7, xmm5
pcmpgtb xmm7, xmm4
pand xmm7, xmm1
paddb xmm7, xmm5
punpcklbw xmm6, xmm7
movdqa xmm5, xmmword ptr [rip + .LCPI2_5]
paddb xmm6, xmm5
movq qword ptr [rsi], xmm6
movd xmm6, dword ptr [rdi + 4]
movdqa xmm7, xmm6
psrlw xmm7, 4
pand xmm7, xmm2
movdqa xmm8, xmm6
pminub xmm8, xmm0
pcmpeqb xmm8, xmm6
pandn xmm8, xmm1
paddb xmm8, xmm7
pand xmm6, xmm3
movdqa xmm7, xmm6
pcmpgtb xmm7, xmm4
pand xmm7, xmm1
paddb xmm7, xmm6
punpcklbw xmm8, xmm7
paddb xmm8, xmm5
movq qword ptr [rsi + 8], xmm8
movd xmm6, dword ptr [rdi + 8]
movdqa xmm7, xmm6
psrlw xmm7, 4
pand xmm7, xmm2
movdqa xmm8, xmm6
pminub xmm8, xmm0
pcmpeqb xmm8, xmm6
pandn xmm8, xmm1
paddb xmm8, xmm7
pand xmm6, xmm3
movdqa xmm7, xmm6
pcmpgtb xmm7, xmm4
pand xmm7, xmm1
paddb xmm7, xmm6
punpcklbw xmm8, xmm7
paddb xmm8, xmm5
movq qword ptr [rsi + 16], xmm8
movd xmm6, dword ptr [rdi + 12]
movdqa xmm7, xmm6
psrlw xmm7, 4
pand xmm7, xmm2
movdqa xmm8, xmm6
pminub xmm8, xmm0
pcmpeqb xmm8, xmm6
pandn xmm8, xmm1
paddb xmm8, xmm7
pand xmm6, xmm3
movdqa xmm7, xmm6
pcmpgtb xmm7, xmm4
pand xmm7, xmm1
paddb xmm7, xmm6
punpcklbw xmm8, xmm7
paddb xmm8, xmm5
movq qword ptr [rsi + 24], xmm8
movd xmm6, dword ptr [rdi + 16]
movdqa xmm7, xmm6
psrlw xmm7, 4
pand xmm7, xmm2
pminub xmm0, xmm6
pcmpeqb xmm0, xmm6
pandn xmm0, xmm1
paddb xmm0, xmm7
pand xmm6, xmm3
movdqa xmm2, xmm6
pcmpgtb xmm2, xmm4
pand xmm2, xmm1
paddb xmm2, xmm6
punpcklbw xmm0, xmm2
paddb xmm0, xmm5
movq qword ptr [rsi + 32], xmm0
ret
encode_as_chunks:
movd xmm7, dword ptr [rdi]
movd xmm5, dword ptr [rdi + 4]
movdqa xmm3, xmm7
psrlw xmm3, 4
movdqa xmm2, xmmword ptr [rip + .LCPI0_0]
pand xmm3, xmm2
movdqa xmm4, xmm5
psrlw xmm4, 4
pand xmm4, xmm2
movdqa xmm0, xmmword ptr [rip + .LCPI0_1]
movdqa xmm6, xmm7
pminub xmm6, xmm0
pcmpeqb xmm6, xmm7
movdqa xmm8, xmm5
pminub xmm8, xmm0
pcmpeqb xmm8, xmm5
movd xmm1, dword ptr [rip + .LCPI0_2]
pandn xmm6, xmm1
paddb xmm6, xmm3
pandn xmm8, xmm1
paddb xmm8, xmm4
movdqa xmm3, xmmword ptr [rip + .LCPI0_3]
pand xmm7, xmm3
pand xmm5, xmm3
movdqa xmm4, xmmword ptr [rip + .LCPI0_4]
movdqa xmm9, xmm7
pcmpgtb xmm9, xmm4
movdqa xmm10, xmm5
pcmpgtb xmm10, xmm4
pand xmm9, xmm1
pand xmm10, xmm1
paddb xmm9, xmm7
punpcklbw xmm6, xmm9
paddb xmm10, xmm5
punpcklbw xmm8, xmm10
punpcklqdq xmm6, xmm8
movdqa xmm5, xmmword ptr [rip + .LCPI0_5]
paddb xmm6, xmm5
movdqu xmmword ptr [rsi], xmm6
movd xmm7, dword ptr [rdi + 8]
movd xmm6, dword ptr [rdi + 12]
movdqa xmm8, xmm7
psrlw xmm8, 4
pand xmm8, xmm2
movdqa xmm9, xmm6
psrlw xmm9, 4
pand xmm9, xmm2
movdqa xmm2, xmm7
pminub xmm2, xmm0
pcmpeqb xmm2, xmm7
pminub xmm0, xmm6
pcmpeqb xmm0, xmm6
pandn xmm2, xmm1
paddb xmm2, xmm8
pandn xmm0, xmm1
paddb xmm0, xmm9
pand xmm7, xmm3
pand xmm6, xmm3
movdqa xmm3, xmm7
pcmpgtb xmm3, xmm4
movdqa xmm8, xmm6
pcmpgtb xmm8, xmm4
pand xmm3, xmm1
pand xmm8, xmm1
paddb xmm3, xmm7
punpcklbw xmm2, xmm3
paddb xmm8, xmm6
punpcklbw xmm0, xmm8
punpcklqdq xmm2, xmm0
paddb xmm2, xmm5
movdqu xmmword ptr [rsi + 16], xmm2
movzx edx, byte ptr [rdi + 16]
mov r8d, edx
shr r8b, 4
xor eax, eax
cmp dl, -96
mov ecx, 39
mov r9d, 0
cmovae r9d, ecx
add r9b, r8b
add r9b, 48
mov byte ptr [rsi + 32], r9b
and dl, 15
cmp dl, 10
mov r8d, 0
cmovae r8d, ecx
add r8b, dl
add r8b, 48
mov byte ptr [rsi + 33], r8b
movzx edx, byte ptr [rdi + 17]
mov r8d, edx
shr r8b, 4
cmp dl, -96
mov r9d, 0
cmovae r9d, ecx
add r9b, r8b
add r9b, 48
mov byte ptr [rsi + 34], r9b
and dl, 15
cmp dl, 10
mov r8d, 0
cmovae r8d, ecx
add r8b, dl
add r8b, 48
mov byte ptr [rsi + 35], r8b
movzx edx, byte ptr [rdi + 18]
mov r8d, edx
shr r8b, 4
cmp dl, -96
mov r9d, 0
cmovae r9d, ecx
add r9b, r8b
add r9b, 48
mov byte ptr [rsi + 36], r9b
and dl, 15
cmp dl, 10
mov r8d, 0
cmovae r8d, ecx
add r8b, dl
add r8b, 48
mov byte ptr [rsi + 37], r8b
movzx edx, byte ptr [rdi + 19]
mov edi, edx
shr dil, 4
cmp dl, -96
mov r8d, 0
cmovae r8d, ecx
add r8b, dil
add r8b, 48
and dl, 15
cmp dl, 10
cmovae eax, ecx
mov byte ptr [rsi + 38], r8b
add al, dl
add al, 48
mov byte ptr [rsi + 39], al
retOptimized LLVM IR of both functions
define void @encode_chunks_exact(ptr noalias nofree noundef readonly captures(none) dereferenceable(20) %bytes, ptr noalias nofree noundef writeonly captures(none) dereferenceable(40) %out) unnamed_addr #0 personality ptr @rust_eh_personality !guid !13 {
_RINvYINtNtNtCskADBYZwJmSU_4core5slice4iter14ChunksExactMuthENtNtNtNtBa_4iter6traits8iterator8Iterator3zipRAhj14_ECs18u2dlOHsT1_15as_chunks_repro.exit:
%wide.load = load <4 x i8>, ptr %bytes, align 1
%0 = lshr <4 x i8> %wide.load, splat (i8 4)
%1 = or disjoint <4 x i8> %0, splat (i8 48)
%2 = icmp ugt <4 x i8> %wide.load, splat (i8 -97)
%3 = select <4 x i1> %2, <4 x i8> splat (i8 39), <4 x i8> zeroinitializer
%4 = add nuw nsw <4 x i8> %1, %3
%5 = and <4 x i8> %wide.load, splat (i8 15)
%6 = or disjoint <4 x i8> %5, splat (i8 48)
%7 = icmp samesign ugt <4 x i8> %5, splat (i8 9)
%8 = select <4 x i1> %7, <4 x i8> splat (i8 39), <4 x i8> zeroinitializer
%9 = add nuw nsw <4 x i8> %6, %8
%interleaved.vec = shufflevector <4 x i8> %4, <4 x i8> %9, <8 x i32> <i32 0, i32 4, i32 1, i32 5, i32 2, i32 6, i32 3, i32 7>
store <8 x i8> %interleaved.vec, ptr %out, align 1
%10 = getelementptr inbounds nuw i8, ptr %out, i64 8
%11 = getelementptr inbounds nuw i8, ptr %bytes, i64 4
%wide.load.1 = load <4 x i8>, ptr %11, align 1
%12 = lshr <4 x i8> %wide.load.1, splat (i8 4)
%13 = or disjoint <4 x i8> %12, splat (i8 48)
%14 = icmp ugt <4 x i8> %wide.load.1, splat (i8 -97)
%15 = select <4 x i1> %14, <4 x i8> splat (i8 39), <4 x i8> zeroinitializer
%16 = add nuw nsw <4 x i8> %13, %15
%17 = and <4 x i8> %wide.load.1, splat (i8 15)
%18 = or disjoint <4 x i8> %17, splat (i8 48)
%19 = icmp samesign ugt <4 x i8> %17, splat (i8 9)
%20 = select <4 x i1> %19, <4 x i8> splat (i8 39), <4 x i8> zeroinitializer
%21 = add nuw nsw <4 x i8> %18, %20
%interleaved.vec.1 = shufflevector <4 x i8> %16, <4 x i8> %21, <8 x i32> <i32 0, i32 4, i32 1, i32 5, i32 2, i32 6, i32 3, i32 7>
store <8 x i8> %interleaved.vec.1, ptr %10, align 1
%22 = getelementptr inbounds nuw i8, ptr %out, i64 16
%23 = getelementptr inbounds nuw i8, ptr %bytes, i64 8
%wide.load.2 = load <4 x i8>, ptr %23, align 1
%24 = lshr <4 x i8> %wide.load.2, splat (i8 4)
%25 = or disjoint <4 x i8> %24, splat (i8 48)
%26 = icmp ugt <4 x i8> %wide.load.2, splat (i8 -97)
%27 = select <4 x i1> %26, <4 x i8> splat (i8 39), <4 x i8> zeroinitializer
%28 = add nuw nsw <4 x i8> %25, %27
%29 = and <4 x i8> %wide.load.2, splat (i8 15)
%30 = or disjoint <4 x i8> %29, splat (i8 48)
%31 = icmp samesign ugt <4 x i8> %29, splat (i8 9)
%32 = select <4 x i1> %31, <4 x i8> splat (i8 39), <4 x i8> zeroinitializer
%33 = add nuw nsw <4 x i8> %30, %32
%interleaved.vec.2 = shufflevector <4 x i8> %28, <4 x i8> %33, <8 x i32> <i32 0, i32 4, i32 1, i32 5, i32 2, i32 6, i32 3, i32 7>
store <8 x i8> %interleaved.vec.2, ptr %22, align 1
%34 = getelementptr inbounds nuw i8, ptr %out, i64 24
%35 = getelementptr inbounds nuw i8, ptr %bytes, i64 12
%wide.load.3 = load <4 x i8>, ptr %35, align 1
%36 = lshr <4 x i8> %wide.load.3, splat (i8 4)
%37 = or disjoint <4 x i8> %36, splat (i8 48)
%38 = icmp ugt <4 x i8> %wide.load.3, splat (i8 -97)
%39 = select <4 x i1> %38, <4 x i8> splat (i8 39), <4 x i8> zeroinitializer
%40 = add nuw nsw <4 x i8> %37, %39
%41 = and <4 x i8> %wide.load.3, splat (i8 15)
%42 = or disjoint <4 x i8> %41, splat (i8 48)
%43 = icmp samesign ugt <4 x i8> %41, splat (i8 9)
%44 = select <4 x i1> %43, <4 x i8> splat (i8 39), <4 x i8> zeroinitializer
%45 = add nuw nsw <4 x i8> %42, %44
%interleaved.vec.3 = shufflevector <4 x i8> %40, <4 x i8> %45, <8 x i32> <i32 0, i32 4, i32 1, i32 5, i32 2, i32 6, i32 3, i32 7>
store <8 x i8> %interleaved.vec.3, ptr %34, align 1
%46 = getelementptr inbounds nuw i8, ptr %out, i64 32
%47 = getelementptr inbounds nuw i8, ptr %bytes, i64 16
%wide.load.4 = load <4 x i8>, ptr %47, align 1
%48 = lshr <4 x i8> %wide.load.4, splat (i8 4)
%49 = or disjoint <4 x i8> %48, splat (i8 48)
%50 = icmp ugt <4 x i8> %wide.load.4, splat (i8 -97)
%51 = select <4 x i1> %50, <4 x i8> splat (i8 39), <4 x i8> zeroinitializer
%52 = add nuw nsw <4 x i8> %49, %51
%53 = and <4 x i8> %wide.load.4, splat (i8 15)
%54 = or disjoint <4 x i8> %53, splat (i8 48)
%55 = icmp samesign ugt <4 x i8> %53, splat (i8 9)
%56 = select <4 x i1> %55, <4 x i8> splat (i8 39), <4 x i8> zeroinitializer
%57 = add nuw nsw <4 x i8> %54, %56
%interleaved.vec.4 = shufflevector <4 x i8> %52, <4 x i8> %57, <8 x i32> <i32 0, i32 4, i32 1, i32 5, i32 2, i32 6, i32 3, i32 7>
store <8 x i8> %interleaved.vec.4, ptr %46, align 1
ret void
}
define void @encode_as_chunks(ptr noalias nofree noundef readonly captures(none) dereferenceable(20) %bytes, ptr noalias nofree noundef writeonly captures(none) dereferenceable(40) %out) unnamed_addr #0 personality ptr @rust_eh_personality !guid !4 {
start:
%0 = getelementptr inbounds nuw i8, ptr %out, i64 8
%1 = getelementptr inbounds nuw i8, ptr %bytes, i64 4
%wide.load = load <4 x i8>, ptr %bytes, align 1
%wide.load6 = load <4 x i8>, ptr %1, align 1
%2 = lshr <4 x i8> %wide.load, splat (i8 4)
%3 = lshr <4 x i8> %wide.load6, splat (i8 4)
%4 = or disjoint <4 x i8> %2, splat (i8 48)
%5 = or disjoint <4 x i8> %3, splat (i8 48)
%6 = icmp ugt <4 x i8> %wide.load, splat (i8 -97)
%7 = icmp ugt <4 x i8> %wide.load6, splat (i8 -97)
%8 = select <4 x i1> %6, <4 x i8> splat (i8 39), <4 x i8> zeroinitializer
%9 = select <4 x i1> %7, <4 x i8> splat (i8 39), <4 x i8> zeroinitializer
%10 = add nuw nsw <4 x i8> %4, %8
%11 = add nuw nsw <4 x i8> %5, %9
%12 = and <4 x i8> %wide.load, splat (i8 15)
%13 = and <4 x i8> %wide.load6, splat (i8 15)
%14 = or disjoint <4 x i8> %12, splat (i8 48)
%15 = or disjoint <4 x i8> %13, splat (i8 48)
%16 = icmp samesign ugt <4 x i8> %12, splat (i8 9)
%17 = icmp samesign ugt <4 x i8> %13, splat (i8 9)
%18 = select <4 x i1> %16, <4 x i8> splat (i8 39), <4 x i8> zeroinitializer
%19 = select <4 x i1> %17, <4 x i8> splat (i8 39), <4 x i8> zeroinitializer
%20 = add nuw nsw <4 x i8> %14, %18
%21 = add nuw nsw <4 x i8> %15, %19
%interleaved.vec = shufflevector <4 x i8> %10, <4 x i8> %20, <8 x i32> <i32 0, i32 4, i32 1, i32 5, i32 2, i32 6, i32 3, i32 7>
store <8 x i8> %interleaved.vec, ptr %out, align 1
%interleaved.vec7 = shufflevector <4 x i8> %11, <4 x i8> %21, <8 x i32> <i32 0, i32 4, i32 1, i32 5, i32 2, i32 6, i32 3, i32 7>
store <8 x i8> %interleaved.vec7, ptr %0, align 1
%22 = getelementptr inbounds nuw i8, ptr %out, i64 16
%23 = getelementptr inbounds nuw i8, ptr %out, i64 24
%24 = getelementptr inbounds nuw i8, ptr %bytes, i64 8
%25 = getelementptr inbounds nuw i8, ptr %bytes, i64 12
%wide.load.1 = load <4 x i8>, ptr %24, align 1
%wide.load6.1 = load <4 x i8>, ptr %25, align 1
%26 = lshr <4 x i8> %wide.load.1, splat (i8 4)
%27 = lshr <4 x i8> %wide.load6.1, splat (i8 4)
%28 = or disjoint <4 x i8> %26, splat (i8 48)
%29 = or disjoint <4 x i8> %27, splat (i8 48)
%30 = icmp ugt <4 x i8> %wide.load.1, splat (i8 -97)
%31 = icmp ugt <4 x i8> %wide.load6.1, splat (i8 -97)
%32 = select <4 x i1> %30, <4 x i8> splat (i8 39), <4 x i8> zeroinitializer
%33 = select <4 x i1> %31, <4 x i8> splat (i8 39), <4 x i8> zeroinitializer
%34 = add nuw nsw <4 x i8> %28, %32
%35 = add nuw nsw <4 x i8> %29, %33
%36 = and <4 x i8> %wide.load.1, splat (i8 15)
%37 = and <4 x i8> %wide.load6.1, splat (i8 15)
%38 = or disjoint <4 x i8> %36, splat (i8 48)
%39 = or disjoint <4 x i8> %37, splat (i8 48)
%40 = icmp samesign ugt <4 x i8> %36, splat (i8 9)
%41 = icmp samesign ugt <4 x i8> %37, splat (i8 9)
%42 = select <4 x i1> %40, <4 x i8> splat (i8 39), <4 x i8> zeroinitializer
%43 = select <4 x i1> %41, <4 x i8> splat (i8 39), <4 x i8> zeroinitializer
%44 = add nuw nsw <4 x i8> %38, %42
%45 = add nuw nsw <4 x i8> %39, %43
%interleaved.vec.1 = shufflevector <4 x i8> %34, <4 x i8> %44, <8 x i32> <i32 0, i32 4, i32 1, i32 5, i32 2, i32 6, i32 3, i32 7>
store <8 x i8> %interleaved.vec.1, ptr %22, align 1
%interleaved.vec7.1 = shufflevector <4 x i8> %35, <4 x i8> %45, <8 x i32> <i32 0, i32 4, i32 1, i32 5, i32 2, i32 6, i32 3, i32 7>
store <8 x i8> %interleaved.vec7.1, ptr %23, align 1
%_3.i.i = getelementptr inbounds nuw i8, ptr %out, i64 32
%_3.i1.i = getelementptr inbounds nuw i8, ptr %bytes, i64 16
%b = load i8, ptr %_3.i1.i, align 1, !noundef !5
%_15 = lshr i8 %b, 4
%_29 = or disjoint i8 %_15, 48
%_31 = icmp ugt i8 %b, -97
%. = select i1 %_31, i8 39, i8 0
%_14 = add nuw nsw i8 %_29, %.
store i8 %_14, ptr %_3.i.i, align 1
%_32 = and i8 %b, 15
%_33 = or disjoint i8 %_32, 48
%_35 = icmp samesign ugt i8 %_32, 9
%_34.sroa.0.0 = select i1 %_35, i8 39, i8 0
%_16 = add nuw nsw i8 %_33, %_34.sroa.0.0
%46 = getelementptr inbounds nuw i8, ptr %out, i64 33
store i8 %_16, ptr %46, align 1
%_3.i.i.1 = getelementptr inbounds nuw i8, ptr %out, i64 34
%_3.i1.i.1 = getelementptr inbounds nuw i8, ptr %bytes, i64 17
%b.1 = load i8, ptr %_3.i1.i.1, align 1, !noundef !5
%_15.1 = lshr i8 %b.1, 4
%_29.1 = or disjoint i8 %_15.1, 48
%_31.1 = icmp ugt i8 %b.1, -97
%..1 = select i1 %_31.1, i8 39, i8 0
%_14.1 = add nuw nsw i8 %_29.1, %..1
store i8 %_14.1, ptr %_3.i.i.1, align 1
%_32.1 = and i8 %b.1, 15
%_33.1 = or disjoint i8 %_32.1, 48
%_35.1 = icmp samesign ugt i8 %_32.1, 9
%_34.sroa.0.0.1 = select i1 %_35.1, i8 39, i8 0
%_16.1 = add nuw nsw i8 %_33.1, %_34.sroa.0.0.1
%47 = getelementptr inbounds nuw i8, ptr %out, i64 35
store i8 %_16.1, ptr %47, align 1
%_3.i.i.2 = getelementptr inbounds nuw i8, ptr %out, i64 36
%_3.i1.i.2 = getelementptr inbounds nuw i8, ptr %bytes, i64 18
%b.2 = load i8, ptr %_3.i1.i.2, align 1, !noundef !5
%_15.2 = lshr i8 %b.2, 4
%_29.2 = or disjoint i8 %_15.2, 48
%_31.2 = icmp ugt i8 %b.2, -97
%..2 = select i1 %_31.2, i8 39, i8 0
%_14.2 = add nuw nsw i8 %_29.2, %..2
store i8 %_14.2, ptr %_3.i.i.2, align 1
%_32.2 = and i8 %b.2, 15
%_33.2 = or disjoint i8 %_32.2, 48
%_35.2 = icmp samesign ugt i8 %_32.2, 9
%_34.sroa.0.0.2 = select i1 %_35.2, i8 39, i8 0
%_16.2 = add nuw nsw i8 %_33.2, %_34.sroa.0.0.2
%48 = getelementptr inbounds nuw i8, ptr %out, i64 37
store i8 %_16.2, ptr %48, align 1
%_3.i.i.3 = getelementptr inbounds nuw i8, ptr %out, i64 38
%_3.i1.i.3 = getelementptr inbounds nuw i8, ptr %bytes, i64 19
%b.3 = load i8, ptr %_3.i1.i.3, align 1, !noundef !5
%_15.3 = lshr i8 %b.3, 4
%_29.3 = or disjoint i8 %_15.3, 48
%_31.3 = icmp ugt i8 %b.3, -97
%..3 = select i1 %_31.3, i8 39, i8 0
%_14.3 = add nuw nsw i8 %_29.3, %..3
store i8 %_14.3, ptr %_3.i.i.3, align 1
%_32.3 = and i8 %b.3, 15
%_33.3 = or disjoint i8 %_32.3, 48
%_35.3 = icmp samesign ugt i8 %_32.3, 9
%_34.sroa.0.0.3 = select i1 %_35.3, i8 39, i8 0
%_16.3 = add nuw nsw i8 %_33.3, %_34.sroa.0.0.3
%49 = getelementptr inbounds nuw i8, ptr %out, i64 39
store i8 %_16.3, ptr %49, align 1
ret void
}Benchmark (src/main.rs; the four functions in src/lib.rs, release profile with codegen-units = 1)
The slice functions are the two above with bytes: &[u8], out: &mut [u8].
//! Best of 9 runs per case; pin it to one core (`taskset -c 2 cargo run --release`).
use as_chunks_repro::*;
use std::hint::black_box;
use std::time::Instant;
fn best(iters: u64, mut f: impl FnMut()) -> f64 {
let mut best = f64::MAX;
for _ in 0..9 {
let t = Instant::now();
for _ in 0..iters {
f();
}
best = best.min(t.elapsed().as_secs_f64() * 1e9 / iters as f64);
}
best
}
fn main() {
let inputs: Vec<[u8; 20]> = (0..1024u32)
.map(|i| core::array::from_fn(|j| (i as u8).wrapping_mul(31).wrapping_add(j as u8 * 7)))
.collect();
for x in &inputs {
let (mut p, mut q) = ([0u8; 40], [0u8; 40]);
encode_chunks_exact(x, &mut p);
encode_as_chunks(x, &mut q);
assert_eq!(p, q);
}
let mut out = [0u8; 40];
let mut k = 0usize;
let ce = best(20_000_000, || {
encode_chunks_exact(black_box(&inputs[k & 1023]), black_box(&mut out));
k += 1;
});
let ac = best(20_000_000, || {
encode_as_chunks(black_box(&inputs[k & 1023]), black_box(&mut out));
k += 1;
});
println!(
"[u8; 20] -> [u8; 40] chunks_exact_mut {ce:7.2} ns as_chunks_mut {ac:7.2} ns ({:+.0}%)",
(ac / ce - 1.0) * 100.0
);
for len in [20usize, 4096] {
let src: Vec<u8> = (0..len).map(|i| (i * 37) as u8).collect();
let mut dst = vec![0u8; 2 * len];
let iters = (400_000_000 / len as u64).max(1);
let ce = best(iters, || encode_chunks_exact_slice(black_box(&src), black_box(&mut dst)));
let ac = best(iters, || encode_as_chunks_slice(black_box(&src), black_box(&mut dst)));
println!(
"slices, {len:4} bytes chunks_exact_mut {ce:7.2} ns as_chunks_mut {ac:7.2} ns ({:+.0}%)",
(ac / ce - 1.0) * 100.0
);
}
}$ taskset -c 6 cargo run --release
[u8; 20] -> [u8; 40] chunks_exact_mut 3.82 ns as_chunks_mut 4.37 ns (+14%)
slices, 20 bytes chunks_exact_mut 4.96 ns as_chunks_mut 3.61 ns (-27%)
slices, 4096 bytes chunks_exact_mut 297.66 ns as_chunks_mut 296.93 ns (-0%)
Other as_chunks codegen issues, about remainder loops: #150647, #147459.
Meta
rustc --version --verbose:
rustc 1.99.1 (58cf8f9c3 2026-10-13)
binary: rustc
commit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67
commit-date: 2026-10-13
host: x86_64-unknown-linux-gnu
release: 1.99.1
LLVM version: 23.1.1
The same instruction counts on rustc 1.101.0-nightly (16d1ffac1 2026-10-26) (LLVM 23.1.4) and on 1.95.0 (LLVM 22.1.2).
Activity