CPU quantize_blockwise stores 0.0 as a neighbouring code, so Adam8bit on CPU moves weights with zero gradient #2118

System Info

Linux x86-64, CPU only (no GPU), Python 3.13, torch 2.10.0 CPU. bitsandbytes built from source with -DCOMPUTE_BACKEND=cpu, main at a616886cacf86701514f0cb1d6be760f967481c0. The same behaviour shows up with the 0.50.2 wheel.

Reproduction

import torch
import bitsandbytes.functional as F

x = torch.zeros(256)
x[0] = 1.0
q, st = F.quantize_blockwise(x, blocksize=256)
print(q[1].item(), F.dequantize_blockwise(q, st)[1].item())

The CPU path looks codes up in a 65,536-entry table indexed by the normalized value. Near zero the dynamic map has codes closer together than one table bin, so a bin can return a neighbouring code instead of the nearest one.

With Adam8bit this matters for any weight whose gradient is always 0. Its first moment is quantized and read back as a small non-zero value, so the weight moves, while torch.optim.Adam leaves it alone. In my run the weight ended at -3.6e-6 after two steps.

Expected behavior

0.0 maps to code index 127 and dequantizes to exactly 0.0, as it does on CUDA and with the default backend (which picks the nearest code).

Observed on main:

131 2.1249998098937795e-05

Expected: 127 0.0.