[jomplox]

Hey @kernelpool! I'm working on running the full released V4.1 weights (FP8 dense, FP4 experts) on DwarfStar with no extra quantization. So far I can get token generation working on the full weights, but only at a barely usable speed (~1 tok/s). I saw that your branch also focuses on performance, so I thought we could combine our work in a single PR. If you'd rather not have it here, just let me know and I'll open the PR against the main repo instead.

Summary

Right now, this engine can only run the released V4.1 checkpoints after the dense weights go through another round of quantization. This PR adds an experimental native recipe plus a single-Mac Metal path that run the checkpoints with their original weight values. That includes the 1017 checkpoint.

Not supported with native. These paths refuse to load it: CUDA, distributed inference, DSpark, vision, and concurrent decode batching.

What's in this PR

Converter

• Dense weights are expanded to F32, which is lossless.
• Expert weights keep their original FP4 codes and UE8M0 scales.
• Every native payload is audited after conversion.
• An optional APFS-clone packer stores a dense matrix as BF16 only when every value fits BF16 exactly. It checks each value and then verifies a bitwise round trip.
• The original model file is never modified.

Metal inference

• The kernels use the same arithmetic as the reference: native activation rounding, scale-block accumulation, and the same HC product/addition boundaries.
• SSD streaming has two modes. It can load the selected experts into the expert cache, within that cache's memory budget. Otherwise it uses two whole-layer read buffers.
• Batched prefill can optionally widen weights to BF16, with checks.
• Native snapshots from an incompatible arithmetic revision are rejected.

Memory fix

• Streamed Metal buffers called from C were not being released. Scoped autorelease pools now free completed buffers. Numbers are under Validation below.

Not touched: the server API and the OpenCode cache. ds4_server.c is unchanged.

Validation

Hardware: Apple M5 Max, 128 GiB, macOS/Metal. Base: 72ccf363174c3e1acf840b3661a956d4fb321071 on ds41-perf.

• Builds: the Metal and CPU builds both pass. The CPU backend was only built; I did not run full-model CPU inference.
• Conversion: all 7 conversion tests pass. They cover exact packing, payloads that must stay unconverted, full auditing of native payloads, and catching corruption outside the legacy sample set.
• Other suites pass: the independent source-arithmetic test, native Metal source checks, the standard V4.1 Metal kernel suite, server unit tests, GGUF layout tests, and Engram tests.
• Small test model (520 tokens, 40 layers): passes in all three modes (resident, uncached SSD, cached SSD). The covered cases are serial and batched snapshot continuation, rejection of obsolete snapshots, recovery after cancellation, and wide/partitioned prefill.
• Bit-exactness: all 33,280 logits from the test model are bit-identical across the three modes and against a fresh build of my original implementation. Compared with an independently generated first-token reference, the largest difference is 5.960464477539063e-08.
• Memory leak fix: I ran the streamed-buffer and command-lifetime checks with Metal validation enabled, and they pass. On the unmodified base, the streamed-buffer test grows from 424 MiB at 32 iterations to 680 MiB at 64. With the fix, it stays at about 177 MiB through 256 iterations.
• Reproducibility: the test-model generator rebuilds the test model byte-for-byte from the pinned reference sources.

Commands run

make -j1 all
make -j1 cpu
python3 tests/test_deepseek41_conversion.py
python3 tests/test_deepseek41_native_reference.py
./ds4_test --server
./tests/test_deepseek41_gguf
./tests/test_engram

# Metal checks, all with MTL_DEBUG_LAYER=1:
./tests/test_deepseek41_metal                       # default and --native-source
./tests/test_metal_command_memory                   # default and buffers
./tests/test_deepseek41_native_graph MODEL TOKENS OUTPUT   # resident, stream, and cache

The commands that generate the test model and source logits are in docs/DEEPSEEK41_NATIVE.md.

Full-model speed and accuracy comparison

This compares Before (my original implementation on 31786cb594121cc1ff9e5f0ad78bc766f6fde17f) with After (this PR on the ds41-perf base above). The unmodified upstream base can't load the native recipe, so it isn't included.

• Model: the full 1017 checkpoint, using the verified lossless BF16 dense weights and the original FP4 experts.
• Run order: Before / After / After / Before. Each run followed the same 16-token warmup (using Before), which was not measured.
• Shared settings: both versions used the same ds4_bench.c, model file, 64 GiB expert cache, 262,144-token context allocation, 8,192-row prefill chunk, public prompt, and diagnostics.
• Per context size: 128 greedy tokens were generated, with EOS skipped.
• Caveat: OS file-cache effects are included, so these are not uncached SSD numbers.

Median throughput (two runs per version):

Context	Metric	Before (tok/s)	After (tok/s)	Change
2,048	Prefill	6.570	6.595	+0.38%
2,048	Generation	0.960	0.960	0.00%
4,096	Prefill (next 2,048 tokens)	6.565	6.575	+0.15%
4,096	Generation	1.050	1.025	−2.38%

• Speed: before running, I set a threshold of a median regression greater than 3% at either context size. The result stays under that threshold. Generation at 4,096 tokens is still 2.4% slower, though, so this is not a speedup.
• Memory: peak sampled RSS was 81.82 GiB, and macOS memory pressure stayed normal.
• Accuracy: each run compared 33,354,240 logits (2 full 129,280-value context-boundary vectors plus 256 generation vectors). All of them were finite and bit-identical across the four runs.

Benchmark command (ARM = before or after):

DS4_BENCH_DECODE_LOGITS_HASH=1 DS4_BENCH_DECODE_LOGITS_DUMP=/tmp/ARM-decode.f32 \
  ./ds4-bench --metal --ssd-streaming --ssd-streaming-cache-experts 64GB \
  -m /path/to/native-lossless-bf16.gguf \
  --prompt-file speed-bench/promessi_sposi.txt --ctx-alloc 262144 \
  --prefill-chunk 8192 --ctx-start 2048 --ctx-max 4096 --step-incr 2048 \
  --gen-tokens 128 --csv /tmp/ARM.csv \
  --dump-frontier-logits-dir /tmp/ARM-logits --show-output

Raw CSVs, in the order they ran

Run 1 (Before):

ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,gen_first_ms,gen_steady_tokens,gen_steady_tps,kvcache_bytes
2048,2048,6.55,128,0.96,5145.187,127,0.99,24118324
4096,2048,6.56,128,1.07,5136.990,127,1.11,0

Run 2 (After):

ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,gen_first_ms,gen_steady_tokens,gen_steady_tps,kvcache_bytes
2048,2048,6.60,128,0.97,5230.672,127,1.00,24118324
4096,2048,6.57,128,1.02,5171.629,127,1.06,0

Run 3 (After):

ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,gen_first_ms,gen_steady_tokens,gen_steady_tps,kvcache_bytes
2048,2048,6.59,128,0.95,5200.361,127,0.98,24118324
4096,2048,6.58,128,1.03,5161.250,127,1.07,0

Run 4 (Before):

ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,gen_first_ms,gen_steady_tokens,gen_steady_tps,kvcache_bytes
2048,2048,6.59,128,0.96,5130.125,127,0.99,24118324
4096,2048,6.57,128,1.03,5156.319,127,1.07,0

After benchmarking, I restored my original runtime. I confirmed that its 39 protected source/runtime files, Git status/diff, model identities, and OpenCode configuration were unchanged.

Known gaps

• CPU reference: after the first token, the logits are not bit-identical to the CPU reference. On the test model, the top-1 token matches at 497 of 520 positions. The docs also list the existing differences between the full checkpoint and the source.
• Not tested: CUDA, multi-host runs, and full coding-task acceptance for this port. Having the original weights and matching the original Metal implementation doesn't prove CUDA parity or coding quality.