Repository navigation
Qwen3.5-4B: quant_generate works but quant_ask produces empty/garbage output #69
Description
Description
Qwen3.5-4B (DeltaNet hybrid) loads successfully and produces coherent output via quant_generate, but quant_ask returns empty/garbage tokens. This breaks the server API since quant-server uses quant_ask for non-streaming requests.
Evidence
quant_generate (CLI) — WORKS
$ ./qwen35_test Qwen3.5-4B-Q4_K_M.gguf "Hello, who are you?"
--- response ---
I am Qwen3.5, a large language model developed by Alibaba Cloud.
I can help you with various tasks such as answering questions,
solving problems, and generating content.quant_ask (server) — BROKEN
$ curl localhost:8080/v1/chat/completions \
-d '{"messages":[{"role":"user","content":"What is gravity?"}]}'
{"content": " -\n.\n- \n1. - "} # empty/whitespace tokensStreaming via server — ALSO BROKEN
data: {"delta":{"content":"!"}}
data: {"delta":{"content":"!"}}
data: {"delta":{"content":"!"}}
# Repeating "!" tokens
Root Cause Hypothesis
quant_ask and quant_generate may handle the prompt/tokenization differently. Possible causes:
quant_askmay apply its own chat template that conflicts with the ChatML template already in the promptquant_askmay not properly initialize DeltaNet state (conv buffer, delta state) between calls- The BOS token handling may differ between the two paths
Steps to Reproduce
// This works:
quant_generate(ctx, "<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\n", cb, NULL);
// This produces garbage:
char* result = quant_ask(ctx, "<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\n");Environment
- quant.cpp: latest main (1e1ea2c)
- Model: unsloth/Qwen3.5-4B-GGUF (Q4_K_M, 2.6GB)
- Architecture: qwen35 (DeltaNet hybrid, 8 attn + 24 DeltaNet layers)
- OS: macOS 15 (Apple M3, 16GB)
Reported by ClawTeam Claw-4 (Optimizer) + Claw-5 (Researcher)
Activity
unamedkr commentedon Apr 12, 2026
Update: quant_generate also produces empty tokens via Python ctypes
Switched the Python server from
quant_asktoquant_generatewith streaming callback — same result: empty/whitespace-only tokens.However, the exact same C code compiled as a standalone binary produces perfect output:
This narrows the issue to the Python ctypes / shared library boundary:
libquant.dylib→ BROKEN for Qwen3.5 (but works for Phi-3.5 and SmolLM2)Hypothesis
Qwen3.5's DeltaNet layers use large FP32 dequantized tensors (24 layers × attn_qkv dequant). The
cc -shared -fPICbuild may handle memory differently thancc -O2static builds:quant.h(static variables) may not initialize correctly in shared library modeSuggested investigation
-fno-picor position-dependent codeUpdated by ClawTeam Claw-5 (Researcher)
feat(libturboquant): port Phi-3 fused QKV/FFN + LongRoPE to split sou…
unamedkr commentedon Apr 13, 2026
Update: Qwen3.5-4B basic fact extraction works (3/3)
Direct tests show Qwen3.5-4B can extract facts correctly:
The Acme 2/7 failure is NOT a model capability issue. Root cause:
RLV pipeline's prompts (ANSWER/NONE format, system prompt) are
optimized for Phi-3.5's output patterns. Qwen3.5-4B produces
longer, more verbose answers that don't match the pipeline's
parsing expectations.
Fix: prompt tuning for Qwen3.5-4B's response style, or a
model-agnostic prompt format.