Skip to content

fit-params: accept --spec-type, -md, -ngld arguments - #30427

Open
claudiubelu wants to merge 2 commits into
ggml-org:masterfrom
claudiubelu:llama-fit-params-fix
Open

claudiubelu wants to merge 2 commits into
ggml-org:masterfrom
claudiubelu:llama-fit-params-fix

Conversation

@claudiubelu

@claudiubelu claudiubelu commented Oct 12, 2026 •

Copy link
Copy Markdown

Overview

llama-server accepts the --spec-type, -md, -ngld parameters and fits the draft / MTP context with the target, but llama-fit-params does not. Anything that estimates a server config with llama-fit-params will get a wrong estimate.

llama-fit-params now accepts the same flags, improving the estimates.

Additional information

Fixes: #30424

Repro:

PS D:\work\llama.cpp> .\build\bin\Release\llama-fit-params.exe -m ..\models\Qwen3.5-9B\Qwen3.5-9B-IQ4_XS-mtp-latin-outq4k.gguf --fit off --fit-print on
0.01.957.464 I llama_fit_params: printing estimated memory in MiB to stdout (device, model, context, compute) ...
CUDA0 3895 8242 344
Host 318 0 272
PS D:\work\llama.cpp> .\build\bin\Release\llama-fit-params.exe -m ..\models\Qwen3.5-9B\Qwen3.5-9B-IQ4_XS-mtp-latin-outq4k.gguf --fit off --fit-print on --spec-type draft-mtp
error: invalid argument: --spec-type

# With this fix:
PS D:\work\llama.cpp> .\build-dev\bin\Release\llama-fit-params.exe -m ..\models\Qwen3.5-9B\Qwen3.5-9B-IQ4_XS-mtp-latin-outq4k.gguf --fit off --fit-print on
0.01.941.872 I llama_fit_params: printing estimated memory in MiB to stdout (device, model, context, compute) ...
CUDA0 3895 8242 344
Host 318 0 272
PS D:\work\llama.cpp> .\build-dev\bin\Release\llama-fit-params.exe -m ..\models\Qwen3.5-9B\Qwen3.5-9B-IQ4_XS-mtp-latin-outq4k.gguf --fit off --fit-print on --spec-type draft-mtp
0.00.745.028 I llama_fit_params: printing estimated memory in MiB to stdout (device, model, context, compute) ...
CUDA0 4039 9417 644
Host 318 0 544

PS D:\work\llama.cpp> .\build-dev\bin\Release\llama-fit-params.exe -m ..\models\Qwen3.5-9B\Qwen3.5-9B-Q4_K_M.gguf --fit off --fit-print on --spec-type draft-mtp
0.01.925.228 I llama_fit_params: printing estimated memory in MiB to stdout (device, model, context, compute) ...
0.03.728.047 W common_fit_print: failed to measure the memory of the extra model, printing the main model alone: the model has no MTP layers (needed by --spec-type draft-mtp)
CUDA0 4861 8393 501
Host 545 0 272

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES, it discovered mismatch between memory usage estimates and actual usage, discovered that it was due to MTP not being accounted for, and it proposed a solution. I have reviewed its changed, tested the binaries, ran the CI (https://pastebin.com/zxaUZDvD).

Assisted-by: Claude Opus 5.5
`llama-server` accepts the `--spec-type`, `-md`, `-ngld` parameters and
fits the draft / MTP context with the target, but `llama-fit-params` does not.
Anything that estimates a server config with `llama-fit-params` will get
a wrong estimate.

`llama-fit-params` now accepts the same flags, improving the estimates.

Assisted-by: Claude Opus 5.5
@github-actions github-actions Bot added testing Everything test related examples labels Oct 12, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

examples testing Everything test related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Misc. bug: llama-fit-params does not accept --spec-type, -md, -ngld arguments

1 participant