Basalt is a local inference engine for the Qwen3.8-Flash-Next mixture-of-experts model. It runs on Linux and Windows with an NVIDIA Blackwell GPU (sm_120); see platforms for limitations.

Experts that do not fit in VRAM live in system RAM, and optionally on a second GPU used as a peer expert tier. A large MoE model runs on one workstation, behind an OpenAI- and Anthropic-compatible API.

Help wanted: have some spare compute to lend for testing? See Contributing.

Requirements

• Linux x86-64 with AVX2 (AVX-512 is used when present), and an NVIDIA Blackwell GPU (compute capability 12.0) with CUDA 13.
• CMake 3.24 or newer, a C++20 compiler, and Python 3.10 or newer.
• Enough system RAM to hold the experts that do not fit in VRAM.

Quick start

git clone https://github.com/jesdga95/basalt && cd basalt
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j                  # bin/basalt, bin/basalt-bench, bin/basalt-probe, bin/basalt-vision
pip install -r requirements.txt
hf download jesdga/Qwen3.8-Flash-Next-Basalt Qwen3.8-FN-NVFP4.basalt --local-dir models

Put your settings in basalt_config.json, with model pointing at the downloaded file:

{
  "server": { "model": "models/Qwen3.8-FN-NVFP4.basalt", "gpu": "0,1", "port": 8080 },
  "engine": { "expert-cache": "auto", "prefill": "auto", "kv": "int8", "resident-experts": true }
}

bin/basalt serve --config basalt_config.json

The server listens on 127.0.0.1 by default.

Models

Ready-to-run packs, with the quality of each measured against Q8_0, are on Hugging Face: Qwen3.8-FN-NVFP4, -UD-Q4_K_XL, -UD-Q5_K_XL, -UD-Q6_K_XL, -Q8_0, -GSQ-RCO-IQ3_S, -GSQ-RCO-IQ3_XXS and -GSQ-RCO-Q2_0 (each .basalt). A model is one file: weights, tokenizer, MTP drafter, expert profile and vision projector.

Docs

• Configuration: every setting.
• Running: parallel requests, prompt chunks, memory floors, image input.
• basalt-probe: measures the machine, the cold decode speed, and suggests values.
• The .basalt model file: the layout, and building a pack yourself (including NVFP4).
• Platforms: Linux and Windows requirements, the build steps and the Windows limitations.
• Changelog: what each release changed.
• Building and testing, Precision and quality, Contributing.

Credits

• Strata: the project Basalt forks (MIT).
• Qwen: the model. The packs are under the Qwen Community License 1.0, not Basalt's MIT license.
• unsloth: the UD and Q8_0 GGUFs the packs are built from.
• IST Austria DAS Lab: the GSQ-RCO quants; please cite GSQ and RCO.
• NVIDIA: the NVFP4 experts.
• ggml and llama.cpp: quantization formats and kernels extracted into src/ (MIT, see THIRD_PARTY_LICENSES).

License

MIT, see LICENSE.