A C/Metal inference engine that streams mixture-of-experts model weights from SSD. Built on JustVugg/colibri, with my work on Apple Silicon execution, serving, and measurement tooling.
It explores a practical question: how much of a large model can stay on disk while the active work fits in a laptop's memory?
Status: experimental. The primary measured configuration is GLM-5.2 int4 on an M5 Max with 128 GB of unified memory. The model occupies roughly 370 GB on disk; building and testing the engine does not require downloading it.
git clone https://github.com/PhilipJohnBasile/iliria.git
cd iliria
makeA C compiler and OpenMP are required. On macOS, the Makefile looks for Homebrew's libomp. Python 3 is used for the test suite and optional serving tools.
make -C c test-cThis runs the portable C tests without downloading model weights. The broader Python regression suite has known failures; verification notes link the results and reproduction commands. Model loading, quality evaluations, and GPU performance are separate checks.
The project includes Apple Silicon paths, an HTTP serving surface, quantization and cache experiments, and tools that compare candidate changes against a fixed reference. The NOTICE credits the upstream engine; the engine guide records which experiments worked, regressed, or remain incomplete.
The core design keeps dense weights resident and reads routed experts as needed. That reduces the memory requirement while putting SSD bandwidth and expert reuse on the critical path.
flowchart LR
A[Prompt] --> B[Resident attention and shared weights]
B --> C[Expert routing]
D[Expert weights on SSD] --> E[Expert cache]
C --> E
E --> F[Next token]
The published M5 Max record reports approximately 1.5 tokens/second warm decode for the 744B model. This is a hardware-specific research result. Prefill latency, cache state, workload, and GPU contention materially affect usability.
The measurement record includes the original conditions and unsuccessful experiments. The performance analysis distinguishes provisional results from changes that passed the project's comparison matrix. Follow the model setup and serving guide when you are ready to run weights.
trailbrake provides the dense MLX tier. racecontrol routes requests between engines and handles fallback. iliria-fm explores Apple's Foundation Models integration.
Source · Validation tools · Roadmap · Release artifacts
Apache 2.0. See NOTICE for upstream attribution and the model provider's terms for weights.