llm-d

Helm charts for llm-d

Self-hosted, distributed, multi-model LLM inference

Kubernetes controllers for fast model actuation using vLLM sleep/wake and launcher-based model swapping

Distributed inference configuration tools

Accelerate reproducible inference experiments for large language models with LLM-D! This lab automates the setup of a complete evaluation environment on OpenShift/OKD: GPU worker pools, core operators, observability, traffic control, and ready-to-run example workloads.

Demo - Distributed Inference on OpenShift (aka llm-d)

Experimental recovery coordination for expert-parallel inference groups in llm-d.

Prove which open model can replace your closed one — on your traffic, your hardware, your budget. Then move you there without a risky cutover.

一个 Kubernetes 原生的高性能分布式大语言模型推理框架

Replay and understand llm-d flow-control experiments across client, EPP, and vLLM layers.

Stack to serve LLM inference based on llm-d, Envoy AI gateway

RecursiveCharacterTextSplitter and context cache with llm-d

Self-hosted LLM inference platform on AWS EKS — vLLM, llm-d, Cosign, ArgoCD, Kyverno, Kagent+MCP

Heterogeneous AI resource integration for Kubernetes + llm-d, with Optimus Prime and FASOP as engine consolidation targets.

A light weight vLLM simulator, for mocking out replicas.

Does prefill/decode disaggregation pay for an MoE model? SLO-first goodput benchmark of Qwen3-30B-A3B on 4x H100 and 8x A100 with vLLM, llm-d and NVIDIA Dynamo, with per-request data.

Verifiable, tenant-isolated LLM inference controls for regulated AI infrastructure.

llm-d benchmark scripts and tooling

End-to-end Bielik model implementation on RHOAI with llm-d

Step-by-step llm-d deployment on GKE with GPU acceleration