nvidia-dynamo

A comprehensive toolkit for deploying production-ready Generative AI infrastructure on Amazon EKS. Includes pre-configured components for: 🚀 AI Gateway (LiteLLM) 🤖 LLM Serving (vLLM, SGLang, Ollama) 📊 Vector Databases, 🔍 Embedding Models (TEI) 📈 Observability (Langfuse, Phoenix) etc. Fast-track your GenAI deployment with Kubernetes

Topology- and workload-aware GPU scheduling with Ray and NVIDIA Dynamo for distributed LLM inference across heterogeneous GPU clusters, evaluated using normalized Job Completion Time.

Cross-node P/D disaggregated serving for Stevens Jarvis using Dynamo/vLLM/Slurm, with topology-aware CPU/NUMA placement and custom Triton kernels

The goal of the project is to benchmark and optimize BERT inference using different backends—PyTorch eager mode, TorchDynamo (Inductor backend), and NVIDIA Triton Inference Server. We use GLUE SST-2 samples for evaluation and compare performance through profiling, kernel timing, and latency analysis.

Serving LLMs on H100s with NVIDIA Dynamo, vLLM and Kubernetes: first-principles performance analysis in seven hands-on exercises

Production-grade Java 25 Virtual Thread inference gateway bridging NVIDIA Triton → Dynamo with Earliest Deadline First (EDF) priority queuing, adaptive batching, and async shadow validation.

Slurm (Slinky) and NVIDIA Dynamo + vLLM inference on one Karpenter GPU pool on Amazon EKS, entirely as code, created and destroyed from CI

Replay cache-event faults against real NVIDIA Dynamo, reduce failures, and share inspectable evidence.

Mini-Dynamo: disaggregated prefill/decode LLM inference platform with KV-cache-aware routing and observability (simulated, no GPU needed)

Does prefill/decode disaggregation pay for an MoE model? SLO-first goodput benchmark of Qwen3-30B-A3B on 4x H100 and 8x A100 with vLLM, llm-d and NVIDIA Dynamo, with per-request data.

Keeps a multi-GPU vLLM engine serving while its nodes change: a lost node's rank taken by a pre-loaded spare, every stream continuing

Rigorous A/B of NVIDIA Dynamo's KV-aware router vs round-robin, driven by the Mooncake trace and profiled with inferscope on top of AIPerf.

Benchmark: how many agentic sessions one 8x H100 node serves on Qwen3-235B-A22B FP8 with NVIDIA Dynamo, vLLM and TensorRT-LLM, and what moves the limit.

GPU-free lab for watching an NVIDIA Dynamo inference fleet recover from failure, autoscale, and absorb traffic spikes on Amazon EKS (Terraform + Helm + Chaos Mesh + k6).