Epic: Slurm batch execution v1 #850

Priority Level

High

Task Summary

Track delivery of the first installable, batch-only Slurm integration for Data Designer.

The integration will ship as an optional data-designer-slurm package under the data_designer.slurm namespace. It will provide a strict configuration-to-result path that plans and submits Data Designer generation jobs, manages model services and client execution inside static Slurm allocations, and persists enough state for observation, retry, merge, and benchmark analysis.

This is a scoped delivery slice of #160. Interactive workflows and dynamic resource management remain future work.

Technical Details & Implementation Plan

Public outcome

• Install with pip install "data-designer[slurm]" without adding Slurm dependencies to a base-only installation.
• Expose an optional, lazily loaded data-designer slurm command group and equivalent Python service APIs.
• Provide profile init/validate, execute, status, cancel, retry, merge, benchmark run/analyze, and image add/ls/rm/info under that command group.
• Accept strict, versioned run, cluster-profile, serving, image, and benchmark inputs.
• Resolve authored input into an immutable execution plan before submission.
• Start model services and a separate zero-GPU Data Designer client within one static allocation per shard attempt.
• Persist run, shard, attempt, result, and output records so status, cancellation, retry, merge, and benchmark analysis work from a fresh process.
• Support deterministic local and fake-runtime tests before real Slurm validation.

Delivery stages

1. Define the public Data Designer boundary and establish optional packaging and CLI discovery.
2. Freeze only the schemas shared across packages, processes, or implementation lanes, integrate the plan and runtime-state boundaries, then add reusable fake infrastructure and golden fixtures.
3. Implement configuration and planning, serving, image management, Slurm control/runtime, the Data Designer client worker, persistent state, and benchmarks in parallel where their contracts allow it.
4. Validate local planning, image creation, one-node generation, multi-node serving, multiple serving images, sharding/retry/merge, and benchmarks.
5. Complete security, packaging, documentation, and release checks against the exact artifacts selected for release.

Initial foundation work

• Define the public Data Designer contract used by the optional Slurm package #851 Define the public Data Designer contract used by the optional Slurm package.
• Add the optional Slurm package and its install extra #852 Add the optional data-designer-slurm package and [slurm] extra.
• Add lazy discovery for optional CLI command groups #853 Add lazy discovery for optional CLI command groups.
• Define Slurm runtime and state record contracts #865 Define shared runtime and state record contracts.
• Define shared Slurm configuration and execution-plan contracts #873 Define shared authored configuration, plan, image, client-result, and benchmark records.
• Integrate Slurm plan and runtime-state contracts #880 Integrate Slurm plan and runtime-state contracts.
• Add deterministic fake infrastructure for Slurm integration tests #872 Add deterministic fake infrastructure for Slurm integration tests.

Later implementation lanes

• Public services, CLI commands, packaging, and documentation.
• Configuration resolution and deterministic plan compilation.
• Resolve deployment-owned vLLM serving plans for Slurm #866 Typed serving resolution with vLLM support.
• Build, inspect, and manage SQSH images for Slurm #867 Container image import, inspection, and registry operations for the Slurm container runtime.
• Implement Slurm submission and allocation runtime #868 Slurm submission and allocation runtime.
• Data Designer client environment and worker.
• Persist Slurm run state, retry shards, and collect winners #869 Persistent state, retry, shard winner publication, and merge.
• Benchmark expansion, execution, observation, and analysis.
• Harden the Slurm runtime and validate sealed release artifacts #870 Runtime security review and sealed-artifact acceptance.

Out of scope for v1

• Interactive sessions, notebooks, tunnels, or detached model-server leases.
• Dynamic allocation resizing, early GPU release, or telemetry-driven resource changes.
• Kubernetes, Ray, a generic scheduler interface, or third-party serving plugins.
• Object-store output semantics.
• Runtime source checkouts, editable installs, or mutable dependency resolution.

Epic quality gates

• A base-only installation does not install or expose the Slurm package.
• The optional package uses only public Data Designer APIs and does not import engine internals.
• CLI extension discovery reads package metadata without importing the optional package until its command is selected.
• Authored inputs and persisted records are strict, versioned, deterministic, and redact secret values.
• Cluster-specific accounts, partitions, paths, mounts, and hardware facts are supplied through configuration rather than source branches or hardcoded defaults.
• Plans, runtime resources, images, state transitions, and published outputs are validated with checksums and atomic publication where applicable.
• Failed or partial attempts cannot become successful shard outputs, and merge selects only validated shard winners.
• Built-wheel installation tests, local/fake integration tests, real Slurm acceptance, documentation checks, dependency review, and source/package scans pass before release.

Investigation / Context

Related roadmap issue: #160.

The repository already uses plan and task issues to track large implementation efforts through dependency-ordered child issues. This epic follows that pattern while keeping the public scope limited to the new Data Designer capability.

Agent Plan / Findings

Start with the three foundation issues for the public Data Designer contract, optional package, and lazy CLI extension. Add issue links to the foundation checklist after those issues are created.

Shared-contract, fake-infrastructure, and feature-lane issues use public, standalone context. The plan/runtime-state boundary is tracked in #880 because it gates multiple downstream lanes; later environment-acceptance milestones remain quality gates on this epic rather than separate implementation issues.

Dependencies

Related to #160. Foundation work can start immediately.