[delamainer]

Summary

The OpenAI SDK transport (#1039) let the SDK build a ChatCompletionChunk pydantic model for every streamed chunk, and the LLM node then dumped each one back to a dict for the stream callbacks. Building and dumping cost about 25 µs per chunk, most of a streamed call's chunk handling: a py-spy profile of the runtime put 8–10 ms per streamed LLM and agent run on it.

Streams now ask the SDK for each chunk's JSON as it is, through its documented raw-response API (with_raw_response.create(...).parse(to=Stream[object])), and _chunk_dict fills in the dict to exactly what ChatCompletionChunk.model_dump() returned, in about 1 µs:

• every field of the model, with its default where the JSON lacks it, then the JSON's extra keys, in the same order;
• an integral number in a float field becomes a float, as the SDK converts it;
• a chunk with a field the SDK has to tell apart, such as moderation (one of several models), still goes through the SDK.

What the callbacks receive, and so what clients receive in stream events, does not change. Around it:

• The stream assembles the response as chunks arrive, from strings and copies, so a callback editing its chunk cannot change the response, as before. The node takes the response from the stream (_build_streamed_response now also gets the stream).
• Usage asked for only to price the call no longer gets written into the last chunk after it was streamed.
• Unstreamed calls are unchanged.

Stacked on #1039.

Measurements

Micro-benchmark: a streamed call of 300 chunks through the node with a mock transport, CPU per call: 14.3 ms before, 5.9 ms after.

End to end in the runtime, after dynamiq 0.70.0: the mono runtime benchmark (services/runtime/benchmark in dynamiq-ai/mono#1565) on the dev cluster. Two images of the runtime from mono main, one with #1039 merged up to dynamiq main (acabdcb) and one with this commit rebased on it, both with DYNAMIQ_OPENAI_NATIVE_SDK=true. One uvicorn worker with 1 CPU requested and 2 as the limit, each workload at two fixed rates of real runs (gpt-5.6-luna). Both images ran at the same time on two nodes of the same type, so they saw the same LLM latency, and swapped nodes for the second round.

CPU per run, the mean of the two rounds:

Workload	Rate (runs/s)	Before	After	Change
Agent, LLM only	4 / 7	54.7 / 54.8 ms	39.6 / 39.8 ms	−28% / −27%
One LLM call, streamed	4 / 7	74.5 / 71.5 ms	56.2 / 55.7 ms	−25% / −22%
Agent, LLM only, streamed	3 / 5	87.6 / 87.0 ms	70.3 / 70.4 ms	−20% / −19%
One LLM call (not streamed)	8 / 15	14.0 / 14.0 ms	14.0 / 14.1 ms	0% / +1%
Input to Output (no LLM)	16 / 28	7.2 / 7.1 ms	7.2 / 7.0 ms	0% / −1%

• A worker sustains about 1000 / (CPU ms per run) runs a second: about 18 streamed LLM calls a second instead of 14, and 25 agent runs instead of 18.
• The benchmark's agent has streaming enabled in its config, so it streams from its LLM on every run, streamed request or not, and saves 15–17 ms per run either way.
• The two workloads that do not stream from the SDK moved by 1% or less, so the comparison itself is clean. Latency did not change, and no run was lost.
• Before the rebase, against perf(utils): parse each JSONPath expression once #1049 at bca4347, the same comparison gave −24% / −27% for the agent, −24% / −23% for a streamed call and −19% for a streamed agent run.

Tests

tests/unit/nodes/llms/test_openai_native_sdk.py:

• _chunk_dict returns what ChatCompletionChunk.construct(**data).model_dump() does, compared as JSON so key order counts, for text, tool call, usage, logprob, extra-key, missing-field and moderation chunks;
• a streamed call never builds a ChatCompletionChunk;
• a callback editing its chunks does not change the response;
• usage asked for only to price the call stays out of the streamed chunks;
• usage is estimated when the stream reports none, now end to end.

The SDK tests pass with openai 2.24 (dynamiq's lock) and 2.43 (the runtime's), and the unit and integration suites pass.

[github-actions]

Coverage Report •

File	Stmts	Miss	Cover	Missing
dynamiq/nodes/llms
_openai_sdk.py	302	20	93%	269, 294, 298, 336, 359–360, 392, 398, 415, 486–487, 501–502, 546–550, 587–588
base.py	638	76	88%	178, 184, 443, 491–492, 538–544, 587, 676–677, 681, 849–858, 883, 960–961, 990–1000, 1006–1010, 1015–1021, 1032–1033, 1043, 1120, 1124, 1214–1215, 1281, 1348–1355, 1357, 1481, 1558, 1561, 1566–1567, 1594–1595
openai.py	107	1	99%	210
TOTAL	54733	10514	81%

Tests	Skipped	Failures	Errors	Time
8092	1 💤	0 ❌	0 🔥	3m 53s ⏱️