[delamainer]

Summary

#1039 serializes a streaming node once per stream_serialization_scope and hands each stream callback a copy of that dict on every event, so a callback editing it cannot change what the next one receives. The copy (_copy_serialized) walks the whole node serialization, prompt included, once per callback per chunk. A py-spy profile of the runtime put it at 16 ms of CPU per streamed agent run and 7 ms per streamed LLM run.

dynamiq's own stream callbacks only read a few keys of serialized: StreamingQueueCallbackHandler reads id, name, group, type and streaming.event, and AgentStreamingParserCallback reads group and id. A new decorator, reads_stream_serialization_only, marks such an on_node_execute_stream, and run_on_node_execute_stream gives a marked callback the scope's serialization itself instead of a copy.

• Unmarked callbacks, third-party ones included, still receive a copy per event, so perf(llms): cut per-token streaming overhead ~5x and add an opt-in OpenAI SDK transport #1039's guarantee holds for them.
• The mark is on the method, not the class: a subclass that overrides a marked on_node_execute_stream receives copies again unless it marks its override too.
• StreamingIteratorCallbackHandler and AsyncStreamingIteratorCallbackHandler inherit the queue handler's method and its mark.

Measurements

Micro-benchmark: run_on_node_execute_stream with a StreamingQueueCallbackHandler inside a scope, for an OpenAI node with a 1 KB system prompt, CPU per event: 15.2 µs before, 4.3 µs after.

End to end in the runtime, after dynamiq 0.70.0: the mono runtime benchmark (services/runtime/benchmark in dynamiq-ai/mono#1565) on the dev cluster. Two images of the runtime from mono main, one with #1039 merged up to dynamiq main (acabdcb) and one with this commit rebased on it, both on dynamiq's default LiteLLM transport, as production runs. One uvicorn worker with 1 CPU requested and 2 as the limit, each workload at two fixed rates of real runs (gpt-5.6-luna). Both images ran at the same time on two nodes of the same type, so they saw the same LLM latency, and swapped nodes for the second round.

CPU per run, the mean of the two rounds:

Workload	Rate (runs/s)	Before	After	Change
Agent, LLM only, streamed	1.5 / 2.5	204.3 / 210.1 ms	192.1 / 190.4 ms	−6% / −9%
One LLM call, streamed	2 / 3	118.7 / 116.9 ms	110.9 / 108.9 ms	−7%
Agent, LLM only	2 / 3	173.6 / 171.1 ms	166.5 / 169.5 ms	−4% / −1%
One LLM call	8 / 15	15.7 / 15.5 ms	15.6 / 15.4 ms	−1%
Input to Output (no LLM)	16 / 28	7.1 / 7.1 ms	7.2 / 7.1 ms	0%

• A streamed LLM call saves about 8 ms and a streamed agent run 12–20 ms, about what it saved on the SDK transport, but a smaller share of a LiteLLM run, which costs more per chunk.
• Input to Output does not stream and did not move, so the comparison itself is clean. Latency did not change, and no run was lost.

On the SDK transport, before the rebase: the same benchmark. Two images from the same runtime code, one with this branch's base (#1049 at bca4347) and one with this commit, both with DYNAMIQ_OPENAI_NATIVE_SDK=true. One uvicorn worker with 1 CPU requested and 2 as the limit, each workload at two fixed rates of real runs (gpt-5.6-luna). Both images ran at the same time on two nodes of the same type, so they saw the same LLM latency, and swapped nodes every round, for four rounds.

CPU per run, the mean of the four rounds:

Workload	Rate (runs/s)	Before	After	Change
Agent, LLM only, streamed	3 / 5	96.5 / 98.8 ms	81.3 / 82.9 ms	−16%
One LLM call, streamed	4 / 7	76.2 / 75.0 ms	66.8 / 67.2 ms	−12% / −10%
Agent, LLM only	4 / 7	62.2 / 64.2 ms	58.8 / 59.4 ms	−6% / −7%
One LLM call	8 / 15	15.2 / 16.2 ms	15.0 / 16.0 ms	−1%
Input to Output (no LLM)	16 / 28	7.3 / 7.5 ms	7.5 / 7.5 ms	+2% / 0%

• A streamed agent run saves about 16 ms of CPU and a streamed LLM call about 9 ms, close to what a py-spy profile under load had put on _copy_serialized (16 ms and 7 ms). A worker sustains about 1000 / (CPU ms per run) runs a second, so about 12 streamed agent runs a second instead of 10.
• The benchmark's agent has streaming enabled in its config, so it streams from its LLM even when the request is not streamed, and saves a few milliseconds there too. A single LLM call that does not stream is unchanged.
• Latency did not change. 14 of about 11,000 non-streamed LLM runs returned a 400, on both images in the same steps, while the model's latency was up.
• The Input to Output app does not stream and measures the comparison itself. One node ran it 10–18% cheaper in two of the four rounds, once for each image, which is why the comparison ran four rounds instead of two.

Tests

tests/unit/nodes/test_stream_serialization.py adds:

• a marked callback receives the scope's serialization itself, with no copy and one to_dict() for the scope;
• a marked callback does not see what unmarked callbacks before or after it edit;
• an unmarked override of a marked method receives a copy;
• dynamiq's four stream handlers are marked.

The existing tests of #1039's isolation still pass, as do the unit and integration suites.

[vitalii-dynamiq]

Reviewed b271ab3: 0 findings (0 critical, 0 high, 0 medium, 0 low).

---

Generated by Claude Code

[github-actions]

Coverage Report •

File	Stmts	Miss	Cover	Missing
dynamiq/callbacks
base.py	62	10	84%	105, 118, 127, 167, 176, 212, 221, 245–246, 248
streaming.py	819	55	93%	196, 209–210, 218–219, 235–236, 274–275, 283, 352–355, 371, 392–395, 446, 528, 531, 535, 577, 597, 600, 766, 775, 869, 1018–1021, 1132, 1134, 1160, 1302, 1315, 1343–1345, 1377, 1440, 1444, 1462, 1464–1472, 1477
dynamiq/nodes
node.py	1169	122	90%	176–177, 197–198, 501, 603, 620, 653, 690, 739–740, 742, 778, 780, 812, 814, 821–822, 905–906, 909–911, 918–920, 928–929, 939–944, 951–952, 1103, 1124–1125, 1140–1141, 1353–1359, 1363–1364, 1367, 1378, 1384, 1589, 1642–1643, 1708–1714, 1896–1903, 1905, 1924, 1936–1937, 2026–2027, 2046, 2080–2081, 2106–2107, 2129–2130, 2158–2159, 2179–2180, 2206–2207, 2232–2233, 2255–2256, 2276–2277, 2340, 2378, 2384, 2387, 2404–2406, 2479–2485, 2487, 2489–2490, 2501, 2521, 2556, 2604, 2610, 2661, 2684–2686
TOTAL	54744	10511	81%

Tests	Skipped	Failures	Errors	Time
8099	1 💤	0 ❌	0 🔥	5m 13s ⏱️