Repository navigation
feat: the tool-call record covers every tool call (follow-up to #880) - #909
Merged
Merged
Conversation
Follow-up to #880 (§7, open question 7): server rows as identity only, scope limited to runs with a resolved store, per-thread prune with a configurable retention, and pendingToolCallIds derived from the record. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…settle and prune Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…, settle and prune Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…t), settle and prune Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…rder Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…settle Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A gate that parks by throwing GraphInterrupt is issued but not settled; the resumed re-execution issues again (no-op on the key) and settles when the tool really returns or throws. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ERROR Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… requires settle and prune Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… carries no ids Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… its own Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…read, derive pendingToolCallIds from the record Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…never shorter than the TTL Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…his run's parks Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…note rolling upgrades Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…oin its sweep Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…st comment Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
blove
enabled auto-merge (squash)
October 3, 2026 14:18
blove
added a commit
that referenced
this pull request
Oct 3, 2026
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This was referenced Oct 3, 2026
blove
added a commit
that referenced
this pull request
Oct 3, 2026
…timodal PR 1) (#912) * docs(specs): AG-UI 1.0 sub-project 3 — multimodal input and content-part tool results Design for #886: skip-and-announce replaces the 422 refusal, all three part sources are carried, modality support derives from LangChain model profiles with a per-provider fallback, tool results carry parts, and the multimodal capability section is derived from the same function that enforces it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(plans): AG-UI multimodal PR 1 — runtime core Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(sdk): B4ContentPart — the AG-UI 1.0 content part shape for tools and middleware Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(ag-ui): pin B4ContentPart to the protocol's ContentPart Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(sdk): B4ContentPart guard rejects null metadata; shape pin checks key sets Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(langchain): resolveModalitySupport — what a model takes, from its LangChain profile Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(langchain): toLangChainContent — content parts to model blocks, with every drop recorded Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): modality support reads through RunnableBinding; file handles per part type Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): multimodal — file handles per part type, legacy image_url for ollama/mistral, profile through bindings Review of the implementation checked the installed converters: google-genai throws on fileId, openai's Chat Completions path maps it only for file blocks, and ollama/mistralai accept only the legacy image_url block. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): content parts honour per-type file handles and the ollama/mistral image_url shape Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(ag-ui): B4Message carries content parts instead of flattening them Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): pdf support per source; openai tool results carry no media on Chat Completions Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): pdf url gating, text-only tool results as strings, ollama url guard Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): multimodal — contentBlocks construction, pdf per source, openai tool-result override, tool text collapse Running real messages through the installed @langchain/openai converters showed: standard blocks translate only under contentBlocks:, a PDF URL vanishes on Chat Completions, and tool-message media never reaches the model there; @langchain/ollama throws on non-string tool content. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(cli): accept content parts on B4Message at the handler and after-middleware seams Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(ag-ui): tool results travel as content parts; dropped parts announced as CUSTOM Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(langchain): carry user content parts to the model; drop and announce what it cannot take Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): multimodal — the content_parts_dropped payload names provider and model Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(ag-ui): pin a dropped-parts CUSTOM queued behind a held orchestration call Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(cli): stop refusing media content parts at the AG-UI envelope stage Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(langchain): tools return content parts; media reaches the model when its profile allows Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(cli): bound Agent Protocol run bodies at the AG-UI ceiling Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(testing): fixtures and matchers read the text of multimodal message content Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): log dropped parts on the non-streaming path; malformed parts are dropped, not stringified Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(testing): contentText sits above the doc it displaced; raw-shape matcher test Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): multimodal — note the fixture first-vs-last user message limitation Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(cli): log every dropped content part once per run; handler tests for parts Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(cli): 413 on every bounded Agent Protocol route, header and streamed Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(ag-ui): kept parts are read out of a Command-wrapped tool result Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): messages with parts serialize with content; tool text keeps its place Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): multimodal — messages carry blocks under content with output_version v1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): multimodal — announce channels per path, subagent drops, assistant-id pointer Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: multimodal content is carried and announced; tools may return content parts Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(cli): the dropped-parts warning links the route's assistant id; subagent drops are logged too Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): the dropped-parts warning names the route without promising a capability section Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(ag-ui): multimodal section stands alone; precise drop reasons and limits Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after rebase Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): multimodal — a tool-result drop's CUSTOM precedes its TOOL_CALL_RESULT Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain,cli): media-aware summarization counts; drop warnings on every path - Summarization renders media blocks as [image]/[audio]/[video]/[file] placeholders (shared message-text helper) and counts each as its placeholder plus MEDIA_TOKEN_ESTIMATE, so base64 is never tokenized or sent to the summarizer. - executeAgentTurn logs subagent.content_parts_dropped too, through the shared pickDroppedPartsReport picker the CLI now also uses. - A client tool answer carrying media warns that only its text is stored. - The dropped-parts warning names the tool result; thread tails render media as placeholders instead of base64. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: changeset and README name every widened contract - The changeset names MiddlewareAfterMessage.content (@b4run/sdk) among the widened types, and that chain/graph/workflow routes now receive messages[].content as a part array whenever the client sent parts. - The ag-ui README lists @b4run/sdk beside @ag-ui/core as a dependency. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): only media blocks count as media in summarization; silent on a malformed drop report - Only image/audio/video/file/document and legacy image_url blocks render as placeholders and add MEDIA_TOKEN_ESTIMATE (image_url as [image]); tool_use, thinking, reasoning and other blocks keep their JSON text. - The CLI's dropped-parts warning is skipped when the chunk carries no parts list, matching executeAgentTurn. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci: build @b4run/ag-ui with its workspace deps; seal the v0.13 package order @b4run/ag-ui now depends on @b4run/sdk, which changes two things CI pins: - The two jobs that built it alone (`pnpm --filter @b4run/ag-ui build`) now build its dependencies too (`--filter @b4run/ag-ui...`); both audited workflow fixtures and the contract test's inline descriptor are transcribed. - The live fixed-group topology orders @b4run/ag-ui after @b4run/sdk. The order v0.13.0/v0.13.1 were published in becomes HISTORICAL_B4_PRE_AGUI_SDK_RELEASE_PACKAGE_ORDER (manifest admission and npm-evidence admission keep accepting it) and CANONICAL_RELEASE_PACKAGE_ORDER follows the live topology. manifest.mjs and npm-evidence.mjs are repinned and the pin-file digest snapshot updated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging main Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(plans): AG-UI multimodal PR 2 — capability section and client-tool parts Spec §4.3 amended: the capability document reads the model's profile off its class (readModelProfile) rather than constructing it, since provider constructors throw without a credential; §3.2: the stores keep a part list in the existing text column as a self-describing envelope. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging main (#909) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging main (#915, #916) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging main (#914) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging main (#918) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci: retrigger validation for #912 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
blove
added a commit
that referenced
this pull request
Oct 3, 2026
…t parts (multimodal PR 2) (#917) * docs(specs): AG-UI 1.0 sub-project 3 — multimodal input and content-part tool results Design for #886: skip-and-announce replaces the 422 refusal, all three part sources are carried, modality support derives from LangChain model profiles with a per-provider fallback, tool results carry parts, and the multimodal capability section is derived from the same function that enforces it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(plans): AG-UI multimodal PR 1 — runtime core Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(sdk): B4ContentPart — the AG-UI 1.0 content part shape for tools and middleware Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(ag-ui): pin B4ContentPart to the protocol's ContentPart Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(sdk): B4ContentPart guard rejects null metadata; shape pin checks key sets Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(langchain): resolveModalitySupport — what a model takes, from its LangChain profile Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(langchain): toLangChainContent — content parts to model blocks, with every drop recorded Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): modality support reads through RunnableBinding; file handles per part type Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): multimodal — file handles per part type, legacy image_url for ollama/mistral, profile through bindings Review of the implementation checked the installed converters: google-genai throws on fileId, openai's Chat Completions path maps it only for file blocks, and ollama/mistralai accept only the legacy image_url block. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): content parts honour per-type file handles and the ollama/mistral image_url shape Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(ag-ui): B4Message carries content parts instead of flattening them Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): pdf support per source; openai tool results carry no media on Chat Completions Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): pdf url gating, text-only tool results as strings, ollama url guard Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): multimodal — contentBlocks construction, pdf per source, openai tool-result override, tool text collapse Running real messages through the installed @langchain/openai converters showed: standard blocks translate only under contentBlocks:, a PDF URL vanishes on Chat Completions, and tool-message media never reaches the model there; @langchain/ollama throws on non-string tool content. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(cli): accept content parts on B4Message at the handler and after-middleware seams Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(ag-ui): tool results travel as content parts; dropped parts announced as CUSTOM Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(langchain): carry user content parts to the model; drop and announce what it cannot take Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): multimodal — the content_parts_dropped payload names provider and model Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(ag-ui): pin a dropped-parts CUSTOM queued behind a held orchestration call Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(cli): stop refusing media content parts at the AG-UI envelope stage Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(langchain): tools return content parts; media reaches the model when its profile allows Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(cli): bound Agent Protocol run bodies at the AG-UI ceiling Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(testing): fixtures and matchers read the text of multimodal message content Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): log dropped parts on the non-streaming path; malformed parts are dropped, not stringified Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(testing): contentText sits above the doc it displaced; raw-shape matcher test Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): multimodal — note the fixture first-vs-last user message limitation Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(cli): log every dropped content part once per run; handler tests for parts Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(cli): 413 on every bounded Agent Protocol route, header and streamed Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(ag-ui): kept parts are read out of a Command-wrapped tool result Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): messages with parts serialize with content; tool text keeps its place Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): multimodal — messages carry blocks under content with output_version v1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): multimodal — announce channels per path, subagent drops, assistant-id pointer Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: multimodal content is carried and announced; tools may return content parts Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(cli): the dropped-parts warning links the route's assistant id; subagent drops are logged too Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): the dropped-parts warning names the route without promising a capability section Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(ag-ui): multimodal section stands alone; precise drop reasons and limits Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after rebase Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): multimodal — a tool-result drop's CUSTOM precedes its TOOL_CALL_RESULT Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain,cli): media-aware summarization counts; drop warnings on every path - Summarization renders media blocks as [image]/[audio]/[video]/[file] placeholders (shared message-text helper) and counts each as its placeholder plus MEDIA_TOKEN_ESTIMATE, so base64 is never tokenized or sent to the summarizer. - executeAgentTurn logs subagent.content_parts_dropped too, through the shared pickDroppedPartsReport picker the CLI now also uses. - A client tool answer carrying media warns that only its text is stored. - The dropped-parts warning names the tool result; thread tails render media as placeholders instead of base64. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: changeset and README name every widened contract - The changeset names MiddlewareAfterMessage.content (@b4run/sdk) among the widened types, and that chain/graph/workflow routes now receive messages[].content as a part array whenever the client sent parts. - The ag-ui README lists @b4run/sdk beside @ag-ui/core as a dependency. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): only media blocks count as media in summarization; silent on a malformed drop report - Only image/audio/video/file/document and legacy image_url blocks render as placeholders and add MEDIA_TOKEN_ESTIMATE (image_url as [image]); tool_use, thinking, reasoning and other blocks keep their JSON text. - The CLI's dropped-parts warning is skipped when the chunk carries no parts list, matching executeAgentTurn. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci: build @b4run/ag-ui with its workspace deps; seal the v0.13 package order @b4run/ag-ui now depends on @b4run/sdk, which changes two things CI pins: - The two jobs that built it alone (`pnpm --filter @b4run/ag-ui build`) now build its dependencies too (`--filter @b4run/ag-ui...`); both audited workflow fixtures and the contract test's inline descriptor are transcribed. - The live fixed-group topology orders @b4run/ag-ui after @b4run/sdk. The order v0.13.0/v0.13.1 were published in becomes HISTORICAL_B4_PRE_AGUI_SDK_RELEASE_PACKAGE_ORDER (manifest admission and npm-evidence admission keep accepting it) and CANONICAL_RELEASE_PACKAGE_ORDER follows the live topology. manifest.mjs and npm-evidence.mjs are repinned and the pin-file digest snapshot updated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging main Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(plans): AG-UI multimodal PR 2 — capability section and client-tool parts Spec §4.3 amended: the capability document reads the model's profile off its class (readModelProfile) rather than constructing it, since provider constructors throw without a credential; §3.2: the stores keep a part list in the existing text column as a self-describing envelope. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging main (#909) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(sdk): client tool results carry content parts; a text-column codec for stores Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(sdk): an empty client tool part list is empty text; decode gates on any JSON object Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(langchain): readModelProfile — a model's LangChain profile without constructing it Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): the dropped-parts warning points at the capability document again Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(storage): client tool results round-trip content parts through SQLite and Postgres Both stores widen their mirrored ClientToolCallRecord.result / answer.result to B4MessageContent and keep a part list in the existing TEXT column via the SDK's encodeClientToolResult/decodeClientToolResult envelope (spec §3.2): no schema migration, text rows decode as text. Both packages take a direct @b4run/sdk dependency for the shared codec (each already had it transitively). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(cli): checkRouteModalitySupport preflight What a route's root model takes as content parts, resolved like the reasoning/response-format preflights and read off the model's profile without constructing it. ok: false for a non-agent route, a raw runnable, an unresolvable provider, or a missing provider package. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * style(postgres-storage): JSDoc continuation lines Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(cli): advertise multimodal.input from the route model's profile GET /agui/:routeId gains a multimodal section from checkRouteModalitySupport, the same profile judgment toLangChainContent applies at run time. file and output are false; the section is omitted for a raw runnable, a non-agent route, a missing provider package, or a boot that cannot load route modules. A derivation test checks the advertised input against what the converter keeps. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): multimodal — capability flags describe the inline source; a broken provider install omits the section Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(cli): client tool results with content parts are stored, screened on their text, and replayed under the tool-result rules Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(cli): a broken provider install omits multimodal instead of failing the document checkRouteModalitySupport turns any readModelProfile throw (a non-missing import error, a profile getter throw) into ok: false, as checkRouteReasoningSupport does for provider/config errors, so GET /agui/:routeId omits the multimodal section and keeps the rest rather than answering 500 or collapsing to unknown. An optional importer seam (threaded from the capabilities handler) lets tests inject the broken install. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(cli): abandon warning fires once per closed call; 413 names what the cap measures Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: the multimodal capability section; client tool results with parts Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: rollback note for part-list results; URL examples are examples Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(plans): AG-UI multimodal PR 3 — research example Spec §7 amended: media is drawn in the app's Transcript beside renderToolCall (CopilotKit's renderer flattens content), the gate is useCapabilities() from the runtime proxy, hydration maps LangChain v1 blocks back to parts, renderChart's image is UI-only on gpt-5-mini. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging main (#915, #916) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging PR 1's branch Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging main (#914) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging PR 1's branch (#914) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging main (#918) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging PR 1's branch (#918) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging main (#912 squash, #739, #920) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
blove
added a commit
that referenced
this pull request
Oct 3, 2026
…te takes one, renderChart (multimodal PR 3) (#919) * docs(specs): AG-UI 1.0 sub-project 3 — multimodal input and content-part tool results Design for #886: skip-and-announce replaces the 422 refusal, all three part sources are carried, modality support derives from LangChain model profiles with a per-provider fallback, tool results carry parts, and the multimodal capability section is derived from the same function that enforces it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(plans): AG-UI multimodal PR 1 — runtime core Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(sdk): B4ContentPart — the AG-UI 1.0 content part shape for tools and middleware Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(ag-ui): pin B4ContentPart to the protocol's ContentPart Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(sdk): B4ContentPart guard rejects null metadata; shape pin checks key sets Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(langchain): resolveModalitySupport — what a model takes, from its LangChain profile Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(langchain): toLangChainContent — content parts to model blocks, with every drop recorded Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): modality support reads through RunnableBinding; file handles per part type Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): multimodal — file handles per part type, legacy image_url for ollama/mistral, profile through bindings Review of the implementation checked the installed converters: google-genai throws on fileId, openai's Chat Completions path maps it only for file blocks, and ollama/mistralai accept only the legacy image_url block. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): content parts honour per-type file handles and the ollama/mistral image_url shape Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(ag-ui): B4Message carries content parts instead of flattening them Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): pdf support per source; openai tool results carry no media on Chat Completions Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): pdf url gating, text-only tool results as strings, ollama url guard Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): multimodal — contentBlocks construction, pdf per source, openai tool-result override, tool text collapse Running real messages through the installed @langchain/openai converters showed: standard blocks translate only under contentBlocks:, a PDF URL vanishes on Chat Completions, and tool-message media never reaches the model there; @langchain/ollama throws on non-string tool content. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(cli): accept content parts on B4Message at the handler and after-middleware seams Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(ag-ui): tool results travel as content parts; dropped parts announced as CUSTOM Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(langchain): carry user content parts to the model; drop and announce what it cannot take Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): multimodal — the content_parts_dropped payload names provider and model Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(ag-ui): pin a dropped-parts CUSTOM queued behind a held orchestration call Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(cli): stop refusing media content parts at the AG-UI envelope stage Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(langchain): tools return content parts; media reaches the model when its profile allows Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(cli): bound Agent Protocol run bodies at the AG-UI ceiling Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(testing): fixtures and matchers read the text of multimodal message content Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): log dropped parts on the non-streaming path; malformed parts are dropped, not stringified Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(testing): contentText sits above the doc it displaced; raw-shape matcher test Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): multimodal — note the fixture first-vs-last user message limitation Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(cli): log every dropped content part once per run; handler tests for parts Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(cli): 413 on every bounded Agent Protocol route, header and streamed Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(ag-ui): kept parts are read out of a Command-wrapped tool result Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): messages with parts serialize with content; tool text keeps its place Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): multimodal — messages carry blocks under content with output_version v1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): multimodal — announce channels per path, subagent drops, assistant-id pointer Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: multimodal content is carried and announced; tools may return content parts Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(cli): the dropped-parts warning links the route's assistant id; subagent drops are logged too Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): the dropped-parts warning names the route without promising a capability section Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(ag-ui): multimodal section stands alone; precise drop reasons and limits Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after rebase Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): multimodal — a tool-result drop's CUSTOM precedes its TOOL_CALL_RESULT Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain,cli): media-aware summarization counts; drop warnings on every path - Summarization renders media blocks as [image]/[audio]/[video]/[file] placeholders (shared message-text helper) and counts each as its placeholder plus MEDIA_TOKEN_ESTIMATE, so base64 is never tokenized or sent to the summarizer. - executeAgentTurn logs subagent.content_parts_dropped too, through the shared pickDroppedPartsReport picker the CLI now also uses. - A client tool answer carrying media warns that only its text is stored. - The dropped-parts warning names the tool result; thread tails render media as placeholders instead of base64. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: changeset and README name every widened contract - The changeset names MiddlewareAfterMessage.content (@b4run/sdk) among the widened types, and that chain/graph/workflow routes now receive messages[].content as a part array whenever the client sent parts. - The ag-ui README lists @b4run/sdk beside @ag-ui/core as a dependency. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): only media blocks count as media in summarization; silent on a malformed drop report - Only image/audio/video/file/document and legacy image_url blocks render as placeholders and add MEDIA_TOKEN_ESTIMATE (image_url as [image]); tool_use, thinking, reasoning and other blocks keep their JSON text. - The CLI's dropped-parts warning is skipped when the chunk carries no parts list, matching executeAgentTurn. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci: build @b4run/ag-ui with its workspace deps; seal the v0.13 package order @b4run/ag-ui now depends on @b4run/sdk, which changes two things CI pins: - The two jobs that built it alone (`pnpm --filter @b4run/ag-ui build`) now build its dependencies too (`--filter @b4run/ag-ui...`); both audited workflow fixtures and the contract test's inline descriptor are transcribed. - The live fixed-group topology orders @b4run/ag-ui after @b4run/sdk. The order v0.13.0/v0.13.1 were published in becomes HISTORICAL_B4_PRE_AGUI_SDK_RELEASE_PACKAGE_ORDER (manifest admission and npm-evidence admission keep accepting it) and CANONICAL_RELEASE_PACKAGE_ORDER follows the live topology. manifest.mjs and npm-evidence.mjs are repinned and the pin-file digest snapshot updated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging main Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(plans): AG-UI multimodal PR 2 — capability section and client-tool parts Spec §4.3 amended: the capability document reads the model's profile off its class (readModelProfile) rather than constructing it, since provider constructors throw without a credential; §3.2: the stores keep a part list in the existing text column as a self-describing envelope. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging main (#909) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(sdk): client tool results carry content parts; a text-column codec for stores Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(sdk): an empty client tool part list is empty text; decode gates on any JSON object Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(langchain): readModelProfile — a model's LangChain profile without constructing it Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(langchain): the dropped-parts warning points at the capability document again Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(storage): client tool results round-trip content parts through SQLite and Postgres Both stores widen their mirrored ClientToolCallRecord.result / answer.result to B4MessageContent and keep a part list in the existing TEXT column via the SDK's encodeClientToolResult/decodeClientToolResult envelope (spec §3.2): no schema migration, text rows decode as text. Both packages take a direct @b4run/sdk dependency for the shared codec (each already had it transitively). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(cli): checkRouteModalitySupport preflight What a route's root model takes as content parts, resolved like the reasoning/response-format preflights and read off the model's profile without constructing it. ok: false for a non-agent route, a raw runnable, an unresolvable provider, or a missing provider package. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * style(postgres-storage): JSDoc continuation lines Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(cli): advertise multimodal.input from the route model's profile GET /agui/:routeId gains a multimodal section from checkRouteModalitySupport, the same profile judgment toLangChainContent applies at run time. file and output are false; the section is omitted for a raw runnable, a non-agent route, a missing provider package, or a boot that cannot load route modules. A derivation test checks the advertised input against what the converter keeps. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): multimodal — capability flags describe the inline source; a broken provider install omits the section Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(cli): client tool results with content parts are stored, screened on their text, and replayed under the tool-result rules Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(cli): a broken provider install omits multimodal instead of failing the document checkRouteModalitySupport turns any readModelProfile throw (a non-missing import error, a profile getter throw) into ok: false, as checkRouteReasoningSupport does for provider/config errors, so GET /agui/:routeId omits the multimodal section and keeps the rest rather than answering 500 or collapsing to unknown. An optional importer seam (threaded from the capabilities handler) lets tests inject the broken install. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(cli): abandon warning fires once per closed call; 413 names what the cap measures Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: the multimodal capability section; client tool results with parts Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: rollback note for part-list results; URL examples are examples Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(plans): AG-UI multimodal PR 3 — research example Spec §7 amended: media is drawn in the app's Transcript beside renderToolCall (CopilotKit's renderer flattens content), the gate is useCapabilities() from the runtime proxy, hydration maps LangChain v1 blocks back to parts, renderChart's image is UI-only on gpt-5-mini. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging main (#915, #916) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging PR 1's branch Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(testing): harness run input accepts content parts Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(research): renderChart returns an inline SVG as a content part A horizontal bar chart tool for the research example and its devkit template: returns a one-line text summary plus an image/svg+xml data part. On gpt-5-mini the image is UI-only (the run emits b4.content_parts_dropped). The coordinator's system prompt gains a renderChart step, and the template's tracked typegen declarations are regenerated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(research): the agent can call renderChart A scripted turn calls renderChart and asserts the split: the model-facing tool result is the text summary only (gpt-5-mini takes no image in a tool message), while the ToolMessage keeps both parts under additional_kwargs.b4_content_parts for the UI. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(research-web): transcript carries content parts and drop notices Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(research): renderChart rejects non-finite values and caps label length NaN or Infinity previously drew width="NaN"; they now throw. The drawn title and labels are cut to 40 characters so long labels no longer overflow the label column (the text summary keeps the full strings). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(research-web): render media parts and dropped-part notices Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(research-web): attach an image when the route takes one; show what the model did not see Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(research-web): media parts allow-list MIME types; composer blocks send while reading Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(research): attach an image, media in the transcript, renderChart Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(research-web): extensionless parts import (next build); image types the model accepts; attachment size cap Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging main (#914) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging PR 1's branch (#914) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging PR 2's branch Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging main (#918) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging PR 1's branch (#918) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging PR 2's branch (#918) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging main (#912 squash, #739, #920) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging PR 2's branch (main through #920) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(web): regenerate SEO lastmod after merging main (#917 squash) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #880 (client-provided AG-UI tools): the retained tool-call record now covers every model tool call on an AG-UI run where the store is resolved, not only client-provided ones. Spec:
docs/superpowers/specs/2026-10-02-tool-call-record-generalization-design.md; plan:docs/superpowers/plans/2026-10-02-tool-call-record-generalization.md.kind: "server": thread, route, run, tool name, issued and settled times; no result text) by the LangChain tool converter around the tool body:issuebefore it runs,settleinfinally. A permission park (GraphInterrupt) does not settle. The client stub stays the only writer of client rows, recognised by a newclientTool: truemarker onB4ToolDefinition. The parenttaskcall is not recorded.server.agui.clientTools,server.agui.clientToolStoreset, or the default.b4/client-tool-calls.sqlitestill present from an earlier opt-in. Apps with no store and non-AG-UI transports are unchanged.role: "tool"message is consumed only when it names an open client row this server issued on this thread. Messages naming a server row, a closed row, or nothing are dropped as history, as before. The message filters are not relaxed.pendingToolCallIdsfrom the record. Read from the store whenRUN_FINISHEDis built, scoped to this run's parks (live), the snapshot's parks (partial), or none (no-op path). The stream normalizer's park callback and the in-memory set are gone.@b4run/ag-ui'spendingToolCallIdsoption may return a Promise; a rejection ends the run asRUN_ERROR.prune({ before }),server.agui.clientToolRetentionMs, hourly sweep on turn settle,b4 client-tools prune, TTL floor on the cutoff). This PR adopts that design unchanged and only extends its predicate: a server row is swept oncesettledAtis behind the window; open rows of either kind are never deleted. The per-thread prune andtoolCallRetentionMsfrom the original spec were dropped at merge (spec §4 note).ClientToolCallRecordgainskindandsettledAt;ClientToolCallStoregainssettle(plus feat: prune settled client tool call records #898'sprune);ClientToolRecordergainsissueandsettle. Memory, SQLite and Postgres stores implement them; SQLite and Postgres append migration 2 (kind text NOT NULL DEFAULT 'client' CHECK (...),settled_at text); existing rows backfill asclient. An operator store missingsettle/prunefails the boot naming them.Behavior changes and risks
issuefails that server tool call, on every AG-UI route. With the store unavailable, server tool calls on AG-UI runs fail until it is back. Previously only client-tool runs depended on the store.kindfilter and reads new server rows as open client rows (it may void them). Nothing becomes answerable; finish the rollout before mixing traffic.ALTER.Test plan
pnpm lint && pnpm build && pnpm typecheck && pnpm testgreen — 9621 passed, 287 skipped, 0 failed.B4_TEST_PGSTORAGE=1), including the v1→v2 backfill throughrunMigrations; the gatedpostgres-storage-dockerlane covers it in CI.node scripts/check-docs.mjs,node scripts/check-changesets.mjs,pnpm --dir apps/web seo:lastmod:checkpass.pendingToolCallIdswere caught in review and fixed with regression tests. The merge resolution was reviewed separately.🤖 Generated with Claude Code