Repository navigation
Description
Checked other resources
- This is a bug, not a usage question. For questions, please use the LangChain Forum (https://forum.langchain.com/).
- I added a clear and detailed title that summarizes the issue.
- I read what a minimal reproducible example is (https://stackoverflow.com/help/minimal-reproducible-example).
- I included a self-contained, minimal example that demonstrates the issue INCLUDING all the relevant imports. The code run AS IS to reproduce the issue.
Example Code
# Minimal Example
# Start a streaming LangGraph run
for chunk in graph.stream(input, config, stream_mode="values"):
process_streamed_value(chunk)
if should_cancel():
cancel_run()
# After cancel, send new message and sync UI:
# => streamed data between last persisted checkpoint and cancel is lost
# Full minimal code can be provided if desired (let me know)Error Message and Stack Trace (if applicable)
Description
Issue Summary
(I am using langgraph platform/ langgraph api)
When canceling a LangGraph run during streaming, the most recent "in-progress" streamed state is not persisted to the backend checkpoint database. As a result, when a new message is sent (triggering the front end to sync with the backend), any streamed data—in some cases, even several seconds or more of streamed output—is lost. The backend reflects only the last completed+persisted checkpoint, not the full state the user saw up to the moment of cancellation.
Steps to Reproduce
- Start a LangGraph run with streaming output (e.g., a chat session, a long node, or token-by-token streaming).
- Observe the front end as streaming updates/tokens arrive and are rendered.
- Cancel/abort the run before it naturally completes or reaches a checkpoint.
- Send a new message or re-sync the UI, causing the front-end to fetch authoritative state from the backend.
- Notice that all the streamed data since the last checkpoint is missing.
Expected Behavior
- On cancellation, the in-progress streamed state (what the user saw up to the abort/cancel) should be persisted to the backend checkpoint database, at least as a "partial"/"aborted" state.
- Resuming the conversation (or rendering the thread after a cancel) should match what the user last saw, not rollback to the last completed checkpoint.
Actual Behavior
- The backend does not persist the streamed/in-memory state at the moment of cancellation.
- The front-end loses all non-checkpointed streamed data on sync, since it relies on the backend as the source of truth after a cancel.
- Practically, this means that users can see output for several seconds (or longer), but after canceling and sending a new message, that output disappears.
Why This Matters
- Users see and may even start reading output as it’s streamed, but lose it entirely if they cancel—even if a majority of the output was streamed.
- Creates an inconsistent, surprising user experience for chat applications or any app with incremental streaming.
- Makes it difficult to build features that require exact “what you saw is what is saved” continuity.
Technical Insight
- Currently, checkpointing only happens at full run/task completion, not on cancel/abort/stream end events.
- There is no mechanism for a "flush/persist to checkpoint right now" on cancel/disconnect.
- Cancelling a run always discards the difference between the last checkpointed state and what was streamed to the client before cancellation.
- This is especially problematic for apps that adopt optimistic/real-time UI updates based on the stream.
Feature Proposal
- Add an option (or enable by default) to persist the latest/active state as a special checkpoint when a run is canceled or the streaming connection is closed. This could be a "partial," "aborted," or "interrupted" checkpoint.
- Optionally, provide a callback or hook for custom flush/final persistence on abort.
Additional context
If you need more information or a deeper technical dive/code pointers, I’m happy to collaborate or prototype a solution!
System Info
OS: macOS 14.5 (ARM64, Apple M1)
LangGraph version: 0.3.31
Runtime: FastAPI, Python 3.11
Deployment: both dev and cloud, similar results
Activity
sethi-ishmeet commentedon Aug 14, 2025
I am running into same issue and was able to pinpoint what was the error , at least in our case. For some reason, the next message sent my user created a new branch. To get the messages, frontend SDK starts a BFS search from the history and looks for a state with no parent i.e.
state.parent_checkpoint == null.However, because it becomes a branch off, the parent checkpoint is not null and it leads to not parsing any messages. This code is written in
stream.jsin methodgetBranchSequence.I confirmed this behaviour by modifying code for a static checkpoint and then it worked.
Ibrahim (@ibbybuilds) - Can you confirm if your frontend is getting the state history by checking the network tab? That's how I reached here.
Someone from Langgraph, can you probably confirm if this is a server issue or client SDK issue? I believe the branching should never have happened at first place. And even if it did, why is the parent checkpoint not null for the first message?
Here is the link to run in Langsmith:
https://smith.langchain.com/public/78163885-e653-4f9f-84c3-e05f4c65ab57/r
FYI, I can see the same issue in Studio. This thread does not render anything in Langgraph Studio.
ibbybuilds commentedon Aug 15, 2025
Ishmeet Singh Sethi (@sethi-ishmeet) I think this is mainly a backend issue. When a run is canceled, the current in-progress streamed state (tokens/messages that the user has already seen in the UI) is never written to the checkpoint database. When the next message is sent, the backend starts from the last saved checkpoint — which could be a human message or an intermediate AI message — instead of the full state the user saw before cancel.
The result is that the backend “history” is missing streamed-but-not-saved output. The frontend may optimistically display that output in real time, but it’s not persisted. On the next message, the UI syncs with backend history and effectively “rolls back” to the last checkpoint, losing several seconds or more of visible output.
Flow of the issue:
Workaround I used on the frontend:
[missing_ai_1, missing_ai_2, new_user_message].of-clausulo commentedon Oct 27, 2025
Ibrahim (@ibbybuilds) Will your workaround work on page reload? I don't think so... Harrison Chase (@hwchase17) Can you please review this? Brace Sproul (@bracesproul) Nuno Campos (@nfcampos)
ibbybuilds commentedon Oct 27, 2025
@ofried-acutis yes it won't work on page reload but you can use local storage to work around that or any prefered persistant storage.
of-clausulo commentedon Oct 27, 2025
Ibrahim (@ibbybuilds) Ok gotcha, thanks for the quick reply. This would mean basically persisting to storage every token streamed as soon as it's streamed... ok thanks!
gautamvarmadatla commentedon Feb 22, 2026
Hi all, I ran into this recently as well.
TLDR is that
Pregel.astream()persists state usingloop.after_tick()and intermediate checkpointeraput_writes()calls scheduled viaAsyncBackgroundExecutor, which are normally drained on a clean exit. When a run is cancelled,AsyncPregelLoop.__aexit__createsexit_task = asyncio.create_task(self.stack.__aexit__(...))and awaits it. If that await raisesCancelledError, it re-raises and only attachesexit_taskto the exception args. Sinceexit_taskis anasyncio.Task, it can continue running after the caller is cancelled, but if the caller does not await it and the event loop/process shuts down before it finishes draining background persistence tasks, pendingaput/aput_writesmay not flush.One possible fix is to shield and await the exit cleanup for a small bounded time before re-raising
CancelledError, so already-enqueued writes are more likely to commit. But I’m not sure if we want this as default behaviour (with a small timeout) or opt-in using a flag. I have some changes made locally and can draft a PR once confirmed.junan-trustarc commentedon Mar 20, 2026
any updates on this????
gautamvarmadatla commentedon Mar 20, 2026
I don’t think so, I haven’t seen much public activity from the maintainers in the past couple of months.
christian-bromann commentedon Apr 15, 2026
GAUTAM V DATLA (@gautamvarmadatla) we are actively looking into this as we are rethinking some of our streaming primitives. Stay tuned.
of-clausulo commentedon Apr 16, 2026
just don't use langchain / langgraph. Models are good enough now to just write all the code with no external dependencies (like langgraph)
fwanggg commentedon Apr 28, 2026
damn langgraph has so many bugs..
in anthropic this is even worse experience it will cause corrupted thread, tool_use_id wont match tool_result
norika1207-lab commentedon Jun 2, 2026
For agent reliability, cancellation should probably be modeled as a state transition with a bounded persistence contract, not only as an asyncio control-flow event.
The key operational question is: after cancellation, which writes are guaranteed durable and which writes were abandoned?
A useful event shape could be:
That would make the failure mode visible to operators and to any recovery UI. Without that distinction, a cancelled streamed run can look recoverable while its latest state never actually reached the checkpointer.
27 remaining items
test(langgraph): assert resume re-runs only the cancelled task
aayushbaluni commentedon Aug 18, 2026
Thanks — and your invariant is a better statement of the bug than mine was. "State survived ≠ termination evidence survived" is exactly it: the durable values were never wrong, what disappeared was the occurrence-level fact that
p2ran at all.I ran your formulation as an executable check against both trees, clause by clause, using precisely the negative control you described.
Unmodified
main(644815f):With the fix branch:
Worth noting what the failure isolates to: exactly one clause, in exactly one durability mode. The other three clauses hold on
maineven in the broken case —p1's work stays committed,p1is not mislabelled, and resume still re-runs onlyp2. So the defect is precisely the loss of termination evidence and nothing else, which is a good sign the fix is not papering over a wider correctness problem.The regression test on the branch already asserts three of your four clauses (
p2error present,p1error absent,p1's value committed) parametrized over all three modes. The one it does not assert is "resume re-runs only p2". That is worth adding on your framing — it is the clause that connects the durable marker to the behaviour a user actually cares about (not paying twice for work that already completed), and it is the one most likely to regress silently if someone later changes how pending writes are replayed. Added in b336f72 on the branch: the test now tracks which nodes execute, clears that record after the cancel, resumes, and asserts only the cancelled node re-ran. Still mutation-checked, and the suite is 313 passed vs 310 on main.Agreed on the two-boundary split, and I'd put it the same way:
The useful property of that split is that the platform boundary becomes testable only after core can prove the first half. Before this fix, an Agent Server integration test asserting "interrupted is not published before the cancellation is durable" could not have distinguished a Platform ordering bug from core never persisting the fact at all — under
durability="exit"there was nothing durable to wait for.Status: the PR (#8634) was auto-closed by the
require-issue-linkbot because I'm not assigned to this issue. The branch and verification stand; it reopens automatically once a maintainer assigns it. LIUwenhao (@LB623) and EMAN YOUSAF (@Eman-Yousaf) both asked for assignment before I did, so I'm still happy to hand this to either of them rather than duplicate.LB623 commentedon Aug 18, 2026
Ayush Baluni (@aayushbaluni) I don't have a PR in flight — please go ahead. Thanks for the thorough write-up and for checking first.
aayushbaluni commentedon Aug 18, 2026
LIUwenhao (@LB623) Thanks — appreciated, and noted. I'll carry it.
One mechanical note for whoever assigns this, because it affects what a reviewer would actually see.
I pushed
b336f72(the fourth invariant clause Aleksey Safonov (@safal207) proposed — asserting a resume re-runs only the cancelled node) after the bot closed #8634. GitHub does not sync a closed PR's head, so the PR still records only7045d72:The branch is correct; the closed PR's view of it is one commit behind.
Reading
reopen_on_assignment.yml, the reopen itself is unaffected — it matches onlabel:missing-issue-linkplus aFixes #5672reference in the body, and #8634 still has both. The workflow's own comment flags exactly this case for the check re-run:So on assignment it should reopen, pick up
b336f72, and re-run cleanly. If for any reason it comes back showing a single commit, ping me and I'll push an empty commit to force the sync rather than leaving a reviewer looking at the older diff.Current state of the branch, for the record:
_loop.py+30,test_pregel_async.py+73 across the two commitsmainexactly one clause fails in exactly one mode_stage_cancelled_task_writes(exc_value)fails theexitcase whileasync/syncstay greentests/test_pregel_async.py: 313 passed vs 310 onmain; full suite 2012 vs 2009, 0 failedruff check/ruff formatcleanBranch: https://github.com/aayushbaluni/langgraph/tree/fix/5672-persist-cancellation-exit-durability
safal207 commentedon Aug 18, 2026
Ayush Baluni (@aayushbaluni) — this is exactly the kind of localization I was hoping the invariant would force.
The strongest result here is not just that the fix gets 4/4. It is that unmodified
mainfails one clause, in one durability mode, while the mutation removes exactly the staging call and reproduces that same failure. That makes the defect much narrower than “cancellation persistence is flaky”: the durable state is intact, replay selection is intact, and the missing fact is specifically the cancellation occurrence evidence underdurability="exit".b336f72is important because it closes the behavioral side of the contract too. I would keep that fourth clause permanently:resume re-runs only the cancelled nodeis where a persistence bug turns into duplicate work / duplicate cost for the user, and it is exactly the sort of property that can silently regress while the checkpoint contents still look plausible.Now that core can prove its half, I think the platform boundary can be stated as a separate executable invariant when that surface is testable:
with the useful negative control:
That gives the full chain a clean proof split:
Before this fix, the second test could not tell whether platform ordering was wrong or core had simply dropped the fact. After this fix, those two failure classes become distinguishable.
And yes — no objection from me to you carrying #8634. The branch evidence is already unusually strong; once assignment reopens/syncs the PR, the reviewer should have a very crisp regression story to inspect.
atomicdjt commentedon Aug 20, 2026
Agreed — carrying the frontier kind explicitly is cleaner. Freezing the model here and moving to the executable crash/recovery fixture sounds like the right next step. Thanks for pushing the distinction to a precise boundary.
scout: the finder kept offering a lead the box would always refuse
safal207 commentedon Aug 23, 2026
One follow-up from the checkpoint/recovery work in this thread: after testing the same explicit-evidence projection contract across several agent runtimes, I found that a single PASS/FAIL score hides an important distinction.
For LangGraph, the source-pinned checkpoint/state boundary can host the complete witness sequence and replay it deterministically, but that does not automatically make the underlying evidence append-only or make every LangGraph application conformant. Those are separate claims supplied at different boundaries.
I wrote the comparison up here:
https://safal207.github.io/RESONANCE/conformance-is-a-vector-not-a-verdict.ru.html
The compact rule is:
projection conformance ≠ persistence ≠ append-only evidence ≠ framework-wide safetyI think this is directly relevant to #5672 because the cancellation problem is exactly where those layers separate: what was visible, what became durable, what can be replayed, and what the host is allowed to treat as authoritative are not the same fact.
The LangGraph row in the article is intentionally bounded to a pinned checkpoint/state surface; it is not a framework ranking or a claim that the issue here is solved.
1320800521 commentedon Sep 1, 2026
I ran a local follow-up specifically on the state boundary behind this behavior using Python 3.11.15, LangGraph 1.2.9, and AsyncSqliteSaver (no model / no external API).
What I observed across 16 deterministic stream cases:
So for applications that promise the user that already-rendered partial output will survive Stop/cancel, I would separate three things: UI stream, an application-owned event ledger for promised partial output, and LangGraph checkpoints for completed workflow state.
Important scope note: this is not a reproduction of the exact LangGraph Platform/API cancellation implementation originally reported here on 0.3.31. It reproduces the local semantic boundary that can produce the same rollback symptom.
Detailed matrix and test shape: https://www.xbstack.com/en/ai/langgraph-cancel-run-streaming-checkpoint-state-loss/?utm_source=github&utm_medium=referral&utm_campaign=langgraph_cancel_state_loss&utm_content=issue_comment
atomicdjt commentedon Sep 1, 2026
This is useful evidence, and I think it both confirms the semantic boundary and narrows the original Platform/API claim in an important way.
The 0/3 result makes the core distinction concrete: a custom stream update can be observable by the consumer without ever becoming a graph-state transition that the checkpointer is entitled to restore.
durability="sync"therefore cannot solve that class of loss; it only changes persistence timing for state changes that have actually completed.That suggests the cancellation contract should not be phrased as “flush the current checkpoint” alone. It needs to reconcile two different frontiers:
If
visible_frontier > state_frontierat explicit cancellation, the runtime/application needs durable evidence for that gap (event ledger / cancellation receipt / equivalent) before later treating the older checkpoint as the full authoritative history.Your
interrupt()observation adds another useful constraint: because code before the interrupt may execute again on resume, any application-owned event ledger should give emitted events an occurrence/sequence identity and make replay/idempotency explicit. Otherwise recovery can preserve the user-visible prefix but accidentally duplicate it on resume.So I would now state the issue more narrowly as:
That seems like a materially stronger and more falsifiable formulation than the original blanket claim that cancellation simply fails to checkpoint the latest streamed state.
1320800521 commentedon Sep 1, 2026
Agreed. Adding run-generation / attempt identity closes an ambiguity that a plain
C < P < Tordering still leaves open: recovery should never be able to satisfy a later terminal transition with a receipt from an earlier run occurrence.For the local visibility-vs-durability fixture I would carry the same identity into any application-owned event ledger as well:
run_id + run_generation + stream_seq. That gives replay an explicit idempotency key and prevents a resumed node from duplicating already-visible partial output.I would also keep transport interruption on a separate causal branch, as you described. My local test still only demonstrates the semantic boundary behind the rollback symptom; it does not independently prove the exact LangGraph Platform/API cancellation path from the original report, so I will keep that scope distinction explicit.
shimjaemandu commentedon Sep 21, 2026
The visible-vs-durable frontier in this thread seems related to a broader execution-evidence question I’m trying to validate.
Would an application benefit from a framework-neutral execution ledger that records a stable logical action/run identity and separately tracks “observable”, “durably recorded”, and “outcome unknown” states across cancel/resume boundaries?
The ledger would not try to replace LangGraph checkpoint semantics or infer recovery automatically; it would only preserve provenance so the application can reconcile later.
Would that be useful production infrastructure, or would you consider it redundant because this state should remain entirely application/LangGraph-owned?
safal207 commentedon Sep 21, 2026
I think there is a useful boundary here, but I would make the ledger earn its existence by keeping its claim deliberately smaller than another source of truth.
For this issue, the interesting case is a fact that is already observable to the application but is not yet represented by a durable LangGraph state transition.
So I would require the ledger to preserve distinctions such as:
with stable occurrence identity across cancel/resume — e.g. logical action/run + attempt/generation + event sequence — while leaving checkpoint semantics, retry, and reconciliation to LangGraph/the application.
A useful negative control would be:
The ledger should still be able to say “observed, not proven durable/outcome unknown”; it must not silently upgrade that fact to durable or succeeded simply because execution later continues.
If a framework-neutral layer cannot preserve that distinction more reliably than ordinary application state, then I would consider it redundant. If it can, that seems like a real execution-evidence boundary rather than a second checkpoint system.
atomicdjt commentedon Sep 24, 2026
Thank you both. The visible/durable/confirmed separation is the right boundary for my Agent Session Bridge work. I’ll keep the record append-only and identify each occurrence by logical run/action, attempt or generation, and event sequence; after cancellation before checkpoint, it should preserve “observed, not proven durable, outcome unknown” and leave checkpoint and recovery semantics to LangGraph/the application. I’ll add this as a concrete LangGraph negative-control case and report back with a minimal repro or results.
atomicdjt commentedon Sep 28, 2026
I added a runnable local negative control in TFB PR #42. It uses LangGraph 1.2.12 with AsyncSqliteSaver 3.1.1 and durability="sync", with no model or hosted API.
The consumer received a synthetic custom stream event; I cancelled before the node returned; then SQLite checkpoint readback still contained only the prior state, without the partial event. The recorded classification is OBSERVED / NOT_PROVEN / UNKNOWN, and the observation timestamp is left absent. The result and reproduction are in
fixtures/runtime/langgraph-cancel-negative-control.result.jsonandexperiments/langgraph_cancel_negative_control/repro.py.Scope: this demonstrates the local stream-versus-checkpoint evidence boundary. It does not reproduce the hosted LangGraph Platform/API cancellation path reported here, establish a platform defect, or characterize other versions/checkpointers.