DECISIONS.md

Decisions

Each entry: what we chose, why, what we rejected, and what would change our mind. Checked on 2026-09-14.

1. Name: plugpull

"Pull the plug" is what the tool does. Short and easy to type.

Candidate	npm	PyPI	GitHub repos with this name
plugpull	free (404)	free (404)	0
pullplug	free	free	0
yankbench	free	free	0
crashbench	free	free	6 (top: ortegaalfredo/crashbench, 14 stars)
killbench	free	free	4 (top: whitecircle/killbench, 22 stars)
agentcrash	free	taken	not checked
halfway, midrun	taken	free	not checked

Checked with https://registry.npmjs.org/<name>, https://pypi.org/pypi/<name>/json, and https://api.github.com/search/repositories?q=<name>+in:name. Change it if the maintainer prefers another name. Nothing is published yet, so renaming is cheap.

2. Language: Python first

• Three of the five frameworks with public failure reports are Python: LangGraph, Pydantic AI, OpenAI Agents SDK. LangGraph has the most reports.
• The harness only needs to start a process, kill it, and read a file. A TypeScript target can plug in later as a command (node target.js start|resume), so Python does not lock out TS.

Rejected for now: TypeScript first. The strongest Vercel AI SDK reports are about the browser (stream resume after tab switch) or about Vercel AI Gateway, a paid hosted service. Both are harder to run locally with no account. Mastra (@mastra/core, Apache-2.0) has a local report (mastra-ai/mastra#22636) and is the first TS candidate.

3. Targets

#	Framework	Version tested	License	Why	Status
1	LangGraph	1.2.11 (+ langgraph-checkpoint-sqlite 3.1.1)	MIT	Most reports: langchain-ai/langgraph#8039, #5672, #3275, all open on 2026-09-14 (#7417 is also open, but it is about LangGraph Cloud, the hosted service, not the local library). Runs fully local with a SQLite checkpointer and no model.	working
2	Pydantic AI	2.43.0	MIT	Has test models built in (no API key). Open reports on approval/resume: pydantic/pydantic-ai#3274, #5536.	chosen, not tried yet
–	OpenAI Agents SDK	0.22.2 (openai-agents, extra openai-agents, needs >=0.22.1)	MIT	Target of lost-session-write (§8): openai/openai-agents-python#4775. Runs local with the SDK's ScriptedModel, a fake API key and tracing off. Not target 2.	working

Rule for target 2: if Pydantic AI cannot save a run and resume it in another process without an API key, switch to OpenAI Agents SDK (openai-agents 0.22.2, MIT; openai/openai-agents-python#4775).

4. What a verdict means

This section is about kill-mid-tool. lost-session-write is in §8.

Version 2, 2026-09-14. Version 1 counted how many times the tool body ran, and killed only after the effect. So the right fix (an idempotency key) got FAIL, and a wrong fix (write "done" to a ledger first, then charge, which can lose the charge) got PASS.

• Effect. An effect is what the provider sees, not a run of the tool body. The tool reports it with record_effect(name, *, key=None) to a fake provider inside the harness. Two calls with the same effect name and the same key are one effect. The same key on two different names is two effects (a charge and a receipt email can share one order id). Every call without a key is a new effect.

• Kill points. kill-mid-tool kills the agent at two points around the provider call, and gives one result per point:

  • before-effect: the tool has started, the provider has not recorded the effect;
  • after-effect: the provider has recorded the effect, the tool has not returned.

  After resume, for every effect name the run used (including the name of the held call), the provider must have seen exactly 1 distinct effect. 0 = lost, 2 or more = duplicated.

• Kill check. The provider holds the tool's call and does not answer it, so the tool cannot return. Before a verdict, the harness checks, in this order, facts it gets from the kernel or from its own child process:

  1. the agent is still running, and the held call is still open;
  2. the process that made the call is in the agent's process group, the group the harness kills. Its pid comes from the socket (LOCAL_PEERPID on macOS, SO_PEERCRED on Linux);
  3. after SIGKILL to that group, the agent's exit status says it died from SIGKILL;
  4. the held call closes within 1 second, so no other process (for example a fork that left the group) keeps it open.

  Nothing the target writes counts as proof. If a check fails, or the platform does not report the caller's pid, the result is ERROR.

• PASS: the scenario ran as designed, and the provider saw exactly 1 effect.

• FAIL: the scenario ran as designed, and the expectation broke: the effect was lost or duplicated, or resume crashed (non-zero exit), or resume hung past the timeout. A FAIL is not always a framework bug. Example: LangGraph documents that "a task that started but did not finish may run again on that resume, so design side effects to be idempotent" (https://docs.langchain.com/oss/python/langgraph/functional-api.md, section Idempotency). A tool with no idempotency key still gets FAIL at after-effect, because the provider saw two effects. When the framework documents the behavior, the FAIL text must link the docs (not for --agent, where plugpull does not know the framework, see §10).

• ERROR: plugpull could not run the scenario as designed: the agent exited before the tool called the provider, the tool did not call the provider before the timeout, the kill check failed, or the framework is not installed. Never counted as PASS or FAIL.

• Exit code: 2 if any ERROR or a requested framework is missing; else 1 if --strict and any FAIL; else 0.

Rejected:

• Counting runs of the tool body (version 1): it cannot tell a deduplicating provider from a double charge.
• One kill point: after-effect alone passes the ledger-first tool; before-effect alone passes a tool with no key.
• A marker file written by the target after the tool returns, as proof that the kill landed in the tool. The kill can land between the effect and the marker, and that run still looks correct.
• Timing alone as proof ("the call closed soon after the kill"). A worker outside the process group that gives up after 0.3 seconds closes the call too, and got PASS or FAIL.
• Keys shared across effect names. It would hide a lost receipt email behind a charge that used the same order id.

What would change our mind: a framework that runs tools outside the agent's process group (for example on a remote worker). The kill check would then always give ERROR, and the harness would need another way to reach the tool.

5. No LLM calls, no keys

Every target uses a fake model. Results must be the same on every machine and cost nothing. We do not promise to measure tokens or cost correctly; that is not possible from the outside (openai/openai-openapi#539). We only show where the behavior breaks.

6. SIGKILL, not SIGTERM

SIGKILL skips all cleanup, like a crash, an OOM kill, or a deploy that stops a container hard. A graceful-shutdown scenario can be added later as a separate scenario. The harness kills the target's whole process group, so a wrapper process (like uv run) does not keep the agent alive.

7. The target interface is not stable before 0.1

start / resume / record_effect(name, *, key=None) may change. Say so in the README until 0.1.

--json prints {"schema": 1, "results": [...]}. The schema number changes when a field is removed or changes meaning. Each result has scenario, target, kill_point, verdict, detail and evidence. kill_point holds the scenario's fault point: before-effect or after-effect for kill-mid-tool, write-lost or ack-lost for lost-session-write. The name stays kill_point in schema 1, although lost-session-write kills nothing.

evidence holds only measured values. A field is missing when the run stopped before it was measured.

• kill-mid-tool: effects (count per effect name), calls (effect, key, caller pid from the kernel, phase), kill (agent pid, caller pid and its process group, agent exit status, seconds until the held call closed), resume_returncode, and stderr when there is output.
• lost-session-write: staged_writes (numbers of the writes that carried the staged input in the first run), lost_write (the one lost in this run), writes (every Session write: phase, number, session, pid, items, and outcome = stored, write-lost or ack-lost; the pid is what the target sends, not a kernel fact), start_code, start_stderr, items (how many times each item written is stored after resume, keyed by its JSON), resume_returncode (as in kill-mid-tool: the exit code of resume, null when it hung past the timeout; added 2026-09-14, schema stays 1), and stderr when resume failed. When the first run already cannot be used, every result is ERROR and evidence has baseline_writes and stderr instead.

stderr (in both scenarios) is the target's stderr, up to the last 60 lines and 4000 characters (a generous cap for a full traceback that still won't let a runaway spew blow up the report; older bytes are dropped from the front, marked ...(truncated)). Fixed 2026-09-14: "resume failed" always included a crash (resume_returncode non-zero), but a hang (resume_returncode null) used to reach its FAIL with no stderr at all — this was already promised by the words above, just not built. Both scenarios now fill stderr on a hang too, from whatever the resume process had written before the timeout killed it. No new field: stderr's meaning ("why this verdict, from the target's own output") already covered it.

The CLI's text output used to print stderr only for ERROR (cli.py, next to the result line, one result may write several lines to stderr). It now also prints it for a FAIL where resume crashed or hung (resume_returncode non-zero or null) — the byte-for-byte promise in the README's example output is only for PASS and for a FAIL where resume exited 0, so this does not touch it.

8. lost-session-write: a Session write that fails

Added 2026-09-14, for openai/openai-agents-python#4775.

• Fault. The agent's Session store is a fake store in the harness (a Unix socket), like a database on another machine. One Session call is one store call. It keeps exactly what it is given: no keys, no deduplication, so the store cannot hide a duplicate. The harness picks one Session write and closes that call without an answer, at one of two fault points:

  • write-lost: the store did not keep the items;
  • ack-lost: the store kept the items.

  The call raises in the agent. The process is not killed.

• Which write. The harness gives the agent a staged input with a random token (PLUGPULL_STAGED_INPUT). A first start with every write answered finds the writes that carry the token. Each of them is lost in turn, at each fault point: one result per (fault point, write), and the text starts with write N:. The agent does not choose.

• Steps. start keeps the run state the way the framework docs say and exits with the error. resume retries, with the store answering normally.

• PASS: after resume, every item ever written is stored exactly once.

• FAIL: an item is stored 0 times (lost) or 2 or more times (duplicated), or resume exits non-zero, or resume hangs past the timeout.

• ERROR: the scenario did not run as designed: the store could not start; the first run did not finish, exited non-zero, wrote no staged input, or already stored an item twice; or in the faulted run, start did not reach the chosen write, did not follow the first run up to it (the chosen write does not carry the staged input, or the writes up to it that carry the staged input are not the same write numbers as in the first run; other items in the writes are not compared), exited 0 (the lost write did not make the run fail), or kept writing to the Session after it. Never counted as PASS or FAIL.

• Exit codes and --strict: as in §4.

• FAIL text. On openai-agents, an ack-lost FAIL where resume exited 0 and an item of the lost write (the staged input) is stored 2 or more times names openai/openai-agents-python#4775 and says the docs promise the opposite: "Across serialization, resume, and replay-safe retries, the SDK preserves one durable InputItem occurrence." (https://openai.github.io/openai-agents-python/results/, section Add input before resuming). It does not say "documented": this is not behavior the SDK asks you to handle. A write-lost FAIL gets no note, because #4775 is about the lost acknowledgment. Neither does an ack-lost FAIL of another kind (resume exited non-zero or hung, an item lost): the CLI decides from resume_returncode, lost_write, writes and items in the evidence, not from the text.

Measured on 0.22.2 (RunState schema 1.17): write-lost PASS, ack-lost FAIL (the staged input stored 2 times). On 0.22.2 with PR #4906 applied (schema 1.18): both PASS. No network connection in either run (counted in tests/openai_agents_rig.py).

Rejected:

• An idempotency key on Session writes. Session stores in the SDK (for example SQLiteSession) take none, and a store that deduplicates would pass a framework that writes twice.
• Letting the target say which write failed. The harness picks the write and checks the target reached it and failed on it; nothing the target writes counts as proof.
• Killing the process at the write, as in kill-mid-tool. #4775 is about an exception the app sees and recovers from as the docs say, not about a crash.

What would change our mind: a framework whose Session layer retries a failed write by itself before the app sees the error. start would then exit 0, and every result would be ERROR.

9. LangGraph durability="async": the kill can land before any checkpoint is saved

Added 2026-09-14, after the first CI run (GitHub Actions run 34855642495, Python 3.10) printed FAIL for LangGraph durability="async" before-effect with the no-key tool, where the tests expected PASS. That is the same line the README once reported with "do not know the cause".

• Mechanism. With durability="async", LangGraph hands every checkpoint save to a thread pool and does not wait for it: self.submit(self._checkpointer_put_after_previous, ...) for a checkpoint and self.submit(self.checkpointer_put_writes, ...) for a task's writes (langgraph/pregel/_loop.py, LangGraph 1.2.11). Only "sync" waits (pregel/main.py: if durability_ == "sync": loop._put_checkpoint_fut.result()). So the tool starts while saves may still be queued. plugpull's kill point does not depend on time (the harness kills when the provider has the tool's call), but how much LangGraph has saved by then does. If at least one checkpoint is saved, resume continues from it (it runs the model or the tool again) and a no-key tool calls the provider once: PASS. If none is saved, the target's app.invoke(None, ...) raises langgraph.errors.EmptyInputError: Received no input for __start__, resume exits 1, and the provider saw 0 effects.

• Evidence. A copy of the SQLite file taken just before resume, on this machine (macOS, Python 3.10, tests/langgraph_rig.py no-key, before-effect):

  • no load, 20 runs: 3 checkpoints saved 1 time, 2 saved 12, 1 saved 7, none 0; 20 PASS;
  • with 16 processes doing write + fsync, 40 runs: none saved in exactly 1 run, and that run is the only FAIL; its stderr is the EmptyInputError above. The same load on Python 3.13 (no snapshot), 40 runs: FAIL 3, each with 0 effects, resume exit 1 and EmptyInputError. So the race is not specific to Python 3.10;
  • every save slowed by 0.2 s (PLUGPULL_RIG_SAVE_DELAY=0.2), 20 runs each, Python 3.10 and 3.13 alike: saves left to LangGraph, FAIL 20 (each with 0 effects, resume exit 1, EmptyInputError, agent killed by SIGKILL); with PLUGPULL_RIG_SAVE=first, PASS 20.

  The CI log does not show the stderr (pytest cut the result short), so for CI this is the mechanism that fits its evidence best (FAIL, so the kill check passed; detail starting with "r", which fits "run could not be resumed" but also "resume hung"), not a proven cause. Neither is it proven for the one local FAIL the README reported, whose output was not saved.

• Verdict: FAIL, unchanged. The kill check passed, so the scenario ran as designed, and resume crashed with the effect lost: that is FAIL in §4. It is not ERROR, which is for plugpull not being able to run the scenario. LangGraph documents the risk: "there is a small risk that LangGraph does not write checkpoints if the process crashes during execution" (https://docs.langchain.com/oss/python/langgraph/checkpointers, section Durability modes). As §4 asks, the FAIL text links it: the CLI adds a note when resume exited non-zero and its stderr has langgraph.errors.EmptyInputError, on a durability=async line only (the CLI reads PLUGPULL_DURABILITY from the environment the variant ran with; the target's default is "async"). Not on a sync line: "sync" waits for every save before the next step, so the same crash there is not the risk the docs name, and we have no documented explanation for it. No new verdict, exit code or field.

• Tests. The async cases in tests/test_langgraph.py depended on this timing, under a test named ..._do_not_depend_on_timing. The rig now takes timing out, in start only: PLUGPULL_RIG_SAVE=first makes the tool wait until LangGraph has saved the checkpoint the tool runs from (a wrapper around SqliteSaver), PLUGPULL_RIG_SAVE=never blocks every save, and PLUGPULL_RIG_SAVE_DELAY makes every save wait, like a slow disk, to reproduce the race on demand. The no-key async case is asserted both ways: PASS/FAIL when saved first, FAIL/FAIL (with the note) when nothing is saved. The save="first" cases also slow every save by 0.1 s: without that, saves usually win the race anyway, and a rig whose tool does not wait still passed every case. Measured with the wait removed, 20 runs per case, all three first cases: delay 0, the cases still matched their verdicts in 20, 20 and 7 runs; delay 0.01, 0.02, 0.05 and 0.1 s, in 0 runs. 0.1 s leaves room for a slower CI runner. uv run plugpull's own target is not changed: its durability=async before-effect line can be PASS or FAIL, and both are true results.

Rejected:

• A retry, a sleep or a longer timeout in the test: the race stays, only rarer.
• ERROR for this result: the scenario ran as designed; ERROR would hide a loss the framework documents.
• Making plugpull's LangGraph target wait for the checkpoint: that tests an app nobody writes, and would hide the loss from users of the default durability.
• A fourth verdict for timing-dependent results: this result is a real FAIL, so it does not need one.

What would change our mind: a run where resume crashes like this although a checkpoint was saved, or a LangGraph version that saves the first checkpoint before the first step runs.

10. Your own agent: plugpull --agent COMMAND

Added 2026-09-15. Like the target interface in §7, this is not stable before 0.1.

• What. kill-mid-tool on an agent plugpull does not ship. The public surface is three things, and each is what the built-in targets already use:

  1. --agent COMMAND (repeatable). plugpull splits COMMAND like a POSIX shell (shlex.split; no shell runs it) and runs it with start, then with resume, added as the last argument. With --agent and no --target, no built-in target runs.
  2. from plugpull import record_effect: record_effect(name, *, key=None) from §4, called where the tool calls the outside service, with the key the tool sends it.
  3. PLUGPULL_WORKDIR, an environment variable: a directory that is new for each kill point and the same for its start and resume. The agent keeps its checkpoints there, so one kill point cannot resume the run of another. It holds only what the agent writes: plugpull keeps its own files (the provider socket, the stderr of start and resume) in another directory, so an agent that deletes everything in its workdir gets the same verdicts.

• Verdicts. The same code runs (run_kill_mid_tool), so §4 holds unchanged: effects counted per distinct (name, key), both kill points, the kill check from kernel facts, PASS/FAIL/ERROR, exit codes, --json schema 1. target holds COMMAND as given. No new verdict, exit code or field.

• Smallest example (the full file is examples/langgraph_agent/agent.py):

  from plugpull import record_effect
  
  @tool
  def charge_card(amount: int, tool_call_id: Annotated[str, InjectedToolCallId]) -> str:
      """Charge the customer's card."""
      record_effect("charge_card", key=tool_call_id)  # where the tool calls the payment API
      return "charged"
  
  db = Path(os.environ["PLUGPULL_WORKDIR"]) / "checkpoints.sqlite"
  app = graph.compile(checkpointer=SqliteSaver(sqlite3.connect(db, check_same_thread=False)))
  config = {"configurable": {"thread_id": "order-42"}}
  if sys.argv[-1] == "start":
      app.invoke({"messages": [HumanMessage("Charge $42.")]}, config, durability="sync")
  elif sys.argv[-1] == "resume":
      app.invoke(None, config, durability="sync")

  plugpull --agent "python agent.py".

• What the agent must give, so the kill check in §4 still holds. The tool runs in the process COMMAND starts, or in a child that stays in its process group. The checkpointer outlives the process (a file, not InMemorySaver). start and resume use the same thread id. resume continues the saved run (invoke(None, config)), it does not start a new one. Compared with the same LangGraph agent without plugpull: 7 lines, 4 steps (README, "Run it on your own LangGraph agent").

• Durability in the sample: "sync". LangGraph waits for each checkpoint before the next step (loop._put_checkpoint_fut.result(), pregel/main.py, LangGraph 1.2.11), so the checkpoint holding the tool call id is saved before the tool runs, and the kill finds it at both kill points. With "async" it may not be (§9): no-key before-effect can FAIL with nothing saved, and the tool call id as key gave after-effect FAIL in 9 of 20 runs. Those are true results for an "async" agent, but not an example whose output a reader can compare. Sample, "sync": no-key PASS / FAIL (2 effects), tool-call-id PASS / PASS, the same as the built-in target's tools in tests/test_langgraph.py.

• ERROR on this path (all in tests/test_user_agent.py): the tool never calls record_effect (the agent exits first, or the timeout passes); the call comes from a process outside the process group, or such a process keeps it open after the kill; COMMAND cannot be started (not found, not executable), a new check in the harness that built-in targets never reach. An empty COMMAND or unbalanced quotes stops before any run, with exit code 2 and a usage error, like a framework that is not installed.

Not supported, on purpose:

• Framework notes. A FAIL from --agent links no docs: plugpull does not know the framework, or its durability, of a command it did not write. The rule in §4 ("the FAIL text must link the docs") is kept where plugpull knows the framework, the built-in targets. The README section links the LangGraph docs for the FAILs a LangGraph agent gets.
• lost-session-write on your own agent, or any scenario other than kill-mid-tool.
• Separate start and resume programs, a shell (pipes, &&, variables), and a timeout option. The timeout is 60 seconds, as for the built-in targets.
• Running the agent outside plugpull with record_effect in it: without PLUGPULL_PROVIDER it raises KeyError. The agent under test is a copy of the tool, or one behind a switch.

Rejected:

• A Python API that loads the user's graph inside plugpull (plugpull.run(app), entry points). The kill must hit a separate process group and resume must be a new process, so plugpull would start a child process anyway; and it would tie plugpull to one framework's API.
• --start COMMAND and --resume COMMAND. Two commands to keep in step, where the harness already takes one command and adds a word.
• A new name for record_effect (for example plugpull.effect): two names for one call that §4 and the built-in targets already name.
• A --framework langgraph option to add the notes: plugpull would call a FAIL "documented" for an agent whose durability and tools it cannot see.
• A plugpull.workdir() helper: one more symbol for what one environment variable already says.

What would change our mind: users whose start and resume must be different programs (then two options); agents that keep checkpoints in a shared database, where a new directory does not separate runs (then plugpull should hand out a thread id too); agents that need more than 60 seconds to reach the tool (then a timeout option); users asking for the framework notes on their own agent.