Support durable external signals for waiting flow steps #660

Activity

1. jumski

   changed the title [-]Await signals with Approval demo tab[/-] [+]Support durable external signals for waiting flow steps[/+] on Aug 28, 2026
2. jumski

   added

   enhancementNew feature or requestNew feature or request

   pkgs/core

   pkgs/edge-worker

   pkgs/dsl

   pkgs/client

   priority:p3Later, research, or demand-gated workLater, research, or demand-gated work

   on Aug 28, 2026
3. jumski commented on Aug 28, 2026

   

   jumski

   on Aug 28, 2026

   Contributor

   More actions

   Thanks @cpursley for taking the time to bring this up and point me to a concrete implementation.

   This is a real gap in pgflow, and one I have wanted to solve for a long time. Approvals, webhook callbacks, and other externally completed work should be possible without keeping a worker or Edge Function alive.

   The proposed approach follows the Oban model:

   handler starts
   → calls await_signal()
   → task parks as waiting
   → signal requeues it
   → handler runs again from the top
   

   That gives useful behavior such as early-signal buffering and a built-in wait deadline. The tradeoff is that this is not a real continuation of the original handler invocation. Code before await_signal() runs again, so every earlier side effect must be idempotent. It also adds a signal store, another task status, timeout recovery, and more queue transitions.

   I have had a simpler and more explicit model in mind for pgflow: represent the wait as a handlerless task without a queue.

   flow.step({
     slug: 'wait_for_approval',
     queue: false,
     output: typeHint<Approval>(),
   });

   The flow remains explicit:

   create_order → wait_for_approval → charge
   

   When wait_for_approval becomes ready, pgflow creates a durable task but sends no PGMQ message and starts no handler. Trusted server-side code completes or fails the task, and normal workflow progression starts charge.

   The main differences are:

   Concern	Inline await_signal()	Manual queue: false task
   Wait location	inside handler code	explicit workflow step
   Worker use	starts and later parks	never starts
   Resume	reruns handler from the top	starts downstream steps
   Earlier side effects	must be idempotent	no pre-wait handler code
   State changes	started → waiting → queued → started	started → completed or failed
   Early signal	buffered by pgflow	needs an optional later layer
   Timeout	built in	application-owned initially
   Runtime model	needs special handler control flow	shared database task transition

   I currently prefer the manual-task direction because it keeps the workflow visible, avoids handler replay, and needs fewer states and races in the core.

   It also aligns with an idea I want pgflow core to leave room for: multi-language flows where each step can be invoked by a worker in a different language. For example, a flow authored in TypeScript could have one step run on a Python GPU worker, another on a TypeScript worker, and an approval task completed by application server code. The durable task protocol should own progression rather than a control-flow exception tied to one worker runtime.

   This does not make the additional signal behavior useless. Early buffering, core-managed deadlines, or a signal() helper could later sit on top of manual tasks if concrete use cases need them. For example, a small signal layer could buffer a payload before the manual task exists and complete it when it becomes ready, without replaying a handler.

   I captured the manual-task direction in #661. It is deliberately a deferred design issue: the exact API, storage model, and release sequence are not final. I want to validate this simpler model before adding mid-handler wait and replay semantics to pgflow core.

   I will keep this issue open while #661 is evaluated. If the manual-task model works, this issue can either close as covered or narrow to the remaining early-buffering, timeout, and named-signal capabilities.

   👍React with 👍1Reacted by Chase Pursley
4. cpursley commented on Aug 28, 2026

   

   cpursley

   on Aug 28, 2026

   Last edited by cpursley

   Author

   More actions

   Awesome, I'll do some thinking. We're not necessarily totally married to the current approach in the PR and open to unifying this if/where it makes sense (in the context of the various language/platform strengths).

   🚀React with 🚀1Reacted by Wojtek Majewski
5. cpursley commented on Sep 14, 2026

   

   cpursley

   on Sep 14, 2026

   Author

   More actions

   @jumski have you thought more about the details of this design/schema? Turns out I need signal await type of feature in my product that uses pgflow and was planning to work on this. Glad to align with your simpler suggestion.
6. jumski commented on Sep 14, 2026

   

   jumski

   on Sep 14, 2026

   Contributor

   More actions

   Yes I have an epic that outlines a path to it and I'm working through the required changes to get there #653

   The one that will allow queue less task is #661

   The signals can he implemented on top of this simpler approach by just observing the changes to steps.

   Can you describe your particular use case so I can understand better if it is a good fit?

   👍React with 👍1Reacted by Chase Pursley