Repository navigation
Worker child is not restarted after --max-tasks-per-child is reached #646
Description
Taskiq version
0.12.4
Python version
Python 3.12
OS
Linux
What happened?
Description
With Taskiq 0.12.4 and taskiq-redis 1.2.3, when --max-tasks-per-child is reached, the worker stops prefetching, waits for running tasks, and starts broker shutdown:
Max number of tasks executed.
Stopping prefetching messages...
No more tasks to wait for. Shutting down.
Shutting down the broker.
However, the child process remains alive. Since ProcessManager only restarts a worker after detecting that it is dead, this worker slot is never respawned and stops consuming tasks.
The only workaround I found was forcing the child to exit using os._exit(...), after which ProcessManager correctly respawned that specific worker.
Expected behavior
The child should exit after completing its admitted tasks, allowing ProcessManager to respawn it.
The documentation also describes this as a restart:
--max-tasks-per-child- maximum number of tasks to be executed by a single worker process before restart.
https://taskiq-python.github.io/guide/cli.html#other-parameters
The option was originally requested in #309 and implemented by #314. Issue #556 also shows the expected worker is dead → scheduling reload → restarted sequence.
Is the intended behavior for the child to exit and be respawned after reaching --max-tasks-per-child? If so, should the worker guarantee termination when graceful receiver or broker shutdown does not complete?
I previously created a discussion about this behavior:
Activity
juanmicl commentedon Sep 1, 2026
I dug into this and reproduced the behavior locally with a minimal custom broker. Summary:
max_tasks_to_execute,ProcessManagerdetects it within about a second and respawns it. Verified with--workers 1 --max-tasks-per-child 2: continuous exit/respawn cycles, new pid each time.shutdown_broker()intaskiq/cli/worker/run.py. I reproduced the exact same signature (worker stays alive, no reload ever scheduled) with two broker-level defects:shutdown()that blocks the event loop thread.shutdown_brokerusesasyncio.wait_for(timeout), but the timeout can only fire if the loop is free to run callbacks, so a blocking shutdown defeats it entirely (verified: no timeout warning after 50+ seconds with a 5 second timeout),To pinpoint which one happens with taskiq-redis,
py-spy dump --pid <stuck-worker-pid>on the hung worker: main thread insiderun_until_complete -> wait_for -> broker.shutdownmeans case 1; main thread inthreading._shutdownor ajoinmeans case 2.There is also a side effect worth noting: while a worker is stuck like this, the manager cannot shut itself down either, because multiprocessing joins non-daemon children at exit. In my repro the manager had to be SIGKILLed.
Design-wise, I think the worker child should guarantee its own exit after the graceful shutdown completes (short grace period, then
os._exit(0), the way gunicorn workers do). That would make--max-tasks-per-childself-healing regardless of broker behavior, and would unblock the manager's shutdown join too. Full write-up with the code path and repro logs: #608Would a PR in that direction be welcome?