Repository navigation
A task with if-dependencies or/and soft-dependencies pulls all of them if any gets pulled #710
Description
Considering the following:
loader: taskgraph.loader.transform:loader
transforms:
- taskgraph.transforms.task
task-defaults:
worker-type: succeed
tasks:
task-a:
label: task-a
optimization:
skip-unless-changed: ["world"]
description: task-a
task-b:
label: task-b
description: task-b
final:
label: final
dependencies:
task-a: task-a
task-b: task-b
if-dependencies:
- task-a
- task-b
soft-dependencies:
- task-a
- task-b
description: finaltask-a gets optimized because of skip-unless-changed but gets pulled back in by final
2025-06-23 17:42:57,207 - DEBUG - no files found matching a pattern in `skip-unless-changed` for "task-a"
2025-06-23 17:42:57,207 - DEBUG - optimize: final kept because of never
2025-06-23 17:42:57,207 - DEBUG - optimize: task-b kept because of never
2025-06-23 17:42:57,207 - DEBUG - optimize: task-a kept because of dependent tasks
2025-06-23 17:42:57,207 - INFO - No tasks removed during optimization
If I optimize task-b away too, then I get an empty graph.
I would expect task-a to not get pulled by final if it's already been optimized.
Activity
ahal commentedon Jun 23, 2025
Is the
soft-dependenciesnecessary to reproduce? I'm surprised to see a task that would define both of them. I guess it's legal, but I always thought of those two things solving two different use cases. I honestly don't know what's expected to happen if both are specified.Eijebong commentedon Jun 23, 2025
Good point, removing either doesn't change the outcome, both get pulled.
With just soft-deps:
With just if-deps:
[-]A task with `if-dependencies` pulls all of them if any gets pulled[/-][+]A task with `if-dependencies` or/and `soft-dependencies` pulls all of them if any gets pulled[/+]ahal commentedon Jun 24, 2025
Slight difference in the reason they both get kept, which is expected as they are hitting different code paths (just something to beware of).
When I implemented this, I was mainly doing it to ensure downstream release tasks don't cause builds to run. E.g, to avoid
build-signingpulling inbuild. These were all one to one mappings in the dependency graph, so I guess I never bothered testing this case, or I decided it was out of scope for my immediate goals.I think the behaviour you expected makes the most intuitive sense, though this would be a breaking change and is worth a bit of extra thought to ensure there's no use cases where the current behaviour would be desired.
ahal commentedon Jun 24, 2025
This might also be a PITA to implement.. As I recall it was quite finicky to get working properly without also breaking other edge cases. Luckily I think I added a bunch of test cases for most of those edge cases, and it's easy to add new ones that cover whatever desired behaviour we land on here:
taskgraph/test/test_optimize.py
Line 248 in 0fc0e7a
Add very-soft-if-deps
fix!: ensure if-dependencies apply at target phase
ahal commentedon Apr 1, 2026
Ok, so upon further testing, my patch does not fully fix this issue. It fixes it in part by taking
if-dependenciesinto account in the target phase. But the described behaviour also exists in the optimization phase.While looking into it, I did discover that there are at least two kinds in Gecko that are explicitly relying on the current
if-dependenciesbehaviour:https://searchfox.org/firefox-main/source/taskcluster/kinds/upload-symbols-dummy/kind.yml
https://searchfox.org/firefox-main/source/taskcluster/kinds/upload-generated-sources-dummy/kind.yml
So if we fix this issue, we'll need to solve what those tasks are accomplishing a different way.
See also:
https://bugzilla.mozilla.org/show_bug.cgi?id=2020718 (we ended up working around this another way)
ahal commentedon May 11, 2026
We had another place where we bumped into this limitation for Enterprise. We wanted to have a
completetask that depends on a bunch of release tasks and then notifies when they're all done. I tried to make it useif-dependencies, but anytime one release task got selected, all of them were added.I'm becoming very convinced that the current behaviour is not desired, and more importantly, that it's worth fixing despite the two kinds relying on it in the previous comment.