Skip to content

Self-triggering workflow issues #16

Description

@poautran

In GitLab by @woutdenolf on May 10, 2024, 13:22 GMT+2:

Reported by @edgar

The example below exposes three issues

  • the "loop" node is not recognized as a start node (issue with ewokscore.graph.analysis.start_nodes) https://gitlab.esrf.fr/workflow/ewoks/ewokscore/-/merge_requests/236
  • we sometimes get "the pool is closed" errors (issue in pypushflow?)
  • "save" cannot have required parameters because it fails when triggered by "config" (ewoksppf input merge actor issue?)
from ewokscore import Task
from ewoks import execute_graph


class ConfigTask(Task, output_names=["config"]):
    def run(self):
        print("config")
        self.outputs.config = {}


class LoopTask(
    Task,
    input_names=["i", "n"],
    optional_input_names=["config"],
    output_names=["i", "keep_looping"],
):
    def run(self):
        self.outputs.i = self.inputs.i + 1
        self.outputs.keep_looping = self.outputs.i < self.inputs.n
        print("loop", self.outputs.i)


class SaveTask(
    Task, input_names=[], optional_input_names=["i", "config"], output_names=["result"]
):
    def run(self):
        if self.missing_inputs.i:
            print("skip save")
            self.outputs.result = False
        else:
            print("save", self.inputs.i)
            FILE.append(self.inputs.i)
            self.outputs.result = True


nodes = [
    {
        "id": "config",
        "task_type": "class",
        "task_identifier": "__main__.ConfigTask",
    },
    {
        "id": "loop",
        "task_type": "class",
        "task_identifier": "__main__.LoopTask",
        "default_inputs": [{"name": "i", "value": 0}, {"name": "n", "value": 10}],
    },
    {
        "id": "save",
        "task_type": "class",
        "task_identifier": "__main__.SaveTask",
    },
]


links = [
    {
        "source": "loop",
        "target": "loop",
        "data_mapping": [{"source_output": "i", "target_input": "i"}],
        "conditions": [{"source_output": "keep_looping", "value": True}],
    },
    {
        "source": "loop",
        "target": "save",
        "data_mapping": [{"source_output": "i", "target_input": "i"}],
    },
    {
        "source": "config",
        "target": "save",
        "data_mapping": [{"source_output": "config", "target_input": "config"}],
    }
]


if True:
    # Because loop cannot be a start node (see ewokscore graph analysis)
    links.append({
        "source": "config",
        "target": "loop",
        "data_mapping": [{"source_output": "config", "target_input": "config"}],
    })


workflow = {"graph": {"id": "testworkflow"}, "nodes": nodes, "links": links}

import logging

# logging.basicConfig(level=logging.DEBUG)

FILE = []
result = execute_graph(workflow, engine="ppf", scaling_workers=False, pool_type="thread")
print(FILE)

Assignees: noordhee

Migrated from GitLab: https://gitlab.esrf.fr/workflow/ewoks/ewoksppf/-/issues/16

Activity

  1. poautran commented on May 13, 2024

    @poautran
    MemberAuthor

    In GitLab by @gkoum on May 13, 2024, 14:44 GMT+2:

    mentioned in issue ewokscore#58

  2. poautran commented on Aug 30, 2024

    @poautran
    MemberAuthor

    In GitLab by @loichuder on Aug 30, 2024, 11:27 GMT+2:

    mentioned in merge request ewokscore!236

  3. poautran commented on Sep 17, 2024

    @poautran
    MemberAuthor

    In GitLab by @loichuder on Sep 17, 2024, 09:37 GMT+2:

    mentioned in merge request ewoks!170

  4. added this to the v3.1.0 milestone on Oct 4, 2026
  5. self-assigned this
    on Oct 6, 2026
  6. added theissue type on Oct 6, 2026
  7. updated priority medium and progress in-progress on Oct 6, 2026
  8. olofsvensson commented on Oct 28, 2026

    @olofsvensson
    Contributor
    • Two of the three problems are resolved. Making a self-triggering node a start node is handled in ewokscore with "force_start_node": True (or the extra config → loop link). The "the pool is closed" errors didn't come back in about 150 runs with thread and process pools, so they look fixed, probably in pypushflow. test_ppf_workflow30 covers both.
    • The third problem is still open: save can't have required inputs. Because loop has a conditional link to itself, ewokscore's link_is_required makes loop → save optional by default. So save runs as soon as config finishes, before loop has produced i, and fails with TaskInputError.
      • Workaround: put "required": True on the loop → save link.
      • Real fix: this needs an ewokscore change. Two strict xfail tests in test_ppf_workflow31 will fail once that change lands, which is the signal to remove them and close the issue.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

priority

medium

progress

in-progress

Projects

Relationships

None yet

Development

No branches or pull requests

Issue actions