Skip to content

Ephemeral (single use) runner registrations #510

Closed
Closed
Ephemeral (single use) runner registrations#510
Labels
Runner ❤️ ContainerenhancementNew feature or request

Description

@MatisseHack
MatisseHack
opened on May 30, 2020

Describe the bug
When starting a self hosted runner with ./run.cmd --once, the runner sometimes accepts a second job before shutting down, which causes that second job to fail with the message:

The runner: [runner-name] lost communication with the server. Verify the machine is running and has a healthy network connection.

This looks like the same issue recently fixed here: microsoft/azure-pipelines-agent#2728

To Reproduce
Steps to reproduce the behavior:

  1. Create a repo, enable GitHub Actions, and add a new workflow

  2. Configure a new runner on your machine

  3. Run the runner with ./run.cmd --once

  4. Queue two runs of your workflow

  5. The first job will run and the runner will go offline

  6. (Optionally) configure and start a second runner

  7. The second job will time out after several minutes with the message:

    The runner: [runner-name] lost communication with the server. Verify the machine is running and has a healthy network connection.

    (where [runner-name] is the name of the first runner)

  8. Also: trying to remove the first runner with the command ./config.cmd remove --token [token] will result in the following error until the second job times out:

    Failed: Removing runner from the server
Runner "[runner-name]" is running a job for pool "Default"

Expected behavior
The second job should run on (and wait for) any new runner that comes online rather than try to run as a second job on the, now offline, original runner.

Runner Version and Platform

2.262.1 on Windows

Runner and Worker's Diagnostic Logs

_diag.zip

Activity

  1. MatisseHack
    added
    bugSomething isn't working
    on May 30, 2020

  2. TingluoHuang commented on May 30, 2020

    @TingluoHuang
    TingluoHuang
    on May 30, 2020
    Member

    @MatisseHack
    How did you find the flag? 😆
    the run once flag --once is not in the official actions doc and better not be used in production, there is a known issue that we haven't fix. 😄

  3. bryanmacfarlane commented on May 30, 2020

    @bryanmacfarlane
    bryanmacfarlane
    on May 30, 2020
    Contributor

    Yeah, this is why it's not part of the documentation or command line help. Because the partially implemented code doesn't work and has races so we didn't ship it.

    We have work on the backlog to register a runner with the service as ephemeral. It has to be a service side feature so the service is cooperative and doesn't assign it another job in the window when the runner or outside / in automation decides to tear it down.

  4. bryanmacfarlane
    added
    enhancementNew feature or request
    and removed
    bugSomething isn't working
    on May 30, 2020

  5. MatisseHack commented on Jun 1, 2020

    @MatisseHack
    MatisseHack
    on Jun 1, 2020
    ContributorAuthor

    I've used the --once flag extensively with Azure Pipelines to create an automatically scaling pool of single use agents. I'm trying to convert that system to work with GitHub Actions. I was pleased to find that the flag still worked (at least partially)!

    Makes sens why it wasn't shipped 😄 . I look forward to using it when it's supported though!

  6. TingluoHuang
    added
    Runner ❤️ Container
    on Jun 6, 2020

  7. rclmenezes commented on Jun 7, 2020

    @rclmenezes
    rclmenezes
    on Jun 7, 2020

    We solved this issue by adding a timeout to our ephemeral workers. The timeout will deregister and kill the worker.

    #!/bin/bash -x
set -eux

/usr/bin/actions/register

# Automatically de-register in 30 min
echo "sudo /usr/bin/actions/deregister && sudo shutdown -h now" | at now + 30 minutes

sudo -i -u actions /home/actions/run.sh --once
/usr/bin/actions/deregister
shutdown -h now

    Actions team: please support --once when you have a chance!

    We currently have ~16x parallelization on our CI to keep builds under 20 minutes. When moving to self-hosted, we had the following options:

    1. Have an autoscaling group that keeps 16 instances around at the minimum and deploy more at peak times (this is a huge waste of money, it's hard to scale out and we have no clue how to scale-in without affecting existing jobs).

    2. Make a Github-hosted job that deploys spot instances to perform the other jobs! This is substantially cheaper, avoids queues at peak times and allows us to parallelize even more when we need to!

    Unfortunately, the second strategy only works well with --once...

    So please consider supporting --once officially! If --once stops working, we'll probably have to consider an alternative CI provider.

  8. emilyalbini commented on Jun 8, 2020

    @emilyalbini
    emilyalbini
    on Jun 8, 2020

    We also use the --once flag as we need a fresh build environment for each build (on AArch64, so we can't use GitHub hosted runners), and while we haven't encountered this issue yet it'd be really nice to get proper support!

  9. rclmenezes commented on Jun 10, 2020

    @rclmenezes
    rclmenezes
    on Jun 10, 2020

    Also @bryanmacfarlane - two more requests that would make ephermeral workers easier:

    1. Have --once workers automatically de-register themselves on completion / timeout. Currently, if a worker errors out before cleanup code, you'll have a ton of zombie "offline" registered workers. I think this needs to be solved on the backend.

    2. If no workers are registered with the proper labels, Github will error out the job immediately. This is a problem if you're spinning up ephemeral workers, because the job will error before they boot up.

    Our hackish solution is to keep a registered worker that does no work:

    image

    It simply re-registers itself once a day to avoid the 30 day cleanup 😂

    Thanks!

  10. nabilcode commented on Jun 18, 2020

    @nabilcode
    nabilcode
    on Jun 18, 2020

    👋 there,

    Do we have a date as to which the --once would be supported?

  11. zacker150 commented on Jul 6, 2020

    @zacker150
    zacker150
    on Jul 6, 2020

    Our hackish solution is to keep a registered worker that does no work:

    image

    It looks GitHub changed the runner service to error out the actions if there are no live runners. This hackish solution no longer works ☹️.

  12. rclmenezes commented on Jul 9, 2020

    @rclmenezes
    rclmenezes
    on Jul 9, 2020

    This really sucks. It's the last nail in the coffin for ephemeral self-hosted workers.

    We're going to switch to permanent self-hosted workers and then probably switch to another CI provider :(

  13. bryanmacfarlane commented on Jul 10, 2020

    @bryanmacfarlane
    bryanmacfarlane
    on Jul 10, 2020
    Contributor

    @rclmenezes ack on #1 and #2. we're currently designing and working on it. The plan is exactly what you laid out, register the runner ephemeral with the backend service so the service auto cleans it up after the job is complete and the runner / container exits.

  14. bryanmacfarlane
    changed the title [-]Run once runner accepts a second job[/-] [+]Ephemeral (single use) runner registrations[/+] on Jul 10, 2020

  15. 11 remaining items

  16. stgraber commented on Jan 28, 2021

    @stgraber
    stgraber
    on Jan 28, 2021

    Forgot to mention but this would also make it easy to run on non-x86 architectures. LXD was similarly used to build up the foreign architecture support of Travis-CI where it's used to run arm64, ppc64el and s390x instances. Something similar could easily be done with Github actions if it wasn't for this limitation :)

  17. xriu commented on Feb 9, 2021

    @xriu
    xriu
    on Feb 9, 2021

    Any updates on this?

  18. kylecarbs commented on Feb 10, 2021

    @kylecarbs
    kylecarbs
    on Feb 10, 2021

    @bryanmacfarlane any updates on this?

  19. 0x2b3bfa0 commented on Feb 14, 2021

    @0x2b3bfa0
    0x2b3bfa0
    on Feb 14, 2021

    We would like to use this feature as part of CML for running ephemeral continuous integration jobs.

  20. david-behnke commented on Feb 17, 2021

    @david-behnke
    david-behnke
    on Feb 17, 2021

    Not having a deterministic build environment is a show stopper for us 👎

  21. rclmenezes commented on Mar 26, 2021

    @rclmenezes
    rclmenezes
    on Mar 26, 2021

    Help us @bryanmacfarlane, you're our only hope! 🙏

    image

  22. korotovsky commented on Aug 20, 2021

    @korotovsky
    korotovsky
    on Aug 20, 2021

    We are experiencing this issue on plain hardware server with 10+ runners, any ideas how to fix it? No --once flag was used.

  23. aidan-mundy commented on Sep 17, 2021

    @aidan-mundy
    aidan-mundy
    on Sep 17, 2021

    Is this resolved with the merge of #660?

  24. joeyparrish commented on Sep 17, 2021

    @joeyparrish
    joeyparrish
    on Sep 17, 2021

    I believe so.

  25. pje commented on Sep 20, 2021

    @pje
    pje
    on Sep 20, 2021
    Contributor

    Closing this as #660 has been merged, --ephemeral support has been fully deployed, and the docs have been updated. 🎉

  26. pje
    closed this as completedon Sep 20, 2021
  27. jonans
    added a commit that references this issue on Oct 2, 2021

    Implement changes for emphemeral fix of actions runner. actions/runne…

    b664f96
  28. jonans
    added a commit that references this issue on Oct 27, 2021

    Ephemeral multiconf

    d407dac

  29. spangaer commented on May 11, 2022

    @spangaer
    spangaer
    on May 11, 2022

    It seems like the --once flag is still around and I'd like to ask keeping it around.

    Registering an --ephemeral runner as part of a webhook, sure has it's own right to exist.

    --once seems to cater for a different use case:

    1. Have the action user home dir and actions working dir in a tempfs.
    2. Reinitialize that tempfs with every run (matching the clean slate the cloud hosted runners produce)
      This includes a non ephemeral registration.
    3. Run above in a loop

    That doesn't work with the ephemeral system because the registration token expires. While you don't need a registration token in the other case as the registration stays valid.

    We're using this to use the same workflows across both cloud and a self hosted runners, because some builds have tie in to the enterprise network (for the time being).

  30. fkorotkov commented on Sep 8, 2022

    @fkorotkov
    fkorotkov
    on Sep 8, 2022

    @pje I'd like to confirm that there is no way to guarantee which workflow an ephemeral runner will pick up in case there are several runners are starting simultaneously for several workflow_job events. I've created a separate feature request #2106 to allow specify a workflow id for a runner to pick.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Runner ❤️ ContainerenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions