Skip to content

OSX machines are de-provisioned during CI / PR runs leading to failures #34472

New issue
New issue
Closed
Closed
OSX machines are de-provisioned during CI / PR runs leading to failures#34472
Labels
area-Infrastructureblocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'blocking-official-build
Milestone
Future

Description

@jaredpar
jaredpar
opened on Apr 2, 2020

Runfo Tracking Issue: OSX deprovision in runtime

Definition Build Kind Job Name
runtime 1288514 PR 52695 CoreCLR Product Build OSX arm64 release
runtime 1284973 PR 57116 Build iOSSimulator x86 Debug AllSubsets_Mono
runtime 1284614 PR 57076 Build MacCatalyst x64 Debug AllSubsets_Mono
runtime 1266365 PR 56406 Libraries Test Run release mono Linux x64 Debug
runtime 1266272 PR 56502 CoreCLR Product Build OSX x64 release
runtime 1266203 PR 56398 Mono Product Build OSX x64 release
runtime 1265838 PR 56537 CoreCLR Product Build OSX x64 checked
runtime 1265709 PR 56531 Mono crossaot Product Build OSX x64 release
runtime 1264483 PR 56431 Mono iOS AOT offsets
runtime 1264469 PR 56411 Build OSX x64 Release AllSubsets_Mono_LLVMJIT
runtime 1264447 PR 56490 Build iOSSimulator x86 Debug AllSubsets_Mono
runtime 1264431 PR 56486 Build tvOSSimulator x64 Debug AllSubsets_Mono
runtime 1264431 PR 56486 Build MacCatalyst arm64 Debug AllSubsets_Mono
runtime 1263889 PR 56471 CoreCLR Product Build OSX arm64 checked
runtime 1263861 PR 56411 Build MacCatalyst arm64 Debug AllSubsets_Mono
runtime 1263853 PR 56316 Build iOSSimulator x86 Debug AllSubsets_Mono
runtime 1262979 PR 56434 Installer Build and Test coreclr OSX_x64 Release
runtime 1246278 PR 55939 Build iOSSimulator x86 Debug AllSubsets_Mono
runtime 1246251 PR 55925 Mono crossaot Product Build OSX x64 release

Build Result Summary

Day Hit Count Week Hit Count Month Hit Count
0 2 18

Activity

  1. jaredpar
    added
    blocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
    on Apr 2, 2020
  2. Dotnet-GitSync-Bot
    added
    untriagedNew issue has not been triaged by the area owner
    on Apr 2, 2020

  3. Dotnet-GitSync-Bot commented on Apr 2, 2020

    @Dotnet-GitSync-Bot
    Dotnet-GitSync-Bot
    on Apr 2, 2020
    Collaborator

    I couldn't figure out the best area label to add to this issue. Please help me learn by adding exactly one area label.

  4. jaredpar
    added
    blocking-official-build
    on Apr 2, 2020
  5. jeffschwMSFT
    added
    area-Infrastructure
    on Apr 2, 2020

  6. jaredpar commented on Apr 2, 2020

    @jaredpar
    jaredpar
    on Apr 2, 2020
    MemberAuthor

    FYI this is not runtime specific. Hitting other repos as well

    Definition Build Kind Timeline Record
    runtime 585548 Rolling Mono Product Build iOS arm64 debug
    runtime 585548 Rolling Mono LLVM Product Build OSX x64 debug
    runtime 585497 PR #33701 Installer Build and Test OSX_x64 Debug
    runtime 585541 PR #34229 Mono LLVM Product Build OSX x64 debug
    coreclr 585166 Rolling CoreCLR Pri1 Test Run OSX x64 checked
    MachineLearning-CI 584884 PR dotnet/machinelearning#4925 MacOS_x64_NetCoreApp21 Release_Build

    Evaluated 500 builds
    Impacted 5 bulids
    Impacted 6 jobs

  7. dougbu commented on Apr 21, 2020

    @dougbu
    dougbu
    on Apr 21, 2020
    Contributor

    This is also tracked at dotnet/core-eng#9448. Correct?

  8. MattGal commented on Apr 21, 2020

    @MattGal
    MattGal
    on Apr 21, 2020
    Member

    This is also tracked at dotnet/core-eng#9448. Correct?

    Correct.

  9. AraHaan commented on May 19, 2020

    @AraHaan
    AraHaan
    on May 19, 2020
    Member

    aww that link 404d for me.

  10. MattGal commented on May 19, 2020

    @MattGal
    MattGal
    on May 19, 2020
    Member

    aww that link 404d for me.

    Private repo; I can publicly tell you however that the problem continues to be investigated and seems to be at the MacOS-VM-provider level.

  11. AraHaan commented on May 19, 2020

    @AraHaan
    AraHaan
    on May 19, 2020
    Member

    Seems on my pr 36694 that this might be a cause when I read this: https://helix.dot.net/api/2019-06-17/jobs/b0b1c468-c4c5-4f2d-a2a6-3c831ed6ef20/workitems/System.Net.Sockets.Tests/console

  12. MattGal commented on May 19, 2020

    @MattGal
    MattGal
    on May 19, 2020
    Member

    Seems on my pr 36694 that this might be a cause when I read this: https://helix.dot.net/api/2019-06-17/jobs/b0b1c468-c4c5-4f2d-a2a6-3c831ed6ef20/workitems/System.Net.Sockets.Tests/console

    No that log is from a plain old crash on a Helix machine. This issue is specific to hosted Azure Devops build macs, which are in no way connected.

  13. CoffeeFlux commented on May 22, 2020

    @CoffeeFlux
    CoffeeFlux
    on May 22, 2020
    Contributor

    Because of the AzDO limitation where you can't rerun individual lanes until all of them are completed, this issue is particularly painful when trying to land a PR :(

  14. ViktorHofer
    removed
    untriagedNew issue has not been triaged by the area owner
    on Jul 12, 2020
  15. ViktorHofer
    added this to the 5.0.0 milestone on Jul 12, 2020

  16. myblindy commented on Jul 13, 2020

    @myblindy
    myblindy
    on Jul 13, 2020

    Any idea why my https://dev.azure.com/dnceng/public/_build/results?buildId=726102 build was included in this? It didn't fail any tests or build steps as far as I can tell.

  17. MattGal commented on Jul 13, 2020

    @MattGal
    MattGal
    on Jul 13, 2020
    Member

    Any idea why my https://dev.azure.com/dnceng/public/_build/results?buildId=726102 build was included in this? It didn't fail any tests or build steps as far as I can tell.

    It does on Attempt #1 :
    image

    So, if you didn't click retry, some nice person did for you :)

  18. MattGal commented on Jul 13, 2020

    @MattGal
    MattGal
    on Jul 13, 2020
    Member

    Update from my end; chatted with AzDO folks in charge of the hosted macs today, the previous ETA of 7/15 to migrate the hosted agents seems to have not been accurate. I am politely asking for an updated time.

  19. ViktorHofer
    modified the milestones: 5.0.0, Future on Aug 4, 2020

  20. directhex commented on Sep 11, 2020

    @directhex
    directhex
    on Sep 11, 2020
    Contributor

    Seemingly still a thing https://dev.azure.com/dnceng/internal/_build/results?buildId=810664&view=logs&j=04b1677b-aa91-548a-e789-89c3d998d0f7

  21. MattGal commented on Sep 11, 2020

    @MattGal
    MattGal
    on Sep 11, 2020
    Member

    Seemingly still a thing https://dev.azure.com/dnceng/internal/_build/results?buildId=810664&view=logs&j=04b1677b-aa91-548a-e789-89c3d998d0f7

    Yes, definitely still a thing (and why the issue is still open). I checked this instance out, and it's definitely on the v2 pool so the migration is not something we can hope for to help us here any more. I did share this specific instance back with the MMS team though they are tracking them en masse as well via telemetry.

  22. ViktorHofer commented on Oct 1, 2020

    @ViktorHofer
    ViktorHofer
    on Oct 1, 2020
    Member

    @MattGal we are now live updating this issues top post. Maybe that helps?

  23. ViktorHofer commented on Feb 12, 2021

    @ViktorHofer
    ViktorHofer
    on Feb 12, 2021
    Member

    Got fixed by AzDO. Closing.

  24. ViktorHofer
    closed this as completedon Feb 12, 2021
  25. ghost locked as resolved and limited conversation to collaborators on May 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-Infrastructureblocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'blocking-official-build

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions