[tests] System.Text.Json.Tests segfault, for Libraries Test Run release coreclr OSX x64 Release #47805

Activity

1. ghost added

   area-System.Text.Json

   on Feb 3, 2021
2. ghost added

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Feb 3, 2021
4. Anipik commented on Feb 3, 2021

   

   Anipik

   on Feb 3, 2021

   ContributorAuthor

   More actions

   seems similar to #46100
5. Anipik commented on Feb 8, 2021

   

   Anipik

   on Feb 8, 2021

   ContributorAuthor

   More actions

   Failed again https://dev.azure.com/dnceng/public/_build/results?buildId=982059&view=logs&j=5daca27f-52cd-532c-7da3-66cd43b2fb44&t=544b6965-1404-5848-fc5a-30d8345c68ff
6. Anipik

   added

   blocking-clean-ciBlocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'

   on Feb 10, 2021
7. Anipik commented on Feb 10, 2021

   

   Anipik

   on Feb 10, 2021

   ContributorAuthor

   More actions

   cc @layomia @steveharter
8. steveharter commented on Feb 24, 2021

   

   steveharter

   on Feb 24, 2021

   Contributor

   More actions

   @ViktorHofer thoughts? I can't seem to find any crash dumps (via an "attachments" tab?) or other information.

   It may be related to #46100, but I can't find what test(s) were running. I logged this previously about the lack of information available during crash dumps; at a minimum having the active test(s) that were running at the time of the crash would be valuable.

   As a last resort, I could try running it on a local macbook repeatedly and hope it triggers the exception (with dump).
9. steveharter

   added

   needs more info

   on Feb 25, 2021
10. danmoseley commented on Feb 25, 2021

    

    danmoseley

    on Feb 25, 2021

    Contributor

    More actions

    crash dump: #46100 (comment)
11. ViktorHofer commented on Mar 15, 2021

    

    ViktorHofer

    on Mar 15, 2021

    Member

    More actions

    @steveharter did the crash dump work for you that @danmoseley shared above?
12. ViktorHofer

    removed

    needs more info

    on Mar 15, 2021
13. danmoseley commented on Mar 17, 2021

    

    danmoseley

    on Mar 17, 2021

    Last edited by danmoseley

    Contributor

    More actions

    We have some fresh dumps for this now, and thanks to recent effort by @safern, getting set up debugging them is practically a single command.

    I used this query --

    https://engsrvprod.kusto.windows.net/engineeringdata?query=H4sIAAAAAAAEAJWSXWvCMBSG7wX%2fQ8hVC127zSGD0YH7EPRibLixyxGb4zza5pTkFBX241erpa3zZncpeZPzvE%2bTAouFwhS0iMWU5q7fi6IfsVmCBTGjwiYgEjKs0DghNbEBjmxhGDOQ%2fd5JMo6FTDDKi3mKSdRNRxYWLlqC0i7KlGOwUpD9%2f0k0cs8o6tGvlnKwjOBaoMkSkjXoFiFTShuw3lsBBYxSVM5v8vWmJLeVfqsWK8ulmnuhvskbXOpqL7e0goT3uiY6EKK5MqjrlHxBRebiXFkHXytHxmtQ%2faDtuQqGT7BAg4xkJrryMbwdVgVWhEas0egYjQHb73mfZNcThsw1qGOLYHS6e1HZweZsVzrOwnfYcjgtp5crx67rriKvTvxVMUpzNBAOrq7DUaaHNydeuHDVmAeln7fI8oyYmnK%2fPpoMxCMZRyl8WPQFmUP0rt8bl0%2bw3ab87HLJJW0umFp%2ftL6%2b07zJn%2fY%2fr%2fL49kuShvYXf5f9RRQDAAA%3d&web=0

    let failed = Jobs
    //| where Source contains "dotnet/runtime"
    | where Source == "ci/public/dotnet/runtime/refs/heads/master" or Source == "ci/public/dotnet/runtime/refs/heads/main"
    // | where Properties contains "checked"
    | where tolower(QueueAlias) contains tolower("osx")
    | where Started > ago(30d)
    | project JobId, QueueAlias, Source // , Props=parse_json(Properties),
    //| where Props.DefinitionId == "686"
    | join kind=inner
    (WorkItems
    | where FriendlyName == "System.Text.Json.Tests"
    // | where QueueName contains tolower("Alpine.312.Amd64")
    | where Status == "BadExit"
    | project JobId, WorkItemId, Started, ConsoleUri) on JobId;
    Files
    | where FileName contains "how-to"
    | where WorkItemFriendlyName contains "System.Text.Json"
    | join kind=inner
    failed on WorkItemId

    JobId	JobName	WorkItemId	WorkItemName	WorkItemFriendlyName	Attempt	EventId	Uri	FileName	UploadSeconds	SizeBytes	JobId2	QueueAlias	Source	JobId1	WorkItemId1	Started	ConsoleUri
    13631952	70a9a136-d7a7-415d-89aa-62f3d1c6c159	570902239	7d86f8f1-3550-4ccb-8390-332c5a391de8	System.Text.Json.Tests	1	5463102564	https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-master-70a9a136d7a7415d89/System.Text.Json.Tests/how-to-debug-dump.md?sv=2019-07-07&se=2021-03-15T00%3A28%3A04Z&sr=c&sp=rl&sig=UbnWtUuINADGpHrkpjFLjG8xKihSz71djxadzUfXrGQ%3D	how-to-debug-dump.md	0	6500	13631952	osx.1013.amd64.open	ci/public/dotnet/runtime/refs/heads/master	13631952	570902239	2021-02-23 00:29:08.3060000	https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-master-70a9a136d7a7415d89/System.Text.Json.Tests/console.515a7b53.log?sv=2019-07-07&se=2021-03-15T00%3A28%3A04Z&sr=c&sp=rl&sig=UbnWtUuINADGpHrkpjFLjG8xKihSz71djxadzUfXrGQ%3D
    13659135	92d9e820-e407-446f-929d-2c105c128dad	572125446	41197480-6a14-479d-b1e4-28fb29e1a6f2	System.Text.Json.Tests	1	5475736431	https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-master-92d9e820e407446f92/System.Text.Json.Tests/how-to-debug-dump.md?sv=2019-07-07&se=2021-03-16T03%3A39%3A55Z&sr=c&sp=rl&sig=KbW5pm6QyreQy1FhYQ4AsO6nvBB4Qyzy1%2F0o%2BssRDnY%3D	how-to-debug-dump.md	0	6500	13659135	osx.1013.amd64.open	ci/public/dotnet/runtime/refs/heads/master	13659135	572125446	2021-02-24 03:41:02.3600000	https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-master-92d9e820e407446f92/System.Text.Json.Tests/console.8c4b7126.log?sv=2019-07-07&se=2021-03-16T03%3A39%3A55Z&sr=c&sp=rl&sig=KbW5pm6QyreQy1FhYQ4AsO6nvBB4Qyzy1%2F0o%2BssRDnY%3D
    13711303	9767a0a4-6430-499d-8f36-12da7a70b4da	574598097	ddd2029-33fd-463b-bd7d-8836a1d1a147	System.Text.Json.Tests	1	5500964830	https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-master-9767a0a46430499d8f/System.Text.Json.Tests/how-to-debug-dump.md?sv=2019-07-07&se=2021-03-18T11%3A48%3A57Z&sr=c&sp=rl&sig=EPQdAOJvD8GCdmmhXP%2B5AiuBl260SjMEae%2FDAIMZrx8%3D	how-to-debug-dump.md	0	6500	13711303	osx.1013.amd64.open	ci/public/dotnet/runtime/refs/heads/master	13711303	574598097	2021-02-26 11:49:59.9200000	https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-master-9767a0a46430499d8f/System.Text.Json.Tests/console.83da681c.log?sv=2019-07-07&se=2021-03-18T11%3A48%3A57Z&sr=c&sp=rl&sig=EPQdAOJvD8GCdmmhXP%2B5AiuBl260SjMEae%2FDAIMZrx8%3D
    13918884	a18c2113-8e70-40f3-967c-ada5df3ab122	584030863	854de7a2-23a1-4d58-8276-fc7a31b9e4a3	System.Text.Json.Tests	1	5596681342	https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-a18c21138e7040f396/System.Text.Json.Tests/how-to-debug-dump.md?sv=2019-07-07&se=2021-03-30T03%3A30%3A16Z&sr=c&sp=rl&sig=RTYiC9aOfCFarBhU5meOO1Wfu3NZuEX9RSMimXvy5L0%3D	how-to-debug-dump.md	0	6465	13918884	osx.1013.amd64.open	ci/public/dotnet/runtime/refs/heads/main	13918884	584030863	2021-03-10 03:31:04.2550000	https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-a18c21138e7040f396/System.Text.Json.Tests/console.fc7cd506.log?sv=2019-07-07&se=2021-03-30T03%3A30%3A16Z&sr=c&sp=rl&sig=RTYiC9aOfCFarBhU5meOO1Wfu3NZuEX9RSMimXvy5L0%3D

    And I clicked the link in the most recent hit, it pulls up how-to-debug-dump.md, which gives me the magic commands to paste in to get dump, symbols, and binaries pulled down, then open it in lldb, eg

    dotnet tool install --global runfo
    dotnet tool update --global runfo
    runfo get-helix-payload -j a18c2113-8e70-40f3-967c-ada5df3ab122 -w System.Text.Json.Tests -o ~/helix_payload/System.Text.Json.Tests\
    lldb --core ~/helix_payload/System.Text.Json.Tests/workitems/System.Text.Json.Tests/core.41566 ~/helix_payload/System.Text.Json.Tests/shared/Microsoft.NETCore.App/6.0.0/dotnet

    It really is super easy now. Unfortunately I don't have a Mac, so I couldn't actually do the lldb part. @steveharter could you please take a look, and tell us how this goes for you?

    cc @jkotas

    ❤️React with ❤️1Reacted by Santiago Fernandez Madero
14. danmoseley commented on Mar 17, 2021

    

    danmoseley

    on Mar 17, 2021

    Contributor

    More actions

    cc @stephentoub in case he's interested in how easy it is to get to dumps on all OS'es by just copy/pasting a couple commands out the log.

    👍React with 👍1Reacted by Stephen Toub
15. eiriktsarpalis

    assigned

    steveharter

    on Apr 12, 2021
16. 3 remaining items

    Load more
18. jkotas

    removed

    area-System.Text.Json

    on Apr 20, 2021
19. jkotas

    unassigned

    steveharter

    on Apr 20, 2021
20. Maoni0

    assigned

    PeterSolMS

    on Apr 21, 2021
21. Maoni0 commented on Apr 21, 2021

    

    Maoni0

    on Apr 21, 2021

    Member

    More actions

    we are trying to get a local repro. @PeterSolMS will be doing the investigation.
22. PeterSolMS commented on Apr 26, 2021

    

    PeterSolMS

    on Apr 26, 2021

    Contributor

    More actions

    I've been able to get a repro on Windows/amd64 by setting the bit more eagerly (ignoring whether background GC needs it or not), and checking whether BGC needs the background mark bit set where we test the bit. This makes it fail apparently because the object is right before a pinned plug, and we save the object and restore it later. At the place where we test the bit in make_free_list_in_brick it has not been restored yet, apparently.
23. Maoni0 commented on Apr 27, 2021

    

    Maoni0

    on Apr 27, 2021

    Member

    More actions

    ahh right we are not doing this for sweep like we do for compact_plug. but I think we should be able to make this case simpler for sweep by making sure we clear these bits when we recover.
24. PeterSolMS commented on Apr 28, 2021

    

    PeterSolMS

    on Apr 28, 2021

    Contributor

    More actions

    I've been able to catch 3 crash dumps so far, all of them have the bad object right in front of a pinned plug. In two cases, the size of the bad object was 32 bytes, in one case it was 24 bytes (a boxed System.Boolean). I wonder if we can simply generalize your fix for is_plug_padded, clearing the bits when we save to saved_pre_plug or saved_post_plug, but not for the copy saved to saved_pre_plug_reloc or saved_post_plug_reloc.

    👍React with 👍1Reacted by Maoni Stephens
25. Maoni0 commented on Apr 29, 2021

    

    Maoni0

    on Apr 29, 2021

    Member

    More actions

    @PeterSolMS yes I think that would work!
26. PeterSolMS commented on May 3, 2021

    

    PeterSolMS

    on May 3, 2021

    Contributor

    More actions

    Proposed fix in PR #52183. With this fix in place, the issue has repro'd in days.
27. PeterSolMS

    added a commit that references this issue on May 4, 2021

    Fix issue 47805 (#52183)

    ...

    b7457e8
28. PeterSolMS commented on May 4, 2021

    

    PeterSolMS

    on May 4, 2021

    Contributor

    More actions

    PR has been merged.
29. jkotas commented on May 4, 2021

    

    jkotas

    on May 4, 2021

    Member

    More actions

    @PeterSolMS Thank you!
30. jkotas

    closed this as completedon May 4, 2021
31. ghost locked as resolved and limited conversation to collaborators on Jun 3, 2021