NativeAOT: Silent unwind info corruption #88292

Activity

1. filipnavara

   added

   os-macosmacOS aka OSXmacOS aka OSX

   area-NativeAOT-coreclr

   on Jul 2, 2023
2. ghost added

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Jul 2, 2023
4. filipnavara commented on Jul 2, 2023

   

   filipnavara

   on Jul 2, 2023

   MemberAuthor

   More actions

   I guess we have three options how to deal with it:

   • Short-term: Bail out with error if the DWARF section is too big (> 0xffffff bytes). This is tricky because the problem is the size of the section in the linked image, not the ILCompiler output .o file.
   • Check again if we can generate compact unwind codes instead of DWARF for the common cases: [NativeAOT] Evaluate use/benefits of compact unwinding on osx-x64 and osx-arm64 #76371
   • Build a different index (eg. Linux .eh_hdr format) if the data are too big, and consume it.
5. jkotas commented on Jul 2, 2023

   

   jkotas

   on Jul 2, 2023

   Last edited by jkotas

   Member

   More actions

   Another option is to compensate for this issue in the libunwind implementation. We can loop over all candidates that match the hint.
6. filipnavara commented on Jul 2, 2023

   

   filipnavara

   on Jul 2, 2023

   MemberAuthor

   More actions

   Another option is to compensate for this issue in the libunwind implementation. We can loop over all candidates that match the hint.

   I don't think that's possible. The offset points into middle of a stream so it's essentially decoding garbage. Sometimes the garbage can make sense, sometimes not, but it's not easy to tell whether it's a false hit.
7. jkotas commented on Jul 2, 2023

   

   jkotas

   on Jul 2, 2023

   Member

   More actions

   Can we build our own hint table from the broken hint table? Something like:

   If the dwarf stream is more than 16MB:

   1. Create copy of the hint table, with 32 offsets
   2. Scan the hint table and fill in the top 8-bit offsets. We should be able to tell that where the offsets wrap around
   3. Use this private hint table for lookups

   This would fix our unwinder, but it would not fix other unwinders. For example, I would expect C++ EH to be still broken.
8. filipnavara commented on Jul 2, 2023

   

   filipnavara

   on Jul 2, 2023

   MemberAuthor

   More actions

   We can probably build our own hint table from scratch during compilation. It needs to cover only "managed code" section and hence doesn't need much of a linker input, as long as the DWARF section is preserved in one piece (I think it is).

   Reconstructing the hints at runtime from the linker output may be possible but at that point you get a penalty similar to not using the hints at all and just creating the cache by sequentially reading the DWARF section. That's incredibly slow even on small executables though.

   This would fix our unwinder, but it would not fix other unwinders. For example, I would expect C++ EH to be still broken.

   That's a fair point. I didn't consider other unwinders. If we want other unwinders to work then we basically have to either 1) generate compact unwinding codes where possible (needs codegen changes), 2) implement "compression" for DWARF by identifying common prolog sequences and sharing their code (possible in the DWARF format but difficult to implement), 3) fix it the linker output to not use the hints when they overflow (the slow lookup is too slow for NativeAOT purposes thought). I didn't check what the new Apple linker (ld-prime) produces in this case.
9. jkotas commented on Jul 2, 2023

   

   jkotas

   on Jul 2, 2023

   Member

   More actions

   Reconstructing the hints at runtime from the linker output may be possible but at that point you get a penalty similar to not using the hints

   The hint table is not big and the reconstructed hint table can be cached. I think the penalty would be fairly small.
10. filipnavara commented on Jul 2, 2023

    

    filipnavara

    on Jul 2, 2023

    MemberAuthor

    More actions

    The hint table is not big and the reconstructed hint table can be cached.

    In the executable from OP the size of __unwind_info (compact unwinding table) is 0x89beb8 bytes. The size of __eh_frame is 0x1bc0e28. So, 9Mb for the hint table may not seem like much but it's definitely going to be noticeable. If it was done lazily then you risk running it during thread hijacking on GC suspend. That would almost certainly take long enough to cause live locks when the threads get hammered with the "suspend all thread hijack" logic.
11. filipnavara commented on Jul 2, 2023

    

    filipnavara

    on Jul 2, 2023

    MemberAuthor

    More actions

    On second thought I don't think it's even reliably possible to reconstruct the DWARF offsets solely from __unwind_info since it's sorted, and hence it's not guaranteed that the DWARF offsets are in order. I suspect that in this case they would be, but it feels fragile.
12. filipnavara commented on Jul 3, 2023

    

    filipnavara

    on Jul 3, 2023

    MemberAuthor

    More actions

    There’s potentially an easy win in terms of the ARM64 DWARF size with folding the extremely common sequences into a DWARF CIE and referencing that. That’s a variation of the “DWARF compression” strategy mentioned earlier, just restricted to specific known sequences.

    For example, the prolog for frame with no callee saved registers (except LR and FP) is this:

      DW_CFA_advance_loc: 4
      DW_CFA_def_cfa_offset: +16
      DW_CFA_offset: W29 -16
      DW_CFA_offset: W30 -8
      DW_CFA_advance_loc: 4
      DW_CFA_def_cfa_register: W29
      DW_CFA_nop:
      DW_CFA_nop:
      DW_CFA_nop:
      DW_CFA_nop:
      DW_CFA_nop:
    

    The code looks like this:

    stp x29, x30, [sp, #-10]!
    mov x29, sp
    …
    ldr x29, x30, [sp], #0x10
    ret
    

    This repeats 20000+ times in the OP executable. Besides being foldable in the DWARF codes it’s also likely expressible as compact unwind code with no codegen changes. We would still need to implement special prolog treatment for asynchronous unwinding with the compact unwind codes though, so the DWARF way could be easier (and benefit other platforms too).
13. filipnavara commented on Jul 11, 2023

    

    filipnavara

    on Jul 11, 2023

    Last edited by filipnavara

    MemberAuthor

    More actions

    I tried to replace the empty frame DWARF sequence with compact unwinding and it saves 32% of the DWARF section size for this particular executable. Similar savings are present for empty iOS app from template (dotnet new ios). It's not enough to push the DWARF size below the problematic size but it's significant enough that it may be an option worth exploring.

    👍React with 👍1Reacted by Ivan Povazan
14. agocke

    added this to AppModelon Aug 9, 2023
15. agocke

    added this to the 9.0.0 milestone on Aug 9, 2023
16. ghost removed

    untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

    on Aug 9, 2023
17. agocke

    modified the milestones: 9.0.0, 10.0.0 on Aug 14, 2024
18. agocke

    modified the milestones: 10.0.0, 11.0.0 on Jul 29, 2025