Tracking issue: 32bit x86 targets lose float NaN payload in return values

On x86 (32bit) targets, returning a floating-point number from a "C" ABI function by-value can change the bits of their NaN payloads. (Specifically, the quiet bit gets set.) This is caused by using x87 registers to pass return values, as mandated by the ABI for this target; this requires a format conversion, and that conversion quiets NaNs. For targets that require SSE2, we have patched the "Rust" ABI to avoid this problem; x86-32 targets where SSE2 is optional have the same problem for "Rust" functions as well.

This is a known and long-standing problem, and very hard to fix. The purpose of this issue mostly is to document its existence and to give it a URL that can be referenced.

A proper fix would require patching LLVM to use different code to load float return values into an x87 register -- specifically, it has to be done in a way that NaN payloads are fully preserved. It would be lovely if we could find someone who both cares enough about x86-32 to make sure it correctly implements the Rust semantics and who has the expertise to write the LLVM patch this requires. :)

Related issues:

x86-32: floating-point return values undergo implicit format conversion llvm/llvm-project#66803: the same bug on the LLVM side.
Tracking issue: 32bit x86 targets without SSE2 have unsound floating point behavior #114479: that's a different problem, also affecting x86-32 but concerning the actual FP computations, not the NaN bits of return values.

Prior issues:

It looks like the 80-bit float load/store operations don't modify the bits, so in principle one could fix this by performing the conversion from 32-bit or 64-bit float to 80-bit float (and back) in software, although I expect that would be prohibitively expensive.

Could we do a quick check whether the float is a NaN or not, and only do the software conversion for NaNs? That should be a lot faster I hope.

Came up with this code for software conversion:

typedef union {
    long double ld;
    struct {
        unsigned long long ull;
        unsigned short us;
    };
} ld_t;

typedef union {
    double d;
    unsigned long long ull;
} d_t;

long double identity(double x) {
    d_t y;
    y.d = x;
    ld_t z;
    z.ull = (y.ull << 11) & 0x7FFFFFFFFFFFFFFFull | 0x8000000000000000ull;
    unsigned short e = (y.ull >> 52) & 0x7FF;
    z.us = ((e == 0x7FF) ? 0x7FFF : e + 0x3C00) | ((y.ull >> 48) & 0x8000);
    return z.ld;
}

identity:
        sub     esp, 28
        movq    xmm1, QWORD PTR [esp+32]
        movdqa  xmm0, xmm1
        psllq   xmm1, 11
        psrlq   xmm0, 32
        movd    eax, xmm0
        movdqa  xmm0, XMMWORD PTR .LC0
        mov     edx, eax
        shr     edx, 20
        por     xmm1, xmm0
        and     dx, 2047
        movq    QWORD PTR [esp], xmm1
        cmp     dx, 2047
        lea     ecx, [edx+15360]
        mov     edx, 32767
        cmovne  edx, ecx
        shr     eax, 16
        and     ax, -32768
        or      eax, edx
        mov     WORD PTR [esp+8], ax
        fld     TBYTE PTR [esp]
        add     esp, 28
        ret
.LC0:
        .long   0
        .long   -2147483648
        .long   0
        .long   0

Or, special casing NaNs:

#include "math.h"

typedef union {
    long double ld;
    struct {
        unsigned long long ull;
        unsigned short us;
    };
} ld_t;

typedef union {
    double d;
    unsigned long long ull;
} d_t;

long double identity(double x) {
    if (!isnan(x)) {
        return x;
    }
    d_t y;
    y.d = x;
    ld_t z;
    z.ull = (y.ull << 11) & 0x7FFFFFFFFFFFFFFFull | 0x8000000000000000ull;
    z.us = (y.ull >> 48) | 0x7FFF;
    return z.ld;
}

identity:
        sub     esp, 44
        fld     QWORD PTR [esp+48]
        fst     QWORD PTR [esp+8]
        fucomi  st, st(0)
        jp      .L2
        add     esp, 44
        ret
.L2:
        fstp    st(0)
        mov     eax, DWORD PTR [esp+8]
        mov     edx, DWORD PTR [esp+12]
        movd    xmm0, eax
        movd    xmm1, edx
        mov     eax, edx
        punpckldq       xmm0, xmm1
        shr     eax, 16
        movdqa  xmm1, XMMWORD PTR .LC0
        psllq   xmm0, 11
        or      ax, 32767
        por     xmm0, xmm1
        mov     WORD PTR [esp+24], ax
        movq    QWORD PTR [esp+16], xmm0
        fld     TBYTE PTR [esp+16]
        add     esp, 44
        ret
.LC0:
        .long   0
        .long   -2147483648
        .long   0
        .long   0```

Nice. :)

So, I wonder -- what would it take to actually use this when returning floats or doubles? Do we have to patch LLVM or can the codegen backend do this?

Btw, can someone explain why this only affects return values? How are float arguments passed?

So, I wonder -- what would it take to actually use this when returning floats or doubles? Do we have to patch LLVM or can the codegen backend do this?

No idea :)

Btw, can someone explain why this only affects return values? How are float arguments passed?

Arguments on x86 (with the C calling convention) are always passed on the stack regardless of type. So the problem still exists as soon as you fld the value, but it's not inherent to the calling convention.

By the same token, for non-extern "C" functions, floating point returns could be changed to go on the stack, even without SSE.

Here there is a claim that this issue only affects signaling NaN return values. Is that true? The example given here does seem to be a signaling NaN.

For reference: the only change made to f32/f64 NaN payloads by x87 load/store is to quieten signalling NaN payloads by setting the quiet bit. Additionally, only f32 and f64 are affected by this issue as neither f16 nor f128 are passed in x87 registers.

Auto merge of rust-lang#123351 - beetrees:x86-ret-snan-rust, r=nikic

Auto merge of rust-lang#123351 - beetrees:x86-ret-snan-rust, r=nikic,…

Until a reasonable fix is chosen and incorporated, shouldn't there be a priority effort to patch these target's generated code so when this specific float value corruption occurs at run-time, either a panic is generated or a run-time error message logged? Or maybe this run-time safety check patch should be added via a feature or config option, so developers can choose to disable it if they aren't concerned and don't want the run-time overhead of the check. Later this all will be removed when the proper fix is incorporated. Otherwise, unless I'm misinterpreting, it sounds as if broken (unsound?) code is currently allowed 'in the wild' (even if in rare circumstances)? Or is this so rare that no-one who it impacts is concerned? If this is the case, then I hope that the relevant language documentation and relevant crate documentation (not just the target Tier documentation) prominently highlights this issue so it's not easily overlooked by anyone developing new code for these targets (granted that new code for these targets is probably itself a rare occurrence). Let me add that I'm very appreciative of the great work the community is doing - I'm very positive about Rust.

Otherwise, unless I'm misinterpreting, it sounds as if broken (unsound?) code is currently allowed 'in the wild' (even if in rare circumstances)? Or is this so rare that no-one who it impacts is concerned?

Yes, this is a miscompilation bug. It looks like it just doesn't affect enough people to motivate anyone to fix it. x86-32 is a fairly niche target these days, and signaling NaNs are quite rare as well, so very, very few people care about the combination of those. Combine that with the fact that very few people have the expertise to fix a problem like this, and those people typically have higher-impact issues to worry about, and you get a perfect recipe for problems like this flying under the radar.

If this is the case, then I hope that the relevant language documentation and relevant crate documentation (not just the target Tier documentation)

This is a bug in how a target implements the language, so the target docs seem like the primary place to mention issues like this. I'm not sure where in the language docs it'd make sense to put this -- I guess we could have a section on this page summarizing target-specific floating point issues?

FWIW this is indeed mentioned in the target documentation but I wonder if it's visible enough or - as you point out - should be mentioned elsewhere as well.

Navigation Menu

Repository navigation

Tracking issue: 32bit x86 targets lose float NaN payload in return values #115567

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Tracking issue: 32bit x86 targets lose float NaN payload in return values #115567

Description

Activity

Diggsey commented on Oct 14, 2023

RalfJung commented on Oct 14, 2023

Diggsey commented on Oct 15, 2023

RalfJung commented on Oct 15, 2023

RalfJung commented on Oct 16, 2023

Diggsey commented on Oct 16, 2023

comex commented on Oct 19, 2023

RalfJung commented on Dec 14, 2023

beetrees commented on Jun 21, 2024

jerrywrice commented on Feb 27, 2025

RalfJung commented on Mar 4, 2026

apiraino commented on Mar 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions