Cached EC curves leak when the Linux module is unloaded after composite key allocation #63

Summary

On Linux, allocating and freeing composite ML-DSA keys initializes cached EC curves that are not released when libsymcrypt.so is subsequently unloaded with dlclose.

A standalone reproducer using only SymCrypt's public APIs leaks 10,240 bytes in two allocations per load/unload cycle after using the P-256 and P-384 variants. Three cycles leak 30,720 bytes in six allocations. No OpenSSL or SCOSSL dependency is required, and key generation/signing is not necessary to reproduce it.

All caller-owned composite keys are freed before unloading the module. This report concerns module-lifetime resource cleanup, not a cryptographic correctness finding.

Environment

• Public main commit: 286762b7730e2b780678f5ab11fef2b1bad639e0 (SymCrypt 103.13.0).
• Ubuntu 24.04.4 LTS, x86-64, running under WSL.
• GCC 13.3.0.
• Normal RelWithDebInfo Linux generic shared-module build.
• SYMCRYPT_FIPS_BUILD=ON and SYMCRYPT_FIPS_POSTPROCESS=ON; no self-tests or integrity checks disabled.
• Only the reproducer is ASan-instrumented. LeakSanitizer intercepts the shared library's allocations; the SymCrypt library itself does not need sanitizer instrumentation.

Reproduction

Build the pinned revision using the normal repository prerequisites, including scripts/requirements.txt:

git checkout 286762b7730e2b780678f5ab11fef2b1bad639e0
cmake -S . -B build-unload -DCMAKE_BUILD_TYPE=RelWithDebInfo -DSYMCRYPT_FIPS_BUILD=ON -DSYMCRYPT_FIPS_POSTPROCESS=ON
cmake --build build-unload -j4

Save the following as symcrypt-unload-repro.c in the repository root:

#include <dlfcn.h>
#include <stdio.h>
#include <string.h>
#include "symcrypt.h"

typedef VOID (SYMCRYPT_CALL *init_fn)(UINT32, UINT32);
typedef PSYMCRYPT_COMPOSITE_MLDSAKEY (SYMCRYPT_CALL *allocate_fn)(
    SYMCRYPT_COMPOSITE_MLDSA_PARAMS);
typedef VOID (SYMCRYPT_CALL *free_fn)(PSYMCRYPT_COMPOSITE_MLDSAKEY);

int main(int argc, char **argv)
{
    if (argc != 3 || (strcmp(argv[2], "allocate") &&
        strcmp(argv[2], "repeat") && strcmp(argv[2], "load-only") &&
        strcmp(argv[2], "keep-loaded")))
    {
        fprintf(stderr, "Usage: %s LIBRARY allocate|repeat|load-only|keep-loaded\n", argv[0]);
        return 2;
    }
    int cycles = strcmp(argv[2], "repeat") == 0 ? 3 : 1;
    for (int i = 0; i < cycles; ++i)
    {
        void *module = dlopen(argv[1], RTLD_NOW | RTLD_LOCAL);
        if (module == NULL)
        {
            fprintf(stderr, "dlopen: %s\n", dlerror());
            return 2;
        }
        init_fn init = (init_fn)dlsym(module, "SymCryptModuleInit");
        allocate_fn allocate = (allocate_fn)dlsym(module, "SymCryptCompositeMlDsakeyAllocate");
        free_fn release = (free_fn)dlsym(module, "SymCryptCompositeMlDsakeyFree");
        if (init == NULL || allocate == NULL || release == NULL)
        {
            fprintf(stderr, "Required exported function missing\n");
            dlclose(module);
            return 2;
        }
        init(103, 12);
        if (strcmp(argv[2], "load-only") != 0)
        {
            PSYMCRYPT_COMPOSITE_MLDSAKEY a = allocate(
                SYMCRYPT_COMPOSITE_MLDSA_PARAMS_MLDSA44_ECDSA_P256_SHA256);
            PSYMCRYPT_COMPOSITE_MLDSAKEY b = allocate(
                SYMCRYPT_COMPOSITE_MLDSA_PARAMS_MLDSA65_ECDSA_P384_SHA512);
            int ok = a != NULL && b != NULL;
            if (a != NULL)
                release(a);
            if (b != NULL)
                release(b);
            if (!ok)
            {
                fprintf(stderr, "Key allocation failed\n");
                dlclose(module);
                return 2;
            }
        }
        if (strcmp(argv[2], "keep-loaded") != 0 && dlclose(module) != 0)
        {
            fprintf(stderr, "dlclose: %s\n", dlerror());
            return 2;
        }
        printf("Completed cycle %d (%s)\n", i + 1, argv[2]);
    }
    return 0;
}

Compile without linking SymCrypt directly, so that dlclose can unload it:

gcc -std=c11 -Wall -Wextra -Werror -Wno-unknown-pragmas -g -O0 -fsanitize=address -fno-omit-frame-pointer -Iinc symcrypt-unload-repro.c -ldl -o symcrypt-unload-repro

Run each mode in a separate process, without preloading SymCrypt:

ASAN_OPTIONS=detect_leaks=1:fast_unwind_on_malloc=0 ./symcrypt-unload-repro ./build-unload/module/generic/libsymcrypt.so load-only
ASAN_OPTIONS=detect_leaks=1:fast_unwind_on_malloc=0 ./symcrypt-unload-repro ./build-unload/module/generic/libsymcrypt.so allocate
ASAN_OPTIONS=detect_leaks=1:fast_unwind_on_malloc=0 ./symcrypt-unload-repro ./build-unload/module/generic/libsymcrypt.so repeat
ASAN_OPTIONS=detect_leaks=1:fast_unwind_on_malloc=0 ./symcrypt-unload-repro ./build-unload/module/generic/libsymcrypt.so keep-loaded

Observed results

The allocate report contains two direct leaks of 5,120 bytes each, with allocation stacks through aligned_alloc and the two calls to SymCryptCompositeMlDsakeyAllocate:

SUMMARY: AddressSanitizer: 10240 byte(s) leaked in 2 allocation(s).

The repeated-unload case reports:

SUMMARY: AddressSanitizer: 30720 byte(s) leaked in 6 allocation(s).

SymCrypt stack frames appear as <unknown module> after dlclose. Keeping the module resident leaves the caches reachable and avoids the report; that is a lifetime workaround, not an unload-cleanup fix.

Relevant source

• SymCryptCompositeMlDsakeyAllocate obtains its curve through SymCryptGetCachedEcurve.
• lib/ec_internal_curves.c lazily allocates curves and stores them in static rgpCachedCurves. It frees a losing allocation during concurrent initialization, but has no cache teardown.
• SymCryptModuleDestructor calls SymCryptRngUninit() but does not release the cached curves.

Expected behavior

Once all caller-owned keys have been freed, unloading the module should release module-owned curve caches, rather than accumulating allocations across repeated loads.

If unloading after composite use is intentionally unsupported, please clarify/document that lifetime requirement. Otherwise, a module-owned teardown path appears appropriate; callers should not free internal shared curve pointers.