HttpListener on Unix: GetContextAsync()/BeginGetContext() racing with Close() never completes #136293

Description

On Linux and macOS (the managed HttpListener implementation), a call to GetContextAsync() / BeginGetContext() that races with Close() (or Stop() / Abort()) can be left pending forever: it is never completed with a context or an exception, so anything awaiting it hangs.

This shows up in a common server-loop pattern, where a background loop calls GetContextAsync() again right after handling a request while another thread shuts the listener down and waits for the loop to finish:

while (true)
{
    try
    {
        var context = await listener.GetContextAsync(); // never completes if it races Close()
        Handle(context);
    }
    catch (Exception) when (disposing)
    {
        return;
    }
}

// elsewhere
disposing = true;
listener.Close();
loopTask.Wait(); // hangs forever

We hit this as an intermittent 10-minute CI hang in the OpenTelemetry .NET test suite on ubuntu / .NET 9.0.21 (job log). The hang dump showed the listener with _state == State.Closed and _asyncWaitQueue still holding one ListenerAsyncResult with _completed == false, which the awaiting loop was blocked on.

Reproduction Steps

HttpListenerCloseRace.csproj:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFrameworks>net9.0;net10.0</TargetFrameworks>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

</Project>

Program.cs:

using System.Collections.Concurrent;
using System.Net;
using System.Net.Sockets;

const int Iterations = 20;
const int Callers = 4;

int orphaned = 0;

for (int i = 0; i < Iterations; i++)
{
    using var listener = new HttpListener();
    listener.Prefixes.Add($"http://localhost:{GetFreePort()}/");
    listener.Start();

    var results = new ConcurrentBag<IAsyncResult>();
    using var started = new CountdownEvent(Callers);

    var callers = Enumerable.Range(0, Callers).Select(_ => new Thread(() =>
    {
        started.Signal();

        while (true)
        {
            try
            {
                results.Add(listener.BeginGetContext(null, null));
            }
            catch (Exception)
            {
                // ObjectDisposedException/InvalidOperationException once the listener is closed
                return;
            }
        }
    })).ToArray();

    foreach (var caller in callers)
    {
        caller.Start();
    }

    started.Wait();
    Thread.Sleep(10);

    listener.Close();

    foreach (var caller in callers)
    {
        caller.Join();
    }

    // Every BeginGetContext() that returned should have been completed by Close().
    Thread.Sleep(1000);
    var pending = results.Count(r => !r.IsCompleted);

    if (pending > 0)
    {
        orphaned++;
        Console.WriteLine($"Iteration {i}: {pending} of {results.Count} BeginGetContext() calls never completed after Close().");
    }
}

Console.WriteLine($"{orphaned} of {Iterations} iterations left pending GetContext operations.");
return orphaned == 0 ? 0 : 1;

static int GetFreePort()
{
    using var socket = new TcpListener(IPAddress.Loopback, 0);
    socket.Start();
    return ((IPEndPoint)socket.LocalEndpoint).Port;
}

Run on Linux:

dotnet run -c Release -f net10.0

The repro keeps several threads calling BeginGetContext() while Close() runs. This makes the race happen every time; in real code it is rare but happens with a single caller (see above).

Expected behavior

Every BeginGetContext() / GetContextAsync() call that returns successfully is eventually completed. Once the listener is closed, the call completes with an exception (as it does for operations that were already pending when Close() started, and as it does on Windows):

0 of 20 iterations left pending GetContext operations.

Actual behavior

On Linux, every iteration leaves operations that never complete (usually one per caller thread):

Iteration 0: 4 of 102307 BeginGetContext() calls never completed after Close().
Iteration 1: 4 of 15150 BeginGetContext() calls never completed after Close().
Iteration 2: 4 of 39004 BeginGetContext() calls never completed after Close().
...
Iteration 9: 3 of 18925 BeginGetContext() calls never completed after Close().
...
Iteration 19: 4 of 26181 BeginGetContext() calls never completed after Close().
20 of 20 iterations left pending GetContext operations.

The same program on Windows (http.sys implementation) prints 0 of 20 iterations left pending GetContext operations.

Regression?

Not as far as I know. The relevant code in HttpListener.Managed.cs hasn't changed in years and is identical on release/9.0 and main.

Known Workarounds

Callers can serialize their own GetContextAsync() calls with shutdown, so that a new call is never started while Close() is running:

// loop
lock (syncRoot)
{
    if (disposing)
    {
        return;
    }

    ctxTask = listener.GetContextAsync(); // BeginGetContext runs synchronously, so the operation is queued before the lock is released
}

// shutdown
lock (syncRoot)
{
    disposing = true;
}

listener.Close();

Alternatively, don't wait unboundedly for a pending GetContextAsync() after closing the listener (for example WaitAsync(cancellationToken) cancelled on shutdown).

Configuration

• .NET 9.0.4 and 10.0.10 (repro), .NET 9.0.21 (original CI hang)
• Ubuntu on WSL2 (kernel 6.18, x64) for the repro; GitHub Actions ubuntu-26.04 x64 for the CI hang
• Specific to the managed HttpListener used on non-Windows platforms; Windows (http.sys) is not affected.

Other information

BeginGetContext() checks the listener state without taking _internalLock, and only then takes the _asyncWaitQueue lock to queue the operation (L273-L299):

public IAsyncResult BeginGetContext(AsyncCallback? callback, object? state)
{
    CheckDisposed();
    if (_state != State.Started) { throw ...; }

    ListenerAsyncResult ares = new ListenerAsyncResult(this, callback, state);

    lock ((_asyncWaitQueue as ICollection).SyncRoot)
    {
        ...
        _asyncWaitQueue.Add(ares);
    }

    return ares;
}

Close() / Dispose() (and Stop() / Abort()) take _internalLock, call Cleanup(), which completes and clears _asyncWaitQueue (L245-L253), and only update _state afterwards in a finally (L125-L149).

So the following interleaving leaves an operation in the queue that nothing will ever complete:

1. Thread A: BeginGetContext() sees _state == State.Started and passes the checks.
2. Thread B: Close() → Cleanup() takes the _asyncWaitQueue lock, completes the queued operations with ObjectDisposedException and clears the queue.
3. Thread A: takes the _asyncWaitQueue lock and adds its ListenerAsyncResult.
4. Thread B: sets _state = State.Closed.

Re-checking _state inside the _asyncWaitQueue lock in BeginGetContext() wouldn't be enough on its own, because _state is still Started at step 3. One possible fix is to record, under the _asyncWaitQueue lock in Cleanup(), that the queue has been drained (or move the state change before Cleanup()). BeginGetContext() would then check that flag under the same lock and throw ObjectDisposedException (or complete the result with it) instead of queuing.