[ssegura]

Related: None

What you can do now that you could not before

You can test an API that answers in HTTP/1.0 and closes the connection after each reply, and get a
reply to every request. Python's own http.server works this way unless told otherwise, and many
small APIs under development run on it. Until now every other request went out on a connection the
API had already closed and was counted under no reply: half of the run's budget was lost, and the
report blamed the API.

Try it yourself

This needs python3 as well as Java and git, for the API: Python's own file server, which answers in
HTTP/1.0.

git fetch && git switch fix/http-1-0-closed-connections
./mvnw -q package -DskipTests
mkdir -p /tmp/http10 && echo hello > /tmp/http10/hello.txt
python3 -m http.server 8010 --bind 127.0.0.1 --directory /tmp/http10 2>/dev/null &
printf 'openapi: 3.0.3\ninfo: {title: Files, version: "1"}\npaths:\n  /hello.txt:\n    get:\n      responses:\n        "200":\n          description: the file\n          content:\n            text/plain:\n              schema: {type: string}\n' > /tmp/http10.yaml
./restest run /tmp/http10.yaml --url http://127.0.0.1:8010 --budget 10s --set engine.maxConcurrency=4
kill %1

Expected, on this branch (Python 3.9.6, macOS):
  50994 requests to 1 operations in 10.0s, 3% of it idle
    opening lap: 1 request in 65ms, 1 of 1 operations answered 2xx
    50994 2xx                                      <- every request answered

On master, the same commands print:
  91310 requests to 1 operations in 10.0s, 5% of it idle
    opening lap: 1 request in 64ms, 1 of 1 operations answered 2xx
    45655 2xx, 45655 no reply                      <- every other one lost

engine.maxConcurrency=4 keeps the run within what Python's server takes at once: it queues at most
five connections waiting to be accepted and refuses the rest (see "Open questions").

The engine's own test shows it without Python:

./mvnw -q verify -pl restest-exec -am -Dtest=ConnectionsTheApiClosesTest -Dsurefire.failIfNoSpecifiedTests=false

How it works

A connection is the line between RESTest and the API. RESTest keeps the line open after a reply, so
that the next request need not open a new one. A server that answers in HTTP/1.0 hangs up after every
reply unless the reply asks to keep the line, but the HTTP library RESTest uses only listens for the
HTTP/1.1 way of saying it will hang up (Connection: close). So it kept the dead line and sent the
next request down it: nobody heard it, and it came back with no reply. The request after that opened
a new line and was answered: exactly half of all requests were lost, every other one.

Now, when a reply arrives in HTTP/1.0 without asking to keep the line, RESTest marks that line as one
it will send nothing more on. Before reusing a line, the library asks whether more can be sent on it;
for a marked line the answer is no, so it lets the line go and opens a new one. A marked line is
ended outright rather than hung up politely: a polite hang-up leaves whoever hung up first holding
the line's local address for half a minute or more, and against such an API RESTest opens a line for
every request, thousands a second, so it would soon have no address left to open the next one. By
then the reply has been read whole, and the API has said it has nothing more to say on that line.

No request is sent twice. Sending a request again after its line failed stays off, as before: the
request may have reached the API, and a POST sent twice is a second creation.

What changed in the code

• The cause. OkHttp 5.5.0 stops reusing a connection only when the request or the reply carries
  Connection: close (CallServerInterceptor); nothing in it treats an HTTP/1.0 reply as closing.
  Its check before reusing a kept connection (RealConnection.isHealthy) asks whether the socket is
  closed or shut down, and probes it for the server's hang-up only for a request other than GET on a
  connection idle for ten seconds or more. With retryOnConnectionFailure(false), every request on a
  closed connection fails, with unexpected end of stream or Connection reset.
• ConnectionsTheApiCloses (new, restest-exec): a network interceptor that, when a reply is
  HTTP/1.0 without Connection: keep-alive, marks the connection's socket as soon as the reply's
  headers are in. The socket is an ApiSocket, a java.net.Socket whose isOutputShutdown() also
  answers true once marked and which is set to SO_LINGER 0 then, made by a SocketFactory the
  engine hands to OkHttp. Over TLS, OkHttp asks the TLS socket laid over it, so HTTPS is not covered.
• OkHttpEngine: the socket factory and the interceptor, with one comment.
• ConnectionsTheApiClosesTest (new): a small HTTP/1.0 server on a ServerSocket (the JDK's
  HttpServer cannot answer in HTTP/1.0) that closes after each reply as Python's does, keeps the
  connection with Connection: Keep-Alive, or leaves the connection for the client to end and notes
  how it did. GETs one after another, POSTs (each must reach the API once), 80 requests 8 at a time,
  a kept connection still reused, a marked connection ended with a reset and not a polite hang-up,
  and the rule itself over eight replies.
• docs/known-limitations.md: over HTTPS, an API that answers in HTTP/1.0 still loses every
  other request.

Evidence

• Tests: 13 added (five tests, one of them over eight replies), all green locally; CI runs on this pull request.
  Before the fix, three fail: 10 of 20 GETs one after another answered, alternating; 10 of 20 POSTs;
  41 of 80 sent 8 at a time. The test on how a marked connection ends fails, with
  ["hung up", "hung up"], both without SO_LINGER 0 and with the half-close of the first version
  (see "Decisions taken").

• Architecture tests: unchanged — none weakened. ./mvnw verify on macOS, JDK 25: BUILD SUCCESS,
  4962 tests, none failed or skipped.

• Mutation score: not applicable, neither the oracles nor the constraints were touched.

• Per-request overhead: not measured separately; one header read per HTTP/1.0 reply, nothing for
  HTTP/1.1 or HTTP/2. The HTTP/1.1 control below sends as much as master.

• Smoke run, two containerised APIs: ./mvnw verify -Psmoke green, here and on master. Both
  answer in HTTP/1.1, so nothing changes: run by hand with the smoke test's seed and budget against
  a fresh container each time, operations covered 19 and 49 (previous 19 and 49), operations with
  faults 17 and 30 (previous 17 and 30), kinds of fault 3 and 3 (previous 3 and 3), no no reply
  in either.

• Idle time: against an API on the same machine that answers in about a millisecond, RESTest
  is the slow side, and its idle share is its own pace: 66% against the HTTP/1.1 control, on master
  and here alike. Against HTTP/1.0 master was less idle, 37–42%, because half its requests failed
  within a third of a millisecond; here 48–52%.

• Artefact: runs of ./restest run against a three-operation items API on Python 3.9.6's
  ThreadingHTTPServer, macOS, 10 s each, 16 requests in flight at most, with a minute between runs:

  Server	master 33846376	this branch
  HTTP/1.0, as it comes (queue of 5)	27371 requests: 9618 2xx, 4068 4xx, 13685 no reply (= 9618 + 4068 − 1); limiter down to 1 in flight	17739 requests: 10325 2xx, 4051 4xx, 3363 no reply, every one a connection refused or reset as it opened
  HTTP/1.0, queue of 128	27061 requests: 9583 2xx, 3948 4xx, 13530 no reply (= 9583 + 3948 − 1); limiter down to 2	13424 requests: 9808 2xx, 3616 4xx, 0 no reply; 16 in flight
  HTTP/1.1, queue of 128 (control)	14351 requests: 10447 2xx, 3904 4xx, 0 no reply; 67% idle	13586 requests: 9865 2xx, 3721 4xx, 0 no reply; 66% idle

  Over 45 s against the queue-of-128 server: 31511 requests, 22743 2xx, 8768 4xx, 0 no reply, and no
  connection waiting after close (TIME_WAIT) on either side at 20 s. Python's server as it comes
  refuses ab -n 10000 -c 16 too (apr_socket_connect()); with a queue of 128 ab gets 10000 of
  10000, at 9540 a second.

Decisions taken

• Retries stay off. OkHttp's retryOnConnectionFailure(true) would also have hidden this, but it
  sends again a request whose connection failed after the request was written, which may have reached
  the API. The engine's comment on it records why it is off (a retry is a request nobody asked for,
  and its timing would describe another attempt); that reason holds, so it is kept.
• Mark the connection; do not half-close it. The first version called shutdownOutput() as soon
  as an HTTP/1.0 reply's headers arrived. It ended the alternation, but made RESTest the side that
  closes first, so each connection then held one of RESTest's local ports for 30 s (TIME_WAIT on
  macOS): a 45 s run against the queue-of-128 server had 13636 of them at 20 s and lost 13389 requests
  to BindException: Can't assign requested address.
• End a marked connection with a reset (SO_LINGER 0). Marking alone, with OkHttp closing the
  connection politely when it lets it go, still left RESTest closing first most of the time: OkHttp
  lets it go when the next request looks for a connection, usually before Python's own hang-up has
  arrived. Against python -m http.server with 4 in flight, 16346 ports were waiting at 9 s and the
  run lost 3313 requests to BindException from second 5 on. With the reset, the same run had none
  waiting on either side and every request answered. A reset discards nothing: OkHttp only lets go of
  a connection whose reply has been read whole, and the API had said it would close it. Master
  avoided the waiting by accident, because the request it sent on the closed connection drew a reset
  from the API. The test pins the reset; it cannot pin TIME_WAIT itself portably.
• A subclass of java.net.Socket, with no internal OkHttp API. OkHttp has no public way to say
  "do not reuse this connection" once a reply has arrived, and where it looks for Connection: close
  the reply's headers can no longer be changed. isOutputShutdown() is asked by OkHttp's reuse check
  and by Okio when it closes its side of a socket, where it then skips a half-close nothing needs. The
  fix relies on OkHttp asking that question before it reuses a connection; an upgrade that stopped
  asking would fail the tests.
• HTTPS is not covered, and says so. For a TLS connection OkHttp asks the TLS socket laid over the
  plain one, whose answer only changes when its sending side is closed, which sends a TLS
  close_notify; a TLS 1.2 server may answer that by dropping the rest of the reply. An HTTP/1.0 API
  behind HTTPS, with a certificate the JVM trusts, seemed rare enough to leave as a known limitation
  rather than take that risk.
• No switch. ADR-0025 asks for a switch on every behavioural lever, a behaviour a run can do
  without. This is HTTP/1.0 obeyed, and no experiment would want half its requests lost.
• No ADR. One class in restest-exec, easy to take out.

Open questions

• Against Python's http.server as it comes, 16 requests in flight still make it refuse some
  connections: about a fifth of a 10 s run's requests, counted under no reply with no fault. It is
  the API's doing (ab -c 16 fails the same way), but a request whose connection could not even be
  opened never reached the API, and the report does not say so. Worth a follow-up: say it in the
  report or the troubleshooting chapter, or send again a request whose connection could not be
  opened, which provably sent nothing?

---

• Targets master
• English throughout, including comments and test names
• The hard rules in AGENTS.md hold
• Nothing from "Out of scope" in docs/DESIGN.md without explicit approval

🤖 Generated with Claude Code