[tonygermano]

Replaces #501, which Mitch handed off. This is his commit with the review changes from #501 squashed in. He's still the author, and I'm listed as co-author.

Problem

TCP connections with keepalive enabled wait for the OS idle time before sending the first probe, typically 2 hours. Common NAT devices and load balancers drop idle connections much sooner (4 minutes for an Azure load balancer, 350 seconds for an AWS NAT gateway), and they do it silently. The next read on such a connection fails with errors like the Connection timed out reported in #495.

Change

TCP Listener and TCP Sender sockets can now get a shorter idle time wherever they already enable keepalive: when "Keep Connection Open" is on, and always for TCP Sender server mode. The new oie.tcp.keepalive.idle system property sets the idle time in seconds and defaults to 120. It's documented in conf/custom.vmoptions and in the "Keep Connection Open" tooltips on both connectors.

When the TCP plugin starts with the server, it reads the OS idle time (net.ipv4.tcp_keepalive_time on Linux) from a new socket and decides once how keepalive sockets are handled:

• OS value higher than the property: each keepalive socket gets TCP_KEEPIDLE set to the property value.
• OS value equal or lower: sockets are left alone and keep the OS value, so an admin who already lowered it keeps their setting.
• OS value can't be read: sockets get the property value.
• Property 0 or less, or TCP_KEEPIDLE not supported (such as Windows on a JDK without JDK-8308593): sockets are left alone.

The decision isn't revisited until the engine restarts. If the OS value changes while the engine is running, sockets that get the property value keep getting it, even if the OS value drops below it. Sockets that are left alone use the OS value, so they follow the change, including a raise above the property value.

Logging:

• An invalid property value logs an ERROR and falls back to 120.
• If the OS rejects the value when it's set on a socket, the first failure logs an ERROR and later ones log at DEBUG.
• The decision is logged once at INFO, with the OS value when it can be read and what a later OS change does. Each message names the TCP connectors, so it isn't read as a server-wide setting.

All of this is logged at server startup, before any TCP connection, because TcpServicePlugin.start() loads SocketUtil.

The failures are ERROR rather than WARN because the default log4j2.properties only shows ERROR for this logger, and either one means the configured idle time isn't in effect.

Keepalive is still disabled explicitly when "Keep Connection Open" is off.

This is a system property rather than a mirth.properties setting because the TCP connector is a plugin. No plugin has its own key in mirth.properties, and reading one would mean adding TCP-specific code to ConfigurationController. WebServiceDispatcher reads http.maxRedirects the same way.

Behavior changes for release notes

• New default. On hosts where the OS keepalive idle time is above 120 seconds (the usual 2-hour default), connections that use keepalive now send their first probe after 120 seconds idle. A lower OS setting is kept. To restore the old behavior, add -Doie.tcp.keepalive.idle=0 to custom.vmoptions.

• The idle time in use is only logged at INFO, which the default log4j2.properties doesn't show. To see it in mirth.log and the Server Log, add a logger for SocketUtil to conf/log4j2.properties:

  logger.tcpSocketUtil.name = com.mirth.connect.connectors.tcp.SocketUtil
  logger.tcpSocketUtil.level = INFO

  An invalid oie.tcp.keepalive.idle value is logged at ERROR and shows without this.

• Dead peers are detected sooner. With Linux's default probe settings (75 seconds apart, 9 probes), a peer that disappears from an idle connection now causes Connection timed out after about 13 minutes instead of about 2h11m. The diagnostic in Allow admin to configure to be able to ignore "Connection timed out" errors shown in console #495 for telling idle drops (about 2h11m) from undelivered ACKs (about 15 minutes) by timing no longer works once this is in.

Relationship to #495

This addresses the most likely cause of the idle-connection drops in #495, but it doesn't change how TcpReceiver logs read errors, which is what #495 asks for. So the issue should stay open.

---

🤖 Drafted with Claude Code

[github-actions]

Test Results

130 files  +1  130 suites  +1   3m 55s ⏱️ + 1m 31s
739 tests +2  739 ✅ +2  0 💤 ±0  0 ❌ ±0 
829 runs  +2  823 ✅ +2  6 💤 ±0  0 ❌ ±0 

Results for commit e1fbc4f. ± Comparison against base commit 6d348a0.

♻️ This comment has been updated with latest results.

[mgaffigan]

You had write access to the original PR, but whatever works.

[pacmano1]

It raises a keepalive the admin already lowered. On a host with net.ipv4.tcp_keepalive_time=60, a TCP Listener socket on this branch gets 120. With -Doie.tcp.keepalive.idle=0 it keeps 60. Anyone who tuned the OS lower, say for a firewall with a short idle timeout, gets 120 after upgrading. Make it a cap: only set it when the socket's current value is higher. Inside the engine, getOption(TCP_KEEPIDLE) on an untouched socket returns the OS value (60 here), so it's one comparison.

Nobody can see it. Nothing is logged when it applies, and a bad value's warning doesn't reach mirth.log or the Server Log either, because the default log4j2.properties only passes ERROR for anything without its own logger line. -Doie.tcp.keepalive.idle=12o fell back to 120 with nothing logged anywhere. Adding a WARN logger for SocketUtil made the warning show up. A logger line in log4j2.properties won't reach upgrades, since conf/ is kept. It needs to show in the server log on a default install, upgraded ones included. The startup banner already prints Java, OS and database, so a line there with the idle time in use and the OS value is one way.

[mgaffigan]

It raises a keepalive the admin already lowered. On a host with net.ipv4.tcp_keepalive_time=60, a TCP Listener socket on this branch gets 120.

This is bordering on paranoia and assumes the value can be read.

Nobody can see it. Nothing is logged when it applies, and a bad value's warning doesn't reach mirth.log

That's a systemwide issue that predates this PR. I suggest that would be a separate issue to open. There are many log messages that go nowhere in the default log4j.

[pacmano1]

This is bordering on paranoia and assumes the value can be read.

Fair, it's a narrow case, but the cap only costs a read and it leaves alone the admin who did tune it. Elasticsearch has done this since 2020 on the connections between its own nodes, with 5 minutes where this PR has 120 seconds. On Linux the OS keepalive is net.ipv4.tcp_keepalive_time, 7200 seconds unless an admin changed it. Elasticsearch reads that value from each connection and lowers it to 300 only if it's higher, so an admin who set 300 or less keeps their value. The Elasticsearch PR that added it gives the reason: "not interfering with system defaults or user-specified settings unless they are deemed to be set too high." That's the cap I'm asking for. The value can be read here too: inside the engine on Linux, getOption(TCP_KEEPIDLE) on an untouched socket read 60 with net.ipv4.tcp_keepalive_time set to 60. Didn't try Windows.

There are many log messages that go nowhere in the default log4j.

True, but this is new code, so we don't have to repeat that. The keepalive in use decides when a dead connection shows up as Connection timed out, which is what #495 is about, so anyone reading those errors would want to know which value is in effect. One line at startup would do it.

[tonygermano]

@pacmano1 Thanks for testing this, and for the Elasticsearch reference. I've pushed an update.

Cap. The idle time is now a cap. When the TCP connector first loads, it reads TCP_KEEPIDLE from a new socket, which is the OS value, and only lowers it to oie.tcp.keepalive.idle if the OS value is higher. A host with net.ipv4.tcp_keepalive_time=60 keeps 60. If the value can't be read, it sets 120 as before. The comparison is made once rather than on every socket. If the sysctl changes while the engine is running, sockets that get 120 keep getting it until a restart; sockets left at the OS value follow the change. I checked it in a 4.6.0 JDK container with tcp_keepalive_time at 7200, 120 and 60, and with the property at 30, 0 and 12o. custom.vmoptions describes the cap.

Visibility. The two messages that mean the configured idle time isn't in effect are now ERROR instead of WARN, so they reach mirth.log and the Server Log on a default install, upgraded ones included:

• an invalid oie.tcp.keepalive.idle value (so 12o is no longer silent);
• the OS rejecting the value. That one logs once at ERROR and after that at DEBUG.

The decision, with the idle time in use and the OS value, is logged once at INFO when the TCP connector first loads. That won't show on a default install, but someone interested in seeing it can change the logging level. I didn't put it in the startup banner because the banner is in core (Mirth.java) and this code ships in the TCP plugin, so the banner would need core code that knows about a TCP plugin property. That's the same reason this is a system property and not a mirth.properties setting.

---

🤖 Drafted with Claude Code

[pacmano1]

The setting is read at the first TCP connection, not at startup. With -Doie.tcp.keepalive.idle=12o, mirth.log stayed empty after a restart until a connection came in, then logged the ERROR. Loading SocketUtil from TcpServicePlugin.start(), which is empty now, moves it to startup: with that change the ERROR logged before the startup banner with no channel deployed. If you'd rather leave it, say so in the release notes.

Document it. The Keep Connection Open tooltip on the TCP Listener and TCP Sender should say keepalive probes start after 120 seconds idle by default (oie.tcp.keepalive.idle), or after the OS keepalive time if that is lower. The release notes should also say the idle time in use is only logged at INFO, which the default log4j2.properties doesn't show. Adding a logger for com.mirth.connect.connectors.tcp.SocketUtil at INFO shows it in mirth.log and the Server Log.

[tonygermano]

@pacmano1 Thanks, both done in the latest push.

Startup. You're right, and my earlier reply was wrong to say the decision was made when the connector loads. TcpServicePlugin.start() now calls SocketUtil.init(), so the property is parsed, the OS value is compared and the result is logged at server startup, before any TCP connection. An invalid value is logged at ERROR right after a restart.

Documentation. The "Keep Connection Open" tooltips on the TCP Listener and TCP Sender now say that, with Yes, keepalive probes start after 120 seconds idle, or after the server's OS keepalive time if that is lower, and that oie.tcp.keepalive.idle changes the 120 seconds. The release-notes section of the PR description now says the idle time in use is only logged at INFO, which the default log4j2.properties doesn't show, and gives the two logger lines for com.mirth.connect.connectors.tcp.SocketUtil that show it in mirth.log and the Server Log. The commit message and PR description no longer say the decision is made when the connector first loads.

Log messages. The INFO messages now say they apply to the TCP connectors, and the ones that compare against the OS value say what a later OS change does: if the OS value is kept, later changes still apply but the property isn't compared again until a restart; if the property value is used, later OS changes are ignored until a restart.

Claude checked, starting a server built from this branch with no channels deployed:

• With -Doie.tcp.keepalive.idle=12o and the default log4j2.properties, mirth.log had ERROR ... SocketUtil: Invalid value '12o' for oie.tcp.keepalive.idle; using 120 seconds. just before the startup banner.
• With net.ipv4.tcp_keepalive_time=60 and the SocketUtil logger at INFO, mirth.log had INFO ... SocketUtil: TCP connectors use the OS keepalive idle time; its current value of 60 seconds is not higher than oie.tcp.keepalive.idle (120 seconds). Later changes to the OS value still apply, but oie.tcp.keepalive.idle is not compared again until the server restarts. just before the banner.

---

🤖 Drafted with Claude Code

[pacmano1]

Thanks for addressing these.