[igorchyts]

Why

Kafka Connect workers write every log line to stdout and also to /opt/kafka/logs/connect.log. That file is an hourly log4j appender that is never deleted, and it sits in the container's writable layer, so busy workers fill ephemeral storage and get evicted within days. The 4.2.0 broker fix (KAFKA_LOG4J_ROOT_LOGLEVEL) doesn't help: only the broker image's launcher reads that variable, and Connect starts connect-distributed.sh directly.

Issue: https://gitlab.com/controlplane/controlplane/-/work_items/5401 (internal)

Change (4.4.1, copied from 4.4.0)

The Connect start script drops connectAppender from the root logger in a copy of connect-log4j.properties, and starts Connect with that copy:

if [[ -f /opt/kafka/config/connect-log4j.properties ]]; then
  sed '/^log4j\.rootLogger=/s/, *connectAppender//' /opt/kafka/config/connect-log4j.properties > /opt/kafka/config/connect-log4j-updated.properties
  export KAFKA_LOG4J_OPTS="${KAFKA_LOG4J_OPTS:--Dlog4j.configuration=file:/opt/kafka/config/connect-log4j-updated.properties}"
fi

• No new values, no new env var.
• Brokers and every other workload render byte-identical to 4.4.0, apart from the version tags.
• 4.x images (log4j2) are unchanged.
• Docs: README, RELEASES, a wizard upgrade note, the briefing.
• e2e: new check connect-stdout-only. Every install in its range runs a Connect worker: the 4.4.1 baseline, and the 4.1.0 and 3.2.0 upgrade sources.

Tested

In our test org (aws-us-east-1), with .github/e2e/run.py:

• Fresh 4.4.1: 11/11 passed.
• Upgrade 4.1.0 → 4.4.1 and 3.2.0 → 4.4.1: connect-stdout-only and every other check passed. Only the final uninstall timed out while volumesets finished deleting; a manual uninstall a few minutes later removed everything.

The narrower sed (it removes only connectAppender) was applied after those runs. On the stock file it produces the same line, log4j.rootLogger=INFO, stdout, verified on the image itself, and CI re-runs the full suite on this push.

[github-actions]

Template e2e tests

⚠️ kafka/4.4.1 — 36/43 passed, 7 warnings (34.1 min, aws-us-east-1)

Scenario	Check	Result
install 4.4.1	install + every workload ready	pass
install 4.4.1	rendered == stored (Console Drift tab)	warn — workload/e2ein002cf10-kafbat-ui spec.containers[0].ports[0].protocol: stored "http", chart omits it; workload/e2ein002cf10-cluster spec.containers[1].readinessProbe.httpGet.scheme: stored "HTTP", chart omits it; workload/e2ein002cf10-cluster spec.rolloutOptions.maxUnavailableReplicas: chart sends "1", API did not keep it; workload/e2ein002cf10-cluster spec.rolloutOptions.terminationGracePeriodS
install 4.4.1	produce-consume	pass
install 4.4.1	replicas-in-sync	pass
install 4.4.1	rest-proxy	pass
install 4.4.1	rest-proxy-rejects-bad-password	pass
install 4.4.1	kafbat-ui	pass
install 4.4.1	log4j-root-loglevel (#5137)	pass
install 4.4.1	connect-stdout-only (#5401)	pass · new
install 4.4.1	no-op upgrade ×2 (no churn)	pass
install 4.4.1	uninstall removes everything	pass
upgrade 4.1.0 → 4.4.1	install 4.1.0 (starting point)	pass
upgrade 4.1.0 → 4.4.1 [before, on 4.1.0]	produce-consume	pass
upgrade 4.1.0 → 4.4.1 [before, on 4.1.0]	replicas-in-sync	pass
upgrade 4.1.0 → 4.4.1 [before, on 4.1.0]	rest-proxy	warn — known issue: Values from 4.1.0 and older carry the old Jetty JAAS class, which cp-kafka-rest:latest rejects, so every basic-auth user gets 401 after the upgrade. Pending a maintainer ruling. | exit 1, expected 0; stdout did not match /"e2e"/ (attempt 4/4): in _call_chain ⏎ result = func(*args) ⏎ File "/usr/lib64/python3.9/urllib/request.py", line 641, in http_error_default ⏎ raise H
upgrade 4.1.0 → 4.4.1 [before, on 4.1.0]	rest-proxy-rejects-bad-password	pass
upgrade 4.1.0 → 4.4.1 [before, on 4.1.0]	kafbat-ui	pass
upgrade 4.1.0 → 4.4.1	upgrade to 4.4.1 + every workload ready	pass
upgrade 4.1.0 → 4.4.1	rendered == stored (Console Drift tab)	warn — workload/e2ein002cf10u410-kafbat-ui spec.containers[0].ports[0].protocol: stored "http", chart omits it; workload/e2ein002cf10u410-cluster spec.containers[1].readinessProbe.httpGet.scheme: stored "HTTP", chart omits it; workload/e2ein002cf10u410-cluster spec.rolloutOptions.maxUnavailableReplicas: chart sends "1", API did not keep it; workload/e2ein002cf10u410-cluster spec.rolloutOptions.termina
upgrade 4.1.0 → 4.4.1	produce-consume	pass
upgrade 4.1.0 → 4.4.1	replicas-in-sync	pass
upgrade 4.1.0 → 4.4.1	rest-proxy	warn — known issue: Values from 4.1.0 and older carry the old Jetty JAAS class, which cp-kafka-rest:latest rejects, so every basic-auth user gets 401 after the upgrade. Pending a maintainer ruling. | exit 1, expected 0; stdout did not match /"e2e"/ (attempt 4/4): in _call_chain ⏎ result = func(*args) ⏎ File "/usr/lib64/python3.9/urllib/request.py", line 641, in http_error_default ⏎ raise H
upgrade 4.1.0 → 4.4.1	rest-proxy-rejects-bad-password	pass
upgrade 4.1.0 → 4.4.1	kafbat-ui	pass
upgrade 4.1.0 → 4.4.1	log4j-root-loglevel (#5137)	pass
upgrade 4.1.0 → 4.4.1	connect-stdout-only (#5401)	pass · new
upgrade 4.1.0 → 4.4.1	uninstall removes everything	pass
upgrade 3.2.0 → 4.4.1	install 3.2.0 (starting point)	pass
upgrade 3.2.0 → 4.4.1 [before, on 3.2.0]	produce-consume	pass
upgrade 3.2.0 → 4.4.1 [before, on 3.2.0]	replicas-in-sync	pass
upgrade 3.2.0 → 4.4.1 [before, on 3.2.0]	rest-proxy	warn — known issue: Values from 4.1.0 and older carry the old Jetty JAAS class, which cp-kafka-rest:latest rejects, so every basic-auth user gets 401 after the upgrade. Pending a maintainer ruling. | exit 1, expected 0; stdout did not match /"e2e"/ (attempt 4/4): in _call_chain ⏎ result = func(*args) ⏎ File "/usr/lib64/python3.9/urllib/request.py", line 641, in http_error_default ⏎ raise H
upgrade 3.2.0 → 4.4.1 [before, on 3.2.0]	rest-proxy-rejects-bad-password	pass
upgrade 3.2.0 → 4.4.1 [before, on 3.2.0]	kafbat-ui	pass
upgrade 3.2.0 → 4.4.1	upgrade to 4.4.1 + every workload ready	pass
upgrade 3.2.0 → 4.4.1	rendered == stored (Console Drift tab)	warn — workload/e2ein002cf10u320-kafbat-ui spec.containers[0].ports[0].protocol: stored "http", chart omits it; workload/e2ein002cf10u320-cluster spec.containers[1].readinessProbe.httpGet.scheme: stored "HTTP", chart omits it; workload/e2ein002cf10u320-cluster spec.rolloutOptions.maxUnavailableReplicas: chart sends "1", API did not keep it; workload/e2ein002cf10u320-cluster spec.rolloutOptions.termina
upgrade 3.2.0 → 4.4.1	produce-consume	pass
upgrade 3.2.0 → 4.4.1	replicas-in-sync	pass
upgrade 3.2.0 → 4.4.1	rest-proxy	warn — known issue: Values from 4.1.0 and older carry the old Jetty JAAS class, which cp-kafka-rest:latest rejects, so every basic-auth user gets 401 after the upgrade. Pending a maintainer ruling. | exit 1, expected 0; stdout did not match /"e2e"/ (attempt 4/4): in _call_chain ⏎ result = func(*args) ⏎ File "/usr/lib64/python3.9/urllib/request.py", line 641, in http_error_default ⏎ raise H
upgrade 3.2.0 → 4.4.1	rest-proxy-rejects-bad-password	pass
upgrade 3.2.0 → 4.4.1	kafbat-ui	pass
upgrade 3.2.0 → 4.4.1	log4j-root-loglevel (#5137)	pass
upgrade 3.2.0 → 4.4.1	connect-stdout-only (#5401)	pass · new
upgrade 3.2.0 → 4.4.1	uninstall removes everything	pass

Run logs