Skip to content

OCPBUGS-127482: set etcd unary request warning threshold - #7667

Open
RaphaelFakhri wants to merge 1 commit into
openshift:mainfrom
RaphaelFakhri:fix-etcd-unary-warn-threshold
Open

RaphaelFakhri wants to merge 1 commit into
openshift:mainfrom
RaphaelFakhri:fix-etcd-unary-warn-threshold

Conversation

@RaphaelFakhri

@RaphaelFakhri RaphaelFakhri commented Nov 3, 2026 •

Copy link
Copy Markdown

Summary

Fixes #7301.

microshift-etcd logs a "level":"warn" request stats line for every unary request, including requests that finish in microseconds. On an idle node this is about 30,000 lines per hour, and it hides the few requests that are actually slow.

etcd warns when a request takes longer than WarningUnaryRequestDuration. The etcd binary defaults this to 300ms through its flag parsing, but embed.NewConfig() in the vendored openshift/etcd leaves it at zero, so every request is over the threshold. Upstream etcd fixed the embed default in etcd-io/etcd#20294; the openshift/etcd fork does not carry that change yet.

  • configure() in etcd/cmd/microshift-etcd/run.go sets WarningUnaryRequestDuration to embed.DefaultWarningUnaryRequestDuration (300ms), the same value the etcd binary uses.
  • New unit test TestConfigureWarningUnaryRequestDuration checks the configured value.

Requests slower than 300ms still log the warning. With debug logging enabled, etcd still logs every request at debug level, as before.

Validation

  • cd etcd && go test ./cmd/microshift-etcd/ -run TestConfigureWarningUnaryRequestDuration -v
    • Without the fix: FAIL with WarningUnaryRequestDuration = 0s, want 300ms.
    • With the fix: PASS.
  • cd etcd && go test ./cmd/..., go vet ./cmd/... and gofmt -l cmd pass.

Summary by CodeRabbit

  • Bug Fixes
    • Corrected the default threshold used for slow-request warnings, making etcd warning behavior consistent with its standard configuration.

embed.NewConfig() in the vendored etcd leaves WarningUnaryRequestDuration
at zero, so microshift-etcd logs every unary request as a "request stats"
warning. Set it to etcd's 300ms default.
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification

This PR uses the pipeline controller for second-stage tests. Selection and triggering follow the repository configuration.

Use /test ? to list jobs and /pipeline help for commands.

@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Nov 3, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@RaphaelFakhri: This pull request references Jira Issue OCPBUGS-127482, which is invalid:

  • expected the bug to target the "5.1.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

Fixes #7301.

microshift-etcd logs a "level":"warn" request stats line for every unary request, including requests that finish in microseconds. On an idle node this is about 30,000 lines per hour, and it hides the few requests that are actually slow.

etcd warns when a request takes longer than WarningUnaryRequestDuration. The etcd binary defaults this to 300ms through its flag parsing, but embed.NewConfig() in the vendored openshift/etcd leaves it at zero, so every request is over the threshold. Upstream etcd fixed the embed default in etcd-io/etcd#20294; the openshift/etcd fork does not carry that change yet.

  • configure() in etcd/cmd/microshift-etcd/run.go sets WarningUnaryRequestDuration to embed.DefaultWarningUnaryRequestDuration (300ms), the same value the etcd binary uses.
  • New unit test TestConfigureWarningUnaryRequestDuration checks the configured value.

Requests slower than 300ms still log the warning. With debug logging enabled, etcd still logs every request at debug level, as before.

Validation

  • cd etcd && go test ./cmd/microshift-etcd/ -run TestConfigureWarningUnaryRequestDuration -v
  • Without the fix: FAIL with WarningUnaryRequestDuration = 0s, want 300ms.
  • With the fix: PASS.
  • cd etcd && go test ./cmd/..., go vet ./cmd/... and gofmt -l cmd pass.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign eslutsky for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Nov 3, 2026
@openshift-ci

openshift-ci Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Hi @RaphaelFakhri. Thanks for your PR.

I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review.

Walkthrough

The embedded etcd configuration now sets WarningUnaryRequestDuration to etcd’s default. A test checks that the configured value matches that default.

Changes

etcd warning threshold

Layer / File(s) Summary
Set and verify warning threshold
etcd/cmd/microshift-etcd/run.go, etcd/cmd/microshift-etcd/run_test.go
EtcdService.configure sets the warning threshold to etcd’s default. The test checks that the configured value matches the default.

Estimated code review effort: 2 (Simple) | ~8 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested labels: ready-for-human-review

Merge Risk: Minimal · up to 28ed0

No actionable merge-blocking risk remains; the configured warning threshold is correctly propagated.

Pre-merge checks: Passed 14 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 14 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title names the Jira key and the change: setting the etcd unary request warning threshold. The raw summary confirms that configure() now sets WarningUnaryRequestDuration to etcd's default, replaci…
Linked Issues check Passed Issue #7301 asks for the embedded etcd warning threshold to use the 300 ms default. EtcdService.configure sets s.etcdCfg.WarningUnaryRequestDuration to etcd.DefaultWarningUnaryRequestDuration. `…
Out of Scope Changes check Passed The reviewed changes are limited to the etcd configuration assignment in etcd/cmd/microshift-etcd/run.go and its unit test in run_test.go. Both changes directly support issue #7301. No unrelated c…
Stable And Deterministic Test Names Passed The pull request adds one test, TestConfigureWarningUnaryRequestDuration. Its name is a static, descriptive identifier with no generated values, timestamps, node or namespace names, IP addresses, UU…
Test Structure And Quality Passed PASS. The new test is a standard Go testing.T unit test, not a Ginkgo It block. It checks one behavior, performs no cluster operations or resource creation, has a meaningful failure message, and d…
Microshift Test Compatibility Passed PASS: The pull request adds one standard Go unit test, TestConfigureWarningUnaryRequestDuration, using testing, config.NewDefault(), and an etcd embedded configuration value. It adds no Ginkgo e…
Single Node Openshift (Sno) Test Compatibility Passed PASS: The pull request adds a standard Go unit test, TestConfigureWarningUnaryRequestDuration, in etcd/cmd/microshift-etcd/run_test.go. It does not add Ginkgo e2e tests or assumptions about nodes,…
Topology-Aware Scheduling Compatibility Passed The pull request changes only embedded etcd configuration and adds a unit test. It introduces no deployment manifests, controllers, replica settings, affinity, topology spread constraints, node select…
Ote Binary Stdout Contract Passed The pull request adds only an etcd configuration assignment and a unit test assertion. It adds no stdout write, logging setup, suite setup, or process-level output code. The changed configure() code…
Ipv6 And Disconnected Network Test Compatibility Passed PASS. The pull request adds a standard Go unit test, TestConfigureWarningUnaryRequestDuration, not a Ginkgo e2e test. The test only constructs local configuration and compares a duration value. It c…
No-Weak-Crypto Passed The pull request only adds the etcd warning-duration assignment and its unit test. The patch introduces no MD5, SHA1, DES, 3DES, RC4, Blowfish, ECB, custom cryptography, or secret/token comparisons. E…
Container-Privileges Passed PASS: The pull request changes only Go configuration and a unit test in etcd/cmd/microshift-etcd. The diff introduces no container or Kubernetes manifest fields for privileged, hostPID, `hostNet…
No-Sensitive-Data-In-Logs Passed The PR does not add sensitive-data logging. It only sets WarningUnaryRequestDuration to etcd's 300 ms default and adds a unit test. The vendored logger already handles request metadata and content; …
Full details: Title check

Explanation

The title names the Jira key and the change: setting the etcd unary request warning threshold. The raw summary confirms that configure() now sets WarningUnaryRequestDuration to etcd's default, replacing the zero value. The title matches the primary change, and it is specific enough for a teammate scanning history.

Full details: Linked Issues check

Explanation

Issue #7301 asks for the embedded etcd warning threshold to use the 300 ms default. EtcdService.configure sets s.etcdCfg.WarningUnaryRequestDuration to etcd.DefaultWarningUnaryRequestDuration. TestConfigureWarningUnaryRequestDuration verifies this value. The issue presents a MicroShift configuration option as desirable, not as a required acceptance condition.

Full details: Out of Scope Changes check

Explanation

The reviewed changes are limited to the etcd configuration assignment in etcd/cmd/microshift-etcd/run.go and its unit test in run_test.go. Both changes directly support issue #7301. No unrelated change is shown.

Full details: Stable And Deterministic Test Names

Explanation

The pull request adds one test, TestConfigureWarningUnaryRequestDuration. Its name is a static, descriptive identifier with no generated values, timestamps, node or namespace names, IP addresses, UUIDs, or legacy metadata tokens. No Ginkgo test titles are introduced.

Full details: Test Structure And Quality

Explanation

PASS. The new test is a standard Go testing.T unit test, not a Ginkgo It block. It checks one behavior, performs no cluster operations or resource creation, has a meaningful failure message, and does not require BeforeEach, AfterEach, timeouts, or ote.Informing().

Full details: Microshift Test Compatibility

Explanation

PASS: The pull request adds one standard Go unit test, TestConfigureWarningUnaryRequestDuration, using testing, config.NewDefault(), and an etcd embedded configuration value. It adds no Ginkgo e2e tests, OpenShift API references, unsupported namespaces, or MicroShift-incompatible assumptions.

Full details: Single Node Openshift (Sno) Test Compatibility

Explanation

PASS: The pull request adds a standard Go unit test, TestConfigureWarningUnaryRequestDuration, in etcd/cmd/microshift-etcd/run_test.go. It does not add Ginkgo e2e tests or assumptions about nodes, topology, scheduling, failover, or HA. The SNO compatibility check is therefore not applicable.

Full details: Topology-Aware Scheduling Compatibility

Explanation

The pull request changes only embedded etcd configuration and adds a unit test. It introduces no deployment manifests, controllers, replica settings, affinity, topology spread constraints, node selectors, tolerations, or PDBs. The topology-aware scheduling check is therefore not applicable.

Full details: Ote Binary Stdout Contract

Explanation

The pull request adds only an etcd configuration assignment and a unit test assertion. It adds no stdout write, logging setup, suite setup, or process-level output code. The changed configure() code only assigns WarningUnaryRequestDuration, and the test reports failures through t.Errorf.

Full details: Ipv6 And Disconnected Network Test Compatibility

Explanation

PASS. The pull request adds a standard Go unit test, TestConfigureWarningUnaryRequestDuration, not a Ginkgo e2e test. The test only constructs local configuration and compares a duration value. It contains no IPv4 assumptions and no external or public network connectivity.

Full details: No-Weak-Crypto

Explanation

The pull request only adds the etcd warning-duration assignment and its unit test. The patch introduces no MD5, SHA1, DES, 3DES, RC4, Blowfish, ECB, custom cryptography, or secret/token comparisons. Existing TLS and cryptomaterial references are unchanged.

Full details: Container-Privileges

Explanation

PASS: The pull request changes only Go configuration and a unit test in etcd/cmd/microshift-etcd. The diff introduces no container or Kubernetes manifest fields for privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, or allowPrivilegeEscalation. The existing privileged-process check in Run is unchanged and is not introduced by this pull request.

Full details: No-Sensitive-Data-In-Logs

Explanation

The PR does not add sensitive-data logging. It only sets WarningUnaryRequestDuration to etcd's 300 ms default and adds a unit test. The vendored logger already handles request metadata and content; this change reduces routine warning volume and leaves debug logging unchanged.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 1 functions across 2 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches
Generate unit tests (beta)
  • Create a new PR

Warning

Some tools did not complete. Review the errors below.

golangci-lint (2.14.0)

level=warning msg="The linter 'gomodguard' is deprecated (since v2.12.0) due to: new major version. Replaced by gomodguard_v2."
level=warning msg="Suggested new configuration:\nlinters:\n enable:\n - gomodguard_v2\n"
level=error msg="Running error: context loading failed: failed to load packages: failed to load packages: failed to load with go/packages: err: exit status 1: stderr: go: inconsistent vendoring in /etcd:\n\tgithub.com/openshift/api@v0.0.0-20261009034342-f9511d3fcb26: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\tgithub.com/openshift/build-machinery-go@v0.0.0-20260902143904-520f675c892b: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\tgithub.com/openshift/microshift@v0.0.0-00010101000000-000000000000: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\tgithub.com/spf13/cobra@v1.10.2: is explicitly required in go.mod, but not marked as explicit in vendor/module

... [truncated 19776 characters] ...

: is replaced in go.mod, but not marked as replaced in vendor/modules.txt\n\tk8s.io/pod-security-admission: is replaced in go.mod, but not marked as replaced in vendor/modules.txt\n\tk8s.io/sample-apiserver: is replaced in go.mod, but not marked as replaced in vendor/modules.txt\n\tk8s.io/sample-cli-plugin: is replaced in go.mod, but not marked as replaced in vendor/modules.txt\n\tk8s.io/sample-controller: is replaced in go.mod, but not marked as replaced in vendor/modules.txt\n\tk8s.io/cri-streaming: is replaced in go.mod, but not marked as replaced in vendor/modules.txt\n\tk8s.io/streaming: is replaced in go.mod, but not marked as replaced in vendor/modules.txt\n\n\tTo ignore the vendor directory, use -mod=readonly or -mod=mod.\n\tTo sync the vendor directory, run:\n\t\tgo mod vendor\n"

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.
Recent review info
Run configuration
  • Source mode: Auto
  • Repository: openshift/microshift
  • Branch: main
  • File: .coderabbit.yaml
  • Commit: d91d74e
  • Layers: Repository YAML (base), Central YAML (inherited)
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: f34ba6d0-8af7-4598-9bb4-63adff3844ea
Commits

Reviewing files that changed from the base of the PR and between d91d74e and 28ed077.

Files selected for processing: 2
  • etcd/cmd/microshift-etcd/run.go
  • etcd/cmd/microshift-etcd/run_test.go

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

@coderabbitai coderabbitai Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Nov 3, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Embedded etcd logs every unary request as a warn (request stats), flooding the journal

2 participants