Skip to content

OCPEDGE-3133: certs metrics dropin - #7664

Open
eggfoobar wants to merge 4 commits into
openshift:mainfrom
eggfoobar:ocpstrat-2899-certs-metrics-dropin
Open

eggfoobar wants to merge 4 commits into
openshift:mainfrom
eggfoobar:ocpstrat-2899-certs-metrics-dropin

Conversation

@eggfoobar

@eggfoobar eggfoobar commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary by CodeRabbit

  • New Features
    • Added an opt-in local Prometheus metrics endpoint for certificate health and renewal status. It binds to a loopback address and is disabled by default.
    • Added optional observability package support for scraping the local metrics endpoint and identifying metrics by host.
  • Documentation
    • Added configuration guidance for enabling, disabling, and customizing the endpoint, plus details about certificate metrics and their collection.
    • Clarified that local metrics are separate from telemetry uploads and do not require the optional OpenTelemetry Collector.

eggfoobar and others added 4 commits October 30, 2026 22:24
Add an opt-in loopback HTTP metrics endpoint backed by an isolated Prometheus registry for local OpenTelemetry collection. Start it independently of etcd and API-server readiness, bound scrape requests, and keep listener failures non-fatal.

Share certificate chain construction and pending metadata parsing with a read-only observation path. Coordinate through a non-creating shared lock without reading private keys, generating certificates, or recovering transactions.

Publish minute-based snapshots of active certificate validity and five-state health, pending renewal, collection success, and last-success time. Omit unavailable series instead of presenting stale health or unknown pending state as a reliable zero.

Add startup-loaded metrics configuration with loopback validation, regenerate configuration and schema, document access and failure semantics, and include focused Go tests and Robot smoke coverage.

Co-authored-by: GPT-5 <noreply@openai.com>
Signed-off-by: ehila <ehila@redhat.com>
Accept a variadic list of collectors through an interface that embeds prometheus.Collector and adds context-aware background collection. Register and start each collector independently, and cancel their shared context when the service stops.

Remove the separate registration method, exercise multiple collector lifecycles in the existing service test, and update the contributor architecture documentation.

Co-authored-by: GPT-5 <noreply@openai.com>
Signed-off-by: ehila <ehila@redhat.com>
Signed-off-by: ehila <ehila@redhat.com>
Package a loopback MicroShift scrape drop-in with the observability RPM, preserving administrator edits with config(noreplace). Reuse the existing scrape pipelines and host resource detection across the small, medium, and large presets while keeping endpoint enablement opt-in.

Extend the observability Robot suite to verify certificate, pending-renewal, and freshness metrics reach the collector and backend. Document enablement, custom targets, upgrade adoption, host identity, and disabled-target behavior.

Co-authored-by: GPT-5 <noreply@openai.com>
Signed-off-by: ehila <ehila@redhat.com>
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Nov 3, 2026
@openshift-ci-robot

openshift-ci-robot commented Nov 3, 2026 •

Copy link
Copy Markdown

@eggfoobar: This pull request references OCPEDGE-3133 which is a valid jira issue.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification

This PR uses the pipeline controller for second-stage tests. Selection and triggering follow the repository configuration.

Use /test ? to list jobs and /pipeline help for commands.

@eggfoobar

Copy link
Copy Markdown
Contributor Author

/hold

Holding until #7641 has merged in

@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review.

Walkthrough

MicroShift adds a disabled-by-default, loopback metrics endpoint for certificate status and renewal data. The change adds read-only certificate observation, Prometheus collection and serving, configuration, observability packaging, and tests.

Changes

Local Certificate Metrics

Layer / File(s) Summary
Centralize certificate-chain construction
pkg/admin/certificates/chains.go, pkg/cmd/certs*.go, pkg/cmd/init.go, pkg/cmd/*_test.go
Adds certificates.NewChains and updates certificate setup, renewal, and inventory call sites and tests to use it.
Add read-only certificate observation
pkg/admin/certificates/{lock,metadata,observation,pending}.go, pkg/admin/certificates/observation_test.go
Adds non-blocking shared-lock observation and pending-renewal metadata parsing. Observation reports inventory and pending status without repairing transactions.
Define metrics configuration
pkg/config/*, cmd/generate-config/config/config-openapi-spec.json, packaging/microshift/config.yaml, docs/user/howto_config.md
Adds metrics settings with a loopback address default and disabled status. Validation limits addresses to literal loopback IPs and ports from 1 to 65535.
Publish certificate metrics
pkg/metrics/*, go.mod, docs/contributor/architecture.md
Adds a private-registry service and a certificate collector that refreshes cached metrics. The service exposes /metrics with request limits and configured timeouts.
Wire and verify observability
pkg/cmd/run.go, packaging/observability/*, packaging/rpm/microshift.spec, docs/contributor/observability_metric_exporters.md, docs/user/howto_certificates.md, test/suites/*
Registers the metrics service at startup, adds the MicroShift scrape target and host resource detection to collector pipelines, and adds documentation and integration coverage.

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant MicroShift as RunMicroshift
  participant MetricsService
  participant CertificateCollector
  participant Observe
  participant CertificateStore
  participant ScrapeClient
  MicroShift->>MetricsService: Register service and certificate collector
  MetricsService->>CertificateCollector: Run collector
  CertificateCollector->>Observe: Refresh with current time
  Observe->>CertificateStore: Read lock, inventory, and pending metadata
  ScrapeClient->>MetricsService: GET /metrics
  MetricsService->>CertificateCollector: Collect cached snapshot
  CertificateCollector-->>MetricsService: Return metrics
  MetricsService-->>ScrapeClient: Return metrics
Loading
Merge Risk: Minimal · up to 341a7

This adds an opt-in, disabled-by-default, loopback-only certificate metrics endpoint with read-only certificate observation. No merge-blocking issue was identified. Existing behavior is unchanged unless the endpoint is explicitly enabled.

Pre-merge checks: Passed 14 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 19.23% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 14 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly identifies the certificate metrics drop-in, which is the main change in the pull request.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names Passed The pull request adds standard Go tests and Robot Framework test cases, but no Ginkgo tests. The added test names are static, and no It, Describe, Context, or When title contains dynamic data …
Test Structure And Quality Passed PASS: The pull request adds standard Go testing tests and Robot Framework tests. It adds no Ginkgo test files or It() blocks. Therefore the Ginkgo-specific requirements do not apply.
Microshift Test Compatibility Passed PASS: The pull request adds Go unit tests and Robot Framework suites, not new Ginkgo e2e tests. The changed Go files contain no Ginkgo imports or It/Describe/Context/When test declarations. Therefore,…
Single Node Openshift (Sno) Test Compatibility Passed The pull request adds Go unit tests and Robot Framework suites, not new Ginkgo e2e tests. The changed tests contain no Ginkgo constructs or multi-node/HA assumptions. Therefore the SNO compatibility c…
Topology-Aware Scheduling Compatibility Passed The pull request does not add or modify deployment manifests, operator scheduling logic, or Kubernetes workload controllers. The changed YAML files configure OpenTelemetry pipelines and a local Promet…
Ote Binary Stdout Contract Passed PASS. The PR adds no OTE/Ginkgo suite entrypoint, suite setup, or process-level stdout write. The only new process-level logging is in pkg/metrics/service.go and uses klog; this repository's klog impl…
Ipv6 And Disconnected Network Test Compatibility Passed The pull request adds Robot Framework suites and standard Go tests, not new Ginkgo e2e tests. The changed Go tests use testing and testify/require; no changed file imports Ginkgo or uses It, `De…
No-Weak-Crypto Passed PASS. The PR diff adds no MD5, SHA1, DES, RC4, 3DES, Blowfish, or ECB implementation or API. The certificate changes configure existing certificate-chain builders and do not implement cryptography. Ha…
Container-Privileges Passed No changed container or Kubernetes manifest introduces a flagged privilege setting. The PR diff contains no privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, allowPrivilegeEscalation, …
No-Sensitive-Data-In-Logs Passed No sensitive-data logging was introduced. The new logs in pkg/metrics/service.go report endpoint state, validation or listener errors, and the validated local listener address. Metrics contains on…
Full details: Stable And Deterministic Test Names

Explanation

The pull request adds standard Go tests and Robot Framework test cases, but no Ginkgo tests. The added test names are static, and no It, Describe, Context, or When title contains dynamic data or legacy metadata tokens.

Full details: Microshift Test Compatibility

Explanation

PASS: The pull request adds Go unit tests and Robot Framework suites, not new Ginkgo e2e tests. The changed Go files contain no Ginkgo imports or It/Describe/Context/When test declarations. Therefore, the MicroShift API compatibility check is not applicable.

Full details: Single Node Openshift (Sno) Test Compatibility

Explanation

The pull request adds Go unit tests and Robot Framework suites, not new Ginkgo e2e tests. The changed tests contain no Ginkgo constructs or multi-node/HA assumptions. Therefore the SNO compatibility check is not applicable.

Full details: Topology-Aware Scheduling Compatibility

Explanation

The pull request does not add or modify deployment manifests, operator scheduling logic, or Kubernetes workload controllers. The changed YAML files configure OpenTelemetry pipelines and a local Prometheus scrape target. The Go changes register a local MicroShift service and do not add replicas, affinity, topology spread, node selectors, tolerations, or PDBs. The topology-aware scheduling check is therefore not applicable.

Full details: Ote Binary Stdout Contract

Explanation

PASS. The PR adds no OTE/Ginkgo suite entrypoint, suite setup, or process-level stdout write. The only new process-level logging is in pkg/metrics/service.go and uses klog; this repository's klog implementation defaults all log statements to standard error. The changed tests have no TestMain or suite setup.

Full details: Ipv6 And Disconnected Network Test Compatibility

Explanation

The pull request adds Robot Framework suites and standard Go tests, not new Ginkgo e2e tests. The changed Go tests use testing and testify/require; no changed file imports Ginkgo or uses It, Describe, Context, or When. Therefore this check is not applicable.

Full details: No-Weak-Crypto

Explanation

PASS. The PR diff adds no MD5, SHA1, DES, RC4, 3DES, Blowfish, or ECB implementation or API. The certificate changes configure existing certificate-chain builders and do not implement cryptography. Hash fields are metadata validation only, with no secret or token comparison. The only crypto/x509 reference is in test code, and the existing crypto/sha256 import is unchanged.

Full details: Container-Privileges

Explanation

No changed container or Kubernetes manifest introduces a flagged privilege setting. The PR diff contains no privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, allowPrivilegeEscalation, or security-context entries. The changed YAML files are application configuration and OpenTelemetry scrape settings.

Full details: No-Sensitive-Data-In-Logs

Explanation

No sensitive-data logging was introduced. The new logs in pkg/metrics/service.go report endpoint state, validation or listener errors, and the validated local listener address. Metrics contains only status and a literal loopback address. No passwords, tokens, API keys, PII, session IDs, internal hostnames, or customer data are logged by the changed code.

Full details: Docstring Coverage

Explanation

Docstring coverage is 19.23% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 26 functions across 22 files. (14 skipped: 14 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches
Generate unit tests (beta)
  • Create a new PR

Warning

Some tools did not complete. Review the errors below.

golangci-lint (2.14.0)

level=warning msg="The linter 'gomodguard' is deprecated (since v2.12.0) due to: new major version. Replaced by gomodguard_v2."
level=warning msg="Suggested new configuration:\nlinters:\n enable:\n - gomodguard_v2\n"
level=error msg="Running error: context loading failed: failed to load packages: failed to load packages: failed to load with go/packages: err: exit status 1: stderr: go: inconsistent vendoring in :\n\tgithub.com/apparentlymart/go-cidr@v1.1.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\tgithub.com/coreos/go-systemd@v0.0.0-20190321100706-95778dfbb74e: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\tgithub.com/google/go-cmp@v0.7.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\tgithub.com/miekg/dns@v1.1.63: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt\n\tgithub.com/openshift/api@v0.0.0-20261009034342-f9511d3fcb26: is

... [truncated 30622 characters] ...

: is replaced in go.mod, but not marked as replaced in vendor/modules.txt\n\tk8s.io/pod-security-admission: is replaced in go.mod, but not marked as replaced in vendor/modules.txt\n\tk8s.io/sample-apiserver: is replaced in go.mod, but not marked as replaced in vendor/modules.txt\n\tk8s.io/sample-cli-plugin: is replaced in go.mod, but not marked as replaced in vendor/modules.txt\n\tk8s.io/sample-controller: is replaced in go.mod, but not marked as replaced in vendor/modules.txt\n\tk8s.io/cri-streaming: is replaced in go.mod, but not marked as replaced in vendor/modules.txt\n\tk8s.io/streaming: is replaced in go.mod, but not marked as replaced in vendor/modules.txt\n\n\tTo ignore the vendor directory, use -mod=readonly or -mod=mod.\n\tTo sync the vendor directory, run:\n\t\tgo mod vendor\n"

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.
Recent review info
Run configuration
  • Source mode: Auto
  • Repository: openshift/microshift
  • Branch: main
  • File: .coderabbit.yaml
  • Commit: d91d74e
  • Layers: Repository YAML (base), Central YAML (inherited)
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: 3b950ff0-88bb-48ca-8738-18dfc4816b31
Commits

Reviewing files that changed from the base of the PR and between d91d74e and 341a71b.

Files selected for processing: 36
  • cmd/generate-config/config/config-openapi-spec.json
  • docs/contributor/architecture.md
  • docs/contributor/observability_metric_exporters.md
  • docs/user/howto_certificates.md
  • docs/user/howto_config.md
  • go.mod
  • packaging/microshift/config.yaml
  • packaging/observability/opentelemetry-collector-large.yaml
  • packaging/observability/opentelemetry-collector-medium.yaml
  • packaging/observability/opentelemetry-collector-small.yaml
  • packaging/observability/scrape.d/microshift.yaml
  • packaging/rpm/microshift.spec
  • pkg/admin/certificates/chains.go
  • pkg/admin/certificates/lock.go
  • pkg/admin/certificates/metadata.go
  • pkg/admin/certificates/observation.go
  • pkg/admin/certificates/observation_test.go
  • pkg/admin/certificates/pending.go
  • pkg/cmd/certs.go
  • pkg/cmd/certs_config_test.go
  • pkg/cmd/certs_pending.go
  • pkg/cmd/certs_pending_test.go
  • pkg/cmd/certs_renew_test.go
  • pkg/cmd/certs_transaction.go
  • pkg/cmd/init.go
  • pkg/cmd/init_test.go
  • pkg/cmd/run.go
  • pkg/config/config.go
  • pkg/config/metrics.go
  • pkg/config/metrics_test.go
  • pkg/metrics/certificates.go
  • pkg/metrics/certificates_test.go
  • pkg/metrics/service.go
  • pkg/metrics/service_test.go
  • test/suites/optional/observability.robot
  • test/suites/standard2/cert-metrics.robot

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@openshift-ci openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 3, 2026
@openshift-ci

openshift-ci Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign pacevedom for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci

openshift-ci Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

@eggfoobar: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants