Skip to content

The changefeed checkpoint lag fluctuates by slightly over one second #6397

Open

Description

@july2993

What did you do?

  • update config cdc.min-ts-interval as 200ms(default 1s) of tikv
  • create changefeed and check the checkpoint latency.

What did you expect to see?

There are no fluctuations, the metrics shoud not includes progress-reporting staleness

What did you see instead?

Image

The changefeed checkpoint lag fluctuates by slightly over one second.

The roughly one-second fluctuation in Changefeed Checkpoint Lag is mainly caused by progress reporting in TiCDC’s new architecture.

time.Since(cfMaintainer.lastReportTime) > time.Second {

Why

The roughly one-second fluctuation in Changefeed Checkpoint Lag is mainly caused by progress reporting in TiCDC’s new architecture.

The Maintainer updates its internal checkpoint frequently, but normally reports that progress to the coordinator only every 1–1.2 seconds. The coordinator persists the reported checkpoint, and the metric calculates:

Checkpoint lag = current PD time − last persisted checkpoint time

Between reports, replication continues, but the coordinator still holds the previous checkpoint. Depending on when the metric is calculated, it sees either a relatively fresh checkpoint or one that is almost a reporting cycle old. This produces the fluctuation.

For example, if the internal checkpoint lag is 300ms, an additional 900ms without a new report makes the owner metric show approximately 1.2s, even though downstream writes have continued.

Therefore, this metric includes progress-reporting staleness; it should not be interpreted as the exact end-to-end replication latency of individual transactions.

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

Release Version: v26.3.6
Git Commit Hash: 9d5bccc0b7bfec250d34932da0fa38622f0bb235
Git Branch: HEAD
UTC Build Time: 2026-09-20 09:42:43
Go Version: go version go1.25.12 linux/amd64
Failpoint Build: false
Kernel Type: Next Generation```

Activity

  1. added
    contributionThis PR is from a community contributor.
    first-time-contributorIndicates that the PR was contributed by an external member and is a first-time contributor.
    on Sep 28, 2026
  2. ti-chi-bot commented on Sep 28, 2026

    @ti-chi-bot

    Welcome @july2993! It looks like this is your first issue to pingcap/ticdc 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

contributionThis PR is from a community contributor.first-time-contributorIndicates that the PR was contributed by an external member and is a first-time contributor.type/bugThe issue is confirmed as a bug.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions