[lidezhu]

What problem does this PR solve?

Issue Number: close #1948

What is changed and how it works?

• Stale DML Event Handling: Introduces logic to ignore Data Manipulation Language (DML) events that are associated with an older schema version than what is currently present in the cloud storage sink, ensuring data consistency.
• Schema Version Detection: Modifies the schema checking mechanism to explicitly indicate if a newer schema version has been detected in the storage, allowing for proactive handling of out-of-order events.
• Resource Management for Ignored Tasks: Adds a new utility function to properly handle and acknowledge ignored DML tasks, preventing potential resource leaks by executing their callbacks.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Please refer to [Release Notes Language Style Guide](https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/release-notes-style-guide.html) to write a quality release note.

If you don't think this PR needs a release note then fill it with `None`.

[gemini-code-assist]

Summary of Changes

Hello @lidezhu, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request, marked as "Do Not Merge" (DNM), introduces logging improvements for cloud storage schema handling and adjusts integration test configurations. The changes appear to be part of a debugging or focused testing effort, specifically narrowing down the scope of CI storage tests to a single partition table scenario.

Highlights

• Enhanced Logging: Added detailed log.Info statements within pkg/sink/cloudstorage/path.go to provide better visibility into schema file existence checks and when existing schema files with matching checksums are identified.
• Test Configuration Adjustment: Modified tests/integration_tests/canal_json_storage_partition_table/conf/diff_config.toml to swap the roles of tidb0 and mysql1 as source and target instances, and updated their respective data source configurations.
• CI Test Scope Narrowing: Updated tests/integration_tests/run_heavy_it_in_ci.sh to make all storage_groups run only the canal_json_storage_partition_table integration test, likely for focused testing or debugging.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces extensive logging in the cloud storage schema handling logic and modifies test configurations. While the added logging is useful for debugging, the implementation in pkg/sink/cloudstorage/path.go has significant code duplication that should be refactored. More critically, the changes in tests/integration_tests/run_heavy_it_in_ci.sh disable most of the storage integration tests, which is a major issue that must be addressed before merging. The changes in diff_config.toml appear to be related to this test setup.

[hongyunyan]

/retest

[lidezhu]

/retest

[lidezhu]

/retest

[lidezhu]

/retest-required

[lidezhu]

/retest

[lidezhu]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a mechanism to ignore stale DML events in the cloud storage sink by checking for a newer schema version before processing. The logic appears sound and is accompanied by relevant tests. My review includes a few suggestions to improve code clarity, API design, and test code hygiene.

[lidezhu]

/gemini summary

[gemini-code-assist]

Summary of Changes

This pull request addresses a potential data consistency issue in the cloud storage sink where DML events might be processed out of order relative to schema changes (DDL events). By implementing a mechanism to detect and ignore DML events that pertain to stale schema versions, the system ensures that only data conforming to the latest schema is written, thereby improving data integrity and reliability in the sink.

Highlights

• Stale DML Event Handling: Introduces logic to ignore Data Manipulation Language (DML) events that are associated with an older schema version than what is currently present in the cloud storage sink, ensuring data consistency.
• Schema Version Detection: Modifies the schema checking mechanism to explicitly indicate if a newer schema version has been detected in the storage, allowing for proactive handling of out-of-order events.
• Resource Management for Ignored Tasks: Adds a new utility function to properly handle and acknowledge ignored DML tasks, preventing potential resource leaks by executing their callbacks.

Changelog

• downstreamadapter/sink/cloudstorage/writer.go
  • The flushMessages function now checks for hasNewerSchemaVersion returned by CheckOrWriteSchema. If true, it logs a warning and calls ignoreTableTask for the current task's messages.
  • A new private method ignoreTableTask was added to iterate through messages in a singleTableTask and execute their callbacks, effectively acknowledging and discarding them.
• pkg/sink/cloudstorage/path.go
  • The CheckOrWriteSchema function's signature was updated to return a bool (hasNewerSchemaVersion) in addition to an error.
  • The function now sets hasNewerSchemaVersion to true if a schema file with a higher version than the current table.TableInfoVersion is found during directory traversal.
  • Error return paths were updated to consistently return false for hasNewerSchemaVersion when an error occurs, improving API clarity.
  • Added a log message when a schema file with an exact version is not found, but the latest available one is used.
• pkg/sink/cloudstorage/path_test.go
  • Updated TestCheckOrWriteSchema to assert the new hasNewerSchemaVersion return value, including a test case where an older DML file should be ignored.
  • Removed fmt.Println debugging statements.
• tests/integration_tests/canal_json_storage_partition_table/conf/diff_config.toml
  • Swapped source-instances and target-instance configurations between tidb0 and mysql1.
• tests/integration_tests/canal_json_storage_partition_table/run.sh
  • Modified the logsuffix for run_cdc_server commands from cdcX to X.

Activity

• Initial bot comment requesting approval from hongyunyan.
• Multiple /retest commands issued by hongyunyan and lidezhu, indicating test failures or re-runs.
• lidezhu also issued /retest-required, /gemini review, and /gemini summary commands.
• Automated review comments from gemini-code-assist[bot] were provided, highlighting: a critical concern about replacing all test suites in storage_groups; suggestions for improving comment clarity and correcting typos in writer.go and path.go; a recommendation to improve API clarity by returning false for the boolean when CheckOrWriteSchema encounters an error; and suggestions to remove debugging fmt.Println statements from path_test.go.

[wuhuizuo]

/retest

[lidezhu]

/retest

[lidezhu]

/retest

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hongyunyan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [hongyunyan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2025-09-09 08:34:43.604024435 +0000 UTC m=+350350.163905932: ☑️ agreed by hongyunyan.

[lidezhu]

/retest