[JeonDaehong]

Purpose

Linked issue: close #1163

Brief change log

KeyValueFileReader::read skips the sort merge for a section with one sorted run under the deduplicate or first-row engine. The run's files don't overlap and keys are unique within a file, so every key has a single version. Each batch still drops retract rows the way the merge handles a lone version, and first-row without ignore-delete rejects them. That's needed because older Rust writers recorded delete_row_count = 0 for files holding DELETE rows. read_with_merge_function (audit log) always merges. Other sections merge as before.

Disjoint level-0 files packed into one split, 2M rows: 157 ms → 60 ms on current main (alternating on one instance, ranges don't overlap). Overlapping sections take the merge path unchanged.

Tests

• pk_single_run_read_test: three disjoint files in one split read back once each, with a value-only projection and a residual filter; overlapping one of them still merges to the newer value.
• single_run_read_drops_retracts_hidden_by_legacy_metadata: a two-file run whose second file holds a DELETE with delete_row_count = 0 reads [1, 2] under deduplicate and first-row with ignore-delete, and errors under first-row.
• dv_merge_on_read_loads_deletion_vectors_per_file asserted the first batch had 128 rows, the merge's output size. Its delete-free files now stream unmerged, so it checks the batch is non-empty; the lazy DV loading it covers is unchanged.

Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, integration, datafusion, python, go).

API and Format

None.

Documentation

None.

[JeonDaehong]

@JingsongLi @leaves12138 could you take a look when you have time? I'm looking at running Paimon with the Rust reader for work, so read-path performance matters to me, and I'd like to keep improving it here.

[JingsongLi]

Requirement fit: SUPPORTED. Implementation: FINDINGS at c8b5893. Avoiding unnecessary merging has end-to-end read value, but the shortcut regresses reads of historical native retract files. Local verification: 55 existing KV/single-run/user-sequence/first-row/DV tests pass. An additional public-API regression test writes actual Parquet, commits historical metadata, and reads a two-file nonraw split through the ordinary table planner: it fails on this head (returns [1, 2, 3]) and passes unchanged on base 46e6871 (returns [1, 2]). Details inline. Formatting and clean integration with current main were also checked.

[JingsongLi]

[P2] Preserve retract filtering for historical native files

delete_row_count == Some(0) does not prove these files contain only add rows. Before 13e370a (#318, May 20), the Rust KV writer accepted _VALUE_KIND but persisted Some(0) unconditionally; normal cross-partition writes also generated DELETE batches. Those files have no writer-version guard here. I reproduced a full table read with two disjoint files in one nonraw split: file A has INSERT keys 1 and 2; file B has a DELETE for key 3 with that historical metadata. Base 46e6871 returns [1, 2], while this head returns [1, 2, 3], exposing the deleted row. The test asserts that the planner marks this two-file split nonraw, so this is an expansion beyond the pre-existing single-file raw path. Please preserve a cheap RowKind filter on the direct path, or restrict the optimization to files whose add-only invariant can actually be verified, and cover this legacy case.

[JeonDaehong]

Good catch, thanks. Fixed in 476312c: the shortcut no longer relies on delete_row_count. Each unmerged batch drops retract rows the same way the merge path handles a lone version of a key. First-row without ignore-delete still errors on them. single_run_read_drops_retracts_hidden_by_legacy_metadata covers your two-file case for deduplicate, first-row and first-row + ignore-delete. Without the filter it returns [1, 2, 3].

While running CI I also found that read_with_merge_function took the shortcut, which dropped -D rows from $audit_log (test_audit_log_system_table_keeps_row_kinds_and_sequence_numbers). The shortcut now applies only to read, so custom merge functions always merge.

Rebased onto current main and re-measured on one instance, alternating: main 157 ms vs this head 60 ms. Ranges don't overlap.

[JingsongLi]

Requirement fit: SUPPORTED. Implementation: FINDINGS at 476312c. The previous historical retract-row issue is fixed, and the custom audit-log path now retains merging. One additional compatibility regression remains for historical aggregation files after an engine change that the Rust catalog currently accepts.

Local validation: 42 KV tests and 35 table-level single-run/user-sequence/first-row/audit-log/DV tests passed. An additional public-API probe reconstructs the pre-#839 aggregation writer's valid Parquet layout, commits two disjoint files through FileSystemCatalog, verifies the aggregation control, then persists ALTER merge-engine=deduplicate and reloads the table. It asserts one two-file non-raw split. This head returns [(1,10),(1,11),(2,20)]; the unchanged probe passes on exact merge-base cea0caa with [(1,11),(2,20)]. Temporary probe removed after validation; formatting and diff checks pass.

The existing CI check failure is an unchanged row_variant_interop/README.md license-header issue.

[JingsongLi]

[P2] Preserve merging for historical files after an engine change

The current read engine does not establish that every file in this run has unique keys. Before #839 (baac87c), the Rust aggregation writer deliberately retained multiple same-key versions in one file. The Rust filesystem catalog currently accepts ALTER merge-engine=deduplicate on such a table. I reproduced that real persisted ALTER/reload with a legacy file containing (key=1,seq=0,value=10) and (1,1,11), plus a disjoint key=2 file. The normal planner produces a two-file non-raw split, but this shortcut returns both key=1 rows. Exact base cea0caa returns only (1,11), as Deduplicate requires; the original aggregation control correctly sums to (1,21). Please retain merging when the files' historical schema/producer does not establish key uniqueness, or explicitly reject the transition before using this shortcut, and cover this accepted Rust ALTER lifecycle.

[JeonDaehong]

Thanks for reproducing it (ca5eb6f). The shortcut now also checks the schema each file was written with, and only skips the merge when that schema's merge engine is deduplicate or first-row. If a file comes from the aggregation schema, the section merges as before. I added the aggregation → deduplicate ALTER case with a two-version file. Without the check it returns key 1 twice, now (1, 11) and (2, 20).

Same clippy note as on #1145: the unused BinaryRow import in table_upsert.rs comes from #1198.