Skip to content

[rust] Allow configuring remote log download directory - #4603

Open
spoon-lz wants to merge 1 commit into
apache:mainfrom
spoon-lz:codex/remote-log-download-directory-20261012
Open

spoon-lz wants to merge 1 commit into
apache:mainfrom
spoon-lz:codex/remote-log-download-directory-20261012

Conversation

@spoon-lz

@spoon-lz spoon-lz commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Fixes #4602.

Allow users to configure the parent directory for remote log segment downloads in the Rust client and its Python and C++ bindings.

The default parent directory is the OS temporary directory plus /fluss, matching the Java client's client.scanner.io.tmpdir default. This change aligns remote log download directory configuration with Java and covers remote log segments, including changelog scanning for KV tables. KV snapshot downloads are outside its scope.

Brief change log

  • Add scanner_io_tmp_dir to the Rust configuration and expose it in Python and C++.
  • Reserve a unique remote-logs-<UUID> subdirectory for each scanner and create directories when an actual download starts.
  • Handle directory creation failures through the existing asynchronous download retry and terminal error flow.
  • Clean up the scanner's subdirectory after its downloads release ownership, preserving the parent directory.
  • Add configuration and directory lifecycle tests, and update both API documentation trees.

Tests

  • Rust workspace unit-test gate: 858 tests passed.
  • C++ FFI configuration conversion tests: 24 tests passed.
  • Python configuration tests: 5 tests passed.
  • Python stubtest with the existing allowlist: passed.
  • Rust workspace Clippy, Rustdoc, formatting, and diff checks: passed.

Tests cover default directory selection, lazy creation, paths containing spaces, directory isolation and cleanup, invalid paths, Unix permission errors, and download retry/error propagation.

A live OSS or Fluss-cluster download was not exercised. Subsequent edits only simplified documentation and changed test path strings from /data to /tmp; the full suite was not rerun after those edits.

API and Format

Adds the following configuration entry points:

  • Rust: Config::scanner_io_tmp_dir
  • Python: Config.scanner_io_tmp_dir and the scanner.io.tmpdir configuration key
  • C++: Configuration::scanner_io_tmp_dir

Rust and Python use None for the default; C++ uses an empty string. No storage format, wire protocol, or RPC changes.

Documentation

Updated the Rust README configuration table and the Rust, Python, and C++ API references in both documentation trees.

Generative AI disclosure

Yes — Codex was used to prepare the implementation, tests, and documentation.

Add a configurable parent for remote log downloads, matching the Java client capability while preserving the Rust OS temporary-directory default. Each scanner owns an isolated temporary subdirectory and retains the existing cleanup lifecycle.

Expose the option in Python and C++ and document it in both API trees. Cover directory creation, isolation, cleanup, invalid paths, Unix permission errors, and binding configuration.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[rust]Support configuring the download directory for remote log segments in the Rust client

1 participant