Skip to content

[Storage] Add optional WAL and checkpoint format extension interfaces - #19002

Merged
HTHou merged 3 commits into
apache:masterfrom
HTHou:codex/wal-format-extension
Oct 26, 2026
Merged

HTHou merged 3 commits into
apache:masterfrom
HTHou:codex/wal-format-extension

Conversation

@HTHou

@HTHou HTHou commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Description

Introduce an optional extension boundary for WAL and checkpoint formats without adding a new built-in file format or an encryption implementation to Apache IoTDB.

Format Dispatch And Ownership

  • Add WALFormatProvider and WALFileContext for format detection, per-file read/write contexts, segment decoding, metadata sealing and resource release.
  • Load at most one optional provider through ServiceLoader. With no provider installed, writers continue to use the existing V2 checkpoint and V3 WAL formats.
  • Use WALFileVersion.EXTENDED only as an internal dispatch marker. The provider owns its on-disk magic, headers and payload layout; no V4 layout, cryptographic algorithm, wallet, key-management implementation or production service registration is included.
  • Reject unavailable formats in the reserved versioned WAL magic namespace instead of interpreting them as V1.

Recovery And Cleanup

  • Add a cause-aware WALFormatException for failures that must not become tail-truncation recovery, metadata reconstruction or file quarantine.
  • Run provider validation before WAL/checkpoint replay or destructive cleanup, propagate failures from asynchronous recovery tasks, and expose an optional recovery-failure diagnostic callback.
  • Release decoded segment buffers before their per-file context, retain logical seek/resume support, and close caller-owned channels when provider initialization fails.
  • Preserve the built-in format codecs and existing recovery diagnostics.

Tests

  • Add provider selection, magic detection, error propagation and cleanup-order tests.
  • Exercise writer/reader/checkpoint dispatch, seek/resume, metadata failures, initialization cleanup and startup recovery with a mock provider in an isolated classloader. The mock provider is not globally registered and cannot conflict with downstream providers.
  • Full-reactor English build with targeted WAL tests: 81 tests passed, with 7 additional extension scenarios checked inside the isolated-runtime test.
  • Full-reactor Chinese test-compile: passed.
  • Spotless, Checkstyle and git diff --check: passed.

No cluster integration test or production encryption-provider test is claimed by this PR.

Checklist

  • Self-reviewed the format boundary and failure paths.
  • Added Javadocs for the extension interfaces and lifecycle contract.
  • Added unit tests and Apache license headers.
  • Kept encryption implementations and concrete extension file formats out of Apache IoTDB.

Comment on lines +57 to +58
throw new IllegalStateException(
StorageEngineMessages.EXCEPTION_UNSUPPORTED_OR_UNAVAILABLE_WAL_FORMAT_4629CFE4);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May include the magic or the provider in the exception.

Comment on lines +100 to +102
if (Holder.PROVIDER == null || !Holder.PROVIDER.isEnabled()) {
throw WALFormatException.unsupported();
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May add some detail

HTHou added 2 commits October 26, 2026 16:41
Leave extension segment allocation ownership with the provider, propagate fatal WAL repair and startup failures, and complete task-owned recovery listeners before rethrowing. Cover pooled direct buffers, repair write/seal failures, listener completion and storage readiness in isolated provider tests.
@HTHou
HTHou merged commit 9eaf5f7 into apache:master Oct 26, 2026
39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants