Skip to content

fix(store): keep GET /v1/partitions working on follower stores - #3232

Merged
imbajin merged 2 commits into
apache:masterfrom
hugegraph:fix/store-partitions-follower-3230
Sep 24, 2026
Merged

imbajin merged 2 commits into
apache:masterfrom
hugegraph:fix/store-partitions-follower-3230

Conversation

@bitflicker64

@bitflicker64 bitflicker64 commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Purpose of the PR

GET /v1/partitions on a Store returns 500 IllegalStateException: Not leader as soon as it reaches a raft group the Store follows. PartitionAPI.getPartitions calls PartitionEngine.getCurrentConf() for every group, and getCurrentConf() builds the Configuration from raftNode.listPeers() and raftNode.listLearners(). jraft's NodeImpl throws Not leader from both unless the node is the leader. With more than one replica per group every Store follows some groups, so the endpoint fails on every Store in the cluster.

Main Changes

  • PartitionAPI.getPartitions: move raft.setConf(...) inside the engine.isLeader() block that already guards setPeers and setLearners. On followers conf is now null, like peers and learners; all other fields (role, leader, term, logIndex, partitions and metrics) are filled as before.
  • isLeader() reads the jraft state without the node lock, so a group can step down between the check and the three leader-only calls. They now run in one try block and are set only if all three succeed; on IllegalStateException the group is logged and reported like a follower instead of failing the whole response.
  • PartitionEngine.getCurrentConf() itself is unchanged, since its other callers (changePeers, onLeaderStart, onStartFollowing) are outside this endpoint.
  • TODO comments, no behavior change, at dead code found nearby: a dead return and an unused DashResponse line in PartitionAPI; a commented-out changeShards block, an unused groupId with an unlogged error in doBlankTaskSync, and a silent replicator onError in PartitionEngine; and HgStoreNodeServiceTest, whose @Test annotations are all commented out.
  • New PartitionAPITest in hg-store-node/src/test, next to the existing ScanUtilTest. It mocks one follower group and one leader group. The follower's Node.listPeers(), Node.listLearners() and getCurrentConf() throw IllegalStateException("Not leader"), which is how jraft behaves. The test checks that both groups are listed, that the follower has conf, peers and learners set to null, and that the leader still reports its conf and peers. testGetPartitionsWhenLeaderStepsDown covers the step-down race: isLeader() returns true while the node methods throw.

Verifying these changes

  • Trivial rework / code cleanup without any test coverage. (No Need)
  • Already covered by existing tests, such as (please modify tests here).
  • Need tests and can be verified as follows:
    • PartitionAPITest#testGetPartitionsOnFollower and PartitionAPITest#testGetPartitionsWhenLeaderStepsDown, run with JDK 11:

      mvn test -pl hugegraph-store/hg-store-node -am -Dtest=PartitionAPITest \
        -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false

      Without the PartitionAPI change it fails with the error from the issue:

      [ERROR]   PartitionAPITest.testGetPartitionsOnFollower:52 » IllegalState Not leader
      [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
      

      With the change, together with the other hg-store-node tests (-Dtest='PartitionAPITest,ScanUtilTest,JraftMetricsTest'):

      Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 - in org.apache.hugegraph.store.node.metrics.JraftMetricsTest
      Tests run: 6, Failures: 0, Errors: 0, Skipped: 0 - in org.apache.hugegraph.store.node.grpc.ScanUtilTest
      Tests run: 2, Failures: 0, Errors: 0, Skipped: 0 - in org.apache.hugegraph.store.node.controller.PartitionAPITest
      Tests run: 9, Failures: 0, Errors: 0, Skipped: 0
      BUILD SUCCESS
      

      testGetPartitionsWhenLeaderStepsDown fails on fc017f8 (before the try block) with IllegalState Not leader and passes on 8dd3665.

    • mvn editorconfig:check -pl hugegraph-store/hg-store-node passes.

    • I have not yet re-run this against a live 3-Store cluster. The verification above is the unit test only.

Does this PR potentially affect the following parts?

Store REST GET /v1/partitions now returns 200 on a Store that follows some groups instead of 500. For those groups conf is null in the response. Nothing changes for groups the Store leads.

Documentation Status

  • Doc - TODO: required documentation is pending; complete it before merging.
  • Doc - Done: documentation is included here or linked below.
  • Doc - No Need: no user-visible documentation is affected.

Documentation files in this PR or paired hugegraph-doc PR:

PartitionAPI.getPartitions built each raft group's conf through
PartitionEngine.getCurrentConf(), which calls jraft Node.listPeers().
jraft only allows that on the leader, so the endpoint failed with a 500
"Not leader" at the first group the store follows. Every store follows
some groups once replicas > 1, so the endpoint failed cluster-wide.

Move the conf into the existing isLeader() block that already guards
peers and learners. Followers now report conf as null, matching peers
and learners.

Also add TODO comments at dead code found nearby, with no behavior
change: a dead return and an unused DashResponse line in PartitionAPI,
a commented-out changeShards block, an unused groupId with an unlogged
error in doBlankTaskSync and a silent replicator onError in
PartitionEngine, and HgStoreNodeServiceTest, whose tests are all
commented out.

Fixes apache#3230
@codecov

codecov Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 41.36%. Comparing base (83ef9f3) to head (8dd3665).

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3232      +/-   ##
============================================
+ Coverage     41.35%   41.36%   +0.01%     
- Complexity     7299     7302       +3     
============================================
  Files           802      802              
  Lines         69688    69688              
  Branches       9291     9291              
============================================
+ Hits          28816    28825       +9     
+ Misses        37576    37575       -1     
+ Partials       3296     3288       -8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bitflicker64
bitflicker64 force-pushed the fix/store-partitions-follower-3230 branch from 517aa51 to fc017f8 Compare September 23, 2026 13:42

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: Moving setConf inside the isLeader() block fixes the follower 500 from #3230, and the new PartitionAPITest reproduces the jraft contract and runs in the store CI job. One non-blocking note: the isLeader() check and the three leader-only calls are not atomic, so a leader step-down during the request can still return 500. Evidence: read PartitionAPI.getPartitions and PartitionEngine.getCurrentConf/isLeader at fc017f8; read NodeImpl.listPeers and isLeader(false) in jraft-core 1.3.13 sources; confirmed PartitionAPITest ran and passed in each hg-store-test step of the store job log; all workflows on this head passed, only codecov/project reports -0.01%.

engine.isLeader() calls jraft isLeader(false), which reads the node
state without the node lock. getCurrentConf(), listPeers() and
listLearners() then take the lock and throw "Not leader" if the group
stepped down in between, for example during a PD leader balance, which
still turned GET /v1/partitions into a 500.

Read the three values inside one try block and set them only if all
succeed. On IllegalStateException, log it and leave conf, peers and
learners null, the same as for a follower.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no actionable issue found after two independent global reviews, one adversarial review and synthesis. Score: 9.2/10. Exact-head Store CI executed and passed both PartitionAPITest cases and all nine node-module tests; all 22 checks passed. The separate single-partition endpoint race is unchanged baseline behavior, outside this fix. No local multi-Store cluster rerun is claimed.

@imbajin
imbajin merged commit db73843 into apache:master Sep 24, 2026
22 checks passed
@imbajin
imbajin deleted the fix/store-partitions-follower-3230 branch September 24, 2026 14:25
bitflicker64 added a commit to hugegraph/hugegraph that referenced this pull request Sep 24, 2026
Reserve the chart-owned checksum/ podAnnotations prefix on every
component: user annotations render after the chart's own and the last
duplicate key wins, so a fixed value would pin the rotation checksum.
Refuse server.securityContext.readOnlyRootFilesystem=true, mirroring the
Hubble guard, because the Server wrapper rewrites rest-server.properties
inside the image. Reject updateStrategy.rollingUpdate options together
with type OnDelete in the schema for PD and Store; Kubernetes refuses
that combination at apply time, which would otherwise surface
mid-upgrade. Unit tests cover the template guards and the CI
reject-invalid-values step covers the schema constraint.

Use the get-with-default pattern for the $exposed NetworkPolicy check,
matching the rest of validateValues. Remove the constant $pdMeta and
$wrapper indirection from the Server Deployment, and remove the unused
server.restServer.minFreeMemory / batchMaxWriteThreads knobs end to end
(template, values, schema, README); the chart is unreleased, so nothing
depends on them. helm template output on the default, single and
cluster presets is byte-identical before and after.

Refresh the docs: the server.testResources default in the README matches
values.yaml again, and the runbook follows apache#3232, apache#3233
and apache#3234 (merged 2026-09-24). The empty-PVC Store retirement was
re-proven on images built from master at dbb6663: all 12 groups
converged onto the replacement 1 s after Tombstone and patrol, with 0
acknowledged writes lost, so the Disaster Recovery section documents the
working procedure with a version caveat instead of a prohibition.
vaijosh pushed a commit to vaijosh/hugegraph that referenced this pull request Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Store GET /v1/partitions answers 500 "Not leader" on any Store that follows at least one raft group

2 participants