Repository navigation
fix(store): keep GET /v1/partitions working on follower stores - #3232
Conversation
PartitionAPI.getPartitions built each raft group's conf through PartitionEngine.getCurrentConf(), which calls jraft Node.listPeers(). jraft only allows that on the leader, so the endpoint failed with a 500 "Not leader" at the first group the store follows. Every store follows some groups once replicas > 1, so the endpoint failed cluster-wide. Move the conf into the existing isLeader() block that already guards peers and learners. Followers now report conf as null, matching peers and learners. Also add TODO comments at dead code found nearby, with no behavior change: a dead return and an unused DashResponse line in PartitionAPI, a commented-out changeShards block, an unused groupId with an unlogged error in doBlankTaskSync and a silent replicator onError in PartitionEngine, and HgStoreNodeServiceTest, whose tests are all commented out. Fixes apache#3230
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3232 +/- ##
============================================
+ Coverage 41.35% 41.36% +0.01%
- Complexity 7299 7302 +3
============================================
Files 802 802
Lines 69688 69688
Branches 9291 9291
============================================
+ Hits 28816 28825 +9
+ Misses 37576 37575 -1
+ Partials 3296 3288 -8 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
517aa51 to
fc017f8
Compare
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: Moving setConf inside the isLeader() block fixes the follower 500 from #3230, and the new PartitionAPITest reproduces the jraft contract and runs in the store CI job. One non-blocking note: the isLeader() check and the three leader-only calls are not atomic, so a leader step-down during the request can still return 500. Evidence: read PartitionAPI.getPartitions and PartitionEngine.getCurrentConf/isLeader at fc017f8; read NodeImpl.listPeers and isLeader(false) in jraft-core 1.3.13 sources; confirmed PartitionAPITest ran and passed in each hg-store-test step of the store job log; all workflows on this head passed, only codecov/project reports -0.01%.
engine.isLeader() calls jraft isLeader(false), which reads the node state without the node lock. getCurrentConf(), listPeers() and listLearners() then take the lock and throw "Not leader" if the group stepped down in between, for example during a PD leader balance, which still turned GET /v1/partitions into a 500. Read the three values inside one try block and set them only if all succeed. On IllegalStateException, log it and leave conf, peers and learners null, the same as for a follower.
imbajin
left a comment
There was a problem hiding this comment.
Blocking: no actionable issue found after two independent global reviews, one adversarial review and synthesis. Score: 9.2/10. Exact-head Store CI executed and passed both PartitionAPITest cases and all nine node-module tests; all 22 checks passed. The separate single-partition endpoint race is unchanged baseline behavior, outside this fix. No local multi-Store cluster rerun is claimed.
Reserve the chart-owned checksum/ podAnnotations prefix on every component: user annotations render after the chart's own and the last duplicate key wins, so a fixed value would pin the rotation checksum. Refuse server.securityContext.readOnlyRootFilesystem=true, mirroring the Hubble guard, because the Server wrapper rewrites rest-server.properties inside the image. Reject updateStrategy.rollingUpdate options together with type OnDelete in the schema for PD and Store; Kubernetes refuses that combination at apply time, which would otherwise surface mid-upgrade. Unit tests cover the template guards and the CI reject-invalid-values step covers the schema constraint. Use the get-with-default pattern for the $exposed NetworkPolicy check, matching the rest of validateValues. Remove the constant $pdMeta and $wrapper indirection from the Server Deployment, and remove the unused server.restServer.minFreeMemory / batchMaxWriteThreads knobs end to end (template, values, schema, README); the chart is unreleased, so nothing depends on them. helm template output on the default, single and cluster presets is byte-identical before and after. Refresh the docs: the server.testResources default in the README matches values.yaml again, and the runbook follows apache#3232, apache#3233 and apache#3234 (merged 2026-09-24). The empty-PVC Store retirement was re-proven on images built from master at dbb6663: all 12 groups converged onto the replacement 1 s after Tombstone and patrol, with 0 acknowledged writes lost, so the Disaster Recovery section documents the working procedure with a version caveat instead of a prohibition.
Purpose of the PR
GET /v1/partitionson a Store returns 500IllegalStateException: Not leaderas soon as it reaches a raft group the Store follows.PartitionAPI.getPartitionscallsPartitionEngine.getCurrentConf()for every group, andgetCurrentConf()builds theConfigurationfromraftNode.listPeers()andraftNode.listLearners(). jraft'sNodeImplthrowsNot leaderfrom both unless the node is the leader. With more than one replica per group every Store follows some groups, so the endpoint fails on every Store in the cluster.Main Changes
PartitionAPI.getPartitions: moveraft.setConf(...)inside theengine.isLeader()block that already guardssetPeersandsetLearners. On followersconfis nownull, likepeersandlearners; all other fields (role,leader,term,logIndex, partitions and metrics) are filled as before.isLeader()reads the jraft state without the node lock, so a group can step down between the check and the three leader-only calls. They now run in onetryblock and are set only if all three succeed; onIllegalStateExceptionthe group is logged and reported like a follower instead of failing the whole response.PartitionEngine.getCurrentConf()itself is unchanged, since its other callers (changePeers,onLeaderStart,onStartFollowing) are outside this endpoint.DashResponseline inPartitionAPI; a commented-outchangeShardsblock, an unusedgroupIdwith an unlogged error indoBlankTaskSync, and a silent replicatoronErrorinPartitionEngine; andHgStoreNodeServiceTest, whose@Testannotations are all commented out.PartitionAPITestinhg-store-node/src/test, next to the existingScanUtilTest. It mocks one follower group and one leader group. The follower'sNode.listPeers(),Node.listLearners()andgetCurrentConf()throwIllegalStateException("Not leader"), which is how jraft behaves. The test checks that both groups are listed, that the follower hasconf,peersandlearnersset to null, and that the leader still reports its conf and peers.testGetPartitionsWhenLeaderStepsDowncovers the step-down race:isLeader()returns true while the node methods throw.Verifying these changes
PartitionAPITest#testGetPartitionsOnFollowerandPartitionAPITest#testGetPartitionsWhenLeaderStepsDown, run with JDK 11:mvn test -pl hugegraph-store/hg-store-node -am -Dtest=PartitionAPITest \ -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=falseWithout the
PartitionAPIchange it fails with the error from the issue:With the change, together with the other
hg-store-nodetests (-Dtest='PartitionAPITest,ScanUtilTest,JraftMetricsTest'):testGetPartitionsWhenLeaderStepsDownfails on fc017f8 (before thetryblock) withIllegalState Not leaderand passes on 8dd3665.mvn editorconfig:check -pl hugegraph-store/hg-store-nodepasses.I have not yet re-run this against a live 3-Store cluster. The verification above is the unit test only.
Does this PR potentially affect the following parts?
Store REST
GET /v1/partitionsnow returns 200 on a Store that follows some groups instead of 500. For those groupsconfisnullin the response. Nothing changes for groups the Store leads.Documentation Status
Doc - TODO: required documentation is pending; complete it before merging.Doc - Done: documentation is included here or linked below.Doc - No Need: no user-visible documentation is affected.Documentation files in this PR or paired hugegraph-doc PR: