HBASE-30357 OpenRegionProcedure#restoreSucceedState ignores persisted transitionCode, forcing OPEN even after a real FAILED_OPEN - #8622
Open
mnpoonia wants to merge 1 commit into
Conversation
mnpoonia
force-pushed
the
HBASE-30357-restoreSucceedState-failed-open
branch
from
September 5, 2026 05:58
2203eb4 to
e2cde5d
Compare
Contributor
Author
|
@virajjasani @apurtell @Apache9 Can you please help with review. |
… transitionCode, forcing OPEN even after a real FAILED_OPEN On master-failover restore, RegionRemoteProcedureBase.stateLoaded() calls restoreSucceedState() for any region whose remote-open report was already persisted with state REGION_REMOTE_PROCEDURE_REPORT_SUCCEED, but before this change OpenRegionProcedure ignored the real persisted TransitionCode and always forced the region to OPEN. If the RS had actually reported FAILED_OPEN and the master crashed before persisting that to hbase:meta, the region would come back as a "phantom" OPEN region on restart: no procedure watching it, no automatic retry, and invisible to the RegionInTransition tracker since OPEN is the only non-RIT terminal state. Widen RegionRemoteProcedureBase#restoreSucceedState to also receive the persisted TransitionCode, and make OpenRegionProcedure branch on it: on FAILED_OPEN, call AssignmentManager#regionFailedOpen(regionNode, false), mirroring what the live (non-restart) reportTransition path already does. This leaves the region non-OPEN so TransitRegionStateProcedure#confirmOpened sees it and drives the normal retry loop instead of finishing silently. CloseRegionProcedure's override is updated to match the new signature but ignores the parameter, since CLOSE has no failure variant. Add TestOpenRegionProcedureRestoreFailedOpen, which reproduces the restore timing directly (invoking the same package-private stateLoaded() hook that a real master restart triggers) and asserts the region is not left in OPEN state after a FAILED_OPEN report.
mnpoonia
force-pushed
the
HBASE-30357-restoreSucceedState-failed-open
branch
from
September 5, 2026 06:32
ded91c9 to
89a3dfb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
OpenRegionProcedure#restoreSucceedState()is called on master-failover restore, onceper region, via
RegionRemoteProcedureBase#stateLoaded(). It unconditionally forced theregion into
OPEN, regardless of the actual persistedtransitionCode, which can beFAILED_OPEN. The method's signature only receivedseqId, nottransitionCode, so itwas structurally unable to branch on the real outcome.
Concretely: an RS reports
FAILED_OPEN; the master persistsstate=REPORT_SUCCEED, transitionCode=FAILED_OPENon theOpenRegionProcedure, but thein-memory
RegionState.Stateis left untouched (stillOPENING, sinceAssignmentManager#regionFailedOpen(regionNode, false)on the live path does not updateRegionState.State). If the master crashes/restarts before this reachesconfirmOpened()and before
hbase:metais updated,restoreSucceedState()sees the region isn'tOPENyet and force-transitions it to
OPENanyway, then persists that (false)OPENstate tohbase:meta. There is no rollback anywhere in this procedure chain (by design -forward-only), so nothing downstream can detect or correct it; the region silently looks
healthy in meta while no RegionServer is actually serving it.
By contrast,
CloseRegionProcedure#restoreSucceedState()is safe doing the equivalentunconditional force, because CLOSE has no failure-variant transition code at the master
side (
checkTransition/updateTransitionWithoutPersistingToMetaboth asserttransitionCode == CLOSED).This is not a regression - the logic is unchanged (modulo spotless formatting) since it
was introduced in HBASE-22365 (2019-05-10).
The fix
RegionRemoteProcedureBase#restoreSucceedState()to also receive the persistedtransitionCode, passed through fromstateLoaded().OpenRegionProcedure#restoreSucceedState()now branches:FAILED_OPENcallsAssignmentManager#regionFailedOpen(regionNode, false), mirroring exactly what the livereportTransition/updateTransitionWithoutPersistingToMetapath already does for thesame transition code; the existing
OPENEDhandling is unchanged.CloseRegionProcedure#restoreSucceedState()accepts the new parameter and ignores it,since CLOSE has no failure variant.
After the fix, a restore-time
FAILED_OPENputs the region back into the sameretryable path (
OPENING, cleared location) thatTransitRegionStateProcedure#confirmOpened()already drives on the live path - the region gets reassigned/retried through the normal
flow instead of being falsely marked
OPEN.Why are the changes needed?
To prevent a region from being silently, durably marked
OPENinhbase:metaafter amaster restart, when in reality no RegionServer opened it. Since HBase has no rollback
mechanism for these forward-only assignment procedures, this bug is otherwise
unrecoverable except by manual detection and intervention.
Does this PR introduce any user-facing change?
No.
Is there a corresponding Apache JIRA?
Yes: HBASE-30357
How was this patch tested?
Added
TestOpenRegionProcedureRestoreFailedOpen, extendingTestAssignmentManagerBase. Acustom mock RS executor reports
FAILED_OPENon the first open attempt, then - whileholding the
RegionStateNodelock, simulating the point right after a crash where thechild procedure has not yet resumed its own
execute()- directly invokes theTransitRegionStateProcedure#stateLoaded()hook that a real master restart would trigger,and records the region's state immediately after. Confirmed the test fails against
unmodified code with the exact predicted
OPENstate, and passes after the fix. Also ranTestAssignmentManager,TestTransitRegionStateProcedure,TestOpenRegionProcedureHang,TestOpenRegionProcedureBackoff,TestRollbackSCP, andTestSCPGetRegionsRacewith noregressions.