PHOENIX-7874 Eliminate replicating index mutations; regenerate on standby - #2566
Merged
tkhurana merged 8 commits intoJul 21, 2026
Merged
Conversation
…ndby Index tables no longer generate their own replication-log records. Instead the active ships only the data-table record plus a per-(row, ts) PRE_IMAGE (the prior row state it observed at lock time), and the standby regenerates every index entry from that record on the replay path. This removes the out-of-order replay hazard: because each replicated batch carries the exact prior-row snapshot it was built against, the standby derives each (row, ts) group's next state from its own shipped pre-image instead of a region scan, so shard files can be replayed in any order (or in parallel) and each group is an independent reproduction of the active preBatchMutate. It also fixes CDC/global/local index rowkeys that embed the encoded data-table region name (PARTITION_ID), which differs between clusters -- the standby rebuilds the rowkey with its own region name. Main changes: - Cell-oriented replication log format (codec + record + MutationCellGrouper) carrying data cells plus a METAFAMILY pre-image cell per row. - IndexRegionObserver: active-side pre-image capture; a forked standby replay path (prepareReplicatedIndexMutations) that groups by (row, ts) and builds index updates from the shipped pre-image. - PreImageLocalTable: a LocalHBaseState that serves the local-index builder's prior-row-state from the shipped pre-image instead of a region scan; new IndexBuildManager.getIndexUpdates overload accepts it. - ReplicationLogProcessor: replay a record as one indivisible batch; do not retry DoNotRetryIOException. Coverage: global, local, uncovered, atomic, conditional-TTL, CDC (plain and behind an EVENTUAL secondary index), concurrent/out-of-order replay, and unit tests for (row, ts) grouping isolation and Put/Delete decomposition.
Thread dataTableName into the standby pre-image decode/lookup so a contract violation names the offending table. decodePreImage and buildReplicatedRowGroups become instance methods reading the dataTableName field instead of taking it as a parameter; add a @VisibleForTesting IndexRegionObserver(String) constructor alongside the public no-arg one HBase loads reflectively. PreImageLocalTable takes the table name in its constructor. Doc notes: decodePreImage javadoc explains the schema-skew case as the one legitimate missing-pre-image path and why failing loud is correct; PreImageLocalTable and MutationCellGrouper get class-level thread-safety notes.
preBatchMutateWithExceptions interleaved the active write path and the standby replay path in one method behind five scattered isReplication guards, so the standby path was implicit (defined by what active-only blocks skip). That fragility surfaced a real bug: getCurrentRowStates was gated only on index flags, not isReplication, so the standby scanned the data-table region under lock for prior row state -- the out-of-order-unsafe read the per-row PRE_IMAGE exists to replace -- and then discarded the result. - Skip getCurrentRowStates on replication: prepareReplicatedIndexMutations reads only the group's PRE_IMAGE/nextState, never dataRowStates, so the scan and its pendingRows/lastConcurrentBatchContext bookkeeping are inert on the standby. - Dispatch to a new preBatchMutateReplication right after currentPhase = PRE and return, so the active body runs only on the active. Drop the five now-dead isReplication guards there. - Extract the shared two-phase commit (preparePreIndexMutations -> unlock -> doPre -> lock -> [wait] -> post) into prepareAndCommitGlobalIndexUpdates so the lockstep protocol stays single-sourced; the active/standby fork lives inside preparePreIndexMutations. The concurrent-batch wait is now reached only when lastConcurrentBatchContext is set, which never happens on the standby. Verified: IndexRegionObserverReplayTest 16/16, ReplicationLogGroupIT 15/15 (incl. testConcurrentUpserts, concurrent same-row standby replay).
… cell filter Both replication paths now drop local-index (L#) cells the same way. The WAL-restore path (replicateEditOnWALRestore) previously shipped the persisted WAL edit verbatim, leaking the L# cells HBase merged into the data mutation's family map -- a recovery-path-only leak no IT exercised. It now filters the cell stream through the shared isReplicableCell predicate. The synchronous path (replicateMutations) reads the batch's now-final cells from miniBatchOp in POST and drops L# cells with a single family-key check per family list, then appends the pre-image cells from WAL slot 0 (filtered through isReplicableCell). capturePreImageCells is now the sole producer of pre-image cells, written only to the WAL edit and read back by both paths, so they ship byte-identical output with no re-derivation. Removes the timing-based snapshot machinery (captureReplicationCells and the replicationCellsByRow field) that the sync path used to exclude L# cells, since the WAL-restore path could not use it.
extractReplicationAttributes must not invent an empty INDEX_UUID for a table without indexes -- an empty UUID would push the standby down the server-cache resolution branch and fail with INDEX_METADATA_NOT_FOUND instead of recognizing the table as non-indexed. Existing coverage only exercised the UUID-present and zero-attribute cases; this locks the contract for a mutation carrying schema/table metadata but no UUID.
…t client attr Remove INDEX_UUID from REPLICATION_ATTR_KEYS and stop normalizing it in extractReplicationAttributes. Whether the standby regenerates indexes is now decided by the active from its own resolved index maintainers (BatchMutateContext.hasIndex()), which stamps an empty INDEX_UUID only for indexed tables via the new MutationCellGrouper.stampIndexAttribute utility. Both the synchronous (replicateMutations) and WAL-restore paths route through buildReplicationAttributes so a non-indexed table never ships an empty UUID that would fail on the standby with INDEX_METADATA_NOT_FOUND.
tkhurana
force-pushed
the
eliminate-index-replication-v2
branch
from
July 17, 2026 06:15
8d74b49 to
06e8aca
Compare
The per-row pre-image cell lives under WALEdit.METAFAMILY. Rename its qualifier from _PhoenixPreImage to PHOENIX::PRE_IMAGE to mirror HBase's own METAFAMILY qualifiers (HBASE::COMPACTION, HBASE::FLUSH, ...) and stay clear of the reserved HBASE:: space / any future qualifier-prefix enforcement (HBASE-8457). Family stays METAFAMILY so HBase continues to skip the cell on recovered-edits replay (the skip keys on family, not qualifier) -- the pre-image must never land in a data store.
virajjasani
approved these changes
Jul 21, 2026
virajjasani
left a comment
virajjasani
left a comment
Contributor
There was a problem hiding this comment.
+1, left one question
Comment on lines
+55
to
+56
| conf1.setBoolean(PHOENIX_INDEX_CDC_CONSUMER_ENABLED, false); | ||
| conf2.setBoolean(PHOENIX_INDEX_CDC_CONSUMER_ENABLED, false); |
Contributor
There was a problem hiding this comment.
do we have coverage for checking if eventually consistent indexes also have rows matching after waiting for some time?
Contributor
Author
There was a problem hiding this comment.
@virajjasani Added the test for eventually consistent index
… replication ReplicationLogGroupEventualIndexIT previously verified only that the CDC index regenerates on the standby; the downstream eventual secondary index was left unverified because the IndexCDCConsumer was disabled on both clusters. Enable the consumer on both, trim its startup/poll/buffer timings, and after replay wait for convergence and assert the two secondary-index tables are byte-identical. The consumer runs independently per cluster (index mutations are never replicated), so cluster 2's copy is materialized solely by its own consumer draining the regenerated CDC index. The index is global+covered (no partition_id) and its cells carry the data row's timestamp, which replication preserves, so exact cross-cluster cell equality must hold.
Himanshu-g81
pushed a commit
to Himanshu-g81/phoenix
that referenced
this pull request
Jul 22, 2026
Brings in PHOENIX-7874 (regenerate index mutations on standby, apache#2566) and PHOENIX-7938 (round-align the replay consistency point, apache#2570). Conflict: phoenix-core/.../replication/ReplicationLogGroupIT.java. Both sides deduplicated the findLogFiles / assertTablesEqualAcrossClusters test helpers: upstream extracted them into a new ReplicationLogGroupBaseIT base class; this branch had extracted them into CrossClusterReplicationTestUtil. Took upstream's version wholesale -- its base-class extraction supersedes the local dedup, and CrossClusterReplicationTestUtil remains for StoreAndForwardFailoverIT (a separate HABaseIT hierarchy). No PHOENIX-7920 logic lived in this file. Production ReplicationLogDiscoveryReplay.java auto-merged cleanly: the C2 recoveryListener/triggerFailover CAS state-machine changes and PHOENIX-7938's getConsistencyPoint round-alignment are orthogonal and consistent (7938 relies on the same SYNC invariant C2 protects). Verified: test-compile BUILD SUCCESS across production + all test/IT sources.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Index tables no longer generate their own replication-log records. The active ships only the data-table record plus a per-
(row, ts)PRE_IMAGE (the prior row state observed at lock time), and the standby regenerates every index entry from that record on the replay path.This targets the
PHOENIX-7562-feature-newfeature branch (builds on the per-batch coalescing / cell-oriented log format, #2540).Why
(row, ts)group's next state from its own shipped pre-image instead of a region scan. Shard files can be replayed in any order or in parallel; each group is an independent reproduction of the activepreBatchMutate.PARTITION_ID), which differs between clusters. Replicating index cells verbatim would carry the active's region name; the standby instead rebuilds the rowkey with its own.Main changes
MutationCellGrouper) carrying data cells plus one METAFAMILY pre-image cell per row.IndexRegionObserver: active-side pre-image capture; a forked standby replay path (prepareReplicatedIndexMutations) that groups by(row, ts)and builds index updates from the shipped pre-image.PreImageLocalTable: aLocalHBaseStateserving the local-index builder's prior-row-state from the shipped pre-image instead of a region scan; newIndexBuildManager.getIndexUpdatesoverload accepts it.ReplicationLogProcessor: replay a record as one indivisible batch; do not retryDoNotRetryIOException.Test coverage
Global, local, uncovered, atomic, conditional-TTL, and CDC (plain and behind an EVENTUAL secondary index) index regeneration; concurrent/out-of-order replay for both global and local indexes; unit tests for
(row, ts)grouping isolation and Put/Delete decomposition.Notes