[client] Back off writes rejected by disk protection

[fhan688]

Purpose

Disk protection rejects writes with the DISK_WRITE_LOCKED error code. Today the Java client immediately retries and resends the full payload, generating excessive network traffic and consuming client CPU without making progress during sustained disk pressure.

This PR introduces bounded exponential backoff for the Java Log and KV writers. It targets the Java client only; Rust-client support is planned separately.

Linked issue: #4278

Brief change log

• Add two client config options:
  • client.writer.disk-write-locked.backoff — initial backoff, default 1s
  • client.writer.disk-write-locked.backoff-max — maximum backoff incl. jitter, default 10s
• Delay doubles with retry count using ±20% jitter, constrained by
  1ms <= initial <= maximum <= 2147483647ms.
• Handle DISK_WRITE_LOCKED in the shared Sender retry path; track backoff deadlines by destination TableBucket; enforce backoff in readiness checks and batch draining; preserve retry limits; wake the Sender after re-enqueueing.
• Extract an @Internal WriteThrottleController that unifies the two per-bucket write-throttling gates — KV backpressure (wall-clock millis) and disk-write backoff (monotonic nanos) — behind a single read side:
  remainingDelayMs(tb) = max(kvRemainingMs, diskRemainingMs) and
  isGated(tb). Both sendability paths, ready() and drain()'s
  shouldSkipBucket(), now consult the controller, so no single gate can be dropped. Within a bucket the gates fold via max(); across buckets ready() keeps the earliest wake-up via min(). This implements the refactor
  @swuferhong requested (agreed in-thread to land in this PR). The public surface and existing throttle/backoff behavior are unchanged.
• Ordering note (leader-first): in bucketReady(), a bucket whose leader is unknown is still reported to unknownLeaderTables even while a throttle gate is pending. A metadata refresh is orthogonal to the gate (it
  sends no data), and refreshing now means the leader is ready to receive the moment the gate clears.

Tests

Validated with JDK 11.0.27 and Maven 3.8.6.

• RecordAccumulatorTest: 27 tests pass (was 25; added coverage for the unified gate and disk/KV independence, and removed one interim test that assumed gate-first ordering).
• SenderTest: 51 tests pass.
• New DiskWriteBackoffITCase: 3 tests pass.
• Full 11-module reactor build is BUILD SUCCESS (checkstyle / spotless / apache-rat / enforcer all green).

Local traffic comparison (single writer / bucket / TabletServer, ~5s simulated protection):

• Log writes: RPCs 73,794 → 3; bytes 306,835,452 → 12,474
• KV writes: RPCs 75,404 → 3; bytes 311,870,944 → 12,408

These are single local samples and do not establish production capacity.

API and format changes

Adds two Java-client config options only. Reuses the existing DISK_WRITE_LOCKED error code; no server protocol upgrade required, but the Java client must be upgraded. Backoff applies per physical table bucket.

Documentation

Updates website/docs/maintenance/configuration.md with the two new options.

[swuferhong]

Hi, @fhan688 I suggest extracting an internal WriteThrottleController to unify the existing disk-write backoff and KV backpressure handling:

• Policies translate errors or pressure signals into a throttling reason, scope, and deadline.
• The controller maintains state separately for each reason, computes the effective deadline, handles cleanup, and wakes the Sender when the next eligible send time moves earlier.
• ready() queries the remaining delay, and drain() checks eligibility again before draining batches. Adding a new policy should not require changes to either path.
• If we later need QPS or bandwidth limits, we can introduce non-blocking permit acquisition before sending requests. If the goal is simply to suppress aggressive retries during write rejection, we could allow only a bounded number of probe requests after the backoff expires.

This would make disk-write backoff one policy within a reusable write-throttling mechanism and provide a natural extension point for future throttling scenarios.

[fhan688]

Hi, @fhan688 I suggest extracting an internal WriteThrottleController to unify the existing disk-write backoff and KV backpressure handling:

• Policies translate errors or pressure signals into a throttling reason, scope, and deadline.
• The controller maintains state separately for each reason, computes the effective deadline, handles cleanup, and wakes the Sender when the next eligible send time moves earlier.
• ready() queries the remaining delay, and drain() checks eligibility again before draining batches. Adding a new policy should not require changes to either path.
• If we later need QPS or bandwidth limits, we can introduce non-blocking permit acquisition before sending requests. If the goal is simply to suppress aggressive retries during write rejection, we could allow only a bounded number of probe requests after the backoff expires.

This would make disk-write backoff one policy within a reusable write-throttling mechanism and provide a natural extension point for future throttling scenarios.

I'd like to confirm that extracting an internal WriteThrottleController in this PR or as a follow-up PR?

[swuferhong]

Hi, @fhan688 I suggest extracting an internal WriteThrottleController to unify the existing disk-write backoff and KV backpressure handling:

• Policies translate errors or pressure signals into a throttling reason, scope, and deadline.
• The controller maintains state separately for each reason, computes the effective deadline, handles cleanup, and wakes the Sender when the next eligible send time moves earlier.
• ready() queries the remaining delay, and drain() checks eligibility again before draining batches. Adding a new policy should not require changes to either path.
• If we later need QPS or bandwidth limits, we can introduce non-blocking permit acquisition before sending requests. If the goal is simply to suppress aggressive retries during write rejection, we could allow only a bounded number of probe requests after the backoff expires.

This would make disk-write backoff one policy within a reusable write-throttling mechanism and provide a natural extension point for future throttling scenarios.

I'd like to confirm that extracting an internal WriteThrottleController in this PR or as a follow-up PR?

I'd prefer to do it in this PR: set up the architecture properly upfront and reduce the amount of refactoring needed later. WDYT @platinumhamburg

[platinumhamburg]

Hi, @fhan688 I suggest extracting an internal WriteThrottleController to unify the existing disk-write backoff and KV backpressure handling:

• Policies translate errors or pressure signals into a throttling reason, scope, and deadline.
• The controller maintains state separately for each reason, computes the effective deadline, handles cleanup, and wakes the Sender when the next eligible send time moves earlier.
• ready() queries the remaining delay, and drain() checks eligibility again before draining batches. Adding a new policy should not require changes to either path.
• If we later need QPS or bandwidth limits, we can introduce non-blocking permit acquisition before sending requests. If the goal is simply to suppress aggressive retries during write rejection, we could allow only a bounded number of probe requests after the backoff expires.

This would make disk-write backoff one policy within a reusable write-throttling mechanism and provide a natural extension point for future throttling scenarios.

I'd like to confirm that extracting an internal WriteThrottleController in this PR or as a follow-up PR?

I'd prefer to do it in this PR: set up the architecture properly upfront and reduce the amount of refactoring needed later. WDYT @platinumhamburg

Agree +1.