Skip to content

util: Fix AdvancedTlsX509KeyManager reloading credentials on every refresh - #13033

Open
TimurRakhmatullin86 wants to merge 1 commit into
grpc:masterfrom
TimurRakhmatullin86:fix/advanced-tls-keymanager-mtime-swap
Open

util: Fix AdvancedTlsX509KeyManager reloading credentials on every refresh#13033
TimurRakhmatullin86 wants to merge 1 commit into
grpc:masterfrom
TimurRakhmatullin86:fix/advanced-tls-keymanager-mtime-swap

Conversation

@TimurRakhmatullin86

Copy link
Copy Markdown
Contributor

Problem

AdvancedTlsX509KeyManager's scheduled credential reloader reloads the key and certificates and rotates the TLS alias on every refresh interval, even when neither file has changed.

UpdateResult's constructor takes (boolean success, long certTime, long keyTime), but readAndUpdate builds it as:

return new UpdateResult(true, newKeyTime, newCertTime);   // args swapped vs the constructor
...
return new UpdateResult(false, oldKeyTime, oldCertTime);  // same swap

So result.certTime actually holds the key file's mtime and result.keyTime the cert file's mtime. LoadFilePathExecution.run() then stores each into the oppositely named field (currentCertTime = result.certTime, currentKeyTime = result.keyTime), so the persisted currentKeyTime/currentCertTime are crossed. On the next tick, readAndUpdate compares the key file's current mtime against the cert file's last-seen mtime and vice versa:

if (newKeyTime != oldKeyTime && newCertTime != oldCertTime) { ... reload ... }
// after the swap this is effectively (K != C && C != K), i.e. (K != C)

With two files whose mtimes differ — the normal case for a cert and a key written at different instants — K != C is always true, so it reloads every interval.

Impact

  • Breaks the method's own documented invariant (comment: "We only update when both the key and the certs are updated.").
  • Re-parses the private key and certificate chain from disk every refresh period for no reason.
  • Rotates the key-<N> alias on every interval, continuously invalidating the per-alias encoded-key-material cache that the alias rotation (util: update AdvancedTlsX509KeyManager to support key alias for reloaded cert #12686) was introduced to enable — the opposite of the intended optimization.

Introduced by #11385, which reordered the UpdateResult constructor parameters but left these two call sites unchanged.

Fix

Pass the times in the order the constructor expects (certTime, keyTime).

Test

scheduledReload_doesNotReloadWhenFilesAreUnchanged gives the cert and key files distinct mtimes, schedules the reloader on a FakeClock, lets one refresh cycle run, then advances several more intervals with the files untouched and asserts the alias is unchanged. It fails before this change (expected:<key-2> but was:<key-3> — the alias keeps rotating) and passes after.

…fresh

The UpdateResult constructor takes (success, certTime, keyTime), but
readAndUpdate constructed it as (success, newKeyTime, newCertTime), so the
returned certTime/keyTime were swapped. LoadFilePathExecution then stored each
into the oppositely named field, so on the next refresh readAndUpdate compared
the key file's mtime against the cert file's last-seen mtime and vice versa.

With two files of different mtimes (the normal case) that cross-comparison is
always true, so the scheduled reloader re-read the key and certificates and
rotated the alias on every refresh interval even when nothing changed. That
breaks the method's documented 'only update when both files changed' invariant
and continuously invalidates the per-alias key-material cache that the alias
rotation was introduced to enable.

Pass the times in the order the constructor expects. Add a test that advances a
FakeClock over several refresh intervals with the files untouched and asserts
the alias is stable; it fails before this change (the alias keeps rotating).

Signed-off-by: Timur Rakhmatullin <174210871+TimurRakhmatullin86@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant