[Bug]: kube_redis_sentinel creds-sync.sh deletes creds Secrets on a pipefail/SIGPIPE race

[wesbragagt]

Prior Search

• I have already searched this project's issues to determine if a bug report has already been made.

What happened?

kube_redis_sentinel deploys a <id>-creds-syncer pod running creds-sync.sh. Every 60s it checks that the VSO-managed Redis users exist and, if one is missing, deletes the matching <id>-{superuser,admin,reader}-creds Secret so VSO recreates it.

The check is:

set -eo pipefail
...
if ! echo "$USERS_SRC" | grep -q "V_KUBERNETES-..._ADMIN"; then
  kubectl -n "$REDIS_NAMESPACE" delete secret "$REDIS_STS_NAME-admin-creds"

grep -q exits on the first match and closes the pipe. If echo is still writing, it dies with SIGPIPE (status 141). With pipefail the pipeline fails, ! inverts it, and the script deletes a Secret whose user does exist.

Effect in our clusters (dev and prod, every redis instance):

• Each of the three creds Secrets is deleted and recreated every 10 to 20 minutes. VSO logs sync_reason="InexistentDestination" and mints a new lease each time.
• Every Deployment that references the Secret carries reloader.stakater.com/auto: "true", so it rolls on every recreation. Our API Deployment went through 82 ReplicaSets in one day.
• Old leases are never revoked, so Vault leases and Redis ACL users pile up for 16h. The longer ACL list makes echo slower and raises the SIGPIPE odds, so the churn grows over time (48/day two weeks ago, 82/day now).

Expected: the Secret is only deleted when the Redis user is actually missing.

Fix: read the variable without a pipe, for example grep -q "$PATTERN" <<<"$USERS_SRC", for the three checks. Measured 0 failures in 3000 runs with the here-string versus 93 with the pipe.

Postgres creds Secrets from kube_pg_cluster are not affected because that module has no syncer.

Steps to Reproduce

1. Deploy kube_redis_sentinel (tested with the module at 40b40dbe; creds-sync.sh is identical on main).
2. Watch kubectl -n <ns> get secret <id>-admin-creds -o jsonpath='{.metadata.creationTimestamp}'. It moves forward every few minutes with no Redis restart, failover, or Vault revocation.
3. Exec into the <id>-creds-syncer pod and run the check in a loop:

set -o pipefail
ACL_RULES_SRC=$(redis-cli -e -h "$SRC_REDIS_HOST" ACL LIST | grep -oP '^user \K.+')
USERS_SRC=$(echo "$ACL_RULES_SRC" | grep -oP '^\S+')
P="V_KUBERNETES-${REDIS_NAMESPACE^^}-${REDIS_STS_NAME^^}_ADMIN"
for i in $(seq 1 3000); do
  echo "$USERS_SRC" | grep -q "$P"; s=$?
  [[ $s -ne 0 ]] && echo "status=$s"
done | sort | uniq -c

4. Repeat with grep -q "$P" <<<"$USERS_SRC"; no failures.

Relevant log output

# in-pod loop, 3000 iterations, 33 ACL users
pipefail + echo | grep -q :  93 failures, all status=141
grep -q <<<"$USERS_SRC"   :   0 failures
no pipefail               :   0 failures

# VSO events on the VaultDynamicSecret
SecretRotated  Secret synced, lease_id="db/creds/admin-<ns>-<id>/...", horizon=8h..., sync_reason="InexistentDestination"
(repeats every 10 to 20 minutes with a new lease_id)