Skip to content

fix(helm): longer embedding liveness window, PDB for replicated embedding; chore(tei): bump to cpu-1.9.4 - #8

Merged
eksrha merged 1 commit into
mainfrom
fix/embedding-liveness-tolerance
Oct 2, 2026
Merged

eksrha merged 1 commit into
mainfrom
fix/embedding-liveness-tolerance

Conversation

@eksrha

@eksrha eksrha commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Under bursts of large /v1/embeddings requests, the embedding pod is restarted by the kubelet although the process is alive and not OOM. The liveness probe queries TEI's /health, which waits in the same queue as the inference requests. While a large batch is being processed (tens of seconds), /health times out, three failures (~90 s) trigger a SIGTERM, and the restart destroys in-flight work and makes the backlog worse.

Changes

  • Embedding liveness failureThreshold 3 -> 10 (~5 min window), so only a genuinely wedged process is restarted. Values comment updated.
  • New PodDisruptionBudget (maxUnavailable: 1) for the embedding deployment, rendered only when embedding.enabled and embedding.replicaCount > 1.
  • TEI base image bump cpu-1.9.3 -> cpu-1.9.4.

Testing

  • helm lint deploy/helm passes
  • helm template with embedding.enabled=true: no PDB at replicaCount=1, one PDB at replicaCount=2; liveness failureThreshold: 10 rendered
  • Tests pass locally: no Go code changed, not run
  • New tests added where applicable: n/a (chart/image only)

Not part of this PR

  • Running 2+ embedding replicas for sustained load (TEI is CPU-bound with one batch queue per replica) and raising the CPU limit are deployment recommendations, not changed defaults here.
  • The new image only takes effect with the next release.

Related Issues

Closes #7

@eksrha
eksrha merged commit 4e261fe into main Oct 2, 2026
2 checks passed
@eksrha
eksrha deleted the fix/embedding-liveness-tolerance branch October 2, 2026 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Embedding pod restarts under load: liveness probe shares the busy /health queue

1 participant