Skip to content

Mount credentials safely, pass provider keys through, and bound stage idle time - #64

Closed
KarthikAvinashFI wants to merge 2 commits into
feat/hosted-harness-e2e-runtimefrom
feat/harness-infra-credentials-and-limits
Closed

Mount credentials safely, pass provider keys through, and bound stage idle time#64
KarthikAvinashFI wants to merge 2 commits into
feat/hosted-harness-e2e-runtimefrom
feat/harness-infra-credentials-and-limits

Conversation

@KarthikAvinashFI

@KarthikAvinashFI KarthikAvinashFI commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

What this changes

Infrastructure fixes found while running the hosted harness end to end against a LiveKit voice
agent. Each one is here because a run failed on it, so the reason is recorded alongside the change.
Separated from the scenario and persona work in #63 so the two can be taken independently. There
is no file overlap between the two branches.

Why each change is needed

A run could not start, because the generated runtime had no Google credential.
The generated runtime needs GOOGLE_APPLICATION_CREDENTIALS. When neither the submitted Compose
mount nor the uploaded runtime configuration pointed at a valid credential file, provisioning
raised and the job ended before the environment existed. A local sandbox does hold that credential
in its own environment, so the fallback reads it from there.

That fallback is a trust boundary, which is why it is gated. The credential belongs to whoever is
running the harness, not to the submitted agent, and mounting it into a container built from
customer source would hand the runner's own key to code it does not control. It is therefore taken
only when ALK_ALLOW_HOST_GOOGLE_CREDENTIALS=1 is set. A hosted runner leaves it unset and keeps
the old refusal. The flag is also reserved, so a job cannot switch it on for itself through
secret_refs.

Calls could not connect, because the worker had no provider credentials.
The generated runtime does not carry the agent's own .env.local, and the runtime configuration
covers datastore settings only. Without LiveKit the worker cannot register or place the call,
without Deepgram there is no speech, and without Vertex there is no model. These are passed
through as bare --env NAME, so values are never written into argv.

An agent's own model could not authenticate.
An agent's Compose commonly mounts its Vertex credential from a variable such as
${VERTEX_CREDENTIALS:-/dev/null}. With the variable unset the placeholder is mounted and the
agent's LLM fails, so it is pointed at the same credential the harness already resolved.

Calls dropped immediately after the greeting.
Voice agents often enable an audio enhancement plugin that is licensed separately. Unauthorized,
it raises on the first inbound audio frame and the worker tears the call down, which showed up as
a two turn conversation with no error attributable to the agent. DISABLE_AI_COUSTICS defaults on
so the plugin is skipped; agents that do not read it are unaffected, and a run can override it
where a license exists.

Jobs were killed in the middle of a healthy suite.
ALK_STAGE_IDLE_TIMEOUT_SECONDS now defaults to 600 rather than 180. Parallel scenario generation
leaves the parent session without an event for longer than three minutes while its writers work,
and the old bound ended the job with write-scenarios produced no event for 180s while the suite
was progressing normally. The bound still measures inactivity rather than total stage duration, so
a genuinely wedged stream still fails rather than hanging.

A world could come up empty and look like an agent failure.
Restoring rows into a Postgres schema that is not there quietly restored nothing, and the scenario
then ran against a blank database. Every check failed, and the failure read as the agent's fault.
It now fails where the problem actually is.

Tool calls failed for agents that ship more than one service.
The generated adapter models only an agent and a datastore, so any other service the agent's tools
are served by, such as an HTTP tools API, was dropped and the world had nothing to forward to.
The agent's own Compose is preferred when it ships one.

Testing

  • tests/test_harness_secrets.py, tests/test_harness_surface_conformance.py and
    tests/test_harness.py pass, 270 tests.
  • Verified end to end locally against a LiveKit voice agent: provisioning succeeds, the runtime
    starts, calls connect and run to a terminal result rather than dropping after the greeting.

Note for review

Based on feat/hosted-harness-e2e-runtime rather than dev, since it builds on the harness
layout introduced there.

@KarthikAvinashFI
KarthikAvinashFI marked this pull request as ready for review August 25, 2026 07:42
@hadarishav

Copy link
Copy Markdown
Contributor

Closing because the safe subset has been selectively integrated into feat/hosted-harness-e2e-runtime as commit 89c6fec (Stabilize hosted runner topology and stage idle bounds).

Integrated:

  • runner-owned Docker network/webhook topology propagation for containerized/Daytona execution;
  • runner-owned agent/user/judge model configuration propagation;
  • stage inactivity default increased from 180s to 600s while retaining the bounded-idle failure;
  • regression tests ensuring jobs cannot override runner topology/model settings;
  • regression tests ensuring host LiveKit/Deepgram/Cartesia credentials are not inherited by submitted jobs.

Intentionally excluded:

  • the Compose fallback, because it bypasses packaging selection and can override an explicit Dockerfile or select development Compose;
  • host Google credential fallback, because the platform now uses explicit encrypted credential-file upload and attempt-scoped secret references;
  • broad provider credential forwarding from runner environment, because it can expose runner-owned credentials to submitted customer code;
  • unconditional DISABLE_AI_COUSTICS=1, because that changes submitted-agent behavior and must remain explicit configuration;
  • VERTEX_CREDENTIALS runtime override, because it does not safely solve Compose-time interpolation and the credential-file mount handles this case;
  • the base Postgres restore change, because AttachedPostgresStore already owns row-only restore and the proposed change can silently accept a missing schema.

Validation on the integrated commit: Ruff passed and the broader harness gate completed with 291 passed. The mixed PR commits were not cherry-picked wholesale because the branch had conflicts with the newer secure credential mounting implementation.

@hadarishav hadarishav closed this Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants