Mount credentials safely, pass provider keys through, and bound stage idle time - #64
Closed
KarthikAvinashFI wants to merge 2 commits into
Closed
Conversation
…nd bound stage idle time
KarthikAvinashFI
marked this pull request as ready for review
August 25, 2026 07:42
Contributor
|
Closing because the safe subset has been selectively integrated into Integrated:
Intentionally excluded:
Validation on the integrated commit: Ruff passed and the broader harness gate completed with |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Infrastructure fixes found while running the hosted harness end to end against a LiveKit voice
agent. Each one is here because a run failed on it, so the reason is recorded alongside the change.
Separated from the scenario and persona work in #63 so the two can be taken independently. There
is no file overlap between the two branches.
Why each change is needed
A run could not start, because the generated runtime had no Google credential.
The generated runtime needs
GOOGLE_APPLICATION_CREDENTIALS. When neither the submitted Composemount nor the uploaded runtime configuration pointed at a valid credential file, provisioning
raised and the job ended before the environment existed. A local sandbox does hold that credential
in its own environment, so the fallback reads it from there.
That fallback is a trust boundary, which is why it is gated. The credential belongs to whoever is
running the harness, not to the submitted agent, and mounting it into a container built from
customer source would hand the runner's own key to code it does not control. It is therefore taken
only when
ALK_ALLOW_HOST_GOOGLE_CREDENTIALS=1is set. A hosted runner leaves it unset and keepsthe old refusal. The flag is also reserved, so a job cannot switch it on for itself through
secret_refs.Calls could not connect, because the worker had no provider credentials.
The generated runtime does not carry the agent's own
.env.local, and the runtime configurationcovers datastore settings only. Without LiveKit the worker cannot register or place the call,
without Deepgram there is no speech, and without Vertex there is no model. These are passed
through as bare
--env NAME, so values are never written into argv.An agent's own model could not authenticate.
An agent's Compose commonly mounts its Vertex credential from a variable such as
${VERTEX_CREDENTIALS:-/dev/null}. With the variable unset the placeholder is mounted and theagent's LLM fails, so it is pointed at the same credential the harness already resolved.
Calls dropped immediately after the greeting.
Voice agents often enable an audio enhancement plugin that is licensed separately. Unauthorized,
it raises on the first inbound audio frame and the worker tears the call down, which showed up as
a two turn conversation with no error attributable to the agent.
DISABLE_AI_COUSTICSdefaults onso the plugin is skipped; agents that do not read it are unaffected, and a run can override it
where a license exists.
Jobs were killed in the middle of a healthy suite.
ALK_STAGE_IDLE_TIMEOUT_SECONDSnow defaults to 600 rather than 180. Parallel scenario generationleaves the parent session without an event for longer than three minutes while its writers work,
and the old bound ended the job with
write-scenarios produced no event for 180swhile the suitewas progressing normally. The bound still measures inactivity rather than total stage duration, so
a genuinely wedged stream still fails rather than hanging.
A world could come up empty and look like an agent failure.
Restoring rows into a Postgres schema that is not there quietly restored nothing, and the scenario
then ran against a blank database. Every check failed, and the failure read as the agent's fault.
It now fails where the problem actually is.
Tool calls failed for agents that ship more than one service.
The generated adapter models only an agent and a datastore, so any other service the agent's tools
are served by, such as an HTTP tools API, was dropped and the world had nothing to forward to.
The agent's own Compose is preferred when it ships one.
Testing
tests/test_harness_secrets.py,tests/test_harness_surface_conformance.pyandtests/test_harness.pypass, 270 tests.starts, calls connect and run to a terminal result rather than dropping after the greeting.
Note for review
Based on
feat/hosted-harness-e2e-runtimerather thandev, since it builds on the harnesslayout introduced there.