Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Preserve source, tests, templates, .github and docs; exclude host state.
.git
**/.git
.venv
**/.venv
venv
env
.env
.env.*
**/.env
**/.env.*
**/*.local.toml
**/*.local.yaml
**/__pycache__
**/*.py[cod]
**/*.egg-info
**/.pytest_cache
**/.ruff_cache
**/.cache
**/.coverage*
**/coverage.xml
**/htmlcov
**/.ipynb_checkpoints
dist
build
artifacts
test-results
output
data
.dd-branch
.cursor
.claude
.codex
.vscode
.idea
working-memory
**/debug_llm_responses.jsonl
**/debug_litellm_requests.jsonl
**/*.pem
**/*.key
.DS_Store

# Private top-level work areas and agent overrides.
_*/
.worktrees/
**/AGENTS.local.md
**/CLAUDE.local.md

# Generated model-specific vectors; retain the source asset banks.
src/usersim/engine/assets/financial_services/**/embeddings.parquet
**/junit.xml
4 changes: 4 additions & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,7 @@
/tests/test_update_license_headers.py @NVIDIA-NeMo/user_sim_maintainers
/docs/engineering/ @NVIDIA-NeMo/user_sim_maintainers
/tests/fixtures/exclusion-policy.json @NVIDIA-NeMo/user_sim_maintainers
/containers/ @NVIDIA-NeMo/user_sim_maintainers
/.dockerignore @NVIDIA-NeMo/user_sim_maintainers
/tests/test_container_runner.py @NVIDIA-NeMo/user_sim_maintainers
/tests/test_coverage_gate.py @NVIDIA-NeMo/user_sim_maintainers
19 changes: 19 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -133,3 +133,22 @@ jobs:
path: test-results/e2e/
if-no-files-found: warn
retention-days: 14

container:
name: CPU container tests
runs-on: ubuntu-latest
timeout-minutes: 65
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Build and run isolated CPU validation
run: make test-container

- name: Container test diagnostics
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: container-test-results
path: test-results/container/
include-hidden-files: true
if-no-files-found: warn
7 changes: 7 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -389,3 +389,10 @@ Participation is governed by the [Contributor Covenant](CODE_OF_CONDUCT.md).
See [Live provider health checks](docs/engineering/health-checks.md) for credential
precedence, local commands, and the pass, skip, and failure outcomes. These checks
make real model requests and are separate from offline contributor validation.

## Container validation

Run `make test-container` to build a fresh CPU test image and run coverage plus
installed-wheel e2e without external networking. See
[CPU container tests](docs/engineering/container-tests.md) for Docker prerequisites,
resource limits, context policy, and exported diagnostics.
12 changes: 12 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -296,3 +296,15 @@ clean: ## Remove caches, coverage output, compiled artifacts, build output and
find . -type f -name '*.py[co]' -not -path './.venv*' -delete 2>/dev/null || true
rm -rf .pytest_cache .ruff_cache .coverage coverage.xml htmlcov build dist src/*.egg-info
@echo "Cleaned."

# Container validation uses only host Python and Docker; dependencies live in the image.
CONTAINER_ARGS ?=
.PHONY: container-audit container-build-test test-container
container-build-test: ## Build the CPU test image with locked dependencies
python3 scripts/run_container_tests.py build $(CONTAINER_ARGS)

test-container: container-build-test ## Build fresh and run offline CPU coverage and e2e
python3 scripts/run_container_tests.py run $(CONTAINER_ARGS)

container-audit: ## Audit the local Docker context before image build
python3 scripts/run_container_tests.py audit $(CONTAINER_ARGS)
22 changes: 22 additions & 0 deletions containers/Dockerfile.test
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Multi-platform manifest verified with docker buildx imagetools inspect.
FROM python:3.12-slim-bookworm@sha256:34386ef0cb081344d7ec1c103ba398e6e9f64e9ab3a1509accc92a4e24a07258 AS tests
RUN apt-get update && apt-get install -y --no-install-recommends git make \
&& python -c 'import shutil; shutil.rmtree("/var/lib/apt/lists")'
WORKDIR /workspace
COPY . .
# pyproject.toml owns the uv version; no second pin belongs here.
RUN python -c 'import tomllib, subprocess; p=tomllib.load(open("pyproject.toml", "rb")); v=p["tool"]["uv"]["required-version"]; assert v.startswith("=="), "container requires an exact uv declaration"; subprocess.run(["python", "-m", "pip", "install", "--no-cache-dir", "uv"+v], check=True)'
ENV UV_PROJECT_ENVIRONMENT=/opt/unit UV_PYTHON_DOWNLOADS=never \
DO_NOT_TRACK=1 HF_HUB_DISABLE_TELEMETRY=1 LITELLM_TELEMETRY=False
RUN uv sync --locked && uv build --wheel --out-dir /opt/wheels \
&& python scripts/run_e2e.py --work-dir /opt/e2e --wheel /opt/wheels/*.whl --prepare-only \
&& chmod -R a+rwX /workspace /opt/e2e
ENV UV_OFFLINE=1 UV_NO_SYNC=1 UV_CACHE_DIR=/tmp/uv-cache HOME=/tmp/home
CMD ["python", "scripts/run_container_tests.py", "inside"]

# Export this stage to inspect exactly the context used by COPY above.
FROM scratch AS context-audit
COPY . /
98 changes: 98 additions & 0 deletions docs/engineering/container-tests.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# CPU container tests

The test image prepares a source unit-test environment and a separate installed-wheel
e2e environment during build. Execution uses the same `make coverage` and
`make test-e2e` commands as native validation, with external networking disabled.

Install Docker with a running daemon. Build dependencies require internet access;
test execution needs no provider credentials. The image uses Python 3.12 pinned by
manifest digest and reads the exact uv version from `pyproject.toml`.

```bash
make container-build-test
make test-container
```

Artifacts appear in fresh `test-results/container/run-*/` directories: build and container logs,
`unit.log`, `unit.xml`, `coverage.xml`, `.coverage`, `cleanup.log`, and the e2e
artifact directory.
A failing suite or timeout returns nonzero. Both suites run so one failure does
not hide the other suite's diagnostics. The host runner allows one hour for image build and another hour for execution;
each suite allows 25 minutes. Runtime limits are two CPUs, two pytest workers,
8 GiB memory with no additional swap, and 512 processes. Temporary storage is
limited to 1 GiB and permits executable test fixtures. Tests run with the caller's UID/GID so exported files remain
owned by the caller.

Only the artifact directory is mounted from the host. The image includes source,
test fixtures, documentation, templates, and workflow files that unit tests read.
`.dockerignore` excludes Git metadata, host environments, private endpoint
configurations, credentials, editor/agent settings, caches and generated output.
Git is installed for tests that create their own temporary repositories.
The e2e runner operates outside `/workspace` against its installed wheel and
prepares its tokenizer cache during build. Docker's disabled external network
still permits the fake provider to use container loopback.

Every image build first runs a required context audit. It exports a synthetic
scratch context with Docker's native ignore semantics and checks every shared
example in `tests/fixtures/exclusion-policy.json`. Generated financial-services
`embeddings.parquet` sidecars stay outside the context; source asset banks remain.
Run this bounded check separately with:

```bash
make container-audit
```

The audit, image build, execution, and cleanup require `/var/run/docker.sock`
to be a Unix socket.
They reject nonempty inherited `DOCKER_*`, `BUILDX_*`, and `BUILDKIT_*` selectors
before invoking Docker, including host, context, client config, TLS, and builder
settings. Unset these variables explicitly when using this local validation
workflow. Each operation uses a temporary empty client config, explicit
`--host unix:///var/run/docker.sock`; image builds use the `default` builder
with the `docker` driver. Execution and cleanup share the same admitted client
configuration and environment, keeping the temporary config alive until cleanup
finishes. Stored default or remote contexts cannot select a different daemon
for running the image. A failed builder check, missing daemon, failed export, or
incorrect canary result returns nonzero. No missing prerequisite becomes a successful skip.
The scratch audit pulls no image and supplies `--network=none`.

Artifacts include `audit-settings.json`, `audit-builder.log`, `audit-export.log`,
and `audit-result.json`; the receipt records every example and its result.
Temporary client config and synthetic context files are removed afterward.
Ordinary pytest mocks Docker orchestration and never invokes a Docker executable.
Actual daemon access belongs to this explicit audit and container validation.

Each `audit`, `build`, or `run` invocation exclusively creates a `run-*` directory
beneath `--artifacts ROOT` and prints its exact path before starting. A `build`
attempt contains its audit and build output; an execution attempt contains the
container, cleanup, unit, and e2e logs. Earlier attempt directories and unrelated
root files are untouched. There are no ownership markers or history migrations;
remove result directories when you no longer need them.

The host mounts only the new execution directory at `/artifacts`. The `inside`
action writes directly into that already-owned directory, without another
container-level directory or cleanup step. Installed e2e creates its own fresh
attempt below `/artifacts/e2e`. Preparation remains in `/opt/e2e`, separate from
results. Failed attempts retain their partial diagnostics without exposing a
previous attempt's passing audit, coverage, JUnit, or scores as current. CI can
continue uploading the selected output root recursively.

`cleanup.log` records the container name, cleanup status, exit code, and captured
output, including partial timeout output. Successful removal and the exact
already-removed response for that container preserve the primary result.
Other removal errors require recovery and return nonzero if the primary run
succeeded. A primary test failure or timeout remains the returned status even
when cleanup fails. The diagnostic identifies the container and local endpoint
for inspection and manual recovery; cleanup never falls back to another daemon.

For direct runner overrides:

```bash
python scripts/run_container_tests.py build --image usersim-tests:local
python scripts/run_container_tests.py run --image usersim-tests:local --artifacts /tmp/usersim-test-artifacts --timeout 3600
```

After changing the base image, resolve and review its public manifest digest,
update the Dockerfile, and rebuild the image before running validation. Changes
to source or fixtures also require rebuilding; execution never mounts the host
checkout over the image.
2 changes: 1 addition & 1 deletion docs/engineering/e2e.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ uv run --frozen python scripts/run_e2e.py --work-dir /tmp/usersim-e2e --prepare-
uv run --frozen python scripts/run_e2e.py --work-dir /tmp/usersim-e2e --run-only
```

`--work-dir` must resolve outside the checkout and cannot be an ancestor of it. Paths containing `..` use their canonical locations for this check; symlinked work and artifact paths are rejected. Reuse the same path for both phases; the prepared environment and copied suite are a snapshot of setup. Changes to tests or wheels require another preparation. A new preparation removes the previous `installed.json` receipt before changing the environment, so an interrupted setup cannot advertise the old wheel as a valid prepared runtime.
`--work-dir` must resolve outside the checkout and cannot be an ancestor of it. Paths containing `..` use their canonical locations for this check; symlinked work and artifact paths are rejected. Reuse the same path for both phases; the prepared environment and copied suite are a snapshot of setup. Changes to tests or wheels require another preparation. A new preparation removes the previous `installed.json` receipt before changing the environment, so an interrupted setup cannot advertise the old wheel as a valid prepared runtime. The container prepares these environments at image build time and runs the second command with external networking disabled.

## Scenarios and assertions

Expand Down
14 changes: 9 additions & 5 deletions docs/engineering/ignore-policy.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ tracked file or prevent setuptools from packaging it.
| Documentation | `scripts/check_doc_links.py` | Tracked Markdown is checked, except private top-level underscore paths. Local links and repository main-branch links resolve against the checkout; external URLs and anchors are not fetched or validated. |
| Coverage | `pyproject.toml`, `Makefile` | Runtime package coverage excludes explicit `pragma: no cover` lines, type-only branches, and abstract `NotImplementedError` branches. These are reporting exclusions, not Git rules. No type-check exclusions are configured yet. |
| Packaging | `MANIFEST.in`, setuptools configuration in `pyproject.toml` | Both `tests/` and `tests_e2e/` are omitted from the sdist. Only `usersim*` packages are discovered; typed markers, shipped model TOML, and required asset banks are included. Generated embedding vectors are explicitly excluded. `make check-dist` inspects both archives and runs an installed wheel. |
| Container context | Not yet configured | No test container or `.dockerignore` exists at this stage. Container work must explicitly exclude credentials, host environments, caches, private files, and Git metadata while including required fixtures and assets, then verify the actual build context. |
| Container context | `.dockerignore`, `containers/Dockerfile.test` | Credentials, host environments, caches, private files, generated artifacts, and Git metadata are excluded. Required source, fixtures, and assets remain included. The required pre-build audit checks shared exclusion examples with Docker itself; see [CPU container tests](container-tests.md). |

`.gitattributes` is not an ignore file. Its uv.lock attributes collapse generated
changes in GitHub and prevent unsafe line-by-line merges; regenerate conflicting
Expand All @@ -32,7 +32,11 @@ Before changing an exclusion, test representative private and required paths wit
and `make check-dist` after changing packaging boundaries. Enroll new test or task
directories in applicable lint and SPDX discovery when they are introduced.

`tests/fixtures/exclusion-policy.json` contains representative private and required
paths. It is test data, not a rule generator or a glob interpreter. Git tests use
`git check-ignore` to check the committed native Git policy. Later boundaries
can add their own independent expectations to the same examples.
`tests/fixtures/exclusion-policy.json` contains representative paths with separate
Git and Docker expectations. The policies intentionally differ: working-memory
plans remain trackable but are excluded from Docker, while test fixtures remain
in Docker but are excluded from published archives. This table is test data,
not a generator or a shared glob interpreter. Git tests use `git check-ignore`;
`make container-audit` checks Docker semantics before image builds. Generated
financial-services embedding sidecars are excluded at both boundaries while
required source asset banks remain included.
Loading
Loading