Skip to content

operations.md

Latest commit

 

History

History
221 lines (152 loc) · 18 KB

operations.md

File metadata and controls

221 lines (152 loc) · 18 KB
Raw
title Operations

Day-to-day tasks for the operator of a Core installation: the oac command, service health, the Core key, backups and uninstalling. Nodes have their own guide, and Troubleshooting lists installer and access messages. Settings are in the Configuration reference.

The oac command

Each installation has its own oac command in its directory (oac.exe on Windows). Run it as the account that installed Core; it needs Docker access, not root. The examples use the default installation directory ~/.oac/core; on Windows, invoke the command with & "$HOME/.oac/core/oac.exe" followed by the same arguments. For a custom directory, replace the path.

Command What it does
oac install Installs an installation or resumes one that stopped before it started; the installer scripts run it
oac apply Checks the data volume and .env, then starts the services and recreates those whose configuration changed; see how oac apply works. A failed check changes no service. On success it prints Services are running with the current settings.
oac core-key [--show] Identifies the key location in the data volume, or prints the key with --show
oac rotate-core-key Replaces the Core key and restarts Core and Web, then says how to read the new key
oac uninstall Deletes the installation and its data after confirmation; see Uninstall

Docker Compose commands manage the services; run them against the installation's Compose file. Stop and restart covers stop and start.

Command What it does
docker compose -f ~/.oac/core/compose.yaml ps Shows the services
docker compose -f ~/.oac/core/compose.yaml logs --tail 200 core Shows a service's log
docker compose -f ~/.oac/core/compose.yaml down Removes the containers. Data is kept

Service health

Use these observations for different questions:

Observation What it establishes
docker compose ps The database accepts its readiness check, and Core and Web each answer their own /healthz
Core /healthz Core's process is alive
An authenticated API read The caller's key works for that resource
Environment connection The Runtime transport is connected
A finished Turn and its results The task's recorded outcome

Only Web is a listener; see Configure the public address. Service health does not show that a Harness or a model works. Use Session, Turn, Items and Usage reads for execution, and Web's Nodes page for node connection, readiness and placement. For local diagnosis, use the installation's own Compose file:

docker compose -f "$HOME/.oac/core/compose.yaml" ps --all
docker compose -f "$HOME/.oac/core/compose.yaml" logs --tail 200 core
docker compose -f "$HOME/.oac/core/compose.yaml" logs --timestamps init

The init service exits after initialization. Its step logs are described in Deployment.

Don't paste docker compose config, docker inspect or raw logs into public issue reports.

Request diagnosis

Open Monitor → Request log to inspect recent requests and copy their request IDs for correlation with Core's service log. Request log defines its scope, privacy and limited in-memory retention. For a request that accepted an asynchronous Turn, inspect Session log for the execution result; HTTP acceptance alone is not task success. Use your existing log collector for retention across restarts or Core instances.

Stop and restart

Let active work settle before a planned restart:

docker compose -f ~/.oac/core/compose.yaml stop
docker compose -f ~/.oac/core/compose.yaml start

Stopping Core stops no node and no sandbox. Node services, their microVMs and Docker containers keep running; stopping is not a way to reclaim compute. A Core restart does not continue an interrupted native tool call transparently. After reconnecting, query the same Session; don't create a new Session to replay uncertain work. Session event streams are live only; recover through Session, Turn and Items reads.

A Web restart, including one caused by oac apply, signs everyone out of the console; see sign-in.

Core key

Each installation has one administrator credential, the Core key. The installer generates a key with the oac_admin_ prefix followed by 64 random lowercase hexadecimal characters in secrets/web/core.key. Read it with oac core-key --show; the file is owned by the container user. The Core key:

  • signs in to Web. The browser gets an HttpOnly session cookie, never the key;
  • authorizes Core API (/core/v1) requests sent as Authorization: Bearer <Core key>;
  • never authorizes the Agents API (/v1). Applications use Project API keys, which in turn can't call /core/v1.

Keep it private. Web reads secrets/web/core.key. Core reads only its SHA-256 from secrets/core/core-key-digests.json. A Core key has at least 32 characters and no whitespace. Web limits failed sign-ins.

Script the Core API

On the Core host, this helper runs curl in Core's network namespace and passes the key on stdin, keeping it off the command line:

core() (  # core METHOD PATH [JSON body]
  cd ~/.oac/core
  ./oac core-key --show | sed 's/^/Authorization: Bearer /' |
    docker run -i --rm --network "container:$(docker compose ps -q core)" curlimages/curl \
      -fsS -X "$1" "http://127.0.0.1:8091/core/v1$2" -H @- -H 'Content-Type: application/json' ${3:+-d "$3"}
)
Task Command
List Projects core GET /projects
Create a Project core POST /projects '{"name": "billing-bot"}'
Issue an API key (shown once, as key) core POST /projects/$PROJECT_ID/keys '{"name": "prod"}'
Revoke a key core DELETE /projects/$PROJECT_ID/keys/$KEY_ID
Archive a Project (revokes all keys) core POST /projects/$PROJECT_ID/archive
See Harnesses and their default models core GET /harnesses
Set Codex's default model core PUT /harnesses/codex/model-configuration '{"model": "your-model-id", "model_provider": {"protocol": "responses", "base_url": "https://provider.example/v1", "api_key": "sk-..."}}'
Installation facts, including the API base URL core GET /installation

The Core administration API lists every route; errors use the Core error envelope.

Rotate the Core key

~/.oac/core/oac rotate-core-key

It runs in the initialization container, updates secrets/web/core.key and secrets/core/core-key-digests.json in the data volume, and restarts Core and Web. The old key stops working as soon as Core restarts, and every console session ends: sign in again and update your scripts.

Projects and API keys

Create Projects and issue keys in Web, on Projects and keys, or through the Core API. How Projects and keys behave is in Projects own assets.

To rotate an application key:

  1. Issue a new key in the same Project.
  2. Update the application to use it.
  3. Revoke the old key.

Archive disables every key of a Project and keeps its assets.

Core records which key made each public resource write; the retention of that history is OAC_WRITE_AUDIT_RETENTION.

Back up

A recoverable backup contains one consistent, stopped-write set. A database dump or Core volume export alone does not include independent workspace storage or node compute state. Restore the data volume together with the installation directory from the same backup, which holds the release that wrote it.

Single-host backup and restore

For an installation whose data all lives in its Docker volume, with no independent workspace storage and no node state to keep, the data volume and the installation directory are the whole backup. This recipe uses a Unix shell and the PostgreSQL image the installation already pulled.

cd ~/.oac/core
docker compose stop
project=$(sed -n 's/^COMPOSE_PROJECT_NAME=//p' .env)
image=$(docker compose images --quiet database)
mkdir -p ~/oac-backup
docker run --rm -v "${project}_data:/data:ro" -v ~/oac-backup:/backup --entrypoint tar "$image" czf /backup/data.tar.gz -C /data .
cp -a ~/.oac/core ~/oac-backup/installation
docker compose start

To restore on this host or another, first uninstall any installation that still uses the project, then:

cp -a ~/oac-backup/installation ~/.oac/core
cd ~/.oac/core
project=$(sed -n 's/^COMPOSE_PROJECT_NAME=//p' .env)
docker compose pull
docker compose up --no-start
image=$(docker compose images --quiet database)
docker run --rm -v "${project}_data:/data" -v ~/oac-backup:/backup:ro --entrypoint tar "$image" xzf /backup/data.tar.gz -C /data
./oac apply

An installation with independent workspace storage or nodes needs the full procedure below.

Backup contents

  • The Docker volume <project>_data, including database/, secrets/ and state/. It holds Projects, key digests, nodes, default models, encrypted credentials and execution history, including large objects. Keep secrets/core/credential.key with its matching database or stored credentials cannot be decrypted.
  • The installation directory, including .env, compose.yaml, Compose overrides and the management command.
  • Every independent workspace namespace selected by current or retained objects, even when it is outside the installation volume. Back up its entire root, including .oac-storage-root, object identities, staging, live data, trash and deletion markers. Preserve numeric ownership, modes, links and extended attributes, including every user.* attribute. Copying only live/data loses lifecycle evidence and can resurrect deleted identities. Workspace storage owns the layout and ownership requirements.
  • For a quota mode namespace, the entire XFS filesystem holding the export as one block-level copy: a storage or LVM snapshot, or an image of the frozen or unmounted device. Only such a copy keeps project IDs, quota limits, inode identities, the filesystem UUID and the oac-workspace-quota journal consistent. A file-level copy, tar, rsync or xfsdump loses them; the command then refuses the restored objects instead of serving them unbounded.
  • Each node's state directory, /var/lib/oac-node/.oac/nodes/<installation-id>/, and its provider storage: Docker volumes or microsandbox's store. Include all compute and snapshot resources still referenced by the database. See when a node host fails.
  • Each private checkpoint_root in its entirety, including .oac-checkpoint-store, checkpoints, operation journals, lock files and tombstones, together with the matching Core database. The checkpoint transfer contract owns its identity, ownership and lifecycle.

Establish a stopped-write window

  1. Block new application input, live file access and administrative mutations at the installation's ingress. Let active execution, initialization, file operations and native cleanup finish. Stop other applications or host processes that can write to the same filesystem.
  2. While Core and nodes remain available, confirm that every owned writer has stopped and all Create, Kill, snapshot cleanup and filesystem mutations have settled. An offline node, timeout, missing instance or stopped service is insufficient. A confirmed suspension stops its source compute but retains a checkpoint that must be included in the backup. For the simplest cold recovery baseline, wait until eligible retained Sessions have completed checkpoint expiry cleanup and their allocations are released; retain their filesystem objects and native history under the cold replacement contract.
  3. Stop Core and Web, then stop the node control services to prevent new lifecycle work. Recheck the exact native resources and keep all writers stopped until every part of the backup is complete. Stopping Core or a node service does not stop its sandboxes. The node service uses KillMode=process; microVMs and other provider-managed compute may continue to run independently.
  4. Flush the stopped filesystem's pending writes using the storage platform's supported procedure and take its snapshot or export. For quota mode, freeze the XFS filesystem with xfs_freeze -f or unmount it while copying its device, then thaw it. Export the Core data volume and installation directory, plus required node state and provider stores, within the same window. Stop PostgreSQL before copying its database/ files. Record the release, backup time, component inventory and checksums together, and keep the backup outside the installation being backed up.
  5. Restore service availability only after all copies finish. Bring up storage before Core and nodes, verify readiness, then reopen ingress. Do not replay uncertain native work to make a backup pass.

There is no one-command, lossless maintenance drain. Archive and deployment reset change Session lifecycle state; they are not substitutes for a backup pause that preserves resumability. If any writer or mutation remains unknown, keep the backup incomplete and resolve that ownership first.

For a logical database copy during the stopped-write window, leave only the database service running and use:

docker compose -f "$HOME/.oac/core/compose.yaml" exec -T database \
  pg_dump -U oac oac > oac-backup.sql

A logical dump still needs the matching secrets, state, independent filesystems and required node resources. Docker Desktop's Volumes export is a way to copy a stopped data volume; it does not coordinate these other components.

Restore and rehearse

Keep application ingress and Core/node services stopped while restoring. Fence or shut down every old writer before exposing the restored filesystem; an unreachable old host is not a fence. A rehearsal must use an isolated installation and storage copy that cannot connect to the original nodes or share their writable namespace.

Restore matching secrets and database state, the complete independent filesystem namespaces and any required node identities/provider stores before starting Core. Preserve the database's configuration and namespace UUIDs, object identities and deletion markers; do not run first-time namespace initialization over restored storage. Restore a quota mode filesystem by writing its block-level copy to the backing device and mounting it with prjquota at the configured mount point; never run oac-workspace-quota init over it, and do not mount the copy beside the original, which has the same filesystem UUID. Reestablish the configured mount paths and service UIDs, then validate the mounted filesystem, ownership and xattrs from each participating service context. Set up workspace storage owns deployment setup.

A cold restore resumes only qualified native history after new compute admission. A backup containing retained checkpoints also depends on the matching native runtime, node identity, provider store and external filesystem identities. File-level restore can change inode identity and prevent strict checkpoint restore even when file bytes match. This procedure restores onto the original node identities; checkpoint transfer defines when another node can restore a checkpoint. Preserve unresolved ownership rather than substituting a new sandbox.

Start the database, Core and node services only after their restored dependencies are complete; keep ingress restricted while checking them. In an isolated rehearsal, verify a retained Session's files and native continuation on fresh compute where qualified, confirm that deleted objects cannot be reopened, and check that no old writer can access the restored namespace. Record the results before relying on the backup for recovery.

Uninstall

Remove nodes first; see Remove a node.

~/.oac/core/oac uninstall

On Windows, run & "$HOME/.oac/core/oac.exe" uninstall. The command shows the Compose project, the data volume, the installation directory and the installation ID, and deletes nothing until you type the directory name or yes. It then removes the containers, the network and the data volume, the installation directory and the installation lock (~/.oac/core.lock). On Windows the running oac.exe cannot delete itself: the command prints the Remove-Item that removes the rest, and you run it before you reinstall into the same directory. It keeps the Docker images so that a reinstall doesn't download them again, and prints the docker image rm command that removes them.

All data goes with it: Projects and API keys, Session history, stored credentials and the Core key. To keep the data, stop the installation with docker compose stop instead, or back it up first.

Uninstall stops no sandbox: node sandboxes keep running on their nodes, and E2B sandboxes keep running, and billing, at E2B. While Core is still up, archive their Sessions or reset the deployment and let it complete.

If Core is already gone, run the node uninstall command with --force on each node host, as described there, using the installation ID that oac uninstall printed or the one in the node's unit name oac-node-<installation-id>.service.

If the installation directory was deleted by hand, its containers keep running and keep the Web port. Find the project and remove it without any installation file:

docker compose ls --all
docker compose -p <project> down --volumes --remove-orphans

The project is the oac-… name from the list. Then delete ~/.oac/core.lock (on Windows also ~/.oac/.core.lock.runtime.lock) if it is still there.