diff --git a/docs/getting-started/operations.md b/docs/getting-started/operations.md index 713843fc6..5ebee95a7 100644 --- a/docs/getting-started/operations.md +++ b/docs/getting-started/operations.md @@ -165,7 +165,7 @@ An installation with independent workspace storage or nodes needs the full proce - For a [quota mode](../workspace-provider.md#quota-mode) namespace, the entire XFS filesystem holding the export as one block-level copy: a storage or LVM snapshot, or an image of the frozen or unmounted device. Only such a copy keeps project IDs, quota limits, inode identities, the filesystem UUID and the `oac-workspace-quota` journal consistent. A file-level copy, `tar`, `rsync` or `xfsdump` loses them; the command then refuses the restored objects instead of serving them unbounded. - Each node's state directory, `/var/lib/oac-node/.oac/nodes//`, and its provider storage: Docker volumes or microsandbox's store. Include all compute and snapshot resources still referenced by the database. See [when a node host fails](./nodes.md#when-a-node-host-fails). -- Each private `checkpoint_root` in its entirety, including `.oac-checkpoint-store`, archives, operation journals, lock files and tombstones, together with the matching Core database. It is separate from the native SDK store and independent workspace namespace. Preserve its identity and private ownership; the [checkpoint transfer contract](../sandbox-provider.md#checkpoint-transfer) owns its lifecycle. +- Each private `checkpoint_root` in its entirety, including `.oac-checkpoint-store`, checkpoints, operation journals, lock files and tombstones, together with the matching Core database. It is separate from the native SDK store and independent workspace namespace. Preserve its identity and private ownership; the [checkpoint transfer contract](../sandbox-provider.md#checkpoint-transfer) owns its lifecycle. ### Establish a stopped-write window diff --git a/docs/sandbox-provider.md b/docs/sandbox-provider.md index 9dada3267..79f914bb2 100644 --- a/docs/sandbox-provider.md +++ b/docs/sandbox-provider.md @@ -256,11 +256,11 @@ A Provider relies on these rules: A checkpoint-capable generation reports `CheckpointCompatibility`: opaque `ArtifactDomain` and `ExecutionClass` tokens. Every verified `SnapshotIdentity` carries the same qualification. Core compares these values without interpreting CPU features, filesystem paths or storage implementations. A restore destination must be online and ready for the allocation's immutable deployment generation, match both tokens, and have active capacity plus a retained slot when moving from another node. The allocation, Device and Session identities remain unchanged. The Session and deployment transaction commits the destination route, placement and exact restore intent before target-side native work. No capacity or compatible destination leaves the retained snapshot owned until its configured deadline. The target must already hold that exact generation; Core does not automatically prepare historical generations on new nodes. A node keeps a superseded generation's provider until its checkpoint retention obligations end. -`Suspend` may report `SourceStopped` only after publishing a verified full archive durably, stopping the exact source compute and settling source-local capture and cleanup obligations. Core persists the snapshot before marking it suspended; native absence alone is insufficient. `Suspend.ObserveOnly` may finish archive publication and source cleanup for an already verified snapshot of the same operation, but cannot capture another snapshot or stop a source that has resumed running. If a previously published archive is missing or damaged, `ObserveOnly` may return its exact durable ownership receipt with `Status: unknown` and `SourceStopped: false` for cleanup; that receipt proves neither current archive usability nor source absence. `Resume` still verifies the archive before execution. After the source-settlement barrier, source-node unavailability does not prevent restoration. Deletion and ordinary expiry can transfer an idle suspended allocation's cleanup route to an online node in the same artifact domain; deletion does not require execution-class compatibility. The cleanup destination must already serve the same immutable generation and have retained capacity. Without such a node, cleanup stays pending and ownership is not released; operators must restore an eligible node or its capacity. An unknown running or restoring writer never moves merely because its node is unreachable. +`Suspend` may report `SourceStopped` only after publishing a verified full checkpoint durably, stopping the exact source compute and settling source-local capture and cleanup obligations. Core persists the snapshot before marking it suspended; native absence alone is insufficient. `Suspend.ObserveOnly` may finish checkpoint publication and source cleanup for an already verified snapshot of the same operation, but cannot capture another snapshot or stop a source that has resumed running. If a previously published checkpoint is missing or damaged, `ObserveOnly` may return its exact durable ownership receipt with `Status: unknown` and `SourceStopped: false` for cleanup; that receipt proves neither current checkpoint usability nor source absence. `Resume` still verifies the checkpoint before execution. After the source-settlement barrier, source-node unavailability does not prevent restoration. Deletion and ordinary expiry can transfer an idle suspended allocation's cleanup route to an online node in the same artifact domain; deletion does not require execution-class compatibility. The cleanup destination must already serve the same immutable generation and have retained capacity. Without such a node, cleanup stays pending and ownership is not released; operators must restore an eligible node or its capacity. An unknown running or restoring writer never moves merely because its node is unreachable. -Recovery uses `Resume.ObserveOnly` for the persisted target and operation. `RestoreAttemptClosed` is an exact operation ID, not a generic absence flag: the adapter may return it only after durably closing an attempt that never entered native execution and fencing every late request for that ID. An adapter may also close a durably admitted attempt when it can prove that its current invocation failed before dispatching native restoration. The microsandbox adapter does this only when the fresh request's wire deadline expires before native dispatch; archive corruption, qualification errors and opaque SDK failures do not authorize an automatic retry. Core then persists a new operation ID before attempting execution. Once native restoration has been dispatched, an unknown outcome retains the same target, ownership and original retention deadline; a missing native listing, helper death or timeout does not authorize replay. Such a restore can remain unavailable until explicit deletion or ordinary retention expiry, and pending inputs can reach their public deadline while waiting. User deletion still requires exact native settlement before resources and ownership can be released. +Recovery uses `Resume.ObserveOnly` for the persisted target and operation. `RestoreAttemptClosed` is an exact operation ID, not a generic absence flag: the adapter may return it only after durably closing an attempt that never entered native execution and fencing every late request for that ID. An adapter may also close a durably admitted attempt when it can prove that its current invocation failed before dispatching native restoration. The microsandbox adapter does this only when the fresh request's wire deadline expires before native dispatch; checkpoint corruption, qualification errors and opaque SDK failures do not authorize an automatic retry. Core then persists a new operation ID before attempting execution. Once native restoration has been dispatched, an unknown outcome retains the same target, ownership and original retention deadline; a missing native listing, helper death or timeout does not authorize replay. Such a restore can remain unavailable until explicit deletion or ordinary retention expiry, and pending inputs can reach their public deadline while waiting. User deletion still requires exact native settlement before resources and ownership can be released. -The microsandbox adapter requires an explicit private `checkpoint_root`, outside every guest-accessible filesystem. Its directory is mode `0700`; a private namespace marker identifies the actual archive store. The default installation creates a node-local private store. Cross-node restoration requires operators to mount the same private store on each participating node; mount paths may differ. Runtime state directories and workspace bindings are never used to infer this root. The adapter verifies the archive, external filesystem identity and native execution class again before restoration. Restored RAM and file descriptors do not promise continuity of external TCP connections or replay safety for application side effects. +The microsandbox adapter requires an explicit private `checkpoint_root`, outside every guest-accessible filesystem. Its directory is mode `0700`; a private namespace marker identifies the actual checkpoint store. The default installation creates a node-local private store. Cross-node restoration requires operators to mount the same private store on each participating node; mount paths may differ. Each retained checkpoint is an uncompressed native snapshot of the guest's memory and root-disk layers, so a shared store carries that volume for every suspension and cross-node restore. Runtime state directories and workspace bindings are never used to infer this root. The adapter verifies the checkpoint, external filesystem identity and native execution class again before restoration. Restored RAM and file descriptors do not promise continuity of external TCP connections or replay safety for application side effects. ### Cold replacement after checkpoint retention diff --git a/docs/zh/getting-started/operations.md b/docs/zh/getting-started/operations.md index d3d8b8234..2c3171110 100644 --- a/docs/zh/getting-started/operations.md +++ b/docs/zh/getting-started/operations.md @@ -1,7 +1,7 @@ --- title: "运维" source: docs/getting-started/operations.md -source_hash: 2f6435641f47d00db1ae4556594d99367223270013dfdf282c16648f8a6cd275 +source_hash: c61c4d11097be8e0718c6c095c55b10a47020e5a2f83f24c504eb6753ca00b5c --- Core 安装运维人员的日常任务:`oac` 命令、服务健康状态、Core 密钥、备份和卸载。节点有[专门的指南](./nodes.md),[故障排查](./troubleshooting.md)列出了安装程序和访问方面的提示信息。设置见[配置参考](../configuration.md)。 @@ -167,7 +167,7 @@ docker run --rm -v "${project}_data:/data" -v ~/oac-backup:/backup:ro --entrypoi - 对于[配额模式](../workspace-provider.md#quota-mode)命名空间,将承载导出的整个 XFS 文件系统作为一份块级副本备份:存储或 LVM 快照,或已冻结或已卸载设备的镜像。只有这样的副本能保持 project ID、配额限制、inode 标识、文件系统 UUID 和 `oac-workspace-quota` 日志库的一致。文件级复制、`tar`、`rsync` 或 `xfsdump` 会丢失这些信息;此时该命令会拒绝恢复后的对象,而不是在无上限状态下提供它们。 - 各节点状态目录 `/var/lib/oac-node/.oac/nodes//` 及其提供商存储:Docker 卷或 microsandbox 存储。包含数据库仍然引用的全部计算资源和快照。参见[节点主机故障时](./nodes.md#when-a-node-host-fails)。 -- 每个私有 `checkpoint_root` 的全部内容,包括 `.oac-checkpoint-store`、归档、operation journal、lock 文件与 tombstone,并与匹配的 Core 数据库一起保留。它独立于原生 SDK store 和工作区 namespace。保留其身份和私有所有权;[检查点转移契约](../sandbox-provider.md#checkpoint-transfer)定义其生命周期。 +- 每个私有 `checkpoint_root` 的全部内容,包括 `.oac-checkpoint-store`、检查点、operation journal、lock 文件与 tombstone,并与匹配的 Core 数据库一起保留。它独立于原生 SDK store 和工作区 namespace。保留其身份和私有所有权;[检查点转移契约](../sandbox-provider.md#checkpoint-transfer)定义其生命周期。 ### 建立停止写入窗口 {#establish-a-stopped-write-window} diff --git a/docs/zh/sandbox-provider.md b/docs/zh/sandbox-provider.md index 4886b9aaa..4c153364b 100644 --- a/docs/zh/sandbox-provider.md +++ b/docs/zh/sandbox-provider.md @@ -1,7 +1,7 @@ --- title: "沙箱提供商协议" source: docs/sandbox-provider.md -source_hash: 9d36546c75d10805c104092536d8bb13a87b887bef1573f2f4fd5ee94d1673c7 +source_hash: 6476bd4c551ee002c2006eba645cfd6ebe3f107b9369570d2fa386fb9c0ef132 --- 沙箱提供商协议连接 Core 与为 Core 管理的(`openai_hosted`)Environment 提供计算资源的 adapter,并由该 adapter 运行启动 Runtime daemon 的有界引导流程。本页面向 adapter 作者:说明提供商必须实现什么,以及可以依赖 Core 做什么。Docker、microsandbox 和 E2B 实现了该协议。 @@ -258,11 +258,11 @@ Core 在初始化完成、没有 root 或 Subagent Turn 处于排队、进行中 支持检查点的 generation 报告 `CheckpointCompatibility`,包含不透明的 `ArtifactDomain` 与 `ExecutionClass` token。每个已验证 `SnapshotIdentity` 携带同样的资格。Core 只比较这些值,不解释 CPU 特性、文件系统路径或存储实现。恢复目标必须在线、对 allocation 的不可变 deployment generation 已就绪、两个 token 均匹配,并有 active 容量;跨 node 时还需 retained slot。allocation、Device 和 Session 身份保持不变。Session 与 deployment 事务先提交目标路由、placement 和精确 restore intent,再执行目标侧原生操作。没有容量或兼容目标时,保留快照的所有权一直持续到配置期限。目标必须已持有该精确 generation;Core 不会在新节点自动准备历史 generation。节点会保留被取代的 generation 的提供商,直到检查点保留义务结束。 -`Suspend` 只有在持久发布已验证的完整归档、停止精确 source compute,并结清 source 本地 capture 与 cleanup 义务后,才能报告 `SourceStopped`。Core 先持久化快照,再标记 suspended;仅原生资源不存在并不足够。`Suspend.ObserveOnly` 可以为同一 operation 已验证的快照完成归档发布和源清理,但不能重新捕获快照,也不能停止已经恢复 running 的源。此前已发布的归档缺失或损坏时,`ObserveOnly` 可以返回其精确持久 ownership receipt,标记 `Status: unknown` 和 `SourceStopped: false`,供清理使用;该 receipt 不证明归档当前可用,也不证明源不存在。`Resume` 仍在执行前验证归档。完成源结清屏障后,source node 不可用不再阻止恢复。删除和正常到期可以将空闲 suspended allocation 的清理路由转给同一 artifact domain 内的在线 node;删除不要求 execution class 匹配。清理目标必须已提供同一不可变 generation,且有 retained 容量。没有这样的 node 时,清理保持 pending,不释放所有权;运维需恢复符合条件的 node 或其容量。未知 running 或 restoring writer 不会仅因 node 不可达而迁移。 +`Suspend` 只有在持久发布已验证的完整检查点、停止精确 source compute,并结清 source 本地 capture 与 cleanup 义务后,才能报告 `SourceStopped`。Core 先持久化快照,再标记 suspended;仅原生资源不存在并不足够。`Suspend.ObserveOnly` 可以为同一 operation 已验证的快照完成检查点发布和源清理,但不能重新捕获快照,也不能停止已经恢复 running 的源。此前已发布的检查点缺失或损坏时,`ObserveOnly` 可以返回其精确持久 ownership receipt,标记 `Status: unknown` 和 `SourceStopped: false`,供清理使用;该 receipt 不证明检查点当前可用,也不证明源不存在。`Resume` 仍在执行前验证检查点。完成源结清屏障后,source node 不可用不再阻止恢复。删除和正常到期可以将空闲 suspended allocation 的清理路由转给同一 artifact domain 内的在线 node;删除不要求 execution class 匹配。清理目标必须已提供同一不可变 generation,且有 retained 容量。没有这样的 node 时,清理保持 pending,不释放所有权;运维需恢复符合条件的 node 或其容量。未知 running 或 restoring writer 不会仅因 node 不可达而迁移。 -恢复用 `Resume.ObserveOnly` 观察已持久化的 target 与 operation。`RestoreAttemptClosed` 是精确 operation ID,不是通用 absence 标志:adapter 只有在持久关闭从未进入原生执行的 attempt,并隔离该 ID 的所有迟到请求后才能返回。如果 adapter 能证明当前调用在派发原生恢复之前失败,也可以关闭已持久 admitted 的 attempt。microsandbox adapter 仅在 fresh 请求的 wire deadline 于原生派发之前到期时执行这种关闭;归档损坏、资格错误和不透明 SDK 失败都不授权自动重试。Core 随后先持久化新的 operation ID,再尝试执行。一旦已派发原生恢复,结果未知的 attempt 保留相同 target、所有权与原始 retention deadline;原生列表缺失、helper 死亡或超时都不授权重放。这种恢复可能一直不可用,直到显式删除或正常保留期到期;等待中的 input 可能达到其公共 deadline。用户删除仍需精确原生 settlement,之后才能释放资源与所有权。 +恢复用 `Resume.ObserveOnly` 观察已持久化的 target 与 operation。`RestoreAttemptClosed` 是精确 operation ID,不是通用 absence 标志:adapter 只有在持久关闭从未进入原生执行的 attempt,并隔离该 ID 的所有迟到请求后才能返回。如果 adapter 能证明当前调用在派发原生恢复之前失败,也可以关闭已持久 admitted 的 attempt。microsandbox adapter 仅在 fresh 请求的 wire deadline 于原生派发之前到期时执行这种关闭;检查点损坏、资格错误和不透明 SDK 失败都不授权自动重试。Core 随后先持久化新的 operation ID,再尝试执行。一旦已派发原生恢复,结果未知的 attempt 保留相同 target、所有权与原始 retention deadline;原生列表缺失、helper 死亡或超时都不授权重放。这种恢复可能一直不可用,直到显式删除或正常保留期到期;等待中的 input 可能达到其公共 deadline。用户删除仍需精确原生 settlement,之后才能释放资源与所有权。 -microsandbox adapter 要求显式的私有 `checkpoint_root`,位于所有 guest 可访问文件系统之外。目录权限为 `0700`,私有 namespace marker 标识真实归档存储。默认安装创建 node 本地私有 store。跨 node 恢复要求运维为参与 node 挂载同一个私有 store,挂载路径可以不同。Runtime state 目录与 workspace binding 都不用于推导该 root。adapter 在恢复前再次验证归档、外部文件系统身份和原生 execution class。恢复 RAM 与文件描述符不保证外部 TCP 连接连续,也不保证应用副作用可安全重放。 +microsandbox adapter 要求显式的私有 `checkpoint_root`,位于所有 guest 可访问文件系统之外。目录权限为 `0700`,私有 namespace marker 标识真实检查点存储。默认安装创建 node 本地私有 store。跨 node 恢复要求运维为参与 node 挂载同一个私有 store,挂载路径可以不同。每个保留的检查点都是 guest 内存与根磁盘层的未压缩原生快照,因此共享 store 要为每次挂起和跨 node 恢复承载这部分数据量。Runtime state 目录与 workspace binding 都不用于推导该 root。adapter 在恢复前再次验证检查点、外部文件系统身份和原生 execution class。恢复 RAM 与文件描述符不保证外部 TCP 连接连续,也不保证应用副作用可安全重放。 ### 检查点保留期后的冷替换 {#cold-replacement-after-checkpoint-retention} diff --git a/services/core/internal/sandbox/microsandbox/checkpoint_linux.go b/services/core/internal/sandbox/microsandbox/checkpoint_linux.go index ff7412118..78ae4412e 100644 --- a/services/core/internal/sandbox/microsandbox/checkpoint_linux.go +++ b/services/core/internal/sandbox/microsandbox/checkpoint_linux.go @@ -16,7 +16,7 @@ import ( "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" ) -// CheckpointStore opens only an explicitly configured, private archive root. +// CheckpointStore opens only an explicitly configured, private checkpoint root. // Its marker is an operator-owned identity, never inferred from workspace paths. func CheckpointStore(c Config) (string, error) { if !filepath.IsAbs(c.CheckpointRoot) || filepath.Clean(c.CheckpointRoot) != c.CheckpointRoot { diff --git a/services/core/internal/sandbox/sandbox_provider.go b/services/core/internal/sandbox/sandbox_provider.go index d4fc5b8e3..712eaf4fe 100644 --- a/services/core/internal/sandbox/sandbox_provider.go +++ b/services/core/internal/sandbox/sandbox_provider.go @@ -152,7 +152,7 @@ type Compute struct { RestoredFrom *SnapshotIdentity } -// CheckpointCompatibility identifies an adapter-verified private archive domain +// CheckpointCompatibility identifies an adapter-verified private checkpoint domain // and execution compatibility class. Core compares these opaque tokens only. type CheckpointCompatibility struct { ArtifactDomain string `json:"artifact_domain"` @@ -190,7 +190,7 @@ type ComputeState struct { Status string BootstrapComplete bool Snapshot *SnapshotIdentity - // SourceStopped proves the archive is durable and all source-local compute + // SourceStopped proves the checkpoint is durable and all source-local compute // and capture obligations have settled, so another node can own restoration. SourceStopped bool // RestoreAttemptClosed echoes the exact Resume OperationID only after a diff --git a/services/core/tools/microsandbox-provider/README.md b/services/core/tools/microsandbox-provider/README.md index bf7d44563..c307b0fdf 100644 --- a/services/core/tools/microsandbox-provider/README.md +++ b/services/core/tools/microsandbox-provider/README.md @@ -14,7 +14,7 @@ The runtime home must be private (mode 0700), short, on local persistent storage The Runtime image is an immutable `repository@sha256:<64 lowercase hex>` reference that the node's `provider.json` records as `native.image`; bare image IDs and mutable tags are rejected. The node installer imports the distribution's image under that reference. To load an image by hand, `msb image load --tag repository@sha256:` must register the digest reference explicitly, with the manifest digest from `image inspect`, not the Docker image config ID. The image carries the daemon, Python 3, the native Harnesses and the shared Runtime helpers. The provider installs no registry credentials. Creation uses the SDK's `never` pull policy. -The checkpoint root is a separate adapter-owned directory, mode 0700, with a private `.oac-checkpoint-store` UUID marker. It is never derived from a workspace binding and must not be reachable through any guest mount. Archives contain confidential RAM and root-disk state. The marker and installation identity define the artifact domain; native release and binary hashes, Runtime image, kernel, CPU features and guest geometry define the conservative execution class. Filesystem paths, hostnames and local SDK homes do not define compatibility. Nodes can share this private root at different mount paths; the backing store must preserve durable writes, atomic rename and cross-host file locks. The installer's local default only advertises its own domain. +The checkpoint root is a separate adapter-owned directory, mode 0700, with a private `.oac-checkpoint-store` UUID marker. It is never derived from a workspace binding and must not be reachable through any guest mount. Snapshots contain confidential RAM and root-disk state. The marker and installation identity define the artifact domain; native release and binary hashes, Runtime image, kernel, CPU features and guest geometry define the conservative execution class. Filesystem paths, hostnames and local SDK homes do not define compatibility. Nodes can share this private root at different mount paths; the backing store must preserve durable writes, atomic rename and cross-host file locks. The installer's local default only advertises its own domain. ## Create and bootstrap @@ -30,17 +30,17 @@ A helper response carries `CreateSettled` with a configuration rejection only af Core persists operation IDs, source and target generations, exact identities and snapshot evidence before it depends on them. `Initial` and `NewCompute` only construct references and allocate nothing. -- **Suspend** pauses the exact VM, captures and verifies the full checkpoint closure, then exports an SDK archive without the Runtime image into the private checkpoint root; the execution class pins that image, which every compatible node holds. The adapter hashes the archive once as it publishes it and synchronizes the archive and its digest, size and ownership receipt before force-stopping the exact paused source, removing its writable disks and deleting its local SDK snapshot. Only after that cleanup is synchronized does `SourceStopped` authorize transfer. A matching artifact is completed instead of captured again; a running source is never killed to finish an old capture. +- **Suspend** pauses the exact VM and has the SDK capture the full checkpoint closure directly into its own group directory `s-` in the private checkpoint root (`SnapshotCreateOptions.DestDir`). The SDK makes the snapshot durable (the closure is hard-linked on the same filesystem, otherwise copied and fsynced, then its directories, descriptor and group head are synced). The adapter verifies the snapshot once, writes the ownership receipt next to it, force-stops the exact paused source and removes its writable disks. Only after that cleanup is synchronized does `SourceStopped` authorize transfer. The snapshot excludes the Runtime image; the execution class pins that image, which every compatible node holds. A matching snapshot is completed instead of captured again; a running source is never killed to finish an old capture. - **KillCompute** fences future admission for the exact target and checks its incarnation before stopping and removing it. Under that dispatch fence the SDK serializes against native launch, waits for the runtime lifecycle lock on Kill, and rechecks the terminal incarnation under that lock on Remove. A final typed absence permits a durable cleanup receipt, including for an admitted restore whose launcher died. This proves the writer is gone, never that it did not execute. -- **Restore** requires the source-settled archive receipt, matching artifact domain and execution class, and the complete archive hash before importing through the SDK, then the imported snapshot's full verification. Before native restore, the snapshot's captured image digest must equal the pinned Runtime image digest and that image must be in the local image store; otherwise restore fails as invalid without starting the target. This metadata check does not prove that every cached image object is present. The pinned SDK's restore has no pull-policy option and uses `if-missing`; missing cache objects can trigger a registry request pinned to the captured manifest digest. A failed resolution leaves the restore unresolved for exact observation and cleanup; it never substitutes another image or cold-boots the Session. Restore then creates the precommitted target name. An existing target is adopted only when its immutable ID, if known, and its persisted `snapshot_parent` agree. Upstream restore defaults to public networking, so restore passes the same explicit host policy as creation, and no undeclared host resource or mount is inherited. +- **Restore** requires the source-settled receipt, matching artifact domain and execution class, and the snapshot's full verification, then restores directly from the snapshot directory; another node that mounts the same checkpoint root does the same. Before native restore, the snapshot's captured image digest must equal the pinned Runtime image digest and that image must be in the local image store; otherwise restore fails as invalid without starting the target. This metadata check does not prove that every cached image object is present. The pinned SDK's restore has no pull-policy option and uses `if-missing`; missing cache objects can trigger a registry request pinned to the captured manifest digest. A failed resolution leaves the restore unresolved for exact observation and cleanup; it never substitutes another image or cold-boots the Session. Restore then creates the precommitted target name. An existing target is adopted only when its immutable ID, if known, and its persisted `snapshot_parent` agree. Upstream restore defaults to public networking, so restore passes the same explicit host policy as creation, and no undeclared host resource or mount is inherited. - Native restore leaves the managed root size unset because the target inherits the verified full snapshot. The helper accepts that only with a matching snapshot resource proof and the exact source and target identities, and it checks the target's CPU and memory before keeping the inherited proof. A missing root size never counts as unlimited capacity, and retained state is never resized. - Fresh restore and retry share one completion: verify the original artifact and resource proof, inspect the running target's resources and ancestry, persist its missing derived resource-proof label, then strictly reread the same native ID ([`restore_completion.go`](restore_completion.go)). A conflicting proof is an error. Native restore does not copy the source's ownership labels; ancestry supplies that evidence. There is no ordinary Start, replacement, disk-only restore or cold boot. - **ResumeCompute** thaws the same resident source after an aborted suspension. The pinned SDK handle method is name-based; the allocation lock and ID checks before and after the call fence every managed replacement. Manual lifecycle changes in the managed namespace are unsupported. -- **DeleteSnapshot** accepts only the derived operation selector and matching artifact identity. It compares the SDK descriptor digest, which covers the checkpoint root, instead of re-hashing payloads, so damaged content never blocks owned cleanup. It fences delayed publication and restoration with a durable deletion receipt before deleting the local SDK snapshot and portable archive. An unsettled admitted restore blocks deletion. The small ownership tombstone remains; Core owns retention and cleanup order. +- **DeleteSnapshot** accepts only the derived operation selector and matching artifact identity. It fences delayed publication and restoration with a durable deletion receipt, then removes everything else in the operation directory without opening the snapshot, so damaged content or a staging directory left by an interrupted capture never blocks owned cleanup. An unsettled admitted restore blocks deletion. The small ownership tombstone remains; Core owns retention and cleanup order. -After a lost response Core uses `ObserveOnly`. It never starts a capture or restore. For an existing verified capture it may complete archive publication and exact paused-source cleanup, but never captures again or kills a running source. Source termination is recorded before local removal so a lost cleanup response can be settled from the archive receipt. A valid ownership receipt remains discoverable when archive bytes are missing or corrupt: observation returns its snapshot identity with unknown source state and `SourceStopped=false`, so ordinary explicit cleanup can proceed. Restore still requires complete archive verification. An interrupted restore may finish the derived proof on the exact running target, but never restarts a stopped target or restores again. +After a lost response Core uses `ObserveOnly`. It never starts a capture or restore. For an existing verified capture it may complete the receipt and exact paused-source cleanup, but never captures again or kills a running source. Source termination is recorded before local removal so a lost cleanup response can be settled from the receipt. A valid ownership receipt remains discoverable when the snapshot is missing or corrupt: observation returns its snapshot identity with unknown source state and `SourceStopped=false`, so ordinary explicit cleanup can proceed. Restore still requires full snapshot verification. An interrupted restore may finish the derived proof on the exact running target, but never restarts a stopped target or restores again. -Before native import or restore, the helper synchronizes an admission record bound to the request operation, snapshot and target. Observation of a never-admitted operation writes a permanent closed record and returns that exact `RestoreAttemptClosed`; late dispatch is rejected. A fresh dispatch checks its existing request deadline after admission and immediately before calling native Restore. If the deadline has elapsed, it durably closes its exact attempt while still holding the allocation lock and returns without invoking Restore. Native FFI waits remain uncancelled. Import or qualification errors keep the admission open, avoiding repeated attempts against an invalid archive. This proof is never inferred from a previous admission or a native Restore error. After admission, helper death or native absence alone never establishes no effect. Observation checks an admitted target before reading the archive; typed absence or a non-running target remains unconfirmed without hashing or importing the archive. A running target still requires complete archive verification. Successful resume requires an observable exact running target. A failed or uncertain admitted restore retains its artifact and owner until explicit deletion or the original retention deadline; the normal cleanup operation uses the native lifecycle fence and records completion. It never shortens retention or substitutes a cold restart. +Before native restore, the helper synchronizes an admission record bound to the request operation, snapshot and target. Observation of a never-admitted operation writes a permanent closed record and returns that exact `RestoreAttemptClosed`; late dispatch is rejected. A fresh dispatch checks its existing request deadline after admission and immediately before calling native Restore. If the deadline has elapsed, it durably closes its exact attempt while still holding the allocation lock and returns without invoking Restore. Native FFI waits remain uncancelled. Qualification errors keep the admission open, avoiding repeated attempts against an invalid snapshot. This proof is never inferred from a previous admission or a native Restore error. After admission, helper death or native absence alone never establishes no effect. Observation checks an admitted target before reading the snapshot; typed absence or a non-running target remains unconfirmed without verifying it. A running target still requires full snapshot verification. Successful resume requires an observable exact running target. A failed or uncertain admitted restore retains its artifact and owner until explicit deletion or the original retention deadline; the normal cleanup operation uses the native lifecycle fence and records completion. It never shortens retention or substitutes a cold restart. GetCompute, commands, cleanup and the next suspension verify restored provenance from the persisted VM configuration after the consumed artifact is deleted. diff --git a/services/core/tools/microsandbox-provider/backend.go b/services/core/tools/microsandbox-provider/backend.go index 2bf23bfeb..3da947ba6 100644 --- a/services/core/tools/microsandbox-provider/backend.go +++ b/services/core/tools/microsandbox-provider/backend.go @@ -57,7 +57,7 @@ func (b backend) run(ctx context.Context) (wire.Response, error) { s, e := b.resume(ctx, *b.q.Resume) return wire.Response{State: &s}, e case "delete_snapshot": - return wire.Response{}, b.deleteArchive(ctx, *b.q.Snapshot) + return wire.Response{}, b.deleteCheckpoint(ctx, *b.q.Snapshot) } return wire.Response{}, sandbox.ErrInvalid } diff --git a/services/core/tools/microsandbox-provider/archive.go b/services/core/tools/microsandbox-provider/checkpoint.go similarity index 66% rename from services/core/tools/microsandbox-provider/archive.go rename to services/core/tools/microsandbox-provider/checkpoint.go index 1d1671ba9..92af874bc 100644 --- a/services/core/tools/microsandbox-provider/archive.go +++ b/services/core/tools/microsandbox-provider/checkpoint.go @@ -4,8 +4,6 @@ package main import ( "context" - "crypto/sha256" - "encoding/hex" "encoding/json" "errors" "io" @@ -21,11 +19,11 @@ import ( ) // This adapter-private namespace is never a guest mount. The allocation lock -// serializes archive publication, admission and deletion across all its nodes. -type archiveReceipt struct { +// serializes checkpoint publication, admission and deletion across all its nodes. +// The native snapshot is the checkpoint: it lives in the group directory of its +// operation, next to the receipt that proves ownership. +type checkpointReceipt struct { Snapshot wire.SnapshotIdentity `json:"snapshot"` - SHA256 string `json:"sha256"` - Size int64 `json:"size"` SourceTerminated bool `json:"source_terminated"` SourceStopped bool `json:"source_stopped"` Deleted bool `json:"deleted"` @@ -34,7 +32,7 @@ type archiveReceipt struct { func (b backend) storeDirectory() string { return filepath.Join(b.q.Config.CheckpointRoot, wire.Name(b.q.Config, b.q.Reference, 0)) } -func (b backend) archiveDirectory(operation string) string { +func (b backend) checkpointDirectory(operation string) string { return filepath.Join(b.storeDirectory(), "s-"+operation) } func privateDirectory(path string) error { @@ -114,91 +112,17 @@ func readPrivateJSON(path string, value any) error { } return nil } -func (b backend) receipt(operation string) (archiveReceipt, error) { - var r archiveReceipt - e := readPrivateJSON(filepath.Join(b.archiveDirectory(operation), "receipt.json"), &r) - if e == nil && (r.Snapshot.OperationID != operation || wire.ValidateSnapshot(b.q.Config, b.q.Reference, r.Snapshot) != nil) { +func (b backend) receipt(operation string) (checkpointReceipt, error) { + var r checkpointReceipt + e := readPrivateJSON(filepath.Join(b.checkpointDirectory(operation), "receipt.json"), &r) + if e == nil && (r.Snapshot.OperationID != operation || wire.ValidateSnapshot(b.q.Config, b.q.Reference, r.Snapshot) != nil || (r.SourceStopped && !r.SourceTerminated)) { e = sandbox.ErrOwnership } - if e == nil && !r.Deleted { - decoded, err := hex.DecodeString(r.SHA256) - if err != nil || len(decoded) != 32 || r.Size <= 0 || (r.SourceStopped && !r.SourceTerminated) { - e = sandbox.ErrOwnership - } - } return r, e } -func archiveDigest(path string) (string, int64, error) { - real, e := filepath.EvalSymlinks(path) - if e != nil { - return "", 0, e - } - if real != path { - return "", 0, sandbox.ErrOwnership - } - st, e := os.Lstat(path) - if e != nil { - return "", 0, e - } - if !st.Mode().IsRegular() || st.Mode().Perm() != 0600 { - return "", 0, sandbox.ErrOwnership - } - f, e := os.Open(path) - if e != nil { - return "", 0, e - } - defer f.Close() - h := sha256.New() - n, e := io.Copy(h, f) - return hex.EncodeToString(h.Sum(nil)), n, e -} -func (b backend) publishArchive(ctx context.Context, a *sdk.SnapshotArtifact, s wire.SnapshotIdentity) (archiveReceipt, bool, error) { - prior, e := b.receipt(s.OperationID) - if e == nil { - if prior.Snapshot != s || prior.Deleted { - return prior, false, sandbox.ErrOwnership - } - return prior, false, nil - } - if !errors.Is(e, os.ErrNotExist) { - return prior, false, e - } - dir := b.archiveDirectory(s.OperationID) - if e = privateDirectory(dir); e != nil { - return prior, false, e - } - if e = syncDirectory(b.storeDirectory()); e != nil { - return prior, false, e - } - archive := filepath.Join(dir, "archive.msb") - // A publication interrupted before its receipt can be rebuilt from the exact - // verified native artifact. The shared allocation lock excludes consumers. - if e = os.Remove(archive); e != nil && !errors.Is(e, os.ErrNotExist) { - return prior, false, e - } - if e = a.SaveTo(ctx, archive, sdk.SnapshotSaveOptions{}); e != nil { - return prior, false, e - } - if e = os.Chmod(archive, 0600); e != nil { - return prior, false, e - } - f, e := os.Open(archive) - if e != nil { - return prior, false, e - } - e = f.Sync() - f.Close() - if e != nil { - return prior, false, e - } - digest, size, e := archiveDigest(archive) - if e != nil { - return prior, false, e - } - r := archiveReceipt{Snapshot: s, SHA256: digest, Size: size} - return r, true, durableJSON(filepath.Join(dir, "receipt.json"), r) -} -func (b backend) importArchive(ctx context.Context, want wire.SnapshotIdentity) (*sdk.SnapshotArtifact, error) { + +// restoreSource returns the verified checkpoint of a source-settled receipt. +func (b backend) restoreSource(ctx context.Context, want wire.SnapshotIdentity) (*sdk.SnapshotArtifact, error) { r, e := b.receipt(want.OperationID) if e != nil { return nil, e @@ -213,45 +137,17 @@ func (b backend) importArchive(ctx context.Context, want wire.SnapshotIdentity) if class != want.Compatibility { return nil, sandbox.ErrInvalid } - path := filepath.Join(b.archiveDirectory(want.OperationID), "archive.msb") - hash, size, e := archiveDigest(path) - if e != nil { - return nil, e - } - if hash != r.SHA256 || size != r.Size { - return nil, sandbox.ErrOwnership - } - artifact, e := b.verifiedSnapshot(ctx, want) - if sdk.IsKind(e, sdk.ErrSnapshotNotFound) { - if _, e = sdk.Snapshot.LoadWithOptions(ctx, path, sdk.SnapshotLoadOptions{Group: wire.Name(b.q.Config, b.q.Reference, 0)}); e != nil { - return nil, e - } - return b.verifiedSnapshot(ctx, want) - } - return artifact, e -} -func (b backend) deleteArchive(ctx context.Context, want wire.SnapshotIdentity) error { - return b.deleteArchiveWithNative(want, func() error { - _, err := b.ownedSnapshot(ctx, want) - if err == nil { - err = sdk.Snapshot.Remove(ctx, want.Reference, false) - } - if sdk.IsKind(err, sdk.ErrSnapshotNotFound) { - return nil - } - return err - }) + return b.verifiedSnapshot(ctx, want) } - -// Native artifact removal must settle before the private archive is unlinked. -func (b backend) deleteArchiveWithNative(want wire.SnapshotIdentity, removeNative func() error) error { +func (b backend) deleteCheckpoint(ctx context.Context, want wire.SnapshotIdentity) error { if e := b.requireSettledRestores(nil, &want); e != nil { return e } + dir := b.checkpointDirectory(want.OperationID) r, e := b.receipt(want.OperationID) if errors.Is(e, os.ErrNotExist) { - r = archiveReceipt{Snapshot: want} - if e = privateDirectory(b.archiveDirectory(want.OperationID)); e != nil { + r = checkpointReceipt{Snapshot: want} + if e = privateDirectory(dir); e != nil { return e } if e = syncDirectory(b.storeDirectory()); e != nil { @@ -263,19 +159,31 @@ func (b backend) deleteArchiveWithNative(want wire.SnapshotIdentity, removeNativ if r.Snapshot != want { return sandbox.ErrOwnership } - // Fence consumers before removing either local or portable artifact. Keep the - // small tombstone, including its ownership identity, against delayed helpers. + // Fence consumers before removing the checkpoint. Keep the small tombstone, + // including its ownership identity, against delayed helpers. r.Deleted = true - if e = durableJSON(filepath.Join(b.archiveDirectory(want.OperationID), "receipt.json"), r); e != nil { + if e = durableJSON(filepath.Join(dir, "receipt.json"), r); e != nil { return e } - if e = removeNative(); e != nil { + entries, e := os.ReadDir(dir) + if e != nil { return e } - if e = os.Remove(filepath.Join(b.archiveDirectory(want.OperationID), "archive.msb")); e != nil && !errors.Is(e, os.ErrNotExist) { - return e + for _, entry := range entries { + if entry.Name() == "receipt.json" { + continue + } + // The SDK removal also drops its index row; a damaged snapshot or a + // staging directory left by an interrupted capture is removed directly. + path := filepath.Join(dir, entry.Name()) + if strings.HasPrefix(entry.Name(), "snap_") && sdk.Snapshot.Remove(ctx, path, false) == nil { + continue + } + if e = os.RemoveAll(path); e != nil { + return e + } } - return syncDirectory(b.archiveDirectory(want.OperationID)) + return syncDirectory(dir) } type restoreAdmission struct { @@ -417,17 +325,6 @@ func (b backend) requireSettledRestores(target *wire.Compute, snapshot *wire.Sna return nil } -func (b backend) verifyArchive(r archiveReceipt) error { - hash, size, e := archiveDigest(filepath.Join(b.archiveDirectory(r.Snapshot.OperationID), "archive.msb")) - if e != nil { - return e - } - if hash != r.SHA256 || size != r.Size { - return sandbox.ErrOwnership - } - return nil -} - // The native transition/lifecycle locks fence an admitted launcher and its // detached child. This receipt means cleanup completed, never "did not run". func (b backend) recordRestoreCleanup(target wire.Compute) error { diff --git a/services/core/tools/microsandbox-provider/deployment.go b/services/core/tools/microsandbox-provider/deployment.go index 322aacaf3..01870c4bb 100644 --- a/services/core/tools/microsandbox-provider/deployment.go +++ b/services/core/tools/microsandbox-provider/deployment.go @@ -74,7 +74,7 @@ func qualifySnapshotResources(config wire.Config, labels map[string]string) erro return nil } -// Archives exclude the image. The restored root must resolve from this node's +// Checkpoints exclude the image. The restored root must resolve from this node's // installed image by the captured digest; a missing image record fails before // Restore. This metadata check cannot verify the SDK's physical image cache. func qualifyRuntimeImage(ctx context.Context, config wire.Config, captured string) error { diff --git a/services/core/tools/microsandbox-provider/deployment_test.go b/services/core/tools/microsandbox-provider/deployment_test.go index 3ab23c595..441aa7edc 100644 --- a/services/core/tools/microsandbox-provider/deployment_test.go +++ b/services/core/tools/microsandbox-provider/deployment_test.go @@ -103,7 +103,7 @@ func TestSnapshotProofBindsResourcesWithoutChangingCleanupOwnership(t *testing.T } } -// Archives exclude the image, so Restore must find the pinned image locally. +// Checkpoints exclude the image, so Restore must find the pinned image locally. // The real SDK's empty local registry proves absence fails typed, without a pull. func TestRestoreRequiresPinnedLocalImage(t *testing.T) { if !isolatedSDKTest(t) { diff --git a/services/core/tools/microsandbox-provider/snapshot.go b/services/core/tools/microsandbox-provider/snapshot.go index ce29aa2b9..11f138ac6 100644 --- a/services/core/tools/microsandbox-provider/snapshot.go +++ b/services/core/tools/microsandbox-provider/snapshot.go @@ -10,6 +10,7 @@ import ( "os" "path/filepath" "strconv" + "strings" "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox" wire "github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/microsandbox" @@ -25,15 +26,44 @@ func (b backend) snapshotLabels(operation string, source wire.Compute) map[strin return labels } -// openSnapshot opens only an allocation/operation-derived selector and returns +// snapshotPath finds the one native snapshot the SDK published in the +// operation's group directory. +func (b backend) snapshotPath(operation string) (string, error) { + dir := b.checkpointDirectory(operation) + entries, e := os.ReadDir(dir) + if errors.Is(e, os.ErrNotExist) { + return "", &sdk.Error{Kind: sdk.ErrSnapshotNotFound, Message: dir} + } else if e != nil { + return "", e + } + var found string + for _, entry := range entries { + if !strings.HasPrefix(entry.Name(), "snap_") { + continue + } + if found != "" || !entry.IsDir() { + return "", sandbox.ErrOwnership + } + found = entry.Name() + } + if found == "" { + return "", &sdk.Error{Kind: sdk.ErrSnapshotNotFound, Message: dir} + } + return filepath.Join(dir, found), nil +} + +// openSnapshot opens only an allocation/operation-derived directory and returns // its ownership identity without the checkpoint root, which needs Verify. func (b backend) openSnapshot(ctx context.Context, operation string, source wire.Compute) (*sdk.SnapshotArtifact, wire.SnapshotIdentity, error) { - selector := wire.SnapshotReference(b.q.Config, b.q.Reference, operation) - artifact, e := sdk.Snapshot.Open(ctx, selector) + path, e := b.snapshotPath(operation) if e != nil { return nil, wire.SnapshotIdentity{}, e } - if artifact.Scope() != sdk.SnapshotScopeFull || artifact.SourceSandbox() == nil || *artifact.SourceSandbox() != source.Name { + artifact, e := sdk.Snapshot.Open(ctx, path) + if e != nil { + return nil, wire.SnapshotIdentity{}, e + } + if artifact.ID() != filepath.Base(path) || artifact.Scope() != sdk.SnapshotScopeFull || artifact.SourceSandbox() == nil || *artifact.SourceSandbox() != source.Name { return nil, wire.SnapshotIdentity{}, sandbox.ErrOwnership } for key, value := range b.snapshotLabels(operation, source) { @@ -50,7 +80,7 @@ func (b backend) openSnapshot(ctx context.Context, operation string, source wire return nil, wire.SnapshotIdentity{}, sandbox.ErrOwnership } compatibility := sandbox.CheckpointCompatibility{ArtifactDomain: artifact.Labels()["io.oac.artifact_domain"], ExecutionClass: artifact.Labels()["io.oac.execution_class"]} - return artifact, wire.SnapshotIdentity{Compatibility: compatibility, Reference: selector, ID: artifact.ID(), Digest: artifact.Digest(), CheckpointID: state.Checkpoint.CheckpointID, OperationID: operation, SourceName: source.Name, SourceID: source.ID, SourceGeneration: source.Generation}, nil + return artifact, wire.SnapshotIdentity{Compatibility: compatibility, Reference: wire.SnapshotReference(b.q.Config, b.q.Reference, operation), ID: artifact.ID(), Digest: artifact.Digest(), CheckpointID: state.Checkpoint.CheckpointID, OperationID: operation, SourceName: source.Name, SourceID: source.ID, SourceGeneration: source.Generation}, nil } // inspectSnapshot verifies a fresh capture before its identity is published. @@ -85,23 +115,6 @@ func (b backend) verifiedSnapshot(ctx context.Context, want wire.SnapshotIdentit return a, nil } -// ownedSnapshot matches a published identity for cleanup, where payload -// integrity is irrelevant. The SDK descriptor digest covers the checkpoint -// root, so an equal digest binds the published root without re-hashing. -func (b backend) ownedSnapshot(ctx context.Context, want wire.SnapshotIdentity) (*sdk.SnapshotArtifact, error) { - a, got, e := b.openSnapshot(ctx, want.OperationID, wire.Compute{Name: want.SourceName, ID: want.SourceID, Generation: want.SourceGeneration}) - if e != nil { - return nil, e - } - if got.Digest != want.Digest { - return nil, sandbox.ErrOwnership - } - got.CheckpointRoot = want.CheckpointRoot - if got != want || wire.ValidateSnapshot(b.q.Config, b.q.Reference, got) != nil { - return nil, sandbox.ErrOwnership - } - return a, nil -} func (b backend) suspend(ctx context.Context, q wire.SuspendRequest) (wire.State, error) { receipt, receiptErr := b.receipt(q.OperationID) if receiptErr == nil { @@ -114,7 +127,7 @@ func (b backend) suspend(ctx context.Context, q wire.SuspendRequest) (wire.State } // The host allocation flock spans pause, capture, verification and source kill. - // A surviving completed artifact is observed, never overwritten or recaptured. + // A surviving completed snapshot is observed, never overwritten or recaptured. artifact, snap, e := b.inspectSnapshot(ctx, q.OperationID, q.Source) if sdk.IsKind(e, sdk.ErrSnapshotNotFound) { if q.ObserveOnly { @@ -160,7 +173,7 @@ func (b backend) suspend(ctx context.Context, q wire.SuspendRequest) (wire.State labels[key] = captured.Labels[key] } _, err = sdk.Snapshot.Create(ctx, sdk.SnapshotCreateOptions{ - Name: "s-" + q.OperationID, Group: wire.Name(b.q.Config, b.q.Reference, 0), FromSandbox: q.Source.Name, + Name: "s-" + q.OperationID, Group: "s-" + q.OperationID, DestDir: b.storeDirectory(), FromSandbox: q.Source.Name, Full: true, RecordIntegrity: true, GuestFlush: sdk.GuestFlushSkip, Labels: labels}) if err != nil { return wire.State{}, err @@ -176,20 +189,22 @@ func (b backend) suspend(ctx context.Context, q wire.SuspendRequest) (wire.State if e = qualifySnapshotResources(b.q.Config, artifact.Labels()); e != nil { return wire.State{}, e } - receipt, hashed, e := b.publishArchive(ctx, artifact, snap) - if e != nil { + // The SDK has published the snapshot durably and inspectSnapshot has just + // verified it, so the receipt only records ownership. + receipt = checkpointReceipt{Snapshot: snap} + if e = durableJSON(filepath.Join(b.checkpointDirectory(q.OperationID), "receipt.json"), receipt); e != nil { return wire.State{}, e } - return b.settleSource(ctx, q, receipt, hashed) + return b.settleSource(ctx, q, receipt, true) } -// hashed means this process has just written and hashed the archive under the -// allocation lock; an existing receipt is always checked against its bytes. -func (b backend) settleSource(ctx context.Context, q wire.SuspendRequest, r archiveReceipt, hashed bool) (wire.State, error) { - if !hashed { - if e := b.verifyArchive(r); e != nil { +// verified means this process has just verified the snapshot under the +// allocation lock; an existing receipt is always checked against its snapshot. +func (b backend) settleSource(ctx context.Context, q wire.SuspendRequest, r checkpointReceipt, verified bool) (wire.State, error) { + if !verified { + if _, e := b.verifiedSnapshot(ctx, r.Snapshot); e != nil { if q.ObserveOnly { - // Ownership comes from the durable receipt, independently of archive + // Ownership comes from the durable receipt, independently of snapshot // usability. Expose cleanup identity without authorizing source transfer, // claiming its current state, or weakening the restore integrity check. return wire.State{Compute: q.Source, Status: "unknown", Snapshot: &r.Snapshot}, nil @@ -223,7 +238,7 @@ func (b backend) settleSource(ctx context.Context, q wire.SuspendRequest, r arch } if !r.SourceTerminated { r.SourceTerminated = true - if e = durableJSON(filepath.Join(b.archiveDirectory(q.OperationID), "receipt.json"), r); e != nil { + if e = durableJSON(filepath.Join(b.checkpointDirectory(q.OperationID), "receipt.json"), r); e != nil { return wire.State{}, e } } @@ -235,11 +250,8 @@ func (b backend) settleSource(ctx context.Context, q wire.SuspendRequest, r arch if _, e = sdk.GetSandbox(ctx, q.Source.Name); !sdk.IsKind(e, sdk.ErrSandboxNotFound) { return wire.State{}, wire.ErrUnconfirmed } - if e = sdk.Snapshot.Remove(ctx, r.Snapshot.Reference, false); e != nil && !sdk.IsKind(e, sdk.ErrSnapshotNotFound) { - return wire.State{}, e - } r.SourceStopped = true - if e = durableJSON(filepath.Join(b.archiveDirectory(q.OperationID), "receipt.json"), r); e != nil { + if e = durableJSON(filepath.Join(b.checkpointDirectory(q.OperationID), "receipt.json"), r); e != nil { return wire.State{}, e } return wire.State{Compute: q.Source, Status: "suspended", BootstrapComplete: true, Snapshot: &r.Snapshot, SourceStopped: true}, nil @@ -295,7 +307,7 @@ func (b backend) resume(ctx context.Context, q wire.ResumeRequest) (wire.State, } // Admission is durable before any native import/restore operation. A prior // admitted operation with no visible target remains unknown, never replayed. - artifact, e := b.importArchive(ctx, q.Snapshot) + artifact, e := b.restoreSource(ctx, q.Snapshot) if e != nil { return wire.State{}, e } diff --git a/services/core/tools/microsandbox-provider/snapshot_observation_test.go b/services/core/tools/microsandbox-provider/snapshot_observation_test.go index 74810cc67..ee4b6b353 100644 --- a/services/core/tools/microsandbox-provider/snapshot_observation_test.go +++ b/services/core/tools/microsandbox-provider/snapshot_observation_test.go @@ -83,31 +83,6 @@ func TestRestoreJournalRejectsPartialAndSymlinkRecords(t *testing.T) { }) } } -func TestArchiveDigestRejectsGuestReadableAndSymlinkArtifacts(t *testing.T) { - dir := t.TempDir() - path := filepath.Join(dir, "archive") - if err := os.WriteFile(path, []byte("private RAM"), 0600); err != nil { - t.Fatal(err) - } - hash, size, err := archiveDigest(path) - if err != nil || len(hash) != 64 || size != 11 { - t.Fatal(hash, size, err) - } - if err = os.Chmod(path, 0644); err != nil { - t.Fatal(err) - } - if _, _, err = archiveDigest(path); !errors.Is(err, sandbox.ErrOwnership) { - t.Fatal(err) - } - link := filepath.Join(dir, "link") - if err = os.Symlink(path, link); err != nil { - t.Fatal(err) - } - if _, _, err = archiveDigest(link); !errors.Is(err, sandbox.ErrOwnership) { - t.Fatal(err) - } -} - func TestAdmittedRestoreBlocksCleanupUntilExactCompletion(t *testing.T) { request := initialInfoRequest(t) guard, err := allocationLock(request) @@ -184,7 +159,11 @@ func TestTargetCleanupFenceSurvivesReopenBeforeAndAfterAdmission(t *testing.T) { } } -func TestLostCaptureResponseStillExposesOwnedIdentityWhenArchiveIsDamaged(t *testing.T) { +func TestLostCaptureResponseStillExposesOwnedIdentityWhenCheckpointIsDamaged(t *testing.T) { + if !isolatedSDKTest(t) { + return + } + t.Setenv("MSB_HOME", t.TempDir()) for _, settled := range []bool{false, true} { for _, damage := range []string{"corrupt", "missing"} { t.Run(fmt.Sprintf("settled=%t/%s", settled, damage), func(t *testing.T) { @@ -203,29 +182,21 @@ func TestLostCaptureResponseStillExposesOwnedIdentityWhenArchiveIsDamaged(t *tes t.Fatal(err) } snapshot := wire.SnapshotIdentity{Compatibility: compatibility, Reference: wire.SnapshotReference(request.Config, request.Reference, operation), ID: "exact-snapshot", Digest: "digest", CheckpointID: "checkpoint", CheckpointRoot: "root", OperationID: operation, SourceName: source.Name, SourceID: source.ID} - dir := b.archiveDirectory(operation) + dir := b.checkpointDirectory(operation) if err = privateDirectory(dir); err != nil { t.Fatal(err) } - archive := filepath.Join(dir, "archive.msb") - if err = os.WriteFile(archive, []byte("verified full archive"), 0600); err != nil { - t.Fatal(err) - } - hash, size, err := archiveDigest(archive) - if err != nil { - t.Fatal(err) - } - receipt := archiveReceipt{Snapshot: snapshot, SHA256: hash, Size: size, SourceTerminated: settled, SourceStopped: settled} + receipt := checkpointReceipt{Snapshot: snapshot, SourceTerminated: settled, SourceStopped: settled} if err = durableJSON(filepath.Join(dir, "receipt.json"), receipt); err != nil { t.Fatal(err) } if damage == "corrupt" { - err = os.WriteFile(archive, []byte("damaged"), 0600) - } else { - err = os.Remove(archive) - } - if err != nil { - t.Fatal(err) + if err = os.Mkdir(filepath.Join(dir, "snap_damaged"), 0700); err == nil { + err = os.WriteFile(filepath.Join(dir, "snap_damaged", "snapshot.json"), []byte("damaged"), 0600) + } + if err != nil { + t.Fatal(err) + } } // Core lost the capture response and therefore has no Snapshot yet. It // needs this ownership identity to reach ordinary Kill/DeleteSnapshot. @@ -233,11 +204,11 @@ func TestLostCaptureResponseStillExposesOwnedIdentityWhenArchiveIsDamaged(t *tes if err != nil || state.Snapshot == nil || *state.Snapshot != snapshot || state.Compute != source || state.SourceStopped || state.Status != "unknown" || state.BootstrapComplete { t.Fatalf("cleanup identity hidden or execution authorized: %+v %v", state, err) } - if artifact, err := b.importArchive(t.Context(), snapshot); err == nil || artifact != nil { - t.Fatal("damaged archive was usable for restore", err) + if artifact, err := b.restoreSource(t.Context(), snapshot); err == nil || artifact != nil { + t.Fatal("damaged checkpoint was usable for restore", err) } if _, err = b.suspend(t.Context(), wire.SuspendRequest{Reference: request.Reference, OperationID: operation, Source: source}); err == nil { - t.Fatal("non-observation accepted unusable archive") + t.Fatal("non-observation accepted unusable checkpoint") } foreign := source foreign.ID = "local:foreign" @@ -245,15 +216,11 @@ func TestLostCaptureResponseStillExposesOwnedIdentityWhenArchiveIsDamaged(t *tes if !errors.Is(err, sandbox.ErrOwnership) || denied.Snapshot != nil { t.Fatal("foreign request recovered owned identity", denied, err) } - nativeRemoved := false - if err = b.deleteArchiveWithNative(*state.Snapshot, func() error { nativeRemoved = true; return nil }); err != nil { - t.Fatal("owned damaged archive could not be deleted", err) - } - if !nativeRemoved { - t.Fatal("native cleanup was skipped") + if err = b.deleteCheckpoint(t.Context(), *state.Snapshot); err != nil { + t.Fatal("owned damaged checkpoint could not be deleted", err) } - if _, err = os.Lstat(archive); !errors.Is(err, os.ErrNotExist) { - t.Fatal("archive remains", err) + if _, err = os.Stat(filepath.Join(dir, "snap_damaged")); !errors.Is(err, os.ErrNotExist) { + t.Fatal("damaged checkpoint content was kept", err) } deleted, err := b.receipt(operation) if err != nil || !deleted.Deleted || deleted.Snapshot != snapshot { @@ -288,27 +255,19 @@ func TestPermanentRestoreImportFailureKeepsOneAdmission(t *testing.T) { t.Fatal(err) } q.Snapshot = wire.SnapshotIdentity{Compatibility: compatibility, Reference: wire.SnapshotReference(request.Config, request.Reference, q.OperationID), ID: "id", Digest: "digest", CheckpointID: "checkpoint", CheckpointRoot: "root", OperationID: q.OperationID, SourceName: wire.Name(request.Config, request.Reference, 0), SourceID: "local:source"} - dir := b.archiveDirectory(q.OperationID) + dir := b.checkpointDirectory(q.OperationID) if err = privateDirectory(dir); err != nil { t.Fatal(err) } - archive := filepath.Join(dir, "archive.msb") - if err = os.WriteFile(archive, []byte("original"), 0600); err != nil { - t.Fatal(err) - } - hash, size, err := archiveDigest(archive) - if err != nil { - t.Fatal(err) - } - if err = durableJSON(filepath.Join(dir, "receipt.json"), archiveReceipt{Snapshot: q.Snapshot, SHA256: hash, Size: size, SourceStopped: true, SourceTerminated: true}); err != nil { + if err = durableJSON(filepath.Join(dir, "receipt.json"), checkpointReceipt{Snapshot: q.Snapshot, SourceStopped: true, SourceTerminated: true}); err != nil { t.Fatal(err) } - if err = os.WriteFile(archive, []byte("corrupt!"), 0600); err != nil { + if err = os.MkdirAll(filepath.Join(dir, "snap_damaged"), 0700); err != nil { t.Fatal(err) } } if _, err = b.resume(t.Context(), q); err == nil { - t.Fatal("missing archive accepted") + t.Fatal("unusable checkpoint accepted") } path := filepath.Join(b.storeDirectory(), "r-"+q.OperationID+".json") var record restoreAdmission @@ -390,7 +349,7 @@ func isolatedSDKTest(t *testing.T) bool { return false } -func TestAdmittedAbsentObservationSkipsArchiveAndStaysUnknown(t *testing.T) { +func TestAdmittedAbsentObservationSkipsCheckpointAndStaysUnknown(t *testing.T) { if !isolatedSDKTest(t) { return } @@ -406,7 +365,7 @@ func TestAdmittedAbsentObservationSkipsArchiveAndStaysUnknown(t *testing.T) { if _, err = b.admitRestore(q); err != nil { t.Fatal(err) } - // Even a missing archive must not be consulted: its ENOENT would differ from + // Even a missing checkpoint must not be consulted: its ENOENT would differ from // the typed unknown result returned for this existing admitted target. path := filepath.Join(b.storeDirectory(), "r-"+q.OperationID+".json") before, err := os.ReadFile(path)