Log credential verification failure phases

contracts/agents-api/sandbox-deployment.md

@@ -127,6 +127,8 @@ Initial setup requires the selected template to appear in the key's team-owned t
 
 No provider text or credential is returned. The write and its `change` or `replace_credential` audit entry share one transaction. Core derives administrator provenance from the authenticated request, with the optional `X-Core-Console-Actor` header as its actor label. A replacement briefly fences provider calls, waits for helper processes to actually exit even after the caller cancelled, and verifies again before committing; a helper's exit does not prove that a remote Create settled. The fence and reads are bounded, and a failure keeps the old key and lifecycles. After the successful response, all retained-generation management uses the committed key; only then revoke the old key in E2B, never before cleanup. Template and resource changes do not drain lifecycles.
 
+Credential verification failures emit a structured `sandbox credential verification failed` log with fixed `phase` and `error_class` values, numeric `generation`, and `refs_count`. The phases distinguish loading the current selection, checking the committed ownership anchor, checking the candidate key against the current selection, checking the candidate selection, reading retained generations, checking a retained generation, reading allocations, and checking allocation references. Generation is zero when a read has not selected a generation; the candidate selection reports its supplied generation, not a newly committed one. Logs contain no provider error text, configuration, template, endpoint or credential.
+
 ## Generation ownership and rollout
 
 `runtime_deployment` holds the current specification. Superseded rows keep only immutable specification, build and endpoint metadata, never another E2B key. An E2B allocation binds its generation at reservation; a node placement binds when the scheduler reserves compatible capacity, and its allocation copies that generation, even across later updates. Inspection, renewal, commands and cleanup use the allocation's original specification and endpoint with the current key; a missing generation never falls back to the current specification. Released generation identifiers stay reserved.

contracts/agents-api/zh/sandbox-deployment.md

@@ -1,7 +1,7 @@
 ---
 title: "沙箱部署"
 source: contracts/agents-api/sandbox-deployment.md
-source_hash: 39cf0dbef5b3ec0a214625144d70393f134b5612f7664aedcf0353cc0add9bec
+source_hash: 3f24f44e53649f44dbd91addeafdcbcdd3cb3ba82679aacff822ebc757809729
 ---
 
 沙箱部署为 Core 管理的 `openai_hosted` 执行选择沙箱提供商、每个沙箱的资源以及不可变的 Runtime 发行版。PostgreSQL 为每个安装维护一个当前有效选择；Web 和 Core API 写入同一配置。节点文件保存其已安装副本和特定于主机的路径，且不能覆盖其资源或 Runtime。该选择独立于 Harness。部署可以保持未配置状态，没有节点；此时它拒绝托管准入。
@@ -130,6 +130,8 @@ POST 会在持久保存候选配置之前对其进行验证，并且不会创建
 
 不会返回提供商文本或凭据。写入操作及其 `change` 或 `replace_credential` 审计条目共用一个事务。Core 从已认证请求中派生管理员审计来源，并使用可选的 `X-Core-Console-Actor` 请求头作为操作者标签。替换操作会短暂对提供商调用设置栅栏，即使调用方已取消，也会等待辅助进程实际退出，并在提交前再次验证；辅助进程退出并不能证明远程 Create 已结算。栅栏和读取均有时间边界，失败时会保留旧密钥和生命周期。成功响应后，所有保留代次管理都使用已提交的密钥；只有在清理完成后，才能在 E2B 中撤销旧密钥，绝不能提前撤销。模板和资源更改不会排空生命周期。
 
+凭据验证失败会输出结构化 `sandbox credential verification failed` 日志，字段包括固定的 `phase` 与 `error_class`、数字 `generation` 和 `refs_count`。阶段区分读取当前选择、验证已提交的所有权锚点、使用候选密钥检查当前选择、检查候选选择、读取保留代次、检查某个保留代次、读取分配，以及检查分配引用。读取尚未选定代次时 generation 为零；候选选择报告其输入代次，不代表新提交的代次。日志不包含提供商错误文本、配置、模板、端点或凭据。
+
 ## 代次所有权与推出 {#generation-ownership-and-rollout}
 
 `runtime_deployment` 保存当前 specification。被取代的行仅保留不可变的 specification、构建和端点元数据，绝不保留另一个 E2B 密钥。E2B 分配在预留时绑定其代次；节点放置在调度器预留兼容容量时绑定，其分配会复制该代次，即使之后发生更新也是如此。检查、续期、命令和清理使用分配的原始 specification 和端点以及当前密钥；缺失代次绝不会回退到当前 specification。已释放代次的标识仍会保留。

services/core/cmd/server/managed_generations.go

@@ -6,6 +6,7 @@ import (
 	"maps"
 	"time"
 
+	"github.com/MiniMax-AI/OpenAgentCore/internal/obs/log"
 	"github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment"
 	"github.com/MiniMax-AI/OpenAgentCore/services/core/internal/execution"
 	"github.com/MiniMax-AI/OpenAgentCore/services/core/internal/providercontract"
@@ -114,7 +115,13 @@ func (s *managedSetup) routeGenerations(candidate execution.PreparedRuntimeDeplo
 		}
 		return release, nil
 	}
-	candidate.VerifyCredential = func(ctx context.Context) error {
+	candidate.VerifyCredential = func(ctx context.Context) (verificationErr error) {
+		phase, generation, refsCount := "load_current", uint64(0), 0
+		defer func() {
+			if verificationErr != nil {
+				log.Warn(ctx, "sandbox credential verification failed", "phase", phase, "generation", generation, "refs_count", refsCount, "error_class", credentialVerificationErrorClass(verificationErr))
+			}
+		}()
 		ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
 		defer cancel()
 		// Preserve the committed credential until its ownership anchor is verified.
@@ -127,6 +134,7 @@ func (s *managedSetup) routeGenerations(candidate execution.PreparedRuntimeDeplo
 		if err != nil {
 			return err
 		}
+		phase, generation = "committed_anchor", current.Generation
 		if current.Provider != setup.Provider {
 			return &deployment.ResetRequiredError{CurrentProvider: current.Provider, RequestedProvider: setup.Provider}
 		}
@@ -145,20 +153,24 @@ func (s *managedSetup) routeGenerations(candidate execution.PreparedRuntimeDeplo
 			}
 			return verify(value, refs)
 		}
+		phase = "candidate_current"
 		if err := withCandidateKey(current, nil); err != nil {
 			return err
 		}
+		phase, generation = "candidate_selection", setup.Generation
 		if err := verify(setup, nil); err != nil {
 			return err
 		}
 		generations := map[uint64]deployment.Setup{current.Generation: current}
 		for after := int64(-1); ; {
+			phase, generation = "load_generations", 0
 			page, err := s.deployment.GenerationPage(ctx, after)
 			if err != nil {
 				return err
 			}
 			for _, g := range page {
 				generations[g.Generation] = g
+				phase, generation = "retained_generation", g.Generation
 				if err := withCandidateKey(g, nil); err != nil {
 					return err
 				}
@@ -169,6 +181,7 @@ func (s *managedSetup) routeGenerations(candidate execution.PreparedRuntimeDeplo
 			}
 		}
 		for after := ""; ; {
+			phase, generation, refsCount = "load_allocations", 0, 0
 			page, err := s.allocations.CredentialAllocations(ctx, after)
 			if err != nil {
 				return err
@@ -178,8 +191,9 @@ func (s *managedSetup) routeGenerations(candidate execution.PreparedRuntimeDeplo
 				refsByGeneration[a.DeploymentGeneration] = append(refsByGeneration[a.DeploymentGeneration], sandbox.Reference{TenantID: a.TenantID, EnvironmentID: a.EnvironmentID, AllocationID: a.ID})
 				after = a.ID
 			}
-			for generation, refs := range refsByGeneration {
-				owner, ok := generations[generation]
+			for ownerGeneration, refs := range refsByGeneration {
+				phase, generation, refsCount = "allocation_references", ownerGeneration, len(refs)
+				owner, ok := generations[ownerGeneration]
 				if !ok {
 					return sandbox.ErrConfigurationUnconfirmed
 				}
@@ -195,3 +209,24 @@ func (s *managedSetup) routeGenerations(candidate execution.PreparedRuntimeDeplo
 	}
 	return candidate, nil
 }
+
+// Keep diagnostic classes bounded; provider errors may contain private text.
+func credentialVerificationErrorClass(err error) string {
+	var reset *deployment.ResetRequiredError
+	switch {
+	case errors.Is(err, context.Canceled):
+		return "cancelled"
+	case errors.Is(err, context.DeadlineExceeded):
+		return "timeout"
+	case errors.As(err, &reset):
+		return "reset_required"
+	case errors.Is(err, sandbox.ErrCredentialRejected):
+		return "credential_rejected"
+	case errors.Is(err, sandbox.ErrCredentialOwnership):
+		return "credential_ownership"
+	case errors.Is(err, sandbox.ErrConfigurationUnconfirmed):
+		return "verification_unconfirmed"
+	default:
+		return "internal_error"
+	}
+}

services/core/cmd/server/managed_generations_observation_test.go

@@ -0,0 +1,173 @@
+package main
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"log/slog"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"github.com/MiniMax-AI/OpenAgentCore/services/core/internal/deployment"
+	"github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox"
+	"github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/e2b"
+	"github.com/MiniMax-AI/OpenAgentCore/services/core/internal/sandbox/providers"
+	"github.com/google/uuid"
+)
+
+func TestCredentialVerificationFailureObservation(t *testing.T) {
+	for _, tc := range []struct {
+		fail             int
+		phase            string
+		generation, refs int
+	}{
+		{1, "committed_anchor", 3, 0}, {2, "candidate_current", 3, 0},
+		{3, "candidate_selection", 3, 0}, {4, "retained_generation", 1, 0},
+		{5, "allocation_references", 1, 1}, {0, "", 0, 0},
+		{-1, "load_current", 0, 0}, {-2, "load_generations", 0, 0}, {-3, "load_allocations", 0, 0},
+	} {
+		t.Run(fmt.Sprint(tc.fail), func(t *testing.T) {
+			var logs bytes.Buffer
+			previous := slog.Default()
+			slog.SetDefault(slog.New(slog.NewJSONHandler(&logs, nil)))
+			defer slog.SetDefault(previous)
+			state := filepath.Join(t.TempDir(), "e2b")
+			if err := os.MkdirAll(state, 0700); err != nil {
+				t.Fatal(err)
+			}
+			helper := filepath.Join(t.TempDir(), "helper")
+			script := fmt.Sprintf(`#!/usr/bin/env python3
+import json,sys,pathlib
+q=json.load(sys.stdin)
+code=''
+if q['Operation']=='verify_credential':
+ p=pathlib.Path(q['Config']['StateDir'])/'count'
+ count=int(p.read_text())+1 if p.exists() else 1
+ p.write_text(str(count))
+ if count==%d: code='team_mismatch'
+out={'Version':q['Version'],'ErrorCode':code,'DeploymentValid':not code}
+if q['Operation']=='validate_deployment': out['TemplateBuild']={'Status':'ready','CPUs':2,'MemoryMiB':2048}
+print(json.dumps(out))
+`, tc.fail)
+			if err := os.WriteFile(helper, []byte(script), 0700); err != nil {
+				t.Fatal(err)
+			}
+			current := deployment.Setup{InstallationID: uuid.NewString(), Provider: "e2b", Mode: "direct", Generation: 3, UsesCredential: true,
+				Specification: sandbox.DeploymentSpec{Resources: sandbox.Resources{CPUs: 2, MemoryMiB: 2048}},
+				Configuration: &e2b.DeploymentConfiguration{APIKey: "PRIVATE-KEY", Template: "PRIVATE-TEMPLATE:" + uuid.NewString()}}
+			retained := current
+			retained.Generation = 1
+			ref := deployment.Allocation{ID: uuid.NewString(), TenantID: uuid.NewString(), EnvironmentID: uuid.NewString(), DeploymentGeneration: 1}
+			privateErr := errors.New("PRIVATE-ERROR")
+			setups := &fakeDeploymentSetups{t: t,
+				setup: func(context.Context) (deployment.Setup, error) {
+					if tc.fail == -1 {
+						return deployment.Setup{}, privateErr
+					}
+					return current, nil
+				},
+				withCredential: credentialService(t),
+				generationPage: func(context.Context, int64) ([]deployment.Setup, error) {
+					if tc.fail == -2 {
+						return nil, privateErr
+					}
+					return []deployment.Setup{retained}, nil
+				},
+			}
+			allocations := &fakeGenerationAllocations{t: t, credentialAllocations: func(context.Context, string) ([]deployment.Allocation, error) {
+				if tc.fail == -3 {
+					return nil, privateErr
+				}
+				return []deployment.Allocation{ref}, nil
+			}}
+			s := &managedSetup{capacity: testSandboxCapacity(t), processPaths: testProviderPaths(t, helper, state), registry: providers.Builtin(), installationID: current.InstallationID, deployment: setups, allocations: allocations}
+			candidate, err := s.prepare(t.Context(), current)
+			if err != nil {
+				t.Fatal(err)
+			}
+			logs.Reset()
+			err = candidate.VerifyCredential(t.Context())
+			if tc.fail == 0 && err != nil {
+				t.Fatal(err)
+			}
+			if tc.fail < 0 && err != privateErr {
+				t.Fatal("read error identity changed")
+			}
+			if tc.fail == 1 {
+				var reset *deployment.ResetRequiredError
+				if !errors.As(err, &reset) {
+					t.Fatal("anchor mapping changed")
+				}
+			}
+			if tc.fail > 1 && !errors.Is(err, sandbox.ErrCredentialOwnership) {
+				t.Fatal("ownership mapping changed")
+			}
+			observations := 0
+			for _, line := range strings.Split(strings.TrimSpace(logs.String()), "\n") {
+				if line == "" {
+					continue
+				}
+				var row map[string]any
+				if err := json.Unmarshal([]byte(line), &row); err != nil {
+					t.Fatal(err)
+				}
+				if row["msg"] != "sandbox credential verification failed" {
+					continue
+				}
+				observations++
+				if row["phase"] != tc.phase || row["generation"] != float64(tc.generation) || row["refs_count"] != float64(tc.refs) {
+					t.Fatalf("wrong failure location: %v", row)
+				}
+				wantClass := "credential_ownership"
+				if tc.fail == 1 {
+					wantClass = "reset_required"
+				}
+				if tc.fail < 0 {
+					wantClass = "internal_error"
+				}
+				if row["error_class"] != wantClass {
+					t.Fatalf("wrong error class: %v", row)
+				}
+				for key := range row {
+					switch key {
+					case "time", "level", "msg", "phase", "generation", "refs_count", "error_class":
+					default:
+						t.Fatalf("unexpected diagnostic field %q", key)
+					}
+				}
+			}
+			want := 1
+			if tc.fail == 0 {
+				want = 0
+			}
+			if observations != want {
+				t.Fatalf("got %d observations, want %d", observations, want)
+			}
+			for _, private := range []string{"PRIVATE-", current.InstallationID, ref.ID, ref.TenantID, ref.EnvironmentID} {
+				if strings.Contains(logs.String(), private) {
+					t.Fatal("private diagnostic value leaked")
+				}
+			}
+		})
+	}
+}
+
+func TestCredentialVerificationErrorClasses(t *testing.T) {
+	for _, tc := range []struct {
+		err  error
+		want string
+	}{
+		{context.Canceled, "cancelled"}, {context.DeadlineExceeded, "timeout"},
+		{&deployment.ResetRequiredError{CurrentProvider: "PRIVATE", RequestedProvider: "PRIVATE"}, "reset_required"},
+		{sandbox.ErrCredentialRejected, "credential_rejected"}, {sandbox.ErrCredentialOwnership, "credential_ownership"},
+		{sandbox.ErrConfigurationUnconfirmed, "verification_unconfirmed"}, {errors.New("PRIVATE"), "internal_error"},
+	} {
+		if got := credentialVerificationErrorClass(fmt.Errorf("PRIVATE: %w", tc.err)); got != tc.want {
+			t.Fatalf("got %q, want %q", got, tc.want)
+		}
+	}
+}