Skip to content

feat(security): transition from static policies.yaml to dynamic relat…#224

Open
aojea wants to merge 20 commits into
google:mainfrom
aojea:policies
Open

feat(security): transition from static policies.yaml to dynamic relat…#224
aojea wants to merge 20 commits into
google:mainfrom
aojea:policies

Conversation

@aojea

@aojea aojea commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

…ional SQL policy API

  • Replaced local file-based policies.yaml with centralized SQL tables (roles and bindings) in Postgres/SQLite on the Control Plane.
  • Exposed a REST API (/policies) to dynamically retrieve and update policies on the Hub.
  • Refactored the policy compiler to build Biscuit Datalog rules from relational DB snapshots locally on the node.
  • Added support for periodic policy sync interval configuration flag (--policy-sync-interval) on the node.
  • Gracefully handled OIDC callback local server shutdown to prevent TCP reset issues during enrollment.
  • Removed obsolete --policy-file flags, volumes, and ConfigMaps from all GKE template and kind development configurations.
  • Integrated automated policy seeding via background port-forward curl commands post-rollout in GitHub actions and local kind runner scripts.
  • Updated policy and control plane website guides to document the new JSON API and database model.
  • Fixed all golangci-lint issues across the codebase.
  • Verified that all unit, integration, and E2E test suites pass successfully.

…ional SQL policy API

- Replaced local file-based policies.yaml with centralized SQL tables (roles and bindings) in Postgres/SQLite on the Control Plane.
- Exposed a REST API (/policies) to dynamically retrieve and update policies on the Hub.
- Refactored the policy compiler to build Biscuit Datalog rules from relational DB snapshots locally on the node.
- Added support for periodic policy sync interval configuration flag (--policy-sync-interval) on the node.
- Gracefully handled OIDC callback local server shutdown to prevent TCP reset issues during enrollment.
- Removed obsolete --policy-file flags, volumes, and ConfigMaps from all GKE template and kind development configurations.
- Integrated automated policy seeding via background port-forward curl commands post-rollout in GitHub actions and local kind runner scripts.
- Updated policy and control plane website guides to document the new JSON API and database model.
- Fixed all golangci-lint issues across the codebase.
- Verified that all unit, integration, and E2E test suites pass successfully.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the mesh policy configuration, moving from a static file-based bootstrap (policies.yaml) to a dynamic REST API and database-backed model where nodes periodically sync policies from the Hub and compile them locally. The review feedback highlights several critical issues: a break statement in VerifyBiscuitAndGetKey that breaks key rotation, multiple logging statements with missing arguments, and several bugs in the node's policy compiler, including unhandled sam:system:authenticated members, missing target_restricted() facts for specific allowed targets, and ignored CustomDatalog configurations.

Comment thread internal/identity/biscuit.go Outdated
Comment on lines +197 to +198
lastErr = err
break // Signature is valid, but authorization failed. No need to try other keys.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-critical critical

This break statement completely breaks key rotation and multi-key verification.

If there are multiple trusted keys (e.g., during key rotation), and the token is signed by a key other than the first one in the list, the verification with the first key will fail (due to an invalid signature). Because of this break, the loop will abort immediately without trying the other trusted keys, causing valid tokens to be incorrectly rejected.

Please remove this break so that the loop continues to try other trusted keys if verification fails.

Suggested change
lastErr = err
break // Signature is valid, but authorization failed. No need to try other keys.
lastErr = err

Comment thread internal/controlplane/server.go Outdated
bBytes, err := identity.MintBootstrapBiscuitToken(privKey, pID, nodeRecord.Role, biscuitExpiry)
if err != nil {
logger.Errorf("Failed to mint refreshed token for node %s: %v", nodeRecord.PeerID, err)
logger.Errorf("Failed to mint refreshed token for node %s: %v")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The log message contains format specifiers %s and %v but the corresponding arguments nodeRecord.PeerID and err were removed. This will result in missing arguments in the log output.

Suggested change
logger.Errorf("Failed to mint refreshed token for node %s: %v")
logger.Errorf("Failed to mint refreshed token for node %s: %v", nodeRecord.PeerID, err)

Comment thread internal/controlplane/server.go Outdated
biscuitBytes, err := identity.MintBootstrapBiscuitToken(privKey, pID, tokenRecord.Role, time.Now().Add(api.BiscuitTokenTTL))
if err != nil {
logger.Errorf("Failed to mint bootstrap biscuit: %v", err)
logger.Errorf("Failed to mint bootstrap biscuit: %v")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The log message contains a %v format specifier but the err argument was removed. This will result in a missing argument in the log output.

Suggested change
logger.Errorf("Failed to mint bootstrap biscuit: %v")
logger.Errorf("Failed to mint bootstrap biscuit: %v", err)

Comment thread internal/controlplane/server.go Outdated
biscuitBytes, err := identity.MintBootstrapBiscuitToken(privKey, pID, tokenRecord.Role, time.Now().Add(api.BiscuitTokenTTL))
if err != nil {
logger.Errorf("Failed to mint bootstrap biscuit: %v", err)
logger.Errorf("Failed to mint bootstrap biscuit: %v")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The log message contains a %v format specifier but the err argument was removed. This will result in a missing argument in the log output.

Suggested change
logger.Errorf("Failed to mint bootstrap biscuit: %v")
logger.Errorf("Failed to mint bootstrap biscuit: %v", err)

Comment thread internal/node/policy.go
Comment on lines +17 to +31
for _, m := range b.Members {
parts := strings.SplitN(m, ":", 2)
if len(parts) == 2 {
memberType := parts[0]
memberVal := parts[1]
rules = append(rules, biscuit.Rule{
Head: biscuit.Predicate{
Name: api.FactRole,
IDs: []biscuit.Term{biscuit.String(b.Role)},
},
Body: []biscuit.Predicate{
{Name: memberType, IDs: []biscuit.Term{biscuit.String(memberVal)}},
},
})
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The policy compiler does not handle the special sam:system:authenticated member (defined as api.SystemAuthenticated). Previously, any authenticated identity was granted the corresponding role. Now, "sam:system:authenticated" is split into a predicate named "sam" with value "system:authenticated". Since there is no "sam" predicate/fact in Biscuit, this binding will never match and is completely broken.

Please add explicit handling for api.SystemAuthenticated to grant the role to any authenticated user or node (for example, by generating rules with a wildcard/variable body for user or node facts).

Comment thread internal/node/policy.go
Comment on lines +99 to +142
for _, t := range role.AllowedTargets {
if t == "*" {
rules = append(rules, biscuit.Rule{
Head: biscuit.Predicate{
Name: api.FactTargetUnrestricted,
IDs: []biscuit.Term{},
},
Body: []biscuit.Predicate{
{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
},
})
continue
}

// Try to parse as fact:value
parts := strings.SplitN(t, ":", 2)
if len(parts) == 2 {
factName := parts[0]
factVal := parts[1]
rules = append(rules, biscuit.Rule{
Head: biscuit.Predicate{
Name: api.FactGrantedTargetExact,
IDs: []biscuit.Term{biscuit.String(factName), biscuit.String(factVal)},
},
Body: []biscuit.Predicate{
{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
},
})
} else {
// Fallback, if it doesn't have a colon, we'll just treat it as an unrestricted node check?
// Wait, earlier tests might use "node:foo", or maybe just a peerID?
// For compatibility, if no colon, assume it's a domain/prefix check?
// Actually, Phase 2 tests mostly use FactGrantedTargetExact.
rules = append(rules, biscuit.Rule{
Head: biscuit.Predicate{
Name: api.FactGrantedTargetExact,
IDs: []biscuit.Term{biscuit.String("node"), biscuit.String(t)},
},
Body: []biscuit.Predicate{
{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
},
})
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

When a role has specific allowed_targets (other than *), the policy compiler should generate a rule concluding api.FactTargetRestricted (i.e., target_restricted()) for that role. Without this, the node's authorizer won't know that targets are restricted for the role, which will cause authorization checks to fail or behave incorrectly.

Comment thread internal/node/policy.go
Comment on lines +35 to +143
for _, role := range roles {
if role == nil {
continue
}
roleName := role.Name

for _, svc := range role.AllowedServices {
svcType, svcName := api.ParseServiceTarget(svc)

if svcType == "*" && svcName == "*" {
rules = append(rules, biscuit.Rule{
Head: biscuit.Predicate{
Name: api.FactGrantedServiceAllTypes,
IDs: []biscuit.Term{},
},
Body: []biscuit.Predicate{
{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
},
})
} else if svcName == "*" {
rules = append(rules, biscuit.Rule{
Head: biscuit.Predicate{
Name: api.FactGrantedServiceAll,
IDs: []biscuit.Term{biscuit.String(svcType)},
},
Body: []biscuit.Predicate{
{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
},
})
} else if strings.HasPrefix(svcName, "*.") {
suffix := svcName[1:]
rules = append(rules, biscuit.Rule{
Head: biscuit.Predicate{
Name: api.FactGrantedServiceSuffix,
IDs: []biscuit.Term{biscuit.String(svcType), biscuit.String(suffix)},
},
Body: []biscuit.Predicate{
{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
},
})
} else if strings.HasSuffix(svcName, ".*") {
prefix := svcName[:len(svcName)-1]
rules = append(rules, biscuit.Rule{
Head: biscuit.Predicate{
Name: api.FactGrantedServicePrefix,
IDs: []biscuit.Term{biscuit.String(svcType), biscuit.String(prefix)},
},
Body: []biscuit.Predicate{
{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
},
})
} else {
rules = append(rules, biscuit.Rule{
Head: biscuit.Predicate{
Name: api.FactGrantedServiceExact,
IDs: []biscuit.Term{biscuit.String(svcType), biscuit.String(svcName)},
},
Body: []biscuit.Predicate{
{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
},
})
}
}

for _, t := range role.AllowedTargets {
if t == "*" {
rules = append(rules, biscuit.Rule{
Head: biscuit.Predicate{
Name: api.FactTargetUnrestricted,
IDs: []biscuit.Term{},
},
Body: []biscuit.Predicate{
{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
},
})
continue
}

// Try to parse as fact:value
parts := strings.SplitN(t, ":", 2)
if len(parts) == 2 {
factName := parts[0]
factVal := parts[1]
rules = append(rules, biscuit.Rule{
Head: biscuit.Predicate{
Name: api.FactGrantedTargetExact,
IDs: []biscuit.Term{biscuit.String(factName), biscuit.String(factVal)},
},
Body: []biscuit.Predicate{
{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
},
})
} else {
// Fallback, if it doesn't have a colon, we'll just treat it as an unrestricted node check?
// Wait, earlier tests might use "node:foo", or maybe just a peerID?
// For compatibility, if no colon, assume it's a domain/prefix check?
// Actually, Phase 2 tests mostly use FactGrantedTargetExact.
rules = append(rules, biscuit.Rule{
Head: biscuit.Predicate{
Name: api.FactGrantedTargetExact,
IDs: []biscuit.Term{biscuit.String("node"), biscuit.String(t)},
},
Body: []biscuit.Predicate{
{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
},
})
}
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The policy compiler completely ignores CustomDatalog (defined in api.PolicyRole). Custom Datalog facts are saved in the database and sent to the node, but they are never compiled into Biscuit rules on the node.

Please add support for parsing and compiling CustomDatalog facts into Biscuit rules/facts in BuildPolicyRules.

aojea added 5 commits July 23, 2026 16:13
…tegration

- Created a customizable Helm chart under charts/sam-mesh to easily deploy Control Plane (Hub), Router, Console, Database, and Dex.
- Automated baseline policy seeding and router bootstrap token generation via a dedicated Kubernetes Job in the chart.
- Cleaned up raw Kubernetes YAML manifests from development/kind/.
- Configured development/kind/run.sh to boot the local cluster using the Helm chart, with dynamic local helm fallback binary check.
- Parameterized node.template.yaml to leverage Helm-based service discovery.
- Confirmed that Helm lints successfully and all E2E / integration test suites pass.
… race condition and fix router local port-forwarding addresses
@aojea

aojea commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request transitions the Sovereign Agent Mesh (SAM) from static file-based policy bootstrapping to a dynamic, database-backed policy API managed via the Control Plane. It introduces structured protobuf messages for roles and bindings, updates the storage layer to use relational tables, adds a Helm chart for deployment, and implements periodic policy synchronization on the nodes. The review feedback highlights several critical issues: the role resolution logic is missing support for email and node prefixes, the policy update endpoint lacks validation, and updates are not immediately broadcasted to the mesh. Additionally, the node's initial policy sync needs a retry loop with exponential backoff, the OIDC server shutdown should use a timeout context, and the Helm chart's database port and SSL mode should be configurable.

Comment thread internal/controlplane/server.go Outdated
Comment on lines +1698 to +1742
func resolveRoles(claims jwt.MapClaims, bindings []*api.PolicyBinding) []string {
oidcRoles := toStringSlice(claims["roles"])
oidcGroups := toStringSlice(claims["groups"])
oidcSub, _ := claims["sub"].(string)
oidcEmail, _ := claims["email"].(string)

resolvedRoles := make(map[string]bool)
for _, b := range bindings {
if b == nil {
continue
}
for _, m := range b.Members {
if m == api.SystemAuthenticated {
resolvedRoles[b.Role] = true
continue
}
parts := strings.SplitN(m, ":", 2)
if len(parts) != 2 {
continue
}
prefix, value := parts[0], parts[1]
switch prefix {
case api.FactGroup:
for _, g := range oidcGroups {
if g == value {
resolvedRoles[b.Role] = true
}
}
case api.FactUser:
if oidcSub == value || oidcEmail == value {
resolvedRoles[b.Role] = true
}
}
}
}
for _, r := range oidcRoles {
resolvedRoles[r] = true
}

var res []string
for r := range resolvedRoles {
res = append(res, r)
}
return res
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

The resolveRoles function is missing support for api.FactEmail ("email") and api.FactNode ("node") prefixes. Additionally, the check for oidcEmail == value is incorrectly nested under api.FactUser, which means any binding defined with "email:xxx" will fail to resolve because its prefix is "email", not "user".

Furthermore, node-specific bindings (e.g., "node:12D3KooW...") cannot be resolved because the "node" prefix is not handled, and the peer ID is not passed to resolveRoles. We should update the signature to accept peerID string and handle both prefixes correctly. Note that the callers of resolveRoles at lines 345 and 562 must also be updated to pass pID.String().

func resolveRoles(claims jwt.MapClaims, bindings []*api.PolicyBinding, peerID string) []string {
	oidcRoles := toStringSlice(claims["roles"])
	oidcGroups := toStringSlice(claims["groups"])
	oidcSub, _ := claims["sub"].(string)
	oidcEmail, _ := claims["email"].(string)

	resolvedRoles := make(map[string]bool)
	for _, b := range bindings {
		if b == nil {
			continue
		}
		for _, m := range b.Members {
			if m == api.SystemAuthenticated {
				resolvedRoles[b.Role] = true
				continue
			}
			parts := strings.SplitN(m, ":", 2)
			if len(parts) != 2 {
				continue
			}
			prefix, value := parts[0], parts[1]
			switch prefix {
			case api.FactGroup:
				for _, g := range oidcGroups {
					if g == value {
						resolvedRoles[b.Role] = true
					}
				}
			case api.FactUser:
				if oidcSub == value {
					resolvedRoles[b.Role] = true
				}
			case api.FactEmail:
				if oidcEmail == value {
					resolvedRoles[b.Role] = true
				}
			case api.FactNode:
				if peerID == value {
					resolvedRoles[b.Role] = true
				}
			}
		}
	}
	for _, r := range oidcRoles {
		resolvedRoles[r] = true
	}

	var res []string
	for r := range resolvedRoles {
		res = append(res, r)
	}
	return res
}

Comment thread internal/controlplane/server.go Outdated
return
}

resolvedRoles := resolveRoles(claims, bindings)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

Update the call to resolveRoles to pass pID.String() to support node-specific bindings and correct prefix resolution.

Suggested change
resolvedRoles := resolveRoles(claims, bindings)
resolvedRoles := resolveRoles(claims, bindings, pID.String())

Comment thread internal/controlplane/server.go Outdated
http.Error(w, "Internal server error", http.StatusInternalServerError)
return
}
resolvedRoles := resolveRoles(claims, bindings)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

Update the call to resolveRoles to pass pID.String() to support node-specific bindings and correct prefix resolution.

Suggested change
resolvedRoles := resolveRoles(claims, bindings)
resolvedRoles := resolveRoles(claims, bindings, pID.String())

Comment thread internal/controlplane/server.go

if err := s.store.SavePolicy(r.Context(), &policy); err != nil {
if err := s.store.SaveMeshPolicy(r.Context(), req.Roles, req.Bindings); err != nil {
logger.Errorf("Failed to save policy: %v", err)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The MeshEvent_POLICY_UPDATE event is defined in the proto and handled by the node to trigger an immediate policy sync. However, when the policy is successfully updated via POST/PUT /policies, the control plane does not broadcast/publish this event to the mesh. This forces nodes to wait for their periodic sync interval (up to 1 hour) to receive policy updates, defeating the dynamic nature of the policy API. Please ensure that a MeshEvent_POLICY_UPDATE event is published/broadcasted to the mesh upon successful policy updates.

Comment thread internal/node/node.go
Comment thread internal/node/oidc.go Outdated
Comment thread charts/sam-mesh/templates/control-plane-deployment.yaml Outdated
aojea added 2 commits July 24, 2026 14:57
…scuit minting

Translates AllowedServices, AllowedTargets, and CustomDatalog from SQL mesh policy roles into authority Datalog facts during Biscuit token minting, resolving Biscuit verification check failures in find_remote_tools.
Adds sam:role:node role with unrestricted services and targets to default-policy.yaml, matching the Helm bootstrap job policy and ensuring Android Flutter mobile E2E test node enrollment receives valid capability facts.
@aojea

aojea commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request transitions the Sovereign Agent Mesh (SAM) policy configuration from a static YAML file to a dynamic, database-backed policy system managed via a REST API, introducing a Helm chart for deployment and enabling periodic policy synchronization on nodes. The review feedback highlights several critical issues: the resolveRoles function silently ignores email and node-based bindings; the policy update endpoint lacks validation for incoming configurations; and session-level PostgreSQL advisory locks are incorrectly executed on a connection pool rather than a dedicated connection. Additionally, the database operations lack robust error handling, such as missing rows.Err() checks, missing nil pointer checks, and potential failures in SQLite cascade deletes if foreign keys are disabled. Other improvements include addressing a race condition in Kubernetes secret recreation and adding a timeout to the server shutdown context.

Comment thread internal/controlplane/server.go Outdated
Comment thread internal/controlplane/server.go Outdated
return
}

resolvedRoles := resolveRoles(claims, bindings)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

Update the call to resolveRoles to pass the peer ID pID.String() to support node-based bindings.

Suggested change
resolvedRoles := resolveRoles(claims, bindings)
resolvedRoles := resolveRoles(claims, bindings, pID.String())

Comment thread internal/controlplane/server.go Outdated
http.Error(w, "Internal server error", http.StatusInternalServerError)
return
}
resolvedRoles := resolveRoles(claims, bindings)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

Update the call to resolveRoles to pass the peer ID pID.String() to support node-based bindings.

Suggested change
resolvedRoles := resolveRoles(claims, bindings)
resolvedRoles := resolveRoles(claims, bindings, pID.String())

Comment on lines 855 to 866
var req api.PolicyConfigUpdateRequest
if err := proto.Unmarshal(body, &req); err != nil {
http.Error(w, "Invalid request format", http.StatusBadRequest)
return
}

var policy api.PolicyConfig
if err := yaml.Unmarshal([]byte(req.YamlContent), &policy); err != nil {
http.Error(w, "Invalid YAML policy format: "+err.Error(), http.StatusBadRequest)
return
}

// Run validation similar to Hub config loading
if err := ValidatePolicyConfig(&policy); err != nil {
http.Error(w, "Invalid policy structure: "+err.Error(), http.StatusBadRequest)
return
if strings.HasPrefix(r.Header.Get("Content-Type"), "application/json") {
if err := protojson.Unmarshal(body, &req); err != nil {
http.Error(w, "Invalid JSON format: "+err.Error(), http.StatusBadRequest)
return
}
} else {
if err := proto.Unmarshal(body, &req); err != nil {
http.Error(w, "Invalid request format", http.StatusBadRequest)
return
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

There is no validation performed on the incoming PolicyConfigUpdateRequest (roles and bindings) before saving it to the database. This allows administrators to accidentally save invalid role names, malformed service/target patterns, or bindings referencing non-existent roles, which can cause runtime errors or security bypasses on the nodes. Consider implementing a validation helper similar to the old ValidatePolicyConfig to validate the structure and formats of the new relational policies.

Comment thread internal/storage/sql_store.go Outdated
Comment on lines +608 to +635
for _, r := range roles {
if _, err := tx.ExecContext(ctx, s.rebind("INSERT INTO roles (name, description, created_at) VALUES (?, '', ?)"), r.Name, time.Now().UnixMilli()); err != nil {
return err
}
for _, target := range r.AllowedTargets {
if _, err := tx.ExecContext(ctx, s.rebind("INSERT INTO role_permissions (role_name, resource_type, resource_value) VALUES (?, 'target', ?)"), r.Name, target); err != nil {
return err
}
}
for _, svc := range r.AllowedServices {
if _, err := tx.ExecContext(ctx, s.rebind("INSERT INTO role_permissions (role_name, resource_type, resource_value) VALUES (?, 'service', ?)"), r.Name, svc); err != nil {
return err
}
}
for _, dl := range r.CustomDatalog {
if _, err := tx.ExecContext(ctx, s.rebind("INSERT INTO role_permissions (role_name, resource_type, resource_value) VALUES (?, 'custom_datalog', ?)"), r.Name, dl); err != nil {
return err
}
}
}

for _, b := range bindings {
for _, member := range b.Members {
if _, err := tx.ExecContext(ctx, s.rebind("INSERT INTO role_bindings (role_name, member) VALUES (?, ?)"), b.Role, member); err != nil {
return err
}
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Enforce defensive programming by adding nil checks for r and b before accessing their fields (e.g., r.Name, b.Role, b.Members). If the input slices contain any nil elements, this will cause a runtime panic.

Comment on lines +641 to 710
func (s *SQLStore) GetMeshPolicy(ctx context.Context) ([]*api.PolicyRole, []*api.PolicyBinding, error) {
rolesRows, err := s.db.QueryContext(ctx, s.rebind("SELECT name FROM roles"))
if err != nil {
return nil, nil, err
}
defer func() { _ = rolesRows.Close() }()

rolesMap := make(map[string]*api.PolicyRole)
var roles []*api.PolicyRole

for rolesRows.Next() {
var name string
if err := rolesRows.Scan(&name); err != nil {
return nil, nil, err
}
role := &api.PolicyRole{Name: name}
rolesMap[name] = role
roles = append(roles, role)
}

permsRows, err := s.db.QueryContext(ctx, s.rebind("SELECT role_name, resource_type, resource_value FROM role_permissions"))
if err != nil {
return nil, err
return nil, nil, err
}
defer func() { _ = permsRows.Close() }()

var policy api.PolicyConfig
if err := yaml.Unmarshal([]byte(data), &policy); err != nil {
return nil, err
for permsRows.Next() {
var roleName, resType, resValue string
if err := permsRows.Scan(&roleName, &resType, &resValue); err != nil {
return nil, nil, err
}
if r, ok := rolesMap[roleName]; ok {
switch resType {
case "target":
r.AllowedTargets = append(r.AllowedTargets, resValue)
case "service":
r.AllowedServices = append(r.AllowedServices, resValue)
case "custom_datalog":
r.CustomDatalog = append(r.CustomDatalog, resValue)
}
}
}
return &policy, nil

bindingsRows, err := s.db.QueryContext(ctx, s.rebind("SELECT role_name, member FROM role_bindings"))
if err != nil {
return nil, nil, err
}
defer func() { _ = bindingsRows.Close() }()

bindingsMap := make(map[string]*api.PolicyBinding)
for bindingsRows.Next() {
var roleName, member string
if err := bindingsRows.Scan(&roleName, &member); err != nil {
return nil, nil, err
}
b, ok := bindingsMap[roleName]
if !ok {
b = &api.PolicyBinding{Role: roleName}
bindingsMap[roleName] = b
}
b.Members = append(b.Members, member)
}

var bindings []*api.PolicyBinding
for _, b := range bindingsMap {
bindings = append(bindings, b)
}

return roles, bindings, nil
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

In GetMeshPolicy, the database rows iteration loops (rolesRows, permsRows, bindingsRows) do not check rows.Err() after completion. It is a standard Go best practice to check rows.Err() to ensure that the loop didn't terminate prematurely due to a database connection or data error.

Comment thread internal/node/policy.go
Comment on lines +180 to +197
for _, dl := range role.CustomDatalog {
trimmed := strings.TrimRight(strings.TrimSpace(dl), ";")
if trimmed == "" {
continue
}
r, err := parser.FromStringRule(trimmed)
if err == nil {
rules = append(rules, r)
} else {
f, err := parser.FromStringFact(trimmed)
if err == nil {
rules = append(rules, biscuit.Rule{
Head: f.Predicate,
Body: []biscuit.Predicate{},
})
}
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If a custom datalog string fails to parse as both a rule and a fact, the error is silently ignored. This can lead to silent security policy failures (fail-open or fail-closed) if there is a typo in the policy. Consider logging a warning when parsing fails.

		for _, dl := range role.CustomDatalog {
			trimmed := strings.TrimRight(strings.TrimSpace(dl), ";")
			if trimmed == "" {
				continue
			}
			r, err := parser.FromStringRule(trimmed)
			if err == nil {
				rules = append(rules, r)
			} else {
				f, err2 := parser.FromStringFact(trimmed)
				if err2 == nil {
					rules = append(rules, biscuit.Rule{
						Head: f.Predicate,
						Body: []biscuit.Predicate{},
					})
				} else {
					logger.Warnf("Failed to parse custom datalog %q as rule (%v) or fact (%v)", trimmed, err, err2)
				}
			}
		}

Comment on lines +118 to +124
# Delete existing secret first (ignore error if doesn't exist)
curl -s --cacert $CACERT -X DELETE \
-H "Authorization: Bearer $K8S_TOKEN" \
"https://kubernetes.default.svc/api/v1/namespaces/${NAMESPACE}/secrets/${SECRET_NAME}" > /dev/null || true

# Create new secret
curl -s --cacert $CACERT -X POST \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Deleting a Kubernetes secret and immediately recreating it via POST is highly prone to race conditions because Kubernetes deletion is asynchronous. The POST request might fail with a 409 Conflict if the deletion is still in progress. A much more robust approach is to use a single PATCH request with Server-Side Apply (application/apply-patch+yaml or application/merge-patch+json) to create-or-update the secret in a single atomic operation.

Comment thread internal/node/oidc.go
aojea added 2 commits July 24, 2026 17:00
…utdown, and k8s secret apply

- Handle email and role binding prefixes in resolveRoles in server.go
- Add 5s timeout context to OIDC server shutdown in oidc.go
- Use atomic Server-Side Apply (PATCH) for Kubernetes secret creation in bootstrap-job.yaml
- Remove obsolete --policy-file flag and volume mount from mobile_e2e.sh
- Add --admin-token to sam-control-plane in mobile_e2e.sh
- Seed initial mesh policies via POST /policies REST API once control plane is ready
@aojea

aojea commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator Author

/gemini review and update the comments that are stale

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request transitions the Sovereign Agent Mesh (SAM) policy management from a static file-based bootstrapping mechanism (policies.yaml) to a dynamic, database-backed model managed via the Control Plane REST API. It introduces structured protobuf messages for roles and bindings, database migrations for relational storage, dynamic policy synchronization on the node side, and a Helm chart for deployment. The review feedback highlights several critical issues, including a lack of policy validation on the REST endpoint, missing rows.Err() checks during database queries, unhandled storage.ErrNotFound errors during node registration/refresh, potential orphaned rows in SQLite due to disabled foreign key cascades, and the use of context.Background() in background sync goroutines which could cause resource leaks on shutdown.

Comment on lines 855 to 871
var req api.PolicyConfigUpdateRequest
if err := proto.Unmarshal(body, &req); err != nil {
http.Error(w, "Invalid request format", http.StatusBadRequest)
return
}

var policy api.PolicyConfig
if err := yaml.Unmarshal([]byte(req.YamlContent), &policy); err != nil {
http.Error(w, "Invalid YAML policy format: "+err.Error(), http.StatusBadRequest)
return
}

// Run validation similar to Hub config loading
if err := ValidatePolicyConfig(&policy); err != nil {
http.Error(w, "Invalid policy structure: "+err.Error(), http.StatusBadRequest)
return
if strings.HasPrefix(r.Header.Get("Content-Type"), "application/json") {
if err := protojson.Unmarshal(body, &req); err != nil {
http.Error(w, "Invalid JSON format: "+err.Error(), http.StatusBadRequest)
return
}
} else {
if err := proto.Unmarshal(body, &req); err != nil {
http.Error(w, "Invalid request format", http.StatusBadRequest)
return
}
}

if err := s.store.SavePolicy(r.Context(), &policy); err != nil {
if err := s.store.SaveMeshPolicy(r.Context(), req.Roles, req.Bindings); err != nil {
logger.Errorf("Failed to save policy: %v", err)
http.Error(w, "Internal server error", http.StatusInternalServerError)
return

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-high high

The previous policy validation logic (ValidatePolicyConfig) has been completely removed in this transition. Currently, the /policies endpoint accepts any payload without verifying if the roles are unique, if the bindings reference valid roles, or if the allowed services/targets conform to expected formats. Saving an invalid policy can lead to silent failures or security bypasses when nodes attempt to parse and apply the rules. Please re-introduce a validation step before calling SaveMeshPolicy.

Comment on lines +641 to 710
func (s *SQLStore) GetMeshPolicy(ctx context.Context) ([]*api.PolicyRole, []*api.PolicyBinding, error) {
rolesRows, err := s.db.QueryContext(ctx, s.rebind("SELECT name FROM roles"))
if err != nil {
return nil, nil, err
}
defer func() { _ = rolesRows.Close() }()

rolesMap := make(map[string]*api.PolicyRole)
var roles []*api.PolicyRole

for rolesRows.Next() {
var name string
if err := rolesRows.Scan(&name); err != nil {
return nil, nil, err
}
role := &api.PolicyRole{Name: name}
rolesMap[name] = role
roles = append(roles, role)
}

permsRows, err := s.db.QueryContext(ctx, s.rebind("SELECT role_name, resource_type, resource_value FROM role_permissions"))
if err != nil {
return nil, err
return nil, nil, err
}
defer func() { _ = permsRows.Close() }()

var policy api.PolicyConfig
if err := yaml.Unmarshal([]byte(data), &policy); err != nil {
return nil, err
for permsRows.Next() {
var roleName, resType, resValue string
if err := permsRows.Scan(&roleName, &resType, &resValue); err != nil {
return nil, nil, err
}
if r, ok := rolesMap[roleName]; ok {
switch resType {
case "target":
r.AllowedTargets = append(r.AllowedTargets, resValue)
case "service":
r.AllowedServices = append(r.AllowedServices, resValue)
case "custom_datalog":
r.CustomDatalog = append(r.CustomDatalog, resValue)
}
}
}
return &policy, nil

bindingsRows, err := s.db.QueryContext(ctx, s.rebind("SELECT role_name, member FROM role_bindings"))
if err != nil {
return nil, nil, err
}
defer func() { _ = bindingsRows.Close() }()

bindingsMap := make(map[string]*api.PolicyBinding)
for bindingsRows.Next() {
var roleName, member string
if err := bindingsRows.Scan(&roleName, &member); err != nil {
return nil, nil, err
}
b, ok := bindingsMap[roleName]
if !ok {
b = &api.PolicyBinding{Role: roleName}
bindingsMap[roleName] = b
}
b.Members = append(b.Members, member)
}

var bindings []*api.PolicyBinding
for _, b := range bindingsMap {
bindings = append(bindings, b)
}

return roles, bindings, nil
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-high high

In Go's database/sql package, it is critical to check rows.Err() after iterating with rows.Next(). If a database error or connection drop occurs mid-stream, the loop will terminate early and silently, returning a partial/incomplete policy. This is a significant security risk as some roles or bindings might be omitted. Please ensure Err() is checked for rolesRows, permsRows, and bindingsRows.

func (s *SQLStore) GetMeshPolicy(ctx context.Context) ([]*api.PolicyRole, []*api.PolicyBinding, error) {
	rolesRows, err := s.db.QueryContext(ctx, s.rebind("SELECT name FROM roles"))
	if err != nil {
		return nil, nil, err
	}
	defer func() { _ = rolesRows.Close() }()

	rolesMap := make(map[string]*api.PolicyRole)
	var roles []*api.PolicyRole

	for rolesRows.Next() {
		var name string
		if err := rolesRows.Scan(&name); err != nil {
			return nil, nil, err
		}
		role := &api.PolicyRole{Name: name}
		rolesMap[name] = role
		roles = append(roles, role)
	}
	if err := rolesRows.Err(); err != nil {
		return nil, nil, err
	}

	permsRows, err := s.db.QueryContext(ctx, s.rebind("SELECT role_name, resource_type, resource_value FROM role_permissions"))
	if err != nil {
		return nil, nil, err
	}
	defer func() { _ = permsRows.Close() }()

	for permsRows.Next() {
		var roleName, resType, resValue string
		if err := permsRows.Scan(&roleName, &resType, &resValue); err != nil {
			return nil, nil, err
		}
		if r, ok := rolesMap[roleName]; ok {
			switch resType {
			case "target":
				r.AllowedTargets = append(r.AllowedTargets, resValue)
			case "service":
				r.AllowedServices = append(r.AllowedServices, resValue)
			case "custom_datalog":
				r.CustomDatalog = append(r.CustomDatalog, resValue)
			}
		}
	}
	if err := permsRows.Err(); err != nil {
		return nil, nil, err
	}

	bindingsRows, err := s.db.QueryContext(ctx, s.rebind("SELECT role_name, member FROM role_bindings"))
	if err != nil {
		return nil, nil, err
	}
	defer func() { _ = bindingsRows.Close() }()

	bindingsMap := make(map[string]*api.PolicyBinding)
	for bindingsRows.Next() {
		var roleName, member string
		if err := bindingsRows.Scan(&roleName, &member); err != nil {
			return nil, nil, err
		}
		b, ok := bindingsMap[roleName]
		if !ok {
			b = &api.PolicyBinding{Role: roleName}
			bindingsMap[roleName] = b
		}
		b.Members = append(b.Members, member)
	}
	if err := bindingsRows.Err(); err != nil {
		return nil, nil, err
	}

	var bindings []*api.PolicyBinding
	for _, b := range bindingsMap {
		bindings = append(bindings, b)
	}

	return roles, bindings, nil
}

Comment thread internal/controlplane/server.go Outdated
Comment on lines +596 to 606
func (s *SQLStore) SaveMeshPolicy(ctx context.Context, roles []*api.PolicyRole, bindings []*api.PolicyBinding) error {
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
return err
}
defer func() { _ = tx.Rollback() }()

var query string
if s.isPostgres() {
query = s.rebind(`
INSERT INTO policies (id, content, updated_at)
VALUES ('mesh-policy', ?, ?)
ON CONFLICT (id)
DO UPDATE SET content = EXCLUDED.content, updated_at = EXCLUDED.updated_at`)
} else {
query = s.rebind(`
INSERT INTO policies (id, content, updated_at)
VALUES ('mesh-policy', ?, ?)
ON CONFLICT (id)
DO UPDATE SET content = excluded.content, updated_at = excluded.updated_at`)
// For simplicity in replacing policy, we clear all and insert new.
if _, err := tx.ExecContext(ctx, "DELETE FROM roles"); err != nil {
return err
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

In SQLite, foreign key constraints are disabled by default unless explicitly enabled on every connection. If they are disabled, DELETE FROM roles will not cascade to role_permissions and role_bindings, leaving orphaned rows in the database. To ensure database-agnostic correctness and prevent orphaned rows, please explicitly delete from all three tables within the transaction.

func (s *SQLStore) SaveMeshPolicy(ctx context.Context, roles []*api.PolicyRole, bindings []*api.PolicyBinding) error {
	tx, err := s.db.BeginTx(ctx, nil)
	if err != nil {
		return err
	}
	defer func() { _ = tx.Rollback() }()

	// Explicitly delete from all tables to prevent orphaned rows, especially in SQLite where foreign keys might be disabled by default.
	if _, err := tx.ExecContext(ctx, "DELETE FROM role_permissions"); err != nil {
		return err
	}
	if _, err := tx.ExecContext(ctx, "DELETE FROM role_bindings"); err != nil {
		return err
	}
	if _, err := tx.ExecContext(ctx, "DELETE FROM roles"); err != nil {
		return err
	}

Comment thread internal/node/node.go
aojea added 3 commits July 24, 2026 22:57
… postgres locks, and event context

- Explicitly delete role_permissions and role_bindings before roles in SaveMeshPolicy
- Use transaction-scoped pg_advisory_xact_lock in initSchema for Postgres
- Pass active node context to syncMeshPolicy in POLICY_UPDATE event handler
- Add validatePolicyConfig in server.go to validate target and service formats in policy update requests
- Check rows.Err() after all SQL query loops in GetMeshPolicy to prevent returning truncated policies on stream errors
- Tolerate storage.ErrNotFound across GetMeshPolicy calls in server.go
…thorizer

- Record errors from addFact for allowed services, allowed targets, and fallback facts
- Parse and record custom Datalog fact errors instead of ignoring them
- Validate authorizer.Authorize error in VerifyAndExtractPeerID
@aojea

aojea commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator Author

/gemini review again and close comments that are no longer relevant

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request transitions the Sovereign Agent Mesh (SAM) policy configuration from a static file-based bootstrapping mechanism to a dynamic, database-backed REST API system, introducing a new Helm chart for deployment and enabling periodic policy synchronization on nodes. Key feedback includes addressing security concerns around unconditionally enabling --insecure-skip-tls-verify in the control plane, restoring missing input validation for custom Datalog rules and binding members, fixing a busybox compatibility issue with nc -z in the init container, ensuring router identity persistence by avoiding emptyDir for router data, and optimizing control plane handlers by removing redundant database queries.

Comment thread charts/sam-mesh/templates/control-plane-deployment.yaml
Comment on lines +1832 to +1837
for _, target := range r.AllowedTargets {
if err := api.ValidateTargetFormat(target); err != nil {
return fmt.Errorf("invalid allowed_target %q in role %s: %w", target, r.Name, err)
}
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-high high

The new validatePolicyConfig function completely omits validation of CustomDatalog rules/facts. This is a regression from the old policy validation logic, allowing administrators to save malformed Datalog rules/facts that will be silently ignored by the nodes. Add validation using the Biscuit parser.

		for _, target := range r.AllowedTargets {
			if err := api.ValidateTargetFormat(target); err != nil {
				return fmt.Errorf("invalid allowed_target %q in role %s: %w", target, r.Name, err)
			}
		}
		for _, dl := range r.CustomDatalog {
			trimmed := strings.TrimRight(strings.TrimSpace(dl), ";")
			if trimmed == "" {
				continue
			}
			if _, err := parser.FromStringRule(trimmed); err != nil {
				if _, err := parser.FromStringFact(trimmed); err != nil {
					return fmt.Errorf("invalid custom datalog %q in role %s: %w", dl, r.Name, err)
				}
			}
		}
	}

Comment on lines +1843 to +1849
if strings.TrimSpace(b.Role) == "" {
return fmt.Errorf("binding role cannot be empty")
}
if !roleNames[b.Role] {
return fmt.Errorf("binding references undefined role: %s", b.Role)
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-high high

The new validatePolicyConfig function completely omits validation of binding members. This is a regression from the old policy validation logic, allowing administrators to save bindings with invalid formats or unsupported identity prefixes. Add validation to ensure members are non-empty and have valid prefixes.

		if strings.TrimSpace(b.Role) == "" {
			return fmt.Errorf("binding role cannot be empty")
		}
		if !roleNames[b.Role] {
			return fmt.Errorf("binding references undefined role: %s", b.Role)
		}
		if len(b.Members) == 0 {
			return fmt.Errorf("binding for role %q must specify at least one member", b.Role)
		}
		for _, member := range b.Members {
			if member == api.SystemAuthenticated {
				continue
			}
			parts := strings.SplitN(member, ":", 2)
			if len(parts) != 2 {
				return fmt.Errorf("member %q in binding for role %q is invalid, must be in format 'type:value' or %q", member, b.Role, api.SystemAuthenticated)
			}
			prefix := parts[0]
			if _, validPrefix := api.ValidMemberPrefixes[prefix]; !validPrefix {
				return fmt.Errorf("member prefix %q in member %q is invalid, expected one of the standard identity facts", prefix, member)
			}
		}
	}

Comment thread charts/sam-mesh/values.yaml
Comment thread charts/sam-mesh/templates/control-plane-deployment.yaml Outdated
Comment on lines +80 to +81
- name: router-data
emptyDir: {}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using emptyDir for router-data means the router's private key (/data/router.key) will be deleted whenever the pod restarts. This causes the router to generate a new identity (PeerID) on every restart, which disrupts peer connections and breaks any persistent routing configurations. Consider using a volumeClaimTemplate to persist the router's identity, or make it configurable.

Comment thread internal/controlplane/server.go Outdated
Comment thread internal/controlplane/server.go Outdated
Comment thread internal/controlplane/server.go Outdated
Comment thread internal/controlplane/server.go Outdated
…ingle DB query, and Helm templates

- Add CustomDatalog rule/fact parsing and member format/prefix validation in validatePolicyConfig
- Support node-specific (node:<peer_id>) bindings in resolveRoles and pass peerID
- Consolidate GetMeshPolicy database queries in HandleRegister and HandleRefresh
- Make --insecure-skip-tls-verify conditional in Helm deployment and remove busybox unsupported -z flag from nc init command
@aojea

aojea commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request transitions the Sovereign Agent Mesh (SAM) from a static, file-based policy configuration to a dynamic, database-backed policy system managed via the Control Plane REST API. It introduces new database schemas, a /policies endpoint, and periodic policy syncing on nodes. Feedback on these changes highlights several critical issues: a bug in BuildPolicyRules where wildcard targets are treated as exact matches, a security vulnerability allowing revoked nodes with unexpired tokens to still fetch policies, a potential thundering herd problem during policy updates, and duplicate custom access roles being minted into Biscuit tokens during registration and refresh.

Comment thread internal/node/policy.go
Comment on lines +52 to +178
for _, svc := range role.AllowedServices {
svcType, svcName := api.ParseServiceTarget(svc)

if svcType == "*" && svcName == "*" {
rules = append(rules, biscuit.Rule{
Head: biscuit.Predicate{
Name: api.FactGrantedServiceAllTypes,
IDs: []biscuit.Term{},
},
Body: []biscuit.Predicate{
{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
},
})
} else if svcName == "*" {
rules = append(rules, biscuit.Rule{
Head: biscuit.Predicate{
Name: api.FactGrantedServiceAll,
IDs: []biscuit.Term{biscuit.String(svcType)},
},
Body: []biscuit.Predicate{
{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
},
})
} else if strings.HasPrefix(svcName, "*.") {
suffix := svcName[1:]
rules = append(rules, biscuit.Rule{
Head: biscuit.Predicate{
Name: api.FactGrantedServiceSuffix,
IDs: []biscuit.Term{biscuit.String(svcType), biscuit.String(suffix)},
},
Body: []biscuit.Predicate{
{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
},
})
} else if strings.HasSuffix(svcName, ".*") {
prefix := svcName[:len(svcName)-1]
rules = append(rules, biscuit.Rule{
Head: biscuit.Predicate{
Name: api.FactGrantedServicePrefix,
IDs: []biscuit.Term{biscuit.String(svcType), biscuit.String(prefix)},
},
Body: []biscuit.Predicate{
{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
},
})
} else {
rules = append(rules, biscuit.Rule{
Head: biscuit.Predicate{
Name: api.FactGrantedServiceExact,
IDs: []biscuit.Term{biscuit.String(svcType), biscuit.String(svcName)},
},
Body: []biscuit.Predicate{
{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
},
})
}
}

hasUnrestricted := false
hasSpecificTargets := false
for _, t := range role.AllowedTargets {
if t == "*" {
hasUnrestricted = true
} else {
hasSpecificTargets = true
}
}

if hasUnrestricted {
rules = append(rules, biscuit.Rule{
Head: biscuit.Predicate{
Name: api.FactTargetUnrestricted,
IDs: []biscuit.Term{},
},
Body: []biscuit.Predicate{
{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
},
})
}

if hasSpecificTargets {
rules = append(rules, biscuit.Rule{
Head: biscuit.Predicate{
Name: api.FactTargetRestricted,
IDs: []biscuit.Term{},
},
Body: []biscuit.Predicate{
{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
},
})
}

for _, t := range role.AllowedTargets {
if t == "*" {
continue
}

// Try to parse as fact:value
parts := strings.SplitN(t, ":", 2)
if len(parts) == 2 {
factName := parts[0]
factVal := parts[1]
rules = append(rules, biscuit.Rule{
Head: biscuit.Predicate{
Name: api.FactGrantedTargetExact,
IDs: []biscuit.Term{biscuit.String(factName), biscuit.String(factVal)},
},
Body: []biscuit.Predicate{
{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
},
})
} else {
// Fallback, if it doesn't have a colon, we'll just treat it as an unrestricted node check?
// Wait, earlier tests might use "node:foo", or maybe just a peerID?
// For compatibility, if no colon, assume it's a domain/prefix check?
// Actually, Phase 2 tests mostly use FactGrantedTargetExact.
rules = append(rules, biscuit.Rule{
Head: biscuit.Predicate{
Name: api.FactGrantedTargetExact,
IDs: []biscuit.Term{biscuit.String("node"), biscuit.String(t)},
},
Body: []biscuit.Predicate{
{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
},
})
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The BuildPolicyRules function contains a critical bug where wildcard targets (e.g., group:backend.*) are always treated as exact matches (api.FactGrantedTargetExact) instead of using the appropriate wildcard predicates. Additionally, there is a significant amount of duplicate parsing logic for services and targets that can be simplified by leveraging the existing helper functions api.BuildServiceDatalogFact and api.BuildTargetDatalogFact from api/datalog.go.

		for _, svc := range role.AllowedServices {
			fact := api.BuildServiceDatalogFact(svc)
			rules = append(rules, biscuit.Rule{
				Head: fact.Predicate,
				Body: []biscuit.Predicate{
					{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
				},
			})
		}

		hasUnrestricted := false
		hasSpecificTargets := false
		for _, t := range role.AllowedTargets {
			if t == "*" {
				hasUnrestricted = true
			} else {
				hasSpecificTargets = true
			}
		}

		if hasUnrestricted {
			rules = append(rules, biscuit.Rule{
				Head: biscuit.Predicate{
					Name: api.FactTargetUnrestricted,
					IDs:  []biscuit.Term{},
				},
				Body: []biscuit.Predicate{
					{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
				},
			})
		}

		if hasSpecificTargets {
			rules = append(rules, biscuit.Rule{
				Head: biscuit.Predicate{
					Name: api.FactTargetRestricted,
					IDs:  []biscuit.Term{},
				},
				Body: []biscuit.Predicate{
					{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
				},
			})
		}

		for _, t := range role.AllowedTargets {
			if t == "*" {
				continue
			}
			targetStr := t
			if !strings.Contains(t, ":") {
				targetStr = "node:" + t
			}
			fact := api.BuildTargetDatalogFact(targetStr)
			rules = append(rules, biscuit.Rule{
				Head: fact.Predicate,
				Body: []biscuit.Predicate{
					{Name: api.FactRole, IDs: []biscuit.Term{biscuit.String(roleName)}},
				},
			})
		}

Comment on lines +782 to +802
isNode := false
if !isAdmin {
// Try checking if it's a valid node biscuit
authHeader := r.Header.Get("Authorization")
if strings.HasPrefix(authHeader, "Bearer ") {
biscuitBytes, err := base64.StdEncoding.DecodeString(strings.TrimPrefix(authHeader, "Bearer "))
if err == nil {
validKeys, err := s.store.GetAllValidKeys(r.Context())
if err == nil {
var trustedKeys []ed25519.PublicKey
for _, k := range validKeys {
trustedKeys = append(trustedKeys, k.Public)
}
_, err := identity.VerifyAndExtractPeerID(trustedKeys, biscuitBytes)
if err == nil {
isNode = true
}
}
}
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-high high

In HandlePolicies (GET), the endpoint authenticates nodes solely by verifying the signature and expiration of their Biscuit token via identity.VerifyAndExtractPeerID. However, it does not query the database to check if the node is still active or has been revoked/deleted. A revoked node with an unexpired token would still be able to retrieve the entire mesh policy. Consider verifying the node's active enrollment status in the store before returning the policies.

Comment thread internal/node/node.go
Comment on lines +1133 to 1140
case api.MeshEvent_POLICY_UPDATE:
logger.Infof("[Mesh Event] Received POLICY_UPDATE event from %s, triggering sync", msg.ReceivedFrom)
go func() {
if err := n.syncMeshPolicy(ctx); err != nil {
logger.Warnf("Failed to sync mesh policy after event: %v", err)
}
}()
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

When a POLICY_UPDATE event is received, all nodes concurrently trigger a policy sync by calling n.syncMeshPolicy(ctx) without any delay. In a large mesh, this can cause a thundering herd problem and temporarily overload the control plane database. Introducing a small random jitter/delay (e.g., 0 to 5 seconds) before fetching the policy would distribute the load and protect the control plane.

Comment on lines +346 to +357
resolvedRoles := resolveRoles(pID.String(), claims, bindings)
var hasCapabilityRoles bool
var customAccessRoles []string
resolvedMap := make(map[string]bool)
for _, r := range resolvedRoles {
resolvedMap[r] = true
if strings.HasPrefix(r, "sam:role:") {
hasCapabilityRoles = true
} else {
customAccessRoles = append(customAccessRoles, r)
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

There is a regression/bug where the requested role is duplicated in finalRoles if it is a custom access role (i.e., does not start with "sam:role:"). This happens because the check r != req.RequestedRole was omitted when building customAccessRoles. Adding this check prevents duplicate roles from being processed and minted into the Biscuit token.

Suggested change
resolvedRoles := resolveRoles(pID.String(), claims, bindings)
var hasCapabilityRoles bool
var customAccessRoles []string
resolvedMap := make(map[string]bool)
for _, r := range resolvedRoles {
resolvedMap[r] = true
if strings.HasPrefix(r, "sam:role:") {
hasCapabilityRoles = true
} else {
customAccessRoles = append(customAccessRoles, r)
}
}
resolvedRoles := resolveRoles(pID.String(), claims, bindings)
var hasCapabilityRoles bool
var customAccessRoles []string
resolvedMap := make(map[string]bool)
for _, r := range resolvedRoles {
resolvedMap[r] = true
if strings.HasPrefix(r, "sam:role:") {
hasCapabilityRoles = true
} else if r != req.RequestedRole {
customAccessRoles = append(customAccessRoles, r)
}
}

Comment on lines +562 to +573
resolvedRoles := resolveRoles(pID.String(), claims, bindings)
var hasCapabilityRoles bool
var customAccessRoles []string
resolvedMap := make(map[string]bool)
for _, r := range resolvedRoles {
resolvedMap[r] = true
if strings.HasPrefix(r, "sam:role:") {
hasCapabilityRoles = true
} else {
customAccessRoles = append(customAccessRoles, r)
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Similar to HandleRegister, the requested role is duplicated in finalRoles during token refresh if it is a custom access role, because the check r != nodeRecord.Role was omitted when building customAccessRoles. Adding this check prevents duplicate roles from being processed and minted into the Biscuit token.

Suggested change
resolvedRoles := resolveRoles(pID.String(), claims, bindings)
var hasCapabilityRoles bool
var customAccessRoles []string
resolvedMap := make(map[string]bool)
for _, r := range resolvedRoles {
resolvedMap[r] = true
if strings.HasPrefix(r, "sam:role:") {
hasCapabilityRoles = true
} else {
customAccessRoles = append(customAccessRoles, r)
}
}
resolvedRoles := resolveRoles(pID.String(), claims, bindings)
var hasCapabilityRoles bool
var customAccessRoles []string
resolvedMap := make(map[string]bool)
for _, r := range resolvedRoles {
resolvedMap[r] = true
if strings.HasPrefix(r, "sam:role:") {
hasCapabilityRoles = true
} else if r != nodeRecord.Role {
customAccessRoles = append(customAccessRoles, r)
}
}

aojea added 3 commits July 25, 2026 10:56
- Add port (default 5432) and sslmode (default disable) to values.yaml database.postgres configuration
- Update control-plane-deployment.yaml init container and --db-dsn argument to reference configurable port and sslmode
…nt publishing

- Introduce MeshAdapter interface with NopMeshAdapter and P2PMeshAdapter implementations
- Automatically sign and publish MeshEvent (POLICY_UPDATE, BANNED, KEY_ROTATION) over GossipSub on HTTP API calls
- Add unit tests in internal/controlplane/mesh_test.go
- Add end-to-end integration test in tests/integration/controlplane_pubsub_integration_test.go
@aojea

aojea commented Jul 26, 2026

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request transitions the Sovereign Agent Mesh from a static, file-based policy configuration to a dynamic, API-driven mesh policy system. It replaces the --policy-file flag with a REST API (/policies) for managing roles and bindings, introduces a MeshAdapter interface to broadcast policy updates and bans via libp2p GossipSub, and enables nodes to periodically and dynamically sync these policies. Additionally, it packages the deployment into a Helm chart with an automated bootstrapping job. The review feedback highlights several critical safety improvements, including potential nil pointer dereferences in policy resolution, database storage, and node authentication, a potential race condition in the interactive OIDC login server, and a missing nil check for the storage parameter in the P2P mesh adapter.

Comment on lines +1738 to +1742
func resolveRoles(peerID string, claims jwt.MapClaims, bindings []*api.PolicyBinding) []string {
oidcRoles := toStringSlice(claims["roles"])
oidcGroups := toStringSlice(claims["groups"])
oidcSub, _ := claims["sub"].(string)
oidcEmail, _ := claims["email"].(string)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The resolveRoles function accesses claims["roles"] and other fields without checking if claims is nil. If claims is nil (which can happen if a node is refreshed or registered without OIDC claims), this will cause a nil pointer dereference panic. Please add a nil check for claims at the beginning of the function.

func resolveRoles(peerID string, claims jwt.MapClaims, bindings []*api.PolicyBinding) []string {
	if claims == nil {
		claims = make(jwt.MapClaims)
	}
	oidcRoles := toStringSlice(claims["roles"])
	oidcGroups := toStringSlice(claims["groups"])
	oidcSub, _ := claims["sub"].(string)
	oidcEmail, _ := claims["email"].(string)

Comment on lines +610 to +638
for _, r := range roles {
if _, err := tx.ExecContext(ctx, s.rebind("INSERT INTO roles (name, description, created_at) VALUES (?, '', ?)"), r.Name, time.Now().UnixMilli()); err != nil {
return err
}
for _, target := range r.AllowedTargets {
if _, err := tx.ExecContext(ctx, s.rebind("INSERT INTO role_permissions (role_name, resource_type, resource_value) VALUES (?, 'target', ?)"), r.Name, target); err != nil {
return err
}
}
for _, svc := range r.AllowedServices {
if _, err := tx.ExecContext(ctx, s.rebind("INSERT INTO role_permissions (role_name, resource_type, resource_value) VALUES (?, 'service', ?)"), r.Name, svc); err != nil {
return err
}
}
for _, dl := range r.CustomDatalog {
if _, err := tx.ExecContext(ctx, s.rebind("INSERT INTO role_permissions (role_name, resource_type, resource_value) VALUES (?, 'custom_datalog', ?)"), r.Name, dl); err != nil {
return err
}
}
}

for _, b := range bindings {
for _, member := range b.Members {
if _, err := tx.ExecContext(ctx, s.rebind("INSERT INTO role_bindings (role_name, member) VALUES (?, ?)"), b.Role, member); err != nil {
return err
}
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The SaveMeshPolicy function iterates over roles and bindings and accesses their fields (e.g., r.Name, b.Role) without checking if the elements themselves are nil. Since validatePolicyConfig in server.go simply skips nil elements during validation but does not filter them out of the slices, SaveMeshPolicy can receive slices containing nil elements, leading to a nil pointer dereference panic. Please add nil checks inside the loops.

	for _, r := range roles {
		if r == nil {
			continue
		}
		if _, err := tx.ExecContext(ctx, s.rebind("INSERT INTO roles (name, description, created_at) VALUES (?, '', ?)"), r.Name, time.Now().UnixMilli()); err != nil {
			return err
		}
		for _, target := range r.AllowedTargets {
			if _, err := tx.ExecContext(ctx, s.rebind("INSERT INTO role_permissions (role_name, resource_type, resource_value) VALUES (?, 'target', ?)"), r.Name, target); err != nil {
				return err
			}
		}
		for _, svc := range r.AllowedServices {
			if _, err := tx.ExecContext(ctx, s.rebind("INSERT INTO role_permissions (role_name, resource_type, resource_value) VALUES (?, 'service', ?)"), r.Name, svc); err != nil {
				return err
			}
		}
		for _, dl := range r.CustomDatalog {
			if _, err := tx.ExecContext(ctx, s.rebind("INSERT INTO role_permissions (role_name, resource_type, resource_value) VALUES (?, 'custom_datalog', ?)"), r.Name, dl); err != nil {
				return err
			}
		}
	}

	for _, b := range bindings {
		if b == nil {
			continue
		}
		for _, member := range b.Members {
			if _, err := tx.ExecContext(ctx, s.rebind("INSERT INTO role_bindings (role_name, member) VALUES (?, ?)"), b.Role, member); err != nil {
				return err
			}
		}
	}

Comment thread internal/node/oidc.go
Comment on lines +231 to +237
shutdownSrv := func() {
if srv != nil {
shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
_ = srv.Shutdown(shutdownCtx)
cancel()
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

There is a potential race condition between the background goroutine starting the HTTP server and the main goroutine calling shutdownSrv(). If ctx.Done() or an error is received before the background goroutine assigns srv, srv will be nil and shutdownSrv() will do nothing. Later, the server will start and run indefinitely, leaking the port and resources. Consider initializing srv in the main goroutine before launching the background goroutine.

Comment on lines +809 to +812
nodeRecord, nodeErr := s.store.GetNode(r.Context(), peerID.String())
if nodeErr == nil && !nodeRecord.Banned {
isNode = true
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If nodeErr == nil but nodeRecord is nil (e.g., due to database inconsistency or test mocks), accessing nodeRecord.Banned will cause a panic. It is safer to check nodeRecord != nil before accessing its fields.

Suggested change
nodeRecord, nodeErr := s.store.GetNode(r.Context(), peerID.String())
if nodeErr == nil && !nodeRecord.Banned {
isNode = true
}
nodeRecord, nodeErr := s.store.GetNode(r.Context(), peerID.String())
if nodeErr == nil && nodeRecord != nil && !nodeRecord.Banned {
isNode = true
}

Comment on lines +98 to +101
func NewP2PMeshAdapter(h host.Host, ps *pubsub.PubSub, store storage.Store) (*P2PMeshAdapter, error) {
if h == nil || ps == nil {
return nil, fmt.Errorf("host and pubsub cannot be nil")
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The store parameter is required to retrieve the signing key for event publishing. If store is nil, PublishEvent will silently publish unsigned events, which might be rejected by other nodes or lead to security bypasses. Please add a nil check for store in NewP2PMeshAdapter.

Suggested change
func NewP2PMeshAdapter(h host.Host, ps *pubsub.PubSub, store storage.Store) (*P2PMeshAdapter, error) {
if h == nil || ps == nil {
return nil, fmt.Errorf("host and pubsub cannot be nil")
}
func NewP2PMeshAdapter(h host.Host, ps *pubsub.PubSub, store storage.Store) (*P2PMeshAdapter, error) {
if h == nil || ps == nil || store == nil {
return nil, fmt.Errorf("host, pubsub, and store cannot be nil")
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant