egress: MITM tunneled TLS with per-SNI certificate minting by haiyanmeng · Pull Request #871 · agent-substrate/substrate

haiyanmeng · 2026-08-11T20:45:10Z

New changes:

atenet sdsmint — an SDS server that mints a short-lived TLS leaf per SNI on demand. Runs as a native sidecar beside Envoy, sharing the atenet image but staying a separate container so the MITM signing key never lands on the data plane.
mitm_listener — the CONNECT route now hands the upgraded stream here instead of to the forward proxy. Two chains,
picked by tls_inspector: tls terminates using Envoy 1.37's on_demand_secret selector + sni cert mapper (hence the
1.34 → 1.37 bump), raw_buffer proxies plaintext http:// and enforces the allowlist as an :authority match, since
there's no mint on that path to refuse.
CA plumbing — internal/localca gains GenerateCA(GenerateOptions); kubectl-ate admin make-ca-pool gains --key-type,
--common-name; install-ate.sh creates the egress-mitm-ca-pool secret before the Deployment.
e2e (internal/e2e/suites/sdsmint) covers the tunnelled handshake, the leaf Envoy serves, and actor identity across the MITM, against the new egressprobe fixture. These e2e tests are skipped for now. We will enable them after figuring out how to test them in presubmit github workflows.

Not included:

how to make actors trust egress-mitm-ca-pool

Followups:

move functionalities common to CA pools cmd/atenet/internal/sdsmint/certauth/certauth.go from to internal/localca
figure out how to run sdsmint e2e tests in github workflow
look into whether we need to use go code to generate the egress envoy config

This is to address #823

It's a good idea to open an issue first for discussion.

Tests pass
Appropriate changes to documentation are included in the PR

Bowei Du (bowei)

Remove the intermediate CA code
Some of the options seem like we don't need to expose. If so, then the logic inside should be simplified.
Move the CA generating logic into its own package (suggest sdsmint/certauth) that exports a very narrow interface out to the server. Seem like the only things that need to be exported are a Get() and Forget()
I don't know if we need an optional minter.Forgetter() -- this comment applies to other places in the code that seem to have more options than we need.
Go through the comments and scrub for the agent-based self-conversation. The comments are pretty verbose in some places and look more like agent thinking tokens than code comments.

I mostly looked at the server code. Once you get the simplifications in, I will take a more detailed look at the rest of the change.

haiyanmeng · 2026-08-12T14:51:07Z

Remove the intermediate CA code

Done

Some of the options seem like we don't need to expose. If so, then the logic inside should be simplified.

Done

Move the CA generating logic into its own package (suggest sdsmint/certauth) that exports a very narrow interface out to the server. Seem like the only things that need to be exported are a Get() and Forget()

Done

I don't know if we need an optional minter.Forgetter() -- this comment applies to other places in the code that seem to have more options than we need.

minter.Forgetter() iis called through the optional interface in cmd/atenet/internal/sdsmint/deltastream.go:391:

  if f, ok := d.srv.minter.(Forgetter); ok {
      for _, name := range expired {
          f.Forget(name)
      }
  }

Go through the comments and scrub for the agent-based self-conversation. The comments are pretty verbose in some places and look more like agent thinking tokens than code comments.

Done

Taahir Ahmed (ahmedtd) · 2026-08-12T23:12:32Z

+
+	cmd.Flags().StringVar(&cfg.UDSPath, "uds-path", "", "unix socket to listen on; required, and the only transport offered, because leaf private keys transit this channel")
+	cmd.Flags().StringVar(&cfg.CAPoolPath, "ca-pool-path", "", "path to a localca pool JSON holding the MITM CA, the format substrate mounts its other CAs in")
+	cmd.Flags().StringVar(&cfg.CAID, "ca-id", "", "which CA in the pool to sign with; empty takes the first")


Let's leave the behavior to just take the first --- localca.Pool should be tracking which localca.CA is active for signing. I will send a PR to do this (and add rotation commands for the CA secrets).

Taahir Ahmed (@ahmedtd) , are you suggesting removing this flag and take the first CA from the pool by default?

Does your PR need to block this PR?

Can we default to first for now or it doesn't work without this flag.

It works as is. Here is how the sdsmint container in the egress gateway Pod sets it

args: - "sdsmint" - "--uds-path=/var/run/sdsmint/sdsmint.sock" - "--ca-pool-path=/run/ca-state/mitm-pool.json" - "--ca-id=mitm"

Yeah, we can just default to the first CA for now.

haiyanmeng · 2026-08-14T20:26:27Z

I rebased after #959 is merged. We should expect TestActorEgressHTTPS to fail.

haiyanmeng · 2026-08-14T21:19:38Z

https://github.com/agent-substrate/substrate/actions/runs/31837847684/job/94888039437?pr=871

The test TestActorEgressHTTPS indeed failed with {"error":"request failed: Get \"[https://example.com/\](https://example.com//)": tls: failed to verify certificate: x509: certificate signed by unknown authority"}:

=== RUN   TestActorEgressHTTPS
    networking_test.go:117: creating actor networking-e2e/egress-https-1786739670953880925
    networking_test.go:117: resumed actor networking-e2e/egress-https-1786739670953880925
    networking_test.go:128: Actor HTTPS egress fetch returned HTTP 502, want 200; body: {"error":"request failed: Get \"[https://example.com/\](https://example.com//)": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
--- FAIL: TestActorEgressHTTPS (1.31s)

Bowei Du (@bowei) , let us merge #926 first. Then I will rebase and update this PR.

Lior Lieberman (LiorLieberman)

Keith Mattix II (@keithmattix) has done a nice thing on #715 where he builds the listener in go (see xds.go in his PR) based on whether someone enabled connect (using a flag).

we should do something similar for MITM listener and other config changes.

Taahir Ahmed (ahmedtd) · 2026-08-14T22:47:10Z

We should also be careful about exhausting randomness sources.

Go's random sources cannot be exhausted --- they are a CSPRNG keyed with initial randomness read from /dev/urandom. /dev/urandom itself is a CSPRNG seeded with a few bytes of random derived from startup or the RDRAND instruction.

Linux does a bunch of accounting of entropy sources, and tries to reseed /dev/random and /dev/urandom once many values are read from them. This is widely considered to be superstitious hokum.

Taahir Ahmed (ahmedtd) · 2026-08-14T22:47:51Z

+
+	cmd.Flags().StringVar(&cfg.UDSPath, "uds-path", "", "unix socket to listen on; required, and the only transport offered, because leaf private keys transit this channel")
+	cmd.Flags().StringVar(&cfg.CAPoolPath, "ca-pool-path", "", "path to a localca pool JSON holding the MITM CA, the format substrate mounts its other CAs in")
+	cmd.Flags().StringVar(&cfg.CAID, "ca-id", "", "which CA in the pool to sign with; empty takes the first")


Yeah, we can just default to the first CA for now.

Taahir Ahmed (ahmedtd) · 2026-08-14T23:11:31Z

LGTM with some nits

haiyanmeng · 2026-08-16T18:41:37Z

Keith Mattix II (Keith Mattix II (@keithmattix)) has done a nice thing on #715 where he builds the listener in go (see xds.go in his PR) based on whether someone enabled connect (using a flag).

we should do something similar for MITM listener and other config changes.

Thanks for sharing. Filed an issue #1003.

Max Thompson (thompsonmax) · 2026-08-13T22:33:17Z

+		return nil, errors.New("empty host")
+	}
+
+	ca := s.active


Should we defensively validate the CA is not expired, to avoid signing certs with an invalid CA?

Let us do it in a follwoup PR since we plan to move selectCA to internal/localca in a follow up PR (see line 77)

Max Thompson (thompsonmax) · 2026-08-13T22:38:25Z

+	if err != nil {
+		return nil, err
+	}
+	if err := active.Validate(); err != nil {


Should we also validate that the CA is not expired/past NotBefore here (or in ca.Validate())?

Let us do it in a follwoup PR since we plan to move selectCA to internal/localca in a follow up PR (see line 77)

Lior Lieberman (LiorLieberman)

A few more comments:

I think grpc and websocket break with this change. The issue is that we are trying to now parse in the inner connection, and we dont have any support for anything thats not h1 AFAICT. This breaks egress traffic that was not http in many ways
related to 1 - we need to add e2e tests for non http traffic..

Lior Lieberman (LiorLieberman) · 2026-08-17T20:02:20Z

+      # Two chains, selected by what the tunnel actually carries. tls_inspector
+      # tags a ClientHello "tls" and anything else "raw_buffer", so a cleartext
+      # HTTP tunnel gets an HTTP chain instead of being fed to a TLS transport
+      # socket, which would fail the handshake and close the tunnel.


What about non http traffic? Actors sends non http all the time, I think with this config it would not work for them. common example includes git clone ssh:// but any other tcp traffic as well.

I created #1017 to add e2e test coverage for non-HTTP traffic.

To unblock this PR, I will do a manual verification of running git clone git@github.com:agent-substrate/substrate.git from an actor.

haiyanmeng · 2026-08-17T20:47:25Z

I think grpc and websocket break with this change. The issue is that we are trying to now parse in the inner connection, and we dont have any support for anything thats not h1 AFAICT. This breaks egress traffic that was not http in many ways

I filed #1018 to add support for non-HTTP traffic when sdsmint is enabled. We can address it as a followup.

haiyanmeng · 2026-08-17T20:58:29Z

I think grpc and websocket break with this change. The issue is that we are trying to now parse in the inner connection, and we dont have any support for anything thats not h1 AFAICT. This breaks egress traffic that was not http in many ways

What does h1 mean? Lior Lieberman (@LiorLieberman)

Bowei Du (bowei) · 2026-08-17T21:06:22Z

h1 is HTTP/1.1. Mostly people call HTTP/2 as h2.

Bowei Du (bowei) · 2026-08-18T01:32:12Z

The experimental flag guards enablement, so we should not see any issues with this change.

Bowei Du (bowei) · 2026-08-18T01:33:08Z

For some reason, the Google CLA is not working
I bumped the PR to see if this clears the issue.

Envoy now terminates the TLS session inside an actor's CONNECT tunnel with a leaf minted on demand for the SNI the client asked for, instead of forwarding the tunneled bytes opaquely. That puts the plaintext request on a filter chain where policy can be applied later. A new atenet sdsmint subcommand serves those leaves over DELTA_GRPC SDS on a unix socket. Opt-in: --experimental-use-sdsmint selects atenet-egress-with-sdsmint.yaml and creates the egress-mitm-ca-pool secret. The default egress manifest and install path are unchanged. Also: localca gains CA.Validate, GenerateCA options and a crypto.Signer SigningKey; kubectl-ate admin make-ca-pool gains --key-type and --common-name; atunnel returns ErrGatewayHandshake and ConnectRejectedError instead of formatted strings. The sdsmint e2e suite is skipped for now.

haiyanmeng · 2026-08-18T02:40:16Z

Bowei Du (@bowei) , I force-pushed the branch. The Google CLA is green now.

haiyanmeng marked this pull request as draft August 11, 2026 20:45

haiyanmeng force-pushed the sds-1 branch from 7cb6eb3 to 8fb773f Compare August 11, 2026 21:01

Eitan Yarmush (EItanya) requested review from Lior Lieberman (LiorLieberman) and Bowei Du (bowei) August 11, 2026 23:04

haiyanmeng force-pushed the sds-1 branch 2 times, most recently from 195d6c4 to 7976e5a Compare August 11, 2026 23:41

haiyanmeng marked this pull request as ready for review August 11, 2026 23:42

haiyanmeng requested review from Eitan Yarmush (EItanya), Taahir Ahmed (ahmedtd) and Max Thompson (thompsonmax) August 11, 2026 23:43

haiyanmeng mentioned this pull request Aug 11, 2026

localca: accept external signers and make CA generation configurable #821

Closed

haiyanmeng commented Aug 11, 2026

View reviewed changes

Comment thread manifests/ate-install/atenet-egress.yaml Outdated

haiyanmeng force-pushed the sds-1 branch from 7976e5a to 6d731dd Compare August 11, 2026 23:54

haiyanmeng added the area/network label Aug 11, 2026

haiyanmeng force-pushed the sds-1 branch 3 times, most recently from 6080f7e to 7b74711 Compare August 12, 2026 01:08

Bowei Du (bowei) reviewed Aug 12, 2026

View reviewed changes

Bowei Du (bowei) self-assigned this Aug 12, 2026

Taahir Ahmed (ahmedtd) reviewed Aug 12, 2026

View reviewed changes

haiyanmeng requested review from Taahir Ahmed (ahmedtd) and Bowei Du (bowei) August 12, 2026 15:07

Bowei Du (bowei) reviewed Aug 12, 2026

View reviewed changes

Comment thread cmd/atenet/internal/sdsmint/cmd.go Outdated

Bowei Du (bowei) reviewed Aug 12, 2026

View reviewed changes

Taahir Ahmed (ahmedtd) reviewed Aug 12, 2026

View reviewed changes

Comment thread cmd/atenet/internal/sdsmint/certauth/certauth.go Outdated

Comment thread cmd/atenet/internal/sdsmint/certauth/certauth.go Outdated

Comment thread cmd/atenet/internal/sdsmint/certauth/certauth.go Outdated

Comment thread cmd/atenet/internal/sdsmint/certauth/leafkey.go Outdated

Taahir Ahmed (ahmedtd) reviewed Aug 12, 2026

View reviewed changes

Bowei Du (bowei) reviewed Aug 12, 2026

View reviewed changes

Comment thread cmd/atenet/internal/sdsmint/minter.go Outdated

Taahir Ahmed (ahmedtd) reviewed Aug 12, 2026

View reviewed changes

Comment thread cmd/atenet/internal/sdsmint/cmd.go Outdated

Comment thread cmd/atenet/internal/sdsmint/deltastream.go

Lior Lieberman (LiorLieberman) reviewed Aug 14, 2026

View reviewed changes

Taahir Ahmed (ahmedtd) reviewed Aug 14, 2026

View reviewed changes

haiyanmeng requested review from Lior Lieberman (LiorLieberman) and Taahir Ahmed (ahmedtd) August 16, 2026 19:15

haiyanmeng force-pushed the sds-1 branch 2 times, most recently from e492ea2 to e69d2a7 Compare August 17, 2026 17:12

Max Thompson (thompsonmax) reviewed Aug 17, 2026

View reviewed changes

haiyanmeng force-pushed the sds-1 branch from f0f94d2 to 58fae4f Compare August 17, 2026 18:01

This was referenced Aug 17, 2026

Investigate whether we need to use xDS for egress Envoy config #1003

Open

Investigate how to run sdsmint e2e tests in presubmit github workflow #1004

Open

haiyanmeng requested a review from Max Thompson (thompsonmax) August 17, 2026 19:39

Lior Lieberman (LiorLieberman) reviewed Aug 17, 2026

View reviewed changes

haiyanmeng mentioned this pull request Aug 17, 2026

Support non-HTTP traffic when sdsmint is enabled #1018

Open

haiyanmeng mentioned this pull request Aug 17, 2026

Add e2e test coverage for non-HTTP traffic #1017

Open

Bowei Du (bowei) closed this Aug 18, 2026

Bowei Du (bowei) reopened this Aug 18, 2026

haiyanmeng force-pushed the sds-1 branch from 58fae4f to 530cc75 Compare August 18, 2026 02:28

Bowei Du (bowei) merged commit d4c68d7 into agent-substrate:main Aug 18, 2026
9 checks passed

Uh oh!

Conversation

haiyanmeng commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Bowei Du (bowei) left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

haiyanmeng commented Aug 12, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

haiyanmeng commented Aug 14, 2026

Uh oh!

haiyanmeng commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Lior Lieberman (LiorLieberman) left a comment

Choose a reason for hiding this comment

Uh oh!

Taahir Ahmed (ahmedtd) commented Aug 14, 2026

haiyanmeng commented Aug 11, 2026 •

edited

Loading

Bowei Du (bowei) left a comment •

edited

Loading

haiyanmeng commented Aug 14, 2026 •

edited

Loading

haiyanmeng commented Aug 16, 2026 •

edited

Loading