EVM proxy forwards through a 2-connection pool and exhausts the validator's ephemeral ports

[bdchatham]

The validator-to-validator EVM proxy dials with no HTTP client, so it inherits http.DefaultTransport and its MaxIdleConnsPerHost of 2. Under load each validator opens and closes a socket for nearly every forwarded transaction and runs out of ephemeral ports.

Where

sei-tendermint/internal/p2p/giga_router_common.go:281:

client, err := ethrpc.DialContext(ctx, addr.EVMRPC.String())

No rpc.WithHTTPClient(...), so go-ethereum falls back to client = new(http.Client) (rpc/http.go:153-156). A zero-value http.Client has a nil Transport and therefore uses http.DefaultTransport, where DefaultMaxIdleConnsPerHost = 2 (Go net/http/transport.go:60).

The forward itself is giga/evmonly/rpc/server.go:55, taken whenever EvmProxy(sender) returns a client — that is, whenever the receiving validator does not own the sender.

Measured

4-validator EVM-only Autobahn chain, blockInterval: 400ms, ~15k tx/s offered, clients routing by address modulo so ~75% of transactions were forwarded. Ephemeral range 32768–60999 (28,232 ports), tcp_tw_reuse=2.

/proc/net/sockstat on each validator:

v0: tw 79789    v1: tw 79685    v2: tw 84468    v3: tw 80069

Each validator held ~80,000 sockets in TIME_WAIT against 28,232 ports, and roughly 40% of forwarded sends failed with connect: cannot assign requested address. Those errors surface at the client as a eth_sendRawTransaction failure naming a validator pod FQDN the client never dialed, which makes them easy to misattribute to the load generator.

Removing the proxy traffic — by having the client send to the owning validator instead — dropped TIME_WAIT to 23–156 per validator and raised throughput from 15,246 to 37,929 tx/s while validator CPU halved, from 23.8–27.4 of 31.85 cores to 13.3–14.1. So the forward was costing roughly half of each validator's CPU.

Suggested fix

Pass a tuned client, as runEvmProxy maintains one long-lived client per committee member:

ethrpc.DialOptions(ctx, addr.EVMRPC.String(), ethrpc.WithHTTPClient(&http.Client{
    Transport: &http.Transport{
        MaxIdleConns:        <peers * perHost>,
        MaxIdleConnsPerHost: <perHost>,
        IdleConnTimeout:     90 * time.Second,
    },
}))

Sizing it to the expected concurrent forwards per peer is enough; anything above the default of 2 removes the port churn.

Two related notes:

• The forward is untimed, so there is no metric attributing this latency to the proxy. Adding one would have made the diagnosis immediate rather than by subtraction.
• enable_evm_proxy defaults to true and is not reachable from the SeiNetwork CRD, so a chain cannot opt out without correct client-side shard routing.