Real Client IP Behind Istio Ambient Gateway: Secure X-Forwarded-For in Go

Here is a log line from a Go auth service running behind such a gateway on Azure Kubernetes Service(AKS):

{
  "level": "INFO",
  "msg": "http_request",
  "method": "POST",
  "path": "/v1/otp/verify",
  "status": 200,
  "remote_ip": "10.0.16.4",
  ...
}

10.0.16.4 is not a user. Every request from every user logs the same handful of internal pod CIDR addresses. It now breaks the IP-based rules, ie. a per-IP rate limit or an audit trail. For an auth service, it collapses every distinct caller into one bucket.

Why RemoteAddr lies

The request path looks like this:

Client
  -> Cloud L4 load balancer
  -> Istio ingress gateway (Envoy)
  -> ztunnel (L4, mTLS tunnel)
  -> our pod

By the time request reach our Go handler, the TCP peer is the gateway (or the local ztunnel), so RemoteAddr is a cluster IP. The real client address only survives if something carries it forward in a header, and that something is X-Forwarded-For.

The obvious fix, and why it is not enough

Read X-Forwarded-For, sure. But X-Forwarded-For is a spoofable, ordered list: each proxy appends the address it saw, and the original client controls the left-most entries. chi’s middleware.RealIP reads the left-most entry, so a caller can just type a fake one:

X-Forwarded-For: 1.2.3.4, 203.0.113.7
                 ^          ^
attacker typed this         appended by Envoy (the truth)

So the common rule is we trust the right side of the list, the part our own infrastructure appended, not the left. Drop middleware.RealIP, and read from the right.

That is correct, but if we stop there in an ambient mesh, we get bitten by the part nobody writes down.

The intermittent twist

After switching to a right-most parse, the logs got better, but not consistently. Some requests now showed the real client. Others still showed an internal IP. Same endpoint, same client, different result:

{ 
  "msg": "http_request",
  "path": "/v1/login/pin",
  "status": 200,
  "remote_ip": "103.87.212.169",
  "forwarded_for": "103.87.212.169"
}

{
  "msg": "http_request",
  "path": "/v1/login/pin",
  "status": 200,
  "remote_ip": "10.244.4.198",
  "forwarded_for": "10.0.16.7"
}

Two things to notice. First, adding the raw X-Forwarded-For to the log (forwarded_for) is what cracked the case: on the broken request, XFF itself is 10.0.16.7, a single internal address. The real client IP is not later in the header. It is not in the header at all.

Second, 10.0.16.7 is not random. On this cluster it is a node internal IP:

$ kubectl get nodes -o wide 
aks-data-...vmss000000     10.0.16.7
aks-system-...vmss000000   10.0.16.6
aks-user-...vmss00001d     10.0.16.4

The XFF “client” on the broken requests is always one of the node IPs. That is the fingerprint of source NAT.

The real culprit: externalTrafficPolicy Cluster

The gateway is exposed by a Kubernetes Service of type LoadBalancer. Its externalTrafficPolicy decides what happens to the client’s source IP:

• Cluster (the default): the cloud LB may deliver a connection to any node. If that node is not running a gateway pod, kube-proxy forwards the connection to another node that hosts a gateway pod, and SNATs the source address to the receiving node’s IP on the way. Envoy then sees a node IP as its peer and writes that into X-Forwarded-For. The client is gone.
• Local: the LB only sends traffic to nodes that host a gateway pod, and no SNAT happens. The original source survives to Envoy.

That is the whole story. With Clusterthe real IP depends on which node the LB happened to pick for that connection:

• LB lands on the node running the gateway pod, no cross-node hop, no SNAT, Envoy sees the real client. Real IP logged.
• LB lands on any other node, cross-node forward, SNAT to that node’s IP, Envoy sees a node IP. Node IP logged.

Same client, different node, different result. The intermittency was at the infrastructure layer, not the app. We confirmed it directly: the broken requests carried node IPs (10.0.16.6, 10.0.16.7) that belonged to nodes other than the one running the single gateway pod.

The fix is one line, at the right layer

# the gateway's LoadBalancer Service
spec:
  externalTrafficPolicy: Local

If you provision the gateway through the Kubernetes Gateway API (as ambient encourages), you do not edit the generated Service directly, you steer it from the Gateway via an infrastructure ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: gw-options
data:
  service: |
    spec:
      externalTrafficPolicy: Local
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
spec:
  gatewayClassName: istio
  infrastructure:
    parametersRef:
      group: ""
      kind: ConfigMap
      name: gw-options

Local has a bonus on AKS: it also fixes the health probe. Under Cluster, Azure derives the LB probe from the listener’s appProtocol and does an HTTP “GET /” that Istio answers with a 404, so the backend flaps unhealthy unless you force a TCP probe with per-port annotations. Under Local, Azure probes the kube-proxy healthCheckNodePort instead, which is exactly “does this node have a ready gateway pod?”, so the workaround annotations disappear too.

Now, and only now, parse the header, but parse it for ambient

With Local in place the client reaches Envoy and lands in X-Forwarded-For. But there is a second ambient-specific gotcha in how you read it. The usual rule is “count N trusted hops from the right”. That assumes a fixed number of appending proxies. Ambient does not guarantee that: depending on the path, waypoints and gateway hops can append a variable number of internal entries to the right of the real client. A fixed “second from the right” lands on a pod IP again.

The robust rule is to walk from the right and return the first public address, skipping trailing private and mesh IPs:

func clientIP(r *http.Request, trustXFF bool) string {
 if trustXFF {
  if xff := r.Header.Get("X-Forwarded-For"); xff != "" {
   parts := strings.Split(xff, ",")
   for i := len(parts) - 1; i >= 0; i-- {
    addr, err := netip.ParseAddr(strings.TrimSpace(parts[i]))
    if err != nil {
     continue
    }
    if addr.IsPrivate() || addr.IsLoopback() ||
     addr.IsLinkLocalUnicast() || addr.IsUnspecified() {
     continue // private ip, skip
    }
    return addr.String()
   }
  }
 }
 host, _, err := net.SplitHostPort(r.RemoteAddr)
 if err != nil {
  return r.RemoteAddr
 }
 return host
}

Why this is better than a hop count:

• It tolerates a variable number of trailing internal hops, which is exactly the ambient failure mode. A right-most-of-N parse does not.

It stays spoof-safe. A client can inject a public IP, but Envoy appends the real client to the right of it, and every internal hop after that is private and skipped, so we land on the real client. The injected value is always further left and never reached.

This assumes the real clients are public, which is true for an internet-facing mobile API. If a legitimate caller is itself on a private network, it would fall through to the peer address, and decide that as a rule.