👋 Hi! I’m Bibin Wilson. In each edition, I share practical tips, guides, and the latest trends in DevOps and MLOps to make your day-to-day DevOps tasks more efficient. If someone forwarded this email to you, you can subscribe here to never miss out!

Today, we are going deep into NAT in Kubernetes networking.

This is one of those fundamental networking concepts you should understand if you manage production Kubernetes clusters, troubleshoot networking issues, or prepare for technical interviews.

We’ll cover:

By the end, you should have a much clearer picture of what happens to a packet as it moves through a Kubernetes cluster.

In one of the GKE projects I worked on, Pods using VPC secondary IP ranges were unable to connect to services running on our organization's private network.

While troubleshooting the iptables rules, we found that traffic leaving the cluster still had the Pod IP as the source IP. The problem was that the Pod IP range was not routable from the organization's private network, so the return traffic couldn't reach the Pods.

We fixed this by configuring ip-masq-agent to masquerade the traffic using the node IP. Since the node IP is in the subnet's primary range, pod traffic could reach the org's private network.

The fix was a single NAT configuration change. But understanding why it worked meant understanding what Kubernetes was actually doing to the packets behind the scenes.

So, before we get into Kubernetes, let's quickly cover the fundamentals of NAT. Once you understand these concepts, Kubernetes networking becomes much easier to follow.

Let's get started.

When a device on a private network needs to access the internet, NAT translates its private IP address into a public IP address. The following image illustrates it better.

NAT also tracks the connection. When the response returns, NAT uses that information to forward the packet to the correct device.

The external server only sees the translated public IP. But this brings up an important question:

Which IP address does NAT actually change: the source or the destination?

That depends on the type of NAT being used. Let's start with SNAT.

SNAT is the process of rewriting the source IP of an outgoing packet. NAT changes the source IP before forwarding the packet and reverses the translation when the response comes back.

For example, imagine a VM in a private subnet that needs to download packages from the internet. The VM doesn't have a public IP, so a NAT gateway replaces its private source IP with a public IP before sending the packet out.

When the response returns, NAT translates it back and forwards it to the original VM.

Since many hosts can share the same public IP, NAT can also rewrite the source port to keep each connection unique. This is called PAT (Port Address Translation), also known as NAPT.

But there is a limit.

NAT has a finite number of source ports available for translated connections. A large number of connections, especially to the same destination, can eventually cause SNAT port exhaustion.

When that happens, new connections may fail until ports become available again.

So far, we've assumed that NAT knows which source IP to use for the translation.

But what if the public IP address used for the translation changes?

That's where MASQUERADE comes in.

In the SNAT example above, the NAT IP is static, so normal SNAT works well. But what if the IP assigned to the NAT device's outgoing interface changes? That's where IP MASQUERADE comes in.

Instead of specifying the translated IP manually, MASQUERADE automatically uses the IP assigned to the outgoing network interface.

A classic example is a home router. Your ISP may change the public IP assigned to the router. Instead of configuring a fixed SNAT address, MASQUERADE automatically uses the IP currently assigned to the router's outgoing interface.

The following image illustrates it better.

So far, we've looked at changing the source address. Now let's look at what happens when the destination address needs to change.

DNAT (Destination Network Address Translation) is the opposite of SNAT. Instead of changing where a packet came from, it rewrites where it’s going.

If you have published a Docker container port on a typical bridge network, you have seen this same DNAT concept in action.

The following image illustrates DNAT using Docker port forwarding.

When we run a container, it gets a private IP on the container network, something like 172.17.0.2, and the app inside listens on port 80. That address only exists on the VM. Nothing outside can route to it.

So clients connect to the VM instead, on 203.0.113.10:8080.

When that packet arrives, the VM rewrites the destination from 203.0.113.10:8080 to 172.17.0.2:80 and forwards it to the container. Both the address and the port change.

That rewrite is DNAT.

DNAT is commonly used for things such as port forwarding and sending traffic from a public-facing address to servers running inside a private network.

Now that SNAT, MASQUERADE, and DNAT are clear, we can see how these same concepts are used inside Kubernetes.

Now let's connect those NAT fundamentals to Kubernetes.

Kubernetes doesn't introduce a new type of NAT. The same SNAT, MASQUERADE, and DNAT concepts we just covered are used under the hood of Kubernetes networking.

What changes is where and why the translation happens.

For example,

The following parts of the Kubernetes networking stack make this happen.

Now, let's look at the two most common scenarios where we can see NAT in action.

Pods have their own IP addresses, but those addresses may not be routable outside the cluster network.

When a Pod connects to an external service, its source IP can be MASQUERADEd behind the node's IP.

The external service sees the request as coming from 192.168.1.20, not 10.1.1.5 as illustrated in the image below.

The exact behavior depends on the cluster networking configuration. For example, Calico can SNAT Pod traffic to the node IP, while some clusters use ip-masq-agent to control which destination ranges should be masqueraded.

Now let's look at the other direction.

Suppose a NodePort Service exposes an application on 192.168.1.20:30080, and the backend Pod is listening on 10.1.1.5:8080

When a client connects to the NodePort, iptables rewrites the destination to the backend Pod. The packet is then forwarded to the Pod.

The following image illustrates the traffic flow when the externalTrafficPolicy is set to Cluster (default policy).

And what happens when the Pod sends the response?

That's where conntrack comes into the picture. Linux already knows that the original connection was translated from 192.168.1.20:30080 to 10.1.1.5:8080. It uses that connection state to apply the reverse NAT translation to the response.

So, from the client's point of view, it is still communicating with 192.168.1.20:30080

If you work with Kubernetes, these networking fundamentals would really help with your day-to-day work and cluster administration.

They are also common topics in Kubernetes, DevOps, SRE, and platform engineering interviews because they test whether you understand what actually happens to network traffic inside a cluster.

You don't have to be a networking expert. But if you manage Kubernetes in production, these are fundamentals worth knowing.