In this edition, we will look beyond the basic VPA configuration and understand how it works internally.

It covers,

and more..

Vertical Pod Autoscaler is an add-on for Kubernetes that recommends and, depending on its update mode, applies CPU and memory resource values to containers.

In simpler terms, VPA right-sizes your application pods by increasing or decreasing their resource allocations based on actual usage.

This is especially useful for ensuring the pods have sufficient resources and avoiding out-of-memory kills or CPU throttling without over-provisioning them.

Vertical Pod Autoscaler has three key components that work together to enable vertical scaling of pods.

Let’s look at each component in detail.

The Recommender collects CPU and memory usage for containers within Pods. It gets this data from the Metrics Server. It looks at both current and past usage patterns, including Out-of-Memory (OOM) events.

The Recommender uses a decaying histogram-based algorithm to calculate optimal resource values and generate recommendations.

A histogram is like a bar chart that groups data into "buckets" (bins) and counts occurrences in each bucket.

For example, suppose the Recommender collects many CPU usage samples for a container. Instead of storing and evaluating each value separately, it groups the samples into histogram buckets based on their CPU usage as shown in the image below.

The histogram uses a decay function, meaning older samples lose importance over time.

After calculating the best values, it stores these recommendations in the status.recommendation section of the VPA Custom Resource.

The following diagram illustrates how the Recommender works.

Now that the recommended resource usage is available, how does it get applied to the pod?

This is where the Updater comes into the picture.

The Updater's role is to make sure the running pods match the recommendations. It checks whether the targeted deployment pods match the latest recommendations.

If it detects that a running pod's allocated resources differ from the Recommender's recommendations, the Updater sends an eviction request to the API Server via the Eviction API.

The API Server receives the eviction request, checks the PodDisruptionBudget (if available) to ensure application availability, and then terminates the pod.

Once the pod is evicted, the relevant Deployment/StatefulSet controller detects the missing pod and creates a new pod creation request.

When the pod is recreated, it returns with new recommended CPU and memory values.

The following diagram shows how the Updater works.

When new pods are created, how do they receive the latest recommended CPU and memory values?

This is where the Admission Controller comes into the picture.

Upon creation, the VPA Admission Controller registers itself as a Mutating Admission Webhook. This means it can patch/modify a pod's spec (resource requests) during pod creation.

The following diagram shows how the VPA Admission Controller works.

For every pod creation, the API Server sends an HTTPS webhook request to the Admission Controller. The Admission Controller receives the pod creation request and checks if there is a VPA object whose targetRef matches the pod's Deployment.

If it matches, the Admission Controller retrieves the current recommendations from the VPA object and creates a JSON patch with the updated CPU and memory values.

The Admission Controller sends this patch back to the API Server as a response.

Once the API Server receives the patch, it applies it to the pod spec before the pod is created, then persists the modified pod to etcd.

The following workflow shows VPA operating in Recreate mode. Other update modes behave differently.

Here is how it all works together.

One limitation of VPA when it applies a new recommendation is that it evicts the running pod and recreates it. A new pod with the updated resources comes up, the old one gets deleted.

For many workloads, this is acceptable. But for stateful apps, apps like Wordpress, long-running jobs, or anything sensitive to restarts, that eviction may cause issues.

This is where VPA In-Place Pod Resize comes in

In-Place Pod Resize feature allows you to change CPU and memory requests and limits on a running pod without deleting or recreating it.

When a resize is needed, the kubelet updates the container's Linux cgroup directly, without deleting the pod or restarting the container, unless you explicitly request it.

The following image illustrates how the VPA In-Place Pod Resize works behind the scenes.

We have a detailed hands-on blog that explains VPA In-Place Pod Resize using practical examples.

Read it here: In-Place Pod Resize with VPA

In the next edition, we will build an AI agent from scratch to understand how agents work and explore their key building blocks.

The core idea is learning. By building one ourselves, we will understand how to work with AI agents and use them safely for infrastructure and DevOps tasks.