👋 Hi! I’m Bibin Wilson. In each edition, I share practical tips, guides, and the latest trends in DevOps, MLOps & LLMOps to make your day-to-day tasks and learning more efficient. If someone forwarded this email to you, you can subscribe here to never miss out!

In today's MLOps deep dive, we'll cover the Kubernetes DRA feature every DevOps engineer should know.

It covers,

and more..

Sponsored by Stategraph

Terraform keeps your infrastructure in a state file behind a single lock. As infrastructure grows, Terraform's state locking can slow teams down. Splitting state into smaller pieces adds complexity.

Stategraph takes a different approach. It stores state in a database as a dependency graph and locks only the resources you’re changing. It can manage changes across multiple parts of your infrastructure with a single plan and apply. Your existing Terraform and OpenTofu code stays the same.

Join Stategraph on November 4 for a 30-minute live demo and Q&A.

👉 Save Your Seat Now (Limited slots)

Your AI/ML workload needs an NVIDIA GPU with at least 80 GB of memory.

How do you tell Kubernetes that?

For example, usually when using the Device Plugin, workloads request a GPU using nvidia.com/gpu: 1, which allocates an entire available GPU to the workload, whether the workload is big or small.

Dynamic Resource Allocation (DRA) lets you describe device requirements in a Kubernetes claim. Kubernetes uses that claim to find a suitable device and a node where your Pod can run.

In our previous GPU edition, we explored time-slicing, MPS, and MIG. This time, we will look at how workloads request GPUs and how those GPUs become available inside containers.

When your application needs storage, you create a PersistentVolumeClaim (PVC) with requirements such as capacity and a StorageClass. Kubernetes then binds it to suitable storage, provisioning a new volume when needed and supported.

DRA follows a similar approach, but for hardware devices. Here is how it works.

These are built-in Kubernetes objects. (Covered in detail in the upcoming sections)

The diagram below shows the similarities between dynamic volume provisioning and DRA.

Now let’s understand how Kubernetes knows which GPUs exist in the cluster and what they offer.

DRA Driver is the key component that exposes GPU information to the Kubernetes cluster. It runs as a DaemonSet on nodes that have the hardware devices.

A key thing to understand is that DRA Drivers are vendor-specific. Meaning, for vendors like NVIDIA, AMD, and Intel, there are individual DRA drivers, and they are maintained by the specific vendors.

For NVIDIA GPUs, the NVIDIA DRA driver discovers the devices and publishes their details through ResourceSlice objects.

Think of ResourceSlices as the cluster's device inventory. They describe details such as the device name, model, memory capacity, and the node where the device can be accessed.

The driver maintains this inventory, so you normally do not create ResourceSlices yourself. The scheduler uses it, together with existing allocations, to work out which devices can satisfy a new request.

For our example, a GPU with 40 GB of memory would not match. A GPU with at least 80 GB could match, provided it is available, and the Pod can run on the node that has it.

So far, we have looked at DRA's backend implementation. For pods to make use of the available GPU, pods use custom resources that are part of DRA.

As I described in the introduction, DRA uses the following objects to manage GPUs.

DeviceClass is a cluster-scoped object that acts as a catalog for devices that workloads can request. Think of it like a StorageClass for hardware devices like GPUs.

Just as StorageClass contains the configurations to create available volume types (AWS gp2, gp3, etc.), DeviceClass contains the configurations for available GPUs.

With DeviceClass, developers/platform engineers can choose the required GPU type for the workloads.

The following image shows two DeviceClasses pointing to two different GPU types (standard and MIG-based GPU).

For our training job example, we can start with the NVIDIA GPU DeviceClass. However, we still need to add the 80 GB memory requirement. That belongs in the ResourceClaim.

How can a pod request GPU with specific memory requirements?

This is where the ResourceClaim object comes into the picture. In our example, the ML workload needs,

You can define these workload requirements in a ResourceClaim.

The Pod references the claim, which must be in the same namespace

When Kubernetes schedules the Pod, the scheduler selects a matching GPU and records the allocation in the claim. If no suitable GPU is available, the Pod waits.

Also, multiple pods can use a single ResourceClaim. If the GPU is attached to one node, those Pods must run there. The driver and configuration control how they share it,

As the name suggests, ResourceClaimTemplate is a template for creating ResourceClaims. As we saw earlier, multiple pods can share the same GPU allocation through a ResourceClaim.

But what if each pod needs its own GPU allocation? It is a typical use case in ML training.

For example, in Kubeflow Trainer, we can run multiple training pods, each needing its own GPU to process part of the training workload.

So, instead of manually creating a separate ResourceClaim for every pod, we can use a ResourceClaimTemplate.

The following diagram shows how the ResourceClaim Controller created dedicated ResourceClaims for each training pod.

If a pod using the ResourceClaimTemplate is deleted, Kubernetes also deletes the ResourceClaim.

There is one more important piece of the DRA workflow to understand is the Container Device Interface (CDI).

Just as CNI provides a standard way for networking solutions like Cilium and Calico to configure container networking, CDI provides a standard way for hardware vendors like NVIDIA, AMD, and Intel to describe how their devices should be made available to containers.

The CDI specification contains information like device paths, mount paths, environment variables, and other runtime configuration needed to expose a GPU inside a container.

The following image shows how the DRA driver from NVIDIA exposes an NVIDIA GPU and how k8s components like kubelet and container runtime use the CDI spec to allocate GPUs for pods during scheduling.

Here is how it works.

One thing I like about Kubernetes is how it lets different teams focus on different responsibilities (separation of concerns).

Think about storage. The platform team provides StorageClasses, and application teams use PVCs to request the storage they need. They don’t have to manage the underlying storage system themselves.

DRA brings a similar approach to GPUs.

As shown in the following diagram, different teams have different responsibilities.

In the next edition, we will put this into practice with an end-to-end guide to GPU sharing with DRA. We will look at the driver settings and claims needed to let multiple workloads use the same GPU.

See you in the next edition.

Thanks for reading.