👋 Hi! I’m Bibin Wilson. In each edition, I share practical tips, guides, and the latest trends in DevOps and MLOps to make your day-to-day DevOps tasks more efficient. If someone forwarded this email to you, you can subscribe here to never miss out!

In today's MLOps deep dive, we will look at,

In a previous Kubernetes GPU edition, I shared how Kubernetes schedules GPU workloads.

As more AI workloads move to Kubernetes, managing GPUs efficiently with GPU-sharing strategies is becoming an important infrastructure skill.

In this edition, we will look at the fundamentals of GPU sharing strategies

By default, when you install the Device Plugin on a Kubernetes node, it checks the number of GPUs and reports the details to Kubernetes, as shown in the image below.

For example, if there is one GPU, the Device Plugin advertises it as nvidia.com/gpu: 1

When a workload requests a GPU, Kubernetes allocates an entire GPU by default, even if it uses only a fraction of its capacity.

This means other workloads cannot use the remaining GPU capacity. They may stay in a Pending state until another GPU becomes available. As a result, GPU resources can remain underutilized, increasing infrastructure costs.

This is where GPU sharing comes in.

GPU sharing allows multiple workloads to use the same physical GPU. This helps improve GPU utilization and can reduce infrastructure costs.

However, GPU sharing doesn't always mean physically dividing a GPU into smaller parts. Depending on the strategy, sharing can happen at the software or hardware level.

There are three main GPU sharing strategies:

Let's look at how each strategy works and when to use it.

Time-Slicing is the easiest sharing strategy.

GPU Time-Slicing allows multiple workloads to share the same physical GPU by taking turns using its resources. You can configure this in the Device plugin configuration.

For example, in the NVIDIA Device Plugin, you can configure a single physical GPU to be advertised as four GPU resources using replicas: 4 configuration.

Now, Kubernetes sees this as 4 GPUs, and it can schedule 4 GPU workloads on them. Also, 4 is not the limit. You can increase or decrease the replica count based on your workload requirements and available GPU resources.

The diagram below illustrates how GPU time-slicing allows multiple workloads to share a single physical GPU.

As shown in the image above, the workloads take turns using the GPU.

The GPU's scheduling mechanism runs one workload's GPU operations for a short period, then switches to another workload. During this switch, the GPU saves and restores the execution context needed to resume processing.

The switching happens so rapidly, like 1000 times per second, that the workloads seem to run continuously. However, time-slicing does not divide the GPU into fixed compute or memory partitions.

For example, configuring four replicas does not mean each workload gets 25% of the GPU's compute power or memory.

The actual GPU usage depends on the workloads. If one workload is idle, another can use more of the available GPU compute capacity. When multiple workloads are active, they compete for GPU resources, which can affect performance.

The next strategy is Multi-Process Service (MPS). MPS allows multiple workloads to run on the same physical GPU concurrently, while controlling how much resources each workload can use.

Similar to the time-slicing strategy, the Device Plugin exposes a single GPU as multiple replicas like nvidia.com/gpu: 4.

However, MPS works differently behind the scenes. MPS uses a client-server architecture.

When MPS sharing is enabled through the NVIDIA Device Plugin, an MPS control daemon manages access to the shared GPU.

The control daemon manages an MPS server process, which handles GPU operations submitted by multiple client processes as shown in the image below.

Another important difference is how MPS controls GPU resource usage. With the NVIDIA Device Plugin's MPS sharing configuration, it divides compute and memory limits equally among the configured replicas.

For example, if there are 4 GPU replicas, each gets 25% of the resources. As a result, it does not provide complete fault isolation. This means that if an error in a single workload crashes the GPU, it will affect the other workloads as well.

Unlike Time-Slicing and MPS, MIG slices the GPU at the hardware level. Here, slicing means dividing one physical GPU into multiple smaller MIG instances. Each instance gets its own portion of GPU resources.

Each instance gets dedicated compute and memory resources. This means workloads can run independently without competing for the same GPU resources. This provides hardware-level resource and fault isolation between MIG instances.

In Kubernetes, the MIG Manager Pod applies the GPU partition configuration, and the NVIDIA Device Plugin exposes the resulting MIG instances as schedulable resources. This allows Kubernetes to assign individual MIG instances to different workloads.

In MIG, GPU profiles define how much compute power and memory each MIG instance gets. For example, consider the 1g.10gb profile on an NVIDIA H100 80GB GPU.

An H100 80GB GPU can split up to seven 10 GB GPU slices as shown below.

You may be wondering why there are only 7 slices, not 8.

This is because the H100 has seven available GPU compute slices and eight memory portions. Each 1g.10gb instance requires one compute slice and one memory portion.

After creating seven instances, all available compute slices are allocated. Even though one memory portion remains, no compute slice is available to create an eighth instance.

This is the hardware limit for how many MIG instances an H100 can be split into.

Also, not every MIG profile is supported on every GPU. Refer to NVIDIA's official MIG profile documentation for supported configurations.

In Kubernetes, the NVIDIA Device Plugin supports two MIG strategies: Single and Mixed. These strategies determine how MIG instances are exposed to Kubernetes.

When using MIG Single Strategy, the GPU will be split into equal replicas, and each pod gets the same-sized slice. For example, 1g.10gb.

MIG Single is useful when you need to split the GPU into equal sizes. The image below shows how a GPU is split into equal slices.

The Mixed strategy allows different MIG profiles to coexist on the same GPU.

For example, an H100 GPU can be configured with both 1g.10gb and 3g.40gb instances and shown in the image below.

As shown in the image, the GPU will be split into different sizes. With this split, a workload that needs more GPU can pick the 3g.40gb profile, and smaller workloads can use the 1g.10gb profile.

If you want to try different sharing strategies and have access to a Kubernetes NVIDIA GPU cluster, we have a guide with step-by-step instructions.

Even if you don't have access to a GPU cluster, you can still follow the guide to understand the configuration and workflow.

👉 Read the Detailed Guide Here

In this edition, you have seen how NVIDIA GPUs can be shared using Time-Slicing, MPS, and MIG.

To implement these strategies, you don’t have to configure it at the device plugin level. Kubernetes provides a native resource called DRA (Dynamic Resource Allocation).

DRA became a stable resource in Kubernetes 1.35.

In the next edition, I will cover how DRA works, its main components, and how the DRA driver allocates NVIDIA GPUs.

Thanks for reading.

If you need GPU sharing beyond NVIDIA-specific features like MIG or MPS, HAMi offers a Kubernetes-native approach to sharing heterogeneous accelerators.

HAMi lets multiple workloads share GPU compute and memory while supporting resource isolation and cross-vendor scheduling across NVIDIA, AMD, and other vendors.

It is also a CNCF Incubating project and supports GPU, NPU, MLU, and other device types.

👉 Project: https://project-hami.io/