How gVisor Enhances K8s Security with Pod Sandboxing

👋 Hi! I’m Bibin Wilson. In each edition, I share practical tips, guides, and the latest trends in DevOps and MLOps to make your day-to-day DevOps tasks more efficient. If someone forwarded this email to you, you can subscribe here to never miss out!

In yesterday's edition, we looked at sandboxed containers.

In this edition, we will look at how to implement it in Kubernetes using the gVisor runtime.

gVisor is used by big organizations to secure their workloads. Towards the end, I have added information on how organizations are using it in production.

What is gVisor?

gVisor is an open-source container sandbox container Runtime technology developed by Google.

It acts as a lightweight, user-space kernel that intercepts and manages system calls from containers, preventing them from directly interacting with the host operating system.

Does this mean gVisor is similar to container runtimes like containerd and CRI-O?

Not exactly. gVisor and containerd serve different purposes, but they work together in a Kubernetes environment.

For example, containerd is the main runtime that Kubernetes interacts with, while gVisor acts as an optional sandboxed runtime.

gVisor Architecture

Here is the high level architecture of gVisor.

gVisor consists of the following key components.

1. Sentry (Core of gVisor)

The Sentry is the main component that runs in user space and implements most Linux system calls.

Instead of directly interacting with the Linux kernel, applications interact with Sentry, which processes system calls safely within user space.

2. Gofer (Filesystem Proxy)

gVisor not have direct file system access; instead, it uses a proxy component (Gofer).

Gofer handles filesystem interactions between the sandboxed application and the host. Meaning, file system operations (like open, create, stat, read, and write) are delegated to Gofer to ensure controlled.

In short, it acts as a broker, checking and passing only safe file operations to the system.

Seccomp further restricts Gofer’s access to only essential file operations to enforce minimal privileges.

gVisor on Kubernetes

If you want to use gVisor with Kubernetes, the nodes must have the gVisor runtime installed, just like you would install containerd or CRI-O.

Once the installation is done, you need to create a RuntimeClass object in Kubernetes. It is a native object in Kubernetes.

It is a cluster-scoped object, so we can use a single RuntimeClass for any namespace object.

For example,

cat <<EOF | kubectl apply -f -
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor 
handler: runsc
EOF

Now to deploy a pod in sandboxed mode using gVisor runtime, all you have to do it add the runtimeClassName: gvisor to the pod spec as shown below.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: test-pod
  name: test-pod
spec:
  runtimeClassName: gvisor
  containers:
  - image: nginx
    name: test-pod
EOF

Once the Pod is deployed, you can check which Container Runtime is used using dmesg as shown below.

Default Runtime vs Sandboxed Runtime

To test the networking access of sandboxed pod vs normal pod, i deployed two differnent pods one iwth gVisor and one with normal runtime.

I tried to list contents of /proc/sys/net/ipv4

Here are the results.

• For gVisor, the available kernel parameters (sysctls) are limited, showing only a subset of networking settings.

• For default runtime, the pod has access to a much larger set of networking configurations, showing that the default runtime allows more control over networking stack configurations.

gVisor Security

gVisor provides better security than standard container runtimes but is not invulnerable.

Wiz research found out a vulnerability in NVIDIA Container Toolkit which in sep 2024.

The vulnerability allows files and directories from the host to be mounted into the sandboxed container, which can lead to a container escape.

Organization's Using gVisor

1. Ant group is using gVisor Sandbox Container Runtime for their payment services to enhance security.

2. Blink is a DevOps platform that uses gVisor for the EKS Pods to run securely.

3. Cloudflare is using gVisor for its container environment to reduce the host attack surface.

4. Google has developed the gVisor runtime and they are so using many sandbox containers on their cloud environment (GKE).

5. OpenAI is using gVisor runtime to run some of its high-risk tasks in its research infrastructure.

Wrapping Up

If you're looking for enhanced security without the overhead of full VM isolation, gVisor is a great option to explore.

I haven’t had a chance to deploy this in any projects yet.

But this is something I will consider for clients who require a sandboxed environment to run arbitrary or untrusted code and applications.