The way to Run Remoted Tenant Kubernetes Clusters on Shared GPU Infrastructure

Working a devoted Kubernetes cluster per crew typically ends in extra isolation than a company requires. Whereas one cluster may be efficiently shared throughout many groups, the coordination prices enhance because the variety of groups grows. Challenges embrace conflicting CRD variations, overlapping RBAC, and no clear method to carve GPU capability into team-level budgets. At a sure scale, groups would possibly begin asking for their very own clusters simply to regain autonomy.

This publish offers a sample that preserves crew autonomy with out splitting the {hardware}. This answer entails a single management aircraft cluster with a GPU pool, GPU sharing with per-team quotas, and remoted Kubernetes management aircraft per crew together with an API server, controller, knowledge retailer, syncer, and scheduler. This may be achieved utilizing two open supply instruments: KAI Scheduler and vCluster. 

Comply with together with the tutorial steps and also you’ll have three groups operating actual GPU Kubernetes pods in their very own tenant clusters, all sharing a single bodily GPU. You’ll additionally be capable to confirm that every crew solely sees their very own workloads.

Tutorial conditions and notes

To maintain the method reproducible for customers which have restricted sources, this tutorial makes use of a cluster with one NVIDIA L40S GPU and three groups sharing fractions of it. This makes the transferring components simple to see and check out. The method works the identical means on a bigger cluster with tons of of GPU nodes and dozens of groups. You possibly can scale the node pool, the queue hierarchy, and the variety of tenant clusters. 

KAI Scheduler is a strong, environment friendly, and scalable topology-aware Kubernetes scheduler that was purpose-built for optimizing GPU useful resource allocation for AI workloads. It’s designed to handle large-scale GPU clusters, together with hundreds of nodes, and a excessive throughput of workloads. With KAI Scheduler, you possibly can dynamically allocate GPU sources to workloads. It may run alongside the default kube-scheduler. Any pod with schedulerName: kai-scheduler is dealt with by KAI Scheduler. Every thing else goes by means of the standard kube-scheduler course of. 

The vCluster Kubernetes platform provisions absolutely remoted tenant clusters in your infrastructure or instantly on naked steel. Every tenant cluster will get its personal API server, customized useful resource definitions (CRDs), and role-based entry management (RBAC), indistinguishable from a devoted Kubernetes cluster, whereas sharing the underlying nodes and {hardware}. The virtualized management aircraft is invisible to tenants: no shared management aircraft nodes, no in-cluster agent pods, and no lateral path between environments. That makes vCluster a pure match for GPU infrastructure the place groups want their very own clear cluster expertise with out splitting the {hardware}.

The vCluster shared-nodes mannequin is used for this tutorial, so groups share the GPU node whereas every will get its personal remoted management aircraft, the suitable match for trusted inner groups. For untrusted tenants needing node-, network-, and storage-level separation, the identical sample extends to vCluster personal nodes.

The instance on this publish makes use of three groups: NLP Group, Imaginative and prescient Group, and Recommender System Group. The NLP Group needs to put in their very own CRDs. The Imaginative and prescient Group needs cluster-admin to debug scheduling. The Recommender Group is on a unique Kubeflow model. No one needs to share a kubectl context and unintentionally break each other’s environments.

Utilizing vCluster, every crew will get their very own remoted management aircraft, RBAC, namespaces, and CRDs. They will every have cluster-admin entry. Beneath, all tenant clusters share the identical nodes and GPUs.

Demo setting

This demo runs on an NVIDIA Brev GPU occasion on Nebius with:

One NVIDIA L40S, 40 vCPUs, 160 GiB RAM, 256 GiB disk (48 GB VRAM)

Ubuntu 24.04.4 LTS

MicroK8s v1.36.2 – Kubernetes was preconfigured by Brev, together with the MicroK8s gpu addon, which pre-installs the NVIDIA GPU Operator into the gpu-operator-resources namespace

KAI Scheduler v0.16.4

vCluster CLI 0.35.1

Notice: For various setups, the cluster-creation and GPU Operator set up steps will differ on GKE/EKS/AKS/vanilla k8s/k3s. Step 3 (KAI Scheduler) onward is an identical on any Kubernetes that has the NVIDIA GPU Operator operating with Container Machine Interface (CDI) enabled.

Set up standalone kubectl and helm to keep away from prefixing each command with microk8s. Then wire up a kubeconfig and pin MicroK8s so snap doesn’t auto-upgrade the management aircraft in the course of the demo.

Subsequent, be certain the required MicroK8s addons are enabled:

Then confirm: 

Step 2: Add the Helm repo 

Add the NVIDIA Helm repo:

Step 3: Verify the GPU Operator

In case you’re utilizing an older GPU Operator, improve in place to 26.3.x:

If wanted, set up NVIDIA GPU Operator instantly:

Step 4: Set up KAI Scheduler

Subsequent, set up the KAI Scheduler:

Then confirm:

Step 5: Outline crew queues

KAI Scheduler makes use of a Queue CRD to mannequin an org → crew hierarchy. This step entails creating one mother or father (ml-org, with a complete funds of 1 GPU) and three youngster queues. Every is assured 0.33 of the GPU and allowed to extend to the total GPU when others are idle. 

Save the next as create-queues.yaml:

Right here, quota is the assured minimal, restrict is the utmost allowed, and overQuotaWeight controls how surplus is break up.

Subsequent, apply and record:

Notice that default-parent-queue and default-queue are created mechanically by KAI Scheduler on first set up. They’re the fallback queue for any pod that doesn’t specify one.

Step 6: Spin up a vCluster per crew

Subsequent, set up the vCluster CLI:

Outline the vCluster config. The crucial setting is setOwner: false. The KAI Scheduler pod-grouper walks possession chains (Job → Pod, Deployment → ReplicaSet → Pod) to auto-group workloads. Disabling vCluster proprietor rewriting permits KAI Scheduler to see the actual hierarchy.

Then create one vCluster per crew:

Confirm that every one three vClusters are operating:

Every crew sees the actual nodes, together with the GPU node:

Step 7: Deploy a workload from every crew

Every crew deploys their GPU workload from their very own vCluster. The pod spec is straightforward, with three fields telling KAI Scheduler what to do:

Do the identical to deploy from every crew, altering the queue identify. 

Now, confirm:

Three pods, three totally different vcluster-team-* namespaces, all operating on the identical bodily node brev-8dq0cch1j. 

Every crew sees solely their very own pod from inside their vCluster:

Lastly, the queue confirms the allocation. You possibly can test all three on the identical time:

You will note the identical end result for all:

Notice that KAI Scheduler handles scheduling—which pods land on which GPU and in what quantity. It doesn’t implement GPU reminiscence isolation on the {hardware} degree when GPU sharing is used. Functions have to respect their reminiscence quantity (for instance, setting –gpu-memory-utilization in vLLM).

Beneath the hood, the GPU time-slices between CUDA contexts from every pod at kernel boundaries. For laborious reminiscence isolation on supported {hardware}, NVIDIA Multi-Occasion GPU (MIG) offers hardware-level partitioning, which may be scheduled by KAI Scheduler as nicely.

Get began operating remoted tenant Kubernetes clusters

KAI Scheduler decides pretty who will get GPU slices and schedules them as a gaggle utilizing GPU sharing and DRA Driver assist, hierarchical queues with assured quotas plus over-quota capabilities, gang scheduling, and topology consciousness. As soon as AI workload scheduling is solved, groups need their very own clusters. vCluster offers every crew its personal remoted Kubernetes management aircraft with out the price of separate infrastructure. 

Collectively, KAI Scheduler and vCluster present a devoted cluster expertise for 3 groups with zero waste on a single GPU. The reply isn’t at all times extra GPUs, however higher utilization of infrastructure.

Able to get began? Try KAI Scheduler, vCluster, and NVIDIA GPU Operator on GitHub.

Study extra about KAI Scheduler and vCluster integration at KubeCon 2026 North America, November 9-12.