Skip to content
Closed
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 135 additions & 2 deletions tutorials/labs/gpu-partitioning.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ tags:
toc_max_heading_level: 2
---

import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem';

This lab continues from [Lab 1](./online-install.md). You have one physical Tesla T4 with 15360 MiB of VRAM. In this lab you will run multiple Pods on that single card, each with its own enforced VRAM and compute limit, and verify that the isolation is real: a Pod that tries to allocate past its slice gets a CUDA OOM while its neighbors keep running.

Every command and output in this lab was captured from a live cluster built with Lab 1 (HAMi v2.9.0, GPU Operator v25.3.0, Kubernetes v1.34).
Expand Down Expand Up @@ -92,11 +94,50 @@ spec:

`gpumem-pod-b.yaml` is identical except for the name. Apply both:

<Tabs groupId="cloud-provider">
<TabItem value="aws" label="AWS">

Open a seperate terminal.
Comment thread
Creativeklvn marked this conversation as resolved.
Outdated

Use the same PEM key file and `NODE_PUBLIC_IP` defined in **Step 1.3: SSH Into the Instance** to copy the YAML files to the EC2 instance.

```bash
scp -i ~/<your-pem-key-file-path>/hami-eks.pem \
Comment thread
Creativeklvn marked this conversation as resolved.
Outdated
tutorials/labs/examples/03-gpu-partitioning/gpumem-pod-a.yaml \
tutorials/labs/examples/03-gpu-partitioning/gpumem-pod-b.yaml \
ubuntu@<your-vm-public-ip>:/home/ubuntu/
```

Return to the EC2 terminal.

```bash
ls /home/ubuntu/
```

Expected output, you should see:

```plaintext
gpumem-pod-a.yaml
gpumem-pod-b.yaml
```

```bash
kubectl apply -f /home/ubuntu/gpumem-pod-a.yaml -f /home/ubuntu/gpumem-pod-b.yaml
kubectl get pods gpumem-pod-a gpumem-pod-b -o wide
```

</TabItem>
<TabItem value="gcp" label="GCP">

```bash
kubectl apply -f tutorials/labs/examples/03-gpu-partitioning/gpumem-pod-a.yaml -f tutorials/labs/examples/03-gpu-partitioning/gpumem-pod-b.yaml

kubectl get pods gpumem-pod-a gpumem-pod-b -o wide
```

</TabItem>
</Tabs>

```plaintext
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
gpumem-pod-a 1/1 Running 0 27s 10.244.218.156 hami-workshop <none> <none>
Expand Down Expand Up @@ -173,10 +214,35 @@ spec:
nvidia.com/gpumem: 4000 # limit this Pod to 4000 MiB of VRAM
```

<Tabs groupId="cloud-provider">
<TabItem value="aws" label="AWS">

Open a seperate terminal

Use the same PEM key file and `NODE_PUBLIC_IP` defined in **Step 1.3: SSH Into the Instance** to copy the YAML files to the EC2 instance.

```bash
scp -i ~/<your-pem-key-file-path>/hami-eks.pem \
tutorials/labs/examples/03-gpu-partitioning/oom-test-pod.yaml \
ubuntu@<your-vm-public-ip>:/home/ubuntu/
```

Return to the EC2 terminal.

```bash
kubectl apply -f /home/ubuntu/oom-test-pod.yaml
```

</TabItem>
<TabItem value="gcp" label="GCP">

```bash
kubectl apply -f tutorials/labs/examples/03-gpu-partitioning/oom-test-pod.yaml
```

</TabItem>
</Tabs>

While the image pulls, watch the HAMi scheduler make its decision:

```bash
Expand All @@ -191,17 +257,29 @@ kubectl describe pod oom-test-pod | tail -3

> `FilteringSucceed` shows the scheduler scoring nodes (here `hami-workshop:7.21`), and `BindingSucceed` shows it binding the Pod. These events come from hami-scheduler, not the default scheduler. Note it found a fit even though two Pods already occupy the GPU: 8000 of 15360 MiB are reserved, so a third 4000 MiB slice still fits.

Wait for the Pod to complete, then read its logs:
Wait for the Pod to be completed, then read its logs:

```bash
kubectl get pods oom-test-pod -o wide
```

Expected output:

```plaintext
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
oom-test-pod 0/1 Completed 0 6m17s 10.244.109.31 ip-172-31-8-251 <none> <none>
Comment thread
Creativeklvn marked this conversation as resolved.
Outdated
```

```bash
kubectl logs oom-test-pod | tail -8
```

```plaintext
Allocated 2048 MiB
Allocated 2560 MiB
Allocated 3072 MiB
Allocated 3584 MiB
[HAMI-core ERROR (pid:1 thread=... allocator.c:52)]: Device 0 OOM 4399824896 / 4194304000
[HAMI-core ERROR (pid:1 thread=128984570163776 allocator.c:52)]: Device 0 OOM 4399824896 / 4194304000
Hit the limit after 3584 MiB:
CUDA out of memory
```
Expand Down Expand Up @@ -239,10 +317,57 @@ resources:
nvidia.com/gpucores: 30 # limit this Pod to 30% of GPU compute
```

<Tabs groupId="cloud-provider">
<TabItem value="aws" label="AWS">

Open a seperate terminal

Use the same PEM key file and `NODE_PUBLIC_IP` defined in **Step 1.3: SSH Into the Instance** to copy the YAML files to the EC2 instance.

```bash
scp -i ~/<your-pem-key-file-path>/hami-eks.pem \
tutorials/labs/examples/03-gpu-partitioning/gpucores-pod.yaml \
ubuntu@<your-vm-public-ip>:/home/ubuntu/
```

Return to the EC2 terminal.

```bash
ls /home/ubuntu/
```

Expected output, you should see:

```plaintext
gpucores-pod.yaml gpumem-pod-b.yaml
gpumem-pod-a.yaml oom-test-pod.yaml
```

```bash
kubectl apply -f /home/ubuntu/gpucores-pod.yaml
```

</TabItem>
<TabItem value="gcp" label="GCP">

```bash
kubectl apply -f tutorials/labs/examples/03-gpu-partitioning/gpucores-pod.yaml
```

</TabItem>
</Tabs>

```bash
kubectl get pod gpucores-pod -o wide
```

Expected output:

```plaintext
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
gpucores-pod 1/1 Running 0 2m 10.244.14.160 ip-172-31-9-124 <none> <none>
```

Check the environment HAMi injected into the container:

```bash
Expand Down Expand Up @@ -299,6 +424,14 @@ kubectl exec -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 -c
>
> Without the `force` env, you would see this same workload run at 100% utilization on an idle card, which is intentional: HAMi gives idle capacity away rather than wasting it.

(Optional) Access the WebUI via port forwarding:

```bash
kubectl port-forward --address 0.0.0.0 service/my-hami-webui 3000:3000 --namespace=kube-system
```
Comment thread
Creativeklvn marked this conversation as resolved.

Visit `http://<your-vm-public-ip-address>:3000` to open the HAMi WebUI.

Comment thread
coderabbitai[bot] marked this conversation as resolved.
## Step 6: Cleanup

```bash
Expand Down