π³ Docker 360 β Week 15 of 16 | GPUs in containers + experiment tracking
π― Learning Objective
Machine learning ka kaam GPUs ke baghair adhura hai β aur GPUs ka setup βmeray machine par CUDA version mismatch haiβ wali museebat ke liye mashhoor hai. Is week me tum NVIDIA Container Toolkit se GPU ko container ke andar expose karna seekhoge, CUDA base images ke tags samjhoge, nvidia-smi ko container me chala kar verify karoge, aur MLflow tracking server ko Compose par khada karke ek sample experiment log karoge. End tak tumhare paas ek reproducible ML lab hoga.
π¨ Visual Diagrams
GPU access ka rasta β host se container tak:
Host Machine (NVIDIA GPU + driver installed)
ββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β NVIDIA driver (kernel level β host par hi hota β
β hai, container me driver install NAHI hota!) β
β β
β NVIDIA Container Toolkit β
β (nvidia-container-runtime: GPU devices ko β
β container me "inject" karta hai) β
β β
β ββββββββββββββββββββββββββββββββββββββββββββββββ β
β β Container (nvidia/cuda:xx.x-base) β β
β β ββββββββββββββββββββββββββββββββββββββ β β
β β β CUDA libraries (user-space) β β β
β β β /dev/nvidia0 βββ toolkit se aya β β β
β β β nvidia-smi β torch.cuda β β β β
β β ββββββββββββββββββββββββββββββββββββββ β β
β ββββββββββββββββββββββββββββββββββββββββββββββββ β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Golden rule: driver HOST par, CUDA libraries CONTAINER me.
MLflow tracking ka flow:
ββββββββββββββββββββ log_param/log_metric ββββββββββββββββββββ
β training β βββββββββββββββββββββββββΊ β mlflow server β
β container β http://mlflow:5000 β (compose) β
β (train.py) β β β
ββββββββββββββββββββ β ββββββββββββββ β
β β mlflow.db β β params,
β β (sqlite) β β metrics
β ββββββββββββββ€ β
β β artifacts/ β β models,
β β (volume) β β plots
β ββββββββββββββ β
ββββββββββββββββββββ
β UI :5000
βΌ
browser me compare karo π
π§ Theory Deep Dive
NVIDIA Container Toolkit kya karta hai?
Toolkit me do hisse hain: driver-level kaam host par pehle se hota hai (tumhe host par NVIDIA driver install karna hi parhta hai), aur nvidia-container-runtime container start hote waqt GPU device nodes (/dev/nvidia0, etc.) aur driver libraries ko container ke andar mount kar deta hai. Tumhe container me driver install karne ki zaroorat nahi β sirf user-space CUDA libraries chahiye hoti hain, jo CUDA base images me pehle se hoti hain.
Naye Docker versions me ye CDI (Container Device Interface) ke zariye bhi hota hai, lekin --gpus all flag sab se simple tareeqa hai aur wahi hum lab me use karenge.
CUDA image tags β base vs runtime vs devel
| Tag suffix | Kya included hai | Kab use karo |
|---|---|---|
nvidia/cuda:12.4.1-base-ubuntu22.04 | Sirf CUDA runtime libs (minimal) | GPU check, chhoti utilities |
...-runtime-... | Base + extra math libs (cuDNN etc.) | Inference / PyTorch runtime images |
...-devel-... | Runtime + compilers, headers (nvcc) | Khud CUDA code compile karna ho |
Iska matlab ye hai ke production inference ke liye runtime kafi hai β devel image be-wajah bhaari hogi.
MLflow ke 3 core concepts
- Tracking: har run ke params, metrics aur artifacts ek central server par log hote hain.
- Backend store: metadata kahan save hogi β hum SQLite file use karenge (volume par, taake restart par data na urhe).
- Artifact store: models aur files kahan save honge β ek named volume.
β οΈ Common gotchas
- βcould not select device driver nvidiaβ β iska matlab NVIDIA Container Toolkit install nahi hai ya Docker daemon restart nahi hua. Toolkit install karke
sudo systemctl restart dockerchalao. - WSL2 users: Docker Desktop ke bajaye WSL2 me native Docker + toolkit zyada reliable hai GPU ke liye.
- Image size:
pytorch/pytorchCUDA images kai GB ki hoti hain β pehli pull me waqt lagega, ghabrana nahi. - MLflow artifacts bahar se: agar training container alag host par ho, to
--default-artifact-rootko S3 jaisi shared storage par point karo, local volume par nahi.
π» Hands-On Lab
Note: GPU labs ke liye host par NVIDIA GPU + driver + NVIDIA Container Toolkit installed hona chahiye. Agar GPU nahi hai, to MLflow wala hissa (Lab 3-4) phir bhi poora kar sakte ho β wo CPU par chalta hai.
Lab 1 β Toolkit check
Step 1: Verify karo ke Docker GPU dekh sakta hai:
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv
β Expected result: Tumhare GPU ka naam, driver version aur memory β container ke andar se! Agar error aaye to toolkit install nahi hai.
Step 2: (Optional, bhaari image) PyTorch se CUDA test:
docker run --rm --gpus all pytorch/pytorch:2.5.0-cuda12.4-cudnn9-runtime \
python -c "import torch; print('cuda available:', torch.cuda.is_available()); print('device:', torch.cuda.get_device_name(0))"
β
Expected result: cuda available: True aur GPU ka naam.
Lab 2 β Khud GPU test image banao
Step 1:
mkdir -p /tmp/gpu-lab && cd /tmp/gpu-lab
cat > Dockerfile <<'EOF'
FROM nvidia/cuda:12.4.1-runtime-ubuntu22.04
RUN apt-get update && apt-get install -y --no-install-recommends python3 && rm -rf /var/lib/apt/lists/*
CMD ["nvidia-smi", "-L"]
EOF
docker build -t gpu-test .
Step 2:
docker run --rm --gpus all gpu-test
β
Expected result: GPU 0: <tumhara GPU naam> β tumhari apni built image GPU dekh rahi hai.
Lab 3 β MLflow server on Compose
Step 1: Project banao:
mkdir -p /tmp/mlflow-lab && cd /tmp/mlflow-lab
cat > Dockerfile.mlflow <<'EOF'
FROM python:3.11-slim
RUN pip install --no-cache-dir "mlflow==2.16.0"
EXPOSE 5000
CMD ["mlflow", "server", "--host", "0.0.0.0", "--port", "5000", \
"--backend-store-uri", "sqlite:////mlflow/mlflow.db", \
"--default-artifact-root", "/mlflow/artifacts"]
EOF
Step 2: compose.yaml:
services:
mlflow:
build:
context: .
dockerfile: Dockerfile.mlflow
ports:
- "5000:5000"
volumes:
- mlflow-data:/mlflow
volumes:
mlflow-data:
Step 3: Start karo:
docker compose up --build -d
sleep 10
curl -s localhost:5000/ | head -c 200; echo
β
Expected result: HTML response β MLflow UI http://localhost:5000 par live hai. Browser me kholo to khaali experiments list nazar aayegi.
Lab 4 β Sample experiment log karo
Step 1: Training script likho (host par venv ya system python me pip install mlflow scikit-learn chahiye):
pip install -q mlflow scikit-learn
cat > train.py <<'EOF'
import mlflow
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("iris-baseline")
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
for C in [0.1, 1.0, 10.0]:
with mlflow.start_run(run_name=f"C={C}"):
model = LogisticRegression(C=C, max_iter=200)
model.fit(X_train, y_train)
acc = accuracy_score(y_test, model.predict(X_test))
mlflow.log_param("C", C)
mlflow.log_metric("accuracy", acc)
mlflow.sklearn.log_model(model, "model")
print(f"C={C} -> accuracy={acc:.4f}")
EOF
python train.py
β
Expected result: Teenon runs ke accuracy scores print honge, aur MLflow UI me iris-baseline experiment ke andar 3 runs β params, metrics aur saved models ke sath. Har run compare ki ja sakti hai!
Step 2: Data persist hai ye verify karo β server restart karke dekho:
docker compose restart
sleep 8
curl -s "localhost:5000/api/2.0/mlflow/experiments/search" | head -c 300; echo
β Expected result: Experiment ab bhi mojood hai β volume ki wajah se restart par data nahi urha.
Step 3: (GPU walo ke liye bonus) Training ko GPU container me chalao:
docker run --rm --gpus all --network host \
-v /tmp/mlflow-lab:/work -w /work \
pytorch/pytorch:2.5.0-cuda12.4-cudnn9-runtime \
python -c "import torch; print(torch.cuda.get_device_name(0))"
Step 4: Safai:
cd /tmp/mlflow-lab && docker compose down -v
β Checkpoint
Q1: GPU container me NVIDIA driver install karne ki zaroorat kyun nahi hoti? A: Kyunke driver kernel-level host par hota hai; NVIDIA Container Toolkit sirf device nodes aur user-space libraries ko container me expose karta hai.
Q2: nvidia/cuda ke base, runtime aur devel tags me kya farq hai?
A: Base minimal hai, runtime me inference libraries (cuDNN waghera) hoti hain, devel me compilers/headers (nvcc) β devel sirf tab jab khud CUDA code compile karna ho.
Q3: MLflow me backend store aur artifact store me kya farq hai? A: Backend store (humari sqlite file) me params/metrics ka metadata hota hai; artifact store (volume) me models, plots jaisi files save hoti hain.
Q4: mlflow.set_tracking_uri("http://localhost:5000") ki kya zaroorat hai?
A: Iske baghair MLflow local ./mlruns folder me log karta hai β tracking URI se saare runs central server par jate hain jahan team compare kar sakti hai.
π Next Week
Final week! Week 16 me batch jobs, model ko API ke peeche serve karna, replicas ke sath scaling β aur poore 16-week safar ka wrap-up. π