Cloud Infrastructure for AI Workloads
GPU Clusters & MLOps

Executive Summary

Enterprise AI/ML workloads demand specialized infrastructure that balances computational power, scalability, and cost efficiency. This white paper provides architects with detailed technical guidance for designing cloud infrastructure that supports the full ML lifecycle—from experimentation through production inference—while optimizing for the unique constraints of APAC deployment scenarios.

GPU Cluster Architecture

Modern AI workloads, particularly large language model training and inference, require carefully designed GPU infrastructure:

Instance Selection by Workload

• Training (Large models): AWS p5.48xlarge (8x H100), Azure ND H100 v5, GCP a3-highgpu-8g
• Training (Medium models): AWS p4d.24xlarge (8x A100), Azure NC A100 v4, GCP a2-ultragpu-8g
• Inference (High throughput): AWS inf2.48xlarge (Inferentia2), Azure NC v4 (T4), optimized instances
• Inference (Low latency): Single GPU instances with TensorRT optimization

Network Topology for Distributed Training

Large-scale training requires high-bandwidth, low-latency networking:

• Intra-node: NVLink/NVSwitch provides 900 GB/s GPU-to-GPU bandwidth (H100)
• Inter-node: 3200 Gbps EFA (AWS) or InfiniBand NDR (Azure) for gradient synchronization
• Placement groups: Cluster placement ensures co-located instances for minimal latency

Distributed Training Strategies

Scaling training across multiple GPUs requires appropriate parallelization strategies:

Data Parallelism

• Use case: Models that fit in single GPU memory; most common approach
• Implementation: PyTorch DDP, Horovod, or DeepSpeed ZeRO Stage 1
• Scaling efficiency: Near-linear to 64-128 GPUs with proper gradient compression

Model Parallelism

• Use case: Models too large for single GPU memory (>80GB)
• Tensor parallelism: Split layers across GPUs (Megatron-LM approach)
• Pipeline parallelism: Different layers on different GPUs with micro-batching
• Implementation: DeepSpeed, Megatron-LM, FSDP (PyTorch 2.0+)

Hybrid Approaches

• 3D parallelism: Combine data, tensor, and pipeline parallelism for trillion-parameter models
• ZeRO optimization: Partition optimizer states, gradients, and parameters across data-parallel ranks

Model Serving Architecture

Production inference requires different infrastructure considerations than training:

Serving Frameworks

• NVIDIA Triton: Multi-framework support, dynamic batching, model ensemble
• vLLM: Optimized for LLM inference with PagedAttention
• TensorRT-LLM: NVIDIA's optimized LLM inference with INT8/FP8 quantization
• AWS SageMaker: Managed inference with auto-scaling and multi-model endpoints

Scaling Patterns

• Horizontal scaling: Multiple inference pods behind load balancer; works for most models
• Model sharding: Large models split across GPUs with tensor parallelism for inference
• Caching: KV-cache optimization for conversational AI; semantic caching for common queries

MLOps Pipeline Architecture

Mature ML organizations require automated pipelines for the full model lifecycle:

Core Components

• Feature store: Feast, Tecton, or SageMaker Feature Store for consistent feature engineering
• Experiment tracking: MLflow, Weights & Biases, or Neptune for reproducibility
• Model registry: Versioned model storage with deployment approval workflows
• Pipeline orchestration: Kubeflow Pipelines, Airflow, or Step Functions for workflow automation
• Monitoring: Model performance tracking, data drift detection, alert management

Cost Optimization Strategies

AI infrastructure costs can escalate rapidly; implement these optimization techniques:

Compute Cost Reduction

• Spot/Preemptible instances: 60-90% savings for fault-tolerant training jobs with checkpointing
• Reserved capacity: 1-3 year commitments for baseline production inference
• Right-sizing: Match instance types to actual utilization; avoid over-provisioning
• Scheduling: Shut down dev/test environments outside business hours

Model Efficiency

• Quantization: INT8/INT4 inference reduces memory and compute requirements by 2-4x
• Distillation: Smaller student models for production; larger teacher for training
• Pruning: Remove low-impact weights for faster inference

APAC-Specific Considerations

Deploying AI infrastructure in APAC presents unique challenges:

• GPU availability: H100/A100 capacity constrained in ap-southeast regions; consider ap-northeast-1 (Tokyo) or US regions with acceptable latency
• Data sovereignty: Vietnam, Indonesia require local data storage; architect for data localization with cross-border inference
• Network latency: Consider Singapore as regional hub with sub-30ms latency to Vietnam, Malaysia, Thailand
• Cost arbitrage: APAC regions often 10-20% more expensive than US; weigh against data residency requirements