# Building and Operating Large-Scale GPU Clusters for LLM Training at Meta ## Background and Context Meta has traditionally focused on training recommendation systems for ads, feed, and ranking that required between 8 to 500 GPUs. However, with the advent of generative AI and specifically LLaMA 3, they faced a fundamental shift in computational requirements. While LLaMA 2 was trained on 2 trillion tokens, LLaMA 3 increased to 15 trillion high-quality tokens, necessitating thousands of GPUs running continuously for months. ## Infrastructure Evolution and Challenges ### Hardware Infrastructure - Transitioned from clusters optimized for numerous small/medium training jobs to massive single-job infrastructure - Built the Granton platform with key modifications: ### Hardware Reliability and Operations - Achieving 95% efficiency on 24-hour rolling window basis for LLaMA 3 70B model training - Key challenges addressed: - Implementation of checkpoint and recovery systems - Close collaboration with vendors to address unprecedented scale-related issues ## Network Architecture and Optimization ### Cluster Design - Built two 24K GPU clusters: - Architecture breakdown: ### Network Performance Optimization - Implemented three-pronged optimization strategy: - Network topology awareness integrated across: ## Parallelization Strategies ### Multi-level Parallelism Implementation - Pipeline Parallelism: - Tensor Parallelism: - Data Parallelism: ## System Integration and Performance ### Performance Monitoring and Optimization - Real-time monitoring of GPU performance and thermal conditions - Implementation of automated threshold tuning - Quick failure detection and recovery mechanisms - Efficiency metrics tracking: ### Software Stack - Heavy reliance on PyTorch for rapid research-to-production development - Custom optimizations for collective operations - Integration of topology-aware scheduling - Automated failure handling and recovery systems ## Future Directions and Challenges ### Scaling Beyond Current Limits - Planning for order of magnitude increase in GPU requirements - Addressing reliability at larger scales - Managing increased network distances and latency - Focus on open ecosystems and commodity hardware - Commitment to collaborative problem-solving within the industry ## Key Learnings and Best Practices - Importance of full-stack optimization approach - Critical role of hardware reliability at scale - Significance of network topology in performance - Value of automated monitoring and recovery systems - Need for close vendor collaboration - Balance between power, cooling, and performance requirements The case study demonstrates Meta's comprehensive approach to scaling LLM infrastructure, highlighting the intricate interplay between hardware, networking, and software components necessary for successful large-scale model training operations.