# Replit's LLM Infrastructure Optimization Case Study ## Company Background and LLM Usage Replit is an online web-based development environment that leverages LLMs for various features including: - Code completion - Code transformation - Code explanation - Debugging assistance The company both self-trains and hosts their own models, with some being available open-source on Hugging Face. ## Infrastructure Challenge ### Initial Problem - A100 GPUs, required for low-latency LLM inference, cost approximately $3,000 per month - Spot/preemptable instances offer significant cost savings (around $1,000 per month, a 66% reduction) - However, preemptable instances come with significant operational challenges: ### Risk Mitigation Strategy The team implemented a three-pronged approach: - Multi-zone distribution to spread risk - Fallback to on-demand nodes - Optimization of server startup times ## Technical Deep Dive: Server Startup Optimization ### Initial State The original server startup process took approximately 18 minutes: - 2 minutes for node initialization and driver installation - 11 minutes for application container startup - 5 minutes for model weight loading and health checks ### Container Optimization Initial improvements focused on reducing container size: - Removed pip cache (unnecessary in production) - Eliminated dev/test dependencies - Removed unnecessary CUDA libraries from PyTorch installation - Switched to slim base images - Optimized Triton Inference Server by removing unused framework support - Cleaned up build artifacts ### GKE Image Streaming Implementation - Enabled Google Kubernetes Engine (GKE) image streaming feature - Dramatically reduced container pull time from minutes to seconds - Benefits extended to system containers and Kubernetes components - Works by streaming file contents on-demand rather than pulling entire container ### Model Loading Optimization ### Initial Attempt: - Identified slow model loading from Google Cloud Storage (GCS) - Attempted to optimize by using locally attached NVMe SSDs - Initially saw no improvement with only 50MB/s transfer speeds ### Breakthrough Solution: - Discovered critical issue with container image choice - Switched from Alpine to slim-based Google Cloud SDK image - Uncovered hidden multi-processing limitation in Alpine-based gsutil - Achieved 5x improvement in transfer speeds - Reduced model loading time from 4 minutes to under 30 seconds ## Results and Impact ### Performance Improvements - Total startup time reduced from 18 minutes to under 2 minutes - Key components: ### Business Impact - Successfully implemented preemptable GPU infrastructure - Maintained service uptime despite preemption challenges - Achieved 66% cost reduction in GPU infrastructure ## Technical Implementation Details ### Tools and Technologies Used - Google Kubernetes Engine (GKE) - Triton Inference Server - Google Cloud Storage (GCS) - NVMe SSDs for local storage - Container optimization tools - gsutil for model transfer ### Best Practices Identified - Use container optimization techniques for faster startup - Leverage cloud provider features like image streaming - Carefully choose base images and understand their limitations - Use local SSDs effectively for model loading - Monitor and optimize each component of the startup process - Test thoroughly in preemptable environments ## Future Considerations - Potential for further optimization of startup times - Exploration of H100 GPUs for improved performance - Continuous monitoring and optimization of preemption handling - Balance between cost savings and operational complexity This case study demonstrates how careful optimization of infrastructure and deployment processes can enable the use of cost-effective preemptable instances for LLM serving, despite their inherent challenges. The success of this implementation shows that with proper engineering and optimization, it's possible to run highly available services on preemptable infrastructure while maintaining reliability and significantly reducing costs.