# LLM Production Deployment Challenges and Solutions at Neeva ## Overview This case study presents insights from Tanmay, a machine learning expert at Neeva (previously at TikTok), discussing the challenges and solutions for deploying LLMs in production environments. The presentation provides a comprehensive framework for understanding and addressing both infrastructural and output-related challenges in LLM deployments. ## Key Challenges ### Infrastructural Challenges - **Speed and Latency** - **Build vs. Buy Decision Making** - **API Reliability Issues** - **Evaluation Challenges** ### Output-Related Challenges - **Format Variability** - **Reproducibility Issues** - **Trust and Safety Concerns** ## Solutions and Best Practices ### Infrastructure Solutions - **Speed Optimization** - **Cost Management Strategy** - **Reliability Enhancement** - **Evaluation Framework** ### Output Management Solutions - **Format Control** - **Reproducibility Enhancement** - **Trust and Safety Measures** ## Implementation Strategy ### Project Positioning - **Focus on Non-Critical Workflows** - **Latency Considerations** ### Long-term Planning - **Scale Planning** - **User Experience Focus** ## Best Practices for Success - **Start Small** - **Plan for Scale** - **User-Centric Approach** - **Technical Robustness**