# Productionalizing LLMs for Stripe's Support Operations ## Background and Problem Statement Stripe, a global payments company serving millions of customers worldwide, handles tens of thousands of support cases weekly through their support operations team. With most support interactions being text-based, they identified LLMs as a promising solution to improve support efficiency. The primary goal was to help support agents solve cases more quickly by providing them with AI-generated relevant responses to customer questions, while maintaining human interaction and ensuring accuracy and proper tone. ## Initial Approach and Key Challenges ### The Oracle Fallacy - Initial attempts using out-of-box GPT-DaVinci revealed limitations - Raw LLM responses were plausible but often incorrect due to: - Simple prompt engineering proved insufficient for the complexity and scope of Stripe's support domain ### Solution Framework Evolution The team developed a multi-stage approach breaking down the complex problem into manageable ML components: - Question Validation Stage - Topic Classification Stage - Answer Generation Stage - Tone Modification Stage ### Fine-tuning Implementation - Required approximately 500 labels per class - Relied on expert agent annotations - Successfully mitigated hallucination issues - Improved reliability and interpretability of results ## Production Deployment and Monitoring ### Online Feedback Challenges - Initial development relied heavily on offline blacktest evaluations - Expert agents performed manual review and labeling - Comprehensive user testing and training data collection - Gap in online accuracy monitoring capabilities - Lower than expected adoption rates in production ### Monitoring Solutions - Developed heuristic-based match rate metric - Implementation of comprehensive monitoring dashboard - Emphasis on shipping components in shadow mode for sequential validation ## Data Management and Scaling ### Data-Centric Approach - Confirmed 80/20 rule in development effort - Data quality improvements yielded better results than advanced model architectures - Error patterns showed domain-specific issues rather than general language understanding gaps ### Evolution of the Solution - Second iteration moved away from generative components - Adopted simpler classification approach - Implemented weak supervision techniques ### Long-term Data Strategy - Established subject matter expertise program - Created maintenance procedures for data freshness - Developed "living oracle" concept for continuous updates - Focused on keeping model responses current with product evolution ## Key Lessons Learned ### LLM Implementation - LLMs require careful problem decomposition - Domain expertise integration is crucial - Simple architectures with good data often outperform complex solutions ### Production Considerations - Early UX team engagement is critical - Online monitoring should be prioritized equally with development - Proxy metrics are valuable when exact measurements aren't possible - Sequential shadow deployment enables better debugging and validation ### Data Management - Data quality and quantity remain fundamental - Sustainable data collection and maintenance strategies are essential - Domain-specific training data is crucial for accuracy - Weak supervision can enable efficient scaling ## Impact and Future Directions The project demonstrated the potential for LLMs in customer support while highlighting the importance of proper productionalization practices. Stripe's experience shows that successful LLM deployment requires: - Strong focus on data quality and maintenance - Robust monitoring and feedback systems - Careful attention to user experience and adoption - Pragmatic approach to model complexity - Long-term strategy for keeping the system current These lessons continue to inform Stripe's ongoing development of support automation tools and their broader ML infrastructure.