This case study provides an in-depth look at how Loom, a video communication platform, approaches the challenge of evaluating and deploying LLM-generated content in production, specifically focusing on their automatic video title generation feature. The case study is particularly valuable as it demonstrates a systematic approach to one of the most critical aspects of LLMOps: ensuring consistent quality of AI-generated content in production systems. Loom's approach to LLMOps is centered around a robust evaluation framework that combines both automated and LLM-based assessment methods. What makes their approach particularly noteworthy is their focus on user-centric evaluation criteria rather than just model performance metrics. This represents a mature understanding of what matters in production AI systems - the actual user experience rather than merely technical metrics. The core of their LLMOps implementation revolves around a four-step process for developing and implementing evaluation systems: First, they begin with a thorough analysis of what makes for high-quality output in their specific use case. For video titles, this includes aspects like conveying the main idea, being concise yet descriptive, and maintaining engagement while avoiding hallucinations. This step is crucial for establishing clear, actionable quality criteria that align with business objectives. Second, they incorporate standard quality measures that are common across LLM applications. Their framework considers multiple dimensions: * Relevance and faithfulness to source material * Readability and clarity of language * Structural correctness and formatting * Factual accuracy, particularly important for RAG-based systems * Safety considerations including bias and toxicity checks * Language correctness The third step involves implementing deterministic, code-based scorers wherever possible. This is a crucial LLMOps best practice as it reduces costs, improves reliability, and speeds up evaluation processes. For instance, they use code-based checks for structural elements like emoji placement or JSON schema validation, reserving more expensive LLM-based evaluation for more nuanced aspects that require human-like judgment. The fourth step focuses on iterative improvement through careful testing and refinement. They employ a combination of: * LLM-as-judge evaluators with chain-of-thought reasoning enabled * Distinct scorers for different quality aspects * Weighted averaging for combining multiple quality scores * Flexible scoring scales (binary vs. multi-point) based on the specific quality aspect being evaluated From an LLMOps perspective, several key technical implementation details stand out: Their use of the Braintrust platform for managing evaluations shows an understanding of the importance of having systematic tools and processes for LLM quality assessment. The platform allows them to run both offline evaluations during development and online evaluations in production, enabling continuous monitoring and improvement of their AI features. The team's emphasis on chain-of-thought reasoning in their LLM-based evaluators is particularly noteworthy. This approach not only provides scores but also explanations for those scores, making it easier to debug and improve the system. This transparency is crucial for maintaining and improving production AI systems. The case study also demonstrates sophisticated handling of evaluation complexity through: * Careful separation of concerns in their scoring functions * Implementation of weighted averaging for different quality aspects * Use of appropriate granularity in scoring scales * Integration of both objective and subjective measures From a production operations perspective, their system appears well-designed for scaling and maintenance. The combination of fast, cheap code-based evaluators with more sophisticated LLM-based judges allows for efficient use of computational resources while maintaining high quality standards. The results of their approach demonstrate the value of systematic LLMOps practices. By implementing this comprehensive evaluation framework, Loom has been able to: * Ship AI features faster and more confidently * Maintain consistent quality in production * Systematically improve their AI features over time * Scale their evaluation process effectively While the case study focuses primarily on video title generation, the framework they've developed is clearly applicable to a wide range of LLM applications. Their approach shows how careful attention to evaluation and quality control can create a foundation for successful LLM deployments in production. The case study also highlights some limitations and areas for potential improvement. For instance, while they mention online evaluations, there's limited detail about how they handle real-time monitoring and adjustment of their systems in production. Additionally, more information about how they manage evaluation costs at scale would be valuable. Overall, this case study provides valuable insights into how to systematically approach LLM evaluation and quality control in production systems. It demonstrates that successful LLMOps requires not just technical sophistication but also careful attention to process and methodology.