# Slack's LLM Evaluation Framework for Production Systems ## Company and Use Case Overview Slack, a leading business communication platform, has integrated LLM-powered features into their product, focusing primarily on two main applications: - Message summarization capabilities (summarizing channels, threads, or time periods) - Natural language search functionality (allowing users to ask questions in the search bar and receive LLM-generated responses) The presentation was delivered by Austin, a Staff Software Engineer on Slack's machine learning modeling team, who shared their approach to evaluating and improving generative AI products at scale. ## The Challenge of LLM Evaluation Slack identified several key challenges in evaluating LLM outputs: ### Subjectivity Issues - Different users have varying preferences for output style and detail level - What constitutes a "good" summary varies based on user needs and context - The same output could be considered excellent or poor depending on the use case ### Objective Quality Measures - Accuracy of information - Coherence and language quality - Grammatical correctness - Relevance to user queries - Integration with Slack-specific features and formatting ## Evaluation Framework Architecture ### Breaking Down Complex Problems Slack's approach involves decomposing large evaluation challenges into smaller, more manageable components: - Hallucination detection and management - Slack-specific integration requirements - System integration validation - Output quality scoring ### Automated Quality Metrics System The team developed a comprehensive set of automated quality metrics that: - Generate individual scores for different quality aspects - Combine into composite quality scores - Can be applied consistently across different products and features ### Hallucination Management Specific focus areas include: - Extrinsic hallucination detection (preventing the LLM from generating content outside provided context) - Citation accuracy verification - Reference validation ### Evaluation Methods Multiple approaches are used for evaluation: - LLM-based evaluators for complex assessment - Natural language inference modeling for scale - Sampling techniques to manage computational resources - Integration with existing Slack systems ## Three-Tier Evaluation Process ### 1. Golden Set Testing - Small sample of carefully curated messages - Allows for quick prototyping - Provides immediate feedback on changes - Visible underlying data for detailed analysis ### 2. Validation Set Testing - 100-500 samples - More representative of real-world usage - Blind testing (underlying data not visible) - Automated quality metric assessment - Used for verification before larger-scale deployment ### 3. A/B Testing - Production-level testing - Quality metrics integrated into experiment analysis - Used to validate actual user impact - Confirms continuous product improvement ## Development Lifecycle Integration ### Stage-Gate Process - Enables rapid prototyping - Allows for quick failure identification - Supports iterative improvement - Maintains quality standards throughout development ### Example Implementation The team successfully implemented extractive summarization as a preprocessing technique: - Addressed large context size challenges - Demonstrated quality improvements through automated metrics - Showed enhanced format capabilities - Resulted in improved user experience ## Quality Assurance Practices ### Continuous Evaluation - Regular assessment of all generative products - Standardized development cycle - Focus on rapid prototyping capabilities - Emphasis on failing fast and learning quickly ### System-Level Assessment - Evaluation goes beyond just model outputs - Considers entire system integration - Includes user experience metrics - Accounts for technical integration requirements ## Key Takeaways The success of Slack's LLM evaluation framework demonstrates: - The importance of systematic evaluation in production LLM systems - Benefits of breaking down complex evaluation tasks - Value of automated quality metrics - Need for multi-tiered testing approaches - Importance of considering both subjective and objective quality measures - Benefits of standardized development cycles