Whatnot's LLM Implementation for Trust and Safety

Whatnot, a rapidly growing live shopping marketplace, has implemented an innovative approach to trust and safety using Large Language Models (LLMs). This case study details their journey from a traditional rule-based system to an advanced LLM-powered solution that handles content moderation, fraud detection, and policy enforcement.

System Evolution and Architecture

Initial Rule Engine

• Started with a centralized rule engine as the foundation
• Focused on data-driven policy enforcement
• Processed multiple data sources for violation detection
• Handled routine tasks like shipping delays, refunds, and cancellations
• Limited to scalar values and struggled with contextual understanding

Enhanced System with LLMs

• Integrated LLMs to overcome rule engine limitations
• Created a three-phase architecture:

Technical Implementation Details

Scam Detection System

• Implements contextual analysis of user interactions
• Uses multiple signals to trigger LLM analysis:

Prompt Engineering

• Carefully crafted prompts for scam detection
• Structured JSON output format:
• Example prompt structure includes:

Integration with Existing Systems

• LLM outputs feed back into rule engine
• Decision matrix combines multiple factors:
• Uses Kafka for system updates and product access changes

Advanced Features and Adaptations

Multimodal Processing

• Implemented OCR for analyzing text in images
• Handles evolving fraud tactics
• Adapts to different messaging patterns
• Expanded to cover multiple policy violations:

Machine Learning Approach

• Utilizes zero-shot and few-shot learning
• Plans for fine-tuning in specific use cases
• Maintains human-in-the-loop verification
• Combines AI insights with human judgment

System Performance and Results

Metrics and Achievements

• 95% detection rate for scam attempts
• 96% precision in fraud detection
• Rapid detection within minutes of attempt
• High recall rate in identifying violations

Operational Implementation

• Three-tier enforcement system:
• Automated notification system for violations
• Dynamic product access modifications

Technical Stack and Integration

Trust and Safety LLM Stack

• Modular architecture for different violation types
• Scalable processing pipeline
• Integration with existing ML models
• Real-time data processing capabilities

Data Processing Pipeline

• Event data collection
• User data integration
• Order history analysis
• ML model output incorporation

Future Developments

Planned Improvements

• Moving towards unified Gen AI system
• Enhanced fine-tuning for support cases
• Expansion to new use cases
• Stronger integration between rule engine and enforcement
  The system demonstrates a sophisticated approach to LLMOps in production, combining traditional rule-based systems with modern LLM capabilities. The architecture maintains flexibility while ensuring robust policy enforcement, showing how LLMs can be effectively deployed in critical trust and safety applications while maintaining human oversight and control.