Whatnot's LLM Implementation for Trust and Safety
Whatnot, a rapidly growing live shopping marketplace, has implemented an innovative approach to trust and safety using Large Language Models (LLMs). This case study details their journey from a traditional rule-based system to an advanced LLM-powered solution that handles content moderation, fraud detection, and policy enforcement.
System Evolution and Architecture
Initial Rule Engine
- Started with a centralized rule engine as the foundation
- Focused on data-driven policy enforcement
- Processed multiple data sources for violation detection
- Handled routine tasks like shipping delays, refunds, and cancellations
- Limited to scalar values and struggled with contextual understanding
Enhanced System with LLMs
- Integrated LLMs to overcome rule engine limitations
- Created a three-phase architecture:
Technical Implementation Details
Scam Detection System
- Implements contextual analysis of user interactions
- Uses multiple signals to trigger LLM analysis:
Prompt Engineering
- Carefully crafted prompts for scam detection
- Structured JSON output format:
- Example prompt structure includes:
Integration with Existing Systems
- LLM outputs feed back into rule engine
- Decision matrix combines multiple factors:
- Uses Kafka for system updates and product access changes
Advanced Features and Adaptations
Multimodal Processing
- Implemented OCR for analyzing text in images
- Handles evolving fraud tactics
- Adapts to different messaging patterns
- Expanded to cover multiple policy violations:
Machine Learning Approach
- Utilizes zero-shot and few-shot learning
- Plans for fine-tuning in specific use cases
- Maintains human-in-the-loop verification
- Combines AI insights with human judgment
System Performance and Results
Metrics and Achievements
- 95% detection rate for scam attempts
- 96% precision in fraud detection
- Rapid detection within minutes of attempt
- High recall rate in identifying violations
Operational Implementation
- Three-tier enforcement system:
- Automated notification system for violations
- Dynamic product access modifications
Technical Stack and Integration
Trust and Safety LLM Stack
- Modular architecture for different violation types
- Scalable processing pipeline
- Integration with existing ML models
- Real-time data processing capabilities
Data Processing Pipeline
- Event data collection
- User data integration
- Order history analysis
- ML model output incorporation
Future Developments
Planned Improvements
- Moving towards unified Gen AI system
- Enhanced fine-tuning for support cases
- Expansion to new use cases
- Stronger integration between rule engine and enforcement
The system demonstrates a sophisticated approach to LLMOps in production, combining traditional rule-based systems with modern LLM capabilities. The architecture maintains flexibility while ensuring robust policy enforcement, showing how LLMs can be effectively deployed in critical trust and safety applications while maintaining human oversight and control.