Company
Whatnot
Title
LLM-Enhanced Trust and Safety Platform for E-commerce Content Moderation
Industry
E-commerce
Year
2023
Summary (short)
Whatnot, a live shopping marketplace, implemented LLMs to enhance their trust and safety operations by moving beyond traditional rule-based systems. They developed a sophisticated system combining LLMs with their existing rule engine to detect scams, moderate content, and enforce platform policies. The system achieved over 95% detection rate of scam attempts with 96% precision by analyzing conversational context and user behavior patterns, while maintaining a human-in-the-loop approach for final decisions.

Whatnot's LLM Implementation for Trust and Safety

Whatnot, a rapidly growing live shopping marketplace, has implemented an innovative approach to trust and safety using Large Language Models (LLMs). This case study details their journey from a traditional rule-based system to an advanced LLM-powered solution that handles content moderation, fraud detection, and policy enforcement.

System Evolution and Architecture

Initial Rule Engine

  • Started with a centralized rule engine as the foundation
  • Focused on data-driven policy enforcement
  • Processed multiple data sources for violation detection
  • Handled routine tasks like shipping delays, refunds, and cancellations
  • Limited to scalar values and struggled with contextual understanding

Enhanced System with LLMs

  • Integrated LLMs to overcome rule engine limitations
  • Created a three-phase architecture:

Technical Implementation Details

Scam Detection System

  • Implements contextual analysis of user interactions
  • Uses multiple signals to trigger LLM analysis:

Prompt Engineering

  • Carefully crafted prompts for scam detection
  • Structured JSON output format:
  • Example prompt structure includes:

Integration with Existing Systems

  • LLM outputs feed back into rule engine
  • Decision matrix combines multiple factors:
  • Uses Kafka for system updates and product access changes

Advanced Features and Adaptations

Multimodal Processing

  • Implemented OCR for analyzing text in images
  • Handles evolving fraud tactics
  • Adapts to different messaging patterns
  • Expanded to cover multiple policy violations:

Machine Learning Approach

  • Utilizes zero-shot and few-shot learning
  • Plans for fine-tuning in specific use cases
  • Maintains human-in-the-loop verification
  • Combines AI insights with human judgment

System Performance and Results

Metrics and Achievements

  • 95% detection rate for scam attempts
  • 96% precision in fraud detection
  • Rapid detection within minutes of attempt
  • High recall rate in identifying violations

Operational Implementation

  • Three-tier enforcement system:
  • Automated notification system for violations
  • Dynamic product access modifications

Technical Stack and Integration

Trust and Safety LLM Stack

  • Modular architecture for different violation types
  • Scalable processing pipeline
  • Integration with existing ML models
  • Real-time data processing capabilities

Data Processing Pipeline

  • Event data collection
  • User data integration
  • Order history analysis
  • ML model output incorporation

Future Developments

Planned Improvements

  • Moving towards unified Gen AI system
  • Enhanced fine-tuning for support cases
  • Expansion to new use cases
  • Stronger integration between rule engine and enforcement
    The system demonstrates a sophisticated approach to LLMOps in production, combining traditional rule-based systems with modern LLM capabilities. The architecture maintains flexibility while ensuring robust policy enforcement, showing how LLMs can be effectively deployed in critical trust and safety applications while maintaining human oversight and control.

Start your new ML Project today with ZenML Pro

Join 1,000s of members already deploying models with ZenML.