# Building Privacy-First Recommendation Systems at Slack Slack developed a sophisticated yet pragmatic approach to building and deploying recommendation systems at scale while maintaining strict privacy controls around customer data. This case study explores their journey in creating a generic recommendation API that serves multiple internal use cases while protecting sensitive business communications. ## System Overview and API Design - Started with a simple, generic API interface that could handle different recommendation contexts - API accepts queries with context about users and channels, returns recommended users or channels - Deliberately simplified interface to make it accessible to non-ML teams - Hid complexity of embeddings and ML concepts behind clean API boundaries - Enabled rapid prototyping and integration for product teams ## Privacy-First Architecture - Privacy considerations were fundamental to the system design - Strict avoidance of using customer message content in models to prevent data leaks - Focus on interaction patterns rather than semantic content - Careful feature engineering to avoid even hinting at private communications - Models trained to avoid memorizing sensitive information - Separate handling of public vs private channel data ## Feature Engineering and Training - Hundreds of available features for users and channels - Most models effectively use only a few dozen key features - Online feature computation with logging for training data collection - Log and wait approach to avoid feature training-serving skew - Features focused on interaction patterns rather than message content - Signal service built to provide reusable feature computation ## Cold Start Strategy - Innovative use of hand-tuned models for initial deployment - Hand-tuned models serve as: - Makes system more interpretable to product managers - Provides clean transition path from heuristics to ML ## ML Team Structure and Organization - ML Engineers handle full vertical stack: - Team members specialize while maintaining broad capabilities - Mix of engineers from both data science and software engineering backgrounds - Focus on knowledge sharing and paired learning ## Infrastructure and Deployment - Built internal model serving infrastructure - Parsec system for embedding-based nearest neighbor lookups - Test pages for quick result validation - Fast iteration cycles for model improvements - Integration with company-wide experimentation platform - Infrastructure designed to scale across multiple use cases ## Results and Impact - Over 10% improvement when switching from hand-tuned to ML models - Successfully deployed across multiple recommendation use cases - High adoption from internal teams due to simple API design - Rapid iteration capability for fixing issues and improving results - Strong privacy preservation while maintaining recommendation quality ## Lessons Learned - Simple APIs can hide complex ML systems effectively - Hand-tuned models provide excellent starting points - Privacy considerations must be built in from the start - Vertical integration of ML teams can work well - Internal tooling often preferable for large-scale B2B companies - Experimentation and quick iteration are crucial for success ## Future Considerations - Potential integration of large language models while maintaining privacy - Continued focus on differentiating engineering efforts - Balance between building internal tools vs using external services - Evolution of ML infrastructure as industry matures - Maintaining privacy standards while expanding capabilities