# Building Secure Enterprise LLM Infrastructure at Slack ## Company Overview and Challenge Slack, a leading enterprise communication platform, faced the challenge of implementing AI features while maintaining their strict security standards and privacy requirements. As a FedRAMP Moderate authorized platform, they needed to ensure that their AI implementation would meet rigorous compliance requirements while delivering value to their users. ## Technical Architecture and Implementation ### Core Principles The implementation was guided by four key principles: - Customer data must never leave Slack - No training of LLMs on customer data - AI operations limited to user-accessible data only - Maintenance of enterprise-grade security and compliance ### Infrastructure Setup - Utilized AWS SageMaker to host closed-source LLMs in an escrow VPC - Implemented complete control over customer data lifecycle - Prevented model providers from accessing customer data - Maintained data within Slack-controlled AWS VPCs ### Model Strategy and RAG Implementation - Chose to use off-the-shelf models instead of fine-tuning - Implemented Retrieval Augmented Generation (RAG) for stateless operation - Selected models based on context window size and latency requirements - Combined traditional ML models with generative models for improved results ### Security and Access Control Features - Built on existing Slack security infrastructure - Implemented strict access control using user ACLs - AI-generated outputs visible only to the requesting user - Integrated with existing compliance features: ### Data Privacy Measures - Ephemeral storage for AI outputs where possible - Minimal data storage following least-data principle - Integration with existing tombstoning mechanisms - Automatic invalidation of derived content when source content is removed ## Technical Challenges and Solutions ### Model Selection Challenges - Required large context windows for processing channel data - Needed to balance latency with processing capability - Evaluated multiple models for summarization and search use cases ### Integration Challenges - Built on top of existing Slack feature sets - Reused existing security libraries and infrastructure - Created new compliance support for derived content - Implemented special handling for DLP and administrative controls ### Data Processing - Implemented context-aware processing for channel summarization - Built secure data fetching mechanisms using existing ACLs - Developed systems for handling spiky demand - Created evaluation frameworks for model performance ## Implementation Results and Impact ### Security Achievements - Successfully maintained FedRAMP Moderate authorization - Prevented customer data leakage outside trust boundary - Implemented comprehensive compliance controls - Maintained existing security standards ### Performance Outcomes - 90% of AI feature adopters reported higher productivity - Successfully addressed key user pain points: ### Architecture Benefits - Scalable and secure AI infrastructure - Maintenance of data privacy guarantees - Enterprise-grade security compliance - Flexible foundation for future AI features ## Technical Best Practices and Learnings ### Security First Approach - Start with security requirements first - Build on existing security infrastructure - Implement strict data access controls - Maintain compliance requirements throughout ### Architecture Decisions - Use of RAG over fine-tuning for privacy - Hosting models in controlled environments - Implementation of stateless processing - Integration with existing security frameworks ### Data Handling - Implement strict data lifecycle management - Use ephemeral storage where possible - Maintain detailed data lineage - Integrate with existing data protection mechanisms ### Future Considerations - Monitoring and evaluation of model performance - Handling increasing context window sizes - Managing prompt optimization - Scaling for demand spikes ## Infrastructure and Tooling ### AWS Integration - SageMaker for model hosting - VPC configuration for data security - Integration with existing AWS infrastructure - Compliance maintenance in cloud environment ### Security Tools - Access Control Lists (ACLs) - Data Loss Protection (DLP) - Encryption key management - Data residency controls ### Development Framework - Built on existing Slack infrastructure - Integration with core services - Reuse of security libraries - Custom compliance tooling This case study demonstrates how enterprise-grade AI features can be implemented while maintaining strict security and compliance requirements. Slack's approach shows that with careful architecture and implementation choices, organizations can leverage powerful AI capabilities while protecting sensitive customer data.