Wealthsimple, a Canadian financial technology company, has implemented a comprehensive approach to leveraging Large Language Models (LLMs) in production while maintaining strong security and privacy controls. Their journey with LLMs focuses on three main streams: employee productivity, operational optimization, and platform enablement. ## Core Philosophy and Architecture The company's LLM implementation philosophy centers around three key themes: * Accessibility * Security * Optionality across different foundation models The centerpiece of their LLM infrastructure is their LLM Gateway, an internal tool that acts as an intermediary between users and various LLM providers (both external and self-hosted). The gateway was developed in response to early concerns about inadvertent data sharing with external LLM providers like OpenAI. Key features of the LLM Gateway include: * PII (Personally Identifiable Information) redaction using an in-house developed model * Support for multiple LLM providers including OpenAI, Cohere, and Google's Gemini * Self-hosted open-source models deployment * Conversation checkpointing functionality allowing blended conversations across different models * Multimodal input support, particularly through Gemini models with their large context windows ## Security and Privacy Measures The company has implemented several security measures to protect sensitive data: * Custom PII redaction model that processes all inputs before sending to external LLM providers * Self-hosted models within their cloud environment for handling sensitive data without redaction * Different security tiers for different cloud environments (AWS vs GCP) * Role-based access control for knowledge bases ## Knowledge Management and RAG Implementation They've developed a tool called "Booster Pack" that implements Retrieval Augmented Generation (RAG) with three types of knowledge bases: * Public - accessible to all employees, including source code, help articles, and financial newsletters * Private - personal document storage and querying * Limited - shared with specific roles and working groups The system includes nightly jobs that update public knowledge sources from various internal documentation systems. ## Deployment and Adoption Metrics The platform has seen significant adoption within the organization: * Over 50% of the company actively uses the LLM Gateway * Strongest adoption (about 50%) comes from Research and Development teams * Usage is uniform across tenure and organizational levels * Primary use cases: programming support, content generation, and information retrieval ## Production Use Cases One notable production implementation is in customer experience ticket triaging: * Integration of Whisper for voice transcription of customer calls * Automated classification and routing of support tickets * Enhancement of existing ML-based classification systems with LLM-generated metadata * Self-hosted models for internal processing ## Build vs Buy Philosophy Their approach to building versus buying LLM solutions considers three main factors: * Security and privacy requirements * Time to market and cost considerations * Unique leverage points with proprietary data The company acknowledges that some tools they built initially might not be built today given the maturation of vendor offerings, but the experience gained has been valuable for their LLM operations. ## Challenges and Lessons Learned Key challenges and insights include: * PII redaction sometimes interferes with legitimate use cases * Concerns about LLM reliability, bias, and hallucinations * Need for better awareness and education about LLM capabilities * Integration points matter significantly - tools are most valuable when embedded in existing workflows * Multiple separate tools create confusion; consolidation is important ## Technical Implementation Details The platform is primarily built on AWS, with some components running on Google Cloud Platform. They've managed to streamline their model deployment process, reducing deployment time from two weeks for their first self-hosted model to just 20 minutes for recent deployments. For system reliability, they've implemented: * Fallback mechanisms for model availability * Quota management for external API calls * Monitoring of usage patterns and engagement metrics ## Future Directions The team is focusing on: * Consolidating tools into more integrated experiences * Exploring fine-tuning opportunities with proprietary data * Improving the user experience around PII masking * Further automation of operational workflows * Expanding use cases based on user feedback and business priorities The case study demonstrates a thoughtful approach to implementing LLMs in a heavily regulated industry, balancing security requirements with user needs while maintaining flexibility for future developments.