This case study details Figma's journey in implementing AI-powered search capabilities within their design platform, offering valuable insights into the practical challenges and solutions of deploying LLMs in production for visual and semantic search. The project began with an interesting pivot - what started as an AI hackathon project for design autocomplete evolved into a comprehensive search solution after user research revealed more pressing needs. This demonstrates the importance of validating AI use cases against real user problems rather than implementing AI for its own sake. The technical implementation faced several crucial challenges that are common in LLMOps: **Data Processing and Selection** The team had to carefully consider what content to index and search. The infinite canvas nature of Figma's platform presented unique challenges - they couldn't feasibly index everything due to cost and performance constraints. They developed sophisticated heuristics to identify and index only relevant UI designs: * Looking at common UI frame dimensions * Detecting duplicate designs and files to avoid redundant indexing * Implementing a 4-hour delay before indexing to ensure designs were somewhat finalized * Creating "quality signals" to prioritize completed, production-ready designs over experimental work **RAG Implementation** The team utilized Retrieval Augmented Generation (RAG) as a core technology, both for the initial autocomplete prototype and the eventual search implementation. This demonstrates how RAG can be effectively used to ground AI outputs in relevant, existing content. The implementation required careful consideration of how to store and retrieve design examples efficiently. **Quality Evaluation System** One of the most interesting aspects of this case study is their approach to AI quality evaluation: * They built custom evaluation tools using their plugin API * Created a visual grading system for search results * Developed specific test queries based on designer workflows * Implemented different similarity levels for evaluation * Created a continuous feedback loop for model improvement **Infrastructure Scaling** The team had to solve significant infrastructure challenges: * Generating and managing billions of embeddings * Optimizing storage and retrieval systems * Balancing cost considerations with performance requirements * Implementing efficient indexing strategies **Production Deployment Strategy** Their deployment approach followed several key principles that are valuable for any LLMOps implementation: * Focus on existing workflows rather than creating entirely new ones * Rapid iteration with continuous updates to staging environment * Systematic quality checks through custom evaluation tools * Cross-disciplinary collaboration between product, content, engineering, and research teams **User Interface Considerations** The team made several important decisions about how to present AI capabilities to users: * Created a unified interface for both visual and semantic search * Implemented features like 'peek' for quick previews * Added metadata filters and creation information * Designed the interface to support both exploration and execution modes **Testing and Iteration** The deployment process included: * Closed beta testing * Continuous refinement using their eval plugin * Regular monitoring of search result accuracy * Iterative improvements based on user feedback **Cost Management** The case study highlights important considerations around managing costs in production AI systems: * Selective indexing to reduce storage and processing needs * Efficient embedding generation and storage * Careful balance between coverage and cost * Strategic decisions about what content to make searchable **Future Considerations** The team is planning several expansions: * Extending visual and semantic search to the Figma Community * Potential return to the original autocomplete feature * Continued refinement of search quality and performance This case study provides valuable insights into the practical challenges of implementing AI features in a production environment, particularly in the context of visual design tools. It demonstrates the importance of starting with user needs, building robust evaluation systems, and carefully managing infrastructure and costs. The team's approach to quality assessment and their focus on solving real workflow problems rather than just implementing AI features provides a good model for other organizations implementing LLMOps.