Company
Wix
Title
Dynamic Knowledge and Instruction RAG System for Production Chatbots
Industry
Tech
Year
2024
Summary (short)
Wix developed an innovative approach to enhance their AI Site-Chat system by creating a hybrid framework that combines LLMs with traditional machine learning classifiers. They introduced DDKI-RAG (Dynamic Domain Knowledge and Instruction Retrieval-Augmented Generation), which addresses limitations of traditional RAG systems by enabling real-time learning and adaptability based on site owner feedback. The system uses a novel classification approach combining LLMs for feature extraction with CatBoost for final classification, allowing chatbots to continuously improve their responses and incorporate unwritten domain knowledge.
Wix's implementation of AI Site-Chat represents a sophisticated approach to deploying LLMs in production, specifically focusing on creating more adaptive and customizable chatbot systems. This case study showcases several important aspects of LLMOps and provides valuable insights into handling LLMs at scale. The core challenge Wix aimed to address was the limitation of traditional chatbot systems in incorporating business-specific knowledge and preferences that aren't explicitly documented. Their solution introduces two key innovative concepts that are particularly relevant to LLMOps practitioners: ## Hybrid Classification System The first major component is their hybrid classification approach, which cleverly combines the strengths of LLMs with traditional machine learning. This design choice demonstrates important LLMOps considerations: * Instead of relying solely on LLMs for classification, which can be expensive and prone to hallucination, they use LLMs to extract features through targeted yes/no questions * These features then feed into a CatBoost classifier, providing more reliable and interpretable results * This approach addresses several common LLM production challenges: * Cost optimization by using smaller, focused prompts that can run in parallel * Improved reliability by combining LLM capabilities with traditional ML * Better interpretability through feature importance analysis * Reduced prompt fatigue by breaking down complex tasks into smaller components ## Dynamic Knowledge Management (DDKI-RAG) The second major component is their Dynamic Domain Knowledge and Instruction RAG (DDKI-RAG) system, which represents an evolution of traditional RAG approaches. Key LLMOps innovations include: * Real-time knowledge base updates without requiring full reindexing * Dynamic prompt modification based on feedback * Separation of knowledge types into regular documents, knowledge documents, and instruction documents * Vector database integration for efficient retrieval The system's architecture demonstrates several production-ready features: * Feedback Loop Integration: * Captures site owner feedback * Classifies feedback type * Generates appropriate knowledge or instruction documents * Updates vector embeddings in real-time * Document Processing Pipeline: * Clear separation between indexing and inference modes * Structured approach to document creation and classification * Efficient vector comparison for retrieval * Prompt Management: * Dynamic prompt modification capabilities * Template-based and additive modification options * Context-aware prompt adjustments From an LLMOps perspective, the system addresses several critical production concerns: * Scalability: The hybrid approach allows for efficient processing of queries by breaking down complex tasks into manageable components * Maintainability: Clear separation of concerns between classification, knowledge management, and prompt handling * Monitoring: The feedback system provides built-in monitoring of system performance and accuracy * Adaptation: Continuous learning capabilities through the feedback loop * Cost Management: Efficient use of LLM resources through targeted prompting and hybrid architecture The implementation also shows careful consideration of error handling and edge cases: * Classification includes a "don't know" option to handle uncertainty * Multiple document types allow for flexible knowledge representation * Structured approach to prompt modifications reduces the risk of prompt injection or corruption Best Practices Demonstrated: * Separation of concerns between different system components * Use of vector databases for efficient similarity search * Hybrid architecture combining multiple AI/ML approaches * Structured feedback incorporation * Dynamic knowledge base updates * Prompt management system From a technical implementation standpoint, the system shows sophisticated use of: * Embedding systems for document vectorization * Vector databases for similarity search * Boosting algorithms (CatBoost) for classification * Template-based prompt management * Document processing pipelines The case study also reveals important considerations about LLM system design: * The importance of balancing automation with human feedback * The value of hybrid approaches combining different AI/ML techniques * The need for flexible, adaptable knowledge management systems * The importance of efficient prompt management in production systems One particularly noteworthy aspect is how the system handles the challenge of incorporating "unwritten knowledge" - information that exists in business owners' minds but isn't documented anywhere. This is a common challenge in real-world LLM applications, and Wix's solution provides a structured way to capture and utilize this knowledge. The results, while not explicitly quantified in the case study, suggest significant improvements in chatbot performance and adaptability. The system's ability to continuously learn and adapt based on feedback, while maintaining efficient operation, demonstrates a mature approach to LLMOps that could be valuable for other organizations deploying LLMs in production environments.

Start your new ML Project today with ZenML Pro

Join 1,000s of members already deploying models with ZenML.