This case study provides an in-depth examination of the practical challenges and solutions in implementing Retrieval Augmented Generation (RAG) systems in production environments, based on the experiences of Prolego's engineering team. The discussion brings together multiple engineers to share their real-world insights, moving beyond the typical simplified tutorial examples found online. The discussion begins with a practical example of using RAG for employee policy lookup, demonstrating how RAG can enhance LLM responses by providing relevant context from company documents. This example shows how a general query about bringing a dog to work can be augmented with specific company policy information to provide accurate, contextualized responses. Several key technical challenges and considerations are explored in detail: **Document Processing and Chunking Challenges** * The team emphasizes that one of the most significant challenges is the initial document processing phase, which is often oversimplified in tutorials * Real-world documents come in various formats (PDFs, Word docs, SharePoint sites, PowerPoint slides) and contain complex structures including tables * Maintaining document hierarchy is crucial as context can be spread across different sections * The team discusses the importance of preserving structural relationships, such as when rules have exceptions or conditions that appear in different sections **Chunking Strategies and Implementation** * Finding the optimal chunk size is critical - too small (like single sentences) loses context, too large makes embeddings less effective * The team recommends using overlapping windows to ensure context isn't lost between chunks * Implementation often requires flattening hierarchical structures while maintaining references to the original hierarchy * They suggest using unique identifiers that combine file names, chapter names, and section information to maintain traceability **Retrieval and Embedding Considerations** * Context window limitations of LLMs necessitate careful selection of relevant information * The team discusses the importance of finding the right balance in embedding chunk sizes * Performance trade-offs must be considered when designing the retrieval system * They emphasize that the retrieval step is often the most critical part of a RAG system **Evaluation Challenges** * The team discusses the complexity of evaluating RAG systems, comparing it to grading an essay * They introduce the concept of "faithfulness" - ensuring responses are based on retrieved information rather than the LLM's general knowledge * Multiple evaluation approaches are discussed: * Automated metrics vs. human evaluation * The challenge of evaluating responses beyond simple multiple-choice scenarios * The importance of testing with diverse query formulations **Performance Improvement Techniques** * Various approaches to improve RAG performance are discussed: * Using metadata and rules alongside embeddings * Implementing document summarization techniques * Query expansion through LLM-based rephrasing * Fine-tuning embedding models or using adapters * Specialized summarization techniques that preserve key entities **Production Considerations** * The team emphasizes the importance of handling edge cases and user behavior * They discuss practical solutions like implementing reasonableness checks * The importance of domain-specific optimization is highlighted, noting that different techniques may work better for different industries or use cases The discussion concludes with insights into emerging techniques and ongoing research in the field, such as specialized summarization methods and embedding model fine-tuning. The team emphasizes that RAG implementation is not a one-size-fits-all solution, and success often requires a combination of techniques tailored to specific use cases and document types. This case study provides valuable insights for organizations looking to implement RAG systems in production, highlighting the importance of careful consideration of document processing, chunking strategies, and evaluation methods. It demonstrates that while RAG is a powerful technique, successful implementation requires addressing numerous practical challenges that go beyond the typical tutorial examples.