This case study presents a detailed exploration of Retrieval Augmented Generation (RAG) systems in production, specifically focusing on their limitations when handling complex, context-dependent documents. The research, conducted through OpenGPA's implementation, provides valuable insights into the practical challenges of deploying RAG systems in real-world applications. The study begins with a fundamental implementation of RAG in OpenGPA, which follows the standard architecture of document chunking, embedding generation, and vector search. This implementation proves effective for straightforward documents containing independent facts or definitions, demonstrating the basic utility of RAG in production environments. The system employs vector-based similarity search, which offers advantages over traditional keyword-based approaches by capturing semantic relationships between concepts. However, the core of the case study reveals significant limitations in current RAG implementations when dealing with temporal and context-dependent documents, specifically exemplified through movie scripts. The research identifies two primary challenges in production RAG systems: Context Management Limitations: * The current chunk-based approach fails to maintain temporal context across document segments * Chunks are processed in isolation, losing critical contextual information needed for accurate question answering * Simple questions requiring temporal awareness (like identifying a character's dog's name in a specific year) cannot be accurately answered due to context loss * The system struggles to differentiate between similar entities appearing in different temporal contexts Relationship Processing Challenges: * The system fails to capture indirect relationships between entities * Complex queries requiring understanding of implicit connections (like tracking character movements across locations) cannot be properly processed * The current implementation struggles with questions requiring aggregation of information across multiple contexts The case study then explores potential solutions, including the implementation of Graph RAG, a more sophisticated approach developed by Microsoft Research. This implementation uses: * LLM-powered entity and relationship extraction * Graph database (Neo4J) integration for relationship modeling * Combined vector and graph-based search strategies While Graph RAG shows promise in capturing entity relationships, the study reveals that even this advanced approach falls short in handling complex temporal contexts. This leads to a broader discussion of potential improvements needed in production RAG systems: Proposed Enhancements for Production Systems: * Development of context-aware chunking strategies * Implementation of hierarchical context management * Integration of document-specific context models (temporal, geographical, legal, etc.) * Enhanced methods for context summarization and propagation The case study is particularly valuable for practitioners as it proposes a novel benchmark for evaluating RAG systems using movie scripts and trivia questions. This benchmark would test: * Temporal context handling * Entity relationship tracking * Cross-reference capability * Context-dependent information retrieval From an LLMOps perspective, the study highlights several critical considerations for deploying RAG systems in production: 1. The importance of thorough system evaluation using domain-specific test cases 2. The need for careful consideration of document structure and context requirements 3. The potential benefits and limitations of different RAG architectures 4. The role of specialized databases and data structures in supporting RAG implementations The research emphasizes that successful deployment of RAG systems in production requires more than just implementing the basic RAG architecture. It needs careful consideration of: * Document characteristics and context requirements * Appropriate database selection and configuration * Context management strategies * System evaluation and testing approaches The study concludes by suggesting that future RAG implementations need to move beyond simple chunk-based approaches to more sophisticated context management systems. This might include: * Development of context-aware chunking strategies * Implementation of multi-level context hierarchies * Integration of specialized context models for different document types * Enhanced methods for context summarization and propagation This case study provides valuable insights for organizations looking to implement RAG systems in production, highlighting both the current limitations and potential future directions for improvement. It demonstrates the importance of thorough testing and evaluation of RAG systems, particularly when dealing with complex, context-dependent documents. The proposed movie script benchmark could serve as a valuable tool for evaluating and improving RAG systems in production environments.