This case study from Microsoft details the implementation of a production-grade RAG (Retrieval Augmented Generation) system designed to handle complex financial documents, including analyst reports and SEC filings. The speakers, who are development managers at Microsoft, share their experiences and challenges in building a real-world RAG implementation that goes beyond the typical "simple RAG in minutes" demonstrations. The system needed to handle a massive corpus of financial documents - tens of thousands of documents, with each document containing hundreds of pages. These documents included various complex elements like tables, graphs, charts, and financial data spanning different time periods and fiscal years. Key Technical Challenges and Solutions: **Context Window and RAG Necessity** The team first addresses why RAG was necessary despite increasing context windows in modern LLMs. While models like GPT-4 can handle 18K tokens and Claude can process up to 2M tokens, simply stuffing all documents into the context window isn't practical due to: * Cost considerations - sending full documents with every query would be expensive * Latency impact - processing large amounts of text affects response time * Loss of accuracy due to position bias - research shows that model performance degrades based on where information appears in the context window **Metadata Handling** One of the major challenges was handling document metadata correctly. Financial documents often require precise temporal context - knowing exactly which fiscal quarter or year a particular piece of information relates to. The team discovered that: * Much of the crucial metadata appears in the first few pages of documents * They needed to handle both calendar years and fiscal years (which vary by company) * Simple metadata extraction from file properties was often unreliable * They developed specialized prompt engineering approaches to handle fiscal year conversions and temporal reasoning **Chart and Graph Analysis** The team implemented sophisticated handling of visual elements: * They used a multi-modal model to analyze charts, graphs, and tables * Developed a basic image classifier to determine which images contained relevant financial information * Created structured JSON representations of visual data to make it searchable * Balanced the cost and latency implications of multi-modal processing by being selective about which images to analyze **Document Processing Pipeline** The team built a sophisticated document processing pipeline that includes: * Document parsing using Azure Document Intelligence * Specialized handling for different element types (text, tables, charts) * Metadata extraction and enrichment * Efficient storage and indexing of processed content **Evaluation-Driven Development** Perhaps most importantly, the team developed a comprehensive evaluation methodology: * They created detailed evaluation datasets * Implemented nuanced scoring that goes beyond simple binary correct/incorrect * Developed specialized evaluation for numerical answers, considering business context * Created automated evaluation pipelines for continuous testing * Built separate evaluation sets for different document types and query categories Key Learnings and Best Practices: * The importance of handling edge cases and special conditions in production systems * The need for robust evaluation frameworks before making system changes * The value of breaking down answers into factual components for better accuracy assessment * The importance of domain-specific considerations (like fiscal vs. calendar years) * The need to balance perfect accuracy with practical approximations (like handling numerical precision appropriately) The case study emphasizes that building production RAG systems is significantly more complex than tutorial examples suggest. Success requires careful attention to: * Document processing and metadata extraction * Handling of multi-modal content * Robust evaluation frameworks * Domain-specific knowledge integration * Performance and cost optimization The team's approach of evaluation-driven development provides a valuable framework for others building similar systems. Their experience shows that while basic RAG implementations might be simple, building production-grade systems requires sophisticated handling of edge cases, robust evaluation frameworks, and careful attention to domain-specific requirements.