# Automating Post Incident Review Summaries at Canva with GPT-4 ## Company and Use Case Overview Canva, a leading online graphic design platform serving over 150 million monthly active users across 100+ languages, implemented GPT-4 to automate their Post Incident Review (PIR) summarization process. This case study demonstrates a practical application of LLMs in production for improving operational efficiency in incident management. ## Technical Implementation Details ### Model Selection Process - Evaluated three potential models: - Selection criteria included: - Key factors in choosing GPT-4 chat: ### Production Pipeline - Data Flow: ### Prompt Engineering Strategy - Implemented a structured prompt architecture: - Key prompt components: ### Quality Control and Optimization - Temperature setting of 0 to ensure consistent, factual outputs - Character limit guidance (600 characters) implemented through prompts - Multiple example PIRs to prevent rigid format copying - Monitoring of human modifications to assess quality - Resubmission process for oversized summaries ## Production Results and Metrics ### Performance Indicators - Most AI-generated summaries remained unmodified by engineers - Consistent quality across different types of incidents - Successful removal of sensitive information - Cost-effective implementation ($0.6 maximum per summary) ### Integration Benefits - Improved data quality and consistency - Reduced engineering workload - Enhanced searchability and accessibility of incident information - Successful integration with existing tools (Confluence, Jira) - Comprehensive reporting capabilities through data warehouse integration ## Technical Challenges and Solutions ### Data Processing - Implemented HTML parsing for Confluence content - Developed preprocessing pipeline for sensitive data removal - Created robust error handling for various PIR formats ### Model Interaction - Designed multi-step prompt system with examples - Implemented failsafes for summary length control - Created verification system through Jira integration ### Infrastructure Integration - Established data flow between multiple systems: ## Best Practices and Learnings ### Prompt Engineering - Use of system messages for context setting - Multiple examples for format guidance - Clear constraints and requirements - Blameless approach enforcement ### Quality Assurance - Regular monitoring of human modifications - Comparison of AI-generated vs. human-modified summaries - Integration with existing quality control processes ### System Design - Focus on modularity and scalability - Integration with existing tools and workflows - Emphasis on data security and privacy - Cost-effective implementation strategies ## Production Infrastructure ### Components - Confluence integration for PIR extraction - Text preprocessing pipeline - GPT-4 API integration - Data warehouse storage - Jira webhook integration ### Monitoring and Maintenance - Tracking of summary modifications - Cost monitoring per summary - Quality assessment through human review - System performance monitoring This implementation demonstrates a successful production deployment of LLM technology in an enterprise setting, showing how careful prompt engineering, system design, and integration can create significant operational improvements while maintaining high quality standards.