# ChatLTV: An AI Teaching Assistant Implementation at Harvard Business School ## Project Overview Harvard Business School implemented an AI teaching assistant called ChatLTV for their Launching Tech Ventures (LTV) course, serving approximately 250 MBA students. The system was designed to help students prepare for classes, understand course materials, and handle administrative queries, while providing faculty with insights into student learning patterns. ## Technical Architecture and Implementation ### Core Components - Azure OpenAI Service for LLM capabilities - Pinecone for vector database storage - Langchain as middleware - Slack for user interface - Custom CMS for content management ### Development Scope - Backend: 8000 lines of code - CMS: 9000 lines of code ### Data Sources and Training - Course corpus of ~200 documents containing 15 million words - Content types included: ## Security and Compliance Considerations - Utilized Azure OpenAI Service instead of direct OpenAI APIs - Implemented data privacy controls to protect copyrighted content - Leveraged SOC2 Type II compliant Pinecone database - Chunked content delivery to minimize exposure - Plans for migration to Harvard's private LLM ## Testing and Quality Assurance - Created 500 Q&A test queries - Implemented dual evaluation approach: - Iterative prompt engineering process - Content indexing optimization ## Production Implementation ### User Interface - Slack-based chatbot integration - Support for both private and public queries - Source attribution for answers - Parallel access to public ChatGPT ### Administrative Features - Custom CMS for content management - Query monitoring capabilities - Usage analytics - Student interaction tracking ## Results and Impact ### Usage Statistics - 170 active users (>50% adoption) - Over 3000 queries during the semester - ~130 queries per case study - 99% private query preference - 40% of users rated quality 4-5 out of 5 ### Benefits - Enhanced student preparation - Improved class discussions - Administrative query handling - Faculty insights into student learning - Time-efficient case preparation ### Faculty Benefits - Insight into student comprehension - Ability to identify struggling students - Better cold-calling decisions - Understanding of student preparation patterns - Administrative workload reduction ## Technical Evolution ### Custom GPT Implementation - Created "HBS LTV Feedback" custom GPT - Zero-code implementation - Trained for academic evaluation - Used for final project feedback - Rapid deployment (2-hour setup) ## Challenges and Solutions - Copyright protection for course materials - Query quality consistency - Administrative content handling - Student privacy concerns ## Best Practices Identified - Comprehensive testing before deployment - Regular prompt optimization - Source attribution for answers - Privacy-first design approach - Integration with existing workflows - Content management system importance - Multiple evaluation methods ## Future Considerations - Migration to Harvard's private LLM - Expansion to other courses - Enhanced feedback mechanisms - Additional custom GPT applications - Improved administrative capabilities The implementation demonstrates a successful integration of LLMOps practices in an educational setting, combining careful attention to security, usability, and pedagogical effectiveness while maintaining technical robustness and scalability.