This case study examines how Philadelphia Union, a Major League Soccer (MLS) team, implemented a production-grade GenAI system to streamline their roster management operations. The implementation showcases several important aspects of LLMOps and provides valuable insights into deploying LLMs in a real-world business context. The core business challenge revolved around the complexity of MLS Roster Composition Rules and Regulations, which were extensive and filled with legalistic details that slowed down decision-making processes. The team needed a solution that could provide quick, accurate interpretations of these rules while maintaining compliance and enabling faster strategic decisions. ## Technical Architecture and Implementation The solution is built on a Retrieval-Augmented Generation (RAG) architecture, with several key LLMOps components working together: * **Data Pipeline and Vector Storage**: The system includes a continuous ingestion mechanism for roster rule PDFs into Databricks Volumes, ensuring new documents are automatically processed. The text extraction pipeline generates embeddings using the Databricks Foundation Model API, which are then indexed in Vector Search for efficient retrieval. * **Model Selection and Deployment**: The team chose DBRX Instruct, an open-source LLM based on a Mixture of Experts (MoE) architecture, accessed through the Databricks Foundation Model API. This choice eliminated the need for self-hosting model infrastructure while providing strong performance on benchmarks like MMLU. * **Application Architecture**: The RAG chatbot is deployed using the Mosaic AI Agent Framework, which orchestrates the various components into a production-ready chain hosted on a Databricks Model Serving endpoint. The user interface is implemented through Databricks Apps, providing a ChatGPT-like experience. ## LLMOps Best Practices The implementation demonstrates several important LLMOps best practices: ### Testing and Evaluation The team implemented a robust evaluation process using the Mosaic AI Agent Framework's built-in Evaluations feature. This included: * Collection of human feedback through a review app * Validation of RAG solution effectiveness before deployment * Continuous quality assessment of responses ### Monitoring and Governance The solution incorporates several governance and monitoring features: * Unity Catalog integration for secure data handling * Trace logging for response tracking * Feedback capture mechanisms * Performance monitoring ### Development and Deployment Workflow The team established an efficient LLMOps workflow that included: * Fast iteration cycles for model development * Seamless testing procedures * MLflow integration for experiment tracking * Streamlined deployment processes ## Production Considerations Several key production aspects were addressed in the implementation: ### Security and Compliance * Implementation of enterprise governance standards * Secure handling of sensitive player and roster information * Compliance with sports industry regulations ### Scalability and Performance * Efficient processing of large volumes of data * Ability to handle historical trends and future scenarios * Real-time response capabilities for user queries ### Integration and Accessibility * No-code interface for front office staff * Integration with existing workflows * Easy access through familiar chat-like interface ## Results and Impact The implementation delivered several significant benefits: * **Development Efficiency**: The team achieved rapid time-to-model, developing and deploying the RAG system in just days. * **Operational Improvements**: Automated extraction and analysis of roster rules significantly reduced manual work. * **Decision Support**: The system enables instant interpretation of complex regulations, accelerating decision-making processes. ## Lessons Learned and Best Practices The case study reveals several important lessons for LLMOps implementations: * **Platform Selection**: Using an integrated platform (Databricks) simplified the implementation and reduced technical complexity. * **RAG Architecture**: The choice of RAG architecture proved effective for maintaining accuracy while working with domain-specific content. * **Evaluation First**: The team's focus on testing and evaluation before deployment ensured reliability and user trust. * **Governance Integration**: Building governance and security considerations into the architecture from the start ensured sustainable deployment. ## Future Considerations The implementation creates opportunities for future enhancements: * Expansion to other aspects of sports management * Integration with additional data sources * Enhanced analytics capabilities * Potential for cross-team knowledge sharing This case study demonstrates how careful attention to LLMOps practices can deliver a successful production AI system, even in a specialized domain like sports management. The implementation balances technical sophistication with practical usability, while maintaining necessary governance and security controls.