The North Dakota University System (NDUS) represents a fascinating case study in implementing LLMs in a higher education context, specifically focusing on solving the complex challenge of policy and regulatory document search across a large university system. NDUS manages 11 institutions, including community colleges, regional universities, and research universities, serving approximately 80,000 students, faculty, and staff. ### Initial Challenge and Context The organization faced a significant challenge in managing and searching through thousands of policy documents, state laws, contracts, procedures, and codes. Without a modern data infrastructure, staff members were spending hours manually searching through multiple sources to find relevant policy information, leading to inefficiencies and potential compliance risks. This manual process was particularly problematic given the critical nature of regulatory compliance in higher education. ### Technical Implementation The technical implementation of their LLM solution is particularly interesting from an LLMOps perspective, with several key components: * **Model Selection Process**: The team conducted a systematic evaluation of different open-source LLMs, with their selection criteria focusing on: * Performance as the primary metric * Inference time * Model size * Cost considerations They ultimately selected Llama 2 as their primary model, though they mentioned plans to explore DBRX for future consolidation. * **Infrastructure and Platform**: They leveraged the Databricks Data Intelligence Platform on Microsoft Azure, which provided several advantages: * Existing Azure integration, reducing procurement complexity * Built-in MLOps capabilities through MLflow * Vector Search functionality for automatic data synchronization * Unity Catalog for governance and access control * **Data Pipeline and Processing**: * They implemented a system to process over 3,000 public PDF documents * Created vector embeddings for efficient search * Set up automated synchronization to ensure LLM outputs remain current * Established DLT (Delta Live Tables) pipelines for handling daily data updates ### Production Deployment and Operations The production deployment of their "Policy Assistant" application demonstrates several LLMOps best practices: * **Governance and Security**: They implemented comprehensive governance through Unity Catalog, ensuring: * Unified access controls * Secure collaboration * Appropriate data and model access restrictions * Compliance with educational data handling requirements * **Testing and Validation**: * Used MLflow for local testing * Implemented a systematic approach to running ML and GenAI applications * Started with a low-risk application to prove the concept * **API Integration**: * Developed an API interface allowing users to query the system using natural language * Implemented response generation with proper citations, including page numbers and links * Built automated distribution systems for internal audit reports ### Results and Impact The implementation has shown significant measurable improvements: * Search operations are now 10-20x faster than the previous manual process * Development cycle time reduced from one year to six months * Complete elimination of procurement-related delays by leveraging existing Azure infrastructure * Automated daily report generation and distribution * Enhanced ability to track and predict enrollment patterns ### Scaling and Future Development NDUS has taken a thoughtful approach to scaling their LLM operations: * Regular educational events to promote understanding and adoption * Planned expansion into predictive enrollment forecasting * Development of domain-specific LLMs for specialized use cases * Integration with unstructured news data ### Critical Analysis While the case study presents impressive results, there are several aspects worth analyzing: * **Model Choice Trade-offs**: The selection of Llama 2 represents a balance between performance and cost, but the planned transition to DBRX suggests they may be seeking more optimal solutions. * **Governance Considerations**: The implementation appears to have strong governance controls, which is crucial in an educational setting handling sensitive data. * **Scalability Approach**: Their phased approach, starting with a low-risk application before expanding to more critical functions, demonstrates good LLMOps practices. * **Integration Strategy**: The use of existing cloud infrastructure (Azure) and Databricks platform shows a practical approach to rapid deployment while maintaining security and compliance. This case study highlights the importance of careful planning, systematic evaluation of models, and strong governance in implementing LLMs in production, particularly in regulated environments like higher education. The success of this implementation suggests that similar approaches could be valuable for other educational institutions facing comparable challenges with policy and regulatory document management.