WVU Medicine represents a comprehensive health system spanning 25 hospitals across multiple regions, including North Central West Virginia, Southern Pennsylvania, Western Ohio, and Eastern Maryland. Their implementation of NLP technology for automated HCC code extraction showcases a practical application of AI in healthcare operations, demonstrating both the opportunities and challenges of deploying language models in a clinical setting. ## System Overview and Business Context The primary challenge addressed by this implementation was the burden of manual HCC coding on healthcare providers. HCC coding is crucial for both patient care and regulatory compliance, as it helps predict future healthcare needs and ensures appropriate funding for patient care. However, the process of evaluating chronic conditions and documenting relevant HCC codes from unstructured clinical notes is time-consuming and prone to missing codes despite providers' best efforts. The AI team at WVU Medicine leveraged John Snow Labs' Healthcare NLP module to build an automated system that could: * Extract diagnoses from clinical notes * Convert these diagnoses to CPT codes * Map CPT codes to HCC codes * Present these codes to providers through their EMR system ## Technical Implementation The system architecture demonstrates several key LLMOps considerations: ### Data Processing Pipeline The solution implements a sophisticated workflow that begins with querying their EMR database for upcoming patient appointments (14-day window) and retrieving associated radiology notes. This approach ensures timely and relevant data processing while managing computational resources effectively. ### Model Customization and Deployment The team customized the John Snow Labs' Healthcare NLP model to better suit their specific needs by: * Refining the token extraction process * Removing irrelevant details like family history and suggestive comments * Implementing filters for codes not applicable to their patient population * Incorporating confidence scoring to improve accuracy ### Confidence Scoring and Validation The system includes a confidence scoring mechanism where: * Scores range from 0 to 1 * Higher scores indicate greater confidence in the diagnosis extraction * This scoring system is used to filter and prioritize extracted codes ### Integration with Production Systems The implementation shows careful consideration of production requirements: * Automated loading of extracted codes into their EMR system * Real-time best practice alerts during patient visits * Provider feedback mechanism with three options: * Accept the code * Reject the code * Mark the code as invalid * Tracking of provider responses for future validation and audit ### Performance Monitoring and Results Since going live in December 2023, the system has demonstrated significant impact: * Processed over 27,000 HCC codes * Achieved 18.4% provider acceptance rate * Positively impacted more than 5,000 patients * Complete processing runtime under one hour ### Document Tracking and Efficiency To optimize performance, the system includes: * Record keeping of scanned documents * Avoidance of duplicate scanning * Efficient runtime management ## Challenges and Lessons Learned The implementation revealed several important considerations for healthcare LLMOps: ### Model Updates and Infrastructure The team identified challenges in their current model update process: * Complex procedures for upgrading to new versions of the John Snow Labs model * Working on infrastructure improvements to simplify model updates to a single line code change ### Validation and Partnership The case study emphasizes the importance of: * Strong operational business partners * Thorough clinical data validation * Meticulous workflow evaluation * Leveraging vendor expertise for smaller AI teams ## Future Development The team has outlined several future improvements: * Development of version 2 based on the latest John Snow Labs Healthcare NLP model * Expansion beyond radiology documents to include progress notes * Infrastructure improvements for easier model updates * Exploration of additional use cases like document classification and incidental finding identification ## Key LLMOps Takeaways This implementation demonstrates several crucial LLMOps best practices: * Careful consideration of production system integration * Implementation of feedback loops for model improvement * Balance between automation and human oversight * Importance of confidence scoring and validation * Need for efficient document processing and deduplication * Value of strategic partnerships for smaller AI teams The case study particularly highlights how LLM-based solutions can be effectively deployed in highly regulated environments like healthcare, where accuracy and validation are crucial. The implementation shows a thoughtful balance between automation and human oversight, with clear mechanisms for provider feedback and validation. The success of this project also demonstrates the value of starting with a focused scope (radiology notes) while planning for future expansion. This approach allowed the team to validate their approach and demonstrate value before scaling to additional document types and use cases.