This case study explores how Propel is developing and implementing an AI-powered system to help recipients of SNAP (Supplemental Nutrition Assistance Program) benefits better understand official notices they receive from government agencies. The project represents a careful and thoughtful approach to deploying LLMs in a high-stakes environment where user outcomes directly affect access to essential benefits. # Context and Problem Space SNAP notices are official government communications that inform beneficiaries about important changes or requirements related to their benefits. These notices are often confusing and filled with legal language that can be difficult for recipients to understand. This leads to several problems: * Recipients may miss important deadlines or requirements * Benefits may be unnecessarily lost or reduced due to misunderstandings * State agencies face increased call volumes from confused recipients * Staff time is consumed explaining notices rather than processing applications # Technical Implementation Propel's solution leverages several key LLMOps components: * Primary Model: Anthropic's Claude 3.5 Sonnet * Development Framework: Streamlit for rapid prototyping and iteration * Carefully engineered prompts that frame the AI as a legal aid attorney specializing in SNAP benefits * Two-part structured output focusing on: - Importance assessment (High/Medium/Low) - Clear action items in simple language The system is designed to process both the notice content and specific user questions about notices. The implementation includes several technical safeguards: * Strict prompt engineering to ensure responses are grounded in the actual notice content * Potential implementation of local redaction models (like Microsoft's Presidio) to handle PII * Consideration of additional verification layers to catch potential errors or policy violations # Production Safety Considerations Propel has implemented a robust safety framework for this sensitive use case: * Initial testing phase limited to expert review rather than direct user access * Focus on processing existing notice content rather than generating novel responses to reduce hallucination risks * Careful consideration of information filtering to balance cognitive load with comprehensive coverage * PII handling protocols to protect sensitive user information * Awareness of and mitigation strategies for incorrect source notices # Deployment Strategy The deployment approach shows careful consideration of the high-stakes nature of benefits administration: * Phased rollout starting with expert review * Collection of real-world examples from social media to test edge cases * Plans for passive background processing of notices in future iterations * Integration with broader SNAP navigation assistance tools # Technical Challenges and Solutions Several key technical challenges were addressed: * Managing External Context: Balancing the need to provide additional helpful information while maintaining accuracy * Information Filtering: Developing systems to highlight critical information without omitting legally required details * Privacy Protection: Implementing PII handling protocols while maintaining functionality * Error Detection: Building systems to identify potentially incorrect notices # Future Development Plans The case study outlines several areas for future development: * Integration of external contextual information (such as known issues with phone systems) * Development of background processing capabilities for passive notice monitoring * Expansion into broader SNAP navigation assistance * Enhanced verification and safety systems # Results and Impact While still in development, initial results show promise: * Successful interpretation of complex notices into clear, actionable guidance * Effective handling of specific user questions about notices * Positive feedback from initial expert review * Potential for significant reduction in unnecessary agency calls and benefit losses # Lessons Learned Key takeaways from this implementation include: * The importance of domain expertise in prompt engineering * Benefits of a cautious, phased deployment approach for sensitive applications * Value of real-world testing data in development * Need for robust safety protocols when dealing with government benefits This case study demonstrates a thoughtful approach to implementing LLMs in a high-stakes government services context, with careful attention to both technical implementation and user safety. The project shows how AI can be leveraged to improve government service delivery while maintaining appropriate safeguards for vulnerable populations.