# LLM Agents for User Story Quality Enhancement at Austrian Post Group IT ## Overview Austrian Post Group IT implemented an innovative LLM-based system to enhance user story quality in their agile development processes. The company developed a reference model for LLM-based agents and implemented it through an Autonomous LLM-based Agent System (ALAS) to improve the quality of user stories in their mobile delivery project. ## System Architecture and Implementation ### Reference Model Components - Task - Initiates interaction and defines work scope/objectives - Agents - LLM instances with specific role profiles - Shared Knowledge Base - Repository containing task information and conversation history - Response - Output generated by agents based on task description ### ALAS Implementation Details - Two-phase implementation approach: - Utilized multiple prompt engineering techniques: ### Agent Profiles and Roles - Agent PO (Product Owner) - Agent RE (Requirements Engineer) ### Technical Implementation - Models used: - Token management considerations for long conversations - Prompt optimization for context retention ## Production Deployment and Evaluation ### Experimental Setup - Tested with 25 synthetic user stories - Mobile delivery application context - Comprehensive task context including: - Structured interaction flow between agents ### Quality Assessment Framework - Based on INVEST criteria - Seven key evaluation metrics: ### Evaluation Results - Positive outcomes: - Challenges identified: ## Production Challenges and Solutions ### Technical Challenges - Token limit management in extended conversations - Model parameter tuning for creativity vs accuracy - Context retention across agent interactions - Integration with existing agile workflows ### Implementation Solutions - Iterative prompt optimization - Temperature parameter tuning for controlled creativity - Manual validation steps by Product Owner - Integration of domain expertise in prompt preparation ### Quality Control Measures - Expert review of generated content - Validation against project context - Acceptance criteria verification - Business value alignment checks ## Future Improvements and Recommendations ### System Enhancement Opportunities - Integration of additional specialized agents: - Parameter optimization for reduced hallucination - Enhanced context awareness capabilities ### Best Practices Identified - Regular prompt refinement based on feedback - Balance between automation and human oversight - Clear role definition for agents - Structured evaluation frameworks ## Operational Considerations ### Production Deployment Guidelines - Careful prompt preparation with domain expert input - Regular validation of generated content - Integration with existing agile processes - Clear communication of system capabilities and limitations ### Monitoring and Maintenance - Performance tracking against quality metrics - Regular review of agent interactions - Prompt updates based on project evolution - Continuous improvement based on team feedback ## Impact and Results ### Measurable Outcomes - Improved user story quality metrics - Enhanced clarity and completeness - Better alignment with business objectives - Streamlined requirements process ### Business Value - Reduced time in user story refinement - Improved communication between stakeholders - Better requirements quality leading to improved development outcomes - Enhanced agile process efficiency