This case study explores how Cato Networks, a leading SASE (Secure Access Service Edge) provider, successfully implemented a natural language search capability in their production environment using Large Language Models (LLMs). The implementation showcases a practical approach to deploying LLMs in a business-critical application while ensuring reliability, accuracy, and performance. The Challenge and Business Context: Cato Networks operates a SASE management console where users need to analyze potentially millions of events through their events page. Previously, users had to manually construct queries using specific filters, requiring deep product knowledge and considerable time investment. This created a significant barrier, especially for new users unfamiliar with the product's terminology and structure. Technical Solution Architecture: The solution leverages Amazon Bedrock as the core LLM infrastructure, integrated with several key components: * A Natural Language Search (NLS) service deployed on Amazon EKS * A GraphQL server backend * The main management console frontend * JSON schema validation layer The technical implementation follows a well-thought-out approach to ensure reliability and accuracy: Prompt Engineering Strategy: Instead of directly generating GraphQL queries, the team implemented a two-step approach: * First, the LLM converts natural language to a well-defined JSON format * Then, this JSON is validated against a schema before being converted to GraphQL This approach provides better control and validation compared to direct query generation. The prompt design includes: * Comprehensive schema definitions * Available keys, operators, and values * Specific instructions about numeric operators and other constraints * Context about current time for temporal queries Quality Assurance and Validation: The team implemented a robust validation strategy: * Created a benchmark with hundreds of test queries and expected outputs * Defined clear success criteria: * Valid JSON syntax * Schema compliance * Full match of filters * Established an error rate threshold of 0.05 as a release criterion * Implemented different policies for handling various types of failures Model Selection and Optimization: After evaluating several foundation models available through Amazon Bedrock, they selected anthropic.claude-3-5-sonnet-20241022-v2:0 based on: * Meeting the error rate criterion * Achieving the highest success rate * Maintaining reasonable costs * Acceptable latency characteristics Production Implementation: The production system handles queries through a sophisticated pipeline: * User inputs are processed through the NLS service * The service constructs complete prompts with necessary context * Responses are validated against the JSON schema * Valid responses are transformed into GraphQL queries * Results are returned to the user interface Error Handling and Resilience: The system implements graceful degradation: * Missing required fields trigger translation failure notifications * Invalid filter values are handled by removing problematic filters while maintaining valid ones * Users receive appropriate feedback about partial translations Multilingual Support: A notable advantage of the chosen foundation model is its native support for multiple languages, making the system accessible to non-English speaking users without requiring additional development effort. Performance and Business Impact: The implementation has shown significant positive results: * Query time reduced from minutes to near-instant * Near-zero time to value for new users * Positive customer feedback, especially from new users * Improved accessibility for non-native English speakers * Higher user satisfaction and reduced learning curve Lessons and Best Practices: The case study reveals several important lessons for LLMOps implementations: * Using JSON schema validation as an intermediate step provides better control and reliability * Prompt engineering alone can be sufficient for certain use cases, though alternatives like fine-tuning might be better for complex schemas * Setting clear performance criteria (error rates, success rates) helps guide model selection * Building robust validation and error handling is crucial for production deployments Future Considerations: The team noted that for more complex JSON schemas, alternative approaches might be worth considering: * Fine-tuning models to reduce prompt size and costs * Exploring more efficient ways to embed product knowledge * Potential for expanding the natural language interface to other parts of the system This implementation demonstrates a practical, production-ready approach to integrating LLMs into an enterprise application. The focus on validation, error handling, and clear success criteria shows mature LLMOps practices, while the achieved business results validate the approach taken.