This case study captures insights from a panel discussion featuring leaders at several prominent companies implementing LLMs in production: Nubank (fintech), Harvey AI (legal tech), Galileo (ML infrastructure), and Convirza (conversation analytics). The discussion provides a comprehensive view of how different organizations are approaching LLMOps challenges and evolving their strategies as the technology matures. The panel revealed several key themes in how organizations are successfully deploying LLMs in production: **Evolution from Prototypes to Production Systems** The discussion highlighted how organizations typically start with larger proprietary models like OpenAI's GPT for prototyping, then optimize and transition to more specialized solutions as they move to production. This progression is driven by needs around cost optimization, latency requirements, and specialized functionality. Nubank, for example, follows this pattern with their customer service applications - starting with proprietary models to validate product-market fit, then optimizing with smaller, specialized models once the use case is proven. **Modularization and System Architecture** A crucial insight shared by multiple panelists was the importance of breaking down complex LLM workflows into smaller, more manageable components. Rather than trying to solve everything with a single large model, successful implementations tend to decompose problems into discrete tasks that can be handled by specialized models or even traditional rule-based systems. Harvey AI emphasized how this modularization helps with evaluation, monitoring, and maintaining high accuracy in their legal applications. **Model Selection and Optimization** The panelists discussed a sophisticated approach to model selection based on multiple criteria: * Quality/accuracy of results * Latency and throughput requirements * Cost considerations * Technical debt implications Rather than focusing solely on the open source vs. proprietary debate, organizations are making pragmatic choices based on these criteria for each specific use case. Nubank shared how they maintain detailed metrics on each component of their system to drive these decisions. **Evaluation and Quality Assurance** The discussion revealed mature approaches to evaluation that go beyond simple accuracy metrics: * Human-in-the-loop feedback is crucial both during development and in production * Organizations are building comprehensive telemetry systems to monitor model performance * Evaluation needs to happen at multiple levels - from individual components to end-to-end workflows * Companies like Galileo are developing specialized smaller models for efficient evaluation **Cost Management** Cost considerations emerged as a major driver of architectural decisions. Organizations are finding that while large proprietary models are excellent for prototyping, they often become cost-prohibitive at scale. This is driving innovation in several areas: * Use of smaller, specialized models * Fine-tuning for specific tasks * Optimizing inference platforms * Careful consideration of where to spend computation budget **Human Feedback and Continuous Improvement** All panelists emphasized the importance of human feedback in their systems: * Harvey AI maintains human oversight in their legal applications * Convirza uses human feedback for model improvement in their conversation analytics * Galileo has built continuous learning from human feedback into their evaluation platform * Nubank maintains internal labeling teams for ongoing model improvement **Future Trends** The panel identified several emerging trends in LLMOps: * Increasing use of smaller, specialized models * Growth in edge deployment of models * More sophisticated inference-time compute strategies * Better infrastructure for model evaluation and monitoring * Integration of multiple models in complex workflows **Practical Lessons** Some key practical takeaways from the discussion: * Start with proven proprietary models for prototyping but plan for optimization * Build strong telemetry and evaluation systems from the beginning * Consider modularizing complex workflows instead of trying to solve everything with one model * Maintain human oversight and feedback mechanisms * Consider the full cost/performance/latency tradeoff space when making architectural decisions The discussion revealed that successful LLMOps implementations require a sophisticated balance of technical, operational, and business considerations. Organizations are moving beyond simple API calls to build complex systems that combine multiple models, strong evaluation frameworks, and continuous feedback loops. The trend toward smaller, specialized models appears to be gaining momentum, but organizations are primarily focused on finding the right tool for each specific use case rather than following any particular technological dogma.