## Overview This case study is drawn from a podcast-style conversation between Raj Ricky, a Principal Product Manager at Microsoft Azure Data, and the host Demetrius. The discussion centers on the practical considerations for building and deploying AI agents in production environments. Rather than presenting a specific implementation case, this conversation provides a framework of principles and patterns that practitioners should consider when architecting agentic AI systems. Raj brings a unique perspective, having led the development of Microsoft's first deep reinforcement learning platform as a service. ## Historical Context and Foundations Raj emphasizes that understanding agents requires appreciating their lineage. Agents as a concept have existed in academic disciplines since the 1950s, rooted in reinforcement learning, psychology, and dynamic programming. At its core, an agent is an entity—software or otherwise—that takes actions autonomously with decision-making power and capabilities. This historical perspective is valuable because many principles from reinforcement learning apply directly to modern LLM-based agents. The conversation draws important parallels between traditional reinforcement learning concepts and modern agentic AI development. Key concepts include defining the environment (the universe in which the agent operates), specifying the action space (what tools and capabilities the agent has), and establishing reward signals (the success criteria by which agent behavior is evaluated). ## First Principles for Agent Development ### Start with Pen and Paper One of the strongest recommendations from Raj is to begin agent development before writing any code. This means sitting down and defining what you want to accomplish, what the intended behavior should be, and what success looks like. This exercise effectively creates an MVP for an agent without any programming. As Raj notes, programming is merely a means to an end—it's a tool that has libraries and off-the-shelf components to accomplish goals, but it's not the starting point. ### Define the Environment The environment is the bounded universe in which the agent operates. This could be a video game, a screen with pixels, Microsoft Fabric, Microsoft Office, or any other constrained context. Setting these boundaries is crucial because it establishes guardrails for agent behavior. The environment definition should come before deciding on tools or capabilities. ### Specify Tools and Capabilities Once the environment is defined, the next step is to determine what capabilities the agent has. In agentic implementations, this takes the form of function calls or tool calls. These represent the actions an agent can take—driving to a grocery store, picking up an item, accessing a database, making an API call, or invoking another model for inference. The tool specification describes what the agent is capable of doing within its environment. ### Establish Success Criteria Perhaps most critically, success criteria must be defined clearly before development begins. In reinforcement learning, this is handled through reward functions. For agentic AI, this means defining measurable parameters for success. Without clear success criteria, developers risk getting lost in a "vortex of tools" and losing sight of the actual objectives. ## The Tension Between Precision and Flexibility A significant theme in the discussion is the tradeoff between narrowly defining success criteria versus leaving room for interpretation. Over-specification can inadvertently limit an agent's effectiveness, but under-specification leads to unreliable behavior. Raj notes that the tools for making these determinations are becoming more sophisticated, mature, and abstract—which benefits everyone involved. The conversation references AutoGPT's Forge interface from about six months prior as an example of how these abstractions have evolved. While functional, it wasn't as user-friendly as modern interfaces. The ecosystem has since produced cleaner libraries, more mature code, and better documentation. ## Orchestration Considerations Orchestration is identified as a critical consideration for production agents. Key questions include: Who is doing the planning? Are you offloading planning to a service like Azure OpenAI (which has a planner behind the scenes)? Are you using OpenAI's Assistant API or Claude's function calling API? Or are you implementing your own planning logic using frameworks like Semantic Kernel, LangChain, or LangGraph? If you're running planning yourself, you have to figure out what that plan looks like. This may involve implementing your own planning algorithms. The choice of orchestration approach has significant implications for system complexity, reliability, and maintainability. ## Understanding LLM Stochasticity Raj makes an important point about the fundamental nature of LLMs that is often misunderstood. LLMs are probabilistic and stochastic by design—they are meant to be random. What many people call "hallucinations" and view as bugs are actually the product working as intended. The goal for practitioners is to figure out how to effectively harness this fabrication capability. This doesn't mean deterministic behavior is impossible. Developers can create affordances and capabilities that are more deterministic and planned, but this requires intentional design and governance. The architect of an agentic system—whether they're formally an architect or not—is responsible for defining and governing these behaviors. ## Practical Debugging and Evaluation Strategies ### Keep Humans in the Loop For MVP implementations, the recommendation is always to keep a human in the loop. While agents are designed to act autonomously, that doesn't mean they must immediately action every decision. Developers can create stop-gaps before actions are taken and ask the agent to explain its planned next steps. LLMs are quite good at outputting debug logs of what they're going to do and why. ### Constrain the State Space Before deploying broadly, constrain the environment and see how successful the agent is with a small landing space. Raj uses an analogy of testing a cleaning product on an inconspicuous area of a couch before applying it to a visible chocolate milk stain. For a support agent, this means not immediately giving it the entire compendium of documentation and firing it at all support tickets. Instead, cherry-pick lower priority, clearly defined support cases and understand behavior before expanding scope. ### Never Give Raw Web Access A specific anti-pattern called out is giving agents raw web access out of the box. Regardless of safeguards, this is identified as a bad idea. Agents need directives—giving them unlimited access without constraints is compared to handing a tablet to a five-year-old and telling them to explore the internet. ### Start Simple, Then Iterate If your MVP is pen and paper, the step beyond MVP should not be coding a complex chain from scratch. The recommendation is to use off-the-shelf tools first. For non-code-first people, this means exploring no-code solutions like Copilot Studio after completing the pen-and-paper exercise. For code-native folks, it means using established function calling APIs or tool APIs before building custom solutions. The analogy given is that you wouldn't start writing your own database just because a relational database isn't sufficient for your use case—that would be "literally crazy." The same principle applies to agent frameworks. ## Evaluating Tools and Frameworks When evaluating whether a tool or framework is meeting your needs, the process starts with understanding what success looks like. The question becomes: Are you able to accomplish, with a clear mind and clear intent, the success criteria you spelled out on paper? If you're struggling with a tool like AutoGPT, you're not alone—even experienced practitioners have had setup difficulties. Documentation may be good in some areas but challenging in others. The recommendation is not to twist yourself into knots unnecessarily. If a no-code solution with all the bells and whistles out of the box doesn't get you there, try the next step up, but always maintain clarity about success criteria. ## Real-World Application Areas The conversation covers several promising use cases for agentic AI: - **Fraud Detection in Finance**: Identifying suspicious transactions in real-time, stopping them, and contacting users proactively - **Supply Chain Optimization**: Handling bin packing problems with real-time reflexive action as new information arrives - **Patient Monitoring in Healthcare**: Proactively monitoring EHR data (charts, doctor's notes, intake forms, patient history) for issues rather than waiting for human recognition - **Agricultural Automation**: Using sensor telemetry from farms to automatically adjust irrigation or apply nitrogen based on soil conditions - **Personal Shopping Assistants**: Connecting social media and wardrobe data to scour the web for brand recommendations and sales ## Addressing Latency and Cost Concerns A significant discussion addresses the apparent tension between LLM inference latency and real-time use cases like fraud detection. Raj identifies small language models (SLMs) as a major investment opportunity that hasn't been fully realized. The smaller the model, the less expensive, potentially more accurate, faster, and more computationally efficient it becomes. Parallel movements around quantization and parameter-efficient fine-tuning are also making inference more practical. The key insight is that if your application reaches the point where inference speed and cost are the primary problems, you've already won—you've solved the value delivery problem, and there are tools and techniques available to address performance. ## Future Outlook Raj expresses strong conviction that agentic workflows will be fundamentally transformative to software. The evolution from "What can this do for me?" (upon first encountering ChatGPT) to "How can I get it to do it for me?" (enabled by function calls and tools) represents a fundamental shift in how we interact with AI systems. However, he emphasizes that we are early in the agentic software development lifecycle. Practitioners should give themselves and the tools grace, but also remain persistent. The "magic" of early LLM interactions is still there—it just requires more effort to unlock in agentic contexts. The best systems, whether AI-based or traditional software, solve problems simply. If you can get an agent to solve a problem simply, you've won.