The case study examines two companies' approaches to deploying LLMs for code generation at scale: Stackblitz's Bolt.new achieving over $8M ARR in 2 months with their browser-based development environment, and Qodo's enterprise-focused solution handling complex deployment scenarios across 96 different configurations. Both companies demonstrate different approaches to productionizing LLMs, with Bolt.new focusing on simplified web app development for non-developers and Qodo targeting enterprise testing and code review workflows.
This podcast transcript covers two complementary but distinct approaches to deploying LLMs for code generation in production: StackBlitz’s Bolt.new, a consumer-facing browser-based application builder, and Qodo’s enterprise-focused code agents for testing and code review. Both companies have achieved significant production deployments and offer valuable insights into the operational challenges of running LLM-powered development tools at scale.
StackBlitz spent seven years building WebContainers, a custom operating system that runs entirely within the browser using WebAssembly and service workers. This technology allows full Node.js execution, web servers, and development environments to run client-side without any server infrastructure per user. The key insight was that the browser had evolved sufficient APIs (WebAssembly, service workers, etc.) to support running an operating system natively.
The Bolt.new product was conceived earlier in 2024 but shelved because the available LLMs at the time were not capable enough for accurate code generation without extensive RAG infrastructure. When newer models (implied to be Claude Sonnet) became available, the team saw the code generation quality had crossed a threshold that made the product viable. This demonstrates the critical importance of model capability thresholds in LLMOps—the same product architecture was not viable months earlier due to model limitations.
The WebContainer technology provides several LLMOps advantages:
Unified environment: Because StackBlitz wrote the entire OS from scratch, they have complete control over instrumentation at every level (process, runtime, etc.). This is impossible to achieve reliably on local development environments where OS configurations vary wildly.
Error capture and self-healing loops: The instrumented environment captures errors from build processes, the Node.js runtime, and browser applications. These are fed back to the agent with a “fix it” button that provides the full application state and error telemetry, enabling effective self-correction loops.
Zero marginal infrastructure cost: Since WebContainer runs in the user’s browser, serving one user versus a million users has essentially the same server-side cost (minus inference). This is a significant operational advantage compared to server-side sandboxing solutions.
Compact footprint: The custom OS is approximately 1MB compared to 60-100MB for Docker-to-Wasm conversions, enabling fast load times essential for the consumer experience.
The team relies heavily on frontier models, specifically Claude Sonnet, for code generation. They describe the relationship between model capability and prompt engineering as multiplicative: the model provides roughly a “10x multiplier” on base capability, while prompt engineering and multi-agent approaches can squeeze out an additional “3-4x” improvement.
Key engineering decisions include:
The team notes that the same prompting approach works less well on weaker models—the task decomposition approach helps normalize performance across different models by making each individual step simpler.
Bolt.new integrates with Netlify for one-click deployment using an API that allows anonymous deployments without user login. Users can deploy a live website and only claim it to a Netlify account if they want to persist it. This frictionless deployment path is cited as critical to the product’s success, enabling non-technical users to go from idea to live website in a single session.
The pricing evolved rapidly from an initial $9/month plan (carried over from StackBlitz’s developer-focused product) to tiered plans at $20, $50, $100, and $200 plus usage-based token purchases. The company explicitly states they are not taking margin on inference costs, reinvesting all value into user experience.
The high-context approach (sending complete application state with each request) consumes significantly more tokens than traditional code completion tools like GitHub Copilot, which intentionally minimize context to keep costs low. This represents a fundamental trade-off: higher inference costs but dramatically better output quality that justifies premium pricing.
Usage-based billing accounts for an additional 20-30% of revenue beyond subscriptions, indicating power users (especially web development agencies) are willing to pay significantly more for increased capability.
Qodo has evolved from a single IDE extension for unit testing to a platform of specialized agents:
The philosophy is explicitly against general-purpose agents. Specialized agents allow for proper permission management, different data source access, dedicated guardrails, and targeted approval workflows—all requirements for enterprise deployment.
Qodo operates four distinct models in production:
The models are named “Codium” after the company’s rename from Codium AI to Codo. This multi-model approach allows optimization for each specific task rather than relying on a single general-purpose model.
The case study reveals the extreme complexity of enterprise LLMOps deployments. Qodo supports 96 different deployment configurations across multiple dimensions:
Each enterprise customer presents unique networking configurations, and seemingly simple requirements (like “AWS only, but GitHub Enterprise on-premise”) create complex integration challenges requiring private links and custom solutions.
Qodo champions “flow engineering” based on their open-source AlphaCodium project, which achieved 95th percentile on CodeForces competitions. The approach involves:
This task decomposition is presented as a necessity for supporting multiple deployment targets—when you can’t control which model a customer will use, reducing task complexity normalizes output quality across models. Even OpenAI’s O1 (reasoning model) benefits from this decomposition, suggesting current models are not truly “System 2” thinkers despite marketing claims.
Working with Fortune 500 code bases presents unique challenges:
The solution involves allowing tech leads to provide markdown files with best practices that the agents incorporate, preventing suggestions that violate established conventions.
The transcript includes a notable claim about GitHub Copilot enterprise retention rates of 38-50%, attributed to the disconnect between simple autocomplete and the complex, context-rich needs of enterprise development. This suggests that enterprise LLMOps products require significantly more sophisticated context management than consumer tools.
The conversation highlights an interesting dichotomy in AI code generation:
Both approaches require testing and validation loops, but the nature of those loops differs dramatically. Bolt.new can leverage visual inspection and simple error catching; enterprise tools require formal testing frameworks, code review automation, and integration with existing CI/CD pipelines.
The open-source strategy also differs: Bolt.new open-sourced their core agent to build community and demonstrate confidence in their execution ability (citing Geohot’s philosophy that if you’re confident you can “crush it,” keeping things closed is unnecessary). Qodo maintains open-source versions alongside commercial products, accepting the risk of competitors copying their work (including finding their typos in competitor UIs) as the cost of community engagement.
Both companies demonstrate that successful LLMOps requires not just model access but deep integration with the execution environment—whether that’s a custom browser-based OS or enterprise-grade deployment infrastructure supporting dozens of configuration combinations.
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
This case study examines Cursor's implementation of reinforcement learning (RL) for training coding models and agents in production environments. The team discusses the unique challenges of applying RL to code generation compared to other domains like mathematics, including handling larger action spaces, multi-step tool calling processes, and developing reward signals that capture real-world usage patterns. They explore various technical approaches including test-based rewards, process reward models, and infrastructure optimizations for handling long context windows and high-throughput inference during RL training, while working toward more human-centric evaluation metrics beyond traditional test coverage.
Stripe, processing approximately 1.3% of global GDP, has evolved from traditional ML-based fraud detection to deploying transformer-based foundation models for payments that process every transaction in under 100ms. The company built a domain-specific foundation model treating charges as tokens and behavior sequences as context windows, ingesting tens of billions of transactions to power fraud detection, improving card-testing detection from 59% to 97% accuracy for large merchants. Stripe also launched the Agentic Commerce Protocol (ACP) jointly with OpenAI to standardize how agents discover and purchase from merchant catalogs, complemented by internal AI adoption reaching 8,500 employees daily using LLM tools, with 65-70% of engineers using AI coding assistants and achieving significant productivity gains like reducing payment method integrations from 2 months to 2 weeks.
