Duolingo tackled the challenge of scaling their DuoRadio feature, a podcast-like audio learning experience, by implementing an AI-driven content generation pipeline. They transformed a labor-intensive manual process into an automated system using LLMs for script generation and evaluation, coupled with Text-to-Speech technology. This allowed them to expand from 300 to 15,000+ episodes across 25+ language courses in under six months, while reducing costs by 99% and growing daily active users from 100K to 5.5M.
Duolingo’s DuoRadio case study presents an interesting example of scaling educational content production through generative AI pipelines. DuoRadio is an audio feature that provides podcast-like radio shows to help language learners improve their listening comprehension using Duolingo’s character-driven content. The case study, published in March 2025, describes how the team transformed a labor-intensive manual process into a largely automated end-to-end pipeline, achieving significant scale improvements.
The core problem was straightforward: DuoRadio launched in late 2023 and showed promise for learning outcomes, but the production process was extremely resource-intensive. Creating just 300 episodes for a handful of courses took nearly a year, requiring meticulous scripting, curriculum alignment, voice actors, and specialized audio editing. This bottleneck meant DuoRadio remained a niche offering despite its popularity.
The case study is refreshingly honest about early failures with generative AI. Their first approach—generating original scripts from scratch—produced subpar results requiring extensive manual editing. The second approach, automated translation of existing English episodes, also fell short on translation accuracy and proficiency-level appropriateness. Both required significant human intervention, defeating the purpose of automation.
These failures highlight a common LLMOps lesson: naive application of LLMs to content generation often produces outputs that don’t meet production quality standards without substantial human review.
The key insight came during an internal hackathon. Rather than adding more constraints to prompts (which didn’t work well), the team found that feeding existing curriculum content directly into prompts produced dramatically better results. By supplying the LLM with well-crafted sentences and exercises already created by Learning Designers for Duolingo lessons, the model had specific patterns to follow rather than attempting to interpret complex instructions.
This is a significant prompt engineering insight: providing concrete examples from the target domain often outperforms elaborate instruction-based prompting. The curriculum content served as a form of few-shot learning, grounding the model’s outputs in proven educational material.
A critical component of the production system was the AI-powered evaluation layer. The team recognized that while generative AI could produce many candidate scripts, not all would meet quality standards. To address this, they built a filtering process using additional generative AI prompts designed to assess scripts on multiple criteria:
This multi-model approach—using LLMs both for generation and evaluation—is a common pattern in production LLMOps systems. The Learning Designers iteratively refined these evaluator prompts over time, continuously raising the quality bar. This iterative refinement of evaluation criteria represents a human-in-the-loop approach to maintaining quality standards while scaling automation.
It’s worth noting that the effectiveness of this LLM-as-evaluator approach depends heavily on how well the evaluation prompts capture actual quality criteria. The case study doesn’t provide detailed metrics on false positive/negative rates or how the AI evaluations compare to human expert assessments.
An interesting operational finding was that English-only prompt instructions were less effective when generating content for courses teaching languages other than English. By leveraging language-specific content from each course’s curriculum, they achieved better accuracy and relevance. This suggests that for multilingual content generation, prompts and examples should be tailored to the target language rather than relying on translation or English-centric approaches.
The team found that giving generative AI freedom to sequence and place exercises within episodes produced inconsistent quality. They solved this by leveraging learner session data to determine optimal exercise placement and standardizing the order and structure. This is an example of constraining the LLM’s output space based on empirical data—reducing the degrees of freedom where user behavior data already suggests optimal patterns.
Beyond script generation, the automation extended to audio production. The team integrated advanced Text-to-Speech (TTS) systems to automatically generate voiceovers in multiple languages. They also implemented audio hashing techniques for storing and retrieving pre-generated audio segments (like consistent intros and outros), reducing redundant audio generation and editing time.
The full end-to-end pipeline required zero human intervention post-initiation, covering the entire lifecycle from script creation to final deployment. This level of automation is notable, though the case study doesn’t detail the monitoring, error handling, or rollback capabilities that would typically be necessary for such a hands-off production system.
The case study reports impressive metrics, though these should be considered with appropriate skepticism as they come from the company itself:
These numbers suggest substantial improvements in both reach and efficiency. However, the case study doesn’t detail how quality was measured against the original manually-produced episodes, or provide learner outcome data comparing the automated versus manual content.
The team mentions using “Workflow Builder”—described as their internal content generation prototyping tool—to automatically generate DuoRadio content at scale. This suggests Duolingo has invested in internal tooling infrastructure for LLM-powered content generation, which likely enables rapid iteration and experimentation across content teams.
Several LLMOps patterns emerge from this case study:
Domain-specific grounding: Rather than relying solely on prompt instructions, the most effective approach was grounding the LLM’s outputs in existing domain-specific content (curriculum materials). This reduced hallucination and improved alignment with educational standards.
LLM-as-evaluator: Using generative AI not just for content creation but also for quality assessment, with human experts designing and refining the evaluation criteria over time.
Constraint-based generation: Standardizing structural elements (like exercise placement) based on empirical user data, reducing the problem space where the LLM operates.
Multi-stage pipelines: Combining content generation, quality filtering, and audio synthesis into end-to-end automated pipelines with appropriate handoffs between stages.
Iterative prompt refinement: Learning Designers continuously refined prompts based on output quality, representing an ongoing human oversight role even in highly automated systems.
While the case study presents a success story, a few considerations warrant mention. The reliance on existing curriculum content means this approach may be less applicable to truly novel content creation where no reference material exists. The 99% cost reduction claim is striking but lacks detailed breakdown—it’s unclear whether this accounts for the development investment in building the automation infrastructure.
Additionally, the case study notes that simplification of certain feature aspects was necessary “to make automation more feasible.” This suggests some tradeoffs were made between full feature fidelity and automation capability, though the core educational value was reportedly preserved.
The case study also doesn’t address potential concerns around fully automated content pipelines, such as monitoring for model drift, handling edge cases, or quality assurance sampling in production. For educational content, where accuracy is particularly important, these would typically be important operational considerations.
Overall, this case study demonstrates a thoughtful approach to scaling content production through generative AI, with appropriate emphasis on quality controls and human expertise in designing evaluation criteria, while acknowledging that significant infrastructure and prompt engineering investment was required to achieve production-quality results.
Instead of relying on complex constraint-based prompts, they discovered that feeding existing curriculum content into their LLM yielded better results. This approach provided the model with specific patterns to follow, resulting in more appropriate and accurate content generation. This insight demonstrates the importance of high-quality training data and context in LLM applications.
Duolingo developed a sophisticated end-to-end content generation pipeline with several key components:
The implementation of this LLMOps pipeline delivered impressive results:
The case study highlights several important principles for successful LLM implementation in production:
Particularly noteworthy is their approach to quality assurance, which involved overproducing content and then using LLMs themselves to filter for quality, rather than trying to perfect the generation process itself. This approach acknowledges the probabilistic nature of LLM outputs and builds that understanding into the system design.
The case study also demonstrates the importance of having domain experts (Learning Designers) involved in the process of refining and improving the LLM systems over time. Rather than treating the LLM as a black box, they continuously improved the prompts and evaluation criteria based on expert feedback and learner data.
Duolingo plans to expand this approach to other forms of longform content, suggesting that the pipeline they’ve built is flexible enough to be adapted to different content types while maintaining quality standards. This scalability and adaptability is a crucial aspect of successful LLMOps implementations.Duolingo’s implementation of LLMs to scale their DuoRadio feature represents a comprehensive case study in applying generative AI to solve content generation challenges in education technology. The case study demonstrates how a thoughtful, iterative approach to LLMOps can transform a manual, resource-intensive process into an efficient, automated system while maintaining high quality standards.
The initial challenge faced by Duolingo was significant: their DuoRadio feature, which provided podcast-like audio content for language learning, required extensive manual effort for script creation, voice acting, and audio editing. This manual process limited their ability to scale, with just 300 episodes taking nearly a year to produce.
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
Roblox has implemented a comprehensive suite of generative AI features across their gaming platform, addressing challenges in content moderation, code assistance, and creative tools. Starting with safety features using transformer models for text and voice moderation, they expanded to developer tools including AI code assistance, material generation, and specialized texture creation. The company releases new AI features weekly, emphasizing rapid iteration and public testing, while maintaining a balance between automation and creator control. Their approach combines proprietary solutions with open-source contributions, demonstrating successful large-scale deployment of AI in a production gaming environment serving 70 million daily active users.
Smartling operates an enterprise-scale AI-first agentic translation delivery platform serving major corporations like Disney and IBM. The company addresses challenges around automation, centralization, compliance, brand consistency, and handling diverse content types across global markets. Their solution employs multi-step agentic workflows where different model functions validate each other's outputs, combining neural machine translation with large language models, RAG for accessing validated linguistic assets, sophisticated prompting, and automated post-editing for hyper-localization. The platform demonstrates measurable improvements in throughput (from 2,000 to 6,000-7,000 words per day), cost reduction (4-10x cheaper than human translation), and quality approaching 70% human parity for certain language pairs and content types, while maintaining enterprise requirements for repeatability, compliance, and brand voice consistency.
