Realtime has developed an innovative approach to scaling data journalism through the implementation of LLMs in production. The platform automatically generates news stories by analyzing real-time data updates from various sources such as economic indicators, political polls, environmental data, and sports odds. This case study provides valuable insights into the practical challenges and solutions of deploying LLMs in a production news environment. The system architecture comprises several key components that work together to create a robust LLMOps pipeline: Data Infrastructure and Processing: The foundation of the system is a sophisticated data pipeline that continuously monitors and processes updates from various data sources. Rather than relying on push notifications, they implemented a polling system that checks for new data by comparing current holdings against the latest published information. This pipeline runs on a distributed cloud platform and can be configured per data feed to optimize update frequency. The system standardizes diverse data formats into a unified format for consistent processing. LLM Implementation: The platform primarily uses GPT-4 Turbo from OpenAI for text generation. The LLM processing is structured as a multi-stage pipeline, where different tasks are broken down into separate API calls rather than attempting to handle everything in a single prompt. This approach has shown significant improvements in performance and accuracy, though it comes with additional cost considerations. The prompt engineering strategy is particularly noteworthy. The system constructs dynamic prompts using three main components: * Dataset metadata (title, description, and context) * Latest data features (quantitative summary of updates) * Recent related news articles The team has implemented several important safeguards and best practices in their LLM operations: Prompt Engineering and Quality Control: * They use multiple LLM passes for quality control, including an initial generation pass followed by editing passes * The system includes specific prompt constraints to prevent common LLM mistakes, such as inferring false causation * They've discovered that using more human-readable data formats (like YAML over JSON) in prompts can improve LLM performance * The platform implements structured output generation, using JSON templates for consistent formatting Error Prevention and Transparency: * The system includes checks against common fallacies and mathematical errors * They maintain transparency about AI usage and provide readers with ways to verify information * All generated content includes links to source data and news articles * Visualizations are always shown alongside generated text for verification Cost Optimization: The team had to carefully consider the economics of using commercial LLM APIs. They designed their system to ensure costs remain constant regardless of visitor traffic. This was achieved by: * Caching generated content * Optimizing the number of API calls * Carefully structuring the multi-stage pipeline to balance accuracy and cost Technical Innovations: The platform includes several notable technical features: * A markup system for automated link creation in generated text * Dynamic ranking system for top stories based on data update magnitude and news volume * Integration with visualization tools (Vega and Vega-Lite) * Distributed cloud infrastructure for scalability Challenges and Limitations: The case study honestly addresses several ongoing challenges: * Cost considerations with current commercial LLM APIs * The need for extremely clean and consistently formatted input data * Balancing automation with journalistic integrity * Managing LLM tendencies to create overly dramatic or causation-implying headlines Future Considerations: The team acknowledges that the technology is still evolving and anticipates improvements in: * Model costs and accessibility * Inference speeds * Error rates and accuracy * Integration with newsroom workflows This case study is particularly valuable as it demonstrates a practical, production-ready implementation of LLMs in journalism while maintaining high standards for accuracy and transparency. The team's approach to breaking down complex tasks, implementing multiple verification steps, and maintaining clear documentation of AI usage provides a useful template for other organizations looking to implement LLMs in production environments.