This case study explores an innovative collaboration between multiple news organizations to develop an AI-powered system for detecting newsworthy real estate transactions. The project, known as Real Estate Alerter, demonstrates a practical application of LLMs in production journalism, combining traditional data analysis with modern AI techniques to automate story detection. The project originated from a serendipitous discovery by a Detroit Free Press editor who spotted a significant real estate listing during a morning run. This incident highlighted the need for a more systematic approach to identifying newsworthy real estate transactions, leading to the development of an automated system. ## System Architecture and Technical Implementation The system architecture consists of several key components: * Anomaly Detection Layer: The first layer uses domain knowledge, statistical analysis, and clustering to identify outliers within real estate transaction datasets. This preprocessing step helps filter the vast amount of daily transactions to focus on potentially interesting cases. * Data Enrichment and Preprocessing: The system transforms structured data into natural language format suitable for LLM processing. It also enriches the data with contextual information about geographical areas and their characteristics. * LLM Integration: The core of the system uses large language models with few-shot prompting to determine transaction newsworthiness. The team started with basic prompting but incorporated a human feedback loop to continuously improve performance. * Celebrity Detection Feature: A specialized component uses named entity recognition on archived data to maintain a database of famous individuals, cross-referenced with legal names and biographical data from Wikidata. ## LLMOps Challenges and Solutions The team faced several significant LLMOps challenges: ### Defining Newsworthiness for LLMs One of the primary challenges was defining "newsworthiness" in a way that LLMs could understand and consistently apply. The team discovered that LLMs needed careful direction to avoid false positives. For example, the system initially flagged all properties with ground contamination as newsworthy, when this was actually a common occurrence in Oslo's property market. ### Cold Start Problem The team addressed the initial lack of training data through: * Conducting interviews with real estate reporters to identify key features * Implementing a rule-based backup system for high-confidence cases (celebrity involvement and extreme prices) * Gradually incorporating human feedback to improve the model's alignment with expert judgment ### Data Quality and Feature Engineering The project revealed important lessons about feature engineering for LLMs: * More complex feature sets required more sophisticated prompting strategies * The team found that simpler feature sets often performed better than expected * The relationship between feature complexity and LLM performance suggested potential benefits from fine-tuning ### Identity Resolution The celebrity detection system faced challenges with: * Matching stage names to legal names in property records * Disambiguating common names * Verifying identity through additional data points like birth dates ## Production Implementation The production system includes: * A Slack bot interface for real-time alerts * A dashboard for viewing and filtering transactions * Feedback mechanisms for handling false positives and negatives * Integration with external data sources for identity verification ## Results and Impact The system has shown promising results in production: * Successfully identified missed stories, such as a cross-country star's property sale * Demonstrated significant improvement in accuracy with even basic few-shot prompting * Showed continued performance improvements through human feedback integration ## Future Development Plans The team has identified several areas for future development: * Establishing a more robust production pipeline * Scaling to US and Canadian real estate markets * Adapting the system for different journalistic standards (local vs. national news) * Improving the famous person detection system * Implementing pattern detection for identifying trends over time * Exploring applications in other domains ## Lessons Learned Key insights from the project include: * The importance of maintaining focus on the core problem definition * The effectiveness of combining GenAI with human feedback * The value of simple, well-defined features over complex feature engineering * The need for careful data privacy considerations in cross-organization collaboration The case study demonstrates the practical challenges and solutions in implementing LLMs for production journalism, highlighting the importance of human oversight, continuous feedback, and careful system design. The success of this implementation suggests potential applications beyond real estate journalism to other data-driven news discovery processes.