Summer Health successfully deployed GPT-4 to revolutionize pediatric visit note generation, addressing both provider burnout and parent communication challenges. The implementation reduced note-writing time from 10 to 2 minutes per visit (80% reduction) while making medical information more accessible to parents. By carefully considering HIPAA compliance through BAAs and implementing robust clinical review processes, they demonstrated how LLMs can be safely and effectively deployed in healthcare settings. The case study showcases how AI can simultaneously improve healthcare provider efficiency and patient experience, while maintaining high standards of medical accuracy and regulatory compliance.
Summer Health is a pediatric telehealth company that provides parents with text-message-based access to pediatricians around the clock. This case study describes their implementation of GPT-4 to automate the generation of medical visit notes, addressing a significant pain point in healthcare: the administrative burden of documentation. The company recognized that generative AI could transform a traditionally time-consuming and error-prone process into something more efficient and user-friendly.
The case study, published on OpenAI’s website, presents the implementation as a success story. While the reported metrics are impressive, it’s worth noting that this is a promotional piece and independent verification of the claims is not provided. Nevertheless, the described approach offers valuable insights into deploying LLMs in a highly regulated healthcare environment.
Medical visit note documentation represents a major challenge in healthcare. According to the case study, medical care providers spend over 50% of their time on administrative tasks like writing visit notes. This creates several problems:
For Summer Health specifically, operating as a text-based service, documentation needed to be both fast and clear to match the immediacy of their communication model with parents.
Summer Health built a medical visit notes feature that uses GPT-4 to automatically generate visit notes from a pediatrician’s detailed written observations. The workflow appears to follow this pattern:
The pediatrician conducts the visit (via text message in Summer Health’s case) and writes down their observations in their typical format, which may include medical shorthand. These observations are then processed by the GPT-4-powered system, which generates clear, jargon-free notes suitable for sharing with parents. Critically, the generated notes are reviewed by the pediatrician before being shared, maintaining a human-in-the-loop approach essential for medical applications.
Summer Health chose OpenAI’s platform specifically because it offered two critical capabilities: leading LLM models (specifically GPT-4) and the ability to provide a Business Associate Agreement (BAA) for HIPAA compliance. This latter point is crucial for any healthcare AI deployment in the United States. HIPAA compliance requires that any entity handling protected health information (PHI) has appropriate agreements and safeguards in place.
The case study highlights GPT-4’s “robust capabilities in understanding complex medical language and its adaptability to user requirements” as key factors in the selection. This suggests the company evaluated multiple options before settling on OpenAI’s offering.
The case study mentions that Summer Health “rigorously fine-tuned the model” in collaboration with OpenAI. This indicates they went beyond basic prompt engineering to customize the model’s behavior for their specific use case. Fine-tuning on domain-specific medical documentation would help the model:
Beyond fine-tuning, they implemented a clinical review process to ensure accuracy and relevance in medical contexts. This human-in-the-loop approach is critical for healthcare applications where errors could have serious consequences. The system continues to improve based on expert feedback, suggesting an ongoing feedback loop where pediatrician corrections and suggestions are used to refine the model over time.
The reported results are significant:
Documentation time reduction: Time per visit note dropped from 10 minutes to 2 minutes, a 5x improvement. This means pediatricians only need to quickly review and approve the AI-generated content rather than writing everything from scratch.
Delay reduction: Delays in completing visit notes were reduced by 400%. This metric is somewhat unclear in its presentation (a 400% reduction would be more naturally stated as a 5x improvement in speed or an 80% reduction in delay time), but the intent is clear that notes are now delivered to parents much faster.
Parent satisfaction: Feedback from parents was described as “overwhelmingly positive,” with many stating these were the best visit notes they had ever experienced. Parents appreciated the clarity and felt more informed about their child’s health and recommended follow-up actions.
Provider well-being: The reduction in administrative burden helps reduce end-of-day fatigue for providers and improves note accuracy and consistency.
Several key LLMOps themes emerge from this case study:
The HIPAA compliance requirement underscores the importance of choosing AI infrastructure providers that can meet regulatory requirements. For organizations in healthcare, finance, or other regulated industries, vendor selection must include evaluation of compliance capabilities, not just technical performance.
The clinical review step before notes are shared with parents is essential. In healthcare contexts especially, LLM outputs cannot be trusted without expert verification. The system is designed to augment the physician’s capabilities, not replace their judgment. This is a prudent approach for high-stakes applications.
The mention of ongoing improvement based on expert feedback suggests a feedback loop where physician corrections inform model improvements. This could involve collecting examples of edits physicians make to generated notes and using these to further fine-tune the model or adjust prompts.
A notable quote from the co-founder states: “We thought this was something we would get to in 5 years. Seeing that vision pulled forward has been amazing.” This reflects the rapid pace at which generative AI has made previously complex NLP tasks accessible. What might have required years of custom model development can now be achieved relatively quickly with foundation models.
While the case study presents compelling results, several considerations are worth noting:
Source bias: This is a promotional piece published on OpenAI’s website, so the presentation is naturally favorable. Independent verification of the metrics would strengthen the claims.
Scale not specified: The case study doesn’t mention how many notes are processed, the size of Summer Health’s practice, or how long the system has been in production.
Error rates not disclosed: While the clinical review process is mentioned, there’s no information about how often generated notes require significant edits or corrections.
Cost considerations: No information is provided about the cost of the API calls relative to the time savings achieved.
Edge cases: Healthcare involves many unusual situations. How the system handles rare conditions, complex multi-issue visits, or sensitive topics isn’t discussed.
The Summer Health case study demonstrates a practical application of LLMs in healthcare documentation. The key success factors appear to be: choosing a compliant AI provider, fine-tuning for the specific domain, implementing robust human review processes, and establishing feedback loops for continuous improvement. The reported efficiency gains are substantial, and if accurate, suggest significant potential for LLMs to reduce administrative burden in healthcare while improving the patient experience.
For organizations considering similar implementations, this case study highlights the importance of balancing automation benefits with appropriate safeguards, particularly in regulated industries where accuracy and compliance are non-negotiable.
Snorkel developed a specialized benchmark dataset for evaluating AI agents in insurance underwriting, leveraging their expert network of Chartered Property and Casualty Underwriters (CPCUs). The benchmark simulates an AI copilot that assists junior underwriters by reasoning over proprietary knowledge, using multiple tools including databases and underwriting guidelines, and engaging in multi-turn conversations. The evaluation revealed significant performance variations across frontier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.
OpenAI's Forward Deployed Engineering (FDE) team, led by Colin Jarvis, embeds with enterprise customers to solve high-value problems using LLMs and deliver production-grade AI applications. The team focuses on problems worth tens of millions to billions in value, working with companies across industries including finance (Morgan Stanley), manufacturing (semiconductors, automotive), telecommunications (T-Mobile, Klarna), and others. By deeply understanding customer domains, building evaluation frameworks, implementing guardrails, and iterating with users over months, the FDE team achieves 20-50% efficiency improvements and high adoption rates (98% at Morgan Stanley). The approach emphasizes solving hard, novel problems from zero-to-one, extracting learnings into reusable products and frameworks (like Swarm and Agent Kit), then scaling solutions across the market while maintaining strategic focus on product development over services revenue.
Veradigm, a healthcare IT company, partnered with AWS to integrate generative AI into their Practice Fusion electronic health record (EHR) system to address clinician burnout caused by excessive documentation tasks. The solution leverages AWS HealthScribe for autonomous AI scribing that generates clinical notes from patient-clinician conversations, and AWS HealthLake as a FHIR-based data foundation to provide patient context at scale. The implementation resulted in clinicians saving approximately 2 hours per day on charting, 65% of users requiring no training to adopt the technology, and high satisfaction with note quality. The system processes 60 million patient visits annually and enables ambient documentation that allows clinicians to focus on patient care rather than typing, with a clear path toward zero-edit note generation.
