OpenAI’s HealthBench is Trying to Fix AI’s Biggest Medical Blind Spot

• By John K. Waters
• 05/13/2025

OpenAI has introduced HealthBench, a sweeping new benchmark designed to test how large language models perform in real-world healthcare scenarios. It’s the company’s first standalone project in medicine, and it lands at a time when AI tools are already seeping into clinical workflows, often without the scrutiny they deserve.

"Improving human health will be one of the defining impacts of AGI," OpenAI wrote in a company blog post. "If developed and deployed effectively, large language models have the potential to expand access to health information, support clinicians in delivering high-quality care, and help people advocate for their health and that of their communities."

HealthBench contains 5,000 multi-turn, multilingual conversations built to simulate interactions between patients, providers, and AI systems. Backed by a global cohort of 262 physicians and over 48,000 rubric-based evaluation criteria, it pushes beyond multiple-choice exams and into the domain of messy, ambiguous, and high-stakes care delivery.

"HealthBench conversations are split into seven themes, such as emergency situations, handling uncertainty, or global health," the company says. "Each theme contains relevant examples, each with specific rubric criteria. Each rubric criterion has an axis that defines what aspect of model behavior the criterion grades, such as accuracy, communication quality, or context seeking."

The conversations span domains from emergency medicine to global health and are scored on dimensions like factual accuracy, instruction-following, communication quality, and medical appropriateness. Importantly, the benchmark includes 1,000 intentionally difficult examples, cases where existing AI systems tend to falter—offering a proving ground for future model improvements.

Where previous benchmarks emphasized test performance, HealthBench reframes the question: can AI actually function inside the clinical reality of healthcare systems, across cultures, languages, and specialties? The goal is not just to measure correctness, but to evaluate whether a model’s output is safe, understandable, and genuinely useful in practice.

Behind this push is a broader effort to position OpenAI as a central authority in healthcare AI evaluation. The benchmark is open source and comes with both data and code, signaling a strategic move to shape the standards others will follow. It’s a play not just for credibility, but for influence—at a moment when health systems and startups are racing to deploy generative AI.

HealthBench also reveals the complexity of AI evaluation in sensitive domains. OpenAI used its own model-based graders to assess performance, a choice that raises questions about objectivity and hidden biases. There are concerns that shared blind spots between grader and model could mask systemic issues, especially without sufficient external oversight.

Despite this, HealthBench addresses a long-standing gap in AI testing. Although many existing tools focus on structured medical exams, real clinical encounters are nuanced and non-linear. In these environments, language models must do more than retrieve facts, they must reason, explain, and communicate in ways that match the standards of clinical care.

HealthBench arrives as OpenAI deepens its health partnerships. It is already working with Sanofi and Formation Bio on clinical trial optimization, collaborating with Color Health to personalize cancer treatment, and integrating GPT-4 into hospital administration platforms with Iodine Software. The benchmark aligns with these initiatives by offering a framework to ensure these tools are evaluated rigorously.

Still, HealthBench is only a first step. Broader demographic validation, human-in-the-loop auditing, and more transparency around evaluation methods will be needed before AI can be considered safe for medical decision-making at scale. Its release underscores how far the field has come, and how far it still has to go.

OpenAI’s move is ultimately about shifting the narrative. As generative AI becomes more embedded in daily life, the focus must turn from novelty to reliability. HealthBench plants a flag in that direction, offering a foundation for building systems that are not only smart, but accountable in one of the most consequential domains imaginable: human health.