Microsoft's AI Beats Doctors on Hard Cases. Read the Fine Print

Every few months, an AI company announces that its system has beaten doctors at something. This week it was Microsoft's turn. The company says its diagnostic system, MAI-DxO, solved hard medical cases far more often than physicians did in a simulation. The results appear in a study in Science, and Microsoft AI CEO Mustafa Suleyman calls them a step toward "medical superintelligence."

It's an eye-catching claim, so let's look at what was actually tested, what the numbers say, and where the caveats sit. I'll also say where I think the real story is, which is not quite the one in the headline.

What Microsoft actually built

MAI-DxO is short for Microsoft AI Diagnostic Orchestrator. It isn't a single chatbot giving a single answer. According to TechCrunch, it coordinates several AI agents that behave like a panel of doctors. They question a patient, order tests and debate a diagnosis.

Microsoft's own announcement describes it as a panel of virtual physicians. Each one proposes diagnoses, picks useful questions or tests, and challenges the emerging view. The exchange runs over several rounds, and the system updates its assessment as evidence arrives. It also weighs each test's likely value against its cost.

That last part matters. A real doctor doesn't get to order every test on the menu. Money, time and patient discomfort all count. A system that reaches the right answer by ordering everything isn't very useful, and Microsoft built cost into the evaluation for that reason.

A benchmark that looks more like real medicine

Alongside the system, Microsoft introduced a test called the Sequential Diagnosis Benchmark, or SDBench. The company argues that medical licensing exams, which are the usual yardstick for health AI, don't resemble clinical work. In an exam, all the information arrives up front and you pick one answer. In a clinic, you build the picture piece by piece.

SDBench works the other way round. According to Microsoft, it uses 304 complex cases from the New England Journal of Medicine and a set of more typical emergency-department presentations from a dataset called MIMIC-IV-ED. Each case becomes interactive. The AI or the physician starts with limited information and chooses what to ask or order next. The benchmark scores the final diagnosis, the evidence requested and the estimated cost of the tests.

I like this design. Grading the path as well as the answer is closer to how we judge clinicians. Microsoft says it is open-sourcing the benchmark for select NEJM cases, including the code for the cost estimates, so other researchers can reproduce the evaluation and challenge it. That is a good habit, and I hope it sticks.

The headline numbers

Here is what the company reports. On the complex NEJM cases, MAI-DxO reached 83% diagnostic accuracy. Physicians reached 22% without outside reference material and 36% with it. That works out to about 3.7 times and 2.3 times more accurate. The "up to four times" in some headlines comes from the first comparison.

On the more typical emergency-department cases, Microsoft reports 80% accuracy for MAI-DxO against 47% for physicians. It also says the system reached its answers at half or even a third of the cost, with fewer tests and procedures. One cost-optimized configuration hit 81% on the NEJM cases at less than half the cost of comparative systems.

These are big gaps. Still, a number only means something next to its comparison group, and that is where TechCrunch adds an important note.

The caveat about who the doctors were

TechCrunch points out that the physicians in the comparison were primary care doctors and general internists, working individually with reference materials. Cases this complex are normally handled by specialists or teams of physicians, and they weren't part of the comparison.

So the fair reading is not "AI beats doctors." It is closer to "a coordinated AI system beat individual generalists on a set of puzzle-like cases they would normally hand off." That is still interesting. It is also a long way from replacing an oncologist, or a hospital team that has met the patient.

NEJM case write-ups are also famously tricky, which is why they're published. They make a good stress test. They are not a random sample of what walks into a clinic. The emergency-department set is closer to everyday reality, and I'll come back to why the results there look different.

How much credit goes to the orchestrator?

The most useful part of the TechCrunch report is its look at the study's own data. Its conclusion is that much of the improvement seems to come from the AI models underneath, with Microsoft's orchestration adding gains on top.

Microsoft tested MAI-DxO with models from several companies, including OpenAI, Google, Anthropic, Meta and DeepSeek. The best results came with OpenAI's GPT-5 and GPT-5.5. Here are the figures TechCrunch gives:

• GPT-5 on its own scored 78.9% on complex cases at about $8,400 per case, with a short, basic prompt. With MAI-DxO, the cost dropped by about $3,000 and accuracy rose by only about four points.
• GPT-5.5, a newer model released in April 2026, went from 72% to nearly 87% on complex cases with MAI-DxO, at a slightly lower cost.
• Gemini 2.5 Flash scored 46.4% alone and nearly 69% with the orchestrator. That is a large boost, though the gap between weaker and stronger models was still bigger than the lift from MAI-DxO.
• Emergency department cases showed little improvement in accuracy or cost for GPT-5. For GPT-5.5, both versions scored about 85% at a similar cost.

Microsoft's researchers told TechCrunch that accuracy gains naturally shrink as models approach the theoretical maximum. A spokesperson said the orchestration helps with the broader challenge of optimizing accuracy and cost together. Microsoft's own post says the gains persisted, and improved, when it repeated the evaluation with newer models such as GPT-5.5, which it takes as a sign the approach isn't tied to one model generation.

Both readings can be true. The orchestrator seems to help most where models are weaker and cases are harder, and it can make runs cheaper. But the biggest driver of accuracy looks like the quality of the base model. For a reader, that means the "doctor panel" framing is clever packaging around a more basic fact: the models keep getting better.

From benchmark to bedside

The part I find most promising is not the benchmark. It's the plan to test the system in real hospitals. Microsoft says it has a new partnership with Beth Israel Deaconess Medical Center in Boston. According to TechCrunch, patients will receive standard care, including triage, specialist time, treatments, investigations and the discharge decision. Those who consent will also interact with MAI-DxO, and Microsoft and its partners will compare the system's recommendations with the human doctors'.

Dominic King, vice president of Health at Microsoft AI, said he expects it to be the biggest study of its kind, spanning emergency and urgent care settings. TechCrunch notes that Google published a clinical study this month in The Lancet with Beth Israel, evaluating its AMIE diagnostic AI in real urgent care visits with physicians monitoring in real time. OpenAI and Google have made similar claims about matching or beating doctors, so the race is clearly on.

This is where the evidence will really be decided. Simulations tell us what a system can do with tidy information. Real patients are messy, anxious and sometimes unable to say what's wrong. If you want more background on how machine learning has been entering diagnosis, we covered it in our piece on machine learning in early medical diagnosis.

What Suleyman says is coming

Suleyman, who co-founded DeepMind before joining Microsoft in 2024, is openly ambitious. He told TechCrunch that Microsoft is working with the Mayo Clinic on a foundation model for healthcare, with release planned for next year. He also said he expects healthcare models to get "an order of magnitude better" within 24 to 36 months, and that a flywheel of real-world usage will drive the progress, as it did in coding and chat.

He said he believes 2027 will bring a breakthrough for medical records, with accurate predictions about how a record might unfold. Microsoft already sells dictation and radiology tools to clinicians, and Suleyman said the company wants the best clinical decision support products within a few years.

These are forecasts and company ambitions, not results. I'd treat them the way I treat any roadmap from a company with a product to sell: with interest and a raised eyebrow. A flywheel can also spin in the wrong direction if the gaps it finds are about patient safety.

What I take from this

My honest view is that this is a meaningful step, and also more modest than "superintelligence" suggests. The evaluation design is thoughtful. Cost is part of the measurement. The benchmark is partly open. And Microsoft is moving toward a prospective study with real patients, which is the right thing to do.

At the same time, the comparison group was generalists on cases built for specialists, and the study's own data suggests the base model carries a lot of the weight. The emergency-department results, the setting closest to daily medicine, show the smallest orchestration gains for the strongest models. We should also remember that people have tended to be impressed by AI benchmark wins before real-world conditions humbled the claims. We looked at a similar pattern of enthusiasm and hard questions in our article on AI in mathematics.

The best-case future here is not an AI replacing your doctor. It is a second opinion that is always available, makes cheaper choices and helps close the gap between the demand for medical expertise and the supply of it. That gap is real, and King named it as the biggest challenge in healthcare delivery.

So here is a question to take with you. If an AI offered a second opinion on your diagnosis tomorrow, what would you want to know before trusting it? Who tested it, on whom, and against which doctors? Those are the questions that will matter more and more as these systems arrive in clinics.