Paging Dr. AI to the ER? Artificial intelligence shows promise in emergency room diagnosis

When a patient number one comes through and through the doors of an exigency way, Ontario physician Dr. Nour Khatib says it put up be a puzzle determining a diagnosis, course of treatment and what they might need to be sent home safely.

Khatib, who works in ERs at Oak Valley Health's Markham Stouffville and Uxbridge Hospitals, is like many other physicians who are increasingly relying on artificial intelligence to make that process more efficient.

"It's just another tool to help us give the patient the highest quality care possible," she said.

A new study published Thursday in the journal Science may be a further step toward that.

The study looked at the emergency room performance of large language models (LLMs), which can analyze huge amounts of online information to generate human-like responses. It found that LLMs could diagnose cases as well as, or even better than, actual doctors.

But even as the technology develops, Khatib and other physicians — including this study's author — insist that computers won't replace the eyes, ears and skills of a trained emergency medical professional.

Khatib has already been working with AI scribes, which transcribe exchanges between doctors and patients and creates detailed medical notes. It's a pilot project with Oak Valley health and done with prior consent from patients.

She says hospitals are also exploring the use of self-scheduling using AI, and also chatbots that can help patients get a better understanding of specific illnesses.

The LLM used as part of the recent study is a specialized type known as a reasoning model, which is trained to solve complex tasks by explaining its thinking before giving a final answer. It's already becoming "commonplace" in U.S. Hospitals, says lead author Dr. Adam Rodman, a physician at Beth Israel Deaconess Medical Center in Boston.

When you look at how these "reasoners" make a diagnosis, he says, it's similar to the steps a doctor would have taken to solve a problem.

"Getting a model to think in this way," he says, "it improves the diagnostic accuracy."

How doctors are using AI in the exam room — and why it could become the norm

The researchers carried out several trials with both real patient cases and synthetic cases using “unstructured” data from the records of an emergency department, in an effort to "mirror the high-stakes decisions" that doctors and nurses make in the ER.

They used OpenAI's o1-preview model at a Boston emergency room during three points of patient interaction: initial triage, doctor examination in the ER and admission to the medical floor or intensive care unit. The research relied only on data. None of the testing involved actual doctor-patient interactions and had no effect on real diagnoses or treatments.

With the real patient cases, Rodman says the model was asked at each stage a very narrow set of questions focused on the presentation of symptoms to produce the "most likely" diagnosis.

With the synthetic cases, he explains, the tool was also asked about the reasoning for its output as well as next steps in patient management.

Overall, Rodman's study found that the model identified the exact or a very close diagnosis, at times surpassing the performance of the physicians who participated in the trial at each stage of care.

"It doesn't mean that computers can do medicine, but within this narrow task it can solve diagnoses better than humans," says Rodman.

This Toronto hospital will use AI to save scheduling time for patients

Dr. Amol Verma, an internal medicine physician and scientist at Toronto's St. Michael's Hospital, sees how good AI tools have become at answering medical questions and diagnosing patient cases.

But he says it's a "false comparison" to say they are "better than doctors."

"I don't know a single doctor who makes all of their decisions based purely on text information," he said.

It's the physical examination — how someone looks, sounds and feels — that forms a diagnosis, he says.

Khatib echoes that, offering the example of a recent emergency room patient she treated.

She says the information obtained from the patient during triage provided details about symptoms aligned with an existing disease.

But her understanding of the patient's condition changed when she listened with her stethoscope — something AI isn't going to do.

It's also not going to intubate a patient in an ER or put a cast on an injured limb, she says.

AI better at catching early breast cancer than humans: study

Rodman admits there are limitations to his study and that more work is needed to understand how humans and machines can collaborate effectively in an emergency medical environment.

But he believes this is a first step, even though there will need to be more "robust" studies clinical trials to ensure real-world efficacy and safety.

Verma not only wants to see further evaluation of reasoning models in ERs, but also in Canadian settings.

OpenAI is an American company — something he says he finds concerning, in regards to the privacy of patient information — with the study relying on a model trained on U.S. Data in a largely privatized health-care system.

"It may not apply to the Canadian context," he said.

Although this study helps make the case that a reasoning model can be effective at diagnosing ER patients, in some cases, Khatib says all exploration of AI in hospital settings must be done responsibly and that the right people must be using it safely, securely and accurately.

"We are dealing with AI by putting guardrails first," she said. "We're not chasing AI headlines first."

Are AI chatbots giving good medical advice to Canadians?

Senior Writer

---

---