A powerful clinical artificial intelligence tool developed by University at Buffalo biomedical informatics researchers has demonstrated remarkable accuracy on all three parts of the United States Medical Licensing Exam. Achieving higher scores on the USMLE than most physicians and all other AI tools so far, Semantic Clinical Artificial Intelligence (SCAI, pronounced “Sky”) has the potential to become a critical partner for physicians, says lead author Peter L. Elkin, MD, chair of the Department of Biomedical Informatics in the Jacobs School of Medicine and Biomedical Sciences at UB and a physician with UBMD Internal Medicine. 

Elkin says SCAI is the most accurate clinical AI tool available to date, with the most advanced version scoring 95.2% on Step 3 of the USMLE, while a GPT4 Omni tool scored 90.5% on the same test. As physicians, we are used to using computers as tools, he explains, but SCAI is different; it can add to your decision-making and thinking based on its own reasoning. The researchers tested the model against the USMLE, required for licensing physicians nationwide, which assesses the physician’s ability to apply knowledge, concepts and principles, and to demonstrate fundamental patient-centered skills.

Any questions with a visual component were eliminated. Elkin explains that most AI tools function by using statistics to find associations in online data that allow them to answer a question. We call these tools generative artificial intelligence, he says. Some have postulated that they are just plagiarizing what’s on the internet because the answers they give you are what others have written. However, these AI models are now becoming partners in care rather than simple tools for clinicians to utilize in their practice, he says. 

Source: https://www.buffalo.edu/news/releases/2025/04/AI-evidence-based-medicine-outperforms-USMLE-exams.html