Comparative benchmark assessment of performance of six different large language models in clinical decision-making for vital pulp therapy
Source: PubMed Central Open Access, NCBI / U.S. National Library of Medicine
Aim: Artificial intelligence (AI) chatbots or large language models (LLMs) are adept at generating language, but their increasing use in the healthcare field, including endodontics, raises concerns about their accuracy. The potential of LLMs to assist clinicians in their decision-making processes regarding vital pulp therapy (VPT) is worth exploring. This study aims to evaluate and compare the responses provided by OpenAI GPT-5.1 Instant, DeepSeek-R1, Claude, Google Gemini, Comet, and Perplexity to clinically relevant questions related to VPT according to the guidelines set by the American Association of Endodontists, European Society of Endodontics, and Indian Endodontic Society. Materials and Methods: Twenty-three open-ended questions covering various aspects of VPT were developed and presented to OpenAI GPT-5.1 Instant, DeepSeek-R1, Claude, Google Gemini, Comet, and Perplexity. Two experienced endodontists, who were blinded to the different chatbots, evaluated the answers on a 3-point Likert scale. To assess the reproducibility of these answers, the same questions were presented again after 1 month and subsequently saved in a separate Microsoft Word file. The findings were recorded in an Microsoft Excel Sheet, and then statistical analysis was performed. Results: All the LLMs were able to answering all the questions on VPT with almost similar reproducibility across two different intervals. Conclusion: Most tested LLMs, regardless of whether they are free or subscription-ba
Abstract
Aim: Artificial intelligence (AI) chatbots or large language models (LLMs) are adept at generating language, but their increasing use in the healthcare field, including endodontics, raises concerns about their accuracy. The potential of LLMs to assist clinicians in their decision-making processes regarding vital pulp therapy (VPT) is worth exploring. This study aims to evaluate and compare the responses provided by OpenAI GPT-5.1 Instant, DeepSeek-R1, Claude, Google Gemini, Comet, and Perplexity to clinically relevant questions related to VPT according to the guidelines set by the American Association of Endodontists, European Society of Endodontics, and Indian Endodontic Society. Materials and Methods: Twenty-three open-ended questions covering various aspects of VPT were developed and presented to OpenAI GPT-5.1 Instant, DeepSeek-R1, Claude, Google Gemini, Comet, and Perplexity. Two experienced endodontists, who were blinded to the different chatbots, evaluated the answers on a 3-point Likert scale. To assess the reproducibility of these answers, the same questions were presented again after 1 month and subsequently saved in a separate Microsoft Word file. The findings were recorded in an Microsoft Excel Sheet, and then statistical analysis was performed. Results: All the LLMs were able to answering all the questions on VPT with almost similar reproducibility across two different intervals. Conclusion: Most tested LLMs, regardless of whether they are free or subscription-based, demonstrated high accuracy and reproducibility when evaluated on guidelines-based questions related to VPT.