General-purpose chatbots outperform clinical AI tools on physicians’ real-world questions

A study published in Nature Medicine compared the performance of general-purpose chatbots with specialized clinical artificial intelligence tools. The research included testing on two public benchmarks and real questions from physicians. The results showed that three general-purpose frontier large language models outperformed two leading clinical AI tools. Specialized clinical AI tools achieved similar performance as Google's search AI overview. The study highlights the problem that specialized clinical AI tools are entering medical practice without sufficient independent testing. The research was published online on June 17, 2026.