Diagnostic Accuracy of a locally deployed Large Language Model Algorithm for Automated Code Stroke Pathway Identification in Emergency Department Triage Notes

The study evaluated the accuracy of a large language model (LLM) algorithm for automated identification of patients meeting Code Stroke criteria from free-text emergency department triage notes. The analysis included 3,023 patient presentations over one month (September-October 2023) at an Australian hospital. The best-performing model (Qwen 2.5 14B) achieved sensitivity of 0.890, specificity of 0.993, positive predictive value of 0.858, and negative predictive value of 0.995. Of 140 Code Stroke activations, 83 patients (59.2%) had confirmed stroke diagnosis, 16 patients (11.4%) underwent endovascular thrombectomy, and 4 patients (2.9%) received thrombolysis. The algorithm operates without internet connectivity and without retraining on patient data, supporting its practical feasibility for emergency department integration.