Mind the gap: emergent clinical risk at the interface of two individually safe AI systems in a multilingual ambient scribe

The study examined clinical risk in notes generated by a multilingual AI scribe combining speech recognition and text generation. The research included 77 languages and five complexity levels from simple outpatient consultations to complex multidisciplinary team handovers. When testing only note generation from accurate transcripts, the error rate was low (3.4% serious errors). However, when testing the entire system including speech recognition, the error rate increased significantly, especially for more complex cases (up to 24%) and languages with limited available data (up to 15%). Most errors (87%) originated from speech recognition rather than note generation. The study demonstrated that individual system components can be safe, but their combination creates new risks that are not visible when testing components separately.