A blinded, counterbalanced rater design for evaluating AI-assisted summarisation of tertiary clinical genomics reports: methodology of the QNOMX-VHIR-CPSP-001 Phase 1 study

The QNOMX-VHIR-CPSP-001 Phase 1 study is a clinical performance study designed to evaluate artificial intelligence-assisted systems for summarizing tertiary clinical genomics reports in oncology. Tertiary genomic reports contain complex molecular findings that oncologists must read and interpret, and manual summarization is time-consuming and variable. The study uses qualified raters who assess summaries generated by AI systems and manual workflows against the original report using the six-dimensional Quality Summary Index (QSI) scale. Evaluation is conducted in a blinded manner with counterbalancing and a 14-day washout period between study periods. The primary endpoints are two composite measures - content and presentation - analyzed for non-inferiority using a Bayesian hierarchical model. The study is conducted at Vall d'Hebron Institut de Recerca under ISO 20916:2019 standard and aims to establish a replicable protocol for evaluating AI-assisted summarization in genomic reporting.