Piloting a data-extraction form for verbatim terminology: a blinded inter-reviewer agreement study before a surgical evidence map

The study examined whether surgical terms can be reliably extracted from scientific articles using a specialized form. Two independent reviewers extracted terms from 11 articles in English and French focused on deep endometriosis. The primary measure was the Jaccard index, which needed to reach at least 0.70. Results showed that the form failed to meet the target with a mean value of 0.581. Of 92 extracted terms, 55 matched exactly while 37 did not, due to reasons such as source coverage, text span extent, and language comprehension. After harmonization and form revision, agreement improved to 0.913. The study recommends that forms for extracting verbatim text should be tested using set-overlap measures and a predefined acceptability threshold.