Evaluation of a UMLS Auditing Process of Semantic Type Assignments.

Huanying Helen Gu, George Hripcsak, Yan Chen, C. Paul Morrey, Gai Elhanan, James Cimino, James Geller, Yehoshua Perl

Research output: Contribution to journalArticle

15 Scopus citations

Abstract

The UMLS is a terminological system that integrates many source terminologies. Each concept in the UMLS is assigned one or more semantic types from the Semantic Network, an upper level ontology for biomedicine. Due to the complexity of the UMLS, errors exist in the semantic type assignments. Finding assignment errors may unearth modeling errors. Even with sophisticated tools, discovering assignment errors requires manual review. In this paper we describe the evaluation of an auditing project of UMLS semantic type assignments. We studied the performance of the auditors who reviewed potential errors. We found that four auditors, interacting according to a multi-step protocol, identified a high rate of errors (one or more errors in 81% of concepts studied) and that results were sufficiently reliable (0.67 to 0.70) for the two most common types of errors. However, reliability was low for each individual auditor, suggesting that review of potential errors is resource-intensive.

Original languageEnglish (US)
Pages (from-to)294-298
Number of pages5
JournalAMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium
StatePublished - 2007
Externally publishedYes

    Fingerprint

All Science Journal Classification (ASJC) codes

  • Medicine(all)

Cite this