BSBrett R. South

Brett R. South

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Brett R. South has a total of 39 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Normalizing acronyms and abbreviations to aid patient understanding of clinical texts: ShARe/CLEF eHealth Challenge 2013, Task 2

    by Danielle L. Mowery, Brett R. South, Lee Christensen, Jianwei Leng, Laura-Maria Peltonen, Sanna Salanterä, Hanna Suominen, David Martinez, Sumithra Velupillai, Noémie Elhadad, et al.

    The ShARe/CLEF eHealth challenge lab aims to stimulate development of natural language processing and information retrieval technologies to aid patients in understanding their clinical reports. In clinical text, acronyms and abbreviations, also referenced as short forms, can be difficult for patients to understand. For one of three shared tasks in 2013 (Task 2), we generated a reference standard of clinical short forms normalized to the Unified Medical Language System. This reference standard can be used to improve patient understanding by linking to web sources with lay descriptions of annotated short forms or by substituting short forms with a more simplified, lay term. In this study, we evaluate 1) accuracy of participating systems’ normalizing short forms compared to a majority sense baseline approach, 2) performance of participants’ systems for short forms with variable majority sense distributions, and 3) report the accuracy of participating systems’ normalizing shared normalized concepts between the test set and the Consumer Health Vocabulary, a vocabulary of lay medical terms. The best systems submitted by the five participating teams performed with accuracies ranging from 43 to 72 %. A majority sense baseline approach achieved the second best performance. The performance of participating systems for normalizing short forms with two or more senses with low ambiguity (majority sense greater than 80 %) ranged from 52 to 78 % accuracy, with two or more senses with moderate ambiguity (majority sense between 50 and 80 %) ranged from 23 to 57 % accuracy, and with two or more senses with high ambiguity (majority sense less than 50 %) ranged from 2 to 45 % accuracy. With respect to the ShARe test set, 69 % of short form annotations contained common concept unique identifiers with the Consumer Health Vocabulary. For these 2594 possible annotations, the performance of participating systems ranged from 50 to 75 % accuracy. Short form normalization continues to be a challenging problem. Short form normalization systems perform with moderate to reasonable accuracies. The Consumer Health Vocabulary could enrich its knowledge base with missed concept unique identifiers from the ShARe test set to further support patient understanding of unfamiliar medical terms.


  • Open Access

    Extending an Uncertainty Taxonomy for Suspected Pneumonia Case Review

    by Brett R. South, Heidi S. Kramer, Melissa Castine, Danielle Mowery, Barbara Jones, Wendy Chapman

    We present our efforts towards extending a taxonomy of uncertainty for suspected pneumonia case review. We manually annotated a random sample of 200 cases of suspected pneumonia at the case level, classifying cases as suspected, unlikely or cannot be determined. Within each document, evidence snippets were annotated that support, refute or are uncertain for suspected pneumonia. Finally we conducted a thematic analysis to classify mentions marked as uncertain applying a taxonomy of uncertainty. We observed substantial variability in human agreement at the evidence snippet level but high agreement for classification of evidence snippets to a taxonomy of uncertainty.

  • Open Access

    Classifying Supporting, Refuting, or Uncertain Evidence for Pneumonia Case Review

    by Brett R. South, Heidi S. Kramer Heidi S. Kramer, Barbara Jones, Melissa Tharp, Wendy Chapman

    We found substantial annotator variability in identifying supporting, refuting, or uncertain evidence for the diagnosis of pneumonia in clinical text. Future work will expand these methods to a larger case sample and incorporate a more formal linguistic analysis to identify specific lexical cues thereby extending existing taxonomies of uncertainty and improving automated NLP algorithms.

  • Open Access

    Evaluating the effects of machine pre-annotation and an interactive annotation interface on manual de-identification of clinical text

    by Brett R. South, Danielle Mowery, Ying Suo, Jianwei Leng, Óscar Ferrández, Stephane M. Meystre, Wendy W. Chapman


    • 18 March 2013
      • Vol. 2013, 177

    How much does automatic text de-identification impact clinical problems, tests, and treatments?

    by Stéphane M Meystre, Oscar Ferrández, Brett R South, Shuying Shen, Matthew H Samore

    Clinical text de-identification can potentially overlap with clinical information such as medical problems or treatments, therefore causing this information to be lost. In this study, we focused on the analysis of the overlap between the 2010 i2b2 NLP challenge concept annotations, with the PHI annotations of our best-of-breed clinical text de-identification application. Overall, 0.81% of the annotations overlapped exactly, and 1.78% partly overlapped.

    • 1 January 2013
      • Vol. 192, 1213

    What a catch! traits that define good annotators.

    by Tyler B Forbush, Shuying Shen, Brett R South, Scott L Duvalla

    Human annotation and chart review is an important process in biomedical informatics research, but which humans are best suited for the job? Domain expertise, such as medical or linguistic knowledge is desirable, but other factors may be equally important. The University of Utah has a group of 20+ reviewers with backgrounds in medicine and linguistics, and 10 key traits have surfaced in those best able to annotate quickly and with high quality. To identify reviewers with these key traits, we created a hiring process that includes interviewing candidates, testing their medical and linguistic knowledge, and having them complete an annotation exercise on realistic medical text. Each step is designed to assess the key traits and allow the investigator to choose the skill set required for each project.

  • Open Access

    Evaluating current automatic de-identification methods with Veteran’s health administration clinical documents

    by Oscar Ferrández, Brett R South, Shuying Shen, F Jeffrey Friedlin, Matthew H Samore, Stéphane M Meystre

    The increased use and adoption of Electronic Health Records (EHR) causes a tremendous growth in digital information useful for clinicians, researchers and many other operational purposes. However, this information is rich in Protected Health Information (PHI), which severely restricts its access and possible uses. A number of investigators have developed methods for automatically de-identifying EHR documents by removing PHI, as specified in the Health Insurance Portability and Accountability Act “Safe Harbor” method.


  • Top Cited Papers
    Open Access

    Automatic de-identification of textual documents in the electronic health record: a review of recent research

    by Stephane M Meystre, F Jeffrey Friedlin, Brett R South, Shuying Shen, Matthew H Samore

    In the United States, the Health Insurance Portability and Accountability Act (HIPAA) protects the confidentiality of patient data and requires the informed consent of the patient and approval of the Internal Review Board to use data for research purposes, but these requirements can be waived if data is de-identified. For clinical data to be considered de-identified, the HIPAA "Safe Harbor" technique requires 18 data elements (called PHI: Protected Health Information) to be removed. The de-identification of narrative text documents is often realized manually, and requires significant resources. Well aware of these issues, several authors have investigated automated de-identification of narrative text documents from the electronic health record, and a review of recent research in this domain is presented here.


  • Open Access

    Developing a manually annotated clinical document corpus to identify phenotypic information for inflammatory bowel disease

    by Brett R South, Shuying Shen, Makoto Jones, Jennifer Garvin, Matthew H Samore, Wendy W Chapman, Adi V Gundlapalli

    Natural Language Processing (NLP) systems can be used for specific Information Extraction (IE) tasks such as extracting phenotypic data from the electronic medical record (EMR). These data are useful for translational research and are often found only in free text clinical notes. A key required step for IE is the manual annotation of clinical corpora and the creation of a reference standard for (1) training and validation tasks and (2) to focus and clarify NLP system requirements. These tasks are time consuming, expensive, and require considerable effort on the part of human reviewers.


    • 1 March 2009
      • Vol. 2009, 1-32

    Developing a Manually Annotated Clinical Document Corpus to Identify Phenotypic Information for Inflammatory Bowel Disease

    by Brett R South, Shuying Shen, Makoto Jones, Jennifer Garvin, Matthew H Samore, Wendy W Chapman, Adi V Gundlapalli

    Natural Language Processing (NLP) systems can be used for specific Information Extraction (IE) tasks such as extracting phenotypic data from the electronic medical record (EMR). These data are useful for translational research and are often found only in free text clinical notes. A key required step for IE is the manual annotation of clinical corpora and the creation of a reference standard for (1) training and validation tasks and (2) to focus and clarify NLP system requirements. These tasks are time consuming, expensive, and require considerable effort on the part of human reviewers. Using a set of clinical documents from the VA EMR for a particular use case of interest we identify specific challenges and present several opportunities for annotation tasks. We demonstrate specific methods using an open source annotation tool, a customized annotation schema, and a corpus of clinical documents for patients known to have a diagnosis of Inflammatory Bowel Disease (IBD). We report clinician annotator agreement at the document, concept, and concept attribute level. We estimate concept yield in terms of annotated concepts within specific note sections and document types. Annotator agreement at the document level for documents that contained concepts of interest for IBD using estimated Kappa statistic (95% CI) was very high at 0.87 (0.82, 0.93). At the concept level, F-measure ranged from 0.61 to 0.83. However, agreement varied greatly at the specific concept attribute level. For this particular use case (IBD), clinical documents producing the highest concept yield per document included GI clinic notes and primary care notes. Within the various types of notes, the highest concept yield was in sections representing patient assessment and history of presenting illness. Ancillary service documents and family history and plan note sections produced the lowest concept yield. Challenges include defining and building appropriate annotation schemas, adequately training clinician annotators, and determining the appropriate level of information to be annotated. Opportunities include narrowing the focus of information extraction to use case specific note types and sections, especially in cases where NLP systems will be used to extract information from large repositories of electronic clinical note documents.