Real-Time or Near Real-Time Persisting Daily Healthcare Data Into HDFS and ElasticSearch Index Inside a Big Data Platform

Authors:
  • YCYi Chen
    Yi Chen
  • BBBrian N. Brownlow
    Brian N. Brownlow
  • PKPradip P. Kanjamala
    Pradip P. Kanjamala
  • CACarlos A. Garcia Arredondo
    Carlos A. Garcia Arredondo
  • BRBryan L. Radspinner
    Bryan L. Radspinner
  • MRMatthew A. Raveling
    Matthew A. Raveling
Publisher website
Google Scholar
Abstract
Mayo Clinic (MC) healthcare generates a large number of HL7 V2 messages – 0.7-1.1 million on weekends and 1.7-2.2 million on business days at present. With multiple RDBMS-based systems, such a large volume of HL7 messages still cannot be real-time or near-real-time stored, analyzed and retrieved for enterprise-level clinic and non-clinic usage. To determine if Big Data technology coupled with ElasticSearch technology can satisfy MC daily healthcare needs for HL7 message processing, a BigData platform was developed to contain two identical Hadoop clusters (TDH1.3.2 version) – each containing an ElasticSearch cluster and instances of a storm topology – MayoTopology for processing HL7 messages on MC ESB queues into an ElasticSearch index and the HDFS. The implemented BigData platform can process 62±4 million HL7 messages per day while the ElasticSearch index can provide ultra-fast free-text searching at a speed level of 0.2-second per query on an index containing a dataset of 25 million HL7-derived-JSON-documents. The results suggest that the implemented BigData platform exceeds MC enterprise-level patient-care needs.

This publication has 13 references indexed in Scilit: