David Bernhauer

One of the characteristics of big data is its internal complexity and variety manifested in many types of datasets that are to be managed, searched, or analyzed. In their natural forms, some data entities are unstructured, such as texts or multimedia objects, while some are structured but too complex (e.g., high-dimensional tabular data). Due to the many different forms of data managed in many domain-specific problems, there are many different data representations used – tailored to a specific data form, domain and task. In this paper, we propose a framework for universal visual representations of complex data. The desired property of the visualizations is the ability to visually encode the semantic features of the original data. Hence, processing of visualizations (images) by generic deep learning models results in deep feature vectors that could be uniformly used in standard data retrieval/analytics tasks. Specifically, we develop a semi-automated transfer learning pipeline for transformation of input arbitrary tabular data into visual representations. The visual representations serve for data analytics tasks performed by human users as well as serve for universal data representations used in machine learning models for automated tasks. We show in large study that visual representations of complex data are effective in a number of domains while we also propose a recommender to help with the parameterization of the entire pipeline for certain domains and use cases. In summary, the proposed framework enables rapid prototyping of data representations (in an arbitrary domain) using a shared concept – visual representations applicable in data analytics using generic deep learning models.

In data science and content-based retrieval, we find many domain-specific techniques that employ a data processing pipeline with two fundamental steps. First, data entities are represented by some visualizations, while in the second step, the visualizations are used with a machine learning model to extract deep features. Deep convolutional neural networks (DCNN) became the standard and reliable choice. The purpose of using DCNN is either a specific classification task or just a deep feature representation of visual data for additional processing (e.g., similarity search). Whereas the deep feature extraction is a domain-agnostic step in the pipeline (inference of an arbitrary visual input), the visualization design itself is domain-dependent and ad hoc for every use case. In this paper, we survey and analyze many instances of data visualizations used with deep learning models (mostly DCNN) for domain-specific tasks. Based on the analysis, we synthesize a taxonomy that provides a systematic overview of visualization techniques suitable for usage with the models. The aim of the taxonomy is to enable the future generalization of the visualization design process to become completely domain-agnostic, leading to the automation of the entire feature extraction pipeline. As the ultimate goal, such an automated pipeline could lead to universal deep feature data representations for content-based retrieval.