Philipp Achenbach

Mobile fitness applications provide the opportunity to show users real-time feedback on their current fitness activity. For such applications, it is essential to accurately track the user’s current fitness activity using available mobile sensors, such as inertial measurement units (IMUs). Convolutional neural networks (CNNs) have been shown to produce strong results in different time series classification tasks, including the recognition of daily living activities. However, fitness activities can present unique challenges to the human activity recognition task (HAR), including greater similarity between individual activities and fewer available data for model training. In this paper, we evaluate the applicability of CNNs to the fitness activity recognition task (FAR) using IMU data and determine the impact of input data size and sensor count on performance. For this purpose, we adapted three existing CNN architectures to the FAR task and designed a fourth CNN variant, which we call the scaling fully convolutional network (Scaling-FCN). We designed a preprocessing pipeline and recorded a running exercise data set with 20 participants, in which we evaluated the respective recognition performances of the four networks, comparing them with three traditional machine learning (ML) methods commonly used in HAR. Although CNN architectures achieve at least 94% test accuracy in all scenarios, two traditional ML architectures surpass them in the default scenario, with support vector machines (SVMs) achieving 99.00 ± 0.34% test accuracy. The removal of all sensors except one foot sensor reduced the performance of traditional ML architectures but improved the performance of CNN architectures on our data set, with our Scaling-FCN reaching the highest accuracy of 99.86 ± 0.11% on the test set. Our results suggest that CNNs are generally well suited for fitness activity recognition, and noticeable performance improvements can be achieved if sensors are dropped selectively, although traditional ML architectures can still compete with or even surpass CNNs when favorable input data are utilized.

Human-to-human communication via the computer is mainly carried out using a keyboard or microphone. In the field of virtual reality (VR), where the most immersive experience possible is desired, the use of a keyboard contradicts this goal, while the use of a microphone is not always desirable (e.g., silent commands during task-force training) or simply not possible (e.g., if the user has hearing loss). Data gloves help to increase immersion within VR, as they correspond to our natural interaction. At the same time, they offer the possibility of accurately capturing hand shapes, such as those used in non-verbal communication (e.g., thumbs up, okay gesture, …) and in sign language. In this paper, we present a hand-shape recognition system using Manus Prime X data gloves, including data acquisition, data preprocessing, and data classification to enable nonverbal communication within VR. We investigate the impact on accuracy and classification time of using an outlier detection and a feature selection approach in our data preprocessing. To obtain a more generalized approach, we also studied the impact of artificial data augmentation, i.e., we created new artificial data from the recorded and filtered data to augment the training data set. With our approach, 56 different hand shapes could be distinguished with an accuracy of up to 93.28%. With a reduced number of 27 hand shapes, an accuracy of up to 95.55% could be achieved. The voting meta-classifier (VL2) proved to be the most accurate, albeit slowest, classifier. A good alternative is random forest (RF), which was even able to achieve better accuracy values in a few cases and was generally somewhat faster. outlier detection was proven to be an effective approach, especially in improving the classification time. Overall, we have shown that our hand-shape recognition system using data gloves is suitable for communication within VR.

Augmented reality (AR) provides an opportunity for mobile fitness applications to show users real-time feedback on their current fitness activity. For such applications, it is essential to accurately track the user’s current fitness activity using available mobile sensors, such as inertial measurement units (IMUs). Convolutional neural networks (CNNs) have been shown to produce strong results in different time series classification tasks, including the recognition of activities of daily living. However, fitness activities can present unique challenges to the human activity recognition task (HAR), including greater similarity between individual activities and fewer available data for model training. In this paper, we evaluate the applicability of CNNs to the fitness activity recognition task (FAR) using IMU data and determine the impact of input data size and sensor count on performance. For this purpose, we adapted three existing CNN architectures to the FAR task and designed a fourth CNN variant, which we call the scaling fully convolutional network (Scaling-FCN). We designed a preprocessing pipeline and recorded a running exercise data set with 20 participants, in which we evaluated the respective recognition performances of the four networks, comparing them with three traditional machine learning methods commonly used in HAR. On our data set, all CNN architectures significantly outperformed traditional machine learning methods, reaching up to 97.14±1.36% classification accuracy on the test set for ResNet. Whereas a reduced input data size invariably led to a performance loss, removing specific sensors improved the performance of all CNN architectures, with our Scaling-FCN reaching the highest accuracy of 99.86±0.11%. Our results suggest that CNNs are generally well suited for fitness activity recognition, and noticeable performance improvements can be achieved if sensors are dropped selectively.

Human-to-human communication via the computer is mainly done using a keyboard or microphone. In the field of Virtual Reality (VR), where the most immersive experience possible is desired, the use of a keyboard contradicts this goal, while the use of a microphone is not always desirable (e.g. silent commands during task force training) or simply not possible (e.g. if the user has a hearing loss). Data gloves help to increase immersion within the VR as they correspond to our natural interaction. At the same time, they offer the possibility to accurately capture hand shapes, such as those used in non-verbal communication (e.g. thumbs up, okay gesture, ...) and in sign language. In this paper, we present a hand shape recognition system using Manus Prime X data gloves, including data acquisition, data preprocessing, and data classification to enable nonverbal communication within VR. We investigate the impact on accuracy and classification time of using an Outlier Detection and a Feature Selection approach in our data preprocessing. To obtain a more generalized approach, we also studied the impact of artificial Data Augmentation, i.e., we create new artificial data from the recorded and filtered data to augment the training dataset. With our approach, 56 different hand shapes could be distinguished with an accuracy of up to 93.28%. With a reduced number of 27 hand shapes, an accuracy of up to 95.55% could be achieved. Voting Meta-Classifier (VL2) has proven to be the most accurate, albeit slowest, classifier. A good alternative is Random Forest (RF), which was even able to achieve better accuracy values in a few cases and was generally somewhat faster. Outlier Detection has proven to be a effective approach, especially in improving classification time. Overall, we have shown that our hand shape recognition system using data gloves is suitable for communication within VR.

Hand gestures play an important role in human communication, particularly when auditory communication is limited. Akin to speech recognition, hand gesture recognition can therefore be a useful tool to facilitate communication and for more immersive computer interaction. In this paper, we examine the mobile recognition of hand gestures using data recorded with sensor gloves. We design a system based on Support Vector Machines (SVM), capable of recognizing 5 different hand gestures. In an experiment with 11 participants, we determine applicable hyperparameters based on performance on the training set which translates into 100% classification accuracy on the test set. In an additional practical experiment with 9 participants, our system achieves up to 98% in a personalized and up to 87.5% in a generalized model setting.