Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Masafumi Nishimura has a total of 33 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Simultaneous Speech and Eating Behavior Recognition Using Data Augmentation and Two-Stage Fine-Tuning

    by Toshihiro Tsukagoshi, Masafumi Nishida, Masafumi Nishimura

    Speaking and eating are essential components of health management. To enable the daily monitoring of these behaviors, systems capable of simultaneously recognizing speech and eating behaviors are required. However, due to the distinct acoustic and contextual characteristics of these two domains, achieving high-precision integrated recognition remains underexplored. In this study, we propose a method that combines data augmentation through synthetic data creation with a two-stage fine-tuning approach tailored to the complexity of domain adaptation. By concatenating speech and eating sounds of varying lengths and sequences, we generated training data that mimic real-world environments where speech and eating behaviors co-exist. Additionally, efficient model adaptation was achieved through two-stage fine-tuning of the self-supervised learning model. The experimental evaluations demonstrate that the proposed method maintains speech recognition accuracy while achieving high detection performance for eating behaviors, with an F1 score of 0.918 for chewing detection and 0.926 for swallowing detection. These results underscore the potential of using voice recognition technology for daily health monitoring.


  • Open Access

    Hate Speech Detection by Using Rationales for Judging Sarcasm

    by Maliha Binte Mamun, Takashi Tsunakawa, Masafumi Nishida, Masafumi Nishimura

    The growing number of social media users has impacted the rise in hate comments and posts. While extensive research in hate speech detection attempts to combat this phenomenon by developing new datasets and detection models, reconciling classification accuracy with broader decision-making metrics like plausibility and faithfulness remains challenging. As restrictions on social media tighten to stop the spread of hate and offensive content, users have adapted by finding new approaches, often camouflaged in the form of sarcasm. Therefore, dealing with new trends such as the increased use of emoticons (negative emoticons in positive sentences) and sarcastic comments is necessary. This paper introduces sarcasm-based rationale (emoticons or portions of text that indicate sarcasm) combined with hate/offensive rationale for better detection of hidden hate comments/posts. A dataset was created by labeling texts and selecting rationale based on sarcasm from the existing benchmark hate dataset, HateXplain. The newly formed dataset was then applied in the existing state-of-the-art model. The model’s F1-score increased by 0.01 when using sarcasm rationale with hate/offensive rationale in a newly formed attention proposed in the data’s preprocessing step. Also, with the new data, a significant improvement was observed in explainability metrics such as plausibility and faithfulness.


  • Open Access

    Information-gathering dialog system using acoustic features and user’s motivation

    by Ryota Togai, Takashi Tsunakawa, Masafumi Nishimura

    Recently, as society continues to age, automation of watching elderly people who live apart from their families has been gradually expected. However, we must prevent them from losing their purpose in life, which declines due to lack of communication. Thus, a chat dialog system has attracted widespread attention as a method that achieves both problems: keeping their purpose in life and watching their daily lives. Unlike a task-oriented dialog system, a chat dialog system has explicitly no task to accomplish and makes a conversation to continue communication with the users. Keeping a conversation is essential for elderly people who are mostly unfamiliar with digital devices. Moreover, conversing daily on the chat dialog system provides the opportunity to collect information for their care. This study realizes an information-gathering dialog system, a chat dialog system that collects healthcare information of elderly people. Furthermore, we use the nonverbal acoustic features from their speech, since automatic speech recognition is not necessarily accurate in current systems. This paper illustrates the effectiveness of two important elements, topic change for keeping the talking user motivated with the dialog system and motivation estimation, for attaining an information-gathering dialog system using nonverbal acoustic features.

  • Open Access

    Screening of Mild Cognitive Impairment Through Conversations With Humanoid Robots: Exploratory Pilot Study

    by Kenta Yoshii, Daiki Kimura, Akihiro Kosugi, Kaoru Shinkawa, Toshiro Takase, Masatomo Kobayashi, Yasunori Yamada, Miyuki Nemoto, Ryohei Watanabe, Miho Ota, et al.

    Background: The rising number of patients with dementia has become a serious social problem worldwide. To help detect dementia at an early stage, many studies have been conducted to detect signs of cognitive decline by prosodic and acoustic features. However, many of these methods are not suitable for everyday use as they focus on cognitive function or conversational speech during the examinations. In contrast, conversational humanoid robots are expected to be used in the care of older people to help reduce the work of care and monitoring through interaction. Objective: This study focuses on early detection of mild cognitive impairment (MCI) through conversations between patients and humanoid robots without a specific examination, such as neuropsychological examination. Methods: This was an exploratory study involving patients with MCI and cognitively normal (CN) older people. We collected the conversation data during neuropsychological examination (Mini-Mental State Examination [MMSE]) and everyday conversation between a humanoid robot and 94 participants (n=47, 50%, patients with MCI and n=47, 50%, CN older people). We extracted 17 types of prosodic and acoustic features, such as the duration of response time and jitter, from these conversations. We conducted a statistical significance test for each feature to clarify the speech features that are useful when classifying people into CN people and patients with MCI. Furthermore, we conducted an automatic classification experiment using a support vector machine (SVM) to verify whether it is possible to automatically classify these 2 groups by the features identified in the statistical significance test. Results: We obtained significant differences in 5 (29%) of 17 types of features obtained from the MMSE conversational speech. The duration of response time, the duration of silent periods, and the proportion of silent periods showed a significant difference (P<.001) and met the reference value r=0.1 (small) of the effect size. Additionally, filler periods (P<.01) and the proportion of fillers (P=.02) showed a significant difference; however, these did not meet the reference value of the effect size. In contrast, we obtained significant differences in 16 (94%) of 17 types of features obtained from the everyday conversations with the humanoid robot. The duration of response time, the duration of speech periods, jitter (local, relative average perturbation [rap], 5-point period perturbation quotient [ppq5], difference of difference of periods [ddp]), shimmer (local, amplitude perturbation quotient [apq]3, apq5, apq11, average absolute differences between the amplitudes of consecutive periods [dda]), and F0cov (coefficient of variation of the fundamental frequency) showed a significant difference (P<.001). In addition, the duration of response time, the duration of silent periods, the filler period, and the proportion of fillers showed significant differences (P<.05). However, only jitter (local) met the reference value r=0.1 (small) of the effect size. In the automatic classification experiment for the classification of participants into CN and MCI groups, the results showed 66.0% accuracy in the MMSE conversational speech and 68.1% accuracy in everyday conversations with the humanoid robot. Conclusions: This study shows the possibility of early and simple screening for patients with MCI using prosodic and acoustic features from everyday conversations with a humanoid robot with the same level of accuracy as the MMSE.


  • Open Access

    Screening of Mild Cognitive Impairment Through Conversations with Humanoid Robots: Preliminary Study (Preprint)

    by Kenta Yoshii, Daiki Kimura, Akihiro Kosugi, Kaoru Shinkawa, Toshiro Takase, Masatomo Kobayashi, Yasunori Yamada, Miyuki Nemoto, Ryohei Watanabe, Miho Ota, et al.

    BACKGROUND The rising number of dementia patients has become a serious social problem worldwide. To detect dementia in patients at an early stage, many studies have been made to detect signs of cognitive decline by prosodic and acoustic features. However, many of these methods are not suitable for everyday use as they are focusing examinations for cognitive functions or conversational speech during the examinations. On the other hand, conversational humanoid robots are expected to be used in elderly care to help to reduce the work of care and monitoring through interaction. OBJECTIVE For early detection of mild cognitive impairment (MCI) through conversations between elderly people and humanoid robots without specific examinations such as neuropsychological examination. METHODS We collected the conversation data during neuropsychological examination (MMSE) and daily conversation with a humanoid robot from a total of 94 participants (47 cognitively normal and 47 patients with MCI). We extracted 17 types of prosodic and acoustic features such as duration of response time and jitter from these conversations. We conducted a statistical significance test for each feature to clarify the speech features that are useful when classifying people into cognitively normal and MCI patients. Furthermore, we conducted an automatic classification experiment using SVM to verify whether possibility of automatically classify these two group from conversation with humanoid robot by the features identified in statistical significance test. RESULTS We obtained significant differences in 5 out of 17 types of features from the MMSE conversational speech. Duration of response time, duration of silence periods and proportion of silent periods were shown significant difference (p<.001) and met the reference value r=0.1 (Small) of the effect size. Additionally, filler periods(p<.01) and proportion of fillers (p=0.02) were shown significant difference, however these were not met the reference value of the effect size. In contrast, we obtained significant differences in 16 out of 17 types from the speech in the everyday conversations with the humanoid robot. Duration of response time, duration of speech periods, jitter (local, rap, ppq5, ddp), shimmer (local, apq3, apq5, apq11, dda) and F0cov (Coefficient of variation of the fundamental frequency) were shown significant difference(p<.001). In addition, duration of response time, duration of silence periods, filler period and proportion of fillers were significant difference(p<.05). However, only jitter(local) was met the reference value r=0.1 (Small) of effect size. In automatic classification experiment for classification of MCI patients and cognitively normal elderly adults, the results were shown 66.0% accuracy when MMSE conversational speech and 68.1% in daily conversation with humanoid robot. CONCLUSIONS This study suggests the possibility of early and simple screening for patients with MCI by using prosodic and acoustic features from daily conversation with a humanoid robot with the same level of accuracy as human conversations in MMSE tests.

  • Automatic Cough Detection Using Deep Neural Network and a Throat Microphone

    by Bhuiyan Rakib, Masafumi Nishimura

    Coughing is a common symptom of a variety of respiratory and non-respiratory illnesses. Many studies were conducted to detect cough sounds. This study simulates real-life data, including coughing, speaking, eating-related sounds such as chewing and swallowing, and movement-related sounds related to coughing. Furthermore, the cough data in this study was genuine, generated by using a nebulizer. Prior studies used simulated cough data. We attempted to show the usability of the throat microphone and the feasibility of automatic detection in a real-life environment. As a result of cough detection in a real environment, we obtained 0.82 in F- measure using multi-layer long short-term memory (LSTM) and the throat microphone.

  • Open Access

    Automatic Detection of Chewing and Swallowing

    by Akihiro Nakamura, Takato Saito, Daizo Ikeda, Ken Ohta, Hiroshi Mineno, Masafumi Nishimura

    A series of eating behaviors, including chewing and swallowing, is considered to be crucial to the maintenance of good health. However, most such behaviors occur within the human body, and highly invasive methods such as X-rays and fiberscopes must be utilized to collect accurate behavioral data. A simpler method of measurement is needed in healthcare and medical fields; hence, the present study concerns the development of a method to automatically recognize a series of eating behaviors from the sounds produced during eating. The automatic detection of left chewing, right chewing, front biting, and swallowing was tested through the deployment of the hybrid CTC/attention model, which uses sound recorded through 2ch microphones under the ear and weak labeled data as training data to detect the balance of chewing and swallowing. N-gram based data augmentation was first performed using weak labeled data to generate many weak labeled eating sounds to augment the training data. The detection performance was improved through the use of the hybrid CTC/attention model, which can learn the context. In addition, the study confirmed a similar detection performance for open and closed foods.


  • A Study for Detecting Mild Cognitive Impairment by Analyzing Conversations with Humanoid Robots

    by Kenta Yoshii, Masafumi Nishimura, Daiki Kimura, Akihiro Kosugi, Kaoru Shinkawa, Toshiro Takase, Masatomo Kobayashi, Yasunori Yamada, Miyuki Nemoto, Ryohei Watanabe, et al.

    Many studies have attempted to detect dementia in patients early by identifying certain prosodic and acoustic features during neuropsychological examinations. Conversational humanoid robots are expected to be used in elderly care to help increase their quality of life through interactions. If we can detect cognitive impairments from the elderly's conversations with a robot, it would aid in early detection of dementia. In this study, we collected conversational data to simulate daily conversations with humanoid robots and compared them between healthy older adults and patients with mild cognitive impairments. This study evaluated multiple speech features and observed significant differences in answer length, response length, duration of utterance, jitter, and shimmer. Thus, the study results suggest the possibility of early detection of dementia through conversations with a humanoid robot.


  • Continuous digits recognition leveraging invariant structure

    by Masayuki Suzuki, Gakuto Kurata, Masafumi Nishimura, Nobuaki Minematsu