AIAkinori Ito

Akinori Ito

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Akinori Ito has a total of 140 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Multimodal Manga Tag Estimation Using Image, Text, and Emotion Features

    by Rei Takahashi, Akinori Ito, Takashi Nose

    Manga conveys its content through multiple modalities, including page images, dialogue, and emotional progression throughout a story. This study investigates how image, text, and emotion information contribute to volume-level manga tag estimation and whether combining these modalities improves estimation performance. We estimate three tag types–publication decade, target readership, and genre–for 109 volumes in the Manga109 dataset. We extract image features from two-page spread images using CLIP, derive text features from the lines contained in each spread, and obtain emotion features from valence–arousal emotion arcs generated by a large multimodal model. We examine logistic regression, random forest, and support vector machine classifiers, and combine their class probabilities using weighted late fusion. The results show that image information is the most effective single modality for publication decade and target readership, whereas text information is the most effective for genre. Multimodal fusion did not improve publication-decade or target-readership estimation, whereas combining text and emotion features improved genre estimation. A genre-level analysis further shows that the most effective modality varies across genres: text is effective for many content-based genres, image information contributes to visually distinctive genres, and emotion information complements text for several narrative genres. These results show the importance of selecting and combining modalities based on tag type and individual genre.

  • Japanese Shadowing Training Using Synchronized Partial Captions

    by Syuyu Fang, Akinori Ito, Takashi Nose

    Shadowing is a language practice method where a learner listens to a native speaker's speech and speaks the same content immediately. Speech shadowing is a popular languagelearning technique that is effective for developing listening and speaking skills. This paper proposes using synchronized partial captions (PSC) in the shadowing training. In PSC, the text is synchronized to the speech at the word level, and only selected words are shown in the caption to aid listening comprehension, while the rest are masked to keep the learner listening to the speech. We conducted an experiment where 59 learners exercised shadowing in one of three learning methods: no caption, full caption, and partial caption. The experimental result shows that training in PSC conditions rapidly improves learners' proficiency.

  • Open Access

    The Development of an Emotional Embodied Conversational Agent and the Evaluation of the Effect of Response Delay on User Impression

    by Simon Christophe Jolibois, Akinori Ito, Takashi Nose

    Embodied conversational agents (ECAs) are autonomous interaction interfaces designed to communicate with humans. This study investigates the impact of response delays and emotional facial expressions of ECAs on user perception and engagement. The motivation for this study stems from the growing integration of ECAs in various sectors, where their ability to mimic human-like interactions significantly enhances user experience. To this end, we developed an ECA with multimodal emotion recognition, both with voice and facial feature recognition and emotional facial expressions of the agent avatar. The system generates answers in real time based on media content. The development was supported by a case study of artwork images with the agent playing the role of a museum curator, where the user asks the agent for information on the artwork. We evaluated the developed system in two aspects. First, we investigated how the delay in an agent’s responses influences user satisfaction and perception. Secondly, we explored the role of emotion in an ECA’s face in shaping the user’s perception of responsiveness. The results showed that the longer response delay negatively impacted the user’s perception of responsiveness when the ECA did not express emotion, while the emotional expression improved the responsiveness perception.


  • Open Access

    Robust Human Tracking Using a 3D LiDAR and Point Cloud Projection for Human-Following Robots

    by Sora Kitamoto, Yutaka Hiroi, Kenzaburo Miyawaki, Akinori Ito

    Human tracking is a fundamental technology for mobile robots that work with humans. Various devices are used to observe humans, such as cameras, RGB-D sensors, millimeter-wave radars, and laser range finders (LRF). Typical LRF measurements observe only the surroundings on a particular horizontal plane. Human recognition using an LRF has a low computational load and is suitable for mobile robots. However, it is vulnerable to variations in human height, potentially leading to detection failures for individuals taller or shorter than the standard height. This work aims to develop a method that is robust to height differences among humans using a 3D LiDAR. We observed the environment using a 3D LiDAR and projected the point cloud onto a single horizontal plane to apply a human-tracking method for 2D LRFs. We investigated the optimal height range of the point clouds for projection and found that using 30% of the point clouds from the top of the measured person provided the most stable tracking. The results of the path-following experiments revealed that the proposed method reduced the proportion of outlier points compared to projecting all the points (from 3.63% to 1.75%). As a result, the proposed method was effective in achieving robust human following.


  • Open Access

    Reversible Spectral Speech Watermarking with Variable Embedding Locations Against Spectrum-Based Attacks

    by Xuping Huang, Akinori Ito

    To guarantee the reliability and integrity of audio, data have been focused on as an essential topic as the fast development of generative AI. Significant progress in machine learning and speech synthesis has increased the potential for audio tampering. In this paper, we focus on the digital watermarking method as a promising method to safeguard the authenticity of audio evidence. Due to the integrity of the original data with probative importance, the algorithm requires reversibility, imperceptibility, and reliability. To meet the requirements, we propose a reversible digital watermarking approach that embeds feature data concentrating in high-frequency intDCT coefficients after transforming data from the time domain into the frequency domain. We explored the appropriate hiding locations against spectrum-based attacks with novel proposed methodologies for spectral expansion for embedding. However, the drawback of fixed expansion is that the stego signal is prone to being detected by a spectral analysis. Therefore, this paper proposes two other new expansion methodologies that embed the data into variable locations—random expansion and adaptive expansion with distortion estimation for embedding—which effectively conceal the watermark’s presence while maintaining high perceptual quality with an average segSNR better than 21.363 dB and average MOS value better than 4.085. Our experimental results demonstrate the efficacy of our proposed method in both sound quality preservation and log-likelihood value, indicating the absolute discontinuity of the spectrogram after embedding is proposed to evaluate the effectiveness of the proposed reversible spectral expansion watermarking algorithm. The result of EER indicated that the adaptive hiding performed best against attacks by spectral analysis.


  • Open Access

    We open our mouths when we are silent

    by Shoki Kawanishi, Yuya Chiba, Akinori Ito, Takashi Nose

    Lip syncing is an important technology that enhances the impression of embodied conversational agents. However, there is no study to design the mouth movement of the agent when the agent is silent. Therefore, this paper investigated how human speakers move their mouths when silent in dialogues. As a result, we found three facts. First, a speaker does not completely close mouth even when listening to a partner’s talk. Second, the degree of mouth opening while talking and listening greatly depends on the speaker. Third, the mouth opening is possibly affected by the next state of the speaker.

  • Open Access

    Speaker Intimacy Estimation in Chat-Talks Based on Verbal and Non-Verbal Information

    by Yuya Chiba, Akinori Ito

    Conversations based on mutual intimacy are critical for maintaining positive relationships. A detailed understanding of speaker relationships in dialogues enhances various applications, such as information recommendation systems. Such systems, when interacting with multiple users, can provide more tailored information by understanding the users’ relationships. Furthermore, dialogue systems, which are becoming increasingly prevalent in society, can foster long-term user engagement by recognizing and responding to the intimacy levels of users. This study explores a method for estimating the intimacy levels of speakers and dialogue partners in conversational exchanges. Our approach utilizes a multimodal corpus of natural conversations with 71 Japanese participants, complete with metadata indicating each speaker’s perceived intimacy level. We identified key features for estimating intimacy by analyzing the statistical parameters of these features. Our comprehensive analysis encompassed both verbal and non-verbal information, including prosody, gestures, and facial expressions. The proposed intimacy estimation model combines multimodal features using a multi-stream Bi-directional Long Short-Term Memory (BLSTM) network and grasps the contextual information of conversations with a Context BLSTM. Our model’s effectiveness is demonstrated through comparisons with several baseline models. Experimental results show that our proposed model significantly improves the overall performance compared with other models. Although the RoBERTa-based method (the best baseline model) achieved an F1 score of 0.571, our method had an F1 score of 0.594. In particular, an ablation study shows that combining verbal and non-verbal features is useful for intimacy estimation. The performance was further improved by extending the dialogue context, showing that the proposed model can estimate three levels of intimacy with an F1 score of 0.666 by observing eight utterance exchanges.