Takatsugu Hirayama
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Takatsugu Hirayama has a total of 78 co-authors. Here are the top co-authors by number of shared publications:
- Ichiro Ide
Nagoya University, Nagoya, Japan
16 shared publications - Yasutomo Kawanishi
Guardian Robot Project R-IH, RIKEN
14 shared publications - Marc A. Kastner
Hiroshima City University, Hiroshima, Japan
12 shared publications - Takahiro Komamizu
Nagoya University, Nagoya, Japan
11 shared publications - Chihaya Matsuhira
Nagoya University, Nagoya, Japan
8 shared publications - Daisuke Deguchi
Nagoya University, Nagoya, Japan
8 shared publications
Latest Publications
- Open Access
Small Object Detection for Birds with Swin Transformer
Object detection is the task of detecting objects in an image. In this task, the detection of small objects is particularly difficult. Other than the small size, it is also accompanied by difficulties due to blur, occlusion, and so on. Current small object detection methods are tailored to small and dense situations, such as pedestrians in a crowd or far objects in remote sensing scenarios. However, when the target object is small and sparse, there is a lack of objects available for training, making it more difficult to learn effective features. In this paper, we propose a specialized method for detecting a specific category of small objects; birds. Particularly, we improve the features learned by the neck; the sub-network between the backbone and the prediction head, to learn more effective features with a hierarchical design. We employ Swin Transformer to upsample the image features. Moreover, we change the shifted window size for adapting to small objects. Experiments show that the proposed Swin Transformer-based neck combined with CenterNet can lead to good performance by changing the window sizes. We further find that smaller window sizes (default 2) benefit mAPs for small object detection.
Open AccessSmall Object Detection for Birds with Swin Transformer
Object detection is the task of detecting objects in an image. In this task, the detection of small objects is particularly difficult. Other than the small size, it is also accompanied by difficulties due to blur, occlusion, and so on. Current small object detection methods are tailored to small and dense situations, such as pedestrians in a crowd or far objects in remote sensing scenarios. However, when the target object is small and sparse, there is a lack of objects available for training, making it more difficult to learn effective features. In this paper, we propose a specialized method for detecting a specific category of small objects; birds. Particularly, we improve the features learned by the neck; the sub-network between the backbone and the prediction head, to learn more effective features with a hierarchical design. We employ Swin Transformer to upsample the image features. Moreover, we change the shifted window size for adapting to small objects. Experiments show that the proposed Swin Transformer-based neck combined with CenterNet can lead to good performance by changing the window sizes. We further find that smaller window sizes (default 2) benefit mAPs for small object detection.
- Open Access
- 10 November 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 199615-199629
Semantic Alignment on Action for Image Captioning
Image captioning is a popular task in vision and language processing, which aims to generate textual descriptions for images. Previously, it simply used image and text as input with self-attention to capture global dependencies. Recent research further uses objects detected from the input image, so-called object tags, as anchor points to ease alignment between image and text with the attention mechanism. However, they only consider object information in images, while neglecting the actions and object interactions that also appear in the image, which causes actions not caught properly in image captioning. To tackle this previously underrepresented dimension of the semantic alignment, we take account of actions on the semantic level. Specifically, our work focuses on human actions and interactions, which ensures that more salient parts of the image get captioned. We introduce a new type of tag, called action tag, to anchor the action information. First, we provide a method for obtaining such action tags using an action detection model which predicts actions in the image. Next, we leverage these action tags into the captioning model. Experimental results indicate that the proposed action tags can help learn action semantics and catch the salient actions leading to perceived improvements in common performance. Experimental results on MS-COCO Karpathy test split show that the proposed model achieves good scores in BLEU-4 and CIDEr metrics, using action tags as anchors. Furthermore, the number of action tags (no more than 5) is smaller than that of object tags (commonly more than 20), which means there is a potential to reduce FLOPs by reducing the total sequence length. It indicates the potential for efficient reasoning and may be applied to daily activity scenes in the future.
- 10 November 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 199615-199629
Open AccessSemantic Alignment on Action for Image Captioning
Image captioning is a popular task in vision and language processing, which aims to generate textual descriptions for images. Previously, it simply used image and text as input with self-attention to capture global dependencies. Recent research further uses objects detected from the input image, so-called object tags, as anchor points to ease alignment between image and text with the attention mechanism. However, they only consider object information in images, while neglecting the actions and object interactions that also appear in the image, which causes actions not caught properly in image captioning. To tackle this previously underrepresented dimension of the semantic alignment, we take account of actions on the semantic level. Specifically, our work focuses on human actions and interactions, which ensures that more salient parts of the image get captioned. We introduce a new type of tag, called action tag, to anchor the action information. First, we provide a method for obtaining such action tags using an action detection model which predicts actions in the image. Next, we leverage these action tags into the captioning model. Experimental results indicate that the proposed action tags can help learn action semantics and catch the salient actions leading to perceived improvements in common performance. Experimental results on MS-COCO Karpathy test split show that the proposed model achieves good scores in BLEU-4 and CIDEr metrics, using action tags as anchors. Furthermore, the number of action tags (no more than 5) is smaller than that of object tags (commonly more than 20), which means there is a potential to reduce FLOPs by reducing the total sequence length. It indicates the potential for efficient reasoning and may be applied to daily activity scenes in the future.
- 9 November 2025
- Published by Springer Nature
- p. 16-31
Origami Crease Recognition for Automatic Folding Diagrams Generation
- 9 November 2025
- Published by Springer Nature
- p. 16-31
Origami Crease Recognition for Automatic Folding Diagrams Generation
- 9 November 2025
- Published by Springer Nature
- p. 48-62
Lip Shape-Aware Word Selection for Lyric Translation
- 9 November 2025
- Published by Springer Nature
- p. 48-62
Lip Shape-Aware Word Selection for Lyric Translation
- 8 October 2025
- Published by Springer Nature in Landscape and Ecological Engineering
- Vol. 22  (1) , 143-151
Monetary valuation of tree ecosystem services using i-Tree Eco and UAV: development of a model eliminating the need for DBH data
- 8 October 2025
- Published by Springer Nature in Landscape and Ecological Engineering
- Vol. 22  (1) , 143-151
Monetary valuation of tree ecosystem services using i-Tree Eco and UAV: development of a model eliminating the need for DBH data
- Open Access
- 22 April 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. PP  (15249050) , 1-12
Pre-Instruction for Pedestrians Interacting Autonomous Vehicles With eHMI: Effects on Their Psychology and Walking Behavior
External human-machine interface (eHMI) is considered as a new explicit communication method for pedestrian-AV interactions, particularly in encounter scenarios. Pedestrians without prior negotiation experience with eHMI may misinterpret the driving intentions of the AV, leading to confusion and unpredictable behavior. To address this issue, this study suggests providing pre-instruction on eHMI to enhance comprehension. To compare pedestrians’ subjective feelings and walking behavior changes with and without the use of eHMI, as well as before and after receiving pre-instructions, a road crossing experiment using a within-subject design was conducted. In the experiment, the participants were challenged to recognize situations and experienced uncertainty when encountering AVs lacking eHMI, in contrast to manual driving vehicles. After the pre-instruction, participants could understand the driving intention of an AV with eHMI and predict its driving behavior more easily. Furthermore, participants’ subjective feelings and hesitation to make decisions improved to align with the same criteria as encountered with a manual driving vehicle. Additionally, this study found that the information guidance effect of using eHMI made participants’ walking speeds more consistent over multiple trials, as they gain a comprehensive understanding of eHMI principles through the pre-instruction.
- 22 April 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. PP  (15249050) , 1-12
Open AccessPre-Instruction for Pedestrians Interacting Autonomous Vehicles With eHMI: Effects on Their Psychology and Walking Behavior
External human-machine interface (eHMI) is considered as a new explicit communication method for pedestrian-AV interactions, particularly in encounter scenarios. Pedestrians without prior negotiation experience with eHMI may misinterpret the driving intentions of the AV, leading to confusion and unpredictable behavior. To address this issue, this study suggests providing pre-instruction on eHMI to enhance comprehension. To compare pedestrians’ subjective feelings and walking behavior changes with and without the use of eHMI, as well as before and after receiving pre-instructions, a road crossing experiment using a within-subject design was conducted. In the experiment, the participants were challenged to recognize situations and experienced uncertainty when encountering AVs lacking eHMI, in contrast to manual driving vehicles. After the pre-instruction, participants could understand the driving intention of an AV with eHMI and predict its driving behavior more easily. Furthermore, participants’ subjective feelings and hesitation to make decisions improved to align with the same criteria as encountered with a manual driving vehicle. Additionally, this study found that the information guidance effect of using eHMI made participants’ walking speeds more consistent over multiple trials, as they gain a comprehensive understanding of eHMI principles through the pre-instruction.
- 2 January 2025
- Published by Springer Nature
- p. 428-441
Quantifying Image-Adjective Associations by Leveraging Large-Scale Pretrained Models
- 2 January 2025
- Published by Springer Nature
- p. 428-441
Quantifying Image-Adjective Associations by Leveraging Large-Scale Pretrained Models
- Open Access
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 7307-7315
Investigating Conceptual Blending of a Diffusion Model for Improving Nonword-to-Image Generation
Text-to-image diffusion models sometimes depict blended concepts in the generated images. One promising use case of this effect would be the nonword-to-image generation task which attempts to generate images intuitively imaginable from a non-existing word (nonword). To realize nonword-to-image generation, an existing study focused on associating nonwords with similar-sounding words. Since each nonword can have multiple similar-sounding words, generating images containing their blended concepts would increase intuitiveness, facilitating creative activities and promoting computational psycholinguistics. Nevertheless, no existing study has quantitatively evaluated this effect in either diffusion models or the nonword-to-image generation paradigm. Therefore, this paper first analyzes the conceptual blending in a pretrained diffusion model, Stable Diffusion. The analysis reveals that a high percentage of generated images depict blended concepts when inputting an embedding interpolating between the text embeddings of two text prompts referring to different concepts. Next, this paper explores the best text embedding space conversion method of an existing nonword-to-image generation framework to ensure both the occurrence of conceptual blending and image generation quality. We compare the conventional direct prediction approach with the proposed method that combines k-nearest neighbor search and linear regression. Evaluation reveals that the enhanced accuracy of the embedding space conversion by the proposed method improves the image generation quality, while the emergence of conceptual blending could be attributed mainly to the specific dimensions of the high-dimensional text embedding space.
- 28 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 7307-7315
Open AccessInvestigating Conceptual Blending of a Diffusion Model for Improving Nonword-to-Image Generation
Text-to-image diffusion models sometimes depict blended concepts in the generated images. One promising use case of this effect would be the nonword-to-image generation task which attempts to generate images intuitively imaginable from a non-existing word (nonword). To realize nonword-to-image generation, an existing study focused on associating nonwords with similar-sounding words. Since each nonword can have multiple similar-sounding words, generating images containing their blended concepts would increase intuitiveness, facilitating creative activities and promoting computational psycholinguistics. Nevertheless, no existing study has quantitatively evaluated this effect in either diffusion models or the nonword-to-image generation paradigm. Therefore, this paper first analyzes the conceptual blending in a pretrained diffusion model, Stable Diffusion. The analysis reveals that a high percentage of generated images depict blended concepts when inputting an embedding interpolating between the text embeddings of two text prompts referring to different concepts. Next, this paper explores the best text embedding space conversion method of an existing nonword-to-image generation framework to ensure both the occurrence of conceptual blending and image generation quality. We compare the conventional direct prediction approach with the proposed method that combines k-nearest neighbor search and linear regression. Evaluation reveals that the enhanced accuracy of the embedding space conversion by the proposed method improves the image generation quality, while the emergence of conceptual blending could be attributed mainly to the specific dimensions of the high-dimensional text embedding space.
- Open Access
- 18 March 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 12, 41299-41316
Interpolating the Text-to-Image Correspondence Based on Phonetic and Phonological Similarities for Nonword-to-Image Generation
Text-to-Image (T2I) generation is the task of synthesizing images corresponding to a given text input. The recent innovations in artificial intelligence have enhanced the capacity of conventional T2I generation, yielding more and more powerful models day by day. However, their behavior is known to become unstable in the face of text inputs containing nonwords that have no definition within a language. This behavior not only results in situations where image generation does not match human expectations but also hinders these models from being utilized in psycholinguistic applications and simulations. This paper exploits the human nature of associating nonwords with their phonetically and phonologically similar words and uses it to propose a T2I generation framework robust against nonword inputs. The framework comprises a phonetics-aware language model as well as an adjusted T2I generation model. Our evaluations confirm that the proposed nonword-to-image generation synthesizes images that depict visual concepts of phonetically similar words more stably than comparative methods. We also assess how the image generation results match human expectations, showing a better agreement than the phonetics-blind baseline.
- 18 March 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 12, 41299-41316
Open AccessInterpolating the Text-to-Image Correspondence Based on Phonetic and Phonological Similarities for Nonword-to-Image Generation
Text-to-Image (T2I) generation is the task of synthesizing images corresponding to a given text input. The recent innovations in artificial intelligence have enhanced the capacity of conventional T2I generation, yielding more and more powerful models day by day. However, their behavior is known to become unstable in the face of text inputs containing nonwords that have no definition within a language. This behavior not only results in situations where image generation does not match human expectations but also hinders these models from being utilized in psycholinguistic applications and simulations. This paper exploits the human nature of associating nonwords with their phonetically and phonologically similar words and uses it to propose a T2I generation framework robust against nonword inputs. The framework comprises a phonetics-aware language model as well as an adjusted T2I generation model. Our evaluations confirm that the proposed nonword-to-image generation synthesizes images that depict visual concepts of phonetically similar words more stably than comparative methods. We also assess how the image generation results match human expectations, showing a better agreement than the phonetics-blind baseline.
- Open Access
- 16 December 2023
- Published by Springer Nature in Multimedia Tools and Applications
- Vol. 83  (9) , 26211-26212
Correction to: Computational measurement of perceived pointiness from pronunciation
- 16 December 2023
- Published by Springer Nature in Multimedia Tools and Applications
- Vol. 83  (9) , 26211-26212
Open AccessCorrection to: Computational measurement of perceived pointiness from pronunciation