ITIoannis Tsiamas

Ioannis Tsiamas

*This is an algorithmically generated author record based on publicly available information.

Ioannis Tsiamas has a total of 10 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Speech is More Than Words: Do Speech-to-Text Translation Systems Leverage Prosody?

    by Ioannis Tsiamas, Matthias Sperber, Andrew Finch, Sarthak Garg

    The prosody of a spoken utterance, including features like stress, intonation and rhythm, can significantly affect the underlying semantics, and as a consequence can also affect its textual translation. Nevertheless, prosody is rarely studied within the context of speech-to-text translation (S2TT) systems. In particular, end-to-end (E2E) systems have been proposed as well-suited for prosody-aware translation because they have direct access to the speech signal when making translation decisions, but the understanding of whether this is successful in practice is still limited. A main challenge is the difficulty of evaluating prosody awareness in translation. To address this challenge, we introduce an evaluation methodology and a focused benchmark (named ContraProST) aimed at capturing a wide range of prosodic phenomena. Our methodology uses large language models and controllable text-to-speech (TTS) to generate contrastive examples. Through experiments in translating English speech into German, Spanish, and Japanese, we find that (a) S2TT models possess some internal representation of prosody, but the prosody signal is often not strong enough to affect the translations, (b) E2E systems outperform cascades of speech recognition and text translation systems, confirming their theoretical advantage in this regard, and (c) certain cascaded systems also capture prosodic information in the translation, but only to a lesser extent that depends on the particulars of the transcript's surface form.

  • Open Access

    Speech Is More than Words: Do Speech-to-Text Translation Systems Leverage Prosody?

    by Ioannis Tsiamas, Matthias Sperber, Andrew Finch, Sarthak Garg

    Ioannis Tsiamas, Matthias Sperber, Andrew Finch, Sarthak Garg. Proceedings of the Ninth Conference on Machine Translation. 2024.


  • Open Access

    Pushing the Limits of Zero-shot End-to-End Speech Translation

    by Ioannis Tsiamas, Gerard Gállego, José Fonollosa, Marta Costa-Jussà

    Ioannis Tsiamas, Gerard Gállego, José Fonollosa, Marta Costa-jussà. Findings of the Association for Computational Linguistics ACL 2024. 2024.


  • Open Access

    SegAugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentations

    by Ioannis Tsiamas, José A. R. Fonollosa, Marta R. Costa-Jussà

    End-to-end Speech Translation is hindered by a lack of available data resources. While most of them are based on documents, a sentence-level version is available, which is however single and static, potentially impeding the usefulness of the data. We propose a new data augmentation strategy, SegAugment, to address this issue by generating multiple alternative sentence-level versions of a dataset. Our method utilizes an Audio Segmentation system, which re-segments the speech of each document with different length constraints, after which we obtain the target text via alignment methods. Experiments demonstrate consistent gains across eight language pairs in MuST-C, with an average increase of 2.5 BLEU points, and up to 5 BLEU for low-resource scenarios in mTEDx. Furthermore, when combined with a strong system, SegAugment establishes new state-of-the-art results in MuST-C. Finally, we show that the proposed method can also successfully augment sentence-level datasets, and that it enables Speech Translation models to close the gap between the manual and automatic segmentation at inference time.

  • Open Access

    Speech Translation with Foundation Models and Optimal Transport: UPC at IWSLT23

    by Ioannis Tsiamas, Gerard I. Gállego, José A. R. Fonollosa, Marta R. Costa-Jussà

    This paper describes the submission of the UPC Machine Translation group to the IWSLT 2023 Offline Speech Translation task. Our Speech Translation systems utilize foundation models for speech (wav2vec 2.0) and text (mBART50). We incorporate a Siamese pretraining step of the speech and text encoders with CTC and Optimal Transport, to adapt the speech representations to the space of the text model, thus maximizing transfer learning from MT. After this pretraining, we fine-tune our system end-to-end on ST, with Cross Entropy and Knowledge Distillation. Apart from the available ST corpora, we create synthetic data with SegAugment to better adapt our models to the custom segmentations of the IWSLT test sets. Our best single model obtains 31.2 BLEU points on MuST-C tst-COMMON, 29.8 points on IWLST.tst2020 and 33.4 points on the newly released IWSLT.ACLdev2023.

  • Open Access

    Explaining How Transformers Use Context to Build Predictions

    by Javier Ferrando, Gerard I. Gállego, Ioannis Tsiamas, Marta R. Costa-Jussà

    Language Generation Models produce words based on the previous context. Although existing methods offer input attributions as explanations for a model's prediction, it is still unclear how prior words affect the model's decision throughout the layers. In this work, we leverage recent advances in explainability of the Transformer and present a procedure to analyze models for language generation. Using contrastive examples, we compare the alignment of our explanations with evidence of the linguistic phenomena, and show that our method consistently aligns better than gradient-based and perturbation-based baselines. Then, we investigate the role of MLPs inside the Transformer and show that they learn features that help the model predict words that are grammatically acceptable. Lastly, we apply our method to Neural Machine Translation models, and demonstrate that they generate human-like source-target alignments for building predictions.


  • Open Access

    SegAugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentations

    by Ioannis Tsiamas, José Fonollosa, Marta Costa-Jussà

    Ioannis Tsiamas, José Fonollosa, Marta Costa-jussà. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.


  • Open Access

    End-to-End Speech Translation with Pre-trained Models and Adapters: UPC at IWSLT 2021

    by Gerard I. Gállego, Ioannis Tsiamas, Carlos Escolano, José A. R. Fonollosa, Marta R. Costa-Jussà

    This paper describes the submission to the IWSLT 2021 offline speech translation task by the UPC Machine Translation group. The task consists of building a system capable of translating English audio recordings extracted from TED talks into German text. Submitted systems can be either cascade or end-to-end and use a custom or given segmentation. Our submission is an end-to-end speech translation system, which combines pre-trained models (Wav2Vec 2.0 and mBART) with coupling modules between the encoder and decoder, and uses an efficient fine-tuning technique, which trains only 20% of its total parameters. We show that adding an Adapter to the system and pre-training it, can increase the convergence speed and the final result, with which we achieve a BLEU score of 27.3 on the MuST-C test set. Our final model is an ensemble that obtains 28.22 BLEU score on the same set. Our submission also uses a custom segmentation algorithm that employs pre-trained Wav2Vec 2.0 for identifying periods of untranscribable text and can bring improvements of 2.5 to 3 BLEU score on the IWSLT 2019 test set, as compared to the result with the given segmentation.