Diogo Pernes
Diogo Pernes has a total of 14 co-authors. Here are the top co-authors by number of shared publications:
- Jaime Santos Cardoso
INESC TEC, Faculdade de Engenharia, Universidade do Porto, Porto, Portugal
7 shared publications Afonso Mendes
2 shared publicationsAlfonso Mendes
2 shared publications- Ana Rebelo
Centre for Telecommunications and Multimedia, INESC TEC, Porto, Portugal
2 shared publications - Gonçalo Correia
Priberam Labs Alameda D. Afonso Henriques 41 2º 1000-123 Lisboa Portugal
2 shared publications Gonçalo M. Correia
Priberam Labs Alameda D. Afonso Henriques 41 2º 1000-123 Lisboa Portugal
2 shared publications
Latest Publications
- Open Access
Multi-Target Cross-Lingual Summarization: a novel task and a language-neutral approach
Cross-lingual summarization aims to bridge language barriers by summarizing documents in different languages. However, ensuring semantic coherence across languages is an overlooked challenge and can be critical in several contexts. To fill this gap, we introduce multi-target cross-lingual summarization as the task of summarizing a document into multiple target languages while ensuring that the produced summaries are semantically similar. We propose a principled re-ranking approach to this problem and a multi-criteria evaluation protocol to assess semantic coherence across target languages, marking a first step that will hopefully stimulate further research on this problem.
Open AccessMulti-Target Cross-Lingual Summarization: a novel task and a language-neutral approach
Cross-lingual summarization aims to bridge language barriers by summarizing documents in different languages. However, ensuring semantic coherence across languages is an overlooked challenge and can be critical in several contexts. To fill this gap, we introduce multi-target cross-lingual summarization as the task of summarizing a document into multiple target languages while ensuring that the produced summaries are semantically similar. We propose a principled re-ranking approach to this problem and a multi-criteria evaluation protocol to assess semantic coherence across target languages, marking a first step that will hopefully stimulate further research on this problem.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 12908-12924
Multi-Target Cross-Lingual Summarization: a novel task and a language-neutral approach
Diogo Pernes, Gonçalo M. Correia, Afonso Mendes. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 12908-12924
Open AccessMulti-Target Cross-Lingual Summarization: a novel task and a language-neutral approach
Diogo Pernes, Gonçalo M. Correia, Afonso Mendes. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024.
- Open Access
Supervising the Centroid Baseline for Extractive Multi-Document Summarization
The centroid method is a simple approach for extractive multi-document summarization and many improvements to its pipeline have been proposed. We further refine it by adding a beam search process to the sentence selection and also a centroid estimation attention model that leads to improved results. We demonstrate this in several multi-document summarization datasets, including in a multilingual scenario.
Open AccessSupervising the Centroid Baseline for Extractive Multi-Document Summarization
The centroid method is a simple approach for extractive multi-document summarization and many improvements to its pipeline have been proposed. We further refine it by adding a beam search process to the sentence selection and also a centroid estimation attention model that leads to improved results. We demonstrate this in several multi-document summarization datasets, including in a multilingual scenario.
- Open Access
Towards End-to-end Speech-to-text Summarization
Speech-to-text (S2T) summarization is a time-saving technique for filtering and keeping up with the broadcast news uploaded online on a daily basis. The rise of large language models from deep learning with impressive text generation capabilities has placed the research focus on summarization systems that produce paraphrased compact versions of the document content, also known as abstractive summaries. End-to-end (E2E) modelling of S2T abstractive summarization is a promising approach that offers the possibility of generating rich latent representations that leverage non-verbal and acoustic information, as opposed to the use of only linguistic information from automatically generated transcripts in cascade systems. However, the few literature on E2E modelling of this task fails on exploring different domains, namely broadcast news, which is challenging domain where large and diversified volumes of data are presented to the user every day. We model S2T summarization both with a cascade and an E2E system for a corpus of broadcast news in French. Our novel E2E model leverages external data by resorting to transfer learning from a pre-trained T2T summarizer. Experiments show that both our cascade and E2E abstractive summarizers are stronger than an extractive baseline. However, the performance of the E2E model still lies behind the cascade one, which is object of an extensive analysis that includes future directions to close that gap.
Open AccessTowards End-to-end Speech-to-text Summarization
Speech-to-text (S2T) summarization is a time-saving technique for filtering and keeping up with the broadcast news uploaded online on a daily basis. The rise of large language models from deep learning with impressive text generation capabilities has placed the research focus on summarization systems that produce paraphrased compact versions of the document content, also known as abstractive summaries. End-to-end (E2E) modelling of S2T abstractive summarization is a promising approach that offers the possibility of generating rich latent representations that leverage non-verbal and acoustic information, as opposed to the use of only linguistic information from automatically generated transcripts in cascade systems. However, the few literature on E2E modelling of this task fails on exploring different domains, namely broadcast news, which is challenging domain where large and diversified volumes of data are presented to the user every day. We model S2T summarization both with a cascade and an E2E system for a corpus of broadcast news in French. Our novel E2E model leverages external data by resorting to transfer learning from a pre-trained T2T summarizer. Experiments show that both our cascade and E2E abstractive summarizers are stronger than an extractive baseline. However, the performance of the E2E model still lies behind the cascade one, which is object of an extensive analysis that includes future directions to close that gap.
- Open Access
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 87-96
Supervising the Centroid Baseline for Extractive Multi-Document Summarization
Simão Gonçalves, Gonçalo Correia, Diogo Pernes, Afonso Mendes. Proceedings of the 4th New Frontiers in Summarization Workshop. 2023.
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 87-96
Open AccessSupervising the Centroid Baseline for Extractive Multi-Document Summarization
Simão Gonçalves, Gonçalo Correia, Diogo Pernes, Afonso Mendes. Proceedings of the 4th New Frontiers in Summarization Workshop. 2023.
- Open Access
Improving abstractive summarization with energy-based re-ranking
Current abstractive summarization systems present important weaknesses which prevent their deployment in real-world applications, such as the omission of relevant information and the generation of factual inconsistencies (also known as hallucinations). At the same time, automatic evaluation metrics such as CTC scores have been recently proposed that exhibit a higher correlation with human judgments than traditional lexical-overlap metrics such as ROUGE. In this work, we intend to close the loop by leveraging the recent advances in summarization metrics to create quality-aware abstractive summarizers. Namely, we propose an energy-based model that learns to re-rank summaries according to one or a combination of these metrics. We experiment using several metrics to train our energy-based re-ranker and show that it consistently improves the scores achieved by the predicted summaries. Nonetheless, human evaluation results show that the re-ranking approach should be used with care for highly abstractive summaries, as the available metrics are not yet sufficiently reliable for this purpose.
Open AccessImproving abstractive summarization with energy-based re-ranking
Current abstractive summarization systems present important weaknesses which prevent their deployment in real-world applications, such as the omission of relevant information and the generation of factual inconsistencies (also known as hallucinations). At the same time, automatic evaluation metrics such as CTC scores have been recently proposed that exhibit a higher correlation with human judgments than traditional lexical-overlap metrics such as ROUGE. In this work, we intend to close the loop by leveraging the recent advances in summarization metrics to create quality-aware abstractive summarizers. Namely, we propose an energy-based model that learns to re-rank summaries according to one or a combination of these metrics. We experiment using several metrics to train our energy-based re-ranker and show that it consistently improves the scores achieved by the predicted summaries. Nonetheless, human evaluation results show that the re-ranking approach should be used with care for highly abstractive summaries, as the available metrics are not yet sufficiently reliable for this purpose.
- 15 January 2022
- Published by Elsevier in Expert Systems with Applications
- Vol. 194
Tackling unsupervised multi-source domain adaptation with optimism and consistency
- 15 January 2022
- Published by Elsevier in Expert Systems with Applications
- Vol. 194
Tackling unsupervised multi-source domain adaptation with optimism and consistency
- 31 January 2020
- Published by SPIE-Intl Soc Optical Eng
- Vol. 11433, 114333D
Learning signer-invariant representations with adversarial training
Sign Language Recognition (SLR) has become an appealing topic in modern societies because such technology can ideally be used to bridge the gap between deaf and hearing people. Although important steps have been made towards the development of real-world SLR systems, signer-independent SLR is still one of the bottleneck problems of this research field. In this regard, we propose a deep neural network along with an adversarial training objective, specifically designed to address the signer-independent problem. Concretely speaking, the proposed model consists of an encoder, mapping from input images to latent representations, and two classifiers operating on these underlying representations: (i) the signclassifier, for predicting the class/sign labels, and (ii) the signer-classifier, for predicting their signer identities. During the learning stage, the encoder is simultaneously trained to help the sign-classifier as much as possible while trying to fool the signer-classifier. This adversarial training procedure allows learning signer-invariant latent representations that are in fact highly discriminative for sign recognition. Experimental results demonstrate the effectiveness of the proposed model and its capability of dealing with the large inter-signer variations.
- 31 January 2020
- Published by SPIE-Intl Soc Optical Eng
- Vol. 11433, 114333D
Learning signer-invariant representations with adversarial training
Sign Language Recognition (SLR) has become an appealing topic in modern societies because such technology can ideally be used to bridge the gap between deaf and hearing people. Although important steps have been made towards the development of real-world SLR systems, signer-independent SLR is still one of the bottleneck problems of this research field. In this regard, we propose a deep neural network along with an adversarial training objective, specifically designed to address the signer-independent problem. Concretely speaking, the proposed model consists of an encoder, mapping from input images to latent representations, and two classifiers operating on these underlying representations: (i) the signclassifier, for predicting the class/sign labels, and (ii) the signer-classifier, for predicting their signer identities. During the learning stage, the encoder is simultaneously trained to help the sign-classifier as much as possible while trying to fool the signer-classifier. This adversarial training procedure allows learning signer-invariant latent representations that are in fact highly discriminative for sign recognition. Experimental results demonstrate the effectiveness of the proposed model and its capability of dealing with the large inter-signer variations.
- 20 December 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Systems, Man, and Cybernetics: Systems
- Vol. 51 (9) , 5830-5845
DeSIRe: Deep Signer-Invariant Representations for Sign Language Recognition
As a key technology to help bridging the gap between deaf and hearing people, sign language recognition (SLR) has become one of the most active research topics in the human-computer interaction field. Although several SLR methodologies have been proposed, the development of a real-world SLR system is still a very challenging task. One of the main challenges is related to the large intersigner variability that exists in the manual signing process of sign languages. To address this problem, we propose a novel end-to-end deep neural network that explicitly models highly discriminative signer-independent latent representations from the input data. The key idea of our model is to learn a distribution over latent representations, conditionally independent of signer identity. Accordingly, the learned latent representations will preserve as much information as possible about the signs, and discard signer-specific traits that are irrelevant for recognition. By imposing such regularization in the representation space, the result is a truly signer-independent model which is robust to different and new test signers. The experimental results demonstrate the effectiveness of the proposed model in several SLR databases.
- 20 December 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Systems, Man, and Cybernetics: Systems
- Vol. 51 (9) , 5830-5845
DeSIRe: Deep Signer-Invariant Representations for Sign Language Recognition
As a key technology to help bridging the gap between deaf and hearing people, sign language recognition (SLR) has become one of the most active research topics in the human-computer interaction field. Although several SLR methodologies have been proposed, the development of a real-world SLR system is still a very challenging task. One of the main challenges is related to the large intersigner variability that exists in the manual signing process of sign languages. To address this problem, we propose a novel end-to-end deep neural network that explicitly models highly discriminative signer-independent latent representations from the input data. The key idea of our model is to learn a distribution over latent representations, conditionally independent of signer identity. Accordingly, the learned latent representations will preserve as much information as possible about the signs, and discard signer-specific traits that are irrelevant for recognition. By imposing such regularization in the representation space, the result is a truly signer-independent model which is robust to different and new test signers. The experimental results demonstrate the effectiveness of the proposed model in several SLR databases.