AKAidana Karibayeva

Aidana Karibayeva

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Aidana Karibayeva has a total of 7 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Speech-to-Speech Translation from Kazakh into Turkic Languages: Pretrained Speech Encoders, Turkic Speech Units, and Lessons from Negative Results

    by Aidana Karibayeva, Oleg Myssov, Balzhan Abduali, Dina Amirova, Diana Rakhimova

    Speech-to-speech translation (S2ST) among Turkic languages is rarely studied due to the scarcity of parallel speech corpora and the complex morphology. In this work, we develop low-resource S2ST systems translating from Kazakh to Turkish, Tatar, and Uzbek, and examine which architectural choices proved ineffective and why. Three architectures were trained on a synthetic parallel speech corpus: (i) a scratch-trained, morphology-aware Conformer encoder with a morpheme-level connectionist temporal classification (CTC) loss, a vector-quantized (VQ) codebook, and a semantic anchor; (ii) a pretrained Whisper encoder with a text decoder conditioned on the target language by feature-wise linear modulation (FiLM); and (iii) a text-mediated two-pass model that predicts Turkic speech units from the decoder states and converts them to speech with a Turkic unit vocoder. Text output was scored with BLEU and speech output with ASR-BLEU, that is, BLEU on automatic transcripts. The scratch-trained model and about ten variants scored ASR-BLEU 0. The Whisper-based speech-to-text model reached 19.49 BLEU on synthetic test speech and 3.56 on natural speech. The final speech-to-speech model reached 3.25 ASR-BLEU on about 1500 synthetic test utterances per language, 5.1 times the English-unit baseline, and 0.86 ASR-BLEU on natural speech. These scores show feasibility, not usable quality.

  • Open Access

    Adapting Large-Scale Foundation Models for Turkic Speech-to-Speech Translation: Fine-Tuned Cascade and Direct Approaches

    by Aidana Karibayeva, Vladislav Karyukin, Oleg Myssov, Dina Amirova, Balzhan Abduali, Adina Karybayeva

    This paper presents a novel approach to speech-to-speech (STS) translation for low-resource Turkic languages. Today, STS has progressed rapidly for high-resource languages; the Turkic family remains significantly underrepresented. Consequently, it is quite challenging to develop a reliable speech translation system for these languages. To address this issue, we have developed two speech translation systems (STS) specifically tailored to Turkic languages with limited resources. The first, TurkicCascadeSTS, is a cascaded system that combines a fine-tuned Whisper-medium speech recognition model, GPT translation, and separate speech synthesis models for each language. The second system is a direct speech translation model based on a fine-tuned SeamlessM4Tv2. Both systems have been tested for translation into Turkic languages, using 24,656 audio recordings per language. The TurkicCascadeSTS system delivered far better results: the average BLEU score rose from 4.68 to 30.60; the METEOR score rose from 15.03 to 44.42; and the word error rate (WER) also fell significantly. These improvements are due to the fact that each module of the system was individually fine-tuned to account for the specific characteristics of each language. Although SeamlessM4Tv2 sometimes produces clearer audio, TurkicCascadeSTS generally delivers higher speech and translation quality for all language pairs. This demonstrates that modular, specially tuned systems are an effective solution for translation into Turkic languages, particularly given their complex structure and limited linguistic resources. Such systems could benefit more than 200 million native speakers of Turkic languages.


  • Open Access

    A Lightweight Multimodal Architecture for Punctuation Restoration in Kazakh ASR

    by Aidana Karibayeva, Oleg Myssov, Balzhan Abduali, Dina Amirova, Adina Karybayeva

    In this paper, we first present a multimodal architecture called CrossAttn-v1. This model is designed to recover punctuation marks in Kazakh and combines contextual XLM-RoBERTa-large text embeddings with the Whisper large-v3 encoder states via a cross-attention mechanism. In addition, a 4-dimensional prosodic vector and a CRF output layer are used. The model was trained using an adapted Whisper ASR model on 33,332 utterances from the KazakhTTS2 corpus. After adaptation, the word error rate decreased from 45.7% to 4.25%. On the in-domain test set (56,396 tokens), CrossAttn-v1 achieved F1-macro = 0.8485 for recovering five-class punctuation marks. Furthermore, CrossAttn-v1 outperformed the GPT-4o zero-shot model by +0.294 F1 and the M3 Hybrid model based on prosody alone by +0.070 F1. The class analysis showed that the Whisper encoder states were particularly useful for prosody-dependent punctuation. For example, it outperformed M3 Hybrid by +9.5 percentage points on the QUESTION mark and by +20.2 percentage points on the EXCLAIM mark. On 883 out-of-domain natural speech recordings, the model performed similarly to the text-only baseline model (Δ = −0.041, not significant), suggesting that domain mismatch in the Whisper training corpus was a major factor limiting generalization.