TRTapani Raiko

Tapani Raiko

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Tapani Raiko has a total of 59 co-authors. Here are the top co-authors by number of shared publications:

  • Alexander Ilin

    Department of Information and Computer Science, School of Science at the Aalto University, Finland

    9 shared publications
  • Juha Karhunen

    Aalto University School of Science

    6 shared publications
  • Antti Honkela

    Helsinki Institute for Information Technology HIIT, University of Helsinki, Helsinki, Finland

    4 shared publications
  • KyungHyun Cho

    Aalto University School of Science, Department of Information and Computer Science, PO Box 15400, FI-00076, Finland

    4 shared publications
  • Tommi Vatanen

    Institute of Biotechnology, Helsinki Institute of Life Science, University of Helsinki, Helsinki, Finland

    4 shared publications
  • Matti Tornio

    Adaptive Informatics Research Centre, Department of Information and Computer Science, Helsinki University of Technology, Finland

    3 shared publications

Latest Publications

  • Open Access

    A Character-Word Compositional Neural Language Model for Finnish

    by Matti Lankinen, Hannes Heikinheimo, Pyry Takala, Tapani Raiko, Juha Karhunen

    Inspired by recent research, we explore ways to model the highly morphological Finnish language at the level of characters while maintaining the performance of word-level models. We propose a new Character-to-Word-to-Character (C2W2C) compositional language model that uses characters as input and output while still internally processing word level embeddings. Our preliminary experiments, using the Finnish Europarl V7 corpus, indicate that C2W2C can respond well to the challenges of morphologically rich languages such as high out of vocabulary rates, the prediction of novel words, and growing vocabulary size. Notably, the model is able to correctly score inflectional forms that are not present in the training data and sample grammatically and semantically correct Finnish sentences character by character.

  • Open Access

    DopeLearning: A Computational Approach to Rap Lyrics Generation

    by Eric Malmi, Pyry Takala, Hannu Toivonen, Tapani Raiko, Aristides Gionis

    Writing rap lyrics requires both creativity to construct a meaningful, interesting story and lyrical skills to produce complex rhyme patterns, which form the cornerstone of good flow. We present a rap lyrics generation method that captures both of these aspects. First, we develop a prediction model to identify the next line of existing lyrics from a set of candidate next lines. This model is based on two machine-learning techniques: the RankSVM algorithm and a deep neural network model with a novel structure. Results show that the prediction model can identify the true next line among 299 randomly selected lines with an accuracy of 17%, i.e., over 50 times more likely than by random. Second, we employ the prediction model to combine lines from existing songs, producing lyrics with rhyme and a meaning. An evaluation of the produced lyrics shows that in terms of quantitative rhyme density, the method outperforms the best human rappers by 21%. The rap lyrics generator has been deployed as an online tool called DeepBeat, and the performance of the tool has been assessed by analyzing its usage logs. This analysis shows that machine-learned rankings correlate with user preferences.

  • Open Access

    Ladder Variational Autoencoders

    by Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, Ole Winther

    Variational Autoencoders are powerful models for unsupervised learning. However deep models with several layers of dependent stochastic variables are difficult to train which limits the improvements obtained using these highly expressive models. We propose a new inference model, the Ladder Variational Autoencoder, that recursively corrects the generative distribution by a data dependent approximate likelihood in a process resembling the recently proposed Ladder Network. We show that this model provides state of the art predictive log-likelihood and tighter log-likelihood lower bound compared to the purely bottom-up inference in layered Variational Autoencoders and other generative models. We provide a detailed analysis of the learned hierarchical latent representation and show that our new inference model is qualitatively different and utilizes a deeper more distributed hierarchy of latent variables. Finally, we observe that batch normalization and deterministic warm-up (gradually turning on the KL-term) are crucial for training variational models with many stochastic layers.


  • Open Access

    Techniques for Learning Binary Stochastic Feedforward Neural Networks

    by Tapani Raiko, Mathias Berglund, Guillaume Alain, Laurent Dinh

    Stochastic binary hidden units in a multi-layer perceptron (MLP) network give at least three potential benefits when compared to deterministic MLP networks. (1) They allow to learn one-to-many type of mappings. (2) They can be used in structured prediction problems, where modeling the internal structure of the output is important. (3) Stochasticity has been shown to be an excellent regularizer, which makes generalization performance potentially better in general. However, training stochastic networks is considerably more difficult. We study training using M samples of hidden activations per input. We show that the case M=1 leads to a fundamentally different behavior where the network tries to avoid stochasticity. We propose two new estimators for the training gradient and propose benchmark tests for comparing training algorithms. Our experiments confirm that training stochastic networks is difficult and show that the proposed two estimators perform favorably among all the five known estimators.


  • Two-layer contractive encodings for learning stable nonlinear features

    by Hannes Schulz, Kyunghyun Cho, Tapani Raiko, Sven Behnke


  • Open Access

    Denoising autoencoder with modulated lateral connections learns invariant representations of natural images

    by Antti Rasmus, Tapani Raiko, Harri Valpola

    Suitable lateral connections between encoder and decoder are shown to allow higher layers of a denoising autoencoder (dAE) to focus on invariant representations. In regular autoencoders, detailed information needs to be carried through the highest layers but lateral connections from encoder to decoder relieve this pressure. It is shown that abstract invariant features can be translated to detailed reconstructions when invariant features are allowed to modulate the strength of the lateral connection. Three dAE structures with modulated and additive lateral connections, and without lateral connections were compared in experiments using real-world images. The experiments verify that adding modulated lateral connections to the model 1) improves the accuracy of the probability model for inputs, as measured by denoising performance; 2) results in representations whose degree of invariance grows faster towards the higher layers; and 3) supports the formation of diverse invariant poolings.

  • How to Pretrain Deep Boltzmann Machines in Two Stages

    by Kyunghyun Cho, Tapani Raiko, Alexander Ilin, Juha Karhunen


  • Self-organization and missing values in SOM and GTM

    by T. Vatanen, M. Osmala, T. Raiko, K. Lagus, M. Sysi-Aho, M. Orešič, T. Honkela, H. Lähdesmäki


  • Gaussian-Bernoulli deep Boltzmann machine

    by Kyung Hyun Cho, Tapani Raiko, Alexander Ilin

    In this paper, we study a model that we call Gaussian-Bernoulli deep Boltzmann machine (GDBM) and discuss potential improvements in training the model. GDBM is designed to be applicable to continuous data and it is constructed from Gaussian-Bernoulli restricted Boltzmann machine (GRBM) by adding multiple layers of binary hidden neurons. The studied improvements of the learning algorithm for GDBM include parallel tempering, enhanced gradient, adaptive learning rate and layer-wise pretraining. We empirically show that they help avoid some of the common difficulties found in training deep Boltzmann machines such as divergence of learning, the difficulty in choosing right learning rate scheduling, and the existence of meaningless higher layers.