GBGino Brunner

Gino Brunner

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Gino Brunner has a total of 16 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Of Non-Linearity and Commutativity in BERT

    by Sumu Zhao, Damian Pascual, Gino Brunner, Roger Wattenhofer

    In this work we provide new insights into the transformer architecture, and in particular, its best-known variant, BERT. First, we propose a method to measure the degree of non-linearity of different elements of transformers. Next, we focus our investigation on the feed-forward networks (FFN) inside transformers, which contain 2/3 of the model parameters and have so far not received much attention. We find that FFNs are an inefficient yet important architectural element and that they cannot simply be replaced by attention blocks without a degradation in performance. Moreover, we study the interactions between layers in BERT and show that, while the layers exhibit some hierarchical structure, they extract features in a fuzzy manner. Our results suggest that BERT has an inductive bias towards layer commutativity, which we find is mainly due to the skip connections. This provides a justification for the strong performance of recurrent and weight-shared transformer models.

  • Open Access

    Telling BERT's full story: from Local Attention to Global Aggregation

    by Damian Pascual, Gino Brunner, Roger Wattenhofer

    We take a deep look into the behavior of self-attention heads in the transformer architecture. In light of recent work discouraging the use of attention distributions for explaining a model's behavior, we show that attention distributions can nevertheless provide insights into the local behavior of attention heads. This way, we propose a distinction between local patterns revealed by attention and global patterns that refer back to the input, and analyze BERT from both angles. We use gradient attribution to analyze how the output of an attention attention head depends on the input tokens, effectively extending the local attention-based analysis to account for the mixing of information throughout the transformer layers. We find that there is a significant discrepancy between attention and attribution distributions, caused by the mixing of context inside the model. We quantify this discrepancy and observe that interestingly, there are some patterns that persist across all layers despite the mixing.

  • Attentive Multi-task Deep Reinforcement Learning

    by Timo Bräm, Gino Brunner, Oliver Richter, Roger Wattenhofer


  • Neural Symbolic Music Genre Transfer Insights

    by Gino Brunner, Mazda Moayeri, Oliver Richter, Roger Wattenhofer, Chi Zhang


  • Open Access

    Recognition and Repetition Counting for Complex Physical Exercises with Deep Learning

    by Andrea Soro, Gino Brunner, Simon Tanner, Roger Wattenhofer

    Activity recognition using off-the-shelf smartwatches is an important problem in human activity recognition. In this paper, we present an end-to-end deep learning approach, able to provide probability distributions over activities from raw sensor data. We apply our methods to 10 complex full-body exercises typical in CrossFit, and achieve a classification accuracy of 99.96%. We additionally show that the same neural network used for exercise recognition can also be used in repetition counting. To the best of our knowledge, our approach to repetition counting is novel and performs well, counting correctly within an error of ±1 repetitions in 91% of the performed sets.


  • Disentangling the Latent Space of (Variational) Autoencoders for NLP

    by Gino Brunner, Yuyi Wang, Roger Wattenhofer, Michael Weigelt


  • Open Access

    Natural Language Multitasking: Analyzing and Improving Syntactic Saliency of Hidden Representations

    by Gino Brunner, Yuyi Wang, Roger Wattenhofer, Michael Weigelt

    We train multi-task autoencoders on linguistic tasks and analyze the learned hidden sentence representations. The representations change significantly when translation and part-of-speech decoders are added. The more decoders a model employs, the better it clusters sentences according to their syntactic similarity, as the representation space becomes less entangled. We explore the structure of the representation space by interpolating between sentences, which yields interesting pseudo-English sentences, many of which have recognizable syntactic structure. Lastly, we point out an interesting property of our models: The difference-vector between two sentences can be added to change a third sentence with similar features in a meaningful way.

  • Open Access

    JamBot: Music Theory Aware Chord Based Generation of Polyphonic Music with LSTMs

    by Gino Brunner, Yuyi Wang, Roger Wattenhofer, Jonas Wiesendanger

    We propose a novel approach for the generation of polyphonic music based on LSTMs. We generate music in two steps. First, a chord LSTM predicts a chord progression based on a chord embedding. A second LSTM then generates polyphonic music from the predicted chord progression. The generated music sounds pleasing and harmonic, with only few dissonant notes. It has clear long-term structure that is similar to what a musician would play during a jam session. We show that our approach is sensible from a music theory perspective by evaluating the learned chord embeddings. Surprisingly, our simple model managed to extract the circle of fifths, an important tool in music theory, from the dataset.


  • Open Access

    Teaching a Machine to Read Maps with Deep Reinforcement Learning

    by Gino Brunner, Oliver Richter, Yuyi Wang, Roger Wattenhofer

    The ability to use a 2D map to navigate a complex 3D environment is quite remarkable, and even difficult for many humans. Localization and navigation is also an important problem in domains such as robotics, and has recently become a focus of the deep reinforcement learning community. In this paper we teach a reinforcement learning agent to read a map in order to find the shortest way out of a random maze it has never seen before. Our system combines several state-of-the-art methods such as A3C and incorporates novel elements such as a recurrent localization cell. Our agent learns to localize itself based on 3D first person images and an approximate orientation angle. The agent generalizes well to bigger mazes, showing that it learned useful localization and navigation capabilities.