Temporal Alignment for History Representation in Reinforcement Learning
Authors:
- 21 August 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2172-2178
Abstract
Environments in Reinforcement Learning are usually only partially observable. To address this problem, a possible solution is to provide the agent with information about the past. However, providing complete observations of numerous steps can be excessive. Inspired by human memory, we propose to represent history with only important changes in the environment and, in our approach, to obtain automatically this representation using self-supervision. Our method (TempAl) aligns temporally-close frames, revealing a general, slowly varying state of the environment. This procedure is based on contrastive loss, which pulls embeddings of nearby observations to each other while pushing away other samples from the batch. It can be interpreted as a metric that captures the temporal relations of observations. We propose to combine both common instantaneous and our history representation and we evaluate TempAl on all available Atari games from the Arcade Learning Environment. TempAl surpasses the instantaneous-only baseline in 35 environments out of 49. The source code of the method and of all the experiments is available at https://github.com/htdt/tempal.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Momentum Contrast for Unsupervised Visual Representation Learning2020
- Time-Contrastive Networks: Self-Supervised Learning from Video2018
- Unsupervised Learning of Spatiotemporally Coherent Metrics2015
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- The Arcade Learning Environment: An Evaluation Platform for General AgentsJournal of Artificial Intelligence Research, 2013
- Dimensionality Reduction by Learning an Invariant Mapping2006
- Slow Feature Analysis: Unsupervised Learning of InvariancesNeural Computation, 2002
- Planning and acting in partially observable stochastic domainsArtificial Intelligence, 1998
- Long Short-Term MemoryNeural Computation, 1997
- State of the Art—A Survey of Partially Observable Markov Decision Processes: Theory, Models, and AlgorithmsManagement Science, 1982