Improved Offline Reinforcement Learning via Quantum Metric Encoding
Preprint
Abstract
Reinforcement learning (RL) with limited samples is common in real-world applications. However, offline RL performance under this constraint is often suboptimal. We consider an alternative approach to dealing with limited samples by introducing the Quantum Metric Encoder (QME). In this methodology, instead of applying the RL framework directly on the original states and rewards, we embed the states into a more compact and meaningful representation, where the structure of the encoding is inspired by quantum circuits. For classical data, QME is a classically simulable, trainable unitary embedding and thus serves as a quantum-inspired module, on a classical device. For quantum data in the form of quantum states, QME can be implemented directly on quantum hardware, allowing for training without measurement or re-encoding. We evaluated QME on three datasets, each limited to 100 samples. We use Soft-Actor-Critic (SAC) and Implicit-Q-Learning (IQL), two well-known RL algorithms, to demonstrate the effectiveness of our approach. From the experimental results, we find that training offline RL agents on QME-embedded states with decoded rewards yields significantly better performance than training on the original states and rewards. On average across the three datasets, for maximum reward performance, we achieve a 116.2% improvement for SAC and 117.6% for IQL. We further investigate the $Δ$-hyperbolicity of our framework, a geometric property of the state space known to be important for the RL training efficacy. The QME-embedded states exhibit low $Δ$-hyperbolicity, suggesting that the improvement after embedding arises from the modified geometry of the state space induced by QME. Thus, the low $Δ$-hyperbolicity and the corresponding effectiveness of QME could provide valuable information for developing efficient offline RL methods under limited-sample conditions.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- Does Reinforcement Learning Improve Outcomes for Critically Ill Patients? A Systematic Review and Level-of-Readiness AssessmentCritical Care Medicine, 2023
- An adaptive reinforcement learning-based multimodal data fusion framework for human–robot confrontation gamingNeural Networks, 2023
- Making machine learning matter to clinicians: model actionability in medical decision-makingnpj Digital Medicine, 2023
- Generalization in quantum machine learning from few training dataNature Communications, 2022
- Supervised learning with quantum-enhanced feature spacesNature, 2019
- Quantum autoencoders for efficient compression of quantum dataQuantum Science and Technology, 2017
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- Computing the Gromov hyperbolicity of a discrete metric spaceInformation Processing Letters, 2015
- Direct search algorithms for optimization calculationsActa Numerica, 1998
- Constrained Optimization and Lagrange Multiplier Methods1982