Contrastive Value Learning: Implicit Models for Simple Offline RL

TY - JOUR

T1 - Contrastive Value Learning

T2 - 7th Conference on Robot Learning, CoRL 2023

AU - Mazoure, Bogdan

AU - Eysenbach, Benjamin

AU - Nachum, Ofir

AU - Tompson, Jonathan

N1 - Publisher Copyright: © 2023 Proceedings of Machine Learning Research. All Rights Reserved.

PY - 2023

Y1 - 2023

N2 - Model-based reinforcement learning (RL) methods are appealing in the offline setting because they allow an agent to reason about the consequences of actions without interacting with the environment. While conventional model-based methods learn a 1-step model, predicting the immediate next state, these methods must be plugged into larger planning or RL systems to yield a policy. Can we model the environment dynamics in a different way, such that the learned model directly indicates the value of each action? In this paper, we propose Contrastive Value Learning (CVL), which learns an implicit, multi-step dynamics model. This model can be learned without access to reward functions, but nonetheless can be used to directly estimate the value of each action, without requiring any TD learning. Because this model represents the multi-step transitions implicitly, it avoids having to predict high-dimensional observations and thus scales to high-dimensional tasks. Our experiments demonstrate that CVL outperforms prior offline RL methods on complex robotics benchmarks.

AB - Model-based reinforcement learning (RL) methods are appealing in the offline setting because they allow an agent to reason about the consequences of actions without interacting with the environment. While conventional model-based methods learn a 1-step model, predicting the immediate next state, these methods must be plugged into larger planning or RL systems to yield a policy. Can we model the environment dynamics in a different way, such that the learned model directly indicates the value of each action? In this paper, we propose Contrastive Value Learning (CVL), which learns an implicit, multi-step dynamics model. This model can be learned without access to reward functions, but nonetheless can be used to directly estimate the value of each action, without requiring any TD learning. Because this model represents the multi-step transitions implicitly, it avoids having to predict high-dimensional observations and thus scales to high-dimensional tasks. Our experiments demonstrate that CVL outperforms prior offline RL methods on complex robotics benchmarks.

UR - http://www.scopus.com/inward/record.url?scp=85184351550&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85184351550&partnerID=8YFLogxK

M3 - Conference article

SN - 2640-3498

VL - 229

JO - Proceedings of Machine Learning Research

JF - Proceedings of Machine Learning Research

Y2 - 6 November 2023 through 9 November 2023

ER -