Optimal control with learning on the fly: A toy problem

TY - JOUR

T1 - Optimal control with learning on the fly

T2 - A toy problem

AU - Fefferman, Charles L.

AU - Pegueroles, Bernat Guillen

AU - Rowley, Clarence W.

AU - Weber, Melanie

N1 - Funding Information: Funding. This work was supported by the Air Force Office of Scientific Research, under awards FA9550-19-1-0005 and FA9550-18-1-0069, and partially supported by the National Science Foundation, under grant DMS-1700180. Publisher Copyright: © 2022 European Mathematical Society Publishing House. All rights reserved.

PY - 2022

Y1 - 2022

N2 - We exhibit optimal control strategies for a simple toy problem in which the underlying dynamics depend on a parameter that is initially unknown and must be learned.We consider a cost function posed over a finite time interval, in contrast to much previous work that considers asymptotics as the time horizon tends to infinity. We study several different versions of the problem, including Bayesian control, in which we assume a prior distribution on the unknown parameter; and "agnostic" control, in which we assume nothing about the unknown parameter. For the agnostic problems, we compare our performance with that of an opponent who knows the value of the parameter. This comparison gives rise to several notions of "regret", and we obtain strategies that minimize the "worst-case regret" arising from the most unfavorable choice of the unknown parameter. In every case, the optimal strategy turns out to be a Bayesian strategy or a limit of Bayesian strategies.

AB - We exhibit optimal control strategies for a simple toy problem in which the underlying dynamics depend on a parameter that is initially unknown and must be learned.We consider a cost function posed over a finite time interval, in contrast to much previous work that considers asymptotics as the time horizon tends to infinity. We study several different versions of the problem, including Bayesian control, in which we assume a prior distribution on the unknown parameter; and "agnostic" control, in which we assume nothing about the unknown parameter. For the agnostic problems, we compare our performance with that of an opponent who knows the value of the parameter. This comparison gives rise to several notions of "regret", and we obtain strategies that minimize the "worst-case regret" arising from the most unfavorable choice of the unknown parameter. In every case, the optimal strategy turns out to be a Bayesian strategy or a limit of Bayesian strategies.

KW - Adaptive control

KW - Agnostic control

KW - Competitive ratio

KW - Fuel tax regret

KW - Regret

UR - http://www.scopus.com/inward/record.url?scp=85125189236&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85125189236&partnerID=8YFLogxK

U2 - 10.4171/RMI/1275

DO - 10.4171/RMI/1275

M3 - Article

SN - 0213-2230

VL - 38

SP - 175

EP - 187

JO - Revista Matematica Iberoamericana

JF - Revista Matematica Iberoamericana

IS - 1

ER -