Heterogeneous Crowd Simulation Using Parametric Reinforcement Learning

TY - JOUR

T1 - Heterogeneous Crowd Simulation Using Parametric Reinforcement Learning

AU - Hu, Kaidong

AU - Haworth, Brandon

AU - Berseth, Glen

AU - Pavlovic, Vladimir

AU - Faloutsos, Petros

AU - Kapadia, Mubbasir

N1 - Publisher Copyright: © 1995-2012 IEEE.

PY - 2023/4/1

Y1 - 2023/4/1

N2 - Agent-based synthetic crowd simulation affords the cost-effective large-scale simulation and animation of interacting digital humans. Model-based approaches have successfully generated a plethora of simulators with a variety of foundations. However, prior approaches have been based on statically defined models predicated on simplifying assumptions, limited video-based datasets, or homogeneous policies. Recent works have applied reinforcement learning to learn policies for navigation. However, these approaches may learn static homogeneous rules, are typically limited in their generalization to trained scenarios, and limited in their usability in synthetic crowd domains. In this article, we present a multi-agent reinforcement learning-based approach that learns a parametric predictive collision avoidance and steering policy. We show that training over a parameter space produces a flexible model across crowd configurations. That is, our goal-conditioned approach learns a parametric policy that affords heterogeneous synthetic crowds. We propose a model-free approach without centralization of internal agent information, control signals, or agent communication. The model is extensively evaluated. The results show policy generalization across unseen scenarios, agent parameters, and out-of-distribution parameterizations. The learned model has comparable computational performance to traditional methods. Qualitatively the model produces both expected (laminar flow, shuffling, bottleneck) and unexpected (side-stepping) emergent qualitative behaviours, and quantitatively the approach is performant across measures of movement quality.

AB - Agent-based synthetic crowd simulation affords the cost-effective large-scale simulation and animation of interacting digital humans. Model-based approaches have successfully generated a plethora of simulators with a variety of foundations. However, prior approaches have been based on statically defined models predicated on simplifying assumptions, limited video-based datasets, or homogeneous policies. Recent works have applied reinforcement learning to learn policies for navigation. However, these approaches may learn static homogeneous rules, are typically limited in their generalization to trained scenarios, and limited in their usability in synthetic crowd domains. In this article, we present a multi-agent reinforcement learning-based approach that learns a parametric predictive collision avoidance and steering policy. We show that training over a parameter space produces a flexible model across crowd configurations. That is, our goal-conditioned approach learns a parametric policy that affords heterogeneous synthetic crowds. We propose a model-free approach without centralization of internal agent information, control signals, or agent communication. The model is extensively evaluated. The results show policy generalization across unseen scenarios, agent parameters, and out-of-distribution parameterizations. The learned model has comparable computational performance to traditional methods. Qualitatively the model produces both expected (laminar flow, shuffling, bottleneck) and unexpected (side-stepping) emergent qualitative behaviours, and quantitatively the approach is performant across measures of movement quality.

KW - Multi-agent navigation

KW - crowd simulation

KW - parametric policy learning

KW - reinforcement learning

UR - http://www.scopus.com/inward/record.url?scp=85122304227&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85122304227&partnerID=8YFLogxK

U2 - 10.1109/TVCG.2021.3139031

DO - 10.1109/TVCG.2021.3139031

M3 - Article

C2 - 34965213

SN - 1077-2626

VL - 29

SP - 2036

EP - 2052

JO - IEEE Transactions on Visualization and Computer Graphics

JF - IEEE Transactions on Visualization and Computer Graphics

IS - 4

ER -