Reinforcement Learning-based Adaptive Trajectory Planning for AUVs in Under-ice Environments

Chaofeng Wang, Li Wei, Zhaohui Wang, Min Song, Nina Mahmoudian

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

This work studies online learning-based trajectory planning for multiple autonomous underwater vehicles (AUVs) to estimate a water parameter field of interest in the under-ice environment. A centralized system is considered, where several fixed access points (APs) on the ice layer are introduced as gateways for communications between the AUVs and a remote data fusion center (FC). We model the water parameter field of interest as a Gaussian process (GP) with unknown hyper-parameters. The AUV trajectories for sampling are determined on an epoch-by-epoch basis. At the end of each epoch, the APs relay the observed field samples from all the AUVs to the FC which computes the posterior distribution of the field based on the Gaussian process regression (GPR) and estimates the field hyper-parameters. The optimal trajectories of all the AUVs in the next epoch are determined to minimize a long-term cost that is defined based on the field uncertainty reduction and the AUV mobility cost, subject to the kinematics constraint, the communication range constraint and the sensing area constraint. We formulate the adaptive trajectory planning problem as a Markov decision process (MDP). A reinforcement learning (RL)-based online learning method is designed to determine the optimal AUV trajectories in a constrained continuous space. Simulation results show that the proposed learning-based trajectory planning algorithm has performance similar to a benchmark method that assumes perfect knowledge of the field hyper-parameters.

Original languageEnglish (US)
Title of host publicationOCEANS 2018 MTS/IEEE Charleston, OCEAN 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781538648148
DOIs
StatePublished - Jan 7 2019
EventOCEANS 2018 MTS/IEEE Charleston, OCEANS 2018 - Charleston, United States
Duration: Oct 22 2018Oct 25 2018

Publication series

NameOCEANS 2018 MTS/IEEE Charleston, OCEAN 2018

Conference

ConferenceOCEANS 2018 MTS/IEEE Charleston, OCEANS 2018
CountryUnited States
CityCharleston
Period10/22/1810/25/18

Fingerprint

epontic environment
Autonomous underwater vehicles
autonomous underwater vehicle
Reinforcement learning
reinforcement
Ice
learning
trajectory
Trajectories
Planning
communication
Communication
Data fusion
planning
cost
Costs
Water
Kinematics
kinematics
parameter

All Science Journal Classification (ASJC) codes

  • Oceanography
  • Renewable Energy, Sustainability and the Environment

Cite this

Wang, C., Wei, L., Wang, Z., Song, M., & Mahmoudian, N. (2019). Reinforcement Learning-based Adaptive Trajectory Planning for AUVs in Under-ice Environments. In OCEANS 2018 MTS/IEEE Charleston, OCEAN 2018 [8604754] (OCEANS 2018 MTS/IEEE Charleston, OCEAN 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/OCEANS.2018.8604754
Wang, Chaofeng ; Wei, Li ; Wang, Zhaohui ; Song, Min ; Mahmoudian, Nina. / Reinforcement Learning-based Adaptive Trajectory Planning for AUVs in Under-ice Environments. OCEANS 2018 MTS/IEEE Charleston, OCEAN 2018. Institute of Electrical and Electronics Engineers Inc., 2019. (OCEANS 2018 MTS/IEEE Charleston, OCEAN 2018).
@inproceedings{6c1666711ba04ed98771fbd9ff9e30a8,
title = "Reinforcement Learning-based Adaptive Trajectory Planning for AUVs in Under-ice Environments",
abstract = "This work studies online learning-based trajectory planning for multiple autonomous underwater vehicles (AUVs) to estimate a water parameter field of interest in the under-ice environment. A centralized system is considered, where several fixed access points (APs) on the ice layer are introduced as gateways for communications between the AUVs and a remote data fusion center (FC). We model the water parameter field of interest as a Gaussian process (GP) with unknown hyper-parameters. The AUV trajectories for sampling are determined on an epoch-by-epoch basis. At the end of each epoch, the APs relay the observed field samples from all the AUVs to the FC which computes the posterior distribution of the field based on the Gaussian process regression (GPR) and estimates the field hyper-parameters. The optimal trajectories of all the AUVs in the next epoch are determined to minimize a long-term cost that is defined based on the field uncertainty reduction and the AUV mobility cost, subject to the kinematics constraint, the communication range constraint and the sensing area constraint. We formulate the adaptive trajectory planning problem as a Markov decision process (MDP). A reinforcement learning (RL)-based online learning method is designed to determine the optimal AUV trajectories in a constrained continuous space. Simulation results show that the proposed learning-based trajectory planning algorithm has performance similar to a benchmark method that assumes perfect knowledge of the field hyper-parameters.",
author = "Chaofeng Wang and Li Wei and Zhaohui Wang and Min Song and Nina Mahmoudian",
year = "2019",
month = "1",
day = "7",
doi = "https://doi.org/10.1109/OCEANS.2018.8604754",
language = "English (US)",
series = "OCEANS 2018 MTS/IEEE Charleston, OCEAN 2018",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
booktitle = "OCEANS 2018 MTS/IEEE Charleston, OCEAN 2018",
address = "United States",

}

Wang, C, Wei, L, Wang, Z, Song, M & Mahmoudian, N 2019, Reinforcement Learning-based Adaptive Trajectory Planning for AUVs in Under-ice Environments. in OCEANS 2018 MTS/IEEE Charleston, OCEAN 2018., 8604754, OCEANS 2018 MTS/IEEE Charleston, OCEAN 2018, Institute of Electrical and Electronics Engineers Inc., OCEANS 2018 MTS/IEEE Charleston, OCEANS 2018, Charleston, United States, 10/22/18. https://doi.org/10.1109/OCEANS.2018.8604754

Reinforcement Learning-based Adaptive Trajectory Planning for AUVs in Under-ice Environments. / Wang, Chaofeng; Wei, Li; Wang, Zhaohui; Song, Min; Mahmoudian, Nina.

OCEANS 2018 MTS/IEEE Charleston, OCEAN 2018. Institute of Electrical and Electronics Engineers Inc., 2019. 8604754 (OCEANS 2018 MTS/IEEE Charleston, OCEAN 2018).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Reinforcement Learning-based Adaptive Trajectory Planning for AUVs in Under-ice Environments

AU - Wang, Chaofeng

AU - Wei, Li

AU - Wang, Zhaohui

AU - Song, Min

AU - Mahmoudian, Nina

PY - 2019/1/7

Y1 - 2019/1/7

N2 - This work studies online learning-based trajectory planning for multiple autonomous underwater vehicles (AUVs) to estimate a water parameter field of interest in the under-ice environment. A centralized system is considered, where several fixed access points (APs) on the ice layer are introduced as gateways for communications between the AUVs and a remote data fusion center (FC). We model the water parameter field of interest as a Gaussian process (GP) with unknown hyper-parameters. The AUV trajectories for sampling are determined on an epoch-by-epoch basis. At the end of each epoch, the APs relay the observed field samples from all the AUVs to the FC which computes the posterior distribution of the field based on the Gaussian process regression (GPR) and estimates the field hyper-parameters. The optimal trajectories of all the AUVs in the next epoch are determined to minimize a long-term cost that is defined based on the field uncertainty reduction and the AUV mobility cost, subject to the kinematics constraint, the communication range constraint and the sensing area constraint. We formulate the adaptive trajectory planning problem as a Markov decision process (MDP). A reinforcement learning (RL)-based online learning method is designed to determine the optimal AUV trajectories in a constrained continuous space. Simulation results show that the proposed learning-based trajectory planning algorithm has performance similar to a benchmark method that assumes perfect knowledge of the field hyper-parameters.

AB - This work studies online learning-based trajectory planning for multiple autonomous underwater vehicles (AUVs) to estimate a water parameter field of interest in the under-ice environment. A centralized system is considered, where several fixed access points (APs) on the ice layer are introduced as gateways for communications between the AUVs and a remote data fusion center (FC). We model the water parameter field of interest as a Gaussian process (GP) with unknown hyper-parameters. The AUV trajectories for sampling are determined on an epoch-by-epoch basis. At the end of each epoch, the APs relay the observed field samples from all the AUVs to the FC which computes the posterior distribution of the field based on the Gaussian process regression (GPR) and estimates the field hyper-parameters. The optimal trajectories of all the AUVs in the next epoch are determined to minimize a long-term cost that is defined based on the field uncertainty reduction and the AUV mobility cost, subject to the kinematics constraint, the communication range constraint and the sensing area constraint. We formulate the adaptive trajectory planning problem as a Markov decision process (MDP). A reinforcement learning (RL)-based online learning method is designed to determine the optimal AUV trajectories in a constrained continuous space. Simulation results show that the proposed learning-based trajectory planning algorithm has performance similar to a benchmark method that assumes perfect knowledge of the field hyper-parameters.

UR - http://www.scopus.com/inward/record.url?scp=85061814829&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85061814829&partnerID=8YFLogxK

U2 - https://doi.org/10.1109/OCEANS.2018.8604754

DO - https://doi.org/10.1109/OCEANS.2018.8604754

M3 - Conference contribution

T3 - OCEANS 2018 MTS/IEEE Charleston, OCEAN 2018

BT - OCEANS 2018 MTS/IEEE Charleston, OCEAN 2018

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Wang C, Wei L, Wang Z, Song M, Mahmoudian N. Reinforcement Learning-based Adaptive Trajectory Planning for AUVs in Under-ice Environments. In OCEANS 2018 MTS/IEEE Charleston, OCEAN 2018. Institute of Electrical and Electronics Engineers Inc. 2019. 8604754. (OCEANS 2018 MTS/IEEE Charleston, OCEAN 2018). https://doi.org/10.1109/OCEANS.2018.8604754