Mono-STAR: Mono-Camera Scene-Level Tracking and Reconstruction

Haonan Chang; Dhruv Metha Ramesh; Shijie Geng; Yuqiu Gan; Abdeslam Boularias

doi:https://doi.org/10.1109/ICRA48891.2023.10160778

Mono-STAR: Mono-Camera Scene-Level Tracking and Reconstruction

Haonan Chang, Dhruv Metha Ramesh, Shijie Geng, Yuqiu Gan, Abdeslam Boularias

School of Arts and Sciences, Computer Science

Rutgers, The State University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We present Mono-STAR, the first real-time 3D reconstruction system that simultaneously supports semantic fusion, fast motion tracking, non-rigid object deformation, and topological change under a unified framework. The proposed system solves a new optimization problem incorporating optical-flow-based 2D constraints to deal with fast motion and a novel semantic-aware deformation graph (SAD-graph) for handling topology change. We test the proposed system under various challenging scenes and demonstrate that it significantly outperforms existing state-of-the-art methods. Supplementary material, including videos, can be found at https://github.com/changhaonan/Mono-STAR-demo.

Original language	English (US)
Title of host publication	Proceedings - ICRA 2023
Subtitle of host publication	IEEE International Conference on Robotics and Automation
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	820-826
Number of pages	7
ISBN (Electronic)	9798350323658
DOIs	https://doi.org/10.1109/ICRA48891.2023.10160778
State	Published - 2023
Event	2023 IEEE International Conference on Robotics and Automation, ICRA 2023 - London, United Kingdom Duration: May 29 2023 → Jun 2 2023

Publication series

Name	Proceedings - IEEE International Conference on Robotics and Automation
Volume	2023-May

Conference

Conference	2023 IEEE International Conference on Robotics and Automation, ICRA 2023
Country/Territory	United Kingdom
City	London
Period	5/29/23 → 6/2/23

ASJC Scopus subject areas

Software
Control and Systems Engineering
Electrical and Electronic Engineering
Artificial Intelligence

Access to Document

https://doi.org/10.1109/ICRA48891.2023.10160778

Cite this

Chang, H., Ramesh, D. M., Geng, S., Gan, Y., & Boularias, A. (2023). Mono-STAR: Mono-Camera Scene-Level Tracking and Reconstruction. In Proceedings - ICRA 2023: IEEE International Conference on Robotics and Automation (pp. 820-826). (Proceedings - IEEE International Conference on Robotics and Automation; Vol. 2023-May). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICRA48891.2023.10160778

@inproceedings{3e4bb06970174881bd40ec8ca6bdcedf,

title = "Mono-STAR: Mono-Camera Scene-Level Tracking and Reconstruction",

abstract = "We present Mono-STAR, the first real-time 3D reconstruction system that simultaneously supports semantic fusion, fast motion tracking, non-rigid object deformation, and topological change under a unified framework. The proposed system solves a new optimization problem incorporating optical-flow-based 2D constraints to deal with fast motion and a novel semantic-aware deformation graph (SAD-graph) for handling topology change. We test the proposed system under various challenging scenes and demonstrate that it significantly outperforms existing state-of-the-art methods. Supplementary material, including videos, can be found at https://github.com/changhaonan/Mono-STAR-demo.",

author = "Haonan Chang and Ramesh, {Dhruv Metha} and Shijie Geng and Yuqiu Gan and Abdeslam Boularias",

note = "Funding Information: 1 The authors are with the Department of Computer Science, Rutgers University, 08854 New Brunswick, USA. This work is supported by NSF awards 1846043 and 2132972. Publisher Copyright: {\textcopyright} 2023 IEEE.; 2023 IEEE International Conference on Robotics and Automation, ICRA 2023 ; Conference date: 29-05-2023 Through 02-06-2023",

year = "2023",

doi = "https://doi.org/10.1109/ICRA48891.2023.10160778",

language = "English (US)",

series = "Proceedings - IEEE International Conference on Robotics and Automation",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "820--826",

booktitle = "Proceedings - ICRA 2023",

address = "United States",

}

Chang, H, Ramesh, DM, Geng, S, Gan, Y & Boularias, A 2023, Mono-STAR: Mono-Camera Scene-Level Tracking and Reconstruction. in Proceedings - ICRA 2023: IEEE International Conference on Robotics and Automation. Proceedings - IEEE International Conference on Robotics and Automation, vol. 2023-May, Institute of Electrical and Electronics Engineers Inc., pp. 820-826, 2023 IEEE International Conference on Robotics and Automation, ICRA 2023, London, United Kingdom, 5/29/23. https://doi.org/10.1109/ICRA48891.2023.10160778

Mono-STAR: Mono-Camera Scene-Level Tracking and Reconstruction. / Chang, Haonan; Ramesh, Dhruv Metha; Geng, Shijie et al.
Proceedings - ICRA 2023: IEEE International Conference on Robotics and Automation. Institute of Electrical and Electronics Engineers Inc., 2023. p. 820-826 (Proceedings - IEEE International Conference on Robotics and Automation; Vol. 2023-May).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Mono-STAR

T2 - 2023 IEEE International Conference on Robotics and Automation, ICRA 2023

AU - Chang, Haonan

AU - Ramesh, Dhruv Metha

AU - Geng, Shijie

AU - Gan, Yuqiu

AU - Boularias, Abdeslam

N1 - Funding Information: 1 The authors are with the Department of Computer Science, Rutgers University, 08854 New Brunswick, USA. This work is supported by NSF awards 1846043 and 2132972. Publisher Copyright: © 2023 IEEE.

PY - 2023

Y1 - 2023

N2 - We present Mono-STAR, the first real-time 3D reconstruction system that simultaneously supports semantic fusion, fast motion tracking, non-rigid object deformation, and topological change under a unified framework. The proposed system solves a new optimization problem incorporating optical-flow-based 2D constraints to deal with fast motion and a novel semantic-aware deformation graph (SAD-graph) for handling topology change. We test the proposed system under various challenging scenes and demonstrate that it significantly outperforms existing state-of-the-art methods. Supplementary material, including videos, can be found at https://github.com/changhaonan/Mono-STAR-demo.

AB - We present Mono-STAR, the first real-time 3D reconstruction system that simultaneously supports semantic fusion, fast motion tracking, non-rigid object deformation, and topological change under a unified framework. The proposed system solves a new optimization problem incorporating optical-flow-based 2D constraints to deal with fast motion and a novel semantic-aware deformation graph (SAD-graph) for handling topology change. We test the proposed system under various challenging scenes and demonstrate that it significantly outperforms existing state-of-the-art methods. Supplementary material, including videos, can be found at https://github.com/changhaonan/Mono-STAR-demo.

UR - http://www.scopus.com/inward/record.url?scp=85168677765&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85168677765&partnerID=8YFLogxK

U2 - https://doi.org/10.1109/ICRA48891.2023.10160778

DO - https://doi.org/10.1109/ICRA48891.2023.10160778

M3 - Conference contribution

T3 - Proceedings - IEEE International Conference on Robotics and Automation

SP - 820

EP - 826

BT - Proceedings - ICRA 2023

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 29 May 2023 through 2 June 2023

ER -

Chang H, Ramesh DM, Geng S, Gan Y, Boularias A. Mono-STAR: Mono-Camera Scene-Level Tracking and Reconstruction. In Proceedings - ICRA 2023: IEEE International Conference on Robotics and Automation. Institute of Electrical and Electronics Engineers Inc. 2023. p. 820-826. (Proceedings - IEEE International Conference on Robotics and Automation). doi: https://doi.org/10.1109/ICRA48891.2023.10160778

Mono-STAR: Mono-Camera Scene-Level Tracking and Reconstruction

Abstract

Publication series

Conference

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this