Immersive educational systems with procedure-oriented combinations of real and virtual environments

Zhou Zhang, Shaojin Zhang, Mingshao Zhang, Sven Esche

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Virtual reality (VR) is becoming increasingly popular in educational applications, but insufficient users' feel of immersion often slows the further adoption of VR. Many solutions with a focus on the results rather than the details of the interactions between the objects in the real and virtual worlds have been developed. Therefore, the real procedures are distorted and the users lose their perception of in-person participation. In order to improve the users' feel of immersion further and to simulate more realistic operations in VR, a procedure-oriented approach for the combination of real and virtual environments is proposed here. As its name implies, this approach emphasizes the details of the procedures, namely how to capture, track, operate and interoperate the real and virtual objects in a mixed environment. In order to illustrate this idea, a prototype of mixed real and virtual assembly, in con-junction with object recognition and rigid-object tracking functions based on robotic vision techniques, is presented as an example. This prototype is designed based on a game-based virtual la-oratory system, and the specific implementation is a planetary gear train experiment. In this experiment, all models of the parts with the information required for the assembly are created, labeled and added to the database of the virtual laboratory system. The physical parts are marked in order to facilitate object recognition and object tracking. During the experiment, the main assembly with one missing planetary gear is accomplished in a purely virtual environment. In the real world, the missing planetary gear is tracked by a Kinect while the user is manipulating this gear. Then, the system recognizes this gear based on the markers and couples the corresponding virtual model of that gear with the avatar's hand in the virtual environment. Afterward, the cam-era tracks the real part, and the user can adjust its pose and location to finish the final assembly. The main benefit of this implementation is that the user can take advantage of some simple real parts in conjunction with virtual models of sophisticated parts in order to get realistic experience with the assembly process.

Original languageEnglish (US)
Title of host publicationEngineering Education
PublisherAmerican Society of Mechanical Engineers (ASME)
Volume5
ISBN (Electronic)9780791852064
DOIs
StatePublished - Jan 1 2018
EventASME 2018 International Mechanical Engineering Congress and Exposition, IMECE 2018 - Pittsburgh, United States
Duration: Nov 9 2018Nov 15 2018

Other

OtherASME 2018 International Mechanical Engineering Congress and Exposition, IMECE 2018
CountryUnited States
CityPittsburgh
Period11/9/1811/15/18

Fingerprint

Virtual reality
Gears
Object recognition
Experiments
Cams
Robotics

Cite this

Zhang, Z., Zhang, S., Zhang, M., & Esche, S. (2018). Immersive educational systems with procedure-oriented combinations of real and virtual environments. In Engineering Education (Vol. 5). American Society of Mechanical Engineers (ASME). https://doi.org/10.1115/IMECE2018-86597
Zhang, Zhou ; Zhang, Shaojin ; Zhang, Mingshao ; Esche, Sven. / Immersive educational systems with procedure-oriented combinations of real and virtual environments. Engineering Education. Vol. 5 American Society of Mechanical Engineers (ASME), 2018.
@inproceedings{fe4d9750ad554dde81050be641b54f6d,
title = "Immersive educational systems with procedure-oriented combinations of real and virtual environments",
abstract = "Virtual reality (VR) is becoming increasingly popular in educational applications, but insufficient users' feel of immersion often slows the further adoption of VR. Many solutions with a focus on the results rather than the details of the interactions between the objects in the real and virtual worlds have been developed. Therefore, the real procedures are distorted and the users lose their perception of in-person participation. In order to improve the users' feel of immersion further and to simulate more realistic operations in VR, a procedure-oriented approach for the combination of real and virtual environments is proposed here. As its name implies, this approach emphasizes the details of the procedures, namely how to capture, track, operate and interoperate the real and virtual objects in a mixed environment. In order to illustrate this idea, a prototype of mixed real and virtual assembly, in con-junction with object recognition and rigid-object tracking functions based on robotic vision techniques, is presented as an example. This prototype is designed based on a game-based virtual la-oratory system, and the specific implementation is a planetary gear train experiment. In this experiment, all models of the parts with the information required for the assembly are created, labeled and added to the database of the virtual laboratory system. The physical parts are marked in order to facilitate object recognition and object tracking. During the experiment, the main assembly with one missing planetary gear is accomplished in a purely virtual environment. In the real world, the missing planetary gear is tracked by a Kinect while the user is manipulating this gear. Then, the system recognizes this gear based on the markers and couples the corresponding virtual model of that gear with the avatar's hand in the virtual environment. Afterward, the cam-era tracks the real part, and the user can adjust its pose and location to finish the final assembly. The main benefit of this implementation is that the user can take advantage of some simple real parts in conjunction with virtual models of sophisticated parts in order to get realistic experience with the assembly process.",
author = "Zhou Zhang and Shaojin Zhang and Mingshao Zhang and Sven Esche",
year = "2018",
month = "1",
day = "1",
doi = "https://doi.org/10.1115/IMECE2018-86597",
language = "English (US)",
volume = "5",
booktitle = "Engineering Education",
publisher = "American Society of Mechanical Engineers (ASME)",

}

Zhang, Z, Zhang, S, Zhang, M & Esche, S 2018, Immersive educational systems with procedure-oriented combinations of real and virtual environments. in Engineering Education. vol. 5, American Society of Mechanical Engineers (ASME), ASME 2018 International Mechanical Engineering Congress and Exposition, IMECE 2018, Pittsburgh, United States, 11/9/18. https://doi.org/10.1115/IMECE2018-86597

Immersive educational systems with procedure-oriented combinations of real and virtual environments. / Zhang, Zhou; Zhang, Shaojin; Zhang, Mingshao; Esche, Sven.

Engineering Education. Vol. 5 American Society of Mechanical Engineers (ASME), 2018.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Immersive educational systems with procedure-oriented combinations of real and virtual environments

AU - Zhang, Zhou

AU - Zhang, Shaojin

AU - Zhang, Mingshao

AU - Esche, Sven

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Virtual reality (VR) is becoming increasingly popular in educational applications, but insufficient users' feel of immersion often slows the further adoption of VR. Many solutions with a focus on the results rather than the details of the interactions between the objects in the real and virtual worlds have been developed. Therefore, the real procedures are distorted and the users lose their perception of in-person participation. In order to improve the users' feel of immersion further and to simulate more realistic operations in VR, a procedure-oriented approach for the combination of real and virtual environments is proposed here. As its name implies, this approach emphasizes the details of the procedures, namely how to capture, track, operate and interoperate the real and virtual objects in a mixed environment. In order to illustrate this idea, a prototype of mixed real and virtual assembly, in con-junction with object recognition and rigid-object tracking functions based on robotic vision techniques, is presented as an example. This prototype is designed based on a game-based virtual la-oratory system, and the specific implementation is a planetary gear train experiment. In this experiment, all models of the parts with the information required for the assembly are created, labeled and added to the database of the virtual laboratory system. The physical parts are marked in order to facilitate object recognition and object tracking. During the experiment, the main assembly with one missing planetary gear is accomplished in a purely virtual environment. In the real world, the missing planetary gear is tracked by a Kinect while the user is manipulating this gear. Then, the system recognizes this gear based on the markers and couples the corresponding virtual model of that gear with the avatar's hand in the virtual environment. Afterward, the cam-era tracks the real part, and the user can adjust its pose and location to finish the final assembly. The main benefit of this implementation is that the user can take advantage of some simple real parts in conjunction with virtual models of sophisticated parts in order to get realistic experience with the assembly process.

AB - Virtual reality (VR) is becoming increasingly popular in educational applications, but insufficient users' feel of immersion often slows the further adoption of VR. Many solutions with a focus on the results rather than the details of the interactions between the objects in the real and virtual worlds have been developed. Therefore, the real procedures are distorted and the users lose their perception of in-person participation. In order to improve the users' feel of immersion further and to simulate more realistic operations in VR, a procedure-oriented approach for the combination of real and virtual environments is proposed here. As its name implies, this approach emphasizes the details of the procedures, namely how to capture, track, operate and interoperate the real and virtual objects in a mixed environment. In order to illustrate this idea, a prototype of mixed real and virtual assembly, in con-junction with object recognition and rigid-object tracking functions based on robotic vision techniques, is presented as an example. This prototype is designed based on a game-based virtual la-oratory system, and the specific implementation is a planetary gear train experiment. In this experiment, all models of the parts with the information required for the assembly are created, labeled and added to the database of the virtual laboratory system. The physical parts are marked in order to facilitate object recognition and object tracking. During the experiment, the main assembly with one missing planetary gear is accomplished in a purely virtual environment. In the real world, the missing planetary gear is tracked by a Kinect while the user is manipulating this gear. Then, the system recognizes this gear based on the markers and couples the corresponding virtual model of that gear with the avatar's hand in the virtual environment. Afterward, the cam-era tracks the real part, and the user can adjust its pose and location to finish the final assembly. The main benefit of this implementation is that the user can take advantage of some simple real parts in conjunction with virtual models of sophisticated parts in order to get realistic experience with the assembly process.

UR - http://www.scopus.com/inward/record.url?scp=85060293929&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85060293929&partnerID=8YFLogxK

U2 - https://doi.org/10.1115/IMECE2018-86597

DO - https://doi.org/10.1115/IMECE2018-86597

M3 - Conference contribution

VL - 5

BT - Engineering Education

PB - American Society of Mechanical Engineers (ASME)

ER -

Zhang Z, Zhang S, Zhang M, Esche S. Immersive educational systems with procedure-oriented combinations of real and virtual environments. In Engineering Education. Vol. 5. American Society of Mechanical Engineers (ASME). 2018 https://doi.org/10.1115/IMECE2018-86597