TY - GEN
T1 - A new framework for sign language recognition based on 3D handshape identification and linguistic modeling
AU - Dilsizian, Mark
AU - Yanovich, Polina
AU - Wang, Shu
AU - Neidle, Carol
AU - Metaxas, Dimitris
N1 - Funding Information: For the handshape synthetic dataset, we thank Matt Huener-fauth and the signing participants. The ASLLVD was constructed in collaboration with Ashwin Thangali and Stan Sclaroff (Neidle et al., 2012). Thangali also developed an excellent interface to facilitate verifications of the linguistic annotations and viewing and searching the dataset, which has greatly facilitated our work. We also thank Vas-silis Athitsos, Ben Bahan, Christian Vogler, Iryna Zhu-ravlova, and the many signers, students, and annotators who have contributed to that project, including especially Naomi Berlove, Elizabeth Cassidy, Lana Cook, Braden Painter, Tyler Richard, Dana Schlang, Robert G. Lee, Joan Nash, Tory Sampson, Donna Riggle, Amelia Wisniewski-Barker, and Rachel Benedict, and annotators who have contributed to that project. The work reported here has been partially funded by grants from the National Science Foundation (#IIS-1065013, #IIS-0964385, #EIA-9528985, #CNS-1059218).
PY - 2014
Y1 - 2014
N2 - Current approaches to sign recognition by computer generally have at least some of the following limitations: they rely on laboratory conditions for sign production, are limited to a small vocabulary, rely on 2D modeling (and therefore cannot deal with occlusions and off-plane rotations), and/or achieve limited success. Here we propose a new framework that (1) provides a new tracking method less dependent than others on laboratory conditions and able to deal with variations in background and skin regions (such as the face, forearms, or other hands); (2) allows for identification of 3D hand configurations that are linguistically important in American Sign Language (ASL); and (3) incorporates statistical information reflecting linguistic constraints in sign production. For purposes of large-scale computer-based sign language recognition from video, the ability to distinguish hand configurations accurately is critical. Our current method estimates the 3D hand configuration to distinguish among 77 hand configurations linguistically relevant for ASL. Constraining the problem in this way makes recognition of 3D hand configuration more tractable and provides the information specifically needed for sign recognition. Further improvements are obtained by incorporation of statistical information about linguistic dependencies among handshapes within a sign derived from an annotated corpus of almost 10, 000 sign tokens.
AB - Current approaches to sign recognition by computer generally have at least some of the following limitations: they rely on laboratory conditions for sign production, are limited to a small vocabulary, rely on 2D modeling (and therefore cannot deal with occlusions and off-plane rotations), and/or achieve limited success. Here we propose a new framework that (1) provides a new tracking method less dependent than others on laboratory conditions and able to deal with variations in background and skin regions (such as the face, forearms, or other hands); (2) allows for identification of 3D hand configurations that are linguistically important in American Sign Language (ASL); and (3) incorporates statistical information reflecting linguistic constraints in sign production. For purposes of large-scale computer-based sign language recognition from video, the ability to distinguish hand configurations accurately is critical. Our current method estimates the 3D hand configuration to distinguish among 77 hand configurations linguistically relevant for ASL. Constraining the problem in this way makes recognition of 3D hand configuration more tractable and provides the information specifically needed for sign recognition. Further improvements are obtained by incorporation of statistical information about linguistic dependencies among handshapes within a sign derived from an annotated corpus of almost 10, 000 sign tokens.
KW - 3D hand configurations
KW - ASL
KW - Hand tracking
KW - Handshape dependencies
KW - Structured prediction
UR - https://www.scopus.com/pages/publications/84948403362
UR - https://www.scopus.com/pages/publications/84948403362#tab=citedBy
M3 - Conference contribution
T3 - Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014
SP - 1924
EP - 1929
BT - Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014
A2 - Calzolari, Nicoletta
A2 - Choukri, Khalid
A2 - Goggi, Sara
A2 - Declerck, Thierry
A2 - Mariani, Joseph
A2 - Maegaard, Bente
A2 - Moreno, Asuncion
A2 - Odijk, Jan
A2 - Mazo, Helene
A2 - Piperidis, Stelios
A2 - Loftsson, Hrafn
PB - European Language Resources Association (ELRA)
T2 - 9th International Conference on Language Resources and Evaluation, LREC 2014
Y2 - 26 May 2014 through 31 May 2014
ER -