Detecting visual text

Jesse Dodge; Amit Goyal; Xufeng Han; Alyssa Mensch; Margaret Mitchell; Karl Stratos; Kota Yamaguchi; Yejin Choi; Hal Daumé; Alexander C. Berg; Tamara L. Berg

Detecting visual text

Jesse Dodge
, Amit Goyal
, Xufeng Han
, Alyssa Mensch
, Margaret Mitchell
, Karl Stratos
, Kota Yamaguchi
, Yejin Choi
, Hal Daumé
, Alexander C. Berg
, Tamara L. Berg

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

When people describe a scene, they often include information that is not visually apparent; sometimes based on background knowledge, sometimes to tell a story. We aim to separate visual text - descriptions of what is being seen - from non-visual text in natural images and their descriptions. To do so, we first concretely define what it means to be visual, annotate visual text and then develop algorithms to automatically classify noun phrases as visual or non-visual. We find that using text alone, we are able to achieve high accuracies at this task, and that incorporating features derived from computer vision algorithms improves performance. Finally, we show that we can reliably mine visual nouns and adjectives from large corpora and that we can use these effectively in the classification task.

Original language	American English
Title of host publication	Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics
Subtitle of host publication	Human Language Technologies
Publisher	Association for Computational Linguistics (ACL)
Pages	762-772
Number of pages	11
ISBN (Electronic)	1937284204, 9781937284206
State	Published - 2012
Externally published	Yes
Event	2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2012 - Montreal, Canada Duration: Jun 3 2012 → Jun 8 2012

Publication series

Name	NAACL HLT 2012 - 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference

Conference

Conference	2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2012
Country/Territory	Canada
City	Montreal
Period	6/3/12 → 6/8/12

ASJC Scopus subject areas

Language and Linguistics
Computer Science Applications
Linguistics and Language

Cite this

Dodge, J., Goyal, A., Han, X., Mensch, A., Mitchell, M., Stratos, K., Yamaguchi, K., Choi, Y., Daumé, H., Berg, A. C., & Berg, T. L. (2012). Detecting visual text. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 762-772). (NAACL HLT 2012 - 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference). Association for Computational Linguistics (ACL).

Dodge, Jesse ; Goyal, Amit ; Han, Xufeng et al. / Detecting visual text. Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics (ACL), 2012. pp. 762-772 (NAACL HLT 2012 - 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference).

@inproceedings{2f7156d034d5461a8ebec2a24e0cee83,

title = "Detecting visual text",

abstract = "When people describe a scene, they often include information that is not visually apparent; sometimes based on background knowledge, sometimes to tell a story. We aim to separate visual text - descriptions of what is being seen - from non-visual text in natural images and their descriptions. To do so, we first concretely define what it means to be visual, annotate visual text and then develop algorithms to automatically classify noun phrases as visual or non-visual. We find that using text alone, we are able to achieve high accuracies at this task, and that incorporating features derived from computer vision algorithms improves performance. Finally, we show that we can reliably mine visual nouns and adjectives from large corpora and that we can use these effectively in the classification task.",

author = "Jesse Dodge and Amit Goyal and Xufeng Han and Alyssa Mensch and Margaret Mitchell and Karl Stratos and Kota Yamaguchi and Yejin Choi and Hal Daum{\'e} and Berg, \{Alexander C.\} and Berg, \{Tamara L.\}",

note = "Publisher Copyright: {\textcopyright} 2012 Association for Computational Linguistics.; 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2012 ; Conference date: 03-06-2012 Through 08-06-2012",

year = "2012",

language = "American English",

series = "NAACL HLT 2012 - 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference",

publisher = "Association for Computational Linguistics (ACL)",

pages = "762--772",

booktitle = "Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics",

}

Dodge, J, Goyal, A, Han, X, Mensch, A, Mitchell, M, Stratos, K, Yamaguchi, K, Choi, Y, Daumé, H, Berg, AC & Berg, TL 2012, Detecting visual text. in Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL HLT 2012 - 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, Association for Computational Linguistics (ACL), pp. 762-772, 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2012, Montreal, Canada, 6/3/12.

Detecting visual text. / Dodge, Jesse; Goyal, Amit; Han, Xufeng et al.
Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics (ACL), 2012. p. 762-772 (NAACL HLT 2012 - 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Detecting visual text

AU - Dodge, Jesse

AU - Goyal, Amit

AU - Han, Xufeng

AU - Mensch, Alyssa

AU - Mitchell, Margaret

AU - Stratos, Karl

AU - Yamaguchi, Kota

AU - Choi, Yejin

AU - Daumé, Hal

AU - Berg, Alexander C.

AU - Berg, Tamara L.

PY - 2012

Y1 - 2012

N2 - When people describe a scene, they often include information that is not visually apparent; sometimes based on background knowledge, sometimes to tell a story. We aim to separate visual text - descriptions of what is being seen - from non-visual text in natural images and their descriptions. To do so, we first concretely define what it means to be visual, annotate visual text and then develop algorithms to automatically classify noun phrases as visual or non-visual. We find that using text alone, we are able to achieve high accuracies at this task, and that incorporating features derived from computer vision algorithms improves performance. Finally, we show that we can reliably mine visual nouns and adjectives from large corpora and that we can use these effectively in the classification task.

AB - When people describe a scene, they often include information that is not visually apparent; sometimes based on background knowledge, sometimes to tell a story. We aim to separate visual text - descriptions of what is being seen - from non-visual text in natural images and their descriptions. To do so, we first concretely define what it means to be visual, annotate visual text and then develop algorithms to automatically classify noun phrases as visual or non-visual. We find that using text alone, we are able to achieve high accuracies at this task, and that incorporating features derived from computer vision algorithms improves performance. Finally, we show that we can reliably mine visual nouns and adjectives from large corpora and that we can use these effectively in the classification task.

UR - https://www.scopus.com/pages/publications/84901455535

UR - https://www.scopus.com/pages/publications/84901455535#tab=citedBy

M3 - Conference contribution

T3 - NAACL HLT 2012 - 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference

SP - 762

EP - 772

BT - Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics

PB - Association for Computational Linguistics (ACL)

T2 - 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2012

Y2 - 3 June 2012 through 8 June 2012

ER -

Dodge J, Goyal A, Han X, Mensch A, Mitchell M, Stratos K et al. Detecting visual text. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics (ACL). 2012. p. 762-772. (NAACL HLT 2012 - 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference).

Detecting visual text

Abstract

Publication series

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this