COSMic: A Coherence-Aware Generation Metric for Image Descriptions

TY - GEN

T1 - COSMic

T2 - 2021 Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021

AU - Inan, Mert

AU - Sharma, Piyush

AU - Khalid, Baber

AU - Soricut, Radu

AU - Stone, Matthew

AU - Alikhani, Malihe

N1 - Funding Information: The authors affiliated with Rutgers University were partly supported by NSF Award CCF-19349243. Thanks to Pitt Cyber for supporting this project and the authors from the University of Pittsburgh. We also acknowledge the Center for Research Computing at the University of Pittsburgh for providing the required computational resources for carrying out experiments at the University of Pittsburgh. Publisher Copyright: © 2021 Association for Computational Linguistics.

PY - 2021

Y1 - 2021

N2 - Developers of text generation models rely on automated evaluation metrics as a stand-in for slow and expensive manual evaluations. However, image captioning metrics have struggled to give accurate learned estimates of the semantic and pragmatic success of output text. We address this weakness by introducing the first discourse-aware learned generation metric for evaluating image descriptions. Our approach is inspired by computational theories of discourse for capturing information goals using coherence. We present a dataset of image-description pairs annotated with coherence relations. We then train a coherence-aware metric on a subset of the Conceptual Captions dataset and measure its effectiveness - its ability to predict human ratings of output captions - on a test set composed of out-of-domain images. We demonstrate a higher Kendall Correlation Coefficient for our proposed metric with the human judgments for the results of a number of stateof-the-art coherence-aware caption generation models when compared to several other metrics including recently proposed learned metrics such as BLEURT and BERTScore.

AB - Developers of text generation models rely on automated evaluation metrics as a stand-in for slow and expensive manual evaluations. However, image captioning metrics have struggled to give accurate learned estimates of the semantic and pragmatic success of output text. We address this weakness by introducing the first discourse-aware learned generation metric for evaluating image descriptions. Our approach is inspired by computational theories of discourse for capturing information goals using coherence. We present a dataset of image-description pairs annotated with coherence relations. We then train a coherence-aware metric on a subset of the Conceptual Captions dataset and measure its effectiveness - its ability to predict human ratings of output captions - on a test set composed of out-of-domain images. We demonstrate a higher Kendall Correlation Coefficient for our proposed metric with the human judgments for the results of a number of stateof-the-art coherence-aware caption generation models when compared to several other metrics including recently proposed learned metrics such as BLEURT and BERTScore.

UR - http://www.scopus.com/inward/record.url?scp=85129187749&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85129187749&partnerID=8YFLogxK

M3 - Conference contribution

T3 - Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021

SP - 3419

EP - 3430

BT - Findings of the Association for Computational Linguistics, Findings of ACL

A2 - Moens, Marie-Francine

A2 - Huang, Xuanjing

A2 - Specia, Lucia

A2 - Yih, Scott Wen-Tau

PB - Association for Computational Linguistics (ACL)

Y2 - 7 November 2021 through 11 November 2021

ER -