Image–text coherence and its implications for multimodal AI

Malihe Alikhani, Baber Khalid, Matthew Stone

Research output: Contribution to journalArticlepeer-review


Human communication often combines imagery and text into integrated presentations, especially online. In this paper, we show how image–text coherence relations can be used to model the pragmatics of image–text presentations in AI systems. In contrast to alternative frameworks that characterize image–text presentations in terms of the priority, relevance, or overlap of information across modalities, coherence theory postulates that each unit of a discourse stands in specific pragmatic relations to other parts of the discourse, with each relation involving its own information goals and inferential connections. Text accompanying an image may, for example, characterize what's visible in the image, explain how the image was obtained, offer the author's appraisal of or reaction to the depicted situation, and so forth. The advantage of coherence theory is that it provides a simple, robust, and effective abstraction of communicative goals for practical applications. To argue this, we review case studies describing coherence in image–text data sets, predicting coherence from few-shot annotations, and coherence models of image–text tasks such as caption generation and caption evaluation.

Original languageEnglish (US)
Article number1048874
JournalFrontiers in Artificial Intelligence
StatePublished - 2023
Externally publishedYes

ASJC Scopus subject areas

  • Artificial Intelligence


  • coherence
  • discourse
  • evaluation
  • machine learning
  • multimodality


Dive into the research topics of 'Image–text coherence and its implications for multimodal AI'. Together they form a unique fingerprint.

Cite this