Skip to main navigation Skip to search Skip to main content

Egocentric Daily Video Question Answering with Token-Efficient Storyboard Retrieval

  • Zihao Ding
  • , Tun Yuan Chang
  • , Cheng Hsin Hsu
  • , Yao Liu

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Always-on AI glasses enable continuous egocentric video recording that can support daily video question answering (QA). However, processing hours of personal footage poses significant challenges under the compute and token constraints of modern vision-language models (VLMs). In this paper, we present a token-Aware retrieve-Then-reason architecture for egocentric QA that indexes visual logs and retrieves a small set of relevant evidence for VLM inference. Using a two-Tier evaluation, we first establish an intrinsic reasoning baseline for single video QA, and then measure the end-To-end behavior when retrieval must search a personal video archive. Across two datasets, we identify a critical "modality gap": unlike narrative web videos that rely on audio, egocentric QA depends heavily on visual evidence. Furthermore, we find that storyboards, by packing multiple frames in a chronologically arranged grid, provide a strong token-Accuracy trade-off, achieving similar or better accuracy than frame-based baselines with up to 85% fewer prompt tokens. Tile-level retrieval further improves accuracy at a higher token cost. Our results highlight the importance of evidence packaging and retrieval policies for practical, resource-constrained daily QA.

Original languageEnglish (US)
Title of host publicationNOSSDAV 2026 - Proceedings of the 36th Workshop on Network and Operating System Support for Digital Audio and Video
PublisherAssociation for Computing Machinery, Inc
Pages71-77
Number of pages7
ISBN (Electronic)9798400725340
DOIs
StatePublished - Apr 8 2026
Event36th Workshop on Network and Operating System Support for Digital Audio and Video, NOSSDAV 2026 - Hong Kong, Hong Kong
Duration: Apr 4 2026 → Apr 8 2026

Publication series

NameNOSSDAV 2026 - Proceedings of the 36th Workshop on Network and Operating System Support for Digital Audio and Video

Conference

Conference36th Workshop on Network and Operating System Support for Digital Audio and Video, NOSSDAV 2026
Country/TerritoryHong Kong
CityHong Kong
Period4/4/26 → 4/8/26

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Human-Computer Interaction
  • Software

Keywords

  • Egocentric video
  • Multi-video retrieval
  • Storyboard representations
  • Token-efficient inference
  • Vision-Language Models

Fingerprint

Dive into the research topics of 'Egocentric Daily Video Question Answering with Token-Efficient Storyboard Retrieval'. Together they form a unique fingerprint.

Cite this