TY - GEN
T1 - Egocentric Daily Video Question Answering with Token-Efficient Storyboard Retrieval
AU - Ding, Zihao
AU - Chang, Tun Yuan
AU - Hsu, Cheng Hsin
AU - Liu, Yao
N1 - Publisher Copyright: © 2026 Copyright held by the owner/author(s).
PY - 2026/4/8
Y1 - 2026/4/8
N2 - Always-on AI glasses enable continuous egocentric video recording that can support daily video question answering (QA). However, processing hours of personal footage poses significant challenges under the compute and token constraints of modern vision-language models (VLMs). In this paper, we present a token-Aware retrieve-Then-reason architecture for egocentric QA that indexes visual logs and retrieves a small set of relevant evidence for VLM inference. Using a two-Tier evaluation, we first establish an intrinsic reasoning baseline for single video QA, and then measure the end-To-end behavior when retrieval must search a personal video archive. Across two datasets, we identify a critical "modality gap": unlike narrative web videos that rely on audio, egocentric QA depends heavily on visual evidence. Furthermore, we find that storyboards, by packing multiple frames in a chronologically arranged grid, provide a strong token-Accuracy trade-off, achieving similar or better accuracy than frame-based baselines with up to 85% fewer prompt tokens. Tile-level retrieval further improves accuracy at a higher token cost. Our results highlight the importance of evidence packaging and retrieval policies for practical, resource-constrained daily QA.
AB - Always-on AI glasses enable continuous egocentric video recording that can support daily video question answering (QA). However, processing hours of personal footage poses significant challenges under the compute and token constraints of modern vision-language models (VLMs). In this paper, we present a token-Aware retrieve-Then-reason architecture for egocentric QA that indexes visual logs and retrieves a small set of relevant evidence for VLM inference. Using a two-Tier evaluation, we first establish an intrinsic reasoning baseline for single video QA, and then measure the end-To-end behavior when retrieval must search a personal video archive. Across two datasets, we identify a critical "modality gap": unlike narrative web videos that rely on audio, egocentric QA depends heavily on visual evidence. Furthermore, we find that storyboards, by packing multiple frames in a chronologically arranged grid, provide a strong token-Accuracy trade-off, achieving similar or better accuracy than frame-based baselines with up to 85% fewer prompt tokens. Tile-level retrieval further improves accuracy at a higher token cost. Our results highlight the importance of evidence packaging and retrieval policies for practical, resource-constrained daily QA.
KW - Egocentric video
KW - Multi-video retrieval
KW - Storyboard representations
KW - Token-efficient inference
KW - Vision-Language Models
UR - https://www.scopus.com/pages/publications/105036514254
UR - https://www.scopus.com/pages/publications/105036514254#tab=citedBy
U2 - 10.1145/3798065.3798076
DO - 10.1145/3798065.3798076
M3 - Conference contribution
T3 - NOSSDAV 2026 - Proceedings of the 36th Workshop on Network and Operating System Support for Digital Audio and Video
SP - 71
EP - 77
BT - NOSSDAV 2026 - Proceedings of the 36th Workshop on Network and Operating System Support for Digital Audio and Video
PB - Association for Computing Machinery, Inc
T2 - 36th Workshop on Network and Operating System Support for Digital Audio and Video, NOSSDAV 2026
Y2 - 4 April 2026 through 8 April 2026
ER -