City Research Online

Abstract

The visualization community has invested substantially in deploying Vision-Language Models (VLMs) for visual analytics. However, the complementary question of how exactly do these models reason about visualizations, and how do we know when they are doing so correctly has received comparatively little rigorous attention. In this position paper, we argue that the interpretability methods currently applied to VLMs in the visualization domain inherit a cascade of structural mismatches. Methods ported from computer vision assume a crisp notion of objectness, spatial locality, and to a large extent, context-independence that natural images possess but data visualizations do not. Human gaze studies, often proposed as ground truth to compare and contrast interpretability methods, carry their own fragilities rooted in the reproducibility crisis in perception research. Finally, and perhaps more importantly, VLMs themselves are not vision models as their visual inputs merely serve as adaptors into a language model’s (linguistic) representational space, fundamentally altering the notion of what “visual attention” means. To probe these questions, we introduce a dual causal-chain account of visualization comprehension, distinguishing the designer’s intended reading path from the reader’s actual comprehension trace, we propose a set of necessary conditions for valid interpretability evaluation in this domain. We posit that the visualization community is uniquely positioned to develop novel interpretability methods which can significantly impact both theoretical and empirical advances in the broad field of visual analytics.

Publication Type:	Conference or Workshop Item (Paper)
Additional Information:	© 2026 IEEE. This accepted manuscript is made available under the terms of the Creative Commons Attribution License (CC-BY), which permits unrestricted use, distribution and reproduction in any medium, provided the original work is properly cited.
Publisher Keywords:	Vision-language models, interpretability, saliency, referential chain, shortcut learning
Subjects:	H Social Sciences > HN Social history and conditions. Social problems. Social reform Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Departments:	School of Science & Technology School of Science & Technology > Department of Computer Science School of Science & Technology > Department of Computer Science > giCentre
SWORD Depositor:	Symplectic Administrator

Export

Downloads