Token-Based Affordance Grounding with Large Vision-Language Models

Abstract

Affordance grounding aims to localize image regions that support a specific action, serving as a core capability for physical intelligence and embodied perception. Previous studies have primarily relied on weakly supervised learning with action labels from exocentric images. However, these methods often struggle with visually ambiguous exocentric images containing co-occurring actions; moreover, they fail to distinguish semantically similar actions because existing methods typically rely on brief action phrases that lack rich semantic details for action-specific localization. Although large vision-language models (LVLMs) encode rich action semantics and their action-conditioned textual outputs implicitly contain spatial cues, they do not directly provide action-specific spatial localization. To address these problems, we propose TokAG, a zero-shot affordance grounding framework that exploits the token-level semantic-spatial signals in LVLMs to localize action-relevant regions without external supervision. We observe that attention maps associated with different LVLM output tokens vary significantly, with many attending to irrelevant regions such as the background. Thus, we introduce a spatial-aware token-selection mechanism to systematically evaluate each output token and select the one whose attention maps exhibit dominant activation over the target object, instead of relying on arbitrary attention maps. By extracting these object-focused attention maps, we transform the LVLM’s implicit semantic signals into zero-shot affordance heatmaps. Our zero-shot framework consistently outperforms prior weakly supervised approaches across multiple benchmarks, improving NSS by 10.7% on the unseen split of AGD20K and by 29.7% on HICO-IIF. The code and models will be made publicly available.

Pipeline

Overview of our proposed pipeline.

Given an egocentric image and an action text prompt, TokAG extracts output tokens \(\{t_1, t_2, \ldots, t_N\}\) from the language layers of a frozen LVLM. For each token \(t_i\), token-to-image attention maps are aggregated across all selected layers \(\mathcal{L}\) and attention heads \(\mathcal{H}\): \[ A_i = \frac{1}{|\mathcal{L}||\mathcal{H}|} \sum_{l \in \mathcal{L}} \sum_{h \in \mathcal{H}} A_{i}^{l,h}. \] Using object masks \(M_k\) generated by CLIPSeg, TokAG computes an object-region attention score for each token by measuring how strongly its attention falls inside the target object regions: \[ s_i = \sum_k \frac{\sum_{p} A_i(p) M_k(p)}{\sum_{p} M_k(p)}. \] The token with the highest score is selected, \(t^* = \arg\max_i s_i\), and its object-region attention heatmap becomes the final affordance grounding prediction.

Qualitative and Quantitative Results

Qualitative Results

Here are some qualitative results.

Quantitative Results

Here are the experiment results in zero-shot affordance grounding.

BibTeX

@inproceedings{lee2026tokag,
  title     = {Token-Based Affordance Grounding with Large Vision-Language Models},
  author    = {Lee, Seung il and Lei, Qinqian and Xu, Daguang and Yang, Dong and Tan, Robby T. and Chen, Yixin and Wang, Bo},
  booktitle = {In Proceedings of the European Conference on Computer Vision (ECCV)},
  year      = {2026}
}