Position: Benchmarks for Vision–Language Models in Urban Perception Should Be Reliability-Aware and Negotiated

Year: 2026

ICML · Peer-reviewed position paper

Author: Rashid Mushkani

Original heatmap of model agreement with human judgments across urban perception dimensions. — Figure 5 from the paper: agreement by dimension and model. Source

Mushkani, R. (2026). Position: Benchmarks for Vision–Language Models in Urban Perception Should Be Reliability-Aware and Negotiated. Proceedings of the 43rd International Conference on Machine Learning, PMLR 306, 171558–171571.

Using Montréal street scenes evaluated by community participants and vision–language models, this paper argues that disagreement and abstention carry information. Benchmark categories and scoring rules should remain open to negotiation when their results inform urban governance.

Read the proceedings entry.