ArXiv

The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric

2026-07-20 · 17:59 UTC ·Sheng-Yu Wang, Yotam Nitzan, Aaron Hertzmann... ·1 min read
Authors
Sheng-Yu Wang, Yotam Nitzan, Aaron Hertzmann...
Categories
cs.CV, cs.LG
arXiv
https://arxiv.org/abs/2607.18237v1
PDF
https://arxiv.org/pdf/2607.18237v1

Brief

The paper introduces TPIPS, a text-prompted image perceptual similarity metric designed to capture multiple, context-dependent senses of visual similarity. To train and evaluate TPIPS, the authors collected a large-scale dataset of human judgments on image triplets annotated with free-form semantic aspects, demonstrated that existing VLMs lag behind human consensus, and showed TPIPS better matches human perception and generalizes while enabling text-guided retrieval, compositional search, and finer evaluation of generative models.

Why it matters

Created a large-scale dataset of human similarity judgments on image triplets, where each triplet is annotated across multiple, free-form semantic aspects; benchmarking 'frontier' vision-language models revealed a considerable performance gap versus human consensus.

Key details

  • Fine-tuned a vision-language model to produce TPIPS (Text-Prompted Image Perceptual Similarity), a prompt-conditioned metric that aligns more closely with human perception and generalizes beyond the training distribution.
  • TPIPS enables text-guided retrieval, compositional search, and fine-grained evaluation of generative models; authors (Wang, Nitzan, Hertzmann, Zhu, Shechtman, Efros, Zhang) released code/data/models at https://peterwang512.github.io/TPIPS and posted the paper to arXiv on 2026-07-20.
Source evidence

Abstract

Human visual similarity judgments are context-dependent. For example, two images may be similar in shape but distinct in color. Existing perceptual similarity metrics, however, collapse these nuances into a single scalar value, offering no mechanism to condition on specific aspects. To bridge this gap, we introduce a large-scale dataset of human similarity judgments over image triplets, where each triplet is annotated across multiple, free-form semantic aspects of similarity. Benchmarking a broad range of frontier vision-language models (VLMs) reveals a considerable performance gap compared to human annotators' consensus. Leveraging our data, we fine-tune a VLM to produce our Text-Prompted Image Perceptual Similarity (TPIPS) metric, capturing multiple senses of visual similarity depending on the specified text prompt. We demonstrate that TPIPS aligns more closely with human perception and generalizes reliably beyond the training distribution. Finally, we show that TPIPS unlocks new capabilities in text-guided retrieval, compositional search, and the fine-grained evaluation of generative models. Our code, data, and trained models are at https://peterwang512.github.io/TPIPS

Comment: Project Webpage: https://peterwang512.github.io/TPIPS