Interpretable Measures of Conceptual Similarity by Complexity-Constrained Descriptive Auto-Encoding
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 11062-11071
Abstract
Quantifying the degree of similarity between images is a key copyright issue for image-based machine learning. In legal doctrine however, determining the degree of similarity between works requires subjective analysis, and fact-finders (judges and juries) can demonstrate considerable variabil-ity in these subjective judgement calls. Images that are structurally similar can be deemed dissimilar, whereas images of completely different scenes can be deemed similar enough to support a claim of copying. We seek to define and compute a notion of ‘conceptual similarity’ among images that captures high-level relations even among images that do not share repeated elements or visually similar components. The idea is to use a base multi-modal model to gen-erate 'explanations' (captions) of visual data at increasing levels of complexity. Then, similarity can be measured by the length of the caption needed to discriminate between the two images: Two highly dissimilar images can be dis-criminated early in their description, whereas conceptually dissimilar ones will need more detail to be distinguished. We operationalize this definition and show that it correlates with subjective (averaged human evaluation) assessment, and beats existing baselines on both image-to-image and text-to-text similarity benchmarks. Beyond just providing a number, our method also offers interpretability by pointing to the specific level of granularity of the description where the source data are differentiated.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- An Unsupervised Sentence Embedding Method by Mutual Information Maximization2020
- Semantic Bottleneck for Computer Vision TasksLecture Notes in Computer Science, 2019
- Sentencia de la corte de distrito de Estados Unidos, Distrito Sur, Nueva York, del 24 de junio de 1987: Steinberg V. Columbia Pictures Industries, Inc., 663 f. SUPP. 706 (S.D.N.Y. 1987)Revista La Propiedad Inmaterial, 2018
- Information Dropout: Learning Optimal Representations Through Noisy ComputationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- An Introduction to Kolmogorov Complexity and Its Applications2008
- The Communication Complexity of Correlation2007
- Kolmogorov's Structure Functions and Model SelectionIEEE Transactions on Information Theory, 2004
- Elements of Information Theory2001
- Algorithmic statisticsIEEE Transactions on Information Theory, 2001
- Knowing and guessingNuclear Physics A, 1971