Integrating Prior Knowledge in Contrastive Learning with Kernel
Authors:
- Benoit DufumierBDBenoit Dufumier
- Carlo Alberto BarbanoCBCarlo Alberto Barbano
- Robin LouisetRLRobin Louiset
- Edouard DuchesnayEDEdouard Duchesnay
- Pietro GoriPGPietro Gori
Preprint
Abstract
Data augmentation is a crucial component in unsupervised contrastive learning (CL). It determines how positive samples are defined and, ultimately, the quality of the learned representation. In this work, we open the door to new perspectives for CL by integrating prior knowledge, given either by generative models -- viewed as prior representations -- or weak attributes in the positive and negative sampling. To this end, we use kernel theory to propose a novel loss, called decoupled uniformity, that i) allows the integration of prior knowledge and ii) removes the negative-positive coupling in the original InfoNCE loss. We draw a connection between contrastive learning and conditional mean embedding theory to derive tight bounds on the downstream classification loss. In an unsupervised setting, we empirically demonstrate that CL benefits from generative models to improve its representation both on natural and medical images. In a weakly supervised scenario, our framework outperforms other unconditional and conditional CL approaches.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Decoupled Contrastive Learning2022
- Momentum Contrast for Unsupervised Visual Representation Learning2020
- Lithium prevents grey matter atrophy in patients with bipolar disorder: an international multicenter studyPsychological Medicine, 2020
- A Rigorous Theory of Conditional Mean EmbeddingsSIAM Journal on Mathematics of Data Science, 2020
- Discrete Energy on Rectifiable Sets2019
- Cortical abnormalities in bipolar disorder: an MRI analysis of 6503 individuals from the ENIGMA Bipolar Disorder Working GroupMolecular Psychiatry, 2017
- Kernel Embeddings of Conditional Distributions: A Unified Kernel Framework for Nonparametric Inference in Graphical ModelsIEEE Signal Processing Magazine, 2013
- Self-organizing neural network that discovers surfaces in random-dot stereogramsNature, 1992