Anton Hristov

Finding a template location in a query image is a fundamental problem in many computer vision applications, such as localization of known objects, image registration, image matching, and object tracking. Currently available methods fail when insufficient training data are available or big variations in the textures, different modalities, and weak visual features exist in the images, leading to limited applications on real-world tasks. We introduce Self-Supervised Foundation Model for Template Matching (Self-TM), a novel end-to-end approach to self-supervised learning template matching. The idea behind Self-TM is to learn hierarchical features incorporating localization properties from images without any annotations. As going deeper in the convolutional neural network (CNN) layers, their filters begin to react to more complex structures and their receptive fields increase. This leads to loss of localization information in contrast to the early layers. The hierarchical propagation of the last layers back to the first layer results in precise template localization. Due to its zero-shot generalization capabilities on tasks such as image retrieval, dense template matching, and sparse image matching, our pre-trained model can be classified as a foundation one.

Finding a template location in query image is a fundamental problem in many computer vision applications, such as localization of known objects, image registration, image matching and object tracking. Currently available methods fail when not enough training data is available or big variations in the textures, different modalities and weak visual features exist in the images, leading to limited applications on real-world tasks. We introduce Self-Supervised Foundation Model for Template Matching (Self-TM), a novel end-to-end approach for self-supervised learning template matching. The idea behind Self-TM is to learn hierarchical features incorporating localization properties from images without any annotations. As going deeper in the convolutional neural network (CNN) layers, their filters begin to react to more complex structures and their receptive fields increase. This leads to loss of localization information in contrast to the early layers. The hierarchical propagation of the last layers activations back to the first layer results in precise template localization. Due to its zero-shot generalization capabilities on tasks such as image retrieval, dense template matching and sparse image matching our pre-trained model can be classified as a foundation one.

With the development of Convolutional neural networks (CNN), a myth was formed in recent years that only the last convolutional layers, including the fully connected layers, provide the most essential visual features (visual concepts) of the images at the input. In this paper, we experimentally disprove this prejudice through demonstrations showing that all the convolution layers of a CNN contain essential information that influences the classification. In connection with the experiments conducted, we look at each filter from different convolution layers as an independent detector of a particular visual structure (visual concept), and on this basis we introduce a new notion called the "vector of visual concepts". This concept summarizes the information, what visual constellation detectors are activated by an input image, and how strong they are activated. The classification error obtained by the vector of visual concepts differs by only 0.12% from the results of the last (softmax) layer of the network on the training data domain, which undoubtedly suggests the information saturation of all layers in the network. Additionally, we show by experiments that in input data from a domain other than the training one, the features of the initial and intermediate convolution layers provide a more accurate classification than those of the last convolution layers.