Jan Blaha

A honeybee colony is a complex and dynamic system that emerges out of the interactions of thousands of individuals within a seemingly chaotic and heterogeneous environment. At the figurative core of this system is the honeybee queen, responsible for the growth and reproduction of the eusocial superorganism. In this study, we examine the interaction between the queen and her surrounding environment by analyzing her movement patterns using mathematical models and computational approaches. We employed a visual tracking system to observe three queens of Apis mellifera within their colonies over a three-week period and analyzed sets of quality tracklets to provide observational evidence regarding the queens’ motion-related decision-making. Contrary to expectations, we found that the queen’s short-term motion characteristics—such as speed and turning—were remarkably invariant across distinct hive regions, suggesting a lack of direct environmental modulation at short timescales. Yet, long-term patterns showed structured and strategic behavior. Inter-stop distances followed a power-law distribution, and queens repeatedly revisited specific spatial zones over multi-day timescales. These results indicate a dual-scale movement strategy that is not captured by standard random walk models, highlighting internal state or memory-based navigation. Our findings suggest that queen movement is shaped by temporally layered processes that may support brood nest stability, efficient egg-laying, and colony cohesion.

Digital and mechatronic methods, paired with artificial intelligence and machine learning, are transformative technologies in behavioral science and biology. The central element of the most important pollinator species—honey bees—is the colony’s queen. Because honey bee self-regulation is complex and studying queens in their natural colony context is difficult, the behavioral strategies of these organisms have not been widely studied. We created an autonomous robotic observation and behavioral analysis system aimed at continuous observation of the queen and her interactions with worker bees and comb cells, generating behavioral datasets of exceptional length and quality. Key behavioral metrics of the queen and her social embedding within the colony were gathered using our robotic system. Data were collected continuously for 24 hours a day over a period of 30 days, demonstrating our system’s capability to extract key behavioral metrics at microscopic, mesoscopic, and macroscopic system levels. Additionally, interactions among the queen, worker bees, and brood were observed and quantified. Long-term continuous observations performed by the robot yielded large amounts of high-definition video data that are beyond the observation capabilities of humans or stationary cameras. Our robotic system can enable a deeper understanding of the innermost mechanisms of honey bees’ swarm-intelligent self-regulation. Moreover, it offers the possibility to study other social insect colonies, biocoenoses, and ecosystems in an automated manner. Social insects are keystone species in all terrestrial ecosystems; thus, developing a better understanding of their behaviors will be invaluable for the protection and even the restoration of our fragile ecosystems globally.

Nowadays, robots can operate in environments which are not tailored for them. This allows their deployments in changing and human-populated environments, which recent advances in machine learning methods enabled. The efficiency of these methods is largely determined by the quality of their training data. An up-to-date and well-balanced training dataset is paramount for achieving robust robot operation. To achieve long-term operation, the robot has to deal with perpetual environmental changes, forcing it to keep its models up-to-date. We present an exploration method allowing a mobile robot to gather high-quality data to update its models both while performing its duties and when idle, maximizing effectivity. The robot evaluates the quality of the data gathered in the past and based on that, it creates preferences which influence how often these locations are visited. This exploration method was integrated with a self-supervised visual teach-and-repeat pipeline. We show the precision and robustness of visual-based navigation to improve when using machine-learned models trained by our exploration method. Our research resulted in a robotic navigation system that can not only annotate its training data but also ensure that its training dataset is balanced and up-to-date.

The performance of deep learning-based detection methods has made them an attractive option for robotic perception. However, their training typically requires large volumes of data containing all the various situations the robots may potentially encounter during their routine operation. Thus, the workforce required for data collection and annotation is a significant bottleneck when deploying robots in the real world. This applies especially to outdoor deployments, where robots have to face various adverse weather conditions. We present a method that allows an independent car tansporter to train its neural networks for vehicle detection without human supervision or annotation. We provide the robot with a hand-coded algorithm for detecting cars in LiDAR scans in favourable weather conditions and complement this algorithm with a tracking method and a weather simulator. As the robot traverses its environment, it can collect data samples, which can be subsequently processed into training samples for the neural networks. As the tracking method is applied offline, it can exploit the detections made both before the currently processed scan and any subsequent future detections of the current scene, meaning the quality of annotations is in excess of those of the raw detections. Along with the acquisition of the labels, the weather simulator is able to alter the raw sensory data, which are then fed into the neural network together with the labels. We show how this pipeline, being run in an offline fashion, can exploit off-the-shelf weather simulation for the auto-labelling training scheme in a simulator-in-the-loop manner. We show how such a framework produces an effective detector and how the weather simulator-in-the-loop is beneficial for the robustness of the detector. Thus, our automatic data annotation pipeline significantly reduces not only the data annotation but also the data collection effort. This allows the integration of deep learning algorithms into existing robotic systems without the need for tedious data annotation and collection in all possible situations. Moreover, the method provides annotated datasets that can be used to develop other methods. To promote the reproducibility of our research, we provide our datasets, codes and models online.

Visual teach and repeat navigation (VT&R) is popular in robotics thanks to its simplicity and versatility. It enables mobile robots equipped with a camera to traverse learned paths without the need to create globally consistent metric maps. Although teach and repeat frameworks have been reported to be relatively robust to changing environments, they still struggle with day-to-night and seasonal changes. This paper aims to find the horizontal displacement between prerecorded and currently perceived images required to steer a robot towards the previously traversed path. We employ a fully convolutional neural network to obtain dense representations of the images that are robust to changes in the environment and variations in illumination. The proposed model achieves state-of-the-art performance on multiple datasets with seasonal and day/night variations. In addition, our experiments show that it is possible to use the model to generate additional training examples that can be used to further improve the original model’s robustness. We also conducted a real-world experiment on a mobile robot to demonstrate the suitability of our method for VT&R.

The performance of deep neural networks and the low costs of computational hardware has made computer vision a popular choice in many robotic systems. An attractive feature of deep-learned methods is their ability to cope with appearance changes caused by day–night cycles and seasonal variations. However, deep learning of neural networks typically relies on large numbers of hand-annotated images, which requires significant effort for data collection and annotation. We present a method that allows autonomous, self-supervised training of a neural network in visual teach-and-repeat (VT&R) tasks, where a mobile robot has to traverse a previously taught path repeatedly. Our method is based on a fusion of two image registration schemes: one based on a Siamese neural network and another on point-feature matching. As the robot traverses the taught paths, it uses the results of feature-based matching to train the neural network, which, in turn, provides coarse registration estimates to the feature matcher. We show that as the neural network gets trained, the accuracy and robustness of the navigation increases, making the robot capable of dealing with significant changes in the environment. This method can significantly reduce the data annotation efforts when designing new robotic systems or introducing robots into new environments. Moreover, the method provides annotated datasets that can be deployed in other navigation systems. To promote the reproducibility of the research presented herein, we provide our datasets, codes and trained models online.