David Suter

Robust model fitting aims to estimate model parameters from data contaminated by noise and outliers in computer vision. Traditional RANSAC-based methods suffer from model hypothesis ambiguity and inefficiency due to the problems of neglecting data preference distributions and employing iterative hypothesis sampling. Learning-based methods enhance traditional methods through deep features. However, their reliance on static coordinate representations inherently lacks motion cues, hindering the analysis of complex dynamic scenes. Furthermore, the local receptive fields of CNNs inadequately capture global context. To address these issues, we propose MPCFormer, a motion-aware Transformer method via multi-channel preference filtering and multi-scale consensus smoothing for robust model fitting. It reformulates robust model fitting as a joint optimization of point classification and model estimation by integrating correspondence learning and embedding spatiotemporal motion cues, eliminating iterative hypothesis sampling. Specifically, we design a motion preference filter to explore multi-channel motion information by residual-connected Transformer layers. It explicitly encodes data preference distributions for models via multi-head preference attention, generating confidence scores to adaptively suppress outlier interference and enhance model robustness. Additionally, we present a pyramid consensus smoother with multi-scale Transformer encoding. It hierarchically captures local-to-global motion consistency through a sparse feature pyramid, effectively resolving motion ambiguity from spatial discontinuities. This module enables precise inlier identification and reliable model estimation through multi-head consensus attention. Extensive experiments demonstrate that MPCFormer outperforms state-of-the-art baselines by 4.68% mAP@5°, 1.89% AUC@3 pixel, and 1.52% F-score, even at extreme outlier ratios (up to 95%).

Panoramic images play an important role in indoor robotics, offering a comprehensive spatial understanding of the environment. For tasks such as visual room rearrangement, where agents restore objects to their original positions or states, panoramic views provide a broader understanding of all changes. Existing 2D datasets for scene change understanding are limited to single-view images, which fail to capture the full spatial context, object relationships, and partially visible or obscured objects, making them less suitable for embodied AI applications. To address this, we introduce PanoSCU (Panoramic Scene Change Understanding), a dataset specifically designed to enhance the visual object rearrangement task. PanoSCU supports eight research tasks using single-viewand panoramic inputs: single-viewand panoramic detection, single-view and panoramic segmentation, single-view and panoramic change understanding, embodied object tracking, and change reversal. Our dataset contains 5,300 simulator-generated panoramas, encompassing 48 frequently-seen indoor object classes. We also introduce PanoStitch, a training-free stitching method for automatic panoramic data collection in embodied AI simulators. We evaluate state-of-the-art methods on panoramic segmentation and change understanding tasks. There is a gap in existing methods, as they are not designed for panoramic inputs and struggle with varying ratios and sizes, resulting from the unique challenges of visual object rearrangement. Our findings reveal these limitations and highlight PanoSCU’s challenging nature, emphasizing the need for further research to bridge this gap for panoramic indoor scene understanding. The dataset and the baseline methods for panoramic segmentation task are available publicly a.

Vertebral fracture assessment (VFA) images from bone density machines enable the automated machine learning assessment of abdominal aortic calcification (ML-AAC), a marker of cardiovascular disease (CVD) risk. The objective of this study was to describe the risk of a major adverse cardiovascular event (MACE, from linked health records) in patients attending routine bone mineral density (BMD) testing and meeting specific criteria based on age, BMD, height loss, or glucocorticoid use have a VFA in the Manitoba BMD Registry. The cohort included 10 250 individuals (mean age 75.5 yr, 94% women without CVD) with VFA (February 2010 to March 2017). ML-AAC24 scores were categorized (low <2; moderate 2–<6; high ≥6). Over follow-up (mean 3.9 yr), 1265 people (12.3%) experienced a MACE. Among those with low, moderate, and high ML-AAC24, MACE rates per 1000 person-years were 18.4 (95% CI 16.4-20.5), 34.1 (95% CI 30.9-37.4), and 55.6 (95% CI 50.8-60.1), respectively. A similar gradient was observed after stratifying by age and sex. Incidence rate ratios (IRRs) for low vs moderate and high groups were 1.9 (95% CI 1.6-2.2) and 3.0 (95% CI 2.6-3.5), respectively. In those most likely to benefit from pharmaceutical intervention (<80 yr, not on statins), MACE rates among those with low, moderate, and high ML-AAC24 were 13.5 (95% CI 11.5-15.8), 26.0 (95% CI 22.1-30.3) and 44.1 (95% CI 37.0-52.0). Corresponding IRRs for low vs moderate 1.9 (95% CI 1.5-2.4) and high ML-AAC24 was 3.3 (95% CI 2.6-4.1]), respectively. In routine osteoporosis screening, individuals with moderate and high ML-AAC24 had substantially greater MACE rates compared to those with low ML-AAC24. Consequently, AAC detection during osteoporosis screening (especially in women) may guide intensification of preventative cardiovascular strategies.

Embodied AI aims to develop robots that can \textit{understand} and execute human language instructions, as well as communicate in natural languages. On this front, we study the task of generating highly detailed navigational instructions for the embodied robots to follow. Although recent studies have demonstrated significant leaps in the generation of step-by-step instructions from sequences of images, the generated instructions lack variety in terms of their referral to objects and landmarks. Existing speaker models learn strategies to evade the evaluation metrics and obtain higher scores even for low-quality sentences. In this work, we propose SAS (Spatially-Aware Speaker), an instruction generator or \textit{Speaker} model that utilises both structural and semantic knowledge of the environment to produce richer instructions. For training, we employ a reward learning method in an adversarial setting to avoid systematic bias introduced by language evaluation metrics. Empirically, our method outperforms existing instruction generation models, evaluated using standard metrics. Our code is available at \url{https://github.com/gmuraleekrishna/SAS}.

Whether simultaneous automated ascertainments of prevalent vertebral fracture (auto-PVFx) and abdominal aortic calcification (auto-AAC) on vertebral fracture assessment (VFA) lateral spine bone density (BMD) images jointly predict incident fractures in routine clinical practice is unclear. We estimated the independent associations of auto-PVFx and auto-AAC primarily with incident major osteoporotic and secondarily with incident hip and any clinical fractures in 11 013 individuals (mean [SD] age 75.8 [6.8] years, 93.3% female) who had a BMD test combined with VFA between March 2010 and December 2017. Auto-PVFx and auto-AAC were ascertained using convolutional neural networks (CNNs). Proportional hazards models were used to estimate the associations of auto-PVFx and auto-AAC with incident fractures over a mean (SD) follow-up of 3.7 (2.2) years, adjusted for each other and other risk factors. At baseline, 17% (n = 1881) had auto-PVFx and 27% (n = 2974) had a high level of auto-AAC (≥ 6 on scale of 0 to 24). Multivariable-adjusted hazard ratios (HR) for incident major osteoporotic fracture (95% CI) were 1.85 (1.59, 2.15) for those with compared with those without auto-PVFx, and 1.36 (1.14, 1.62) for those with high compared with low auto-AAC. The multivariable-adjusted HRs for incident hip fracture were 1.62 (95% CI, 1.26 to 2.07) for those with compared to those without auto-PVFx, and 1.55 (95% CI, 1.15 to 2.09) for those high auto-AAC compared with low auto-AAC. The 5-year cumulative incidence of major osteoporotic fracture was 7.1% in those with no auto-PVFx and low auto-AAC, 10.1% in those with no auto-PVFx and high auto-AAC, 13.4% in those with auto-PVFx and low auto-AAC, and 18.0% in those with auto-PVFx and high auto-AAC. While physician manual review of images in clinical practice will still be needed to confirm image quality and provide clinical context for interpretation, simultaneous automated ascertainment of auto-PVFx and auto-AAC can aid fracture risk assessment.

Muraleekrishna Gopinathan, Martin Masek, Jumana Abu-Khalaf, David Suter. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Abdominal aortic calcification (AAC), a recognized measure of advanced vascular disease, is associated with higher cardiovascular risk and poorer long-term prognosis. AAC can be assessed on dual-energy X-ray absorptiometry (DXA)-derived lateral spine images used for vertebral fracture assessment at the time of bone density screening using a validated 24-point scoring method (AAC-24). Previous studies have identified robust associations between AAC-24 score, incident falls, and fractures. However, a major limitation of manual AAC assessment is that it requires a trained expert. Hence, we have developed an automated machine-learning algorithm for assessing AAC-24 scores (ML-AAC24). In this prospective study, we evaluated the association between ML-AAC24 and long-term incident falls and fractures in 1023 community-dwelling older women (mean age, 75 ± 3 years) from the Perth Longitudinal Study of Ageing Women. Over 10 years of follow-up, 253 (24.7%) women experienced a clinical fracture identified via self-report every 4–6 months and verified by X-ray, and 169 (16.5%) women had a fracture hospitalization identified from linked hospital discharge data. Over 14.5 years, 393 (38.4%) women experienced an injurious fall requiring hospitalization identified from linked hospital discharge data. After adjusting for baseline fracture risk, women with moderate to extensive AAC (ML-AAC24 ≥ 2) had a greater risk of clinical fractures (hazard ratio [HR] 1.42; 95% confidence interval [CI], 1.10–1.85) and fall-related hospitalization (HR 1.35; 95% CI, 1.09–1.66), compared to those with low AAC (ML-AAC24 ≤ 1). Similar to manually assessed AAC-24, ML-AAC24 was not associated with fracture hospitalizations. The relative hazard estimates obtained using machine learning were similar to those using manually assessed AAC-24 scores. In conclusion, this novel automated method for assessing AAC, that can be easily and seamlessly captured at the time of bone density testing, has robust associations with long-term incident clinical fractures and injurious falls. However, the performance of the ML-AAC24 algorithm needs to be verified in independent cohorts. © 2023 The Authors. Journal of Bone and Mineral Research published by Wiley Periodicals LLC on behalf of American Society for Bone and Mineral Research (ASBMR).

Humans use their knowledge of common house layouts obtained from previous experiences to predict nearby rooms while navigating in new environments. This greatly helps them navigate previously unseen environments and locate their target room. To provide layout prior knowledge to navigational agents based on common human living spaces, we propose WIN (\textit{W}hat \textit{I}s \textit{N}ear), a commonsense learning model for Vision Language Navigation (VLN) tasks. VLN requires an agent to traverse indoor environments based on descriptive navigational instructions. Unlike existing layout learning works, WIN predicts the local neighborhood map based on prior knowledge of living spaces and current observation, operating on an imagined global map of the entire environment. The model infers neighborhood regions based on visual cues of current observations, navigational history, and layout common sense. We show that local-global planning based on locality knowledge and predicting the indoor layout allows the agent to efficiently select the appropriate action. Specifically, we devised a cross-modal transformer that utilizes this locality prior for decision-making in addition to visual inputs and instructions. Experimental results show that locality learning using WIN provides better generalizability compared to classical VLN agents in unseen environments. Our model performs favorably on standard VLN metrics, with Success Rate 68\% and Success weighted by Path Length 63\% in unseen environments.

Deep convolutional neural networks for dense prediction tasks are commonly optimized using synthetic data, as generating pixel-wise annotations for real-world data is laborious. However, the synthetically trained models do not generalize well to real-world environments. This poor “synthetic to real” (S2R) generalization we address through the lens of shortcut learning. We demonstrate that the learning of feature representations in deep convolutional networks is heavily influenced by synthetic data artifacts (shortcut attributes). To mitigate this issue, we propose an Information-Theoretic Shortcut Avoidance (ITSA) approach to automatically restrict shortcut-related information from being encoded into the feature representations. Specifically, our proposed method minimizes the sensitivity of latent features to input variations: to regularize the learning of robust and shortcut-invariant features in synthetically trained models. To avoid the prohibitive computational cost of direct input sensitivity optimization, we propose a practical yet feasible algorithm to achieve robustness. Our results show that the proposed method can effectively improve S2R generalization in multiple distinct dense prediction tasks, such as stereo matching, optical flow, and semantic segmentation. Importantly, the proposed method enhances the robustness of the synthetically trained networks and outperforms their fine-tuned counterparts (on real data) for challenging out-of-domain applications.