Aaron M. Dollar

The digitization of natural history collections over the past three decades has unlocked a treasure trove of specimen imagery and metadata. There is great interest in making this data more useful by further labeling it with additional trait data, and modern “deep learning” machine learning techniques utilizing convolutional neural nets (CNNs) and similar networks show particular promise to reduce the amount of required manual labeling by human experts, making the process much faster and less expensive. However, in most cases, the accuracy of these approaches is too low for reliable utilization of the automatic labeling, typically in the range of 80-85% accuracy. In this paper, we present and validate an approach that can greatly improve this accuracy, essentially by examining the “confidence” that the network has in the generated label as well as utilizing a user-defined threshold to reject labels that fall below a chosen level. We demonstrate that a naive model that produced 86% initial accuracy can achieve improved performance - over 95% accuracy (rejecting about 40% of the labels) or over 99% accuracy (rejecting about 65%) by selecting higher confidence thresholds. This gives flexibility to adapt existing models to the statistical requirements of various types of research and has the potential to move these automatic labeling approaches from being unusably inaccurate to being an invaluable new tool. After validating the approach in a number of ways, we annotate the reproductive state of a large dataset of over 600,000 herbarium specimens. The analysis of the results points at under-investigated correlations as well as general alignment with known trends. By sharing this new dataset alongside this work, we want to allow biologists to gather insights for their own research questions, at their chosen point of accuracy/coverage trade-off.

Force-sensing capabilities are essential for robot manipulation systems. However, commonly used wrist-mounted force/torque sensors are heavy, fragile, and expensive, and tactile sensors require adding fragile circuitry to the robot fingers while only providing force information local to the contact. Here, we present a vision-based contact force estimator that serves as a more cost-effective and easier-to-implement alternative to existing force sensors by leveraging deformations of compliant hands upon contacts when compliant hands are in use. Our approach uses an estimator that visually observes a specialized compliant robot hand (available open source with easy fabrication through 3D printing) and predicts the contact force on the basis of its elastic deformation upon external forces. Because using wrist-mounted cameras to observe the gripper is common for robot manipulation systems, our method can obtain additional force information provided that the gripper is compliant. We optimized our compliant hand to minimize friction and avoid singularities in finger configurations, and we introduced memory to the estimator to combat the partial observability of the contact forces from the remaining friction and hysteresis. In addition, the estimator was made robust to background distractions and finger occlusions using vision foundation models to segment out the fingers. Although it is less accurate and slower than commercial force/torque sensors, we experimentally demonstrated the accuracy and robustness of our estimator (achieving between 0.2 newton and 0.4 newton error) and its utility during a variety of manipulation tasks using the gripper in the presence of noisy backgrounds and occlusions.

Identifying predictive world models for robots from sparse online observations is essential for robot task planning and execution in novel environments. However, existing methods that leverage differentiable programming to identify world models are incapable of jointly optimizing the geometry, appearance, and physical properties of the scene. In this work, we introduce a novel rigid object representation that allows the joint identification of these properties. Our method employs a novel differentiable point-based geometry representation coupled with a grid-based appearance field, which allows differentiable object collision detection and rendering. Combined with a differentiable physical simulator, we achieve end-to-end optimization of world models or rigid objects, given the sparse visual and tactile observations of a physical motion sequence. Through a series of world model identification tasks in simulated and real environments, we show that our method can learn both simulation- and rendering-ready rigid world models from only one robot action sequence.

The robot manipulation ecosystem currently faces issues with integrating open-source components and reproducing results. This limits the ability of the community to benchmark and compare the performance of different solutions to one another in an effective manner, instead relying on largely holistic evaluations. As part of the COMPARE Ecosystem project, we are developing modular grasping and manipulation pipeline infrastructure in order to streamline performance benchmarking. The infrastructure will be used towards the establishment of standards and guidelines for modularity and improved open-source development and benchmarking. This paper provides a high-level overview of the architecture of the pipeline infrastructure, experiments conducted to exercise it during development, and future work to expand its modularity.

Due to the nature of their implementation, nearly all low-level fabrication processes produce solidly filled structures. However, lattice structures are significantly stronger for the same amount of material, resulting in structures that are much lighter and more materially efficient. Here we propose an approach for fabricating lattice structures that echoes 3D printing techniques. In it, a modular chain of specially designed links is “extruded” onto a substrate to produce various lattices configurations depending on the chosen assembly algorithm, ranging from rigid regular lattices with nodal connectivity of 12, octet-truss, to significantly less dense configurations. Compared to conventional additive manufacturing methods, our approach allows for efficient use of nearly any material or combination of materials to construct lattices with programmed arrangements. We experimentally demonstrate that a 3x3x2 lattice structure (287 total links) is fabricated in 27 minutes via a modified robotic arm and can support approximately 1000 N in compression testing.

Precision gas analyzers are widely used in ecological research for manual measurement of soil carbon flux, a key metric used in the study of climate change. We present a generational update to the first low-cost, autonomous, closed-chamber style soil CO2 flux sensors (Fluxbots). Fluxbot 2.0 is the first such low-cost autonomous flux chamber capable of real-time wireless data transmission, which enables ecologists conducting in situ soil carbon flux surveys to set up their own wireless sensor arrays, reporting carbon flux data in real time at a very high level of temporal resolution. The system’s low cost (less than 500 USD per unit) and long-range cellular data transmission capabilities also allow for greatly improved spatial resolution. Additionally, the updated system consumes significantly less power, resulting in the ability to be deployed for longer than 10 × the battery lifetime of the original version on a single charge.

Joint absence in people with upper-limb-difference leads to compensatory motions. Such compensation has long been a topic of study, but typically only for a single object/user layout, which is unlikely to spatially generalize. We seek to understand how motion varies over a planar workspace for different target orientations and wrist mobility conditions. We therefore present a study that records arm and torso pose during grasping of 49 equally spaced cylindrical targets. Furthermore, we seek to validate the research practice of using wrist-immobilizing bypass sockets on able-bodied participants to simulate prostheses without wrists. Participants were 2 transradial amputees and 7 able-bodied individuals who conducted the study with and without wrist braces, generating 2450 trajectories. Heat-maps illustrate variation over the workspace in Mean Joint Angle, Range of Joint Motion and Distance Travelled by Body Segment. Results indicate that greater wrist restriction primarily exacerbated shoulder internal rotation and elbow flexion, not the trunk. We observed that bypass sockets do not fully simulate amputee behavior. Furthermore, amputee reaching with their intact limb is different to the reaching motion of normative participants, implying that transradial limb-difference affects both sides of the body. Differences in participant behavior were also observed between horizontal and vertical target orientations.

Due to the nature of their implementation, nearly all low-level fabrication processes produce solidly filled structures. However, lattice structures are significantly stronger for the same amount of material, resulting in structures that are much lighter and more materially efficient. In this paper we propose an approach for fabricating lattice structures that echoes commercially successful additive manufacturing/3D printing techniques. In it, a modular chain of specially designed links is “extruded” onto a substrate to produce various lattice configurations depending on the chosen assembly algorithm, ranging from one of the strongest known rigid lattices, the octet-truss, to significantly less dense configurations. Like 3D printing, the process allows material to be compactly stored before being deployed into structures of nearly arbitrary geometry, but unlike it, the approach allows for the use of nearly any material or combination of materials. We demonstrate the concept experimentally with a nearly 300-link chain that is fed from a spool and autonomously laid down onto programmed lattice arrangements via a robotic arm modified for the task. In the current prototype implementation, ~300-link structures are fabricated in 27 minutes, and a 3x3x2 lattice structure (287 total links) is shown to support approximately 1000N in compression testing.

Robot manipulation today generally focuses on motions exclusively with a robot arm or a dexterous hand, but usually not a combination of both. However, complex manipulation tasks can require coordinating arm and hand motions that leverage capabilities of both, much like the coordinated arm and hand motions carried out by humans to perform everyday tasks. In this work, we evaluate unified manipulation with robot arms and dexterous hands, using a motion optimization framework that synthesizes a series of configuration states over the entire manipulation system. We characterize the possible benefits of unifying arm and dexterous hand capabilities within a single model via metrics such as pose accuracy, manipulability, joint-space smoothness, distance to joint-limits, distance to collisions, and more. Several arm-hand combinations are quantitatively compared in simulation on a variety of experiment tasks and performance measures. Our results suggest that combining motions from robot arms and dexterous hands indeed has compelling benefits, highlighting the exciting potential of continued progress in unified arm-hand motion synthesis for robotics applications.