PressureVision: Estimating Hand Pressure from a Single RGB Image
Preprint
Abstract
People often interact with their surroundings by applying pressure with their hands. While hand pressure can be measured by placing pressure sensors between the hand and the environment, doing so can alter contact mechanics, interfere with human tactile perception, require costly sensors, and scale poorly to large environments. We explore the possibility of using a conventional RGB camera to infer hand pressure, enabling machine perception of hand pressure from uninstrumented hands and surfaces. The central insight is that the application of pressure by a hand results in informative appearance changes. Hands share biomechanical properties that result in similar observable phenomena, such as soft-tissue deformation, blood distribution, hand pose, and cast shadows. We collected videos of 36 participants with diverse skin tone applying pressure to an instrumented planar surface. We then trained a deep model (PressureVisionNet) to infer a pressure image from a single RGB image. Our model infers pressure for participants outside of the training data and outperforms baselines. We also show that the output of our model depends on the appearance of the hand and cast shadows near contact regions. Overall, our results suggest the appearance of a previously unobserved human hand can be used to accurately infer applied pressure. Data, code, and models are available online.Keywords
All Related Versions
This publication has 26 references indexed in Scilit:
- A computer vision approach for classifying isometric grip force exertion levelsErgonomics, 2020
- Estimating Fingertip Forces, Torques, and Local Curvatures from Fingernail ImagesRobotica, 2019
- Learning the signatures of the human grasp using a scalable tactile gloveNature, 2019
- The TacTip Family: Soft Optical Tactile Sensors with 3D-Printed Biomimetic MorphologiesSoft Robotics, 2018
- Capturing Hands in Action Using Discriminative Salient Points and Physics SimulationInternational Journal of Computer Vision, 2016
- Flexible and stretchable fabric-based tactile sensorRobotics and Autonomous Systems, 2015
- Stiffness map of the grasping contact areas of the human handJournal of Biomechanics, 2013
- The neural basis of the Weber–Fechner law: a logarithmic mental number lineTrends in Cognitive Sciences, 2003
- Skin Melanin, Hemoglobin, and Light Scattering Properties can be Quantitatively Assessed In Vivo Using Diffuse Reflectance SpectroscopyJournal of Investigative Dermatology, 2001
- Photoplethysmograph fingernail sensors for measuring finger forces without haptic obstructionIEEE Transactions on Robotics and Automation, 2001