Estimating finger grip force from an image of the hand using Convolutional Neural Networks and Gaussian processes
- 1 May 2014
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3137-3142
Abstract
Estimating human fingertip forces is required to understand force distribution in grasping and manipulation. Human grasping behavior can then be used to develop force-and impedance-based grasping and manipulation strategies for robotic hands. However, estimating human grip force naturally is only possible with instrumented objects or unnatural gloves, thus greatly limiting the type of objects used. In this paper we describe an approach which uses images of the human fingertip to reconstruct grip force and torque at the finger. Our approach does not use finger-mounted equipment, but instead a steady camera observing the fingers of the hand from a distance. This allows for finger force estimation without any physical interference with the hand or object itself, and is therefore universally applicable. We construct a 3-dimensional finger model from 2D images. Convolutional Neural Networks (CNN) are used to predict the 2D image to a 3D model transformation matrix. Two methods of CNN are designed for separate and combined outputs of orientation and position. After learning, our system shows an alignment accuracy over 98% on unknown data. In the final step, a Gaussian process estimates finger force and torque from the aligned images based on color changes and deformations of the nail and its surrounding skin. Experimental results shows that the accuracy achieves about 95% in the force estimation and 90% in the torque.Keywords
This publication has 16 references indexed in Scilit:
- Task Dependency of Grip Stiffness—A Study of Human Grip Force and Grip Stiffness Dependency during Two Different Tasks with Same Grip ForcesPLOS ONE, 2013
- Computing grip force and torque from finger nail images using Gaussian processes2013
- Force prediction by fingernail imaging using active appearance models2013
- Relation between object properties and EMG during reaching to graspJournal of Electromyography and Kinesiology, 2013
- Anticipatory grip force control using a cerebellar modelNeuroscience, 2009
- EigenNail for Finger Force Direction RecognitionProceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (cat. No.01ch37164), 2007
- Sparse lumigraph relighting by illumination and reflectance estimation from multi-view images2006
- Measurement of Finger Posture and Three-Axis Fingertip Touch Force Using Fingernail SensorsIEEE Transactions on Robotics and Automation, 2004
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Survey of Texture MappingIEEE Computer Graphics and Applications, 1986