Title
FingerTrak: Continuous 3D Hand Pose Tracking by Deep Learning Hand Silhouettes Captured by Miniature Thermal Cameras on Wrist
Abstract
In this paper, we present FingerTrak, a minimal-obtrusive wristband that enables continuous 3D finger tracking and hand pose estimation with four miniature thermal cameras mounted closely on a form-fitting wristband. FingerTrak explores the feasibility of continuously reconstructing the entire hand postures (20 finger joints positions) without the needs of seeing all fingers. We demonstrate that our system is able to estimate the entire hand posture by observing only the outline of the hand, i.e., hand silhouettes from the wrist using low-resolution (32 x 24) thermal cameras. A customized deep neural network is developed to learn to "stitch" these multi-view images and estimate 20 joints positions in 3D space. Our user study with 11 participants shows that the system can achieve an average angular error of 6.46° when tested under the same background, and 8.06° when tested under a different background. FingerTrak also shows encouraging results with the re-mounting of the device and has the potential to reconstruct some of the complicated poses. We conclude this paper with further discussions of the opportunities and challenges of this technology.
Year
DOI
Venue
2020
10.1145/3397306
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Keywords
DocType
Volume
hand reconstruction,pose recognition
Journal
4
Issue
ISSN
Citations 
2
2474-9567
0
PageRank 
References 
Authors
0.34
32
5
Name
Order
Citations
PageRank
Fang Hu1337.54
Peng He200.68
Songlin Xu300.34
Yin Li479735.85
Cheng Zhang518216.88