Paper Info
Title
Self-supervised CNN for Unconstrained 3D Facial Performance Capture from an RGB-D Camera.
Abstract
We present a novel method for real-time 3D facial performance capture with consumer-level RGB-D sensors. Our capturing system is targeted at robust and stable 3D face capturing in the wild, in which the RGB-D facial data contain noise, imperfection and occlusion, and often exhibit high variability in motion, pose, expression and lighting conditions, thus posing great challenges. The technical contribution is a self-supervised deep learning framework, which is trained directly from raw RGB-D data. The key novelties include: (1) learning both the core tensor and the parameters for refining our parametric face model; (2) using vertex displacement and UV map for learning surface detail; (3) designing the loss function by incorporating temporal coherence and same identity constraints based on pairs of RGB-D images and utilizing sparse norms, in addition to the conventional terms for photo-consistency, feature similarity, regularization as well as geometry consistency; and (4) augmenting the training data set in new ways. The method is demonstrated in a live setup that runs in real-time on a smartphone and an RGB-D sensor. Extensive experiments show that our method is robust to severe occlusion, fast motion, large rotation, exaggerated facial expressions and diverse lighting.
Year
Venue
Field
2018
arXiv: Computer Vision and Pattern Recognition
UV mapping,Motion capture,Pattern recognition,Computer science,Coherence (physics),Regularization (mathematics),Parametric statistics,Facial expression,RGB color model,Artificial intelligence,Deep learning
DocType
Volume
Citations 
Journal
abs/1808.05323
0
PageRank 
References 
Authors
0.34
31
5
Name
Order
Citations
PageRank
Yudong Guo101.01
Juyong Zhang237934.08
Lin Cai342.19
jianfei cai41804147.18
jianmin zheng5102499.03