Paper Info
Title
Protecting World Leader Using Facial Speaking Pattern Against Deepfakes
Abstract
Face forgery instances involving celebrities are on the rise, owing to the ease with which their large quantity of videos may be accessible on the Internet, world leaders particularly. While current face manipulation detectors have achieved impressive results on several open datasets, which incorporate persons with various identities, they show performance degradation on these high-quality ones targeting at celebrities. What is more, these online videos usually undergo compression processing, marking the detection task harder. Besides, more face manipulation techniques arise for celebrities other than face-swap, such as lip-synchronize and image-animation, with which most works have not been concerned. This paper proposes a dual stream learning facial and speaking patterns method to protect celebrities against deepfakes. We design an action unit module based on facial action coding system along with an Action Unit Transformer (AUT) to exploit facial expressions embeddings. Besides, our method's dual stream architecture utilizes a Temporal Convolutional Network (TCN) to extract lip motion pattern and learns the relatedness between facial and speaking patterns. Our method could protect the person of interest (POI) against deepfakes in an end-to-end manner. Extensive experiments show that our method achieves better performance and has a higher resistance to video compression than state-of-the-art detection models.
Year
DOI
Venue
2022
10.1109/LSP.2022.3205562
IEEE SIGNAL PROCESSING LETTERS
Keywords
DocType
Volume
Lips, Deepfakes, Feature extraction, Faces, Gold, Transformers, Detectors, Face manipulation detection, identification of persons, multi modal integration, speech pattern recognition
Journal
29
ISSN
Citations 
PageRank 
1070-9908
0
0.34
References 
Authors
0
5
Name
Order
Citations
PageRank
Beilin Chu100.68
Weike You200.68
Zhen Yang300.34
jing412220.75
Renying Wang500.68