Name
Abstract | ||
|---|---|---|
This paper proposes a real-time lip reading system (consisting of a lip detector, lip tracker, lip activation detector, and word classifier), which can recognize isolated Korean words. Lip detection is performed in several stages: face detection, eye detection, mouth detection, mouth end-point detection, and active appearance model (AAM) fitting. Lip tracking is then undertaken via a novel two-stage lip tracking method, where the model-based Lucas-Kanade feature tracker is used to track the outer lip, and then a fast block matching algorithm is used to track the inner lip. Lip activation detection is undertaken through a neural network classifier, the input for which being a combination of the lip motion energy function and the first dominant shape feature. In the last step, input words are defined and recognized by three different classifiers: HMM, ANN, and K-NN. We combine the proposed lip reading system with an audio-only automatic speech recognition (ASR) system to improve the word recognition performance in the noisy environments. We then demonstrate the potential applicability of the combined system for use within hands free in-vehicle navigation devices. Results from experiments undertaken on 30 isolated Korean words using the K-NN classifier at a speed of 15fps demonstrate that the proposed lip reading system achieves a 92.67% word correct rate (WCR) for person-dependent tests, and a 46.50% WCR for person-independent tests. Also, the combined audio-visual ASR system increases the WCR from 0% to 60% in a noisy environment. |
Year | DOI | Venue |
|---|---|---|
2011 | 10.1016/j.patcog.2010.09.011 | Pattern Recognition |
Keywords | Field | DocType |
proposed lip reading system,lip activation detection,lip detection,novel two-stage lip tracking,real-time lip reading system,audio-visual asr,lip tracking,lip reading,automatic speech recognition,word classifier,lip activation detector,lip tracker,inner lip,lip detector,two-stage lip tracking,isolated korean word recognition,lip motion energy function,lucas kanade,word recognition,active appearance model,real time,face detection,block matching algorithm | Block-matching algorithm,Pattern recognition,Computer science,Word recognition,Speech recognition,Active appearance model,Artificial intelligence,Face detection,Hidden Markov model,Classifier (linguistics),Detector,Lip tracking | Journal |
Volume | Issue | ISSN |
44 | 3 | Pattern Recognition |
Citations | PageRank | References |
19 | 1.08 | 13 |
Authors | ||
3 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Jongju Shin | 1 | 53 | 6.13 |
| Jin Lee | 2 | 19 | 1.08 |
| Daijin Kim | 3 | 1882 | 126.85 |