Paper Info
Title
American Sign Language fingerspelling recognition in the wild.
Abstract
We address the problem of American Sign Language fingerspelling recognition “in the wild”, using videos collected from websites. We introduce the largest data set available so far for the problem of fingerspelling recognition, and the first using naturally occurring video data. Using this data set, we present the first attempt to recognize fingerspelling sequences in this challenging setting. Unlike prior work, our video data is extremely challenging due to low frame rates and visual variability. To tackle the visual challenges, we train a special-purpose signing hand detector using a small subset of our data. Given the hand detector output, a sequence model decodes the hypothesized fingerspelled letter sequence. For the sequence model, we explore attention-based recurrent encoder-decoders and CTC-based approaches. As the first attempt at fingerspelling recognition in the wild, this work is intended to serve as a baseline for future work on sign language recognition in realistic conditions. We find that, as expected, letter error rates are much higher than in previous work on more controlled data, and we analyze the sources of error and effects of model variants.
Year
DOI
Venue
2018
10.1109/slt.2018.8639639
2018 IEEE Spoken Language Technology Workshop (SLT)
Keywords
DocType
Volume
Videos,Assistive technology,Gesture recognition,Detectors,Hidden Markov models,Error analysis,Image recognition
Conference
abs/1810.11438
ISSN
Citations 
PageRank 
2639-5479
1
0.41
References 
Authors
18
7
Name
Order
Citations
PageRank
Bowen Shi1122.66
Aurora Martinez Del Rio210.41
Jonathan Keane361.60
Jonathan Michaux410.41
Diane Brentari542.53
Gregory Shakhnarovich61579106.33
Karen Livescu7125471.43