Paper Info
Title
Gesto: Mapping UI Events to Gestures and Voice Commands
Abstract
Gesto is a system that enables task automation for Android apps using gestures and voice commands. Using Gesto, a user can record a UI action sequence for an app, choose a gesture or a voice command to activate the UI action sequence, and later trigger the UI action sequence by the corresponding gesture/voice command. Gesto enables this for existing Android apps without requiring their source code or any help from their developers. In order to make such capability possible, Gesto combines bytecode instrumentation and UI action record-and-replay. To show the applicability of Gesto, we develop four use cases using real apps downloaded from Google Play-Bing, Yelp, AVG Cleaner, and Spotify. For each of these apps, we map a gesture or a voice command to a sequence of UI actions. According to our measurement, Gesto incurs modest overhead for these apps in terms of memory usage, energy usage, and code size increase. We evaluate our instrumentation capability and overhead using 1,000 popular apps downloaded from Google Play. Our result shows that Gesto is able to instrument 94.9% of the apps without any significant overhead. In addition, since our prototype currently supports 6 main UI elements of Android, we evaluate our coverage and measure what percentage of UI element uses we can cover. Our result shows that our 6 UI elements can cover 96.4% of all statically-declared UI element uses in the 1,000 Google Play apps.
Year
DOI
Venue
2019
10.1145/3300964
Proceedings of the ACM on Human-Computer Interaction
Keywords
DocType
Volume
gestures, mapping UI events, record and replay, voice commands
Journal
3
Issue
Citations 
PageRank 
EICS
0
0.34
References 
Authors
0
7
Name
Order
Citations
PageRank
Chang Min Park163.57
Taeyeon Ki2757.89
Ali J. Ben Ali321.50
Nikhil Sunil Pawar400.34
Karthik Dantu533137.87
Steven Y. Ko647145.08
Lukasz Ziarek729135.09