Name
Abstract | ||
|---|---|---|
We report on our experience scaling up the Mobile Millennium traffic information system using cloud computing and the Spark cluster computing framework. Mobile Millennium uses machine learning to infer traffic conditions for large metropolitan areas from crowdsourced data, and Spark was specifically designed to support such applications. Many studies of cloud computing frameworks have demonstrated scalability and performance improvements for simple machine learning algorithms. Our experience implementing a real-world machine learning-based application corroborates such benefits, but we also encountered several challenges that have not been widely reported. These include: managing large parameter vectors, using memory efficiently, and integrating with the application's existing storage infrastructure. This paper describes these challenges and the changes they required in both the Spark framework and the Mobile Millennium software. While we focus on a system for traffic estimation, we believe that the lessons learned are applicable to other machine learning-based applications. |
Year | DOI | Venue |
|---|---|---|
2011 | 10.1145/2038916.2038944 | SoCC |
Keywords | Field | DocType |
real-world machine,mobile millennium software,cloud computing,mobile millennium,mobile millennium system,mobile millennium traffic information,spark cluster computing framework,spark framework,cloud computing framework,simple machine,machine learning-based application,configuration management,information system,information retrieval,cluster computing,machine learning,robustness,failover | Information system,Failover,Spark (mathematics),Computer science,Software,Configuration management,Computer cluster,Scalability,Cloud computing,Distributed computing | Conference |
Citations | PageRank | References |
12 | 2.12 | 13 |
Authors | ||
8 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Timothy Hunter | 1 | 389 | 17.41 |
| Teodor Moldovan | 2 | 60 | 4.61 |
| Matei Zaharia | 3 | 9101 | 407.89 |
| Samy Merzgui | 4 | 12 | 2.12 |
| Justin Ma | 5 | 2314 | 104.86 |
| Michael J. Franklin | 6 | 17423 | 1681.10 |
| Pieter Abbeel | 7 | 6363 | 376.48 |
| Alexandre M. Bayen | 8 | 1250 | 137.72 |