Name
Abstract | ||
|---|---|---|
A significant proportion of Web traffic is now attributed to Web robots, and this proportion is likely to grow over time. These robots may threaten the security, privacy, functionality, and performance of a Web server due to their unregulated crawling behavior. Therefore, to assess their impact, it must be possible to accurately detect Web robot requests. Contemporary detection approaches, however, may cease to be effective as the behavior of both robots and humans evolves. In this paper, we present a novel detection approach that is based on the contrasts in the resource request patterns of robots and humans. The proposed scheme, which relies on an invariant behavioral difference between humans and robots, builds on the lessons from contemporary approaches. We demonstrate that the proposed approach can accurately detect Web robots and argue that it is expected to remain effective even as they continue their rapid evolution. |
Year | DOI | Venue |
|---|---|---|
2012 | 10.1109/ICMLA.2012.11 | ICMLA (1) |
Keywords | Field | DocType |
crawling behavior,user classification,web server,web robot,web robot request,contemporary detection approach,web mining,humans evolves,web traffic,novel detection approach,computer network security,web log analysis,invariant behavioral difference,resource request pattern,proposed scheme,internet,detection,resource request patterns,web crawler,contemporary approach,detecting web,web robots | Web page,Computer science,Web modeling,Human–computer interaction,Artificial intelligence,Web API,World Wide Web,Robots exclusion standard,Web navigation,Web application security,Web service,Machine learning,Web server | Conference |
Volume | ISBN | Citations |
1 | 978-1-4673-4651-1 | 1 |
PageRank | References | Authors |
0.37 | 0 | 2 |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Derek Doran | 1 | 170 | 21.22 |
| Swapna S. Gokhale | 2 | 860 | 77.93 |