Name
Abstract | ||
|---|---|---|
This paper presents a study on whether the heavy-tailed trends reported in Web traffic are present in the traffic generated by Web robots. The study is motivated by three factors: (i) a significant volume of Web server traffic can now be attributed to Web robots, (ii) the Web is continuing to evolve into a semantic and service-oriented environment where Web robots will play a central role, and (iii) there are fundamental differences in the way robots and humans visit a site and search for information and these differences may lead to contrasts in the statistical patterns of the robots' requests compared to humans. We analyze Web robot traffic from a two-year access log from a Web server in the academic domain and study whether the response sizes, request inter-arrival times, and inter-session times exhibit heavy-tailed properties. In a multi-faceted analysis of the data we find that the response sizes and request inter-arrival times of robot requests do not exhibit heavy-tailed characteristics, contrasting the trends in these metrics in human traffic. However, we find that inter-session times of robots follow heavy-tailed characteristics similar to that of humans. |
Year | DOI | Venue |
|---|---|---|
2010 | 10.1109/QEST.2010.42 | QEST |
Keywords | Field | DocType |
semantic-oriented environment,statistical distributions,request inter-arrival time,web server,web robot,robots,web robot traffic,web services,inter-session time,web traffic,heavy-tailed distributions,data analysis,inter-session times,human traffic,web server traffic,service-oriented environment,heavy tails,multifaceted data analysis,heavy-tailed characteristic,request inter-arrival times,response size,heavy tailed distribution,measurement,weibull distribution,data models,heavy tail | Web traffic,Data modeling,World Wide Web,Computer science,Robots exclusion standard,Web service,Service oriented environment,Robot,Web server | Conference |
ISBN | Citations | PageRank |
978-1-4244-8082-1 | 4 | 0.47 |
References | Authors | |
12 | 2 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Derek Doran | 1 | 170 | 21.22 |
| Swapna S. Gokhale | 2 | 860 | 77.93 |