| Testing crawlerbots indexing speed |
|
|
| Written by European network dynamics principal author | |
|
According to serveral observations, see 1 and 2, many questions are to be asked concerning the incredible reactivity of crawlerbots such as Googlebot. First of all ameasurement is realized then results are discussed about in terms different means to acces newly published information by bots.
According to several concerns, measurements, and observations, the time given by bots before indexing newly published articles has drastically decreased these latest days dropping as low as a few minutes when published on small to average audiences websites (1 and 2 ). To our knowledge, no objective measurements have been objectively realized and publicly published concerning this indexing speed. Therefore we decided to do the following experiments using European network dynamics website and with the help of Webrankinfo.com, a well known french SEO website (3 ) . Hypothesis
Methods The time between the publication of 4 articles within European network dynamics and the passage of an indexing crawling bot such as googlebot were cautionously measured All articles published within European network dynamics are submitted to a RSS feed taken into consideration by bots and feeds agregators. The articles discussed above were writted using 'clean' browser (Firefox2, fresh install)with no helping bar such as google bar, nor alexa bar. A total of 4 articles are written for the study: (URL are given for information purposes only, nothing of interest can be found there) Article 1: totally new article with adsenses present in left column http://www.netdynamics.eu/analytics/testing-googlebots-indexing-speed-1.html Article 2: absolutely similar to Article 1, URL ends with ...-2.html, but the real URL (although easy to guess is not given here), Adsenses present as Article 1 Article 3: New article, with no adsense, using the following URL : http://www.netdynamics.eu/Testing_Googlebots_indexing_speed_3.html Article 4: exactly similar to Article 3 with no Adsenses, and no URL published, although ends with ..-4.html (see article 1 and 2) All four articles were submitted to publication exactly at 11.30 AM on Friday Aug.31st Western European time. In addition, all these 4 articles were also presented and discussed about in (3 ) as a 'promotionnal' discussion (see 4 ). Results An intensive log analysis taking only these 4 articles into consideration, of visitors after 11:30 AM this days gives the following results: Article 1At the time these lines are being written, Articles 3 and 4 have not been indexed yet. No other bots have been indexing these 4 articles. Discussion The speed at which Googlebot indexes new article seems tremendous, About 5 minutes only after an articla has been unveiled it is already indexed. Furthermore, the 'promotionnal' discussion used in (3) was indexed in google as soon as 12:51 on Aug 31st, see (5 Results strongly suggest that adsenses are responsible of the short delay between unveiling and indexing, because Googlebot's first fisit on Article 1 has no referrer. Let us recall that One 'human' visitor came before the bot. Apache log analysis does not reveal this visitor used any helping bar equipped browser, but even if not the case, this visit invoked a new url for Adsense, therefore triggering bot's visit. Main referrences
The authors of this study wish to thank webrankinfo.com for their help in this study. End notes concerning this study: This article was first published on Tuesday, 04 September 2007 22:18 and its first visit by googlebot occured some 30' after crawl-66-249-66-163.googlebot.com www.netdynamics.eu - [04/Sep/2007:22:55:31 +0200] "GET /analytics/testing-crawlerbots-indexing-speed.html HTTP/1.1" 200 10942 "-" "Mediapartners-Google"        |
| < Prev | Next > |
|---|
