1Robots.txt Study Shows Webmasters Favor Google
15 November 2007
The Pennsylvania State University conducted a study that showed Webmasters favored Google over other search engines. The study looked at which robots or crawlers were listed in a web site’s robots.txt file, and Google was listed more often than any other search engine. The paper is named Determining Bias to Search Engines from Robots.txt (PDF) (it may be slow, so here is a local copy) and showed some interesting details.
Post a comment