Search engine exclusion policies: implications on indexing e-commerce websites

Abstract

The aim of this research was to determine how search engine exclusion policies and spam affect the indexing of e-Commerce websites. The Internet has brought along new ways of doing business. The unexpected growth of the World Wide Web made it essential for firms to adopt e-commerce as a means of obtaining a competitive edge. The introduction of e-commerce in turn facilitated the breaking down of physical barriers that were evident in traditional business operations. It is important for e-commerce websites to attract visitors, otherwise the website content is irrelevant. Websites can be accessed through the use of search engines, and it is estimated that 88% of users start with search engines when completing tasks on the web. This has resulted in web designers aiming to have their websites appear in the top ten search engine result list, as a high placement of websites in search engines is one of the strongest contributors to a commercial website’s success. To achieve such high rankings, web designers often adopt Search Engine Optimization (SEO) practices. Some of these practices invariably culminate in undeserving websites achieving top rankings. It is not clear how these SEO practices are viewed by search engines, as some practices that are deemed unacceptable by certain search engines are accepted by others. Furthermore, there are no clear standards for assessing what is considered good or bad SEO practices. This confuses web designers in determining what is spam, resulting in the amount of search engine spam having increased over time, impacting adversely on search engine results.

From the literature reviewed in this thesis, as well as the policies of five top search engines (Google, Yahoo!, AskJeeves, AltaVista, and Ananzi), this author was able to compile a list of what is generally considered as spam. Furthermore, 47 e-commerce websites were analysed to determine if they contain any form of spam. The five major search engines indexed some of these websites. This enabled the author to determine to what extent search engines adhere to their policies. This analysis returned two major findings. A small amount of websites contained spam, and from the pre-compiled list of spam tactics, only two were identified in the websites, namely keyword stuffing and page redirects. Of the total number of websites analysed, it was found that 21.3% of the websites contained spam. From these findings, the research contained in this thesis concluded that search engines adhere to their own policies, but lack stringent controls for the majority of websites that contained spam, and were still listed by search engines. In this study, the author only analysed e-commerce websites, and cannot therefore generalise the results to other websites outside ecommerce.