Evaluating web-based information systems effectiveness : an e-service quality multi-stakeholder perspective

Abstract

Over a period spanning approximately twenty five years Information Systems (IS) researchers have been plagued by the problem of how to evaluate IS effectiveness or success. After the advent of the World Wide Web (WWW) in the 1990s, questions have arisen regarding the relevance of previously established evaluation approaches to the evaluation of web-based IS. Subsequently, firms have invested billions of dollars anually in information systems but the lack of appropriate frameworks for evaluating their effectiveness made it difficult to determine the return on IS investment.

In a period spanning 20 years IS researchers proposed a diverse number of approaches e.g. the communications research of Shannon and Weaver (1949) and the information "influence theory" of Mason (1978). These were subsequently incorporated into a single model in 1992, called the IS Success Model (Delone & McLean, 1992). The principal objective of this research project was to develop a generic methodology for web-based Information Systems (IS) success evaluation from a multi-stakeholder perspective for specific IS contexts. The Delone and McLean IS Success Model (Delone and McLean, 1992) provides an underpinning framework for measuring IS effectiveness. In the latter 1990's the concept of service-quality was introduced into the IS effectiveness literature. In their updated IS Success Model, Delone and McLean (2003) then included service quality as key measure in the evaluation of IS success. Consequently this research project focused on how service quality concepts could be applied as a measure of IS effectiveness within e-commerce.

A 3-pronged approach to IS evaluation was proposed viz. identification of generic stakeholder groups such as e-Customer, sponsoring manager, internal users etc.; identification of the context of the IS evaluation for each stakeholder i.e. what is the main function and context of operation of the IS; and lastly identification of stakeholder specific e-SQ criteria. The study demonstrates that the success of IS deployed within online environments, could be evaluated and measured differently by each stakeholder for the various e-Service Quality (e-SQ) dimensions within a particular IS context.

The study presents the results of an investigation into a web-based IS at a national telecommunications company in South Africa which was evaluated using e-Service Quality (e-SQ) constructs. The study demonstrates the operationalisation of an e-SQ instrument for the purposes of evaluating IS effectiveness amongst multi-stakeholders. Evidence is provided that measuring attitudes of different stakeholders provides a more holistic perspective of IS success.

The primary conclusion reached is that by using a step-by-step methodology of IS success measurement, the objective of establishing whether companies have received a return on web-based IS investment, can be achieved. Furthermore, the outcomes of the study has contributed to existing literature on IS effectiveness measurement. In particular, it will add to the existing body of knowledge regarding the use of e-SQ instrument to evaluate multi-stakeholder perceptions.