Framework for Evaluating IS Success

22 Feb

Group 2 Members:

Christine Coughlan, Clifton Moore, Dermot Lucid, Niamh O’ Farrell &
Ronan Murphy

Introduction

We have created a framework which allows an organisation to evaluate the success of an information system unit. In order to develop our framework, we researched a number of IS success models developed by key authors in the IS field, for example, DeLone & McLean, Sedera, Gable, Seddon and Nelson. In researching these models we have identified both value and flaws within these models and have developed our own framework based on what we think are the most important IS dimensions to evaluate when measuring the success of an IS unit for any organisation today.

Our framework identifies key dimensions which must be measured to evaluate the overall success if an IS unit. Any firm, large or small, can use our success framework to measure the success of their IS unit by choosing suitable metrics to measure each dimension contained in the framework. We have created a framework that we believe to be both flexible and customisable. In our framework we have identified possible metrics used to measure each dimension but it is up to an organisation to decide on the most appropriate metrics to suit their organisational context and IS strategy. It is imperative to choose appropriate and agreed measures or this framework will fail to deliver its potential value.

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

Framework for Evaluating the Success of an IS Unit

IS success framework

Click on the image for a better view

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

Explanation of Dimensions

[1] Context

Seddon et al (1999) presented the Cameron and Whetten 1983 (Fig 1, Below) framework for contextualising and evaluating Organisational Performance, adapted in Seddon et al (1999) to IT Effectiveness. [1] We have adopted the same seven-point framework to contextualise IS Success, as endorsed by Petter, DeLone, and McLean (2008) and outlined in the earlier post ‘Relatively Successful IS’.

Though all seven points are important, we suggest, in line with Seddon et al, that
1. Stakeholder Perspective? 2. Type of Information System, and 7. Against which referent is Success to be judged? are central to contextualising Success. The Context dimension of our Success framework is designed to establish, and justify, what is to be deemed ‘Successful’ from the standpoint of the stakeholder concerned, regarding the relevant IS system, and in the particular situation/organisation. To this end the Context aspect should be regarded as a Canvas to identify and outline the perspective or varying perspectives from which the analysis is based.

Stakeholder

“A stakeholder is a person or group in whose interest the evaluation of IS success is being performed” (Seddon et al, 1999). Seddon believes that due to a range of different individuals within an organisation they are going to evaluate IT success in different ways and perspectives. The inclusion of the “stakeholder” section in the framework is aimed to provide an organisation with the tool to adapt and understand the view of a projects success from all stakeholders’ views. Each stakeholder may use their own dedicated canvas, or if deemed useful, multiple stakeholders may outline their perspectives/concerns on a shared canvas. As is the case with Osterwalder’s Business Model Canvas, participants might post their views into the various categories using ‘stickies’, perhaps colour-coded to their individual stakeholder perspectives etc, to build up a visual representation of where their various priorities lie. In either case, this approach allows for comparison of stakeholders various perspectives, priorities and concerns, and will lead various parties to a more complete understanding of the success/weaknesses of the system in question. For example, a user might acknowledge a manager’s concerns for cost and the IT department’s concerns over reliability, versus their own concerns regarding usability, and vice-versa.

The elements ‘Timeframe’, ‘Type of Data’, and ‘Purpose of Evaluation’, are important for clarity, while acknowledging whether the system is for ‘Voluntary or Mandatory Use’ is a key factor to keep in mind within the backdrop to the evaluation. In the operational canvas these four elements (shaded) might be replaced by more relevant concerns, and so, should be regarded as suggestions. Once the vision of success is established stakeholders can turn their attention to the Quality & Impact sections and prioritise and even assign weighting to the various underlying dimensions. The relevant, prioritised/weighted dimensions can be measured using a Likert Scale against Sedera et al’s 27 corresponding measures Figure 3 (Below) as mentioned in the earlier post ‘ IS Success Canvas’.

The table borrowed from Seddon et al (1999), Fig 2 (below) contains examples of various stakeholders and information systems, and this table can be employed to inform the context dimension of the framework. The table column and row headings are useful as prompts but are not exhaustive of potential perspectives. However, the strength of the canvas approach is that it is blank and can therefore accommodate all stakeholder/perspectives and various information systems. Also, though informative as regards Stakeholder and IS type, we favour Sedera’s refined four-dimensional model and its tested measures (Fig 3, below) over the measures contained in Seddon’s table (Fig 2, below)
In a nutshell, the left side of the framework is a canvas to establish and outline what is to be deemed IS success. The right side of our framework is concerned with evaluating the IS against this established vision of success.

[2] Quality and Impact

The DeLone and McLean original IS success model classified measures of success into six constructs; System-Quality, Information-Quality, Organisational-Impact, Individual-Impact, Satisfaction, and Use.
Gable et al (2008) later proposed that information quality and system quality as identified by McLean and DeLone should be elements of a greater construct – IS Quality, while individual and organisational impact should be sub elements of an IS Impact construct.
Furthermore, Gable et al proposed that the ‘Satisfaction’ and ‘Use’ concepts as identified by DeLone and McLean should only be used as a metric to measure IS Impact and IS Quality and should not be treated as dependent constructs.
Thus, in our framework we considered both models and have confined the 6 constructs identified by DeLone and McLean into 2 key constructs as put forth by Gable et al; IS Impact and IS Quality. These constructs can be seen on the top level of the diagram.

The Impact construct is concerned with the eventual outputs delivered by an IS. The reason organisations invest heavily in information systems is because they expect the IS to have positive impacts on individual users and the organisation as a whole. Individual-Impact looks at how the IS has influenced the productivity and capabilities of individual users. Possible measures which can be used include individual productivity, learning and decision effectiveness.
Organisational-Impact is concerned with how the IS contributes to overall organisational results and capabilities. Business process change, cost reductions and overall productivity can be used to measure organisational impact.

The Quality construct is used to measure the IT-Artefact or technology element of IS.
Information-Quality is concerned with the quality of the information produced by the system, for example in reports and on-screen. Some measures which have been developed and successfully measured according to gable et al (2008) include importance, relevance and accuracy.
System-Quality measures the success of IS from a technical and design perspective. Tried and tested measures of system quality include reliability, flexibility, and potential for customisation.

Underneath Quality and Impact in the diagram we have the structure of the IS unit and Net Benefits.

[3] Structure of IS Unit

The structure or make-up of an IS unit can greatly impact its success, for example the level of commitment and support from top management, the quality of communication, culture and the skills of the employees. We will explain each of these to give a greater view of how the structure of an IS unit can influence IS success.

Top Management Support

It is extremely important that top management do not forget about a project after the planning stage but instead are commitment at the time of system implementation. By being directly involved in a project, top management guides the implementation team, allocating resources for projects, and stepping in to solve critical issues likely to affect implementation.

Communication

Management of an IS unit also affects communication within an organization and ultimately the productivity of users. Communication in an enterprise is vital in managing a company more efficiently, keep close monitor on strategies, strong relationships with employees and to have strong relations with partners/clients.

Culture

Culture within an organisation is also critical in determining success as it can impact how innovation affects IT practices and overall performance. Culture can impact organisations in the following three ways
1) Culture within an organisation can provide unwritten guidelines for employees in how to create a good workplace and strengthen relationships in order to improve the social system in the organisation.
2) Culture in an organisation can also affect the ability to deal successfully with issues from both internal and external integration.
3) It can also determine the differentiating between in-group and out-group people.

Employee Skills and Training

Employee skills being one of the most important factors within an organisation are critical in achieving success. If the employee does not meet the requirements/skills needed to carry out the required tasks, it can affect productivity and efficiency. It is also important that a business has a well-established training program for new employees in order to gain the appropriate skills that may be required specific to the company.

[4] Net Benefits

As a group we felt that Net Benefits is needed within an IS framework to support management teams in determining the success of their IS unit. The Net benefit dimension was also used in the DeLone and McLean model (2003) for organising IS success measurements. [ ] Net benefits are the extent to which IS are adding to the success of individuals, organisations and groups. The support management team needs to identify what their net benefits are. Examples of organisational net benefits may include: improved decision making, productivity, increased sales, reductions in cost, profits, economic development and creation of jobs, (DeLone & McLean, 2008).

Conclusion

Our framework is a synthesis of the key dimensions evident in the IS Success model research, and we feel that the framework is applicable or adaptable to all IS evaluations. The framework is intentionally open in nature with regard to its dimensions and measures making it ideal for quickly establishing and explaining across various stakeholders the success or less successful aspects of a system, while if necessary, thorough quantitative methods may be applied to the various dimensions, depending on the nature of the evaluation.

References:

• Seddon, P. B., Staples, S., Patnayakuni, R., & Bowtell, M. (1999). Dimensions of information systems success. Communications of the AIS, 2(3es)
• Petter,S., DeLone,W. & McLean,E. (2008). Measuring information systems success: models dimensions, measures, and interrelationships.
• Gable, Guy G. and Sedera, Darshana and Chan, Taizan (2008) Re-conceptualizing information system success: the IS-Impact Measurement Model. Journal of the Association for Information Systems.
• DeLone, W. and McLean, E. (2003). The DeLone and McLean Model of Information Systems Success: A Ten-Year Update.

Figure 1: Seven Questions to Answer when Measuring Organisational Performance – Cameron and Whetten (1983)

Figure 1: Seven Questions to Answer when Measuring Organisational Performance – Cameron and Whetten (1983)

Figure 2: IS Effectiveness Measures used for different combinations of Stakeholder and System – Seddon et al, (1999)

Figure 2: IS Effectiveness Measures used for different combinations of Stakeholder and System - Seddon et al, (1999)

Figure 3: Gable et al (2008) Impact Measures

Figure 3: Gable et al (2008) Impact Measures

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: