Tag Archives: Information System

Framework for Evaluating IS Success

22 Feb

Group 2 Members:

Christine Coughlan, Clifton Moore, Dermot Lucid, Niamh O’ Farrell &
Ronan Murphy

Introduction

We have created a framework which allows an organisation to evaluate the success of an information system unit. In order to develop our framework, we researched a number of IS success models developed by key authors in the IS field, for example, DeLone & McLean, Sedera, Gable, Seddon and Nelson. In researching these models we have identified both value and flaws within these models and have developed our own framework based on what we think are the most important IS dimensions to evaluate when measuring the success of an IS unit for any organisation today.

Our framework identifies key dimensions which must be measured to evaluate the overall success if an IS unit. Any firm, large or small, can use our success framework to measure the success of their IS unit by choosing suitable metrics to measure each dimension contained in the framework. We have created a framework that we believe to be both flexible and customisable. In our framework we have identified possible metrics used to measure each dimension but it is up to an organisation to decide on the most appropriate metrics to suit their organisational context and IS strategy. It is imperative to choose appropriate and agreed measures or this framework will fail to deliver its potential value.

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

Framework for Evaluating the Success of an IS Unit

IS success framework

Click on the image for a better view

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

Explanation of Dimensions

[1] Context

Seddon et al (1999) presented the Cameron and Whetten 1983 (Fig 1, Below) framework for contextualising and evaluating Organisational Performance, adapted in Seddon et al (1999) to IT Effectiveness. [1] We have adopted the same seven-point framework to contextualise IS Success, as endorsed by Petter, DeLone, and McLean (2008) and outlined in the earlier post ‘Relatively Successful IS’.

Though all seven points are important, we suggest, in line with Seddon et al, that
1. Stakeholder Perspective? 2. Type of Information System, and 7. Against which referent is Success to be judged? are central to contextualising Success. The Context dimension of our Success framework is designed to establish, and justify, what is to be deemed ‘Successful’ from the standpoint of the stakeholder concerned, regarding the relevant IS system, and in the particular situation/organisation. To this end the Context aspect should be regarded as a Canvas to identify and outline the perspective or varying perspectives from which the analysis is based.

Stakeholder

“A stakeholder is a person or group in whose interest the evaluation of IS success is being performed” (Seddon et al, 1999). Seddon believes that due to a range of different individuals within an organisation they are going to evaluate IT success in different ways and perspectives. The inclusion of the “stakeholder” section in the framework is aimed to provide an organisation with the tool to adapt and understand the view of a projects success from all stakeholders’ views. Each stakeholder may use their own dedicated canvas, or if deemed useful, multiple stakeholders may outline their perspectives/concerns on a shared canvas. As is the case with Osterwalder’s Business Model Canvas, participants might post their views into the various categories using ‘stickies’, perhaps colour-coded to their individual stakeholder perspectives etc, to build up a visual representation of where their various priorities lie. In either case, this approach allows for comparison of stakeholders various perspectives, priorities and concerns, and will lead various parties to a more complete understanding of the success/weaknesses of the system in question. For example, a user might acknowledge a manager’s concerns for cost and the IT department’s concerns over reliability, versus their own concerns regarding usability, and vice-versa.

The elements ‘Timeframe’, ‘Type of Data’, and ‘Purpose of Evaluation’, are important for clarity, while acknowledging whether the system is for ‘Voluntary or Mandatory Use’ is a key factor to keep in mind within the backdrop to the evaluation. In the operational canvas these four elements (shaded) might be replaced by more relevant concerns, and so, should be regarded as suggestions. Once the vision of success is established stakeholders can turn their attention to the Quality & Impact sections and prioritise and even assign weighting to the various underlying dimensions. The relevant, prioritised/weighted dimensions can be measured using a Likert Scale against Sedera et al’s 27 corresponding measures Figure 3 (Below) as mentioned in the earlier post ‘ IS Success Canvas’.

The table borrowed from Seddon et al (1999), Fig 2 (below) contains examples of various stakeholders and information systems, and this table can be employed to inform the context dimension of the framework. The table column and row headings are useful as prompts but are not exhaustive of potential perspectives. However, the strength of the canvas approach is that it is blank and can therefore accommodate all stakeholder/perspectives and various information systems. Also, though informative as regards Stakeholder and IS type, we favour Sedera’s refined four-dimensional model and its tested measures (Fig 3, below) over the measures contained in Seddon’s table (Fig 2, below)
In a nutshell, the left side of the framework is a canvas to establish and outline what is to be deemed IS success. The right side of our framework is concerned with evaluating the IS against this established vision of success.

[2] Quality and Impact

The DeLone and McLean original IS success model classified measures of success into six constructs; System-Quality, Information-Quality, Organisational-Impact, Individual-Impact, Satisfaction, and Use.
Gable et al (2008) later proposed that information quality and system quality as identified by McLean and DeLone should be elements of a greater construct – IS Quality, while individual and organisational impact should be sub elements of an IS Impact construct.
Furthermore, Gable et al proposed that the ‘Satisfaction’ and ‘Use’ concepts as identified by DeLone and McLean should only be used as a metric to measure IS Impact and IS Quality and should not be treated as dependent constructs.
Thus, in our framework we considered both models and have confined the 6 constructs identified by DeLone and McLean into 2 key constructs as put forth by Gable et al; IS Impact and IS Quality. These constructs can be seen on the top level of the diagram.

The Impact construct is concerned with the eventual outputs delivered by an IS. The reason organisations invest heavily in information systems is because they expect the IS to have positive impacts on individual users and the organisation as a whole. Individual-Impact looks at how the IS has influenced the productivity and capabilities of individual users. Possible measures which can be used include individual productivity, learning and decision effectiveness.
Organisational-Impact is concerned with how the IS contributes to overall organisational results and capabilities. Business process change, cost reductions and overall productivity can be used to measure organisational impact.

The Quality construct is used to measure the IT-Artefact or technology element of IS.
Information-Quality is concerned with the quality of the information produced by the system, for example in reports and on-screen. Some measures which have been developed and successfully measured according to gable et al (2008) include importance, relevance and accuracy.
System-Quality measures the success of IS from a technical and design perspective. Tried and tested measures of system quality include reliability, flexibility, and potential for customisation.

Underneath Quality and Impact in the diagram we have the structure of the IS unit and Net Benefits.

[3] Structure of IS Unit

The structure or make-up of an IS unit can greatly impact its success, for example the level of commitment and support from top management, the quality of communication, culture and the skills of the employees. We will explain each of these to give a greater view of how the structure of an IS unit can influence IS success.

Top Management Support

It is extremely important that top management do not forget about a project after the planning stage but instead are commitment at the time of system implementation. By being directly involved in a project, top management guides the implementation team, allocating resources for projects, and stepping in to solve critical issues likely to affect implementation.

Communication

Management of an IS unit also affects communication within an organization and ultimately the productivity of users. Communication in an enterprise is vital in managing a company more efficiently, keep close monitor on strategies, strong relationships with employees and to have strong relations with partners/clients.

Culture

Culture within an organisation is also critical in determining success as it can impact how innovation affects IT practices and overall performance. Culture can impact organisations in the following three ways
1) Culture within an organisation can provide unwritten guidelines for employees in how to create a good workplace and strengthen relationships in order to improve the social system in the organisation.
2) Culture in an organisation can also affect the ability to deal successfully with issues from both internal and external integration.
3) It can also determine the differentiating between in-group and out-group people.

Employee Skills and Training

Employee skills being one of the most important factors within an organisation are critical in achieving success. If the employee does not meet the requirements/skills needed to carry out the required tasks, it can affect productivity and efficiency. It is also important that a business has a well-established training program for new employees in order to gain the appropriate skills that may be required specific to the company.

[4] Net Benefits

As a group we felt that Net Benefits is needed within an IS framework to support management teams in determining the success of their IS unit. The Net benefit dimension was also used in the DeLone and McLean model (2003) for organising IS success measurements. [ ] Net benefits are the extent to which IS are adding to the success of individuals, organisations and groups. The support management team needs to identify what their net benefits are. Examples of organisational net benefits may include: improved decision making, productivity, increased sales, reductions in cost, profits, economic development and creation of jobs, (DeLone & McLean, 2008).

Conclusion

Our framework is a synthesis of the key dimensions evident in the IS Success model research, and we feel that the framework is applicable or adaptable to all IS evaluations. The framework is intentionally open in nature with regard to its dimensions and measures making it ideal for quickly establishing and explaining across various stakeholders the success or less successful aspects of a system, while if necessary, thorough quantitative methods may be applied to the various dimensions, depending on the nature of the evaluation.

References:

• Seddon, P. B., Staples, S., Patnayakuni, R., & Bowtell, M. (1999). Dimensions of information systems success. Communications of the AIS, 2(3es)
• Petter,S., DeLone,W. & McLean,E. (2008). Measuring information systems success: models dimensions, measures, and interrelationships.
• Gable, Guy G. and Sedera, Darshana and Chan, Taizan (2008) Re-conceptualizing information system success: the IS-Impact Measurement Model. Journal of the Association for Information Systems.
• DeLone, W. and McLean, E. (2003). The DeLone and McLean Model of Information Systems Success: A Ten-Year Update.

Figure 1: Seven Questions to Answer when Measuring Organisational Performance – Cameron and Whetten (1983)

Figure 1: Seven Questions to Answer when Measuring Organisational Performance – Cameron and Whetten (1983)

Figure 2: IS Effectiveness Measures used for different combinations of Stakeholder and System – Seddon et al, (1999)

Figure 2: IS Effectiveness Measures used for different combinations of Stakeholder and System - Seddon et al, (1999)

Figure 3: Gable et al (2008) Impact Measures

Figure 3: Gable et al (2008) Impact Measures

Advertisements

Measuring information system quality, from the past to the present

8 Feb

The assessment of information system in the organisation is not well established and the studies show that more research is needed (DeLone & Mclean, 1992; Clark, 1992; Saunders & Jones, 1992; Wells & Wilkes, 1988 ). The IS performance and the organisational performance should be equally and carefully exploreAlso some of the roles of the IS function in the organisation are linked and can be subtle and difficult to measure (Crowton & Treacy, 1986; Niederman et al.). What is required for IS measurement framework is an assessment which connects the organisational performance to IS. The measurement should incorporate all the surrounding functions and factors in order to meet the need of both the organisation and customers.

Studies of IS assessment were initially focused on economic consideration(Ahituv, 1980; Bender, 1986Martin, 1979; the measurement focus shifted from efficiency, ie doing the things rightly, to effectiveness which is doing the right things McLean, 1973,B. Myers et tal, 1997.

Development of IS measurement.

The literature holds that Moad , 1993 presented a 3 by 3 matrix of IS framework for evaluating the IS functions (B. Myers et tal, 1997). Others have developed IS assessment framework (Dickson et al, 1988; Wells, 1987). However the most recent and comprehensive framework was developed by DeLone & McLean (1992) for IS multidimensional Model which was popular for measuring the complex IS ‘independent variable’. This model combines major previous IS Models theories and research studies to produce a multidimensional model called the D & M Success Model shown below in fig3.

Fig 3. D & M Success Model (DeLone & McLean, 1992)

fig3 DeLone & McLean Model

In the IS model in fig 3 above there are 6 interrelated dimensions of information systems success, comprising  the system quality, information quality, service quality, use, user satisfaction, net benefits (DeLone & McLean, 1992). It is also noted that in this model the elements are interconnected and interrelated in their functions. According to Delone  & McLean the IS success Model the “Systems quality” measures the systems technical success. The “information quality” measures the semantic success and the rest of the 4 components; the “use, user satisfaction, individual impacts and organisational impacts” measure the effectiveness success.

There are 3 steps in the “temporal process” of the D & M model, comprising of the following: 1. creation of a rich information system which exhibits varying degrees of information quality; 2. Users of this system are categorised as satisfied or dissatisfied with the information; Thus the individual user is impacted by the use of the system and its information and these individual users collectively impact the organisation which results to certain conditions (DeLone & McLean, 1992)

Also a “causal model studies the covariance of the success relationship to determine if there is a causal relationship in the dimensions. This is whether higher system or information quality results to higher use and user satisfaction to impact the individual and finally the organisation positively or the other way round (DeLone & McLean, 1992). The D & M model establishes the relationships between the components of the models and indicating the interdependencies among the variables of the IS ( Seddon and Kiew, 1996) Also the causal chain is a rigid presumption that higher quality will result to greater satisfaction and impact. What of other environmental factors which affect both the customers and the business.

With this review we shall compare and modify our previous framework in fig 2 in order to accommodate all the prevailing factors and eliminate all the weaknesses. See the comparative view. Fig4.

Fig 4. IS Interactive Componentsnew4

DeLone, W., and McLean, E. Information systems success: The quest for the dependent variable. Information Systems Research, 3, 1 (1992),

Seddon, P.B. A respecification and extension of the DeLone and McLean model of IS success. Information Systems Research, 8, 3 (1997),

P.B., and Kiew, M.-Y. A partial test and development of the DeLone and McLean model of IS success. In J.I. DeGross, S.L. Huff, and M.C. Munro (eds.), Proceedings of the International Conference on Information Systems. Atlanta, GA: Association for Information Systems, 1994, pp. 99–110.30  DELONE AND MCLEAN

What is Information Systems quality and who beholds it?

8 Feb

In this blog we shall review our tentative Fig 1 model to define and include other parameters of the IS measurement model such as quality.

The word ‘quality’ is so frequently used, however its use is somehow ubiquitous, deep and enigmatic. ‘Quality’ is like beauty, it is said to be ‘in the eye of the beholder’. Quality is a comparative attribute or relative characteristics of things which may be observed to be good, bad or ugly; high, medium or low; big, average or small. Quality may be about things and also relative in the way it is observed and described by various people. Therefore we have to determine what is observed and understand its description or attributes in the eye of the observers. In this case there is a product to be observed by people.We have therefore derived some fundamental factors in the theme; that quality comprises of 3 major requirements, the product, the process of observation and the people to observe it. The fundamental questions are therefore, what is the product, how is it observed and who are the people observing.

Based on these questions above we shall analyse the various elements of the products and their qualities and the process of observation and those involved. In this we will also see the information system as a kind of product which comprise of other components. Also we shall understand how it could be observed and why it should be observed, perhaps because there are various misunderstanding and doubts, a kind of murky darkness which requires a brighter light and the need for observation. Also there are various people (the beholders) involved; perhaps the public, the government, the employers and employees ‘in whose eyes the beauty lay’. Finally we shall find how all these fragments of different composites and sub-components can be arranged systematically together as a structure or a framework resulting to a meaningful expression in thought and word from which further actions could be derived.

The overall questions narrows down to the ratings or grading or qualities of the various components of the information system which combine with certain qualities of the functions of business in the environment to produce certain qualities of output service. Therefore the output is a measure of all the qualities of the components which combine to result to a measurable output.

Therefore the system quality measure is equal to the resultant of all the measures of the various components of each system. The organisational quality + IS quality = output quality which also return to feed the other. This is shown in fig 2 below.

Fig 2. Quality components of the information system elements                                                            Fig 2. Quality components of IS Interactive elements IS Interactive Componentsnew

Basic Building Blocks for Determining IS Services and Outputs.

7 Feb

In our previous blog we dis- assembled the various components involved in the measurement of IS management. These entities are further disassembled here and moulded into a certain visual structure which suggests a kind of pattern or suggestive interactive framework in their working relationship.

Narratively the environment comprise of business elements interacting with information systems to produce a certain quality of outputs? Therefore our building blocks will be derived from the following entities:

Environment: cultural, economic, political, demographic, regulatory and technological, consumers, competition etc.

As indicated above the environmental factors are physical, cultural, economic, political, demographic, regulatory and technological factors that affect the business (http://www.businessdictionary.com/definition/environmental-factors.html).

Organisation: planning, organising, staffing, directing, and controlling

Similarly, the management functions compose primarily of five basic functions which are utilised to accomplish the desired goals and objectives of the organisation. The management therefore comprise of planning, organising, staffing, directing, and controlling the people and the process in order to accomplish the desired goals of the organisation (Henri Fayol, 1841-1925) http://en.wikipedia.org/wiki/Fayolism . To these 5 functions we shall add information system management as a factor of business management (http://en.wikipedia.org/wiki/Management). All the above functions are targeted towards some desired goals and objectives which are measurable outputs.

Also the Information System comprises of people, process, IT artefacts.

The Output comprises of measurable values and other characteristics of different qualities

Without going into many details, these business functions have individual, interrelated and collective goals which are tied to the overall organisational mission. Therefore we shall formulate and analyse this integration and interrelationship of business functions and information technology functions in an environment.  Building these entities into a building a framework gives the visual impression shown below in Fig 1 below.

Fig. 1 A framework for the interaction of Information System and other entities in a business.

interactive components

 

 

Developing a framework for management teams to determine the quality of their IS services and outputs.

5 Feb

We should begin the above topic by selecting the basic components and differentiating them into primary constituents. The question requires us to derive a framework (basic structure or building blocks) for management to measure the Information System Quality (ISQ). Usually the Information System and the organisation are intermingled to produce measurable outputs in the environment and the environment also feeds into the system. Therefore the various entities of the various systems have to be disassembled. This will result to some basic questions, such as:  What are the components of the organisation and what are the components of the information technology? How do they interact with themselves and how do they interact with the components of the environment. How do they produce measurable outputs? These questions are related to the narratives of the functions of the members or entities involved in the system. This system therefore comprise of the following; Information System, of an organisation, operating to produce a measurable (quality) output, which feeds back into the environment of people including customers and businesses including competitors.

We will try to understand the nature of the relationships between these entities. The relationships may be one-way relationship, mutual or two-way relationship or opposing relationship. We will also disintegrate these entities into their smaller composite parts. In this case we have to decompose the business organisation and the environment into various functions. Also we have to breakdown information systems into various components to derive a measurable output quality.

The following entities are now part of the requirements analysis for the framework:

Information System: comprising of people, process, IT artefacts

Organisation: comprising all the business functions

Output: comprising of various values and other measurable characteristics (of quality)

Environment: of people including consumers and business including competitors etc

Finally we will relate these major entities to derive a workable framework and to derive some measurable characteristics in their relationships.

%d bloggers like this: