A one-size fits all approach does not exist

14 Nov

There is little doubt that information systems evaluation is problematical (Smithson & Hirscheim, 1988). Berghout and Renkama (1994) list 60 evaluation techniques, while Katz (1993) cites Wilson as having identified over 160. Renkema and Berghout (1997) considered over 65 methods for IT evaluation and conclude that the available non-financial evaluation methods are barely supported by theoretical foundations.

There is not one individual evaluation method that can be used for every company. With so many different evaluation measures, how will a company decide which one to use? When will a company use quantitative measures over qualitative measures or use both together? The answer lies in how the company perceive the value of IT.

Investment decisions are based on perceived value, however measured.  An understanding of how value translates to decisions can be aided by classifying approaches to evaluating IT decisions into three basic techniques that can be used in two different ways. (Bannister & Remenyi 1999)

The first level consists of the basic approaches to evaluation, which can be termed Fundamental, Composite and Meta methods.

Fundamental measures are metrics which attempt to assign parameters to some characteristic or closely related set of characteristics of the investment down to a single measure. These include IRR and ROI as discussed in previous blogs. Measures of this type are not confined to the purely financial, although financial measures are the most common.

Composite approaches combine several fundamental measures to get a ‘balanced’ overall picture of value/investment return. These include portfolio methods as lucid21 discussed  and the Balanced Scorecard of Kaplan and Norton (1996) which sully1210 talked about. There are few organisations that would try to evaluate their information systems activity today without using some variant of the composite approach.

Meta approaches (e.g. Farbey et al 1993; Peters, 1994) attempt to select the optimum set of measures for a context or set of circumstances.  This meta orientation is not usually structured and there is no question of the organisation wishing to use this approach for any sort of benchmarking other than for internal comparison between different projects and or over time when the same meta approach is being applied.

These three approaches may be applied in two different ways:

1. Positivist, where the decision-maker allows the methodology to make the decision.  In this approach the investment with the highest return or with the best overall score in some ranking is chosen.  The decision-maker establishes a series of mechanical operations which reduce the decision to a single score, either by using a preferred basic method, combining several such methods with a composite technique or using a meta approach to select a single method.  The latter two can be combined, ie. using a meta method to select an optimum set of techniques to be used and an ad hoc composite method to combine them.

2.  Hermeneutic, here defined as methods of interpretation of data which use non-structured approaches to both understanding and decision-making.  Here the decision-maker takes on board several different metrics directly and combines them in his or her mind in a manner that cannot be formally stated.  It is in this area that instinct and intuition plays the biggest role.

The concept is illustrated below: (click to enlarge)

If a company perceives the value of IT/IS as purely financial then quantitative measures will be used or if the company perceive the value of IT/IS as being more intangible then qualitative measures will be used. It depends on how the company perceives the value of IT/IS that will determine the evaluation method to use. For example, if the evaluation relates to IT/IS in services, one might need to assign more weight to intangibles, and if it is in manufacturing, one might need to assign higher weight to tangibles.


8 Responses to “A one-size fits all approach does not exist”

  1. agblogail November 14, 2012 at 2:42 pm #

    Great job 1rguru. Recent discussions on “the productivity Paradox” have left em wondering how companies would eventually measure IT, especially with all these “intangibles”. One size not fitting all is an excellent point.

    • steepletoes November 29, 2012 at 1:14 pm #

      Excellent post ‘1rguru’. Your point about how a one size fits all framework does not exist is very relevant to the adoption of evaluation methods and that a custom approach is the right route to take forward.

      • 1rguru November 29, 2012 at 1:39 pm #

        Im glad you agree steepletoes, the main issue is peoples perceived value of IT that will determine the method of use. That is why a customized approach which can be adapted based on peoples percevied value is more applicable.

      • 1rguru November 29, 2012 at 1:53 pm #

        Im glad you agree steepletoes, the main issues is peoples perceived value of IT. A customised evaluation method is better suited because of its adapted nature to deal with all types of perceived Value of IT.


  1. When should the value of an IS investment be measured quantitatively or qualitatively? « So Opinionated … - November 17, 2012

    […] should be measured and as irguru previously outlined, there is no one size fits all model. ((https://sopinion8ed.wordpress.com/2012/11/14/a-one-size-fits-all-approach-does-not-exist/#comments […]

  2. Different evaluation measure for different types of investments « So Opinionated … - November 19, 2012

    […] my last blog: https://sopinion8ed.wordpress.com/2012/11/14/a-one-size-fits-all-approach-does-not-exist/#comments I discussed how there is no one fit solution to evaluating an IS system. That it will depend on how […]

  3. Implications of IS value « So Opinionated … - November 26, 2012

    […] my previous blog https://sopinion8ed.wordpress.com/2012/11/14/a-one-size-fits-all-approach-does-not-exist/ I stated that; it depends on how the company perceives the value of IT/IS that will determine the […]

  4. Towards a generic IS implementation evaluation? « So Opinionated … - November 26, 2012

    […] https://sopinion8ed.wordpress.com/2012/11/14/a-one-size-fits-all-approach-does-not-exist/), while others have outlined different methodologies that are in use […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: