A formal, application-specific evaluation approach

13 Nov

I’m not disagreeing with Sully1210  regarding flexible approaches to evaluation, such as the balanced scorecard (https://sopinion8ed.wordpress.com/2012/11/04/the-use-of-balance-scorecards-in-evaluating-information-systems-investments/), but I’m going to look at a methodology in the literature positioned at the other end of the spectrum and the benefits it appears to offer. The approach suggested by Irani (2002) employs a robust evaluation model which utilises evaluation criteria, both tangible and intangible, specific to the organisation. Irani proposes that this approach would make the evaluation process more manageable by helping to reduce the time needed to make IT investment decisions since non-value adding activities would be removed and focus would remain fixed on the evaluation process.

A case study analysis did indeed suggest that a descriptive application-specific evaluation framework made the process more manageable and provided a high level of focus and depth. The framework also provided a structure to support a learning experience that was central to the successful development and implementation of the custom system. In contrast, Irani argues that reliance on an ad-hoc approach is subjective and lacking in both robustness and depth, which might result in the system being abandoned and ultimately deemed a failure.  Furthermore, the approach’s structure encouraged the participation of different stakeholders within the organisation during the concept and financial justification stages, which led to an increased level of commitment towards the project’s success. Finally, Irani argues that an integrated, generic approach to evaluation (such as BSC) is impossible because of the complex and interacting social and technical factors involved in the process (Irani, 2002).

However, Irani’s findings relate to a specific company and the data used was not systematically sampled which means that it is not possible to generalise these findings to other companies (Irani, 2002). The company in the case study was a custom build manufacturing organisation towards which an application-specific evaluation process may have been suitable. Although a formal methodology may provide more focus and depth on the process, its inflexibility makes adaptation to changes in technology or the competitive market difficult and often fails to consider external factors, in contrast to BSC (https://sopinion8ed.wordpress.com/2012/11/04/the-use-of-balance-scorecards-in-evaluating-information-systems-investments/). Furthermore, the benefits of the learning experience and encouraged participation are not limited to a formal evaluation framework.


2 Responses to “A formal, application-specific evaluation approach”

  1. sully1210 November 13, 2012 at 3:21 pm #

    Great post mcconnell.

    After reading the Irani (2002) article that your blog is based on I cannot disagree with what you are saying. The final statement in the article is one that really caught my eye, and is one you mentioned in your post. Here Irani (2002) states that “there are a wide variety of interacting social and technical factors that complicate the evaluation process. This in turn makes the search for an integrated generic technique impossible”.
    I agree with this statement, as well I think that both business environments and social environments are constantly changing and this means that evaluation methods must also adapt and change along with the times. I would be very interested to see this method applied in other contexts and in other industries as it definitely has potential to be a very influential form of evaluation.

    I was also very interested to read about the similarities between both the BSC and the method you have outlined, for example the use of interviews to gage users opinions on a system.

    You also mentioned that the benefits of learning and experience cannot be limited to a formal evaluation framework, I agree, and as lucid21 pointed out it is ultimately experience that separates a great manager/decision maker from an ordinary one. Do you think the method you discuss above allows for the ‘gut instinct’ of decision makers when it comes to making IS investment decisions?

    • mcoconnell November 15, 2012 at 4:02 pm #

      I don’t think so. The framework seems too stringent to leave decision making open to ‘gut instinct’.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: