A formal, application-specific evaluation approach

I’m not disagreeing with Sully1210  regarding flexible approaches to evaluation, such as the balanced scorecard (https://sopinion8ed.wordpress.com/2012/11/04/the-use-of-balance-scorecards-in-evaluating-information-systems-investments/), but I’m going to look at a methodology in the literature positioned at the other end of the spectrum and the benefits it appears to offer. The approach suggested by Irani (2002) employs a robust evaluation model which utilises evaluation criteria, both tangible and intangible, specific to the organisation. Irani proposes that this approach would make the evaluation process more manageable by helping to reduce the time needed to make IT investment decisions since non-value adding activities would be removed and focus would remain fixed on the evaluation process.

A case study analysis did indeed suggest that a descriptive application-specific evaluation framework made the process more manageable and provided a high level of focus and depth. The framework also provided a structure to support a learning experience that was central to the successful development and implementation of the custom system. In contrast, Irani argues that reliance on an ad-hoc approach is subjective and lacking in both robustness and depth, which might result in the system being abandoned and ultimately deemed a failure.  Furthermore, the approach’s structure encouraged the participation of different stakeholders within the organisation during the concept and financial justification stages, which led to an increased level of commitment towards the project’s success. Finally, Irani argues that an integrated, generic approach to evaluation (such as BSC) is impossible because of the complex and interacting social and technical factors involved in the process (Irani, 2002).

However, Irani’s findings relate to a specific company and the data used was not systematically sampled which means that it is not possible to generalise these findings to other companies (Irani, 2002). The company in the case study was a custom build manufacturing organisation towards which an application-specific evaluation process may have been suitable. Although a formal methodology may provide more focus and depth on the process, its inflexibility makes adaptation to changes in technology or the competitive market difficult and often fails to consider external factors, in contrast to BSC (https://sopinion8ed.wordpress.com/2012/11/04/the-use-of-balance-scorecards-in-evaluating-information-systems-investments/). Furthermore, the benefits of the learning experience and encouraged participation are not limited to a formal evaluation framework.