In my last blog: https://sopinion8ed.wordpress.com/2012/11/14/a-one-size-fits-all-approach-does-not-exist/#comments I discussed how there is no one fit solution to evaluating an IS system. That it will depend on how the company perceives the value of the system that will determine the method of evaluation, i.e. if the company sees the benefit of a new system as increasing staff moral (intangible) then they will engage in using methods such as the balance scorecard. Mirra2, kechy4me and sully1210 have all elaborated on when quantitative and qualitative measures should be used.
The search for one uniform evaluation method becomes unjustified, and a customized evaluation method should be designed for each IS investment. Bannister and Remenyi (1999) call it a “Meta approach‟ (as defined in last blog) and argue that this orientation is not structured and varies from case to case.
The selection of methods most appropriate for particular company’s circumstances is one of the biggest problems in IT/IS evaluation (Remenyi et. al., 2000). Remenyi puts forward a view of how different evaluation methods are chosen by relating the evaluation measure to the investment purpose. For example, if the purpose is to improve efficiency– a cost-benefit analysis should be performed, whilst for strategic investments – strategic analysis would be the suitable tool (Remenyi 2000). The relation of evaluation measure to the investment purpose determines the overall evaluation strategy, rather than the specific tools to be used (Lech 2005). Irani (2002) suggests that evaluation criteria should be dependent on the IT element (i.e an ERP system) that is implemented.
The first question that should be answered before the evaluation process begins is for what reason the IT/IS investment is being made. This is the purpose of implementation that determines general evaluation strategy. The relationship between the investment purpose and aspects that should be subject to evaluation can be seen in the below table. (Click to enlarge)
The Investment purpose first of all determines what should be measured (see above). If the investment is a “must-do” – which means that it is either required by law or is an industry standard, then the main benefit is staying on board. The optimizing criterion for such investment should be to obtain the desired goal at minimum costs (Lech 2005). However if the investment is for competitive advantage then a more strategic analysis is required.
I believe Lechs paper (2005) Evaluation Methods’ Matrix – A Tool for Customized IT Investment Evaluation, has contributed to outlining how companies determine what IS/IT evaluation methods to select when choosing an investment. By following the above figure one can see that an IS investment is split into 4 categories (investment purpose) which is linked to the evaluation aspect and methods used for evaluation. From this it can be concluded that the purpose of the investment will also be related to what evaluation method to use.
So Opinionated ... 
Many senior executives view IT as a necessary cost to be minimized. For example, the CEO of a US conglomerate of petroleum, natural gas, and chemicals expressed his frustration with IT: “All I get are these esoteric benefits and a bunch of baloney on how much technology has advanced. Show me where you put one more dollar on the income statement.” (Lacity et al., 1996 “The Value of Selective IT Sourcing”)
How would you sell the benefits of a IS investment to a CEO, like the one above, when you can’t demonstrate exactly where and how the business will improve in monetary terms? Will the CEO accept evaluations such as strategic analysis and future options analysis as you’ve outlined in your post as suitable determinants to give the go-ahead for the investment, for example?
Hi Steepletoes and thanks for your comment:
Its seems this CEO perceives the value of IT as being purely financial and would be most comfortable using traditional evaluation methods to assess an investment in IT such as IRR and payback methods.
However this CEO must be made aware that IT returns-on-investments are mostly intangibles (esoteric benefits) which make the value of the investment difficult to measure using traditional accounting practice. A common criticism for companies is that they do not give due consideration to intangibles which produce long-term strategic and operational benefits (Anandarajan and Wen, 1999). Oliver (2007) pointed out by including intangible benefits of the project in the evaluation; a better understanding of the true value of the project will emerge.
Providing a strategic analysis to the CEO by using methods such as the balance scorecard or a ranking/scoring system chosen by the stakeholders who will be using the system would enable a broader perspective of the benefits of the system to be seen by the CEO. Weak justification of IT investments may be putting the competitive advantage of many companies at risk.
Thanks for your comprehensive reply ‘1rguru’. Just again on the subject, you recommend that the CEO be provided with a strategic analysis – where the investment with the best overall score in some ranking is chosen. According to Bannister & Ramenyi (2000) this is refered to as the positivist or reductionist approach. They also mention an alternative approach called ‘Hermeneutic’ where instinct and intuition play a big role in the decion-making. What are your views on this approach to decion-making in comparison to the positivist approach?
In my previous blog https://sopinion8ed.wordpress.com/2012/11/14/a-one-size-fits-all-approach-does-not-exist/ I discussed (Bannister & Remenyi 1999) where approaches to evaluating IT decisions are classified into three basic techniques that can be used in two different ways. Providing a strategic analysis to the CEO by using methods such as the balance scorecard are referred to as ‘composite approach’ which combines several fundamental measures to get a ‘balanced’ overall picture of value/investment return. The CEO now has two options of using the results in a positivist way as you stated the best project is chosen based on facts and results. Alternatively the strategic analysis could also be interpreted in a Hermeneutic view, again as you stated using your instinct and intuition. I believe that a Hermeneutic view of using ones ‘gut feeling’ has a role to play in choosing an IS investment. i.e when betting on a horse a person will look at the data about the horse like its form and favourite ground etc however a person may also consider things that don’t relate to the chances of the horse winning like its name and colour and then makes a bet based on all these elements.
Just to clear up my blog https://sopinion8ed.wordpress.com/2012/11/19/different-evaluation-measure-for-different-types-of-investments/ is pointing out a trend in the different types of evaluation measures used based on the type/purpose of the investment. It is not a framework for choosing an evaluation method. On my previous blog: a one size fits all framework does not exist rather a custom evaluation method should be developed.
Interesting analogy comparing the IS investment decision with that of choosing a winning horse. Good example, however, I would also include how horse looks in the pre-race paddock inspection that would evoke stronger levels of instinct and intuition. While name and colour are rather random insignificant elements, negative signs of lameness or nerves; or positive signs of prancing, neck bowed and focused look are indicators more closely linked to instinct. I would believe name and colour to be elements chosen by a novice as a guess. Remember, as Malcolm Gladwell in his book ‘Blink’ says, first comes knowledge and experience, then instinct. And it is the negative and positive signs I’ve indicated above that would shape a betters decision, whether consciously or sub-consciously.
Your other point about how “a one size fits all” approach does not exist is an excellent point and an excellent post, I found it very informative. Indeed a custom evaluation method is the approach to follow.