Archive | Evaluating Information System Investments RSS feed for this section

Business Performance: An Introduction

15 Jan

This series of blogs will examine how the contribution of IS investments affects the overall performance of business. We will commence by taking a look at various definitions of Business Performance and Business Performance Measurement Systems.

A business performance measurement system refers to the use of a multi-dimensional set of performance measures for the planning and management of a business” (1)

BPM systems enable an enterprise to plan, measure, and control its performance and help ensure that sales and marketing initiatives, operating practices, information technology resources, business decision, and people’s activities are aligned with business strategies to achieve desired business results and create shareholder value.” (2)

Professor Eric Brynjolfsson of MIT and Professor Lorin Hitt of University of Pennsylvania co-wrote a paper on the effectiveness of IT on business performance (3). These scholars strongly believe that IT investment is essential to better the performance of firms. They argue:

That a significant component of the value of information technology is its ability to enable complementary organisational investments such as business processes and work practices. These investments, in turn, lead to productivity increases by reducing costs and, more importantly, by enabling firms to increase output quality in the form of new products or in improvements in intangible aspects of existing products like convenience, timelines, quality and variety.

The professors strongly believe that without the existence of IT and IS, other fields would fail to flourish.

Firms previously used financial figures to assess business performance but in recent years complementary approaches have been developed taking many other factors such as market share into consideration to deliver a more accurate measure of a firms’ performance. The most popular of these approaches was developed in the 90’s at Harvard Business School in the form of Balanced Scorecards. The Balanced Scorecards approach takes a range of features into account which makes it a good vehicle for structuring an array of performance measures. (4) They can be defined as:

A balanced scorecard is a comprehensive set of performance measures defined from four different measurement perspectives (financial, customer, internal, and learning and growth) that provides a framework for translating the business strategy into operational terms.”  (5)

This approach will be discussed in detail further on in the series.

IT investments not only lead to improved business performance in factors such as increased productivity but also through the use of Business Intelligence to analyse a firms data, which will also be examined in upcoming blog posts.


(1)    Bourne, M. C. S., Neely, A. D., Mills, J. F. & Platts, K. W, (2003), “Implementing performance measurement systems: a literature review”, International Journal of Business Performance Management, Vol. 5, No. 1, 1-24.

(2)    Maisel, L.S. (2001), Performance Measurement Practices Survey Results, AICPA, US.

(3)    Brynjolfsson, E., & Hitt, L. M. (2000). Beyond computation: Information technology, organizational transformation and business performance. The Journal of Economic Perspectives, 23-48.

(4)    Neely, A. (2002). Business Performance Measurement: Theory and Practice, Cambridge University Press.

(5)    Kaplan, R.S. and Norton, D.P. (1996), “Linking the Balanced Scorecard to Strategy (Reprinted From the Balanced Scorecard)”, California Management Review, Vol. 39, No. 1, pp. 53-79.



Is IS evaluation different to evaluation of other investments?

29 Nov

Taking into account all the various blogs and viewpoints in relation to the evaluation of information system investment it’s fair to say that the process from the proposal stage to the outcome result is a complex, multidimensional and interdependent process. As a result of its diverse nature, the approaches and methods have gone under scrutiny in the academic environment. From the Delphi/VAHP/DEA, PENG Analysis, CCP and Bedell method we can see that there are various perspectives in the “how” part of the evaluation IS investment. Does this make it completely different to other types of investments?

We have seen from kechy4me blog “is IS investment different to evaluation other investments” that IT/IS investments acquire less benefits in contrast to other investments in that IT/IS investments require faster time response, more data and tighter integration which a manager would be less likely be able to quantify the intangibles. Mirra2 also supported similar viewpoints that the interaction between systems and the organization cannot be predicted with accuracy therefore it is different from capital projects where it can easily be predicted. I relatively agree with the above but only in relation to capital investments.

A capital investment being only one form of investment does differ from IS but there are other types of investments that share similar characteristics to IS investments. For example investments in stocks, shares and bonds have allot if not more risks involved in such an investment in comparison with IT/IS. In order to quantify these intangibles and predict future outcomes require an in depth knowledge of the business itself or even the economy in order to be successful, combined with even a faster time response and more data. In fact there are few investments that involve zero risk including capital investments (e.g. investments in housing areas which become “ghost estates”).

The IT/IS environment may involve more uncertainty than other business sectors in that there are so many new dimensions opening up year after year in which IT/IS managers have to operate in. However the fundamental characteristics are still the same. In contrast with investments in stocks/shares, there is a high degree of risk. After the assimilation of information a manager still has to make a final decision on whether to invest or not. Sometimes he/she will eventually have to go by their “gut instinct” as the outcome is unknown. This is also a similar trait to other types of investments. We can see here some of its similarities with other investments, but it is important to note it differs very much in its approaches and its methodologies in the evaluation of IS investments.

The role of ‘gut instinct’ in IS evaluation

28 Nov

According to Bannister and Remenyi (2000), Gut feeling, Instinct or intuition is often a different and subtler kind of reasoning; it takes into account of how the world really is rather than simply what the spreadsheets say. They described that as the limitations of investment  appraisal methods become more evident, decision-makers are falling back on ‘gut feel’ and other non formal/rigorous ways of making IS investment decisions.  They went ahead to say that due to the complexity of the determination of costs and benefits and the difficulty to predict the future success of new technology or new business models and the competitive reaction, top management is asked to commit to those strategic projects as an act of faith. Gut feeling sometimes called ‘vision’ or ‘strategic insight’, replaces qualitative analysis.

Many researchers that have investigated on the practice of IT investment decision making have also found that, when it comes to very complex decisions, managers often rely on methods which do not fall within the traditional boundaries of so-called rational decision making.

Leimeister et al (2008) also pointed out that although a significant amount of work is put into formal evaluations the investment decision itself is often not based on the formal results. They also said that corresponding to other research by (Bannister & Remenyi, 2000; Fabrey et al, 1999) they observed that many decisions are justified as ‘acts of faith’ and the reason behind that could be that, the executives mistrust towards benefit evaluation methods or generally an inconsistent understanding of how IT value is defined.

Also Mercken (2005) said that in a lot of cases the purposes of the evaluation is not to choose in a perfect positivist view the best solution out of a limited list of alternatives, but rather to justify a project that has already, on a basis of gut feeling been likely to be the best choice.

When managers are not relying on their intuition or instincts, studies of decision practise have indicated that managers frequently fall back on a relatively techniques involving some variety of relatively simplistic cost benefit analysis (Willcocks and Lester, 1994; Ballantine and Stray, 1998)

Irrespective of the evaluation approach that has been taken to the IS/IT investment analysis, in due course a decision must be made and instinct plays a role because it is a central part of decision making.

ronnoc90, 1rguru and Mirra2 have also explained on their blogs the importance of the balance of gut instinct and rational decision making in evaluating investment decisions.

Two perspectives on Risk Management

28 Nov

For my final blog I decided to follow on from lucid21’s previous blog post and focus on some videos related to risk management.

Analysts Take: IT Risk Management

This short video is particularly interesting and describes two Gartner Analysts take on IT Risk Management. Lucid21 successfully defined the meaning of the term risk. In this clip Richard Hunter states that “at this point in time if an enterprise has between 50 to 60 enterprise level risks, something like 5 to 10 of them will be IT risks”.  He goes on to say that those IT related risks feature prominently in a wide number of business decisions. These can range from how customers are served to how the company enters new markets. (Ricard Hunter, 2009).

The best thing an organisation can do today to address risk management is to get a good formalized risk assessment process in place.Lucid21 identified a number of ways of doing this. Many companies do not take risks into account and this makes it extremely difficult for them to make the right investment decisions and evaluate their investments effectively (Paul Proctor, 2009).  While identifying risks is essential, it is also important that companies accept that some degree of risk will be associated with an investment. Without taking some form of risk IT investments would not create any value for an organisation. (Paul Proctor, 2009).

Does risk matter? Disengagement from risk management in information systems projects

In this video Elmar Kutsch (2012) describes a risk as being endemic, present and has the potential to derail the success of any IS project. Risk needs to be proactively managed. A wide range of frameworks and tools are available to project managers in order to actively manage risks in advance of the risk materialising. However project managers are often disengaged from this rational and sensible process of managing risks (Elmar Kutsch, 2012).

The risk management process that is being advocated by major professional organisations is a rational, beneficial and sensible approach that project managers should interpret. Lucid21 touched on a number of these in his previous blog posts also. Unfortunately the speaker in this video has found that at any stage of the risk management process, the process itself tends to fall apart. People become disengaged and ultimately no action is taken.(Elmar Kutsch, 2012).

Managers see risks as a fictional entity, something that hasn’t happened yet and might not happen. As a result while most risks are known to the project managers, they are not responded to.  Project managers then end up waiting until this fictional piece of information turns into reality.(Elmar Kutsch, 2012).

What impact does disengagement have on the quality of projects and their evaluation?

Disengagement has an adverse impact on the quality of an IS project.  If a risk management process is not followed through it makes the company more vulnerable to a risk taking place. What Kutsch has found is that while managers disengage from this process they are actually very efficient at managing the resulting risks quickly and appropriately.(Elmar Kutsch, 2012).

How could this be addressed?

Kutsch presents an alternative paradigm for risk management.  He claims that in order to deal with disengagement, one first needs to understand the reasons why project managers disengage from the process in the first place. Instead of focusing on predicting what might happen and preventing that from occurring, Kutsch suggests that companies put in place the necessary processes to deal with the impact of any potential risk. This ensures that if risks do occur they are manageable from the onset.(Elmar Kutsch, 2012).

Risk management and evaluation of IS projects go hand in hand, as all elements of a project ultimately need to be evaluated. These video’s show the complexities involved in managing, preventing and evaluating risk.

An integrated IT/IS investment evaluation method

28 Nov

Despite the vast array of methodologies present in the literature for evaluating IT/IS investments some argue that a comprehensive or integrated approach is needed (Azadeh et al., 2009; Gunasekaran et al., 2006). While Gunasekaran et al. have advocated such an approach and laid some of the groundwork to achieve this through a literature review (, Azadeh et al. have developed an integrated framework which they argue is comprehensive and robust enough to evaluate and justify IS/It investments by all types of organisations.

Given the complexity involved in IT/IS implementation evaluation and the range of stakeholders involved, the framework developed by Azadeh et al. (2009) includes a number of important characteristics:

1)      Aggregates stakeholders’ opinion regarding the different criteria involved in the problem.

2)      Weighs and scores different criteria from the viewpoint of various stakeholders.

3)      Aggregates stakeholders’ opinion regarding alternative IT/IS investments.

4)      Determines the efficiency of alternative investments.

In order to take account of these characteristics the authors integrated three different frameworks (VAHP, Delphi, and DEA) to develop one that would be of use to different types of organisations. Each tool is utilised to deal with different stages of the evaluation process.

The integrated framework involves different stakeholders and can analyse both tangible and intangible benefits, costs and risks, as well as quantitative and qualitative data. Despite involving analytical and mathematical techniques, the authors maintain that it is simple enough to be understood by different stakeholders in the organisation (Azadeh et al., 2009).

Evaluating the risks of information system investments

27 Nov

When a company decides to invest, most often an element of risk is involved. Risk on its own is a complex term therefore we must define it before we can evaluate the risk of IS investments. The meaning of the term risk has been fraught with confusion and controversy (Watson & Hope, 1984). Willcocks & Margetts (1993) refers to risk as being exposed to such consequences as failure to obtain some or all of anticipated benefits due to implementation difficulties. However in order to account for the risks involved in an investment it must be measured/evaluated.

There have been many frameworks formed to assist risk assessment in IT projects. According to Cash et al. (1992) the magnitude of risk involved is influenced by three dimensions within the IT project.

1)      Project size: include costs, number of staff involved, estimated time, the number of company partnerships involved.

2)      Technology Experience: Newer technologies are more prone to failure rather than tested ones.

3)      Project Structure: If a project is well structured and has fixed outputs that are not subject to change acquires less risk.

Another framework that evaluates the risks of IS investments is the Parker framework (1988). It evaluates five risks which including the following:

1)      Organizational risk- how well equipped the company is in terms of implementing an IT project (e.g. experience and skill sets)

2)      IS infrastructure risk- Evaluating the needs required prior to investment in the project.

3)      Definitional uncertainty- Evaluating whether the specifications of the project are known or not known.

4)      Competitive Response- evaluating the risk of not investing in the project (the opportunity cost).

5)      Technical uncertainty- evaluate whether a project is dependent on untested technologies (Willcocks & Margetts, 1993).

Both Parker and Cash et al. frameworks is quiet broad. A more detailed and analytical framework is developed by Pettigrew (2001). The framework highlights the risks involved in the development, introduction and use of IS by examining six conceptual categories.

1)      Outer Context- External factors will have to be evaluated such as political, economic, market and competition factors.

2)      Inner Context- Analysing the structure, management, company relations and human resources.

3)      Content- Evaluating the proposed changes such as the technologies size and its characteristics.

4)      Processes- Calculating the “how” part of implementing a new project or technology and solving perceived issues.

5)      Outcomes- Include both the evaluation of anticipated and unanticipated outcomes assessed in the IS project such as costs, time, utility and the performance of the project.

According to Willcocks and Margetts “computerisation remains a high risk process but the degree and type of risk is influenced considerably by environmental and organization contexts and pressures”. Similar to the methods of evaluating IS investments, there is no one method to evaluate the risks involved in such a decision. When an IT/IS investments involves many risks, there are potentially many “ways to reconfigure the investment using different series of cascading (compound) options” (Benaroch 2002).

Towards a generic IS implementation evaluation?

26 Nov

A number of blogs have already identified that a single methodology is not adequate to evaluate an IS investment (;, while others have outlined different methodologies that are in use (; Yet, some other important evaluation methodologies in IT/IS have not been mentioned. Information Economics, for example, is a variant of cost benefit analysis but it is structured to incorporate intangibles and other non-financial measures in IT/IS investments. Real Options Valuation seeks to place a quantifiable measure on flexibility, helping to gauge the risks involved in an investment having considered that business strategy and system requirements are subject to change. A final example is Multi-Objective Multi-Criteria which highlights the presence of different viewpoints and values among various stakeholders in an organisation. The technique does not employ monetary measures but instead employs a procedure that establishes preferences or utilities. The best investment is determined as that which provides the highest aggregate utility or the highest overall calculation of satisfied preferences (Ranti, 2006).

In the face of such an array of evaluation techniques, one procedure for choosing the appropriate methodology has been put forward on this blog. Using Remenyi (2000), 1rguru demonstrated that an appropriate method for choosing the correct methodology was to ‘relate the evaluation measure to the investment purpose’ ( Gunasekaran et al. (2006) maintain, however, that the need exists ‘for a complete and integrated framework for IT/IS justification to guide researchers and help practitioners understand their full range of choices.’ They developed a framework with the objective of identifying key areas of importance for successfully evaluating investment decisions from the relevant literature. Drawing from this knowledge they confidently proposed in the future ‘to produce a complete, efficient and effective methodology for justifying IT/IS on a small or large scale’ (Gunasekaran et al., 2006). Consequently, future research may develop a generic approach to evaluating IT/IS investment.

Implications of IS value

26 Nov

In my previous blog I stated that; it depends on how the company perceives the value of IT/IS that will determine the evaluation method to use. As Lucid21 points out in his last post one problem facing IS evaluation methods is defining the term “value”.

Cronk and Fitzgerald (2002) demonstrate how the many evaluation measures that have been employed stem from differing philosophical paradigms and interpretations of the term IS business value.

However what is the value of IT/IS? Bannister and Remenyi (2000) pointed out, to be able to assess value; one has to clearly define it. Parker and Benson (1988) stated that IT value is the ability of IT to enhance the business performance of the enterprise.

Brown (2005) stated that it is the benefits of an IS investment that constitute the business value that have proved the most challenging to assess. Organizations invest in and maintain ICT (information and communications technology) applications to create business value. They expect benefits like productivity gains or quality improvements that contribute to the organization’s strategic goals or operational performance. Evaluation exercises are concerned with the question of value. There has been comparatively little attention paid to the definition of IS business value (Cronk and Fitzgerald, 1997).

“Without a clear understanding of this concept estimation of benefits and hence evaluation are unlikely to contribute much to our understanding of the impact of ICT.” (Brown 2005)

Both the actual and potential value of each IS investment will vary from situation to situation and between stakeholders.

Orlikowski (1999) states that; “Technology is not valuable, meaningful or consequential by itself; it only becomes so when people use it.” Davenport (2001) uses examples of organizations that have installed advanced information systems to capture transaction data but have yet to make effective use of it.**

To investigate business value Orlikowski (1999) and Davenport (2001) focus on the specific information systems applications and the organizational context within which they are used. Business value is de facto considered a function of the individual information systems application and the particular context within which it is of applied.

To conclude IS evaluation exercises can also be viewed as projects that involve both the hard factors of evaluation of the technology and soft factors like evaluation of the organizational impact of the technology and organizational processes for decisions. People’s different take on the value of IS will lead to different evaluation methods, customized to capture the perceived value of the investment.

**Davenport Examples: Davenport T, Harris J, De Long D and Jacobson A (2001) ‘Data to Knowledge to Results: Building an analytic Capability’ California Management Review vol 41(2) p117-138

Problems facing IS evaluation methods

24 Nov

As pointed out in my last blog “the evolutionary nature of IS evaluation methods” there are many difficulties facing these type of methods due to the diversifying nature of the economy and its multidimensional composition which alters the current methods. In this blog I will highlight the problems that currently face IS evaluation methods.

We have seen from 1guru “no 1 method” due to the multidimensional complexity of evaluation methods there are complications in choosing a method. There is allot of difficulty when it comes to the evaluation of IS, with companies being unable to quantify the full implication of their current IS infrastructure (Irani 2001). There is allot of standardized literature to support that there is difficulty with IS evaluation however there is allot less empirical research that indicates this mainly due the fact that few companies are willing to publicize their difficulties and failures.

Some of IS evaluation complications consist of the following:

  • Understanding the term “value” and its multidimensional aspects
  • Identifying, managing and controlling both direct and indirect investment related costs
  • Analysing the risks of investment relating strategies
  • The ability to provide adequate technology management resources
  • Assessing the nature of IT/IS intangible, tangible, financial and non-financial benefits
  • The ability to enable a better “technological-fit” of information systems
  • and understanding the human and organization structure of investment decision making within companies

According to Hochstrasser (1992) the reason for a high fail rate in IS/IT failure is partly due to the lack of solid but simple and rather vague management tools in evaluating, prioritising, monitoring and the controlling of IS investments. Griffith and Horchstrasser (1991) claim that many companies are faced with expired and unsuitable techniques in evaluating information systems leading to irrational decision making.

This also refers to my previous blog in relation to the evolutionary nature of IS where methods require up to date enhancements… “ it is necessary to recognize that organizations need to tailor existing evaluation methodologies to incorporate technology-related flexibility” (Irani 2002, pp. 13)

The ArchiValue Project

23 Nov

Whilst commenting on a post made by Lucid21 a few weeks ago, I came across the term Enterprise Architecture Valuation. Enterprise Architecture can be defined as, “Design or ‘blueprint’ of a business that depicts the components of a firm employed in its operations, interrelationships of those components, information flows, and how each component supports the objectives or the strategy of the enterprise.” (Business Dictionary: 2012). The concept was explored by Dutch authors Buschle, and Quartel, (2011), along with an extension of the Bedell method as a means of evaluating investment in IT. I examined the article as I found their critique of Bedell’s approach interesting, however I wanted to find more information on architecture valuation.

My research led me to the work carried out by Dutch Company, “Novay”, who have proposed the , “Archivalue project”, which they provide as a service to companies who wish to evaluate investment in IT. The project offers a, “concrete method for IT portfolio valuation”. The company put forward the case for IT as a value centre and stress how viewing IT as cost has stifled innovation. Excessive budgets used to maintain legacy system has resulted in a failure to accept new IT projects that may facilitate current business practice or create new opportunities. Therefore Novay urge companies to see the different types of Value IT brings to a company. “Organizations can classify their business goals and decide which and how well IT contributes to these goals.” They offer the following as frameworks whereby investment in IT project can be evaluated.

Figure 1.1 “The ArchiValue Framework”


“The bottom layer represents the usage of Enterprise Architecture as a foundation for valuation.” The first column represents how the “EA” currently supports the business. The third column represents the desire future, in which the IT which supports the business activities are improved. Projects are carried out to reach this to-be column, through doing this “the value of IT is increased”.

“The middle layer represents the valuation of IT and project portfolios.” Based on the value given to current projects decisions can be made to align IT with the strategic goals of the company. These result in a change to existing portfolio which changes the portfolio of the future, Portfolio to-be.

“The top layer represents the usage of a valuation profile that reflects the business strategy.” Each value centre (Service, Investment, Cost, Profit) is associated with one or more business goals whereby a valuation technique can be created. “Depending on these criteria different valuation techniques may be selected to analyse IT with respect to these criteria. Distinct IT budgets may be allocated to each value centre and the specific valuations that result from the analyses.”

Figure 1.2 “Project Portfolio Management”


Figure 1.2 displays ArchiValue’s approach to management. According to the framework, the approach consists of two stages, leading toward IT portfolio valuation. It follows the Deming cycle, which can be defined as, “A continuous quality-improvement model in which a sequence of the four repetitive steps (plan, do, check, and act) comprise a feedback loop that allows a business manager to identify and correct deficiencies.” (The American Heritage Dictionary of Business Terms: 2010) As you can see from the diagram the first stage is concerned with IT portfolios. Following the Deming cycle, the valuation profile is planned by defining goals, valuation criteria, and performance indicators to evaluated progress. Under the ‘Do’ step the performance indicators are measured . They are then checked against the desired or ideal values that have been defined in the valuation profile. the results determine actions that will be taken to bridge gaps. “Typically this involves the definition of change goals and the design of a ‘to-be’ situation with higher valuation scores.”

The second stage shows how to manage the implementation of the proposed changes. The plan stage valuates the proposed projects. The projects are introduced and the results of the implementation of these project are checked against the goals that were laid out in the planning stage.

The ArchiValue project offers a lot by way of evaluation and support to Organizations.  Do you think this model achieves it goal of helping IT managers and architects to get a grip on the costs and benefits of IT? Do you think this model is an effective way of measuring IT value? Should this valuation be carried out to determine whether or not to invest in IT that may bring about new business opportunities?

Link to full information leaflet:

Photo Credits: Novay

%d bloggers like this: