post-implementation evaluation matters too

5 Nov

“Generally speaking, most empirical and theoretical articles (with very few exceptions) tend to classify IT/IS evaluation as a planning activity or take a temporal view along the development life cycle only to stop short of the operational phase.” (H. Al-Yaseen , T. Eldabi , R. J. Paul and R. El Haddadeh 2008)

(Irani and Love 2008) discussed the evaluation process by extending the temporal view of the systems life cycle with more concentration put on the operational phase – in order to understand issues related to IT/IS evaluation after project completion.

“Evaluation is a process that takes place at different points in time, or continuously, explicitly searching for (quantitatively or qualitatively) the impact of IT/IS projects” ( Al Yaseen 2006 ). This definition recognizes the different stages in the full life cycle of an Information system in which evaluation is performed.

First view evaluation which can be referred to as ‘ex ante’, ‘formative evaluation’ or ‘prior operational use’ (POU) is used to gain direction in the IS project. Here quantitative techniques such as IRR and NPV can be used. This evaluation doesn’t provide feedback beyond design, implementation and delivery of the project outcomes.

Evaluation can also be considered in terms of the effectiveness of the IT/IS system. What a system actually accomplishes in relation to its stated goals (Eldabi 2003). This form of evaluation draws on real data rather than projected data, and can be used to justify adoption (Love and Irani, 2001). This type of evaluation should be performed during the operational phase of the project. This is referred to as ‘post-implementation’ evaluation (PIE).

The below diagram from (Irani & Love 2008) shows these forms of evaluation with respect to a systems life cycle.

Forms of evaluation with respect to a systems life cycle.

Results from (H. Al-Yaseen , T. Eldabi , R. J. Paul and R. El Haddadeh 2008) suggest that most decision-makers do not place much importance on PIE of their IT/IS systems. “Most managers tend to think of it only as a formality rather than a proper evaluation process. Among the 45 who considered adopting PIE, those companies who undertake it seriously tend to gain considerable benefits, including the validation of their original POU estimates. But more importantly PIE helps those organisations to better appreciate and capture the intangible benefits associated with IT.”

The results of the research indicated that all the participating organisations have carried out a formal POU evaluation, but that only 36.5% currently perform a formal PIE of their IT/IS systems. That suggests only 36.5% of the participating organisations justified their investment in IT/IS and if their IT/IS systems achieved the anticipated benefits. (Irani & Love 2008)

Evaluating an IS investment after the implementation stage makes sense to me as it can be seen how the new system is progressing and can give organisations a more comprehensive evaluation of the investment. After the implementation stage a system may act as an enabler into new products and services that may not have been taught of in the ‘prior operational use’ stage of evaluation.

Do people think that to evaluate the full benefits and cost of an information system investment that using both POU and PIE would be a more comprehensive method of evaluation?


6 Responses to “post-implementation evaluation matters too”

  1. ronnoc90 November 5, 2012 at 8:52 pm #

    The benefits that could be gained as a result of both a POU and PIE evaluation are explicitly obvious. I find it hard to believe that more organizations don’t adopt this method of analysis. Surely it makes sense to return to a project and analyse the post implementation period. Only through evaluation can organizations determine the strengths, and weaknesses of the information system, and ultimately plan for the future.

    • 1rguru November 6, 2012 at 11:34 am #

      Hi ronnoc90, thanks for your post of which I agree. Why don’t more firms perform PIE? Irani & Love (2008) suggest the reason that companies omit the PIE evaluation is because “no interest group is charged with assessing the value of the IT project over its entire life cycle (from inception to decommissioning) which would therefore include PIE.” So evaluation stops when the system becomes operational because the self-contained and budgeted project has then ended. After completion there is nothing else to do.

      “Currently, only organisations who perform serious PIE understand the benefits of it. And there are not many of these, so very little analysis exists on planned costs and actual costs (or benefits).”

      • ronnoc90 November 10, 2012 at 9:22 am #

        Thanks for the reply! It would be interesting to see the results these companies have achieved through the implementation of PIE. Given the current economic crisis, now more than ever it would be in the interest of the organization to assess costly investments post-implementation. Thought provoking post!

  2. sully1210 November 7, 2012 at 9:52 pm #

    Very interesting blog 1rguru, and I agree. Certainly evaluating IS investments should be done at a number of stages, both before and after implementation.

    I think POU and PIE definitely have potential to be very comprehensive measures of IS investments. Especially because they combine both financial and real data elements.

    I think as well that an investment needs to be constantly evaluated. I found it interesting that you mentioned that ” a system may act as an enabler into new products and services that may not have been thought of in the ‘prior operational use’ stage of evaluation”. I think this highlights the need for a number of different evaluation techniques within one method.

    I also agree with your reply above, business environments are changing constantly, with new competitors entering markets and a number of economic risks still being present. For companies to think that once an investment is evaluated that is it, is a really big risk to take. Constant evaluation is necessary.

  3. mcoconnell November 13, 2012 at 2:51 pm #

    Another major benefit could possibly be that PIE could provide a significant learning opportunity for subsequent implementation efforts. Surely it would be easier at the post-implementation stage, particularly if it was done formally, to identify the stages where the team did well or poorly.

  4. kechy4me November 30, 2012 at 4:48 pm #

    Post implementation evaluation matters. The importance of post implementation reviews enables an organization to learn from experience, both good and bad, and become more successful with their IS/IT investments.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: