Making Training More Accountable
By Patricia P. Phillips and Jack J. Phillips

Because of heightened skepticism in reporting, it is more important than ever to develop a training return-on-investment methodology that will stand up under intense scrutiny. The ROI methodology must meet certain operating standards to help ensure that there is consistency in the evaluation process and that a conservative approach is taken. Standards and guiding principles keep the evaluation credible and allow for replication of the methodology.

When implementing training ROI, here are some guiding principles to use as operating standards.

Report the complete story: ROI is a critical measure, but it is only one of many levels of evaluation necessary to explain the full impact of a program. Once the program has been implemented, you should evaluate how participants reacted (including their perceived ability to put the training into action), the extent to which participants improved their knowledge and skill levels, how well people are applying the skills on the job, and finally the business impact. If measurements are not taken at each of these stages, it is difficult to conclude that the results achieved are actually a result of the training and performance improvement program.

Enhance credibility: When collecting and analyzing data, use only the most credible source. Credibility is the most important factor in the measurement and evaluation process. Without it, the results are meaningless. Using the most credible source (often the participants) will enhance the perception of the quality and accuracy of the data analysis and results.

Be conservative: When analyzing data, select the most conservative alternative for calculations. This principle is at the heart of the evaluation process. A conservative approach lowers the ROI but helps build the needed credibility with the target audience. It is always better to be conservative than to provide a generous estimate and have results that are not credible.

Account for other factors: At least one method must be used to isolate the effects of the program. This step is imperative. Without some method to isolate the effects of the program, the evaluation results will be considered highly inaccurate and overstated. Some commonly used strategies include:

• A pilot group of participants in a training program is compared with a control group not participating in the program to isolate the impact of the program.
• Forecasts of anticipated results without the training program are compared to actual post-intervention results.
• Participants estimate the influence a training program has on key measures of impact.

Account for missing data: Sometimes training participants leave the organization or change their job function. If training participants cannot or do not provide post-intervention improvement data, assume that little or no improvement has occurred. It damages the credibility of the evaluation to make assumptions about improvements for which no substantiating data exists.

Adjust estimates for error: It's common to use estimates in reporting financial and cost-benefit information. To enhance the credibility of estimated data, weigh the estimates based on the level of confidence you have in the data and adjust accordingly.

Omit the extremes: Extreme data items can skew results. To eliminate the influence of extreme data items, omit them from the analysis. For example, if you have a list of numbers that all range from 30 to 70 except for one instance of the number 100, the number 100 would be considered an "outlier" or extreme data item and should be eliminated.

Capture annual benefits for short-term programs: Only use the first year of benefits in the ROI analysis of short-term programs. If benefits are not quickly realized for most training and performance improvement programs, they are probably not worth the cost. Therefore, for short-term programs, consider only annual benefits. Reserve multiple-year ROI analysis for more extensive programs where implementation spans a year or more.

Tabulate all program costs: The ROI methodology must include all of the costs associated with the training and performance improvement programs. These costs include the initial needs assessment; development; delivery costs including facilitator, facility and participant costs; opportunity costs associated with employees being absent from their jobs during training; and evaluation costs. Although the term ROI has been used loosely to express any of the benefits of a training and performance improvement program, a credible ROI methodology includes monetary costs. Omitting or understating costs will destroy the credibility of the ROI results.

Collectively, these guiding principles will ensure that the ROI methodology is credible and that it produces accurate values and consistent outcomes. It also ensures that the impact study can be replicated--when two or more practitioners evaluate the same program, they should always result in the same measurement.

Patricia Pulliam Phillips is chairman and CEO of The Chelsea Group, a research and consulting company focused on accountability issues in training, human resources, and performance improvement. Phillips is the author of numerous books and articles on training ROI including, most recently, The Bottomline on ROI: Basics, Benefits, & Barriers to Measuring Training & Performance Improvement (List Price $16.95, 117 pages, 1-879618-25-7, CEP Press, 2002). This new release provides a "CliffsNotes^TM" view of training ROI for individuals who require a basic understanding of the issues surrounding this important topic.

Jack J. Phillips, Ph.D., is with the Jack Phillips Center for Research, a division of Franklin Covey. He is an expert on measurement and evaluation and is the author or editor of over 30 books and 100 articles.

This article appeared in Workforce (www.workforce.com), September 2002.