CEP Press goes 3 for 3! In its first full year as a full-service publisher, CEP Press has earned the International Society for Performance Improvement's prestigious Award of Excellence for Outstanding Instructional Communication on all three of its 2001 titles: The Business of Winning, Making an Impact, and Conquering Organizational Change!

PERFORMANCE CONSULTING: THE BEGINNING OF THE ROI PROCESS

By Ann W. Parkman, Executive Vice President, CEP, & Perry Anne Scott, General Manager of Air Logistics Learning Organization, Delta Air Lines

Accountability is a matter of growing importance in our industry. According to ROI expert Patricia Pulliam Phillips, "increasingly, executives require the training and performance improvement function to show the value it brings to the organization in the same terms as other operational elements" ("The Bottomline on ROI: Basics, Benefits, & Barriers to Measuring Training & Performance Improvement," CEP Press, 2002).

In order to do a good job of evaluation at the end, however, it's important to lay the right groundwork from the beginning. Conducting up-front analysis before engaging in a training or performance improvement solution offers a number of benefits:

• It ensures that the projects you undertake will have real value
• It forces you to clarify your goals in terms of the business results you want to impact and the job performance changes you need to make
• It ensures that the solutions you implement are based on data, not intuition
• It helps to secure needed buy-in from key stakeholders, who will be instrumental in helping you complete your analysis
• It provides meaningful data for use in comparing pre- and post-intervention performance
• It enhances the credibility of your results

Linking Analysis with Evaluation
There are five generally accepted levels of evaluation. These include Kirkpatrick's four evaluation levels-reaction, learning, job application, and business impact-and Jack J. Phillips' fifth level-ROI. Similarly, there are five levels of analysis that link directly to each level of evaluation.

1. Feasibility analysis (which links to ROI evaluation): Is the project worth doing? Are the benefits of the solution likely to outweigh the costs?
2. Business needs analysis, which links to business impact evaluation): What business needs are not being met?
3. Job performance analysis (which links to job application evaluation): What elements of job performance are prohibiting the achievement of these business needs?
4. Learning needs analysis (which links to learning evaluation): Assuming the performance problem is due to a skill or knowledge deficiency, what do people need to learn in order to perform proficiently?
5. Preferences analysis (which links to reaction evaluation): What instructional methods and media will best meet the needs of both learners and management?

The following case study describes how CEP and Delta Air Logistics (DAL) applied the different levels of analysis to develop a true, results-oriented training solution.

Delta Air Logistics: A Case Study
DAL is the cargo arm of Delta Air Lines. While not as well known as its passenger service, DAL currently is responsible for one-third of Delta's net revenue, with tremendous opportunity for growth. For this reason, Delta has recently committed to investing in both DAL personnel and training. This case study focuses on one of these training initiatives-Acceptance Agent new-hire training.

Acceptance Agents (including full-time DAL employees and, in smaller facilities, external contractors) are responsible for accepting cargo and processing shipment requests. Until Delta's investment in DAL training, no analysis had ever been done to determine the needs of new agents. Over time, this lack of analysis resulted in a large discrepancy between DAL's business needs and its existing new-hire training. For example, international cargo represents a large percentage of DAL's business. Yet in the 40 hours of initial training, new hires received virtually no instruction in this area. This training gap resulted in a significant on-the-job learning curve and inconsistent work practices. It also contributed to a high turnover rate by new agents and frustration on the part of management and facilitators alike.

1. Feasibility Analysis
A feasibility analysis is particularly useful when: a) a lack of consensus exists over whether or not to undertake a performance improvement effort, or b) a project is likely to be very expensive. In this case, all key decision makers, including Delta and DAL's leadership team, as well as DAL's design team, strongly believed that the ROI of the new training initiative would prove significantly higher than that of the existing training program.

With respect to cost, CEP agreed to conduct the up-front analysis and design, and to serve as a quality advisor to expedite instructional development, implementation and evaluation by DAL designers. This served two purposes-it minimized the financial investment in the project while speeding the development timeline. For these reasons, DAL did not require a feasibility analysis in order to proceed with this project.

2. Business Needs Analysis
DAL identified a number of critical issues that were negatively impacting business results:

• The number of service failures resulting from incomplete paperwork and other errors was higher than acceptable to DAL
• This service failure rate was negatively impacting the level of insurance claims, regulatory fines and letters of investigation, as well as "pennies per pound" productivity (how much it cost for DAL to move a pound of freight)
• The attrition rate of new agents ranged from an estimated 20% for full-time staff to 100% for external contractors

Even before the start of the training initiative, DAL regularly tracked most of these business measures. This provides a solid pre-intervention baseline that DAL will be able to use to conduct Level 4 evaluation on the business impact of the new training program.

3. Job Performance Analysis
DAL operates 133 stations across the U.S., ranging from small- and medium-sized cities to large metropolitan areas. To gain a representative picture of the performance issues facing all Acceptance Agents, CEP interviewed and observed the supervisory staff, typical and exemplary agents and, where applicable, external contractors, in four of these cities-New York, Cincinnati, Birmingham, and Orlando.

The analysis focused on the specific business issues identified during the business needs analysis. At the same time, it addressed other potential performance issues, such as perceptions of rewards and punishment, feelings toward existing training, and impressions of the Acceptance Agent position overall.

The analysis uncovered a number of gaps between actual and required performance. The identification of performance gaps provides the necessary data to conduct Level 3 evaluation, as it offers a benchmark upon which to measure performance improvement.

The analysis also uncovered the reasons for the performance gaps, which ranged from skill/knowledge deficiencies to motivational and environmental obstacles. Determining the causes of the performance gaps ensured that CEP's recommended solutions would achieve their intended results.

4. Learning Needs Analysis
CEP established the learning objectives for the training program based on the skill/knowledge deficiencies uncovered by the job performance analysis. The completion of Level 2 and Level 3 analysis ensured that these learning objectives were tied directly to both the job needs of performers and the business needs of the organization. At the same time, the establishment of learning objectives will make it easy to answer the overriding question, "Will people learn what we set out to teach them?" (thereby satisfying the measurement criteria for Level 2 evaluation).

In comparing the existing training program to the recommended program, DAL's Learning Organization determined that the former covered only 18% of the critical skills and knowledge needed for Acceptance Agents to perform to DAL's expectations. The ability to precisely quantify the differences between the two programs enhanced the credibility of DAL's Learning Organization while increasing confidence in the approach being taken.

5. Preferences Analysis
While preferences should not drive any solution, being sensitive to the needs of both performers and managers is an important part of designing a program that will fit your target audience's work environment and management's time/resource constraints. While it isn't essential to perform this type of analysis in order to conduct Level 1 evaluation, considering and accommodating preferences can improve learner reactions to the training program overall.

With respect to DAL's Acceptance Agents, learner preferences were clear-minimize classroom-based training as much as possible. DAL's design team easily accommodated this request by developing materials that minimize stand-up lecture. In any given hour, lecture will comprise only 25% of training time, with the remaining time spent on hands-on activities, including ample practice scenarios and skill checks.

Regarding manager preferences, CEP found that many of DAL's supervisory staff expressed interest in career and skill enhancement opportunities. DAL plans to leverage this interest by giving qualified supervisors a chance to serve as "operations coaches." In this volunteer role, supervisors will receive training to enable them to provide structured mentoring and coaching to new hires.

Some training professionals may be reluctant, even afraid, to conduct comprehensive evaluation, particularly at Levels 3-5, because of what the results may or may not show. But by starting off with comprehensive analysis, you can maximize your chances of achieving positive and credible bottom-line results.

This article is based on a presentation of the same title delivered by Ann W. Parkman and Perry Anne Scott at ASTD's June 2001 Conference Within a Conference in Orlando, FL.

(A Brief Update on DAL's Training Initiative: The new Acceptance Agent training program was scheduled for rollout in the fall of 2001, but because of 9/11, implementation has been delayed and ROI results are not available. Nevertheless, both DAL and CEP are confident that the new performance-based training program will virtually guarantee an extremely positive ROI. As CEP President and CEO Dr. Seth Leibler states, "it's not uncommon for the ROI of true performance-based training to completely offset its costs. Linking analysis to evaluation simply makes good business sense.")

Periodically, we send out e-mails regarding special discount offers on workshops and books, as well as information about new products and services that will help you achieve positive bottomline results. If you would like to be added to this distribution list, please click here.

If you would like to be removed from our newsletter mailing list, please send an e-mail to info@cepworldwide.com with "Unsubscribe to Newsletter" in the subject field.