Exposing Common Misconceptions about Criterion-Referenced Instruction

Deep in the mists of history, three misconceptions were somehow born and seem to have grown stronger over time. These wrong-headed notions are:

1. All criterion-referenced instruction (CRI) is self-paced;
2. Self-paced instruction must be delivered in print;
3. Self-paced or group-paced are two mutually exclusive choices.

It's time to expose these misconceptions for what they are - obstacles in the path of efficient, effective training.

All CRI is NOT Self-Paced

First of all, nowhere in the CRI literature will you find self-pacing listed as a necessary characteristic of criterion-referenced instruction.

Second, criterion-referencing has to do with whether or not instruction is designed to accomplish objectives stated in advance of the development; pacing has only to do with how the instruction is delivered. Apples and oranges.

Third, a criterion-referenced course can deliver the instruction through any combination of media and procedures (delivery system) appropriate to accomplishing the objectives. The Mastering the Art of Instructor-Led Training Workshop, for example, is criterion-referenced, that is, it is designed to accomplish pre-specified objectives, but it is mostly group-paced. There is enough self-pacing to allow variable amounts of practice, but the nature of the objectives demands practice in a group situation.

Self-Paced Instruction Does NOT Have to be Delivered in Print

The misconception that all self-paced instruction must be delivered in print probably arises from the fact that most self-paced courses use print as the medium through which to direct students to the other media and activities (as in "Go watch Video C-7;" "Find a colleague with whom to _____.") But like the first misconception, the assertion that self-paced instruction must be delivered in print is also untrue. While print may be cheap and easy to maintain, and while print has no moving parts to get out of whack, it simply isn't always appropriate for the objectives to be accomplished. In the "Select Delivery System" module of the Instructional Module Development Workshop, one learns to select media that are most appropriate for both content and practice of an objective, best suited to the target population, least costly, and most available and easy to use and maintain. Nowhere is a pacing choice confused or entwined with the choice of media.

While it is true that some media control the pace at which students may progress (for example, video or lecturers), nothing prevents the use of these media within a self-paced course. Students who are occasionally directed to view a short videotape or to participate in a discussion when they are ready to do so are engaging in media-paced activities within a self-paced course.

Self-Paced or Group-Paced are NOT Mutually Exclusive Choices

In truth, self-pacing and group-pacing are the poles of a continuum, and most efficient and effective delivery systems make use of a combination of the two modes, depending mainly on the nature of the objectives to be accomplished.

For example, situations such as these suggest a need for group-pacing:

• Skills must be practiced as a group, as when a team of six must wrestle an airplane fuselage onto a trailer cradle, when football teams must practice executing their plays, or when orchestras practice their music.
• Skills must be practiced in the presence of a group, as when an individual delivers a practice presentation to others.
• The session is informational and/or motivational, rather than instructional, as in "awareness" sessions (when no change in behavior is prescribed or expected).

Other situations suggest a need for self-pacing, such as when:

• · Skills are practiced and/or performed individually (for example, interpreting balance sheets, writing reports, riding a unicycle).
• Skills are learned individually, even though ultimately used in group performance, such as learning a musical instrument, practicing a speech, throwing a ball, or making a sales presentation.
• Amount of practice required varies widely within the target population, as with interactive audio-spoken language lessons.
• The target population will have frequent or lengthy interruptions during training, as with most onsite (particularly managerial) training.
• There is reluctance (or it simply isn't possible) to wait to get a group together for training, as in today's slow-to-hire but impatient-for-competency environment.

Often, the best course is a combination of group-paced and self-paced components, of which the Criterion-Referenced Instruction Workshop is an example. Though primarily self-paced, group-paced sessions are included by design, and additional group sessions are planned by the participants themselves.

(Hint: If you are experiencing reactions against self-paced instruction, do a Performance Analysis centering around instructor behaviors. If you were an instructor with strong intuitive stand-up skills who hadn't yet been equipped for the performance-based environment brought about by CRI, you might be reacting unfavorably too!)

This article is adapted from the work of the late Marianne Hoffman, who served as a Mager Associate for many years.

(Almost) Everything I Ever Needed to Know About Effective Instruction I Learned through Correspondence Courses

By Robert F. Mager

The following article is excerpted from Robert F. Mager's newest book, Life in the Pinball Machine (CEP Press, Feb. 2003). This "story behind the story" offers a revealing and humorous look at the experiences and adventures that shaped the mind of one of the most influential people in training and performance improvement.

Long before e-learning, there was c-learning - otherwise known as the correspondence course. I've taken advantage of several over the years. In addition to the subject matter, each course taught me something about how (and how not) to structure effective instruction.

Practice makes perfect and other instructional observations about ventriloquism
Like most kids, I looked at the "Lemme out" magazine ads and fantasized myself a famous ventriloquist making all kinds of things talk. An opportunity finally presented itself when, in 1969, I bought a ventriloquial correspondence course and bought a used vent figure (if you call them "dummies" they'll find ways to get even) from a retiring ventriloquist.

The course wasn't badly constructed, but it was slow going. I'd sit at home with a puppet on my knee trying to practice, talking to myself, and feeling silly. While there was a source of immediate feedback for lip control practice (a mirror), without a video monitor for feedback, it was hard practicing to make the puppet move as though alive. Though my family said they did in fact hear the puppet talking, I didn't. I developed no confidence at all. Things only improved when I began practicing in front of our video camera and could watch the results on the monitor.

Later, at the urging of a ventriloquist I met at a Magicians' Convention, I entered their vent contest with a borrowed puppet I'd been introduced to only the night before. As there were only five contestants, they pleaded with me to fill in as the sixth. Frantic, I practiced in front of the bathroom mirror while improvising a short skit. (The act involved having stage hands push me onto the stage while I resisted, pretending to be scared mute. During this time, the exasperated puppet encouraged the audience to try to applaud me out of my "coma" and talk).

To my amazement, I won the contest and was more than mildly astonished when a psychiatrist approached me from the audience. He praised me for producing "…the best depiction of a catatonic I've ever seen." Gee, thanks a lot!

Even so, I still felt silly when practicing or performing. It was several years later that I found a book revealing the all-important bit of information I had been lacking: ventriloquists themselves can never hear the illusion! No matter how skillful they become at making the puppet talk, they can never avoid hearing in their own head the voices they're making for the puppet. Well, of course! It was suddenly obvious! Knowing that critical fact made it possible for me to develop confidence and quickly learn to act as though the voices were being produced by the puppets themselves.

But such obvious facts are often not obvious to the struggling learner. This experience gave me insight into the reasons why instruction offered by subject matter experts often is flawed and difficult to understand. People highly knowledgeable in a subject are very likely to leave out critical pieces of information when offering explanations. They don't deliberately withhold information; it's just that they learned those critical bits so long ago they're often unaware of what they're doing. It's best, therefore, always to question experts closely when using them as a source of information.

The proof is in the picking
By far the best course I've ever taken was a correspondence course on locksmithing. I found the ubiquitous ad urging me to "Be a Locksmith" in an issue of Popular Mechanics and sent away for the information. When the first lessons arrived, I immediately dived in.

Expecting the course to begin with dreary expositions on the history of locks, or key appreciation, I was pleasantly surprised to learn that lesson number one was on how to pick a lock. A combination lock was provided (the kind often used on high school lockers) along with a suitable pick, and within ten minutes of beginning the course, I was deeply engrossed in practicing a fascinating skill. At the end of the first lesson, I was able to exclaim, "Hey, lemme show you how I can pick this lock!"

A constant stream of lessons arrived in the mail, each of them task-oriented and beautifully orchestrated. I received a series of lock cores, with instructions to change the combination to the numbers given and return them to the instructor. I received a bag of combination locks with instructions to pick them open, tape them in place, and return them to the instructor. At one point I even received part of a car door and was told that the door-locking mechanism had a broken spring. I was instructed to make a new one, install it, and return it to the instructor.

During the lessons, time was taken out to learn what I needed to know (the "theory") just before beginning to practice. And, after each lesson I could do something important to the final goal I couldn't do before. The least interesting skills - paperwork tasks associated with the craft - were saved until the end, when the student could already smell the fresh ink on the diploma.

The mechanics of the course were so motivating I once found myself writing a letter to the instructor: "Your advertising promised that your mailings are scheduled so I'll never be out of lessons, but I've been out of lessons now for three whole days. Hurry up and send more!"

Truly a splendid example of instructional design and delivery. The entire course was built around the tasks to be mastered. Immediate practice of new skills led to growing confidence and eagerness to learn more. Feedback was built into the tasks being practiced (e.g., it was obvious whether you did or did not pick a lock open). Finally, theory was seeded in chunks just large enough to prepare for the practice to follow. In sum, the course was masterfully constructed to implement just about every principle of effective instruction.

Did it work? Did I learn anything useful? The big test came one day in a parking lot, when a colleague discovered he had locked himself out of his car. Having left my picks at home, I borrowed a toolkit, selected a couple of pointy things, and popped the lock open within thirty seconds. The satisfaction of that moment still lingers.

Take a stroll in your students' shoes
Periodically adopting the role of student has been a useful activity, heightening my sensitivity to obvious and subtle do's and don'ts of instructional design. Perhaps more importantly, an occasional stroll in the students' shoes reminds me of the vulnerability they often feel when faced with learning something new.

It was obvious through my experiences with the correspondence courses that the success of a course is not shrouded in the theatrical skill of an instructor, or the media through which it is presented. The magic is in the degree to which the principles of effective instruction are implemented - instruction leading to the ability to perform useful skills, that provides relevant practice and feedback, as well as lessons sequenced to motivate the student rather than to satisfy tradition or instructor preference. With those ingredients in the instructional soup, success is all but guaranteed.

About Robert F. Mager
Dr. Mager is a world-renowned expert on training and performance improvement. He is credited with revolutionizing the industry by creating the movement toward a performance-based approach to improving human performance. One of his most significant contributions is his development (with Peter Pipe) of the Criterion-Referenced Instruction (CRI) methodology. He has written ten books on issues relating to training and performance improvement that have sold over three and a half million copies worldwide.

For more information on Dr. Mager's newest book, Life in the Pinball Machine, click here.

How Blue Cross and Blue Shield of GA
Slashed Customer Care Agent Training by 90%
A CRI Success Story
By Ferdinand Tesoro, Director - Operations Performance Department, BCBSGA

When Blue Cross and Blue Shield of Georgia (BCBSGA) landed a large new account - a leading bank with over 30,000 members (and the potential for over 70,000 more nationwide) - the Operations Performance Department (OPD) knew it was critical to ensure that the open enrollment process would exceed the bank's expectations. We saw this new account as a golden opportunity to partner and collaborate cross-functionally with other key areas of BCBSGA not only to set the right expectations around customer service, but to reduce costs, increase productivity, and improve work processes.

Prior to using the Criterion-Referenced Instruction (CRI) methodology, we would have trained Customer Care Agents (CCAs) on what we thought they needed or "how we used to do it when we were on the floor." This open enrollment training for a new group would have been 40 hours or more.

Instead of diving straight into training development, we followed the CRI methodology and began the training project with upfront analysis. By employing questionnaires and face-to-face interviews, the project team discovered:

• What the CCAs' current job functions and areas of expertise were
• What would be required of CCAs to support the new account
• How much of what they currently know and do would continue with the new account
• What would be required of them that they had never done before

By performing analysis to determine what the CCAs already knew, we were able to create training that targeted only the knowledge and skill gaps, not covering any redundant or irrelevant information. As an added benefit, the CCAs felt ownership and input because they had the opportunity to tell us what they thought they needed to learn.

We also determined that, instead of requiring them to memorize large volumes of complex information or search through hardcopy documents, certain information should be readily accessible to the CCAs during their calls. So we decided to create a performance support tool - called a knowledgebase - for them to use. This Web-based, "just in time" tool allows CCAs to find the information they need in a few seconds with just one or two mouse clicks. It allows them to look up benefits for a certain product, quickly access phone numbers in the event they need to transfer a caller, refer to scripts for certain call scenarios, and complete other key tasks performed while interacting with customers.

The resulting training program reduced required training time to only 4 hours - a 90% reduction! We estimate that the savings from the 36 hours of lost productivity and associated reductions in overtime costs, multiplied across all 1,200 Operations Associates in Georgia, could result in a savings of $648,000!

Of course, the real test of a training program's success is in what learners can do after being trained. The CCAs felt extremely confident on the phones. This is reflected in an improvement in first call resolution metrics, a significant reduction in open comm. logs (unresolved issues not answered or calls that have to be escalated for further research), the percentage of calls placed on hold, and work time as a percentage of total average hold time. The following table compares the results of this project with a similar group that underwent a traditional open enrollment training process.

At the same time, feedback from key stakeholders has proven extremely positive.

Director of Operations, BCBSGA
"The overwhelming majority of the comments support what I have heard from associates - the training and the materials were excellent and they felt prepared for this critical account. I appreciate all the hard work the Training area put into making this a huge success."

A supervisor from the bank's business unit
"Just know that I'm a walking advertisement for you all and the benefit you provide…. We found the new training techniques (OPD) exposed us to refreshing and like nothing we had experienced in the past. I am excited my team was one of the first to benefit from this new approach."

CCA Training Graduate
"The training allowed us to be better prepared for the calls ahead… the internet training (knowledgebase) was exceptional and very beneficial."

In addition to the positive results of this program, it should be noted that the deliverables from this project now serve as a model for servicing open enrollments for future critical accounts. Having this model allows OPD to leverage these deliverables to reduce design and development time. For example, for the four strategic projects we have engaged in OPD since the bank project, we have used this project's deliverables as templates, thereby reducing development time for the Web-based knowledgebase and training by 25-50%.

Ferdinand Tesoro is Director of Operations Performance Department, Blue Cross Blue Shield of Georgia, an independent subsidiary of WellPoint. He has a team of performance consultants, instructional designers and trainers who has started applying the CRI approach to training with positive business results.

For more information on Criterion-Referenced Instruction, click here. For information on the Criterion-Referenced Instruction Workshop, click here.

Is Your Training Organization Performance-Based?

By Ann W. Parkman, Executive Vice President, and
Karen VanKampen, Director - Performance Consulting

In the past few years, the number of consulting clients asking us for help in assessing their training departments has grown markedly. The same holds true with our workshop clients. The underlying request from each side is: "How can we directly impact the profitability of our organizations?"

This question underscores the transition many companies are making from traditional training organizations to performance-based training organizations. What exactly does it mean to become performance-based? It means the primary goal of your department is to encourage your organization to focus on improving job performance as a means of directly affecting the quality of your organization's products and services and the efficiency with which they are produced. To achieve this goal, the following critical elements must be in place:

• The training organization's goals must be aligned with the goals of the business
• A thorough analysis must be conducted before solutions are selected
• Selected solutions must focus on job performance
• When training is identified as the appropriate solution, it is designed using a systematic performance-based instructional design methodology such as Criterion-Referenced Instruction (CRI)

Here are some recommendations for you to consider as you assess ways to transform your department into a true performance-based training organization.

Focus on Results - Improve Job Performance

The performance-based training organization's focus should be to improve job performance and sustain desired levels of performance. In other words, training is not always the answer. In addition, the support to employees does not end once they leave the classroom. To maintain a focus on results, ask yourself if your organization:

• Has a policy in place that requires all training requests to be analyzed to ensure the right solution (training, non-training, or both) is developed and implemented.
• Views training as the right solution only if the cause of the performance problem is due to a lack of skill or knowledge.
• Follows analysis procedures to ensure objectives are derived from job performance needs.
• Gives business leaders training and tools to guide them in providing feedback and coaching to their employees.
• Has procedures and methods to communicate the importance and benefits of a systematic performance improvement approach.

Training/Performance Improvement Methodology

Adopting a best practice, performance-based training methodology will ensure that training focuses solely on job-relevant skills. CEP uses CRI exclusively because it leads each learner to the desired competence level described by the performance objectives. Fully implementing CRI throughout your training organization will ensure competence in skills learned which will, in turn, result in improved job performance. Here are some questions to help you determine how your training organization's methodology compares to CRI:

• Does training focus on providing learners with the skills they need to improve job performance?
• Do practice situations match the actual on-the-job conditions as closely as possible?
• Do learners receive feedback immediately after practice situations so they know what they have done right or wrong?
• Does training include skill checks (not knowledge tests such as multiple choice) to be sure learners have mastered essential skills?

Performance Measurement - Measure Results

A performance-based training organization is in the best position to take the lead in or influence the development of an organization-wide plan for standardizing and aligning performance measures. Work with Human Resources and line management to establish direct links between pre-employment assessments, training skill checks, quality auditing, employee performance reviews, and major business unit goals. Also take steps to measure the return on investment (ROI) of training and non-training performance improvement solutions so you can be certain your solutions are working and are cost effective. To help you assess how your training organization measures up, ask yourself:

• Are employees' job tasks and performance standards aligned with organizational goals?
• Are pre-employment assessments, training skill checks, and performance appraisals aligned with job tasks and standards and with the major business unit goals?
• Are standards of performance stated in quantitative terms, including measurements such as productivity, quality, and service level?
• Do end-of-course evaluations assess the learners' perceived ability to apply their newly learned skills back on the job?
• Do procedures exist to evaluate performance at key stages of learning and development (for example, upon completion of training, after learners have returned to their jobs for a set amount of time, etc.)?
• Do procedures exist to determine the ROI of training and non-training solutions?

If you answer "no" or "don't know" to any of these questions, you may need to close some gaps between your current training focus and your desired performance improvement focus. You can develop targeted strategies for closing these gaps and transition to a true performance-based training organization.

Need help assessing your training department? Contact Paula Alsher at 770-458-4080 or palsher@cepworldwide.com for a free consultation.

Periodically, we send out e-mails regarding special discount offers on workshops and books, as well as information about new products and services that will help you achieve positive bottomline results. If you would like to be added to this distribution list, please click here.

If you would like to be removed from our newsletter mailing list, please send an e-mail to info@cepworldwide.com with "Unsubscribe to Newsletter" in the subject field.

The Center for Effective Performance, Inc.
2300 Peachford Road, Suite 2000, Atlanta, GA 30338
770-458-4080, 1-800-558-4CEP
770-458-9109 (Fax)