How to Evaluate Off-the-Shelf Training Products

© 2001, CEP

These days, there's a mind-boggling array of training products from which to choose. Need "training" on teamwork? Pick from literally dozens of off-the-shelf videos, games, workshops, and other resources. Got customer service problems? You're in luck - there are enough software programs, hands-on simulations, and skill-building activities to make your head spin.

But before pulling out your checkbook, it's important that you be able to evaluate whether an off-the-shelf product will work. After all, that's what you are truly paying for - not the glitzy packaging or the entertainment value. When all is said and done, you need to know whether the product you've selected will solve a specific performance problem in the most cost-effective manner possible.

Below are some questions you should ask yourself when considering an instructional product that has already been developed and packaged. These questions have been extracted from the "Evaluate Proposals" module of Dr. Robert F. Mager's Training Manager Workshop.

1. Has a need for training been verified?

Check to be sure that a need for instruction exists - that is, that there are people who cannot do what they need to be able to do, and that this problem cannot be solved through some other means (such as job aids, improved documentation, or feedback).

2. Are there objectives describing the outcomes accomplished by the instructional product?

A well-designed product will tell you what learners will be able to do after completing the training.  Without a clearly stated description of what the product is intended to accomplish, you will have no basis for evaluating the instructional activities, the media used, or even the product's overall usefulness. If your answer to this question is "No," consider rejecting the product.

3. Is the target population at which the product is aimed clearly defined?

A one-size-fits-all product is likely to fit no one closely enough to be of much value.  When you are offered an instructional product, it should be accompanied by a description of the specific audience(s) for which it is intended.

4. Do the objectives and the target population match your need?

Unless both the objectives and the target population of the proposed product match your own, the product will be unlikely to do you much good. 

5. Has the product been subjected to validation (tryout) testing?

Even though it can be very expensive and time-consuming to complete extensive validation studies, something should have been done to find out whether the product does what it is supposed to do.  Ask how the product was tried out, and by whom.  Expect to hear that:

a. The product was tested and revised during the development phase, and that

b. The materials were revised on the basis of one or more field tryouts

6. Are criterion tests (skill checks) provided by which to determine whether the objectives have been accomplished?

If the product is alleged to have been developed to accomplish stated objectives, it is reasonable to expect to be provided with a means for finding out whether those objectives have been accomplished. Your answer to this question will tell you whether the product has indeed been designed to accomplish specified outcomes, or whether it was developed merely to provide some interesting instructional activities.

The criterion tests or skill checks should match the objectives. If they don't, you won't be able to find out whether the objectives have been accomplished.  Serious mismatches between the criterion tests and the objectives should cause warning bells to ring.

7. Is adequate practice provided?

Probably the most common failing of instruction -- anywhere -- is that it doesn't provide practice in the objectives, or enough practice. So check to see if practice is provided, and if that practice is related to the objectives. If there are no objectives against which to compare the practice, think twice about adopting the product.

8. Are the instructional materials/activities lean - in other words, are excess content and busywork activities kept to a minimum?

Keep the objectives in view as you review the instructional materials, asking yourself whether you can easily tell which objectives each activity is intended to help accomplish.  If too much irrelevant content/activities exist, you will be wasting people's time.  Worse, you will be squandering your credibility.  Learners know when they are being asked to do things that are irrelevant to their needs.

9. Is the delivery system appropriate for your target population?

Does the delivery system require poor readers to read large amounts of material? Do learners have access to VCRs, computers, or other required media? Does the delivery system ensure that learners will engage in relevant practice? If the delivery system isn't a good "fit" with your target population, you should consider rejecting this product.

10. Is the cost reasonable for the number of learners you expect to train?

To make sure the cost is reasonable, first check that the media used to deliver the instruction isn't more elaborate or costly than what is needed to accomplish the objectives.  If the media used is appropriate for accomplishing the objectives, calculate the cost of the product per learner. If the cost fits within your own guidelines, then you should consider the product for adoption.