How to Measure Online Course Effectiveness

Every course or training initiative has at least one of two goals: to bridge knowledge gaps, and/or to transform the learner’s behavior.

Instructional designers aim to provide the necessary content to accomplish these goals, but sometimes the delivery of a course or training isn’t very effective. In these cases, students or trainees may not experience the results that they (or their employers or faculty) were hoping to see, and the course will need to be improved…

… but how can you tell which part(s) of a course are flawed?

Guessing at which aspects of a course need to be improved is a recipe for frustration. Luckily, there’s a proven process that helps you measure the effectiveness of your courses and start to fix any problems in their delivery.

Kirkpatrick’s Four-Level Approach to Assessing Training Outcomes

In his well-known book Four Levels of Training Evaluation, industry expert Donald Kirkpatrick established a trusted method to help training developers and HR specialists measure the effectiveness of their training initiatives.

LEVEL 1: Reaction

The first order of business is to assess how learners react to the course by asking…

  • What did participants think about the course?
  • What did they like the most?
  • What aspects were under-covered or least-liked?
  • Would participants recommend the course to their colleagues / peers?

The objective of Level 1 is to measure favorable and negative reactions, so you can emphasize the good aspects and correct or mitigate the bad.

LEVEL 2: Learning

This level seeks to evaluate whether your learners acquired the information, skills, and knowledge they were expecting to obtain.

  • Did they learn what they intended to learn?
  • Has this training improved their skills, confidence and attitudes?
  • Did this course offer them all the resources they were looking for?

In order to best measure the impact of your courses, you should test learners prior to and after taking the course, so you’ll have a benchmark for comparing students’ achievement of the desired learning outcomes.

LEVEL 3: Behavior

This level seeks to measure the transference of course knowledge into the workforce or student body.

  • Is the new knowledge used in their every-day work environment?
  • Are they now able to share their learning with others in the workplace / classroom?
  • Have supervisors, colleagues, and co-workers seen positive behavioral adjustments?

One important factor to consider when measuring behavioral changes is whether learners have the appropriate environmental support (e.g. organizational culture, structured performance reviews, etc.) to reward behavioral change. The absence of such support may impede any post-training behavioral changes if employees or students don’t believe that those changes will be appreciated or deliver tangible results.

LEVEL 4: Results

These metrics measure whether, and to what degree, planned organizational outcomes were accomplished as a result of training. For example:

  • Did the course increase productivity?
  • Did it help reduce negative outcomes (such as accidents or injuries)?
  • Was the overall error rate reduced?
  • Have sales increased?
  • Are fewer customers complaining?
  • Is customer, employee, or student retention improving?

Once again, to correctly measure these types of results, you must conduct a study of these metrics prior to implementing the course in order to establish a pre-training benchmark.

How to Implement Effectiveness Measurement Plans

Since Kirkpatrick’s approach is based on a graduating scale of precision and complexity, you should begin with Level 1 measurement and work toward Level 4. Starting with the simplest level of evaluation will help your entire team or department get used to the effectiveness measurement process and is likely to increase buy-in from your colleagues when more complex steps — like pre- and post-training measurement comparisons — are introduced.

You should also allow for an adequate lapse of time between measurements. For example, don’t expect significant behavioral changes to be clearly evident immediately after students complete a course.

It should also be noted that this approach to measuring course effectiveness highlights the results of training. It does not provide corrective suggestions on what steps to take to bridge the gaps between planned learning outcomes and actual post-learning behavior. You will need to take appropriate action to close those gaps based on the results of the course effectiveness measurements.

Image: “Improvement” by thinkpublic, via Flickr Creative Commons License

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Leave a Reply

Your email address will not be published. Required fields are marked *