For those who aren’t immersed in the learning measurement world, here’s the gist: Dr. Donald Kirkpatrick, way back in 1959, wrote  his doctoral dissertation on the 4 Levels of learning measurement. His work remains the juggernaut of thinking and execution about measurement since then. If you’re expecting me to flay Dr. Kirkpatrick’s work here…sorry to disappoint, but it’s not going to happen. His work elegantly lays out a progressive measurement approach that makes perfect theoretical sense (yes, there’s a “but” in there). I will say that the way his work is implemented is often different than it was written (specifically Level 4), but I digress.

Level 3 in his model is Evaluating Behavior, particularly behavior on the job. Level 1 measures the learner’s reaction, Level 2 measures whether they learned what they were supposed to learn, Level 3 measures whether they’re using that new knowledge or skill on the job, and Level 4 measures what business results came from all that training. As I said, A great model.

In the training world, Level 4 sometimes gets replaced by Jack Phillips ROI model, which I don’t like very much for a completely different set of reasons. But today, it’s Level 3 upon which I need to geek-out.

For me, Level 3 evaluation (on the job performance) conjures images of supervisors crouching behind file cabinets with clipboards in their hands. Truthfully, I don’t even know whether file cabinets and clipboards are even part of the 21st century office anymore, but that’s the image.

Awright, enough rambling. I believe Level 3, as commonly practiced, is unsustainable. Hard words, I know, but I stand by them.

Up to the point when Level 3 is prescribed, measures of the training (Levels 1 and 2, as well as the others I think are critical earlier in the process) have been of, by, and for the training folks themselves. Once you get to Level 3 however, you’re out in the business…in the productive work flow if you will… and once you introduce a process to the work flow (like asking supervisors to fill out an observation “survey” about their recently trained employees) that doesn’t actually add value to the work flow, you brand it for extinction. Well-meaning training folks will create brilliant Level 3 documents and thrust them like sticks into the rushing flow of the business stream, where they will be snatched from their hands and whirled away. Gracious, that’s a cheesy metaphor. I should write romance novels.

Most companies have some sort of performance evaluation program (if they don’t, shame on them). Some of these are yearly, some are quarterly. Nearly all of them ask that supervisors evaluate employee performance in a number of different categories.

Well heck, if this training is so important to the business (and it is, or we wouldn’t be doing it…right?), then evidence of its successful expression on the job should already be baked into the performance evaluation process.

That said, it’s possible that the training results are a subset of the larger category that actually shows up on the performance evaluation.

(like learning in training how to put a key in the ignition, while the performance evaluation might simply rate how well you drive a car)

In cases like these, a temporary Level 3 survey or observation could be useful in the early stages of the training in order to ensure the link between the training and performance.

(if we find out in the Level 3 that even after training, folks aren’t putting the key in the ignition correctly, then the performance evaluation will fail and that Level 3 observation becomes appropriately predictive)

But as the work flow rushes by, the ideal situation is to have access to the data produced by the (hopefully…please let it be so) electronic performance evaluation process, so you can sort out the one related to the training and get regular data on how well employees are performing in that particular category. This will provide a consistent flow of data that directly reflects what I believe Dr. Kirkpatrick wanted us to learn in Level 3…namely whether people are using what they learned in their regular jobs.

I do believe Dr. Kirkpatrick’s work holds up well after all these years, but I think implementing any learning measurement program requires a bit of real-world scrutiny to see whether we’re creating a measurement process in order to check it off a list, or if we’re creating one that actually allows action to be taken for continuous improvement.

Go forth and measure…

~Geek~

So we set about studying the methodologies: Don Kirkpatrick, Jack & Patti Phillips, Robert Brinkerhoff and more. What does Josh Bersin have to say? Surely Elliott Masie has an opinion! What about Judy Hale? Surely there’s a consultant who can help me unravel this! And there are. There are consultants with opinions sprouting like spring wildflowers, ready to tell us the best way to measure learning effectiveness. I should know; I’m one of them.

But really it’s this: the effectiveness of learning is measured the way anything is measured:

• The measure of my coffee cup’s effectiveness is its ability to hold coffee.
• The measure of my pen’s effectiveness is its ability to deliver ink to paper for as long as my hand can hold out without leaving ink splots on the page.
• The measure of my car’s effectiveness is its ability to transport me to work AND its ability to provide a quite space to listen to music.

So here’s your new Learning Measurement Strategy: How well does it do what it’s supposed to do? This is a simple question, but it’s best understood if we break it down into its component parts. Remember sentence diagramming? (do they still teach that?)

“How well…”

“How well” tells us that we’re being asked to evaluate something. The words measurement, evaluation and assessment sometimes get used interchangeably and that’s not always a bad thing, but let’s come to a common definition. Measuring is counting and counting is easy.

• I have $.75 in my pocket.
• We trained 162 people.
• 37 employees quit within the first 30 days of hire.

But evaluation is more than counting, evaluation asks us to compare the measurement to a target. Anytime we measure how well something does, we’re comparing the measure to a target. So evaluation includes a pass/fail component when the measurement is compared to the target.

• With $.75 in my pocket, I don’t have enough for the bus, which is $1.25. Fail.
• We trained 162 people, which is the entire target audience. Pass.
• 37 Employees quit within 30 days of hire, but last quarter that number was 97. Pass.

You see that targets can be subjective. 37 people quitting within 30 days of hire isn’t likely a crowning achievement, but out of context we really don’t know how to define success. So in our measurement strategy so far we know we’re going to be asked to develop a performance target at some point.

“…does it do…”

This is a two-parter. In the middle is the easy one, “it” is training, or “it” could be mentoring, or executive coaching, or the implementation of a new resume tracking system. “It” is simply the intervention that you put in place to create change in your organization.

The cookie sandwiching the creamy center of “it” makes up our handy action phrase, implying that our intervention is more than theoretical, and it is. We’re not just imagining training, we’re going to actually develop training and then we’re going to deliver it somehow. In short we expect it to DO something.

“…what it’s supposed to…”

This is the big one. Encased in “supposed to” is the business process that our training supports, how that business  process is measured, and the definition of success (i.e. the target) for that measurement.  “…supposed to” could also be “…designed to.” Design cannot be separated from measurement, they’re like bookends: the stuff in the middle won’t hold up if they’re not both in place. “Supposed to” could include:

• Conclusions drawn from the audience analysis
• The items on your boss’s performance evaluation
• Key performance indicators on the company’s annual report to shareholders

And by the way, the most important of all those is “make more money than we spend”.

So our measurement strategy is to determine how well we have designed and delivered an intervention to support a specific business process. Imagine that this is like digging a tunnel through two sides of a mountain with plans to meet in the middle. If we’ve done the analysis and design correctly, those two tunnels should seamlessly meet in the middle of the mountain.

So then, our measurement strategy is “Does it do what it’s supposed to do?”. More specifically: “Does the data we collect when people take this training indicate that the training is driving the correct business process target?” Even more specifically across various examples:

• Did the sales training increase sales of our new product?
• Did the safety training reduce lost-time injuries?
• Did the leadership training in basic financial skill reduce monthly budget inconsistencies?
• Did the customer service training decrease the complaints regarding rude representatives?

By the way, this is not a new idea. Dr. Deming, in teaching our friends in Japan about zero-defect quality, called this the PDCA model. Six Sigma calls it the DMAIC model. Learning calls it the ADDIE model. But no matter what you call it, the process is the same. When you design a solution to meet a need, the effective measurement is the degree to which the solution meets the need.

~Geek~

Not a ton more to say about this other than to ensure that we differentiate the two. Admittedly, I currently make my living measuring the “business impact” of training, but in truth what I do is help clients measure the business processes that are supported by training.

Training measurement books are as ubiquitious as diet books, but like the folks buying diet books without losing weight, buying (and reading) training measurement books don’t seem to be getting folks any closer.

The key is alignment with the business units for whom the training department is developing training. Actually, alignment doesn’t seem to quite get there…I’m talking about collaboration and partnership here. Business units need to collaborate with the training department to drive their key performance indicators.

I’ve said it before (and heaven knows I will again): The right success measure for training is the success measure for the business process it supports.

~Geek~