Level 3 Evaluation
For the learning measurement wonks (and you know who you are), it seems important to take a jab at the venerable Dr. Kirkpatrick today.
For those who aren’t immersed in the learning measurement world, here’s the gist: Dr. Donald Kirkpatrick, way back in 1959, wrote his doctoral dissertation on the 4 Levels of learning measurement. His work remains the juggernaut of thinking and execution about measurement since then. If you’re expecting me to flay Dr. Kirkpatrick’s work here…sorry to disappoint, but it’s not going to happen. His work elegantly lays out a progressive measurement approach that makes perfect theoretical sense (yes, there’s a “but” in there). I will say that the way his work is implemented is often different than it was written (specifically Level 4), but I digress.
Level 3 in his model is Evaluating Behavior, particularly behavior on the job. Level 1 measures the learner’s reaction, Level 2 measures whether they learned what they were supposed to learn, Level 3 measures whether they’re using that new knowledge or skill on the job, and Level 4 measures what business results came from all that training. As I said, A great model.
In the training world, Level 4 sometimes gets replaced by Jack Phillips ROI model, which I don’t like very much for a completely different set of reasons. But today, it’s Level 3 upon which I need to geek-out.
For me, Level 3 evaluation (on the job performance) conjures images of supervisors crouching behind file cabinets with clipboards in their hands. Truthfully, I don’t even know whether file cabinets and clipboards are even part of the 21st century office anymore, but that’s the image.
Awright, enough rambling. I believe Level 3, as commonly practiced, is unsustainable. Hard words, I know, but I stand by them.
Up to the point when Level 3 is prescribed, measures of the training (Levels 1 and 2, as well as the others I think are critical earlier in the process) have been of, by, and for the training folks themselves. Once you get to Level 3 however, you’re out in the business…in the productive work flow if you will… and once you introduce a process to the work flow (like asking supervisors to fill out an observation “survey” about their recently trained employees) that doesn’t actually add value to the work flow, you brand it for extinction. Well-meaning training folks will create brilliant Level 3 documents and thrust them like sticks into the rushing flow of the business stream, where they will be snatched from their hands and whirled away. Gracious, that’s a cheesy metaphor. I should write romance novels.
Most companies have some sort of performance evaluation program (if they don’t, shame on them). Some of these are yearly, some are quarterly. Nearly all of them ask that supervisors evaluate employee performance in a number of different categories.
Well heck, if this training is so important to the business (and it is, or we wouldn’t be doing it…right?), then evidence of its successful expression on the job should already be baked into the performance evaluation process.
That said, it’s possible that the training results are a subset of the larger category that actually shows up on the performance evaluation.
(like learning in training how to put a key in the ignition, while the performance evaluation might simply rate how well you drive a car)
In cases like these, a temporary Level 3 survey or observation could be useful in the early stages of the training in order to ensure the link between the training and performance.
(if we find out in the Level 3 that even after training, folks aren’t putting the key in the ignition correctly, then the performance evaluation will fail and that Level 3 observation becomes appropriately predictive)
But as the work flow rushes by, the ideal situation is to have access to the data produced by the (hopefully…please let it be so) electronic performance evaluation process, so you can sort out the one related to the training and get regular data on how well employees are performing in that particular category. This will provide a consistent flow of data that directly reflects what I believe Dr. Kirkpatrick wanted us to learn in Level 3…namely whether people are using what they learned in their regular jobs.
I do believe Dr. Kirkpatrick’s work holds up well after all these years, but I think implementing any learning measurement program requires a bit of real-world scrutiny to see whether we’re creating a measurement process in order to check it off a list, or if we’re creating one that actually allows action to be taken for continuous improvement.
Go forth and measure…
~Geek~
If you enjoyed this post, please consider to leave a comment or subscribe to the feed and get future articles delivered to your feed reader.
Comments
No comments yet.
Leave a comment