Your New Training Measurement Strategy
Learning measurement has become mired in its own best intentions and is sinking. Everyone of us agree that there should be some significant measurement of the learning function. Why? To get a seat at “the table”! To protect our budgets! To calculate the ROI for the training dollars we DO spend!
So we set about studying the methodologies: Don Kirkpatrick, Jack & Patti Phillips, Robert Brinkerhoff and more. What does Josh Bersin have to say? Surely Elliott Masie has an opinion! What about Judy Hale? Surely there’s a consultant who can help me unravel this! And there are. There are consultants with opinions sprouting like spring wildflowers, ready to tell us the best way to measure learning effectiveness. I should know; I’m one of them.
But really it’s this: the effectiveness of learning is measured the way anything is measured:
- The measure of my coffee cup’s effectiveness is its ability to hold coffee.
- The measure of my pen’s effectiveness is its ability to deliver ink to paper for as long as my hand can hold out without leaving ink splots on the page.
- The measure of my car’s effectiveness is its ability to transport me to work AND its ability to provide a quite space to listen to music.
So here’s your new Learning Measurement Strategy: How well does it do what it’s supposed to do? This is a simple question, but it’s best understood if we break it down into its component parts. Remember sentence diagramming? (do they still teach that?)
“How well…”
“How well” tells us that we’re being asked to evaluate something. The words measurement, evaluation and assessment sometimes get used interchangeably and that’s not always a bad thing, but let’s come to a common definition. Measuring is counting and counting is easy.
- I have $.75 in my pocket.
- We trained 162 people.
- 37 employees quit within the first 30 days of hire.
But evaluation is more than counting, evaluation asks us to compare the measurement to a target. Anytime we measure how well something does, we’re comparing the measure to a target. So evaluation includes a pass/fail component when the measurement is compared to the target.
- With $.75 in my pocket, I don’t have enough for the bus, which is $1.25. Fail.
- We trained 162 people, which is the entire target audience. Pass.
- 37 Employees quit within 30 days of hire, but last quarter that number was 97. Pass.
You see that targets can be subjective. 37 people quitting within 30 days of hire isn’t likely a crowning achievement, but out of context we really don’t know how to define success. So in our measurement strategy so far we know we’re going to be asked to develop a performance target at some point.
“…does it do…”
This is a two-parter. In the middle is the easy one, “it” is training, or “it” could be mentoring, or executive coaching, or the implementation of a new resume tracking system. “It” is simply the intervention that you put in place to create change in your organization.
The cookie sandwiching the creamy center of “it” makes up our handy action phrase, implying that our intervention is more than theoretical, and it is. We’re not just imagining training, we’re going to actually develop training and then we’re going to deliver it somehow. In short we expect it to DO something.
“…what it’s supposed to…”
This is the big one. Encased in “supposed to” is the business process that our training supports, how that business process is measured, and the definition of success (i.e. the target) for that measurement. “…supposed to” could also be “…designed to.” Design cannot be separated from measurement, they’re like bookends: the stuff in the middle won’t hold up if they’re not both in place. “Supposed to” could include:
- Conclusions drawn from the audience analysis
- The items on your boss’s performance evaluation
- Key performance indicators on the company’s annual report to shareholders
And by the way, the most important of all those is “make more money than we spend”.
So our measurement strategy is to determine how well we have designed and delivered an intervention to support a specific business process. Imagine that this is like digging a tunnel through two sides of a mountain with plans to meet in the middle. If we’ve done the analysis and design correctly, those two tunnels should seamlessly meet in the middle of the mountain.
So then, our measurement strategy is “Does it do what it’s supposed to do?”. More specifically: “Does the data we collect when people take this training indicate that the training is driving the correct business process target?” Even more specifically across various examples:
- Did the sales training increase sales of our new product?
- Did the safety training reduce lost-time injuries?
- Did the leadership training in basic financial skill reduce monthly budget inconsistencies?
- Did the customer service training decrease the complaints regarding rude representatives?
By the way, this is not a new idea. Dr. Deming, in teaching our friends in Japan about zero-defect quality, called this the PDCA model. Six Sigma calls it the DMAIC model. Learning calls it the ADDIE model. But no matter what you call it, the process is the same. When you design a solution to meet a need, the effective measurement is the degree to which the solution meets the need.
~Geek~
If you enjoyed this post, please consider to leave a comment or subscribe to the feed and get future articles delivered to your feed reader.
Dear Geek, who’s your audience for the preso? If it’s a bunch of other measurement geeks, then maybe this is too obvious. If it’s L&D types coming for a “101″ level preso, then you’ve got a good start, given your always entertaining presentation personality, IMO.
A couple of other random but loosely related thoughts spurred by your post…
Some of this — for example: “our measurement strategy is to determine how well we have designed and delivered an intervention to support a specific business process” — sounds like a measurement “tactic” to me, rather than “Measurement Strategy,” with initial caps. “We measure everything we do” sounds philosophical. I think of strategy as occupying the middle ground between philosophy and tactic. Is the gist of your talk really strategy? I’m not just being rhetorical to be a pain — i’m struggling with the same issue on a “strategy” talk I have coming up too. What’s the right level of detail for a “strategy”-level presentation??
Lastly, and this is probably another blog post altogether… Closely tied into the idea of what to measure is whether the solution (usually training) is in fact the correct solution to the business need. Often, training is the solution to an indirect problem: people need training on the software because the software is poorly designed; sales are down because the new product isn’t competitive, not because the sale team needs training on how to close a deal.
Now if you come up with a way to measure whether training is the right intervention to a business need, or whether the money would have been better spent elsewhere, then you’ve got a talk that i’d pay to hear!