Metrics Geek Podcast.(8:15) Fewer than 10 minutes and you’ll be a Likert scale expert!

Likert Scales

This morning’s first session was Sarah Bloomfield from Google who talked about, among other things, a different view of assessment. In the corporate learning biz, the focus for assessment (Level 2) is most often individual assessment…which is to say: having the learner react somehow from their own precious brain to see if they learned what they were supposed to learn. Of course, this enables us to know what was accomplished from a learning perspective. (Yay for checking knowledge transfer off the list!).

From a business perspective however, employees are often required to perform tasks with available resources (or as part of a group). So part of the conversation in the session was assessment of the “enhanced individual” or what in grade school (if I recall) was called an “open-book test”. In essence, assessing learners in more real-life situations with the resources that they would have on the job.

Now just then when I wrote “on the job” my inner Kirkpatrick (don’t we all have one?) thought well, if it’s on the job, it’s clearly a Level 3! Yay, you can do both!

But Sarah’s point was that individual assessment is of use to the learning team but not so much to the business. So then I wonder what an enhanced individual or group assessment would look like?

~Geek~

Perhaps a little more specific: competence is “the quality of being competent; adequacy; possession of required skill, knowledge, qualification, or capacity: He hired her because of her competence as an accountant.“  How appropriate that the example given is about someone hired due to competence, given the topic of today’s post.

The most common reason I’m given for developing new-hire or onboarding training is to reduce the time to competence. When I follow with the obvious question (“what is the time to competence now?”) I get all sorts of vague answers. It seems safe to say that very few organizations have operational definitions of competence and then have also gone to the trouble to track the time it takes to get there.

The details of time to competence is for another day. For today, I’m going to say what you already know:

I say you already know this, but sometimes organizations forget it. Specifically they forget that within this formula (if you will), organizations can choose where they want to spend their money:

1. Hire experienced employees at a high cost (salary), provide minimal training and they’ll “hit the ground running.”
2. Hire less experienced employees at a low cost, provide detailed training and they’ll hit the ground informed, but relatively inexperienced.
3. Hire less experienced employees, provide minimal training and expect that they’ll get experience on the job.

I don’t propose that any of these is necessarily the right choice in all instances, but it pays to realize that there are no shortcuts to this rule. One could make the argument that your customers are going to pay one way or another – either through what they pay for your products and services, or what they “pay” for your employees to gain experience through working with them – but it’s unlikely they’re given a choice, so if your organization is operating under #3 above, it’s probably more costly in customer satisfaction (which means repeat business) than you want it to be.

So then…the questions for the day: where does your organization spend its money in this equation? Where should it spend that money? If the answers to these questions are the same, bravo!

~Geek~

For those who aren’t immersed in the learning measurement world, here’s the gist: Dr. Donald Kirkpatrick, way back in 1959, wrote  his doctoral dissertation on the 4 Levels of learning measurement. His work remains the juggernaut of thinking and execution about measurement since then. If you’re expecting me to flay Dr. Kirkpatrick’s work here…sorry to disappoint, but it’s not going to happen. His work elegantly lays out a progressive measurement approach that makes perfect theoretical sense (yes, there’s a “but” in there). I will say that the way his work is implemented is often different than it was written (specifically Level 4), but I digress.

Level 3 in his model is Evaluating Behavior, particularly behavior on the job. Level 1 measures the learner’s reaction, Level 2 measures whether they learned what they were supposed to learn, Level 3 measures whether they’re using that new knowledge or skill on the job, and Level 4 measures what business results came from all that training. As I said, A great model.

In the training world, Level 4 sometimes gets replaced by Jack Phillips ROI model, which I don’t like very much for a completely different set of reasons. But today, it’s Level 3 upon which I need to geek-out.

For me, Level 3 evaluation (on the job performance) conjures images of supervisors crouching behind file cabinets with clipboards in their hands. Truthfully, I don’t even know whether file cabinets and clipboards are even part of the 21st century office anymore, but that’s the image.

Awright, enough rambling. I believe Level 3, as commonly practiced, is unsustainable. Hard words, I know, but I stand by them.

Up to the point when Level 3 is prescribed, measures of the training (Levels 1 and 2, as well as the others I think are critical earlier in the process) have been of, by, and for the training folks themselves. Once you get to Level 3 however, you’re out in the business…in the productive work flow if you will… and once you introduce a process to the work flow (like asking supervisors to fill out an observation “survey” about their recently trained employees) that doesn’t actually add value to the work flow, you brand it for extinction. Well-meaning training folks will create brilliant Level 3 documents and thrust them like sticks into the rushing flow of the business stream, where they will be snatched from their hands and whirled away. Gracious, that’s a cheesy metaphor. I should write romance novels.

Most companies have some sort of performance evaluation program (if they don’t, shame on them). Some of these are yearly, some are quarterly. Nearly all of them ask that supervisors evaluate employee performance in a number of different categories.

Well heck, if this training is so important to the business (and it is, or we wouldn’t be doing it…right?), then evidence of its successful expression on the job should already be baked into the performance evaluation process.

That said, it’s possible that the training results are a subset of the larger category that actually shows up on the performance evaluation.

(like learning in training how to put a key in the ignition, while the performance evaluation might simply rate how well you drive a car)

In cases like these, a temporary Level 3 survey or observation could be useful in the early stages of the training in order to ensure the link between the training and performance.

(if we find out in the Level 3 that even after training, folks aren’t putting the key in the ignition correctly, then the performance evaluation will fail and that Level 3 observation becomes appropriately predictive)

But as the work flow rushes by, the ideal situation is to have access to the data produced by the (hopefully…please let it be so) electronic performance evaluation process, so you can sort out the one related to the training and get regular data on how well employees are performing in that particular category. This will provide a consistent flow of data that directly reflects what I believe Dr. Kirkpatrick wanted us to learn in Level 3…namely whether people are using what they learned in their regular jobs.

I do believe Dr. Kirkpatrick’s work holds up well after all these years, but I think implementing any learning measurement program requires a bit of real-world scrutiny to see whether we’re creating a measurement process in order to check it off a list, or if we’re creating one that actually allows action to be taken for continuous improvement.

Go forth and measure…

~Geek~

One of the most common of these is that I have found most organizations to be “data rich and action poor.” Has a nice ring to it, no?

But poetry and rhythm aside, I believe this is true. As a random corporate drone, I have sat in my cubicle and looked at reports for which I was one of a zillion people on a distribution list. These reports were often beautiful. Colorful graphs lined up like soldiers standing at attention with crisp bright uniforms.

And silent. The soldiers were also silent. Which is to say, the charts were presented as graphic representations of data, but entirely lacking interpretation.

Now, one could make the argument (and many have) that it’s the report recipients’ responsibility (say THAT five times fast) to draw conclusions about the data and usually the report creator has spent so much time with the data that the conclusions are obvious: the equivalent of the uniformed soldier smacking the reader on the noggin with the butt of a rifle.

But I’m going to go ahead and say that the emperor has no clothes. I’m not scared. The vast majority of these reports that arrived in my cube, were little more than an ocean of numbers in which I paddled around looking for meaning for about 3 minutes (if I had extra time on my hands). But more often than not, if the report didn’t come from my boss accompanied by a little red exclamation point, it got filed for future reference (which is to say “ignored”).

My point, and I do have one, is that the purpose of a report is to communicate. This is not art and we are not Leonardo DaVinci painting a cryptic smile on a woman that can be interpreted any number of ways. We are busy folks and we are communicating to busy folks and if we don’t lead those horses to water, they’ll likely wander around the pasture bonking into fences.

(If you have been skimming this posting so far, here is where you should slow down and read) In order to communicate clearly, it is the report creator’s responsibility to interpret every report, every chart and graph, and to include a clear statement about what the report represents.

• “Product sales are generally on the rise, but have leveled off this month.”
• “Employee utilization is low this month, but still on target for the quarter.”
• “Customer satisfaction is down in three of five divisions and flat in the other two.”

It will often seem, as report creators, that these are painfully obvious statements. But I consistently find that what seems obvious to the person who’s just spent a full day structuring a data set is often not at all obvious to the person who has (literally) less than a minute to comprehend a report.

The best part is when someone looks at the report, reads my summary statement and disagrees with it! While this may not seem at first blush to be the best part, it means I’ve accomplished my goal; namely to have report recipients engage with the data enough to form an informed opinion. Yay!

Not that this means one should deliberately draw inaccurate conclusions on reports…folks will believe that you’re obtuse and no one wants that.

OK, much rambling. Here’s the gist (sorry skimmers): Vow today to produce no report without also producing a conclusion about what story the data tells. Similarly, when you receive a report, vow to email back to the sender if there is no interpretation. “Marv, thanks so much for sending this. Before I review the report in detail (yeah, right), what conclusion do YOU draw from this?”

Because of course, coming back to “data rich and action poor”, the purpose of all this data and all this reporting is to take action. So the implied question is “what should I do with this data?”. Unless someone looks at the report and takes some sort of action, then it may as well be a weather report. Interesting to know what sort of storm is coming (or more likely, in organization reports, what storm has just passed), but no ability to do anything about it.

~Geek~

There is a constant effort to align training with organizational goals. It seems that alignment to organization goals is on a continuum, with training being developed completely in the absence of what the organization is trying to accomplish on the far left of the continuum and regular conversation with training requestors somewhere in the middle. Yes, the middle.

Under ideal circumstances in the organizations I’ve worked with, the training leaders and the business unit leaders are in close communication with business unit leaders, some going as far as “embedding” a training professional within the business unit. While this reduces the response time and ideally results in the training pro getting a head-start in responding to requests, it remains a transactional relationship.  The shoddy image below illustrates what I’m talking about here.

So then, what is the ultimate “alignment” for training? It’s the condition where training actually drives change in the organization. The summary of this ideal condition is one where the training team maintains an up-to-date dashboard of the key training indicators that are known drivers of organization measures.

In other words, imagine that the training team knows (through previous measurement) that product release training has to get to sales people within 30 days prior to product release. If, on the day of product release, only half the target audience has received the release training, the training team has the opportunity to go directly to the product team (or whoever) with a data-driven prediction of a gap in expected sales.

Yes, somewhat unformed…but I’m working on it. This is what comes from questioning closely-held assumptions (like the one that “alignment” is the ideal).

~Geek~

• Analyze the audience, any current training or documentation, the organizational need.
• Design a training intervention…turn process documentation and tribal knowledge into useful training.
• Develop the intervention you designed…build web-based training, ajob aid, instructor-led training, etc.
• Implement the intervention…bring it to the people.
• Evaluate the results…by comparing the original needs of the audience with the experience after training.

When followed, these steps should ensure that training is designed with the learner and the organization objectives in mind. If you imagine that these steps boil down to:

1. Determine the current state (Analyze)
2. Determine the ideal state (Design)
3. Build a bridge between those two states (Develop, Implement)
4. Check your work (Evaluate)

Then you’ll start to see the ADDIE model as one facet of what is a pretty common problem solving model.

It seems important first to say that this reductionist model of problem solving isn’t the the only way to approach improvement or change. I won’t attempt to be sufficiently informed about chaos theory and quantum physics, but I can say that there always seems to be more to the world than meets the eye. So if you’re an aficionado of chaos theory as a problem solving methodology, please share it with us here.

For the rest of us, the process of reducing something to its constituent parts for analysis is common. This method can be boiled down even further to:

• “Don’t like what you’re getting? Do something new.”

Did you know that Six Sigma, driver of organizational change and spawner of management books, boils down to the same basic methodology?  By understanding the basic steps in the ADDIE model you already have a basic knowledge of Six Sigma!

Six Sigma uses the DMAIC (pronounced “Duh-May’-Ick”) model, which stands for:

• Define the customer and their requirements
• Measure the current business process performance compared to those customer requirements.
• Analyze the difference between customer requirements and current performance.
• Improve the current process through changes that align to customer requirements.
• Control the achieved benefits by ensuring that the process doesn’t revert to old patterns.

These steps provide a framework for driving efficiency (reducing defects) in business processes. They also map easily to our previous 4 steps:

1. Determine the current state (Define, Measure)
2. Determine the ideal state (Also Define, since customer requirements drive the desired state)
3. Build a bridge between those two states (Analyze, Improve)
4. Check your work (Control)

These and other models like them are simply descriptions of a logical process for making sure something (training, an airplane manufacturing process, a selling strategy) does what it was designed to do…again, we see the relationship between design and measurement.

One of the foundations of Six Sigma is the work done by Dr. Edwards W. Deming in the early part of the 20th century, and one of Dr. Deming’s simple contributions to this idea of reductionist problem solving was his PDCA model, or:

• Plan for what you want to do.
• Do it.
• Check the results against your plan.
• Act on what the check tells you.

Dr. Deming’s work was a cycle…the “Act” portion, fed back into the “Plan” portion, and that’s one of the ways he illustrated the idea of continuous improvement. Six Sigma, includes “Control” as a way to ensure that results of improvements are embedded into regular processes and lead to continuous improvement.

Similarly, although I think this is rare, one of the primary consumers of the “Evaluate” information in the ADDIE model, should be the Instructional Designer(s) for updates to the course and to continue to improve the ability to design to stakeholder needs.

Just for fun, this model isn’t at all limited to business processes and training! In 1965, William Glasser wrote a book called Reality Therapy in which he outlined Choice Theory (described in greater detail through the Glasser Institute). The short version of Glasser’s therapeutic approach is that if you’re not getting what you want out of your life (relationship with your mom, success in school, whatever) then design some new behavior that will result in a different outcome.   The point here of course, is that this reductionist method of problem solving is pervasive.

~Geek~

We know that:

1. Training enables knowledge and skill
2. Knowledge and skill enable job performance
3. Job performance enables business process success
4. Business process success enables organizational / financial success

In each of these items, the purpose of the training measurement program is  as much as possible, to reduce the distance between “enables” and “causes”. Because causal relationships are rare outside of experimental conditions, our primary goal is to balance the cost of measurement with the minimum information needed to make informed decisions and take action.

Existing training measurement maps into these assumptions as well:

1. Training enables knowledge and skill – Kirkpatrick Level 2 (knowledge transfer)
2. Knowledge and skill enables job performance – Kirkpatrick Level 3 (behavior on the job)
3. Job performance enables business process success – Kirkpatrick Level 4 (business results)
4. Business process success enables organizational / financial success – Kirkpatrick Level 4 & Phillips ROI.

Number 4 maps to both Kirkpatrick and Phillips, although in application I believe that both those approaches are lacking at this level. Kirkpatrick’s original work definitely called out business metrics and financial success at Level 4, but he didnt’ break out business process-level measures from organizational measures (the difference between increasing sales and increasing profit). Phillips ROI model produces a percent ROI calculation that I think is misleading, although the path to that calculation is incredibly useful for understanding the interplay between training and other organizational activities.

~Geek~

For those kids and for other folks I deal with regularly, if we don’t agree on our terms – have the same assumptions – then our communication fails. In business, this is also often called setting expectations. Same dealio.

Assumptions are the map through which explanation paths are laid (I love this line…it came to me just before bed last night).

In order to agree, for example, that Interstate 5 treks north through California, Oregon and Washington, we have to agree…have to assume…what the boundaries are for those states and even have to agree that there ARE arbitrary political state lines across the landscape. Generally we take state boundaries for granted, so map makers don’t have to convince us (at least for US maps).

Frankly, even “north” is a social construct, so while we can verify factually that there is a highway laid across the landscape, it’s only the context that allows us to say that it runs “north / south” through “California”, “Oregon” and “Washington”. We could as easily say (in Washington and some of Oregon) that it runs just “west” of the Cascade mountain range.

Why, you’re wondering right now, does this make a damned bit of difference? Well I’m glad you asked…it’s because we regularly try to communicate…to demonstrate a point…so someone else with the expectation that they will come to the same conclusion we do, and outside of agreement on that context (agreement on those expectations) failure is common. Examples:

• “It will take me 1 hour to finish this report.” – There is clarification for “one hour” (some folks believe that 15 minutes past the hour is still on time), and “finish” (is that the first draft, or the final approved draft?).
• “I will clean the kitchen.” – I could take the rest of the day clarifying different interpretations of “clean” (mopping? doing dishes or just loading the dishwasher?), but even “kitchen” may need clarification (where, exactly, does the kitchen end and the next room begin?).
• And a personal favorite “I can create Level 2 web-based training for $20K per finished hour.” This dogged my professional world for better than a year. What’s the industry definition of “Level 2″? What’s the client’s definition of Level 2? Does a “finished hour” include time for participants to take assessments? Does this mean that a half hour is $10K?

I get accused of being unnecessarily literal but I gotta tell you…it takes only a little effort and given the choice between being literal and being vague, I’ll take the former.

~Geek~

So we set about studying the methodologies: Don Kirkpatrick, Jack & Patti Phillips, Robert Brinkerhoff and more. What does Josh Bersin have to say? Surely Elliott Masie has an opinion! What about Judy Hale? Surely there’s a consultant who can help me unravel this! And there are. There are consultants with opinions sprouting like spring wildflowers, ready to tell us the best way to measure learning effectiveness. I should know; I’m one of them.

But really it’s this: the effectiveness of learning is measured the way anything is measured:

• The measure of my coffee cup’s effectiveness is its ability to hold coffee.
• The measure of my pen’s effectiveness is its ability to deliver ink to paper for as long as my hand can hold out without leaving ink splots on the page.
• The measure of my car’s effectiveness is its ability to transport me to work AND its ability to provide a quite space to listen to music.

So here’s your new Learning Measurement Strategy: How well does it do what it’s supposed to do? This is a simple question, but it’s best understood if we break it down into its component parts. Remember sentence diagramming? (do they still teach that?)

“How well…”

“How well” tells us that we’re being asked to evaluate something. The words measurement, evaluation and assessment sometimes get used interchangeably and that’s not always a bad thing, but let’s come to a common definition. Measuring is counting and counting is easy.

• I have $.75 in my pocket.
• We trained 162 people.
• 37 employees quit within the first 30 days of hire.

But evaluation is more than counting, evaluation asks us to compare the measurement to a target. Anytime we measure how well something does, we’re comparing the measure to a target. So evaluation includes a pass/fail component when the measurement is compared to the target.

• With $.75 in my pocket, I don’t have enough for the bus, which is $1.25. Fail.
• We trained 162 people, which is the entire target audience. Pass.
• 37 Employees quit within 30 days of hire, but last quarter that number was 97. Pass.

You see that targets can be subjective. 37 people quitting within 30 days of hire isn’t likely a crowning achievement, but out of context we really don’t know how to define success. So in our measurement strategy so far we know we’re going to be asked to develop a performance target at some point.

“…does it do…”

This is a two-parter. In the middle is the easy one, “it” is training, or “it” could be mentoring, or executive coaching, or the implementation of a new resume tracking system. “It” is simply the intervention that you put in place to create change in your organization.

The cookie sandwiching the creamy center of “it” makes up our handy action phrase, implying that our intervention is more than theoretical, and it is. We’re not just imagining training, we’re going to actually develop training and then we’re going to deliver it somehow. In short we expect it to DO something.

“…what it’s supposed to…”

This is the big one. Encased in “supposed to” is the business process that our training supports, how that business  process is measured, and the definition of success (i.e. the target) for that measurement.  “…supposed to” could also be “…designed to.” Design cannot be separated from measurement, they’re like bookends: the stuff in the middle won’t hold up if they’re not both in place. “Supposed to” could include:

• Conclusions drawn from the audience analysis
• The items on your boss’s performance evaluation
• Key performance indicators on the company’s annual report to shareholders

And by the way, the most important of all those is “make more money than we spend”.

So our measurement strategy is to determine how well we have designed and delivered an intervention to support a specific business process. Imagine that this is like digging a tunnel through two sides of a mountain with plans to meet in the middle. If we’ve done the analysis and design correctly, those two tunnels should seamlessly meet in the middle of the mountain.

So then, our measurement strategy is “Does it do what it’s supposed to do?”. More specifically: “Does the data we collect when people take this training indicate that the training is driving the correct business process target?” Even more specifically across various examples:

• Did the sales training increase sales of our new product?
• Did the safety training reduce lost-time injuries?
• Did the leadership training in basic financial skill reduce monthly budget inconsistencies?
• Did the customer service training decrease the complaints regarding rude representatives?

By the way, this is not a new idea. Dr. Deming, in teaching our friends in Japan about zero-defect quality, called this the PDCA model. Six Sigma calls it the DMAIC model. Learning calls it the ADDIE model. But no matter what you call it, the process is the same. When you design a solution to meet a need, the effective measurement is the degree to which the solution meets the need.

~Geek~