In a previous article, we discussed the possibilities of using the same tools we use for an end of course assessment for measuring the impact of learning. Today we want to expand on that idea to show you some of the ways you can evaluate learning using the Advanced Analytics and LMS Advanced Reporting tools.

While the pundits and consultants rave about advanced analytics, artificial intelligence, and machine learning, almost all the information consumed in business today is operational reporting.

That doesn’t mean that reporting has stood still during the digital transformation. We now have many new ways to access information. Instead of quarterly reports, we now have real time interactive visualizations, and embedded analytical engines in our learning management systems have made it possible for us to deliver real business intelligence immediately to the people who make decisions.

When it comes to evaluating learning, our operational reporting and our assessment tools form a powerful combination that can help us understand our learning activities at a profound level. The process begins with designing assessments, or surveys that produce valid, useful information.

You might be thinking at this point that we proved decades ago that end-of-course evaluations, or “smile sheets,” have little or no value in predicting improvements in performance. That is true. Meta-analysis of 34 scientific studies in 1997 found that smile sheets uncorrelated with learning results, and a similar, larger study in 2008 produced the same result.^[1]

Will Thalheimer (we are fans) declared the traditional smile sheet dead and wrote an autopsy in chapter 2 of Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form. He cites several reasons:^[2]

• Learners are incapable of providing an unbiased assessment of their learning, mostly because it’s presented in a biased context. We just taught you all about widgets. Now, tell us objectively whether you learned enough about widgets to be productive on the job.
• Likert-like scales are ambiguous. According to Thalheimer, they don’t help learners take the responses seriously, create indecision, and produce cognitive fatigue. Respondents tend to give the same answer down the list of questions.
• An assessment administered when the learning is fresh has little bearing on whether they will recall the information a week later. Can you tell today us how much of this article you will remember tomorrow? Not likely.

We first began trying to eradicate Likert scales and other fuzzy measures in the early 2000’s. We were working with organizations seeking to improve not only their learning standards, but also their performance management systems, development plans, and succession management practices. A lot of organizations were experimenting at that time, and our primary focus was using granular behavioral statements that eliminated as much bias and ambiguity as possible. It could be cumbersome.

We get better results when we can reduce questions to behavioral statements and ask the respondent to give us a yes/no response to each. The binary response removes the guesswork of scales or levels and focuses the respondent on the behavior rather than an ambiguous value. Then, we can present the percentage of responses to get a valid assessment.

Whoa! You say. Some things are more important than others we need a way to differentiate between what is merely okay and what is the most desired result.

Thalheimer attacks this problem this by assigning an acceptability level to each statement, beginning with what is acceptable and what is not. We can then define other levels if we so choose. So, if we are evaluating the quality of after-training support, we could have something like this example:

One of the skills you learned in your recent training was ___________. Since ongoing training and support are essential to learning new skills, we want to know what support you are receiving in applying the training.

Check all that apply.

• I have time to apply what I learned to real job tasks. Acceptable
• my supervisor encourages me to apply what I learned to real job tasks. Acceptable
• My supervisor monitors my progress in applying what I learned. Superior
• I have a coach or mentor to help me apply what I learned. Superior
• I receive follow-on training. Acceptable
• I do not have support in applying what I learned. Unacceptable

We can chart it like this:^[3]

We can apply the technique to any stage of the learning process, from design to after-training support, but we can also use this to gauge impact on the business. Let’s take an example of upselling training for customer support staff. We can send a questionnaire to participants, and, at the same time, gather the same information from managers to further validate our findings.

You (your team) recently completed upselling training for our new product line. Please complete the following short questionnaire to help us gauge how effective the training was in helping you (them) to sell the product.

Check all that apply:

• The training did not help me upsell the product, and I will not be able to do so.
• The training had no impact on my ability to upsell the product.
• The training has not helped me upsell the product yet, but I expect it to.
• I may be able to upsell the product, but I need more training.
• I can sell the product if I get more coaching and support.
• The job aids help me upsell the product.
• The training helps me upsell the product, and I have increased sales by more than 10%.
• The training helps me upsell the product, and I have increased sales by less than 10%.

Here’s a little exercise for you. How would you assign acceptability levels to these items? Apply the Thalheimer labels if that fit or create your own.

• Acceptable
• Not acceptable
• Unlikely
• Superior
• Deplorable

We can apply the same methods to any situation where we have observable performance.,

If you don’t already have Thalheimer's book, we recommend you get it and give the principles a try. You have nothing to lose but your useless smile sheets.

References:

1. Thalheimer, William, PhD. "Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form." Kindle location 313. Somerville, MA. Work-Learning Press, 2016.

2. Thalheimer.

3. Thalheimer

Chasma Place, is an independent source for solutions that will help you keep pace with changes in the way your people work without ripping and replacing your existing systems.