How Are You Measuring the Impact of Learning on Your Organization?

Story Introduction

Kathryn is the Chief Learning Officer at a fictional manufacturing company called AshCom, located in Minneapolis. Kathryn has been the CLO for more than five years and, with her team, has produced some amazing learning experiences. But things are changing fast. Rebecca is a consultant from MindSpring who has served as an advisor to Kathryn for the last four years. This is their story.

It was the next question that most concerned Kathryn. She and Rebecca had covered some challenging topics, including:

When Rebecca first listed the questions on Kathryn’s whiteboard, the seventh question stood out: “How are you measuring the impact of learning on your organization?”

Even when AshCom was thriving, it was a challenge to both measure and report the return on investment of learning. When Kathryn sat in meetings with the C-Suite, she was impressed by and even a little envious of other department leaders who could tell a story using only numbers. They could easily demonstrate how an investment in their department generated profit for the company. Kathryn wasn’t able to match them.

Her concern grew as the financial performance of AshCom flattened and then turned red. The CEO and CFO began to take steps to right the ship. One of their first actions was to implement a hiring freeze so that when people retired or left the 7,000-person company, they would not be replaced. This announcement sent a chill through the organization. One of Kathryn’s learning experience designers was unsettled and shortly after accepted another job. Kathryn was sad to lose a valued team member, and she knew the hiring freeze meant she would not be replaced.

The chill turned to fear with the next announcement from the executive leaders. Personnel cuts would be made across the company, impacting every department. Kathryn was required to layoff two members of her team. It was up to her to decide who. The decision was personally painful for Kathryn. She worked closely with her small team and choosing who to let go was the most difficult professional decision she’d ever made.

Her team of ten was now down to seven people, and they were not happy. They were scared. So was Kathryn. Since the layoffs, she hadn’t been sleeping well, waking up around 3:30 am every night thinking about what had happened and what else was to come. She felt powerless. For the first time in her tenure at AshCom, Kathryn told her husband she was considering looking for another position. A year earlier, before the economic downturn, the thought would never have entered her mind. Kathryn loved her job, her team, and her company and had assumed she would be with AshCom until she retired.

Rebecca, the consultant from MindSpring, was spending several days onsite. Measuring the impact of learning was the first topic of the day. When Kathryn arrived at the learning offices shortly before eight, Rebecca was already waiting for her with a cup of coffee and a danish.

“Looks like you did a little shopping this morning and still beat me here,” said Kathryn warmly.

“Every meeting should have food, don’t you think?” replied Rebecca as they entered Kathryn’s office and sat down. “I’ve been doing this for a long time and I’m pretty sure your blood pressure is a little elevated this morning knowing what we need to discuss. Maybe I should have brought you a chai instead?”

Kathryn laughed. “Trust me. Coffee was the right choice. My body wants to sleep but my brain thinks I need to be awake in the middle of the night so I can worry more. Coffee sounds perfect.”

“Before we dive into all the details of measuring and reporting the ROI of learning,” said Rebecca, “I want to make sure I listen to you and understand where you are in all of this. What’s on your mind? I want to encourage you to be completely transparent.”

“I’m so glad you said that,” said Kathryn. “I’ll be transparent with you because, honestly, I don’t really have anyone to talk to about this. I can’t talk to my peers on the leadership team because everyone is anxious and worrying about their own departments. I also can’t talk to my team. I don’t think it would do much good and would only make them worry more.”

“I hear you,” said Rebecca. “Let’s call this a safe space. What’s on your mind?”

“Here goes,” said Kathryn. “I’ll start with the fact that I’m afraid. Our learning wasn’t working as well as it used to because learning is the last thing on people’s minds in the middle of a financial struggle. We’ve already had a hiring freeze and then layoffs. And I don’t know what’s coming next. The first layoffs were companywide. We all wonder if more budget cuts are coming? If so, which departments are most likely to be cut? I’ve heard from some of my learning leader peers who went through cuts after 2008 or in earlier downturns. They say learning teams are usually the first to have their budgets reduced or even eliminated.”

Kathryn paused, wondering if Rebecca would confirm what she’d heard or put her at ease.

“I think you want me to tell you not to worry about that,” said Rebecca, “but then I wouldn’t be doing you any good. Your learning leader friends are right. Deep cuts and even department eliminations almost always begin with the learning teams. There is a logic to it, even though people like us don’t like it. Those who have to decide where to cut don’t want to cut sales or production teams unless it’s absolutely necessary. Those areas generate the money that runs everything else. Instead, they look for the places that will have the least impact on their ability to book sales and deliver.”

Think of how a doctor approaches a problem with a patient’s health. There are three stages. The first is the diagnostic stage, when the doctor identifies the injury or disease by asking questions and performing an exam. In the second stage, the prescriptive stage, the doctor determines a treatment. And the third is the prognostic stage, in which the doctor tells you the likely outcome if you follow the prescription.

“I get it,” said Kathryn, “but my fears don’t end there. I don’t know a lot about the details of AshCom’s financials right now, and I’m not sure I should ask. The finance team looks bedraggled and exhausted. Our CFO is a friend of mine, but I’m not sure now is the right time to ask for a better understanding of where things stand. The whole C-Suite team seems like they have better things to do with their time than explain financials to me.”

“I’m sure that the finance people are doing everything in their power to make good decisions,” Rebecca offered, “but I’m guessing there’s something else on your mind.”

“You’re very good at reading me,” Kathryn said with a faint smile. She took a moment to sip her coffee before continuing. “You know as well as I do that learning teams aren’t in a great position to show value coming from their budgets.”

“I want to be clear about something,” said Rebecca. “You aren’t saying that learning isn’t valuable.”

“Of course not,” said Kathryn with some force.

“So let’s agree that learning does deliver value to AshCom,” said Rebecca. “That isn’t in question. Your challenge is clearly demonstrating that value— making it visible, in other words. And remember, you can be sure every other department at AshCom is also trying to defend their budget. ”

Kathryn smiled weakly. “I know you’re right. Cash is king now, and our leadership team is looking for places to cut expenses with the least amount of short-term damage.”

Rebecca nodded sympathetically. “I know this is tough. Thanks for being transparent. Knowing where your heart and mind are helps me know where to go next. We’re going to stick with looking at the ROI of learning, but I want to approach it from a medical perspective.”

Kathryn raised her eyebrows. “I appreciate your concern about my health, but I’d say medical advice is beyond your job description.”

“Ok, let me put that another way,” said Rebecca with a laugh. “I want to use a medical metaphor. Think of how a doctor approaches a problem with a patient’s health. There are three stages. The first is the diagnostic stage, when the doctor identifies the injury or disease by asking questions and performing an exam. In the second stage, the prescriptive stage, the doctor determines a treatment. And the third is the prognostic stage, in which the doctor tells you the likely outcome if you follow the prescription.”

“Interesting,” said Kathryn. “I think I see where you’re taking this. Can we jump right to prognostication where you tell me everything will be fine?”

“Would you want your doctor to take shortcuts?” said Rebecca.

“Ok, I see you want full commitment to your metaphor,” Kathryn said with a chuckle. “Go ahead. I’ll play along!”

“Thanks for your patience. I think it’ll pay off.” Rebecca continued, “So, just like the doctor, we can’t skip steps. We’ll start with the diagnostic step today, and then we’ll talk about prescriptions to fix it. If you want to think of this in another way, we can simply call the first step ‘discovery.’ To get helpful answers, we need to begin here.”

“I’m all ears, Dr. Rebecca,” said Kathryn with a smile.

“I’m going to make some statements and ask some questions,” said Rebecca. “Again, please answer honestly, as that will help us get to the best solution. Would you say that ROI is very important to your success as a learning team but that you don’t really know how to measure it?”

Kathryn grabbed her notebook and flipped through it. “I’m looking for something I wrote down a few weeks ago after reading an article. Here it is: ‘To be blunt, learning and development teams have not been trained to use data effectively.’”

“So you agree,” asked Rebecca, “that ROI is important but you don’t know how to measure it?”

“I guess I was trying to make the point that we aren’t alone in this,” said Kathryn, “but my answer is yes. It is important, especially right now, and I don’t do a very good job of it. Let me put it another way— I need to justify our learning budget, and I don’t have the hard numbers to do it.”

“Second question,” said Rebecca. “Would you say that you mostly only measure what’s available to you from your learning management system?”

“Yes,” replied Kathryn. “And sometimes what we get from our LMS doesn’t make a lot of sense, isn’t very visual, and doesn’t help us show ROI. We can show that people have taken courses, or that they’ve successfully completed knowledge checks, but that’s about it. No one on the financial team will be impressed by those numbers because they don’t show behavioral changes. Without understanding how learning experiences impact behavior, we really don’t have anything to say about ROI.”

You can’t demonstrate value if you cannot show your finance team that you understand their direction and can demonstrate how a robust learning program will help them get there.

“Now we have a baseline,” said Rebecca. “Your answers aren’t good news, but they’re honest and—to give you some reassurance—they’re very common among L&D leaders.”

“Is that it?” asked Kathryn. “Are we done with this part?”

This made Rebecca laugh out loud. Kathryn’s exaggerated discomfort was a sign she hadn’t lost her sense of humor, even in this difficult conversation. They both took a moment to savor their coffee, which was now at an easily drinkable temperature, then Rebecca continued, “The rest of my questions are going to feel more personal. I’m not trying to point out your weaknesses, but if we’re going to get to a better place, I have to ask them. The first is this: Are you, as the CLO of AshCom, anticipating and understanding C-Suite’s priorities and directives?”

“I think I’ve already answered that one,” said Kathryn. “Clearly I am not. As I said, I can’t even say I fully understand our current financial situation.”

“And you see why that’s a problem,” replied Rebecca. “I know it might seem like the worst time, but I want to encourage you to ask some hard questions of your leadership. I know they’re busy, but you need to request some time with them. You need to understand where they plan to take this company. You can’t demonstrate value if you cannot show your finance team that you understand their direction and can demonstrate how a robust learning program will help them get there.”

“I see what you mean,” said Kathryn after a moment of reflection. “These are hard things, and you’re going to tell me that only I can do this.”

“That’s right,” said Rebecca. “Let’s dig a little deeper. Can you anticipate what their objections will be to your learning plans, and do you have answers for those objections?”

You must show how the learning you and your team create makes the entire company more efficient. And you need to be able to back up your claims with data.

“I haven’t even thought about that,” said Kathryn. “I guess my anxiety is causing a little paralysis. I’m certainly not being proactive.”

“You need to show how learning saves money,” said Rebecca. “You must show how the learning you and your team create makes the entire company more efficient. And you need to be able to back up your claims with data.”

As soon as the words were out of her mouth, Rebecca realized she may have pushed Kathryn a little too far a little too fast. She could see a flash of anger that quickly began to look more like despair.

“Sorry about that,” said Rebecca. “I know you have a lot on your plate, and someone adding to your list of things to do really isn’t helpful. I have more questions, but let’s stop this exercise for now and let me give you a little hope and a possible path forward.” Rebecca pushed the bag with the danish in it across the table, “Please, try a little of this first. Data shows that danishes improve morale.”

“Thank you,” said Kathryn, cutting the danish down the middle with a plastic knife. “Share with me, please,” she pushed the bag back toward Rebecca as she continued. “I do appreciate you and what you’re doing here. I knew I had a problem in this area and that it’s bigger and harder to solve than I knew. So if you can help me and my team get to a better place, that would be the best possible outcome.”

“Then let’s move on to the prescriptive stage,” said Rebecca. “There’s a word we keep saying but haven’t discussed in depth yet, and that’s ‘data.’ When the other department heads at AshCom are reporting their ROI and showing wonderful charts and graphs, they’re basing everything on the data available to them. Your data is largely coming from your LMS and, by your own admission, it’s not very helpful.”

“So what you would prescribe in your medical model,” said Kathryn, “is more data?”

“Sort of,” replied Rebecca. “You need to take this in steps. First, determine what you want to be able to report and what data you’ll need to support it.”

“And you can help us do that?” asked Kathryn.

“We can,” said Rebecca. “We added a group of data analysts to our team to be able to provide that service because so many of our clients are in a situation similar to yours.”

“They help determine what data is needed?” asked Kathryn.

“That’s where they start,” said Rebecca. “but they can do more, as well. That team will help you figure out data sources, which is sometimes more challenging than it should be. They’ll meet with your IT team to figure out what data is available, where it’s stored, and how you can access it.”

“What happens after that?” asked Kathryn, now fully enjoying the danish.

“Once you know what you need and know where to find it,” said Rebecca, “you’ll need to build a model that will help you interpret the information. Having reams of data isn’t very helpful unless you can pull out the points you want to see. Some pieces of information will be more important to you than others, and having an interpretive model will give weight to those.”

“So we decide what we need, we work with your team to determine how we get it, and then we build a model together that helps us show ROI for our learning experiences,” said Kathryn.

“There’s one more important piece,” said Rebecca. “You mentioned earlier that the leaders of other departments can tell stories with numbers. To level the playing field, you need to be able to do the same thing. You need a way to show the impact of learning to your finance team and other C-Suite leaders so that your story demonstrates results— actual, visible results.”

“I’m not sure I know exactly what they’re looking for,” said Kathryn.

“That will be part of the assignment. You’ll need to work with the people to whom you report to determine exactly what they want to see, and even how they want to see it. You’ll be learning to speak their language,” Rebecca said, popping the last of her half of the danish into her mouth. “Does this at least give you a pathway forward to do what you know needs to be done?

“I don’t know if it is just the coffee talking,” said Kathryn, “but I get where you’re going. I know the problem, but I couldn’t figure out a way to fix it because our team is now smaller, and we simply don’t have the expertise needed.”

“The good news is that we do,” said Rebecca. “Your struggle is one of the most common and difficult among people in your position. But there is a way forward.”

“You’re telling me that the prognosis is good, doctor?” said Kathryn.

“That’s what I’m telling you,” said Rebecca, smiling. “This can be done and done well.”

Kathryn sat with that a minute and savored the last of her danish. “I’m much more hopeful than I was when I walked in here this morning,” she said. “What’s next?”

“You’ll find the next session much easier,” said Rebecca. “Our next question is, ‘How are you keeping current or ahead of the curve in the learning industry?’ It should be something right up your alley.”

“Great,” said Kathryn. “Fifteen minute break and then meet back here?”

“Perfect,” said Rebecca. “Before we break, I have something I want to tell you. I want to commend you for your honesty and transparency. You said what’s really on your mind, even when it didn’t put you in the best light, and that’s what enabled us to have a truly productive conversation.”

Kathryn smiled brightly. “I appreciate you. See you in fifteen.”

Interested in learning more from our thought leaders perspective on ROI and Learner Metrics?