I recently attended a talk entitled ‘Evaluation – the Holy Grail of L&D’ and it awoke a passion in me, an opportunity to challenge the status quo and help organisations take steps to improve their return on investment in learning and development.
Evaluation of L&D has received much attention from authors, many of whom stress its importance in L&D effectiveness (Bober & Bartlett, 2004; Noe, 2000). It seems that Dee and Hatton (2006) agreed, stating that “…evaluation is undoubtedly the holy grail of the learning and development world”.
Is evaluation really the ‘Holy Grail’ of L&D
I am a long-time supporter of the need for impact assessment and evaluation following all L&D interventions; but I don’t think we’ve found the ‘Holy Grail’ of L&D?
Medieval legends aside, a quick Google search suggests that…
‘…holy grail is a thing which is eagerly pursued or sought after’.
My challenge to the speaker mentioned above and to Dee and Hatton is that I think they have forgotten that the holy grail is the desired impact/outcome itself. That means:
We eagerly pursue L&D interventions that have a positive impact on one or more of attitudes, behaviours, performance, and of course business results. In other words, learning transfer.
Measuring the impact of ‘softer’ types of learning (Leadership, Management, Personal Development, etc.) is often tricky and subjective. It can be done, but at what cost?
I concede that skill-based interventions are generally easier to measure due to their tangible outcomes, but for ‘softer’ interventions such as management development, there are so many intangible outcomes vital to success. The trend seems to be toward trying to put a pounds and pence figure on these outcomes to prove ROI, whether that be a subjective approach or not.
A New Holy Grail of L&D: Evaluation versus Learning Transfer
For me this presents a matter of prioritisation between impact maximisation (learning transfer) and evaluation (did it work?).
Before spending time and money on developing elaborate evaluation models, which are often subjective, focus your energy on developing a culture, systems and behaviours that maximise impact from learning and development.
If your organisation is investing in developing their people, yet they can’t list the internal interventions that support learning, learning transfer, knowledge sharing and a whole host of other supporting factors, then they are putting the cart before the horse. They are working at developing evaluation systems that will only prove a poor bang for their buck.
Is this the death of evaluation?
Let me be clear; I believe that evaluation is essential and should be an integral part of an organisation’s L&D strategy, to continually refine what we are doing and ensure our investments are achieving their intended outcomes.
The bottom line for any learning development intervention should be:
Has it helped the individual/s, team/s and by extension the organisation, achieve its objectives? In short, has it made a positive difference?
Of course, we need to be able to answer this question, but don’t waste your time building a ground-breaking evaluation methodology until you already have confidence in the work you have done to make learning transfer and stick.
In my experience, from working with hundreds of organisations on learning and development projects and my research into the area of learning transfer and ROI, there are some key ingredients that make the secret sauce for transforming your training results. By all means, if you have the recipe already, get to work on your awesome and accurate evaluation tool!
The Secret Sauce Recipe for Learning Transfer:
I am often asked by prospective clients, early in the discussion about learning, to explain how we evaluate the outcome of the interventions we deliver. Great question, but wrong priority!
I would urge all L&D, HR and Business professionals to start asking a different question of training providers and their internal training staff. First, ask what we do to design a stronger and more effective bridge from learning to business impact and results (more about this later).
Don’t stop there; now ask yourself what you are doing in your organisation, as this is the secret sauce of L&D success, supported by two well-founded frameworks/theories…
Secret Sauce Ingredient 1: The 70:20:10 ratio
The 70-20-10 Model for Learning and Development is a commonly used ratio within the training industry, which describes the optimal sources of learning for successful managers. It holds that individuals obtain:
- 70 per cent of their knowledge from job-related experiences
- 20 per cent from interactions with others
- 10 per cent from formal educational events
This model is often challenged, with critics suggesting that the ratio changes according to the type of business and its requirements. Also, the ratio has changed since its initial inception. I wholeheartedly agree!
What is more important is the principle that structured learning or formal training courses and programmes (whether accounting for 10%, 15%, 20% or somewhere in between) only play a small part in the big picture of developing people to be better, higher performing, better to work with, or whatever it is we need them to be.
When you decide to invest in learning and development for your people first consider what you are doing in your organisation to:
#1. Foster learning and development through day-to-day tasks, challenges and practice (70%)
#2. Facilitate learning and development with and through others from coaching, people exploiting personal networks and other collaborative and co-operative actions (20%)
These two elements, if tackled strategically, will serve to create an environment where learning and development is perpetual, happening every day and creating a learning culture that stands the test of time.
On top of this, the impact of this environment will be a significant improvement on the return on investment received from formal training. That leaves the 10% down to finding the right supplier (or delivering internally of course) and holding them to account on their part of the deal (look out for our future piece on ‘Finding the right learning/training provider and getting the most from them’).
Secret Sauce Ingredient 2: The Learning Transfer System Inventory
A fixation of mine, driven by my MSc thesis topic on the subject, is learning transfer and the factors that impact on training/learning ROI and affect individual and organisational performance.
Learning transfer refers to the degree to which an individual applies learned knowledge and skills on the job. It is the primary reason for formal learning interventions—like courses, as well as informal interventions delivered in the workplace.
Learning that fails to transfer to the workplace is in abundance. According to a survey of HRD professionals, only 34% of trainees apply their learning in the workplace one year following a training intervention (Saks & Belcourt, 2006).
Elwood Holton, a distinguished HRD professor, proposed an evaluation model where L&D outcomes are affected by ability, motivation and environmental influences that impact on three outcome levels: learning, individual performance and organisational performance (Holton, 1996; Holton 2007).
Holton saw learning and improving the transfer of learning as critical outcomes in HRD (Holton, Bates & Ruona, 2000). The first step to improvement of learning transfer is the accurate diagnosis of factors inhibiting it (Holton, Bates, Ruona, 2000). His goal was the development of a valid and generalisable set of transfer scales for use by HRD professionals who wish to improve the return on their investments (Holton, Bates & Ruona, 2000). Who wouldn’t want to do that, right?
The outcome of Holton’s initiative is the Learning Transfer System Inventory (LTSI) that has four constructs: ability, motivation, environment, and secondary influence for learning outcomes and organisational outcomes (Holton, Bates & Ruona, 2000).
The model addresses, according to Holton, a significant risk in Kirkpatrick’s four levels that any failure to achieve is the fault of the learning intervention, without consideration that failure may be due to other variables (Holton, 2005). An example cited by Holton is where behaviours do not change because of an inadequate support climate, not a failed intervention (Holton 2005).
The first step to improving learning and learning transfer outcomes is to use the LTSI as a diagnostic tool, to assess the variables that are affecting learning transfer in your organisation. It can also be used to survey participants at appropriate times.
The 16 factors for improving training ROI
There are 16 factors identified and measured in the LTSI (Holton, Bates & Ruona, 2000) that can have an impact on transfer, 11 that are factors affecting the specific learning intervention and a further five that affect training within the organisation in general. They include:
#1. Learner readiness – the extent to which respondents are prepared to enter and participate in training.
#2. Motivation to transfer – participants’ desire to apply knowledge and skills on the job.
#3. Positive personal outcomes – whether participants feel that application of training on the job will lead to positive outcomes (i.e., rewards).
#4. Negative personal outcomes – whether participants believe that not applying learning will lead to negative outcomes.
#5. Personal capacity for transfer – the extent to which participants feel they have the ability, time, energy and mental space to make changes required to transfer learning.
#6. Peer support – extent to which peers reinforce and support learning on the job.
#7. Supervisor support – extent to which participants feel their supervisors offer support and reinforcement of learning once back on the job.
#8. Supervisor sanctions – the perceived extent to which supervisors impose sanctions when participants apply newly learned skills on the job.
#9. Perceived content validity – the participants’ judgement of the total training content in terms of how accurately it reflects job requirements.
#10. Transfer design – extent to which participants perceive the training is designed to support them in applying learning on the job.
#11. Opportunity to use – degree to which participants feel they are given resources and tasks on the job that enable them to use their learning.
These five general factors aim to gauge the extent to which the LTSI respondents felt about training and their organisation in general.
#12. Transfer effort performance expectations – belief of a trainee that the effort devoted to transferring learning will result in improved job performance.
#13. Performance outcomes expectations – expectation that changes in job performance will lead to valued outcomes.
#14. Openness to change – extent to which participants believe that implementing learning and change is encouraged or discouraged by the culture of the organisation/group.
#15. Performance self-efficacy – the trainee’s belief that they have the capacity/ability to change their performance at will.
#16. Performance coaching – formal and informal indicators, in the form of feedback, from an organisation about an employee’s job performance following learning.
Each of these constructs represents an area that can have an impact on transfer, and thus the effect of a given learning intervention. So many formal interventions fail to achieve the desired objectives because of failure not in learning acquisition, but in the organisation’s design and a broader understanding of what it takes to create sustainable change in attitudes and behaviours.
Conclusion
I urge any HRD practitioner to take a deeper dive on the LTSI, to find ways to improve their return. By pragmatically discussing these 16 ‘leverage points’ and considering what you could do better, you are preparing your business to provide a supportive learning culture that will result in improved results and business impact.
