HumRRO recently created an innovative evaluation methodology that capitalizes on the best aspects of two traditional training evaluation approaches: Kirkpatrick’s Four Levels and The Phillips ROI Methodology. Our new model – Return on Learning Experience (ROLE) – is a dynamic approach that accommodates individual, work unit, and organizational outcomes and takes into account both capacity building and continuous improvement over time. This methodology was applied in a recent effort to evaluate a large-scale leadership development program (LDP).
HumRRO was asked to conduct a comprehensive evaluation of a leadership development program (LDP) by considering two traditional training evaluation methodologies: (1) Kirkpatrick’s Four Levels (i.e., the Business Partnership Model), looking at Return on Expectation (ROE), and (2) the Phillips ROI Methodology, looking at Return on Investment (ROI). The results of an ROI pilot study indicated that neither pre-existing model for training evaluation captured the nuances of the impacts of the LDP.
WHAT WE DID
Based on the results of the pilot study, HumRRO developed the Return on Learning Experience (ROLE) Model, shown in Figure 1. The ROLE Model goes beyond the traditional training evaluation models and characterizes proximal and distal outcomes of learning by recent graduates focusing on impacts at multiple levels of the organization. Further, it acknowledges the potential for those influences on the organization to affect the learning experience of future cohorts.
The participants’ prior experience and the organization’s strategic goals (i.e., individual and cohort baseline conditions) may influence what individuals learn in the LDP and apply on the job. Application of what was learned in the program may have individual, work unit, and organizational outcomes. Individual and work unit outcomes are particularly important since they are leading indicators (if aligned with the organizational goals) that the individuals and work units are moving toward helping the organization achieve desired organizational outcomes. This dynamic model takes into account the potential for graduates’ actions to contribute to the capacity building within the organization, thus impacting the baseline conditions of future program cohorts.
The first challenge was to characterize a set of high-impact behavioral outcomes that graduates derived from their experience in the LDP. We interviewed a sample of recent graduates to help them retrospectively identify learning experiences that influenced their behavior and had an impact on their work unit or organization. Many benefits identified by the graduates were intangible and difficult to measure. In some situations, graduates were reluctant to provide detailed estimates of cost savings. Nevertheless, the interviews revealed substantial tangible benefits in terms of application of learning, work unit outcomes, and cost savings resulting from participants’ improved leadership practices. These were incorporated into our new ROLE Model.
The second challenge was to develop standardized questions that would elicit these kinds of outcomes from a wider array of graduates. Individuals in the pilot study had not reflected on the impact of their actions prior to participating in the interviews and indicated they would not have reported such rich information on a survey. Because of the challenges collecting the needed data in a traditional survey format, and the resources required to conduct interviews with all graduates of such programs, HumRRO is currently developing an interactive online approach to collect these data.
We look forward to rolling out the full methodology this spring, when our innovative, automated and rich, data collection method will facilitate use of the ROLE Model to evaluate programs in greater depth than its predecessors.