IS THERE A PREDICTIVE MODEL FOR TRAINING?
Can A Predictive Model for the Quantitative Benefits of Training Be Established?
By Arnie Witchel
Copyright 2003: Witchel & Associates
The traditional approach to measuring training effectiveness measures the efficacy of training after it has taken place. This article examines whether the quantitative benefits of training and increased efficiency expected from training can be predicted prior to the training taking place. The hypothesis indicates a positive relationship can be established between training and the quantitative benefits and increased efficiency expected from training prior to training taking place. A review of the literature regarding training, training measurement, and the intervening variables is conducted. The results indicate that a mathematical predictor of the positive and quantitative benefits of training can not be established due to the number of unquantifiable intervening variables. Guidelines for assessing variables for training prior to investment are suggested.
Training can be approached as a cost or as an investment in human resources. Sometimes it is eliminated from organizational budgets prior to other items because the benefits of training are difficult to quantify and predict. The traditional approach to training effectiveness measures four criteria for assessing training effectiveness: affective reactions or responses to training, knowledge acquisition and retention, changes in job related behavior, and improvements in organizational results (Kirkpatrick, 1959a, 1959b, 1960a, 1960b). Since Kirkpatrick (1959a, 1959b, 1960a, 1960b) advocated the measurement of training benefits after the training occurs, studies have increasingly sought more sophisticated methods of assessing the effectiveness of training in the workplace. However, this approach only serves to measure utility after the training has taken place.
While some studies have examined Kirkpatrick's taxonomy and suggested improvements and revisions in its design (Alliger, Tannenbaum, Bennett, Traver & Shotland, 1997), others have advocated strategies for evaluating training within organizational constraints (Tannenbaum & Woods, 1992). The literature tends to focus on the reactive evaluation of training as a strategy for measuring its effectiveness. Many articles (Phillips, 1996) have been written advocating the return on investment (ROI) strategy of measuring cost effectiveness. While this formulaic approach applied to Kirkpatrick's model does provide some measure of evaluating the quantitative training benefit after training has taken place, it does not provide predictive value. Other approaches argue that taking a financial approach to training is an error in itself (Brinkerhoff, 1997); by approaching the problem in this manner, training is reduced to mere instrumentality and not linked to a more complicated issue: performance.
More recent trends toward avoiding these mental models of return on investment include approaching training as economic value added to the organization (Burkette & Headley, 1997). The advantages to this approach have allowed companies to resist the temptation to address training as a short term gain in investment and take a long-term approach to training as a capital investment in the organization. Some companies have embraced this approach because it is the only method they can recognize for training expenses to make fiscal sense (Peak, 1997). This gives managers an incentive to make the investment, but it does not give them criteria to predict what the investment will be worth to the organization. Rather, this approach allows companies to measure training as a general investment in the organization's overall growth, similar to the investment in research and development. However, not all investment in research is proven to have cost benefit to the organization.
The problematic approach to training as investment goes beyond the approach organizations take toward training. While the body of literature suggests general approaches to persuade managers and organizations to invest in training or justify training after the fact, the benefit of a model that would allow organizations to estimate the cost benefit of training investment prior to the actual training has doubtless appeal. Based on the research conducted, can a model be constructed that will aid decision makers in quantifying the benefits of training and increased efficiency before training takes place? Can the ratio of increased efficiency and cost benefits from training be mathematically calculated and determined by the cost of training prior to the training taking place?
The problem is significant as Human Resource Managers approach future trends in training. The American Society of Trainers and Developers (ASTD), in its 1997 report on the training industry and trends, predict that the number one trend in the next three years will be a movement from skills training to a shift from training to performance (Bassi, Cheney & Van Buren, 1997). A model that can aid prediction of the efficiency and effectiveness of training prior to training taking place would contribute significantly to that shift in emphasis from measuring the skills attained to measuring performance.
The hypothesis is that a positive quantifiable relationship can be established between training and the benefits of training and the increased efficiency expected from training prior to the training taking place. This research will add to the literature already conducted by investigating whether or not this model can be constructed and the variables that would be significant in its development.
Review of the Literature
The traditional approach to training effectiveness measures four criteria for assessing training effectiveness: the trainees' affective reactions to training, knowledge acquisition and retention, changes in job behavior, and improvements in organizational results (Kirkpatrick, 1959a, 1959b, 1960a, 1960b). This approach to training does not have predictive value, but measures training effectiveness after the fact. Nor does this approach measure the improvement in performance. Rather, it measures the effectiveness of training as if it were an isolated event with direct cause and effect properties, that training causes performance improvement (Brinkerhoff, 1997, p. 15).
One difficulty in assessing quantifiable benefits of training effectiveness prior to training taking place is the difference in approaching training as a pay back proposition versus a pay forward opportunity. The pay back approach views training as "a return on training investment that is measurable in financial (turnover, profit, etc.) or analogous terms (increases in sales, conversions of leads to sales, etc.). Accountants call this 'direct return'" (Lee, 1996, p. 30). This traditional approach assumes that the proper assessment is ROI based, and only tangible, measurable returns can be measured. "The ROI calculation process requires that we 'isolate' the effects of training, so that the value of training itself--the training alone--can be assessed" (Brinkerhoff, 1997, p. 18). The pay forward view, however, sees benefits in training that can not be measured in direct financial terms, but support the organization's ability to learn and change (Lee, 1996). The measurement criteria in the pay forward view moves from a financial base to performance based. However, data and measurement criteria for the pay forward view are difficult to gather and assess.
It is tempting to take the ROI approach and simply project into the future a mathematical formula for calculating the expected return on investment that future behaviors will deliver. This is using the pay back approach and projecting the return expected in increased efficiency and effectiveness through training. This leads to a continuation of the accounting mentality toward training. Odiorne and Rummler (1988, p. 51) go so far as to state "Cost benefit is always to be completed in tangible numbers. If you are confident that there will be some intangible benefits, you can always state those as a 'note' to the statement, but the specific costs and specific benefits should be polished until the numbers are as accurate as possible." This approach assumes total training effectiveness, doesn't consider intervening variables that might affect the training outcomes. It also places the training budget in direct competition for funds with other budgets, including research and development, capital equipment, and marketing (Odiorne & Rummler, 1988). While it does address the numbers issues (what the mathematical increase in efficiency is expected to return for the dollars invested), it does not address the efficiency or effectiveness of the training itself. In other words, it attempts to isolate the after effects of training prior to training taking place, but chooses to ignore the events that may affect the performance gained through the training.
Another difficulty in this area of predictive modeling is establishing predictive performance criteria for measurement of soft skills training, which have less objective outcomes, such as management training. Skills based training, such as technical or sales oriented training, are easier to evaluate than human relations or management training. This may discourage efforts to measure performance and lead to skepticism regarding the ability to measure and evaluate training of this type (Tannenbaum & Woods, 1992). Management training also may have a different effect size and less immediate utility (and is therefore more difficult to measure) versus skills based training (Morrow, Jarrett & Rupinski, 1997). For soft skills or managerial training, which types of performance will the training measure? How will the effect of the training be measured? Over what period of time? Behavioral performances tend to be qualitatively different than skills based training and more difficult to measure in terms of transfer maintenance of training (Morrow, et al., 1997). Skills based training can be measured directly: Either the skill is gained and transferred to the workplace or it isn't. Performance based training is not as easily measured, nor are criteria as easily established.
Intervening variables also complicate the predictive economic value and efficiency of training prior to the training event. These variables are either intrinsic to the trainee or extrinsic. Intrinsic variables include individual characteristics that the trainee may have. Individual characteristics, such as the trainee's attitude and motivation toward the training, are extremely difficult to quantify and predict. However, they do moderate training effectiveness. Not only do the individual's attitude and motivation toward training affect the success of training, so do factors such as perceived situational constraints (Mathieu, Tannenbaum & Salas, 1992). These situational constraints can negatively impact the trainee's motivation and vary from involuntary training (versus willingly participating in the training) to lack of information about the training, lack of time to devote to the training, or lack of necessary materials to complete the training (Mathieu, et al., 1992). As Aliger, et al. point out, this is a step that Kirkpatrick failed to identify in his use of the term "behavior" as a criteria for training effectiveness (1997). There is a difference between can do and does do or will do (Alliger, et al., 1997). The intervening variable (motivation) between retaining the knowledge and transferring the skill affects the measurement of the training's success after the fact, but it is also critical in predicting whether training the individual will lead to increased effectiveness and efficiency on the job, demonstrating an increase in the trainee's performance.
The intrinsic intervening variables are not limited to motivation or perceptions of situational constraints, however. Other individual characteristics that serve as intervening variables in delivering a predictive model include individual knowledge, skills and experience, which should be used as antecedent criteria to evaluate the trainee's readiness for training (Tracey & Tews, 1995). Less clear is the issue of job commitment. Tracey & Tews (1995, p.40) assert that if "individuals possess a high degree of commitment to their jobs and the organization, it is likely that they will view training as worthwhile and be committed to the opportunity to acquire new knowledge and skills." Other studies find no support for the hypothesis that high levels of career planning or job involvement affect higher training motivation (Mathieu, et al., 1992).
Self-efficacy does affect training effectiveness (Mathieu, Martineau & Tannenbaum, 1993). Self-efficacy is not concerned with the skills one learns, but with judgments of what the trainee can do with the skills the trainee possesses or gains. Self-efficacy levels at the conclusion of training, and in the process of training itself, have significant correlations with the transfer of training and the measurement of job performance (Mathieu, et al.). While motivation, skills, knowledge and abilities serve as antecedents for the development of self-efficacy, efficacy is obtained through repeated task related experiences (Bandura, 1986) and is measured during and after the training takes place. Mathieu, et al. hypothesize that self-efficacy is affected by initial performance, achievement motivation, choice and situational constraints.
Although not predictable, there must be some type of positive judgment not only about the efficacy of the training by the trainee, but also positive reactions to the training itself during the training experience. This positive affective reaction moderates the relationship between motivation while training and the actual learning that takes place during training (Mathieu, et. al., 1992).
In contrast to intrinsic intervening variables, there are also variables extrinsic to the trainee that will affect the training's efficiency and effectiveness. Primary among these is the work environment itself. There may be a direct link between an organization's culture and climate and the use of skills learned in training (Tracey, Tannenbaum, & Kavanagh, 1995). The social norms of the organization and open encouragement to use the new skills can also affect the transfer of training into performance (Tracey & Tews, 1995). The maturity of the organization will affect the training outcome if it approaches training as a valuable entity. Firms with little maturity may run training events with no clear objectives, while firms at higher maturity levels view the human resources function, and training, as strategic change and development agents (Lee, 1996).
Other extrinsic factors include accountability, reward, and performance appraisal systems. Trainees report greater intentions to use the training when organizations hold employees accountable for the training they receive (Baldwin & Magjuka, 1991). Training professionals "definitively stated that there must be some type of accountability for trainees to use their newly acquired knowledge and skills. Performance appraisal systems should also be used to account for the training employees are expected to demonstrate" (Tracey & Tews, 1995, p.42).
The construct and nature, or the difficulty and delivery of the training itself, must also be addressed. Organizations that construct training based on sophisticated needs analysis may have more effective training than those that do not (Morrow, et al., 1997). In addition, training must be maintained for some time in order to effect transference (Morrow, et. al.). The nature of the training must be must also be considered. If the training is difficult or difficult to measure, the predictive value and assessment method of the training may be questioned. If the delivery of the training is not done well, the affective reaction to the training may interfere with training application and outcomes (Mathieu, et al., 1992). Reactions to training serve not only as a moderator of training relationships, but also as a mediator (Mathieu, et. al.).
Due to the large number of intervening variables, both intrinsic and extrinsic, a quantitative predictive model of the value of training prior to training taking place can not be established based on the research done to date. There is much room for further study in this area. While tempting, the increased efficiency and cost benefits from training can not be mathematically calculated and determined solely by the cost of training prior to the training taking place. One can not simply set an estimate of the cost benefits expected to be obtained from training and calculate backwards to determine if the investment is worthy. The current body of literature argues for the acceptance of the null hypothesis. Due to the large number of unquantifiable intervening variables, a positive quantifiable relationship can not be established between training and the quantitative benefits of training and the increased efficiency expected from training prior to the training taking place. As Tracey and Tews (1995, p. 42) point out: "Simply put, effective training depends on events that occur before, during, and after a training program, which do not necessarily relate directly to training activities." Perhaps Brinkerhoff (1997, p. 20) is correct in asserting that the cost benefit is not the proper approach to the problem, and the questions we should be asking are: "(1) What is keeping our employees from using learning to improve performance? (2) What is keeping our employees from using performance to learn? (3) What can we do to remove these obstacles and what happens when we try to remove them?"
Managers can evaluate these variables to assess whether the investment into training will produce the expected efficiencies and improved performance for the organization. Considering what the training is trying to achieve in terms of performance, and how the performance will be measured, allows the clarification of needs and enhances clear objectives for the training and assessment methods, including time and effect. Considering and weighing intrinsic variables, such as individual attitude and motivation, skill and readiness for training, self efficacy, as well as situational constraints on the group or individual to be trained, allows managers to identify who should be trained and to what extent. It also aids conservation of training dollars by identifying those who are not ready for training at a given point in time due to the intervening intrinsic variables. The construct and content of the training should be assessed. Whether it is skill based or soft skill based, the difficulty and delivery of the training itself needs consideration. This will aid in indicating not only the short versus long term value of the training, but also the length of time that may be needed to effectively transfer training and assess results of the training. Weighing extrinsic variables, such as the culture of the organization, accountability for training behaviors, the maturity of the organization's training efforts, and the amount of maintenance the training needs to effect transfer, can help identify the future resources that will be needed to bring the training from merely an affective reaction to actual performance within the organization. By considering these variables prior to the investment of training expenditures, the cost versus the expected benefits of training in terms of increased efficiency and effectiveness can more carefully be assessed and better training decisions can be justified for organizational performance goals.
Alliger, G.M., Tannenbaum, S.I., Bennett, W. Jr., Traver, H., & Shotland, A. (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50, 341(18).
Baldwin, T.T. & Magjuka, R.J. (1991). Organizational training and signals of importance: linking program outcomes to pre-training expectations. Human Resource Development, 2, 25-36.
Bandura, A. (1986). Social foundations of thought and action. Englewood Cliffs, NJ: Prentice-Hall.
Bassi, L.J., Cheney, S., & Van Buren, M. (1997). Training industry trends 1997. Training & Development, 51(11), 46-59.
Brinkerhoff, R.O. (1997). Invited reaction: Response to Parsons. Human Resource Development Quarterly, 8(1), 15-21.
Burkette, G.D., & Hedley, T.P. (1997). The truth about economic value added. The CPA Journal, 67(7), 46(4).
Kirkpatrick, D.L. (1959a). Techniques for evaluating training programs. Journal of ASTD, 13,
Kirkpatrick, D.L. (1959b). Techniques for evaluating training programs: Part 2-Learning. Journal of ASTD, 13, 21-26.
Kirkpatrick, D.L. (1960a). Techniques for evaluating training programs: Part 3-Behavior. Journal of ASTD, 14, 13-18.
Kirkpatrick, D.L. (1960b). Techniques for evaluating training programs: Part 4: Results. Journal of ASTD, 14, 28-32.
Lee, R. (1996). The 'pay-forward' view of training. People Management, 2(3), 30(3).
Mathieu, J.E., Martineau, J.W., & Tannenbaum, S.I. (1993). Individual and situational influences on the development of self-efficacy: implications for training effectiveness. Personnel Psychology, 46, 125(23).
Mathieu, J.E., Tannenbaum, S.I., & Salas, E. (1992). Influences of individual and situational characteristics on measures of training effectiveness. Academy of Management Journal, 35, 828(20).
Morrow, C. C., Jarrett, M.Q., & Rupinski, M.T. (1997). An investigation of the effect and economic utility of corporate-wide training. Personnel Psychology, 50, 91-119.
Odiorne, G.S., & Rummler, G.A. (1988). Training and development: a guide for professionals. Chicago: Commerce Clearing House.
Peak, M. H. (1997). Training: No longer for the fainthearted. Management Review, 86(2), 23(5).
Phillips, J.J. (1996). How much is the training worth? Training & Development, 50(4), 20(5).
Tannenbaum, S.I. & Woods, S.B. (1992). Determining a strategy for evaluating training: operating within organizational constraints. Human Resource Planning, 15, 63(17).
Tracey, J.B., Tannenbaum, S.I., & Kavanagh, M.J. (1995). Applying trained skills on the job: the importance of the work environment. Journal of Applied Psychology, 80, 239-252.
Tracey, J.B. & Tews, M.J. (1995). Training effectiveness: accounting for individual characteristics and the work environment. Cornell Hotel & Restaurant Administration Quarterly, 36(6), 36(7).
To return to the White Papers, click here: