Time plays an important role in both the design and interpretation of evaluation studies of training programs. While the start and duration of a training program are closely linked to the evolution of job opportunities, the impact of training programs in the short and longer term changes over time. Neglecting these “dynamics” could lead to an unduly negative assessment of the effects of certain training schemes. Therefore, a better understanding of the dynamic relationship between different types of training and their respective labor market outcomes is essential for a better design and interpretation of evaluation studies.
For improving the understanding of how training programs work, dynamic evaluation approaches that consider changes and impacts of training programs over time are useful.
Job search training is a low-cost intervention relative to other active labor market policies.
Job search training reduces the duration of unemployment for the individual.
Occupational skills training has positive long-term effects on employment stability and earnings, which can persist over many years.
The dynamics and impacts of training participation may be misrepresented in static evaluation approaches, which could underestimate training impacts.
Job search training has no strong long-term effects on employment and earnings.
Occupational skills training ranks among the most expensive active labor market programs.
Occupational skills training initially prolongs unemployment.
Author's main message
Of the many “active” labor market policies designed to help unemployed people find work, occupational skills training ranks amongst the most expensive. However, although skills training is costly to implement and can prolong unemployment, the long-term improvements in employment stability and earnings can persist over many years. In contrast, job search-oriented training is a relatively low-cost intervention and helps to identify suitable employment opportunities for individuals more quickly. Policymakers should be aware that while job search training is the best option to activate job-seekers in the short term, investment in occupational skills training is more effective for addressing individual welfare issues and structural skills mismatch in the labor market in the long term. Moreover, a policy regime that entails the possibility of training participation at some future point during the unemployment spell speeds up the exit from unemployment of job-seekers who have not yet participated in training.
Training programs are an important instrument in the tool box of “active labor market policies”; that is, policies designed to help unemployed people find work. Periods of unemployment, or weak attachment to the labor force, may be accompanied by a deterioration in job search skills, as well as in occupational knowledge and competencies. Training programs can help prevent, or reverse, a situation in which self-sustaining employment appears to be no longer achievable. Short-term, job search-oriented programs focus on speedy job entry, whereas longer-term training supports the comprehensive development of occupational skills.
Given the multiple goals of labor market policy, and the various methods and approaches for achieving them, the relative success of these different approaches—skills development or “work-first” strategies—remains a controversial issue, and assessing their effectiveness can be a complex process.
A better understanding of the dynamic relationship between different types of training and their respective labor market outcomes is key for the design and interpretation of evaluation studies. This perspective on training is essential for formulating policy recommendations that will allow decision makers to balance their choice of different types of training programs optimally.
Discussion of pros and cons
Policy measures for the unemployed are designed to provide workers with an insurance against the risk of loss of earnings and consumption potential during periods of unemployment, and to provide job-seekers with the right incentives to take up employment again. Most existing unemployment measures include a mix of passive policies; for example, benefit schemes that compensate for the earnings loss, and active policies that are designed to help the unemployed back into work. There are two main categories of active employment policy: (i) activation strategies that focus on improving the job search activities of the unemployed (either with direct monitoring or training on job search skills); and (ii) human capital development strategies that provide training in occupational skills in order to improve the productivity and employability of the unemployed.
A major advantage of job search training is its low cost. However, a downside is that the limited set of skills provided may not be sufficient to raise employment stability and earnings in the long term. More comprehensive training programs appear to be better suited to improve long-term outcomes and address the structural mismatch between worker skills and employer demand. However, such programs rank amongst the most expensive active policies, as they are costly to run and because the period of unemployment (and in many cases, unemployment benefit payment) is prolonged.
From a theoretical perspective, the application of training policies should depend on the specific skills of a job-seeker and how their skills evolve or develop during the course of unemployment . Participating in job search training becomes more worthwhile following a period of unsuccessful searching, as it can help to prevent, or reverse, a decline in the effectiveness of the search. Participating in intensive skills training is most beneficial during the early stage of unemployment if it is clear that the occupational skills of a worker have become obsolete when they were made unemployed, for example, because their skills are not relevant for other jobs. If, however, occupational skills depreciate only slowly during the course of unemployment, intensive skills training would be better targeted at the long-term unemployed.
Empirical evaluations provide evidence on the effectiveness of training programs that can guide decision makers to choose optimally between different policy options. State-of-the-art evaluations of training programs are based on control group designs, in which the labor market outcomes of job-seekers assigned to a training program are compared to the outcomes of job-seekers who are not assigned to the program. In social experiments, job-seekers are randomly assigned to “treatment” and “control” groups. In contrast, studies based on observational data rely only on statistical methods to generate the treatment and control groups.
Causal relationships and time play an important role in the design and interpretation of evaluation studies. While the start and duration of a training program are both linked to the emergence of job opportunities before and during the period of training, the effects of the training can change over time. Participating in training programs can directly affect the current job search and labor market outcomes of the participants. In contrast, the longer-term effects of the newly acquired skills resulting from training may materialize only gradually over time.
Empirical evidence on the dynamics of training impacts
Empirical evidence on the effectiveness of training programs has increased significantly since the mid-1990s, mainly in response to a growing political demand for scientific evaluations of labor market policies, particularly in Europe. One notable study summarizes the evaluation results from more than 200 econometric evaluations of active labor market programs worldwide, of which around 50% include results on skills-intensive training and 15% on job search training . The study distinguishes between short-term impacts (less than a year) and long-term impacts (more than two years after the assignment of the program to an individual). On average, the impacts of these programs are close to zero in the short term and become more positive in the long term. The precise time profiles, however, vary across program types.
In the case of training programs, there is initially a phase of negative impact (i.e. the so-called “lock-in” effect when, during program participation, participants do not move into regular employment), which is directly related to the duration of the program. Longer programs obviously exhibit longer and deeper lock-in effects than shorter ones. The lock-in effect reflects the investment component of programs aiming at human capital, or skills, development. In the short term, participants in comprehensive training concentrate on improving their occupational skills and, as a result, miss out on job opportunities. However, to the extent that comprehensive training is effective, the labor market outcomes of participants improve once the program is completed. Positive program impacts in the medium and long term reflect the return to the training investment, as well as the return to work experience. In contrast, job search training programs are usually of short duration and aim at quick re-employment rather than human capital development. If anything, the lock-in effect is not very pronounced in job search training. If job search training is effective, positive impacts appear instantaneously. Since the investment in occupational skills is negligible with job search schemes, there is less scope for positive impacts over the medium and long term.
A few studies directly compare different types of training programs and also evaluate their impacts over longer-term horizons of up to ten years . These studies suggest that job search training raises employment prospects in the short term, whereas more intensive skills training initially reduces employment rates and leads to earnings losses. Over the longer term, comprehensive skills training tends to be more effective than job search training. The positive effects of comprehensive training persist over periods as long as eight or nine years, whereas the effects of job search training can disappear after a couple of years.
The vast majority of evaluation studies focus on cross-sectional outcomes, that is, outcomes that can be measured at one point in time, such as employment rates and average earnings. A few studies consider dynamic outcomes, such as the remaining time in unemployment or the duration of subsequent employment. A focus on dynamic outcomes is necessary in order to better understand how training programs work . If participation in a particular program increases the employment rate at a given point in time, it is not possible to know whether the program generates its beneficial effects through shortening unemployment spells or through increasing employment stability. This knowledge is important though in order to understand better the extent to which training programs activate job-seekers or improve their productivity when re-employed.
A recent study investigates the dynamic effects of short-term, job search-oriented training and long-term, human capital-intensive training in more detail . It considers impacts on time until job entry, stability of employment, and earnings. The analysis uses unique data for Germany, where both types of programs are operated simultaneously (Figure 1). The results of this study are broadly in line with the limited available evidence for other European countries and the US.
Panels (a) and (b) in Figure 1 show how the exit rates evolved since the start of the program, while panels (c) and (d) show how they evolved from the planned program end onwards. Panel (a) suggests that job-seekers exit unemployment at a faster rate with short-term training than without training, from the start of the program onwards. A participation in short-term training, which has a median planned duration of four weeks, has no noticeable lock-in effect. Rather, it raises the job-finding rates of participants instantly. The effect peaks at about 65 days following the start of the program, when the exit rate “with” short-term training exceeds that “without” by 18%. However, this advantage vanishes relatively quickly. Panel (c) shows that the difference between the exit rates with, and without, short-term training is already very small three months after the planned program end. In total, short-term training reduces the remaining time in unemployment by three weeks.
The pattern is different for long-term, human capital-intensive training that has a median planned duration of 201 days. Panel (b) of Figure 1 suggests that, during participation, people exit unemployment at a much lower rate compared to the situation of non-participation. Only after somewhat more than 200 days since program start, by which time the majority of participants have completed their training, does the exit rate increase to a level slightly above that without participation. However, concentrating on the period following the scheduled program end, in panel (d) of Figure 1, long-term training has strong and persistent positive effects on the exit rate to employment. During the first three months after the planned program end, the exit rate out of unemployment “with” participation is about twice as high as “without” participation. This effect slowly decreases over time. One year after the planned program end, the exit rate with participation is still around 40% higher than without training. In total, long-term training programs initially prolong the remaining time in unemployment by about three months.
Figure 2 displays the impacts of training on the probability of remaining employed . It shows that for both training programs the probability of remaining employed is higher “with” training than “without” at longer-elapsed durations. The vertical difference between the survival probabilities, with and without training, increases up to approximately 1.5 years after the beginning of the employment period and remains constant thereafter. After five years, 22% of the people who participated in short-term training have been employed continuously, compared to 17% in the situation without training. After five years, 29% of the people who participated in long-term training have not left their first employment, compared to 20% in the situation without training. In the first year of employment, long-term training increases earnings by 7% on average, whereas short-term training has zero effect on earnings .
The study further looks into the different effects of training along a number of personal characteristics . Participants without a formal education degree and people previously working in low- and medium-skilled manual occupations realize particularly high earnings gains after long-term training, in the order of 10–19%. Participants previously working in medium-skilled analytic and interactive jobs reap substantial gains in terms of employment stability.
In sum, long-term training programs foster the occupational advancement of different groups of job-seekers, including those with weak labor market prospects. The unemployment-prolonging effect of long-term training is smaller for people with a lower chance of exiting unemployment on their own, such as the long-term unemployed and low-skilled. Job search training reduces the time without a job more if it is started earlier during unemployment.
Another aspect that becomes salient when taking on a dynamic perspective are ex ante effects of training participation. These effects materialize already before the actual start of the program because forward-looking job-seekers anticipate that they may or will participate in a training program in the future. It is possible to distinguish between announcement effects and regime effects. The former occur as a reaction to a concrete assignment to a program starting in the future from the time when a job-seeker first learns about their future participation until the actual start of the program. Regime effects arise because job-seekers have some general knowledge about the likelihood with which they are to be assigned to a program in the near future and adjust their search behavior accordingly. For instance, job-seekers who dislike the outlook of a future participation in training may intensify their search and exit from unemployment more quickly than compared with a situation in which there exist no training programs or they are used at a lower intensity.
Evidence on the importance of announcement effects of training is very limited. The main reason for this is that caseworkers usually try to assign programs at short notice precisely to avoid distortions of job search incentives and to ensure that available program capacities are fully used. A rare exception is a study that exploits the staggered rollout of the New Deal for Young People in the UK, a job search training program for unemployment benefit claimants below the age of 25 that was initially offered only to those job-seekers whose elapsed unemployment duration reached a multiple of six months (i.e. 6, 12, 18, and so on) at a given cut-off date . This study documents a drop in the conditional probability to exit unemployment around one month before the announced start of the program. In absolute terms, the (negative) ex ante effect within one month before program start amounts to 40% of the (positive) ex post effect occurring within one month after program start. This suggests that the future program participants are attracted by the outlook of program participation and postpone part of their job search until after the start of the program.
A few studies report evidence on regime effects of active labor market policies in general and training in particular (see  and the references therein). One study uses survey data on newly unemployed job-seekers in Germany that include information on their perceived probability of program participation in the next three months, their reservation wage, and the number of search channels used. The job-seekers’ answers concerning their subjective probability of program participation vary widely: while 22% consider it very unlikely, 17% believe it to be very likely. After adjusting for differences in observable characteristics of the job-seekers, a higher perceived probability of program participation is associated with a lower reservation wage and a higher number of search channels used. This pattern is in line with a “threat effect” of active labor market policies, that is, job-seekers intensify their search efforts and are willing to accept worse paying jobs in order to avoid assignment to a program.
Another study that uses administrative data for Denmark and focuses more on training programs documents a similar threat effect. The authors use an objective measure of the probability of program participation in the near future, that is, the average hazard rate to enrol in a program, of which around three-quarters are training programs, in the upcoming three months. The estimation results suggest that a higher program intensity increases the hazard rate out of unemployment—that is, the probability of exiting unemployment conditional on still being unemployed at a given time—significantly, leading to a reduction in the average unemployment duration by two and a half weeks.
A third and the most detailed study measures “regime effects” at the level of the local employment office and the individual caseworker using administrative data for Switzerland . The study exploits the fact that employment offices and caseworkers have discretionary power in their use of active labor market policies and tend to assign the different programs at varying intensities. A more intensive use of supportive activation programs like job search and occupational skills training speeds up the exit from unemployment and slightly increases post-unemployment earnings, whereas a higher intensity of restricting policies like workfare programs and benefit sanctions increases the exit rate to employment at the cost of reducing post-unemployment earnings. While the regime effects of active labor market policies are an order of magnitude smaller than the effects of actual program participation, they are still economically important because they affect all job-seekers and not just those who actually participate. Overall, the documented regime effects suggest that typical impact estimates of actual participation in training tend to underestimate the effect of actual participation against the alternative of non-participation in the absence of any activation program.
Methodological challenges related to the dynamics of training
Experiments are often considered the gold standard for program evaluation. Classical laboratory experiments, as are common in the natural sciences, are also appealing to social scientists because the researcher has perfect control over the experimental setup, and the effectiveness of the treatment can be inferred by a simple comparison of the mean outcomes of the treated and control units. Evaluations of labor market policies are often designed and interpreted in the same way as classical laboratory experiments, even though the empirical reality may not support this approach.
In social experiments, researchers often cannot ensure compliance with the experimental protocol. It is usually not possible (nor desirable) to compel people in the treatment group to participate in the intended program from start to finish, or to deny people in the control group access to alternative programs similar to the one studied. Rather, treatment group dropout and control group substitution are common phenomena. Moreover, standard delays between the time of program assignment, program start, and program end mean that experimental subjects may revise their decisions on program participation and take-up of employment in-between. Thus, actual participation in training programs is the result of both the initial randomization and the dynamic decisions thereafter.
The same holds true in the absence of randomized assignment when job-seekers themselves, or the counselor at the employment agency, decide on program participation. Program start and continuation are, in general, closely linked to the labor market opportunities of job-seekers. For job-seekers as well as program administrators, re-employment is the primary goal. If the job search is unsuccessful, training becomes an attractive option to improve search effectiveness and occupational skills. In the US, where participants do not receive financial support during program participation, eligible job-seekers tend to enroll in programs when their opportunities to find employment on their own have become very low . In countries with comprehensive support systems for the unemployed, such as Germany and the Nordic countries, job-seekers repeatedly meet the counselor at the employment agency. If they fail to find a job, they are eventually assigned to an active labor market program , . Similarly, program participants may drop out for reasons related to their labor market opportunities.
Consequently, care has to be taken in both the design and interpretation of evaluation studies. First, social experiments of training programs often do not determine the effect of “training” against the alternative of “no training,” because not all subjects in the treatment group take part in the intended program, and some subjects in the control group obtain similar services elsewhere. Thus, the effect determined via a social experiment often corresponds to the effect of “having the option to participate in the intended program,” against the alternative of “potentially participating in a substitute program.” Similarly, some non-experimental methods focus on the effect of “starting training at a given point in time,” against the alternative of “not starting training at that point in time,” which involves the option of future participation . Research for Germany documents that the effect of “training now versus not now, but potentially in the future,” underestimates the effect of “training versus no training” by about 30% . This means that training impacts may appear low just because the alternative against which training is evaluated involves a similarly effective treatment at the same point in time, or in the future.
Second, while the early evaluation literature casts doubt on the credibility of non-experimental evaluations, later research has demonstrated that a general skepticism of non-experimental evaluations is not warranted. However, care is again needed in the design of non-experimental evaluations in order to avoid comparisons between treatment and control group units who differ in systematic ways other than participation status.
Considering the arguments outlined above, among the job-seekers who end up participating in training programs, those whose search efforts are unsuccessful tend to be over-represented, while among non-participants the successful job-seekers tend to be over-represented. Thus, a naïve comparison of labor market outcomes of participants and non-participants is likely to be biased toward finding a negative effect of training. Several studies for the US document that failure to align treatment and comparison group units in their recent earnings and employment histories contributes substantially to the bias of naïve, non-experimental methods , .
Evidence for Germany shows that evaluation results are highly sensitive to the way in which non-experimental methods take into account the unemployment experience of treatment and comparison group units . Failure to align treatment and comparison group units in their unemployment experience prior to the program start, results in impact estimates that are biased downward by a substantial amount . This bias is exacerbated by the fact that training participants are similar to successful job-seekers in terms of their demographic characteristics. Dynamic evaluation methods that model transitions between different labor market states have proven useful in aligning treated and comparison units in their labor market histories.
Third, some important questions of interest to researchers and policymakers cannot be addressed with standard research designs in which program participation is modeled as a “one-time decision,” and impacts are measured by cross-sectional comparisons of labor market outcomes. Static designs cannot be used to study dynamic issues that involve repeated decisions on training participation and take-up of employment over time. Examples of dynamic questions include:
What is the optimal sequence of training programs?
What is the optimal duration of training programs?
What is the optimal timing of training during the period of unemployment?
What is the effect of training on the remaining unemployment duration?
What is the effect of training on the stability of subsequent periods of employment?
What is necessary in order to tackle dynamic evaluation questions such as these are methods that model the time spent in different labor market states and the transitions between them , , , , .
Summary assessment of training programs
The above methodological issues have contributed to a rather pessimistic view among researchers as well as policymakers on the effectiveness and efficiency of training programs. For instance, the youth training programs under the US Job Training Partnership Act (JTPA) were cut by over 80% following a negative assessment in the National JTPA Study, a large-scale social experiment. A re-analysis shows that the experimental impact estimates were so low due to the large fractions of treatment group dropout and control group substitution . After correcting for dropout and substitution bias, however, the picture changed, suggesting that the net returns to the participants are actually large.
A cost–benefit assessment for job search training and occupational skills training in Germany, based on a fully dynamic analysis, also reaches a more positive conclusion . Occupational skills training costs almost €5,850 per participant in Germany, which is a common figure by international comparison. In contrast, job search training costs €560 per participant. In addition to this, the employment agency pays for unemployment compensation and social security contributions, totaling €1,050 per job-seeker, per month. Job search training shortens unemployment, which corresponds to a reduction in the associated transfer payments of approximately €1,850 . In addition, job search training has no adverse effects on employment stability and earnings. This suggests that job search training is cost-effective from a fiscal point of view.
The cost–benefit assessment for occupational skills training is less clear-cut, as it initially prolongs unemployment but then increases employment stability and earnings. With the additional benefit payments during the extra time in unemployment, a long-term training course results in approximately €9,000 of additional costs to the German taxpayer, compared with the situation of non-participation . The substantial positive effects on employment stability and earnings suggest that occupational skills training effectively increases the productivity of job-seekers to a similar extent as initial education. Therefore, it seems reasonable that, in the long term, occupational skills training “pays off” from a fiscal point of view, as the improved employment and earnings outcomes translate into lower transfer payments out of the unemployment insurance system and a higher labor-tax revenue. From the perspective of the job-seeker, higher earnings imply that their anticipated future potential income, and thus their lifetime welfare, increases more with long-term than with short-term training, provided that unemployment compensation during the initial lock-in phase is sufficiently high.
Limitations and gaps
Many important evaluation questions on the dynamics of training cannot be addressed with standard static research designs. Recent research has contributed to the advancement of dynamic evaluation approaches. The empirical evidence on the dynamic effects of training is, however, still limited. Existing studies often lack a comprehensive approach regarding the range of programs and outcomes considered. Evidence on the effect of the timing and sequencing of training in connection with different course contents is also scarce.
Future research could explore designs with repeated random assignment, as well as combined experimental and non-experimental approaches. Moreover, an accurate cost–benefit assessment of different types of training would require follow-up periods as long as ten or 20 years. Such evidence is missing to date.
Summary and policy advice
Training programs are an important, but controversial, instrument of active labor market policy. Early evaluation studies cast doubt on the effectiveness of long-term, skills-oriented programs in particular. The basis of this pessimistic view is rooted in a misperception of the dynamics of training programs. On the one hand, training participation is the outcome of a dynamic decision process that responds to the evolution of job opportunities. On the other hand, the impacts of training programs change dynamically over time.
For job-seekers, as well as for training program administrators, stable and long-term re-employment is the primary goal. Training becomes an attractive option when the job search is unsuccessful. Thus, unsuccessful job-seekers tend to be over-represented among participants in training. In non-experimental evaluations, failure to align treated and comparison units with their recent labor market histories can therefore lead to an underestimation of program impacts. In social experiments, standard delays between random assignment, program start, and program end, mean that experimental subjects may update and revise their decisions on program participation and take-up of employment in-between. Treatment group dropout and control group substitution are common phenomena. Hence, training impacts may appear low because the experimental evaluation does not compare training with no training, but considers two intermediate options, that is, potential participation in the intended program and potential participation in a substitute program.
Concerning the time profile of training impacts, longer-term training initially leads to negative effects, as participants miss out on job opportunities while they improve their occupational skills. However, studies that consider longer follow-up periods of more than two years, and studies that estimate the effects on employment stability, suggest that skills-intensive training generates persistent long-term gains. In contrast, the beneficial effects of short, job search-oriented training materialize instantaneously but do not persist for as long.
It is important, therefore, that policymakers are aware that while job search-oriented training can successfully activate job-seekers in the short term, investment in skills-oriented training is more effective for addressing individual welfare issues, tackling structural mismatches between worker skills and employer demand, and improving labor market outcomes in the long term.
Moreover, training programs as well as other active labor market programs also create ex ante effects that also affect those job-seekers who have not yet participated in a program. A policy regime that entails the possibility of training participation at some future point during the unemployment spell speeds up the exit from unemployment of job-seekers who have not yet participated in training. While the ex ante effects of training are much smaller than the ex post effects of actual participation, they are still economically important because they affect all job-seekers and not just those who receive the training.
The author thanks an anonymous referee and the IZA World of Labor editors for many helpful suggestions on earlier drafts. The author also thanks Bernd Fitzenberger, Volker Grossmann, and Marie Paul. Previous work of the author has been used intensively in major parts of this article . Version 2 of the article introduces a discussion on the effect of training programs on job-seekers who have not yet participated in a program, and adds new “Key references” , .
The IZA World of Labor project is committed to the IZA Guiding Code of Conduct. The author declares to have observed the principles outlined in the code.
© Aderonke Osikominu