The dynamics of training programs for the unemployed Updated

Empirical evidence on the dynamics of training impacts

Empirical evidence on the effectiveness of training programs has increased significantly since the mid-1990s, mainly in response to a growing political demand for scientific evaluations of labor market policies, particularly in Europe. One notable study summarizes the evaluation results from more than 200 econometric evaluations of active labor market programs worldwide, of which around 50% include results on skills-intensive training and 15% on job search training [3]. The study distinguishes between short-term impacts (less than a year) and long-term impacts (more than two years after the assignment of the program to an individual). On average, the impacts of these programs are close to zero in the short term and become more positive in the long term. The precise time profiles, however, vary across program types.

In the case of training programs, there is initially a phase of negative impact (i.e. the so-called “lock-in” effect when, during program participation, participants do not move into regular employment), which is directly related to the duration of the program. Longer programs obviously exhibit longer and deeper lock-in effects than shorter ones. The lock-in effect reflects the investment component of programs aiming at human capital, or skills, development. In the short term, participants in comprehensive training concentrate on improving their occupational skills and, as a result, miss out on job opportunities. However, to the extent that comprehensive training is effective, the labor market outcomes of participants improve once the program is completed. Positive program impacts in the medium and long term reflect the return to the training investment, as well as the return to work experience. In contrast, job search training programs are usually of short duration and aim at quick re-employment rather than human capital development. If anything, the lock-in effect is not very pronounced in job search training. If job search training is effective, positive impacts appear instantaneously. Since the investment in occupational skills is negligible with job search schemes, there is less scope for positive impacts over the medium and long term.

A few studies directly compare different types of training programs and also evaluate their impacts over longer-term horizons of up to ten years [1]. These studies suggest that job search training raises employment prospects in the short term, whereas more intensive skills training initially reduces employment rates and leads to earnings losses. Over the longer term, comprehensive skills training tends to be more effective than job search training. The positive effects of comprehensive training persist over periods as long as eight or nine years, whereas the effects of job search training can disappear after a couple of years.

The vast majority of evaluation studies focus on cross-sectional outcomes, that is, outcomes that can be measured at one point in time, such as employment rates and average earnings. A few studies consider dynamic outcomes, such as the remaining time in unemployment or the duration of subsequent employment. A focus on dynamic outcomes is necessary in order to better understand how training programs work [4]. If participation in a particular program increases the employment rate at a given point in time, it is not possible to know whether the program generates its beneficial effects through shortening unemployment spells or through increasing employment stability. This knowledge is important though in order to understand better the extent to which training programs activate job-seekers or improve their productivity when re-employed.

A recent study investigates the dynamic effects of short-term, job search-oriented training and long-term, human capital-intensive training in more detail [1]. It considers impacts on time until job entry, stability of employment, and earnings. The analysis uses unique data for Germany, where both types of programs are operated simultaneously (Figure 1). The results of this study are broadly in line with the limited available evidence for other European countries and the US.

Panels (a) and (b) in Figure 1 show how the exit rates evolved since the start of the program, while panels (c) and (d) show how they evolved from the planned program end onwards. Panel (a) suggests that job-seekers exit unemployment at a faster rate with short-term training than without training, from the start of the program onwards. A participation in short-term training, which has a median planned duration of four weeks, has no noticeable lock-in effect. Rather, it raises the job-finding rates of participants instantly. The effect peaks at about 65 days following the start of the program, when the exit rate “with” short-term training exceeds that “without” by 18%. However, this advantage vanishes relatively quickly. Panel (c) shows that the difference between the exit rates with, and without, short-term training is already very small three months after the planned program end. In total, short-term training reduces the remaining time in unemployment by three weeks.

The pattern is different for long-term, human capital-intensive training that has a median planned duration of 201 days. Panel (b) of Figure 1 suggests that, during participation, people exit unemployment at a much lower rate compared to the situation of non-participation. Only after somewhat more than 200 days since program start, by which time the majority of participants have completed their training, does the exit rate increase to a level slightly above that without participation. However, concentrating on the period following the scheduled program end, in panel (d) of Figure 1, long-term training has strong and persistent positive effects on the exit rate to employment. During the first three months after the planned program end, the exit rate out of unemployment “with” participation is about twice as high as “without” participation. This effect slowly decreases over time. One year after the planned program end, the exit rate with participation is still around 40% higher than without training. In total, long-term training programs initially prolong the remaining time in unemployment by about three months.

Figure 2 displays the impacts of training on the probability of remaining employed [1]. It shows that for both training programs the probability of remaining employed is higher “with” training than “without” at longer-elapsed durations. The vertical difference between the survival probabilities, with and without training, increases up to approximately 1.5 years after the beginning of the employment period and remains constant thereafter. After five years, 22% of the people who participated in short-term training have been employed continuously, compared to 17% in the situation without training. After five years, 29% of the people who participated in long-term training have not left their first employment, compared to 20% in the situation without training. In the first year of employment, long-term training increases earnings by 7% on average, whereas short-term training has zero effect on earnings [1].

The study further looks into the different effects of training along a number of personal characteristics [1]. Participants without a formal education degree and people previously working in low- and medium-skilled manual occupations realize particularly high earnings gains after long-term training, in the order of 10–19%. Participants previously working in medium-skilled analytic and interactive jobs reap substantial gains in terms of employment stability.

In sum, long-term training programs foster the occupational advancement of different groups of job-seekers, including those with weak labor market prospects. The unemployment-prolonging effect of long-term training is smaller for people with a lower chance of exiting unemployment on their own, such as the long-term unemployed and low-skilled. Job search training reduces the time without a job more if it is started earlier during unemployment.

Another aspect that becomes salient when taking on a dynamic perspective are ex ante effects of training participation. These effects materialize already before the actual start of the program because forward-looking job-seekers anticipate that they may or will participate in a training program in the future. It is possible to distinguish between announcement effects and regime effects. The former occur as a reaction to a concrete assignment to a program starting in the future from the time when a job-seeker first learns about their future participation until the actual start of the program. Regime effects arise because job-seekers have some general knowledge about the likelihood with which they are to be assigned to a program in the near future and adjust their search behavior accordingly. For instance, job-seekers who dislike the outlook of a future participation in training may intensify their search and exit from unemployment more quickly than compared with a situation in which there exist no training programs or they are used at a lower intensity.

Evidence on the importance of announcement effects of training is very limited. The main reason for this is that caseworkers usually try to assign programs at short notice precisely to avoid distortions of job search incentives and to ensure that available program capacities are fully used. A rare exception is a study that exploits the staggered rollout of the New Deal for Young People in the UK, a job search training program for unemployment benefit claimants below the age of 25 that was initially offered only to those job-seekers whose elapsed unemployment duration reached a multiple of six months (i.e. 6, 12, 18, and so on) at a given cut-off date [5]. This study documents a drop in the conditional probability to exit unemployment around one month before the announced start of the program. In absolute terms, the (negative) ex ante effect within one month before program start amounts to 40% of the (positive) ex post effect occurring within one month after program start. This suggests that the future program participants are attracted by the outlook of program participation and postpone part of their job search until after the start of the program.

A few studies report evidence on regime effects of active labor market policies in general and training in particular (see [6] and the references therein). One study uses survey data on newly unemployed job-seekers in Germany that include information on their perceived probability of program participation in the next three months, their reservation wage, and the number of search channels used. The job-seekers’ answers concerning their subjective probability of program participation vary widely: while 22% consider it very unlikely, 17% believe it to be very likely. After adjusting for differences in observable characteristics of the job-seekers, a higher perceived probability of program participation is associated with a lower reservation wage and a higher number of search channels used. This pattern is in line with a “threat effect” of active labor market policies, that is, job-seekers intensify their search efforts and are willing to accept worse paying jobs in order to avoid assignment to a program.

Another study that uses administrative data for Denmark and focuses more on training programs documents a similar threat effect. The authors use an objective measure of the probability of program participation in the near future, that is, the average hazard rate to enrol in a program, of which around three-quarters are training programs, in the upcoming three months. The estimation results suggest that a higher program intensity increases the hazard rate out of unemployment—that is, the probability of exiting unemployment conditional on still being unemployed at a given time—significantly, leading to a reduction in the average unemployment duration by two and a half weeks.

A third and the most detailed study measures “regime effects” at the level of the local employment office and the individual caseworker using administrative data for Switzerland [6]. The study exploits the fact that employment offices and caseworkers have discretionary power in their use of active labor market policies and tend to assign the different programs at varying intensities. A more intensive use of supportive activation programs like job search and occupational skills training speeds up the exit from unemployment and slightly increases post-unemployment earnings, whereas a higher intensity of restricting policies like workfare programs and benefit sanctions increases the exit rate to employment at the cost of reducing post-unemployment earnings. While the regime effects of active labor market policies are an order of magnitude smaller than the effects of actual program participation, they are still economically important because they affect all job-seekers and not just those who actually participate. Overall, the documented regime effects suggest that typical impact estimates of actual participation in training tend to underestimate the effect of actual participation against the alternative of non-participation in the absence of any activation program.

Methodological challenges related to the dynamics of training

Experiments are often considered the gold standard for program evaluation. Classical laboratory experiments, as are common in the natural sciences, are also appealing to social scientists because the researcher has perfect control over the experimental setup, and the effectiveness of the treatment can be inferred by a simple comparison of the mean outcomes of the treated and control units. Evaluations of labor market policies are often designed and interpreted in the same way as classical laboratory experiments, even though the empirical reality may not support this approach.

In social experiments, researchers often cannot ensure compliance with the experimental protocol. It is usually not possible (nor desirable) to compel people in the treatment group to participate in the intended program from start to finish, or to deny people in the control group access to alternative programs similar to the one studied. Rather, treatment group dropout and control group substitution are common phenomena. Moreover, standard delays between the time of program assignment, program start, and program end mean that experimental subjects may revise their decisions on program participation and take-up of employment in-between. Thus, actual participation in training programs is the result of both the initial randomization and the dynamic decisions thereafter.

The same holds true in the absence of randomized assignment when job-seekers themselves, or the counselor at the employment agency, decide on program participation. Program start and continuation are, in general, closely linked to the labor market opportunities of job-seekers. For job-seekers as well as program administrators, re-employment is the primary goal. If the job search is unsuccessful, training becomes an attractive option to improve search effectiveness and occupational skills. In the US, where participants do not receive financial support during program participation, eligible job-seekers tend to enroll in programs when their opportunities to find employment on their own have become very low [7]. In countries with comprehensive support systems for the unemployed, such as Germany and the Nordic countries, job-seekers repeatedly meet the counselor at the employment agency. If they fail to find a job, they are eventually assigned to an active labor market program [8], [9]. Similarly, program participants may drop out for reasons related to their labor market opportunities.

Consequently, care has to be taken in both the design and interpretation of evaluation studies. First, social experiments of training programs often do not determine the effect of “training” against the alternative of “no training,” because not all subjects in the treatment group take part in the intended program, and some subjects in the control group obtain similar services elsewhere. Thus, the effect determined via a social experiment often corresponds to the effect of “having the option to participate in the intended program,” against the alternative of “potentially participating in a substitute program.” Similarly, some non-experimental methods focus on the effect of “starting training at a given point in time,” against the alternative of “not starting training at that point in time,” which involves the option of future participation [9]. Research for Germany documents that the effect of “training now versus not now, but potentially in the future,” underestimates the effect of “training versus no training” by about 30% [10]. This means that training impacts may appear low just because the alternative against which training is evaluated involves a similarly effective treatment at the same point in time, or in the future.

Second, while the early evaluation literature casts doubt on the credibility of non-experimental evaluations, later research has demonstrated that a general skepticism of non-experimental evaluations is not warranted. However, care is again needed in the design of non-experimental evaluations in order to avoid comparisons between treatment and control group units who differ in systematic ways other than participation status.

Considering the arguments outlined above, among the job-seekers who end up participating in training programs, those whose search efforts are unsuccessful tend to be over-represented, while among non-participants the successful job-seekers tend to be over-represented. Thus, a naïve comparison of labor market outcomes of participants and non-participants is likely to be biased toward finding a negative effect of training. Several studies for the US document that failure to align treatment and comparison group units in their recent earnings and employment histories contributes substantially to the bias of naïve, non-experimental methods [11], [12].

Evidence for Germany shows that evaluation results are highly sensitive to the way in which non-experimental methods take into account the unemployment experience of treatment and comparison group units [13]. Failure to align treatment and comparison group units in their unemployment experience prior to the program start, results in impact estimates that are biased downward by a substantial amount [13]. This bias is exacerbated by the fact that training participants are similar to successful job-seekers in terms of their demographic characteristics. Dynamic evaluation methods that model transitions between different labor market states have proven useful in aligning treated and comparison units in their labor market histories.

Third, some important questions of interest to researchers and policymakers cannot be addressed with standard research designs in which program participation is modeled as a “one-time decision,” and impacts are measured by cross-sectional comparisons of labor market outcomes. Static designs cannot be used to study dynamic issues that involve repeated decisions on training participation and take-up of employment over time. Examples of dynamic questions include:

• What is the optimal sequence of training programs?

• What is the optimal duration of training programs?

• What is the optimal timing of training during the period of unemployment?

• What is the effect of training on the remaining unemployment duration?

• What is the effect of training on the stability of subsequent periods of employment?

What is necessary in order to tackle dynamic evaluation questions such as these are methods that model the time spent in different labor market states and the transitions between them [1], [4], [8], [10], [14].

Summary assessment of training programs

The above methodological issues have contributed to a rather pessimistic view among researchers as well as policymakers on the effectiveness and efficiency of training programs. For instance, the youth training programs under the US Job Training Partnership Act (JTPA) were cut by over 80% following a negative assessment in the National JTPA Study, a large-scale social experiment. A re-analysis shows that the experimental impact estimates were so low due to the large fractions of treatment group dropout and control group substitution [15]. After correcting for dropout and substitution bias, however, the picture changed, suggesting that the net returns to the participants are actually large.

A cost–benefit assessment for job search training and occupational skills training in Germany, based on a fully dynamic analysis, also reaches a more positive conclusion [1]. Occupational skills training costs almost €5,850 per participant in Germany, which is a common figure by international comparison. In contrast, job search training costs €560 per participant. In addition to this, the employment agency pays for unemployment compensation and social security contributions, totaling €1,050 per job-seeker, per month. Job search training shortens unemployment, which corresponds to a reduction in the associated transfer payments of approximately €1,850 [1]. In addition, job search training has no adverse effects on employment stability and earnings. This suggests that job search training is cost-effective from a fiscal point of view.

The cost–benefit assessment for occupational skills training is less clear-cut, as it initially prolongs unemployment but then increases employment stability and earnings. With the additional benefit payments during the extra time in unemployment, a long-term training course results in approximately €9,000 of additional costs to the German taxpayer, compared with the situation of non-participation [1]. The substantial positive effects on employment stability and earnings suggest that occupational skills training effectively increases the productivity of job-seekers to a similar extent as initial education. Therefore, it seems reasonable that, in the long term, occupational skills training “pays off” from a fiscal point of view, as the improved employment and earnings outcomes translate into lower transfer payments out of the unemployment insurance system and a higher labor-tax revenue. From the perspective of the job-seeker, higher earnings imply that their anticipated future potential income, and thus their lifetime welfare, increases more with long-term than with short-term training, provided that unemployment compensation during the initial lock-in phase is sufficiently high.