The dynamics of training programs for the unemployed

Empirical evidence on the dynamics of training impacts

Empirical evidence on the effectiveness of training programs has increased significantly since the mid-1990s, mainly in response to a growing political demand for scientific evaluations of labor market policies, particularly in Europe. One notable study summarizes the evaluation results from more than 200 econometric evaluations of active labor market programs worldwide, of which around 50% include results on skills-intensive training and 15% on job-search training [3]. The study distinguishes between short-term impacts (less than a year) and long-term impacts (more than two years after the assignment of the program to an individual). On average, the impacts of these programs are close to zero in the short term and become more positive in the long term. The precise time profiles, however, vary across program types.

In the case of training programs, there is initially a phase of negative impact (i.e. the so-called “lock-in” effect when, during program participation, participants do not move into regular employment), which is directly related to the duration of the program. Longer programs obviously exhibit longer and deeper lock-in effects than shorter ones. The lock-in effect reflects the investment component of programs aiming at human capital, or skills, development. In the short term, participants in comprehensive training concentrate on improving their occupational skills and, as a result, miss out on job opportunities. However, to the extent that comprehensive training is effective, the labor market outcomes of participants improve once the program is completed. Positive program impacts in the medium and long term reflect the return to the training investment, as well as the return to work experience. In contrast, job-search training programs are usually of short duration and aim at quick re-employment rather than human-capital development. If anything, the lock-in effect is not very pronounced in job-search training. If job-search training is effective, positive impacts appear instantaneously. Since the investment in occupational skills is negligible with job-search schemes, there is less scope for positive impacts over the medium and long term.

A few studies directly compare different types of training programs and also evaluate their impacts over longer-term horizons of up to ten years [1]. These studies suggest that job-search training raises employment prospects in the short term, whereas more intensive skills training initially reduces employment rates and leads to earnings losses. Over the longer term, comprehensive skills training tends to be more effective than job-search training. The positive effects of comprehensive training persist over periods as long as eight or nine years, whereas the effects of job-search training can disappear after a couple of years.

The vast majority of evaluation studies focus on cross-sectional outcomes, i.e. outcomes that can be measured at one point in time, such as employment rates and average earnings. A few studies consider dynamic outcomes, such as the remaining time in unemployment or the duration of subsequent employment. A focus on dynamic outcomes is necessary in order to better understand how training programs work [4]. If participation in a particular program increases the employment rate at a given point in time, one does not know whether the program generates its beneficial effects through shortening unemployment spells or through increasing employment stability. This knowledge is important though in order to better understand the extent to which training programs activate job-seekers or improve their productivity when re-employed.

A recent study investigates the dynamic effects of short-term, job-search-oriented training and long-term, human-capital-intensive training in more detail [1]. It considers impacts on time until job entry, stability of employment, and earnings. The analysis uses unique data for Germany, where both types of programs are operated simultaneously (Figure 1). The results of this study are broadly in line with the limited available evidence for other European countries and the US.

Panels (a) and (b) in Figure 1 show how the exit rates evolved since the start of the program, while panels (c) and (d) show how they evolved from the planned program end onwards. Panel (a) suggests that job-seekers exit unemployment at a faster rate with short-term training than without training, from the start of the program onwards. A participation in short-term training, which has a median planned duration of four weeks, has no noticeable lock-in effect. Rather, it raises the job-finding rates of participants instantly. The effect peaks at about 65 days following the start of the program, when the exit rate “with” short-term training exceeds that “without” by 18%. However, this advantage vanishes relatively quickly. Panel (c) shows that the difference between the exit rates with, and without, short-term training is already very small three months after the planned program end. In total, short-term training reduces the remaining time in unemployment by three weeks.

The pattern is different for long-term, human-capital-intensive training that has a median planned duration of 201 days. Panel (b) of Figure 1 suggests that, during participation, people exit unemployment at a much lower rate compared to the situation of non-participation. Only after somewhat more than 200 days since program start, by which time the majority of participants have completed their training, does the exit rate increase to a level slightly above that without participation. However, concentrating on the period following the scheduled program end, in panel (d) of Figure 1, long-term training has strong and persistent positive effects on the exit rate to employment. During the first three months after the planned program end, the exit rate out of unemployment “with” participation is about twice as high as “without” participation. This effect slowly decreases over time. One year after the planned program end, the exit rate with participation is still around 40% higher than without training. In total, long-term training programs initially prolong the remaining time in unemployment by about three months.

Figure 2 displays the impacts of training on the probability of remaining employed [1]. It shows that for both training programs the probability of remaining employed is higher “with” training than “without” at longer-elapsed durations. The vertical difference between the survival probabilities, with and without training, increases up to approximately 1.5 years after the beginning of the employment period and remains constant thereafter. After five years, 22% of the people who participated in short-term training have been continuously employed, compared to 17% in the situation without training. After five years, 29% of the people who participated in long-term training have not left their first employment, compared to 20% in the situation without training. In the first year of employment, long-term training increases earnings by 7% on average, whereas short-term training has zero effect on earnings [1].

The study further looks into the different effects of training along a number of personal characteristics [1]. Participants without a formal education degree and people previously working in low- and medium-skilled manual occupations realize particularly high earnings gains after long-term training, in the order of 10–19%. Participants previously working in medium-skilled analytic and interactive jobs reap substantial gains in terms of employment stability.

In sum, long-term training programs foster the occupational advancement of different groups of job-seekers, including those with weak labor market prospects. The unemployment-prolonging effect of long-term training is smaller for people with a lower chance of exiting unemployment on their own, such as the long-term unemployed and low-skilled. Job-search training reduces the time without a job more if it is started earlier during unemployment.

Methodological challenges related to the dynamics of training

Experiments are often considered to be the gold standard for program evaluation. Classical laboratory experiments, as are common in the natural sciences, are also appealing to social scientists because the researcher has perfect control over the experimental setup, and the effectiveness of the treatment can be inferred by a simple comparison of the mean outcomes of the treated and control units. Evaluations of labor market policies are often designed and interpreted in the same way as classical laboratory experiments, even though the empirical reality may not support this approach.

In social experiments, researchers often cannot ensure compliance with the experimental protocol. It is usually not possible (nor desirable) to compel people in the treatment group to participate in the intended program from start to finish, or to deny people in the control group access to alternative programs similar to the one studied. Rather, treatment-group dropout and control-group substitution are common phenomena. Moreover, standard delays between the time of program assignment, program start, and program end mean that experimental subjects may revise their decisions on program participation and take-up of employment in-between. Thus, actual participation in training programs is the result of both the initial randomization and the dynamic decisions thereafter.

The same holds true in the absence of randomized assignment when job-seekers themselves, or the counselor at the employment agency, decide on program participation. Program start and continuation are, in general, closely linked to the labor market opportunities of job-seekers. For job-seekers as well as program administrators, re-employment is the primary goal. If the job search is unsuccessful, training becomes an attractive option to improve search effectiveness and occupational skills. In the US, where participants do not receive financial support during program participation, eligible job-seekers tend to enroll in programs when their opportunities to find employment on their own have become very low [5]. In countries with comprehensive support systems for the unemployed, such as Germany and the Nordic countries, job-seekers repeatedly meet the counselor at the employment agency. If they fail to find a job, they are eventually assigned to an active labor market program [6], [7]. Similarly, program participants may drop out for reasons related to their labor market opportunities.

As a consequence, care has to be taken in both the design and interpretation of evaluation studies. First, social experiments of training programs often do not determine the effect of “training” against the alternative of “no training,” because not all subjects in the treatment group take part in the intended program, and some subjects in the control group obtain similar services elsewhere. Thus, the effect determined via a social experiment often corresponds to the effect of “having the option to participate in the intended program,” against the alternative of “potentially participating in a substitute program.” Similarly, some non-experimental methods focus on the effect of “starting training at a given point in time,” against the alternative of “not starting training at that point in time,” which involves the option of future participation [7]. Research for Germany documents that the effect of “training now versus not now, but potentially in the future,” underestimates the effect of “training versus no training” by about 30% [8]. This means that training impacts may appear low just because the alternative against which training is evaluated involves a similarly effective treatment at the same point in time, or in the future.

Second, while the early evaluation literature casts doubt on the credibility of non-experimental evaluations, later research has demonstrated that a general skepticism of non-experimental evaluations is not warranted. However, care is again needed in the design of non-experimental evaluations in order to avoid comparisons between treatment and control group units who differ in systematic ways other than the participation status.

Considering the arguments outlined above, among the job-seekers who end up participating in training programs, those whose search efforts were unsuccessful tend to be over-represented, while among non-participants the successful job-seekers tend to be over-represented. Thus, a naïve comparison of labor market outcomes of participants and non-participants is likely to be biased towards finding a negative effect of training. Several studies for the US document that failure to align treatment and comparison group units in their recent earnings and employment histories contributes substantially to the bias of naïve, non-experimental methods [9], [10].

Evidence for Germany shows that evaluation results are highly sensitive to the way in which non-experimental methods take into account the unemployment experience of treatment and comparison group units [11]. Failure to align treatment and comparison group units in their unemployment experience prior to the program start, results in impact estimates that are biased downward by a substantial amount [11]. This bias is exacerbated by the fact that training participants are similar to successful job-seekers in terms of their demographic characteristics. Dynamic evaluation methods that model transitions between different labor market states have proven useful in aligning treated and comparison units in their labor market histories.

Third, some important questions of interest to researchers and policymakers cannot be addressed with standard research designs in which program participation is modeled as a “one-time decision,” and impacts are measured by cross-sectional comparisons of labor market outcomes. Static designs cannot be used to study dynamic issues that involve repeated decisions on training participation and take-up of employment over time. Examples of dynamic questions include:

What is the optimal sequence of training programs? What is the optimal duration of training programs? What is the optimal timing of training during the period of unemployment? What is the effect of training on the remaining unemployment duration? What is the effect of training on the stability of subsequent periods of employment?

What is necessary in order to tackle dynamic evaluation questions such as these are methods that model the time spent in different labor market states and the transitions between them [1], [4], [6], [8], [12].

Summary assessment of training programs

The above methodological issues have contributed to a rather pessimistic view among researchers as well as policymakers, on the effectiveness and efficiency of training programs. For instance, the youth training programs under the US Job Training Partner Ship Act (JTPA) were cut by over 80% following a negative assessment in the National JTPA Study, a large-scale social experiment. A re-analysis shows that the experimental impact estimates were so low due to the large fractions of treatment-group dropout and control-group substitution [13]. After correcting for dropout and substitution bias, however, the picture changed, suggesting that the net returns to the participants are actually large.

A cost–benefit assessment for job-search training and occupational skills training in Germany, based on a fully dynamic analysis, also reaches a more positive conclusion [1]. Occupational skills training costs almost €5,850 per participant in Germany, which is a common figure by international comparison. In contrast, job-search training costs €560 per participant. In addition to this, the employment agency pays for unemployment compensation and social security contributions, totaling €1,050 per job-seeker, per month. Job-search training shortens unemployment, which corresponds to a reduction in the associated transfer payments of approximately €1,850 [1]. In addition, job-search training has no adverse effects on employment stability and earnings. This suggests that job-search training is cost-effective from a fiscal point of view.

The cost–benefit assessment for occupational skills training is less clear-cut, as it initially prolongs unemployment but then increases employment stability and earnings. With the additional benefit payments during the extra time in unemployment, a long-term training course results in approximately €9,000 of additional costs to the German taxpayer, compared with the situation of non-participation [1]. The substantial positive effects on employment stability and earnings suggest that occupational skills training effectively increases the productivity of job-seekers to a similar extent as initial education. Therefore, it seems reasonable that, in the long term, occupational skill training “pays off” from a fiscal point of view, as the improved employment and earnings outcomes translate into lower transfer payments out of the unemployment insurance system and a higher labor-tax revenue. From the perspective of the job-seeker, higher earnings imply that their anticipated future potential income, and thus their lifetime welfare, increases more with long-term than with short-term training, provided that unemployment compensation during the initial lock-in phase is sufficiently high.