Evaluating the impact of a mandatory job search

program: evidence from a large longitudinal dataset

Luis CentenoCEETA, Lisbon

lcenteno@telepac.pt

Mario CentenoBanco de Portugal and ISEG

mcenteno@bportugal.pt

Alvaro A. NovoBanco de Portugal and ISEGI

anovo@bportugal.pt

Draft versionPlease do not quote

16th April 2004

Abstract

We assess the impact on unemployment duration of a large scale employment pro-gram implemented in Portugal during the late 1990’s and early 2000’s. Based on adataset covering over 1.5 million individuals, we construct over time nine sets of treat-ment and control groups to infer the effect of the program. We test the programeffectiveness on the unemployment duration outcome. The results across the ninecomparison sets based on the three econometric methods that we use – matching onpropensity scores, duration models and difference-in-differences methods – point to-wards an average effect of the treatment on the treated raging from 1 to 4 months.Even though this reduction is statistically significant, it is rather small in the contextof quite long unemployment spells, as the ones observed in Portugal.

Keywords: job search assistance; unemployment duration; policy evaluationJEL Classification: J18, J23, J64, J68

1

1 Introduction

It is a well-known fact in many European countries that unemployment duration is rather

long, generating problems associated with long-term unemployment, such as weak labor

market attachment. In Portugal, despite the low level of the unemployment rate, long-term

unemployment is quite common, making the low unemployment level a social and economic

issue. This pattern of unemployment duration can be seen as a trap, making long periods

in that state increasingly harder to leave due to some sort of skills depreciation or because

it sends a negative signal to the market about the unemployed.

In response to the high unemployment figures for specific labor market groups, such as

young workers, women and those aged 45 or more, European Union countries increased their

spending on active labor market policy and most countries implemented large scale programs

targeted to these specific groups. In this paper, we assess the impact on unemployment

duration of a large (mandatory) job search program implemented in Portugal during the late

1990’s and early 2000’s. The Portuguese program’s goal was to increase the employability of

the long-term unemployed (the REAGE program) and to act earlier on youth unemployment

preventing episodes of long-term unemployment at the beginning of their labor market career

(the InserJovem program).

The objective of this paper is to determine the effects of the programs comparatively to

the outcome of the individual had he/she continued to search for a job in the absence of

the job search support provided by the program. The focus is on the direct effects of the

programs; no attempt is made to assess the general equilibrium implications. However, we

should stress that displacement effects, the ones that we would expect to identified in general

equilibrium approaches, are more likely in employment assistance program (for example,

with wage subsidies) than through the kind of job search programs we are evaluating here.

With this objective in mind and the estimation issues that arise in non-experimental

studies, mainly due to the problem of missing data, we select a set of methods developed to

address such settings. The methodologies used include: (i) matching methods (Heckman,

Ichimura & Todd 1998), in which the propensity score matching is based on different defi-

nitions of neighbor; (ii) regression methods – in particular, duration models to estimate the

shift in the unemployment hazard attributable to the program, and, finally, (iii) difference-

in-difference-in-differences (Meyer 1995) modeling strategy to tackle the problems associated

with selection on non-observables.

Previous microeconometric studies of active labor market programs include Blundell,

2

Dias, Meghir & Reenen (2003) and Larsson (2003). The results of these studies are mixed.

Whereas Blundell et al for the UK find an important “program introduction effect”, the

program effect is much larger in the first quarter than later on, Larsson finds no significant

effect in the Swedish programs, and if any thing she finds a negative impact of certain aspects

of the program.

Our results point out to a small effect of the programs on unemployment duration. We

estimate that unemployment duration is reduced by 1 to 4 months, which does not represent

a large decrease in duration given that some workers can spend several years unemployed.

The paper is organized as follows. The evaluation problem, as well as the identification

and estimation of average treatment effects is addressed in section 2. The labor market

program and the data are described in Section 3. Section 5 presents the results and section

6 concludes.

2 Econometric evaluation strategies

The problem of evaluating active labor market programs has been extensively studied in

the literature (Heckman, LaLonde & Smith 1999). In recent years, a wealth of methods

to address the main problem of missing data present in all non-experimental studies has

been proposed. These methods suggest different solutions to the problem of generating

conveniently designed comparison groups necessary to perform program evaluations. Given

the non-experimental feature of these programs, the feasibility of any evaluation exercise

depends crucially on the ability that researchers have to generate such comparison groups

from the data available on the program implementation. The methodologies proposed in-

clude: (i) matching methods (Heckman et al. 1998), in which the propensity score matching

is based on different definitions of neighbor; (ii) regression methods – in particular, duration

models to estimate the shift in the unemployment hazard attributable to the program, and,

finally, (iii) difference-in-difference-in-differences (Meyer 1995) modeling strategy to tackle

the problems associated with selection on non-observables.

2.1 The evaluation problem

Active job search programs have the objective of easying/speeding the transition from un-

employment to employment. The evaluation problem faced in this study is precisely to

measure the impact of such a program in the duration of unemployment spells of the target

population – young experiencing short-lived unemployment (under 25 years and unemploy-

ment spell above 6 months) and older individuals with long-term unemployment (over 25

3

years old and spell over 12 months).

As with all non-experimental cases, this study also faces the problem of missing data in

the evaluation problem. The fact that the same individual is not observed simultaneously

in the two states – treatment and control – conditions the entire evaluation. It is, therefore,

necessary to devise sampling strategies and use methods that attempt to overcome such

observational limitations, which may have deeper statistical consequences (e.g. due to self-

selection, Heckman (1979)) in the estimation process.

2.2 Econometric Methods

2.2.1 Matching as an evaluation estimator

One of the most important issues in the evaluation of non-experimental employment pro-

grams is that of the comparision of individuals who are indeed comparable (Rubin 1977).

How such concern is addressed is key to identifying the effect of the treatment.

For that purpose, the selection of the control group can be achieved with recourse to

the non-parametric method of matching in the propensity score. In short, the methodology

compresses the indivuals’ observable multivariate information, and as such difficult to com-

pare, in an indicator – the propensity score. The distance between the individual propensity

scores is minimized to construct a control group comparable in the observables with the

treatment group (Rubin 1977, Rosenbaum & Rubin 1983).

The propensity score is defined as the conditional probability, on the observable pre-

treatment characteristics, of receiving treatment. Formally,

p(X) = Pr[T = 1|X] = E[T |X], (1)

where T is a binary variable equal to 1 if treatment is received and 0 otherwise and X is a

vector of observable characteristics. The selection of the control in p(X), rather than X, is

possible if the following conditions are verified:

1. Balancing of score.

T ⊥ X | p(X). (2)

2. Unconfoundness. If

Y1, Y0 ⊥ T | X, (3)

then

Y1, Y0 ⊥ T | p(X). (4)

4

Intuitively, the first condition requires that given the propensity score, exposition to the

treatment (or not) is independent of the observable characteristics (X). Relatively to the

unconfoundness condition, this states that if, conditional on X, the treatment status is

independent of the expected outcome resulting from treatment Y1 and from no treatment

Y0, then conditioning on p(X) preserves this important condition of independence. While

the first condition can be tested, and the samples redefined until it is verified, the second

condition cannot be tested.

p(X) is computed with regression models for binary variables, such as the probit and

logit. Because propensity scores are continuous variables, the probability that any two p(X)

are equal is zero, requiring the use of matching methods to construct comparable treatment

and control groups. The most commom matching methods are (i) stratification matching,

(ii) nearest neighbor matching, (iii) radius matching and (iv) kernel matching.1 Applications

of these statistical methods to the evaluation of active labor market programs can be found

in Blundell & Dias (2000) and Larsson (2003).

2.2.2 Duration analysis

The effect employment programs on unemployment spells can be also analyzed with dura-

tion models (see for example Richardson & der Berg (2002) and Lalive, Ours & Zweimueller

(2001)). These models estimate transition rates between the two states of interest (un-

employment and employment) as a function of the unemployment experience and other

observable characteristics of the individuals.

The duration of an unemployment spell can be represented by an hazard rate, h, or

transition rate from the unemployment state to employment. This rate is the product of

the rate at which individuals receive job offers, λ, and the probability that such an offer is

accepted by the individual given his/her reservation wage, w∗. If the distribution of wage

offers is given by F , then the hazard rate is equal to:

h = λ{1− F (w∗)}. (5)

Such models can be estimated with non-parametric methods such as the Kaplan-Meier

estimator (Kaplan & Meier 1958) or with Cox (1972) semi-parametric models. We will

report results based on the latter.

Duration models identify as risks the different exit states available to the individuals.

Our dataset identifies several options to terminate the program. Thus, we will estimate1See Becker & Ichino (2002) for a concise review of each of propensity score matching methods and an

estimation algorithm.

5

two versions of the duration modelos. Initially, we will pool together all risks, resulting in

a model that yields information on the variables that influence the transition rate. Later,

we estimate competing risks models, considering that there are two types of risks, namely,

(i) exit directly attributable to developments/actions in the treatment, and (ii) exit due to

self job search efforts conducted by the treated individuals. We will assume that risks are

independent.

2.2.3 Difference-in-difference-in-differences

The above methods control for observed characteristics to infer the impact of the treatment.

However, there are situations in which the treatment and control groups differ in terms of

unobserved characteristics, resulting in two groups that are not fully comparable. In such

circunstances, the difference-in-differences method offers a possible solution.

Let Y Dit be the potential outcome for individual i at time t given that he/she is in state

D, where D = 1, if the individual received treatment and 0 otherwise. Let treatment take

place at time t = 1. The fundamental identification problem lies in the fact that we do

not observe, at time t = 1, individual i in both states. Therefore, we cannot compute the

individual treatment effect, Y 1i1 − Y 0

i1. One can, however, estimate the average effect of the

treatment on the treated, E[Y 1i1 − Y 0

i1 | D = 1]. In order to achieve

identification, the following assumption is fundamental:

E[Y 0i1 − Y 0

i0 | D = 1] = E[Y 0i1 − Y 0

i0 | D = 0] (6)

It states that the temporal evolution of the outcome variable of treated individuals (D = 1)

in the event that they had not been exposed to the treatment would have been the same

as the observed for the individuals not exposed to the treatment D = 0. If the assumption

expressed in (6) holds, then the average treatment effect on the treated can be estimated by

the sample analogs of

{E[Yi1 | D = 1]− E[Yi1 | D = 0]} − {E[Yi0 | D = 1]− E[Yi0 | D = 0]}. (7)

There are two threats to the validity of the difference-in-differences estimator. First, if

cross-sectional data are used, compositional changes over time may invalidate the results.

Second, if the dynamic evolution of the outcome variable depends on non-observables, iden-

tification breaks down.

Meyer (1995) proposed a difference-in-difference-in-differences approach that attempts

to further correct for non-observables that threatment the validity of the difference-in-

6

Table 1: Difference-in-difference-in-differences estimator (DDD)Group Region Before (t′) After (t)

Treatment R=1 Y 11t′ Y 1

1t Y 11t′ -Y

11t

R=0 Y 10t′ Y 1

0t Y 10t′ -Y

10t

Y 11t′ -Y

10t′ Y 1

1t-Y10t Y 1

1t′ -Y11t-Y

10t′+Y 1

0t

Control R=1 Y 01t′ Y 0

1t Y 01t′ -Y

01t

R=0 Y 00t′ Y 0

0t Y 00t′ -Y

00t

Y 01t′ -Y

00t′ Y 0

1t-Y00t Y 0

1t′ -Y01t-Y

00t′+Y 0

0t

Y 11t′ -Y

10t′ -Y

01t′+Y 0

0t′ Y 11t-Y

10t-Y

01t+Y 0

0t

DDD Y 11t′ -Y

10t′ -Y

11t+Y 1

0t

Estimator -Y 01t′+Y 0

0t′+Y 01t-Y

00t

Note: Superscripts denote treatment status; subscripts refer to space and time.

differences estimator. Suppose that the data have two dimensions – space and time. Then,

it is possible to control for a space-time dimension. For instance, consider Table 1 with a

time dimension (before-after) and a space dimension (treatment region and no-treatment

region), where Y DRt stands for the outcome of group D, treatment or control, in space R,

treatment or no-treatment regions, at time t.

Consider the top panel of Table 1. Reading in row, the difference between the two rows

gives a first measure of the impact of the treatment on the treated. That is, it corrects the

outcome variable evolution of treated individuals (1st row) with the effect on pseudo-treated

(same eligibility criteria) observed in a different space R = 0; it corrects for common factors

influencing the target group.2 The D-in-D estimator is Y 11t′ -Y

11t-Y

10t′+Y 1

0t.

It is, however, possible that there are effects/shocks specific to the space of implementa-

tion, which although related with the treatment are not directly attributable to the program.

The bottom panel of Table 1 estimates such effects by performing the same exercise as in

the top panel, D-in-D, but now for a control group (different eligibility criteria). This mea-

sure, Y 01t′ -Y

01t-Y

00t′+Y 0

0t, is a measure of the non-observables affecting the D-in-D estimator

computed in the region of implementation of the treatment. Thus, the corrected effect of

the treatment is given by DDD estimator Y 11t′ -Y

10t′ -Y

11t+Y 1

0t − Y 01t′+Y 0

0t′+Y 01t-Y

00t.

3 The program and data

3.1 Description of the program

We study a large-scale program, implemented in Portugal in the context of the European

Employment Strategy. Similar programs have been subject to evaluation in other countries

(see, for example, Larsson (2003), for a study of the Swedish Youth Practice Program and

2This resembles the estimation strategy of the matching estimators.

7

Blundell et al. (2003) for the British New Deal Program). The Portuguese program is funda-

mentally a job search support program and its main goal is to improve the employability of

two specific groups of unemployed individuals: those aged less than 25 years old and unem-

ployed more than six months (the Programa InserJovem) and those over 25 and unemployed

longer than 12 months (the Programa REAGE). The program was launched in a period of

quite low unemployment rate, but with a poor performance of the unemployed in terms of

duration, and specific demographic groups were doing poorly in the labor market, namely

the less educated young and females. These two groups are more likely to be registered in

the National Employment Offices, representing a large share of our sample.

The program is composed of intensive job-search assistance and small basic skills courses.

Each individual is enrolled in a number of interviews with placement officers to help her

improve her job-search skills. In case the placement team considers necessary, the individuals

can enter a number of vocational or non-vocational training courses. The whole process

of job-search assistance ended in most cases, but not necessarily, with the elaboration of

a “Personal Job Plan”, that included detailed information on the unemployed job search

effort. According to this Plan, the unemployed was expected to meet on a regular basis

with the placement officer at the local employment office and to actively search for a job.

Unjustified rejection of job offers lead to a cancellation of any subsidies being received by the

unemployed. The program is mandatory in the sense that failing to comply with it results

in a cancellation of the workers registration and possibly of all subsidies he might have been

receiving.

The design of the program does not exclude the possibility of spillover effects from its

implementation to other workers not participating in the program. However, these general

equilibrium effects are likely to be small given the nature of the program, for example, there

are no wage subsidies.

3.2 Description of the data

The Portuguese employment agency collected data from all registered unemployed regard-

less of their “treatment status”. The dataset, comprising over 2 million observations for

over 1.5 million individuals, monitors the different features of the program and individuals

during their complete spells of unemployment. The information in the dataset includes most

demographic variables used in labor market studies (age, sex, nationality, schooling, place

of residence), including also a large number of variables related with previous labor market

experience (previous occupation, desired sector of employment, unemployment duration,

8

Table 2: Summary statitics local Employment OfficesTreatment group Control group

Variable Mean Std. Dev. Mean Std. Dev.Age (in years) 31.9269 12.8478 33.3517 43.6103Sex (Male =1) 0.3728 0.4835 0.4139 0.4941UI recipient 0.2259 0.4182 0.2831 0.4505Foreigner 0.0065 0.0805 0.0152 0.1223Disabled 0.0073 0.0850 0.0075 0.0863Marital status

Married 0.4800 0.4996 0.4867 0.4998Divorced 0.0087 0.0931 0.0101 0.1000Other 0.0253 0.1569 0.0258 0.1586Single 0.4747 0.4994 0.4658 0.4988Widow 0.0114 0.1061 0.0116 0.1069

Schooling4 years 0.2780 0.4480 0.2785 0.44826 years 0.2408 0.4276 0.2249 0.41759 years 0.1708 0.3763 0.1718 0.377211 years 0.0926 0.2898 0.0995 0.299312 years 0.0978 0.2971 0.0977 0.29703 years college 0.0254 0.1572 0.0269 0.16195 years college 0.0277 0.1642 0.0449 0.2072Mestra. 0.0001 0.0075 0.0001 0.0116Ph. D. 0.0000 0.0000 0.0000 0.0045Illiterate 0.0296 0.1695 0.0214 0.1448No school, reads 0.0373 0.1894 0.0341 0.1815

Reason to registerStudent 0.1139 0.3176 0.1027 0.3036Finished school 0.0613 0.2398 0.0510 0.2200Finished training 0.0095 0.0973 0.0045 0.0667Worked at home 0.0136 0.1160 0.0148 0.1207Laid-off 0.2004 0.4003 0.2624 0.4399Quit job 0.0321 0.1762 0.0350 0.1838End job mutual aggreement 0.0164 0.1270 0.0264 0.1605End of temporary job 0.4701 0.2854 0.4516Retired 0.0004 0.0212 0.0005 0.0217Bad working conditions 0.0062 0.0787 0.0048 0.0690Previoulsy self-employed 0.0001 0.0114 0.0001 0.0118New registration 0.0021 0.0459 0.0028 0.0527Law 17 0.0008 0.0277 0.0005 0.0231Other 0.2133 0.4096 0.2090 0.4066

Number of observations 53407 201149

9

reason for job displacement). The unemployed is observed for the complete duration of the

unemployment spell and, at the moment of termination, we can observe the destination

state (either employment, training or out of the labor force). Additionally, we have some

information of the vacancies posted in the labor office, which are possibly matched with each

unemployed.

The main limitations of our data are the lack of wage information and the difficulty

in following up the individuals after they leave the program. In fact, there is no income-

related information at the Employment Services and follow up interviews were not carried

out (although they were part of the original program design).

3.3 Sample construction

We take advantage of the wealth of information the dataset contains and of the character-

istics of the program to construct treatment and control groups using different criteria. In

particular, we explore (i) the longitudinal nature of the data, and (ii) the two sources of

variation coming from the eligibility criteria and the different implementation phases (which

generate spatial/regional and time differences).

The program design and implementation generated a natural way to construct treatment

and control groups along several dimensions. One such dimension is the eligibility criteria

(based on age and unemployment duration) and the other is the phased implementation of

the program across the Portuguese territory that generated a sequence of pilot areas. The

program implementation was gradual. The local offices of the Employment Services were

assigned to the program at different moments in time, divided by June and October of 1998,

February, May, July and November of 1999 and by April, June and September of 2000.

The treatment group includes all individuals eligible to participate in the InserJovem and

REAGE programs in the first six months of their implementation in each locacl Employment

Office. This generates a large group of individuals already unemployed at the moment the

programs were initiated in each office (see Table 2 for the dimensions of treatment and

control groups).

The choice of the comparison group was dictated by the eligibility criteria and the loca-

tion outside the initial pilot areas. Thus the comparison group takes all eligible individuals

living in the area covered by local Offices that did not implement the programs. This allows

us to have nine possible comparisons between treatment and control groups that correspond

to each of the nine dates at which different local offices started implementing the programs.

The timing of program implementation can be considered random. It was not dictated

10

Table 3: ATT∗: matching propensity score estimates, only June 1998Implementation: # #June 1998 treated control ATT Standard error t-statAll 24584 60551 -0.784 0.119 -6.577Female 15261 35191 -0.552 0.152 -3.628Male 9322 25354 -1.175 0.189 -6.208∗ Average treatment effect on the treated in months (stratification).

by special labor market conditions at the regional level (for example, regions of higher

unemployment). This implies that the treatment and control groups, can be though of as a

random draw from the Local Employment Offices at any point in time.

In Table 2, we present summary statistics for the two groups of interest. The two groups

are not very different according to the characteristics presented in the table; however, there

are some differences. Treated individuals are slightly younger, and they are more likely to

be female. Among treated individuals the share of unemployment insurance (UI) recipients

is smaller. The control group has a higher level of education, but the two groups are not

very different along this dimension. The largest differences can be found in the “reason to

register” attribute. The unemployed that were subject to treatment were more likely to

have ended a temporary job than those in the control group, who are much more likely to

have been laid-off prior to registration. These summary statistics are reassuring in terms of

our ability to match individuals in the two groups to perform our evaluation exercise.

3.4 The outcome of interest

An explicit aim of active labor market policies is to improve the employability of the unem-

ployed. Hence, a shorter unemployment duration, a higher probability of future employment

or higher employment attachment – that can operate through better matches, and higher

earnings – are possible measures of a program’s success. Unfortunately, the Employment

Services do not collect information on earnings and we were not able to get this information

from any external source. As a consequence, we will concentrate on unemployment duration

throughout this paper.

4 Empirical results

Our results, using the three different econometric methods presented above, suggest a pos-

itive impact in the employability of the those receiving treatment (youth unemployed and

medium-long term unemployed). The results are quite robust across the different estimation

11

Table 4: ATT’s∗ at the local Employment OfficesImplementation† # #

treated control ATT Standard error t-statJune 1998 24584 60551 -0.784 0.119 -6.577October 1998 2137 59586 -0.664 0.355 -1.571February 1999 3354 36917 -1.350 0.278 -4.850May 1999 12835 34626 -0.576 0.177 -3.259July 1999 2250 33120 -1.675 0.326 -5.135November 1999 852 29549 -0.986 0.498 -1.979April 2000 985 23446 -3.605 0.476 -7.578June 2000 2375 22605 0.851 0.317 2.686September 2000 3448 14217 -0.971 0.272 -3.566∗ Average treatment effect on the treated in months computed with stratification

matching on the propensity scores.† Dates of implementation at the local Employment Office level.

methods. The reduction in the average unemployment spell varies typically between 1 and

4 months.

4.1 Matching on the propensity score

The implementation of the matching method follows Becker & Ichino (2002). We first define

the treatment group of interest (there are nine such treatment groups) and then from workers

registered in local Employment Offices not yet in the program we pick the control group.

The matching process leads to a balanced treatment and control groups. For these two

groups (matched on the observables, implying a common support) the mean unemployment

duration is computed.

Table 3 and 4 present the results obtained for the first group of Offices implementing the

program, namely in June 1998. They show a positive effect of the program, which translates

into smaller unemployment duration for the treated. In Figure 1, we summarize the results

of the matching on the propensity score approach to the evaluation problem for all nine

episodes in the program implementation. The results are quite robust across the different

alternatives for the matching process that we implemented, namely stratification, nearest

neighbor and kernel methods. The estimated impact of the program suggests a reduction in

the average unemployment spell ranging from -0.6 to -3.6 months.3 The results do not show

significant differences for men and women, while the program seems to be more efficient at

reducing the unemployment duration for young individuals.3Except in June 2000, where there is an increase in the duration of 0.8 months.

12

●

● ●

●

●

●

●

●

●

−6

−4

−2

02

4

● YoungOlderAll

9806

9810

9902

9905

9907

9911

0004

0006

0009

All all−exits: ATT − Propensity score matching

●

● ●

●

●

●

●

●

●

−6

−4

−2

02

4

● YoungOlderAll

9806

9810

9902

9905

9907

9911

0004

0006

0009

Men all−exits: ATT − Propensity score matching

●● ●

●

●

●

●

●

●

−6

−4

−2

02

4

● YoungOlderAll

9806

9810

9902

9905 9907

9911

0004

0006

0009

Women all−exits: ATT − Propensity score matching

Figure 1: Propensity scores matching

13

Table 5: Unemployment duration – Cox model – all local Employment OfficesImplementation Dates Hazard Ratio Std. error Z No. of observations

June 1998 1.085 0.010 8.890 85855October 1998 1.078 0.027 3.040 62537February 1999 1.107 0.022 5.040 40903May 1999 1.105 0.014 8.080 48084July 1999 1.125 0.027 5.000 36061November 1999 1.233 0.046 5.570 31694April 2000 1.196 0.042 5.110 29940June 2000 1.000 0.023 0.000 25597September 2000 0.737 0.016 -14.370 17950Note: Coefficients on the dummy for treatment in the Cox model estimation

4.2 Unemployment duration analysis

The reduction in unemployment duration identified above should translate into a higher

unemployment hazard as a consequence of the program. This possibility was analyzed

in the context of a duration model with proportional hazards. Confirming the previous

results, there is also a sizable impact on the unemployment hazard due to the impact of the

treatment (see Figure 2 and Table 5). When we consider a single risk model, the shift in

the unemployment hazard ranges between 10 to 20 percent.

In Table A1 in the appendix, we present the full set of results for the June 1998 experi-

ment. As we can see all the covariates have the expected sign: males are more likely to leave

unemployment, as are younger workers. The disabled and those receiving UI are less likely

to terminate their registration. Also, the interaction between the treated dummy and the UI

indicator point to a direct negative effect of UI recipiency on the program efficiency. Also,

the more educated are more likely to have lower unemployment duration and students find

it harder to leave unemployment than those that already have some working experience.

Additionally, we considered a competing risks framework, explicitly dealing with the

different reasons for unemployment termination. In particular, we considered two possible

destinations: placement by the Employment Services, or self-placement by the unemployed.

The effect of the program is quite different in each of these two cases. The placement

through a program (this is a placement by the Employment Services, for example, filling a

vacancy opened at a specific Office) shows a much larger impact of the treatment than the

termination of a unemployment spell motivated by self-placement of the unemployed (see

Figure 2).

14

● ● ● ● ●● ●

●

●

−3

−2

−1

01

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

Period

1 − βRegionyear

June 1998

October 1998

February 1999

May 1999

July 1999

November 1999

April 2000

June 2000

September 2000

Unemployment duration effects

Pooled risks

Self placement risk

Placement through program risk

Figure 2: Duration models – pooled risks and competing risks

4.3 Selection on unobservables difference-in-difference-in-differen-ces

Finally, Figure 3 and Table 6 present the results from a difference-in-difference-in-differences

(DDD) application. In Table 6, we present the results for the June 1998 experiment to illus-

trate how the DDD was computed. There are two sources of differences in this application,

one is related with the eligibility to the program – based on age and unemployment duration

– and another is related with the timing of implementation of the program, which at any

point in time covers only a fraction of the offices of the national employment agency. Thus,

first we consider only the local employment offices that joined the program at a particular

moment in time and we construct treatment groups. Then, for the same time period, same

eligibility criteria, but at non-pilot locations, we construct a pseudo-treatment group.

The difference between these two groups (before-after behavior) gives us the treatment D-

in-D estimate represented in Figure 3 by the dashed line. As we can see, the treatment seems

to contribute towards a reduction in the average unemployment spell. There is, however, the

15

Table 6: DDD Estimates (in months) – Local Employment Offices, June 1998Treatment group

Before AfterTime difference

for locationExperimental Offices 24.31 24.46 0.15

0.06 0.12 0.13[85,008] [24,873]

Non-Experimental Offices 25.35 26.3 0.950.04 0.08 0.08

[194,044] [61,019]

Location difference -1.04 -1.84at a point in time 0.07 0.14

Difference-in-Differences -0.80.16

Control groupBefore After

Time differencefor location

Experimental Offices 4.59 4.31 -0.280.01 0.01 0.02

[79,539] [36,618]

Non-Experimental Offices 4.63 4.29 -0.340.01 0.01 0.01

[157,357] [69,038]

Location difference -0.04 0.02at a point in time 0.01 0.02

Difference-in-Differences 0.060.02

Difference-in-Difference-in-Differences -0.860.16

Notes:

1) Standard errors are in italics; sample size in square brackets.

2) Cells contain the averages of registered unemployment duration for the group

identified in the table aheadings.

3) The computation of the difference-in-differences in each panel is the locational

difference (or the time difference of each location) before and after the treatment

in Experimental Offices and Non Experimental Offices.

4) The computation of the difference-in-difference-in-differences is the difference-

in-differences from the upper panel minus the difference-in-differences from the

lower panel.

16

●●

●

● ●

●

●

●

●

−4

−3

−2

−1

01

●

●

●

● ●

●

●

●

●●

●

●

● ●

●

●● ●

Period

Months

June 1998

October 1998

February 1999

May 1999July 1999

November 1999

April 2000

June 2000

September 2000

Unemployment duration effects

D−in−D effect (control group)

D−in−D effect (treatment group)

DDD effect (treatment group)

Figure 3: D-in-D and DDD Estimates

17

possibility that at the time of implementation of the program there were shock to the pilot

areas that, even though might not be directly attributable to the program, are correlated

with the program. For example, the simple awareness of the program might increase the

firms’ willingness to hire. Thus, we create another level of control. For that purpose, we

repeat the above procedure, but this time we consider only the individuals that do not

meet the eligibility criteria to participate in the program. Thus, we have control groups

at pilot-agencies and at non-pilot agencies. We obtain the D-in-D estimate represented

in the Figure 3 by the dotted line. Therefore, if we take the difference between the two

D-in-D estimates, we obtain the final corrected measure of the impact of the program on

the average unemployment spell, the solid line labeled DDD, which again results in values

similar to those reported earlier.

5 Conclusions

The purpose of this study has been to evaluate labor market programs for youth and long-

term unemployed in Portugal using employment probability as a measure of effectiveness.

The programs evaluated are REAGE and InserJovem, which are mainly job-search support

programs.

We identified the average treatment effect based on the hypothesis that participation in

the various treatments, including the no-treatment state, is independent of the post-program

outcomes conditional on observable exogenous factors (as well as non-observable factors in

our diff-in-diff-in-diffs implementation). The mandatory and phased implementation char-

acteristics imposed on the design of the program allows us to be quite confident about our

identification strategy, namely the comparability of our treatment and control groups.

The results from our analysis point to a positive, but small, effect of the treatment on

unemployment duration on the treated group. We estimate a reduction between 1 and

4 months on unemployment duration. Given the generally high levels of unemployment

duration in Portugal (that can reach several years) these numbers are not impressive. They

are in fact, in line with what has been obtained for other countries and surveyed in Heckman,

Lalonde and Smith (1999).

In order to check the robustness of these findings we plan to carry out several sensitivity

checks with the data. First, looking at male and female separately. Secondly, looking

individually at the two programs might reveal some age related heterogeneity, improving

the results for the younger unemployment program. More important would be to carry out

the analysis using different measures of effectiveness, namely wages. This would be possible

18

if we can match data from different administrative sources available in Portugal.

References

Becker, S. O. & Ichino, A. (2002), ‘Estimation of average treatment effects based on propen-

sity scores’, The Stata Journal 2(4), 358–377.

Blundell, R. & Dias, M. (2000), ‘Evaluation methods for non-experimental data’, Fiscal

Studies March.

Blundell, R., Dias, M., Meghir, C. & Reenen, J. V. (2003), ‘Evaluating the employment im-

pact of a mandatory job search assistance program’, University College London 03/20.

Cox, D. R. (1972), ‘Regression models and life-tables’, Journal of the Royal Statistical Society

Series B 34, 187–220.

Heckman, J. (1979), ‘Sample selection bias as a specification error’, Econometrica 47, 153–

161.

Heckman, J., Ichimura, H. & Todd, P. (1998), ‘Matching as an economic estimator’, The

Review of Economic Studies 62(2), 261–294.

Heckman, J., LaLonde, R. & Smith, J. (1999), ‘The economics and econometrics of active

labor market programs in’, Handbook of Labor Economics 3A.

Kaplan, E. L. & Meier, P. (1958), ‘Nonparametric estimation from incomplete observations’,

Journal of the American Statistical Association 53, 457–481.

Lalive, R., Ours, J. C. & Zweimueller, J. (2001), ‘The impact of active labor market programs

on the duration of unemployment’, IEW, University of Zurich 51, Working paper.

Larsson, L. (2003), ‘Evaluation of swedish youth labor market programs’, 38(4), 891–927.

Meyer, B. D. (1995), ‘Natural and quasi-experiments in economics’, Journal of Business &

Economic Statistics 13, 151–162.

Richardson, K. & der Berg, G. V. (2002), ‘The effect of vocational employment training on

the individual transition rate from unemployment to work’, IFAU 2002:8.

Rosenbaum, P. & Rubin, D. (1983), ‘The central role of the propensity score in observational

studies for causal effects’, Biometrika 70, 41–55.

19

Rubin, D. (1977), ‘Assignment to a treatment group on the basis of a covariate’, Journal of

Educational Statistics 2, 1–26.

20

Appendix

Table A1 – Unemployment duration: local Employment Offices, June 1998No. of individuals: 85855 No. of observations: 85855Number of exits: 85855 Log likelihoood: -849391.44Time at risk (months): 2212025 Likel. Ratio (χ2

36): 84579.08

Cox proportional hazard model Haz. Ratio Std. error Z Pr > |Z|Participation 1.085 0.010 8.890 0.000Sex = Male 1.173 0.009 21.550 0.000Age 0.899 0.002 -51.660 0.000Age2 1.001 0.000 37.740 0.000Unemployment insurance 0.808 0.008 -21.780 0.000U. insurance x Participation 0.976 0.016 -1.460 0.144Foreigner 1.103 0.038 2.880 0.004Disability 0.738 0.029 -7.780 0.000No. of posted local vacancies 1.005 0.000 130.340 0.000No. of individual registrations 1.077 0.003 23.380 0.000Schooling

6 years 0.990 0.011 -0.920 0.3569 years 1.049 0.012 4.130 0.00011 years 0.937 0.012 -4.910 0.00012 years 1.654 0.027 31.140 0.0003 years college 1.237 0.027 9.550 0.0005 years college 1.251 0.026 10.730 0.000Master 1.161 0.519 0.330 0.739Ph. D. 1.728 1.728 0.550 0.584Illiterate 1.040 0.024 1.690 0.092No school, reads 1.026 0.021 1.270 0.202

Reason to registerFinished school 1.531 0.028 23.120 0.000Finished training 1.679 0.083 10.530 0.000Worked at home 1.263 0.044 6.740 0.000Laid-off 1.125 0.016 8.530 0.000Quit job 1.136 0.024 5.970 0.000End job mutual aggreement 1.660 0.061 13.800 0.000End of temporary job 1.134 0.014 9.900 0.000Retired 1.504 0.263 2.340 0.019Bad working conditions 0.924 0.049 -1.510 0.132Law 17 3.560 0.652 6.940 0.000Other 1.014 0.013 1.030 0.301

Marital StatusDivorced 1.846 0.091 12.400 0.000Single 0.906 0.009 -10.020 0.000Widow 1.191 0.040 5.200 0.000Other 1.048 0.022 2.270 0.023

Registration Year = 1999 18.986 0.338 165.250 0.000

Note: For categorical variables the excluded classes are:

Schooling: 4 years; Reason to register: Student; Marital Status: Married

Participation equals 1 for the unemployed individuals registered in local Employment

Offices (EO) that implemented the program in June 1998 and 0 for the unemployed

registered in the other local EO that by that date had not implemented the program.

21