NBER WORKING PAPER SERIES

HUMAN DECISIONS AND MACHINE PREDICTIONS

Jon KleinbergHimabindu Lakkaraju

Jure LeskovecJens Ludwig

Sendhil Mullainathan

Working Paper 23180http://www.nber.org/papers/w23180

NATIONAL BUREAU OF ECONOMIC RESEARCH1050 Massachusetts Avenue

Cambridge, MA 02138February 2017

We are immensely grateful to Mike Riley for meticulously and tirelessly spearheading the data analytics, with effort well above and beyond the call of duty. Thanks to David Abrams, Matt Alsdorf, Molly Cohen, Alexander Crohn, Gretchen Ruth Cusick, Tim Dierks, John Donohue, Mark DuPont, Meg Egan, Elizabeth Glazer, Judge Joan Gottschall, Nathan Hess, Karen Kane, Leslie Kellam, Angela LaScala-Gruenewald, Charles Loeffler, Anne Milgram, Lauren Raphael, Chris Rohlfs, Dan Rosenbaum, Terry Salo, Andrei Shleifer, Aaron Sojourner, James Sowerby, Cass Sunstein, Michele Sviridoff, Emily Turner, and Judge John Wasilewski for valuable assistance and comments, to Binta Diop, Nathan Hess, and Robert Webberfor help with the data, to David Welgus and Rebecca Wei for outstanding work on the data analysis, to seminar participants at Berkeley, Carnegie Mellon, Harvard, Michigan, the National Bureau of Economic Research, New York University, Northwestern, Stanford and the University of Chicago for helpful comments, to the Simons Foundation for its support of Jon Kleinberg's research, to the Stanford Data Science Initiative for its support of Jure Leskovec’s research, to the Robert Bosch Stanford Graduate Fellowship for its support of Himabindu Lakkaraju and to Tom Dunn, Ira Handler, and the MacArthur, McCormick and Pritzker foundations for their support of the University of Chicago Crime Lab and Urban Labs. The main data we analyze are provided by the New York State Division of Criminal Justice Services (DCJS), and the Office of Court Administration. The opinions, findings, and conclusions expressed in this publication are those of the authors and not those of DCJS. Neither New York State nor DCJS assumes liability for its contents or use thereof. The paper also includes analysis of data obtained from the Inter-University Consortium for Political and Social Research at the University of Michigan. Any errors and all opinions are our own. The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research.

NBER working papers are circulated for discussion and comment purposes. They have not been peer-reviewed or been subject to the review by the NBER Board of Directors that accompanies official NBER publications.

© 2017 by Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. All rights reserved. Short sections of text, not to exceed two paragraphs, may be quoted without explicit permission provided that full credit, including © notice, is given to the source.

Human Decisions and Machine PredictionsJon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil MullainathanNBER Working Paper No. 23180February 2017JEL No. C01,C54,C55,D8,H0,K0

ABSTRACT

We examine how machine learning can be used to improve and understand human decision-making. In particular, we focus on a decision that has important policy consequences. Millions of times each year, judges must decide where defendants will await trial—at home or in jail. By law, this decision hinges on the judge’s prediction of what the defendant would do if released. This is a promising machine learning application because it is a concrete prediction task for which there is a large volume of data available. Yet comparing the algorithm to the judge proves complicated. First, the data are themselves generated by prior judge decisions. We only observe crime outcomes for released defendants, not for those judges detained. This makes it hard to evaluate counterfactual decision rules based on algorithmic predictions. Second, judges may have a broader set of preferences than the single variable that the algorithm focuses on; for instance, judges may care about racial inequities or about specific crimes (such as violent crimes) rather than just overall crime risk. We deal with these problems using different econometric strategies, such as quasi-random assignment of cases to judges. Even accounting for these concerns, our results suggest potentially large welfare gains: a policy simulation shows crime can be reduced by up to 24.8% with no change in jailing rates, or jail populations can be reduced by 42.0%with no increase in crime rates. Moreover, we see reductions in all categories of crime, including violent ones. Importantly, such gains can be had while also significantly reducing the percentage of African-Americans and Hispanics in jail. We find similar results in a national dataset as well. In addition, by focusing the algorithm on predicting judges’ decisions, rather than defendant behavior, we gain some insight into decision-making: a key problem appears to be that judges to respond to ‘noise’ as if it were signal. These results suggest that while machine learning can be valuable, realizing this value requires integrating these tools into an economic framework: being clear about the link between predictions and decisions; specifying the scope of payoff functions; and constructing unbiased decision counterfactuals.

Jon KleinbergDepartment of Computer Science Department of Information Science Cornell UniversityIthaca, NY 14853kleinber@cs.cornell.edu

Himabindu LakkarajuStanford University

Gates Computer Science Building 353 Serra MallStanford, CA 94305himalv@stanford.edu

Jure LeskovecStanford University

Gates Computer Science Building 353 Serra MallStanford, CA 94305jure@cs.stanford.edu

Jens LudwigUniversity of Chicago1155 East 60th StreetChicago, IL 60637and NBERjludwig@uchicago.edu

Sendhil MullainathanDepartment of EconomicsLittauer M-18Harvard UniversityCambridge, MA 02138and NBERmullain@fas.harvard.edu

758, 027

12.0 33.2

13.0 18.7

6.3

13.0

32.9 18.7

48.2 75.8

https://storage.googleapis.com/vera-web-assets/downloads/Publications/the-price-of-jails-measuring-the-taxpayer-cost-of-local-incarceration/legacy_downloads/price-of-jails.pdf

18.8

24.5

758, 027

203, 338

x y

m y y = m(x)

L(y, y)

m(x) 0

1

y

m(x)

L[(yi,m(xi)] = −[yi × log(m(xi)) + (1− yi)× log(1−m(xi))]

β

y x

x

βj

m(xi)

0.51

xi

zi h(xi, zi)

m(xi)

m(xi) yi

m(xi) h(xi, zi)

Ri ∈ 0, 1

m(xi)

Ri = f(h(xi, zi)) = 1(h(xi, zi) < h∗)

m(xi) = yi

xi

zi

E[yi|m(xi)]

y = 56.3

Ri = f(h(xi, zi))E[f(h(xi, zi))|m(xi)] = P [h(xi, zi) < h∗|m(xi)] Corr(Ri,m(xi)) < 0

Corr(h(xi, zi),m(xi)) > 0.

1st

25th 1st

25th

h(xi, zi)

h(xi, zi)

m(xi)

6.6 9.6

13.5 22.3

E[RL] = 0.9 E[RS ] = 0.8E[RL|m(x)] E[RS |m(x)] E[RL|m(x)] −

E[RS |m(x)] = P [hL(xi, zi) < h∗L]− P [hS(xi, zi) < h∗S ]

m(xi) = yi

12.0 33.2

6.6 12.0

12.0 33.2

zi

9.9 6.6

2.8

6.6

20.1

103.0

xi ziyi = αxi + βzi α > 0 β > 0 xi = 1

zi = 0 zi = 1α α + βE[zi|xi = 1]

E[yi|RLi = 1,m(xi)] < E[yi|RS

i = 1,m(xi)]xi = 1

αα+ βE[zi|xi = 1]

RJ JRJ

c yicJm(xicJ )

yicJ = γ0 + γ1m(xicJ ) +γ2[RJ

c − RJ ] + ϵicJ γ2 < 0γ2

γ1

.397 .351

.338 .307

2.9 11.8

4.4

−16.9 5.0

−18.7 6.7

−23.4

x

x

x

y

RL(y) y

y

min{1,αy}

α = ∞α

α = ∞

i R(h(xi, zi)) = Ri

J(xi)

m(xi)

J(xi)

h(xi, zi) xi zi

z

h(xi, zi) h(x) = E[h(x, z)|x]z h(xi, zi)

R(h(xi, zi)) J(xi)

z

J

9.9 6.6

4.7 28.8

6.6

13.4 35.5

J

J

J

(J)

y = 59.2 y = 56.3

1

76.6 52.5

61.6

18.8

24.5

1

25

40

→ →

→

→

758, 027

{ } {}

R2

−

∆ ∆

∆ ∆ ∆ ∆

y = min(1,αy)

α· · · ∞

∆ ∆

∆ ∆ ∆ ∆

yijc xijc

yijc

Qjc

Fc

yijc = β0 + β1Qjc + ϵijc

F ∗ck

Pc =1

M

!

k

1(F ∗ck > Fc)

Fc

Pc

y

yP [yijc < y∗]] = .737

Dijc = 1 yijc < y∗

y