Linking Prediction and Prevention

Bibliographic Data

ID12776956
AuthorsGreg Ridgeway (0000-0001-6911-0804, National Institute of Justice, corresponding author)
Year2013
Volume12
Issue3
Pages545-550
Publication date2013-08-01
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueCriminology & Public Policy (JOURNAL)
Journal identifiersISSN: 1538-6473 • E-ISSN: 1745-9133
PublisherWiley (PUBLISHER • GB)
DOI10.1111/1745-9133.12057
OpenAlexW1892987261
LanguageEN
Citations received8
References cited4

First, I want to congratulate Berk and Bleich (2013, this issue) for their article that advances the use of modern statistical prediction methodology in criminal justice. Opportunities abound in the criminal justice system to use statistical prediction methods to improve decision making. Success depends on the quality of the prediction models, the quality of the prevention efforts, and the quality of the link between prediction and prevention. This policy essay will focus on these three components, which are essential to the effective blend of humans and machines in making criminal justice decisions. The science of prediction continues to evolve. This evolution includes the development of new statistical theory and methods, instantiation of those methods in software, improvements in data collection, and evaluation of these new data and methods on practical criminal justice issues. Berk and Bleich (2013) cover numerous foundational issues that should be well understood by quantitative criminal justice scholars. These issues include recognizing the distinction between prediction and explanation, the need for validating prediction models in terms of out-of-sample predictive performance, and that not all prediction errors carry the same cost. In addition, Berk and Bleich note that prediction methods have advanced greatly. No longer should traditional regression be the only prediction tool in the criminal justice analyst's toolbox. Tools such as random forests and boosting are readily (and freely) available thanks to open-source implementations, including my implementation of generalized boosted models that now has a large user and developer community (Ridgeway, 2013). The current criminal justice literature has sufficient examples of the use of these tools so that no mystery should remain around them. Statistical learning is a dynamic field with new methods proposed regularly. How do we know which method is right for our application? The short answer is that we cannot know until we put them to the test as Berk and Bleich (2013) did in comparing out-of-sample predictive performance of logistic regression, random forests, and boosting on a particular example. Comfort with the traditional regression methods should not prevent analysts from experimenting with machine learning methods. Machine learning is the discipline that studies how to make computers learn without explicitly programming them (Mitchell, 1997). Machine learning algorithms are methods that absorb experiences (e.g., data sets) and improve by some performance measure (e.g., misclassification rate) in conducting a task (e.g., prediction) as experiences accumulate. Classification trees and even regression are machine learning techniques as they both meet this definition, although newer algorithms are vastly superior in their ability to learn. Regardless of the origin of the method and the label attached to the method, criminal justice analysts need to stay abreast of emerging higher quality prediction methodologies so that criminal justice can reap the benefits of the new advancements. We need to monitor three issues as prediction models become more prevalent (as I am assuming that they will): concept drift, adaptive predictions, and the effect of existing prediction models. First, concept drift occurs when the definitions of features and outcomes and their relationships change over time. The probation risk-assessment model that might be working splendidly now will need an update in the future as the nature of crime and the criminal justice community changes. Changes in the economy, new technologies, new criminal opportunities, and demographic shifts can degrade the performance of existing prediction models. In practice, simply letting older data expire addresses this problem, but when to update and how much data to let expire can be challenging to determine. Second, many criminal justice issues to which we could apply prediction models are dynamic. Modern probation case management systems, for example, can collect streams of data on offenders including attendance in substance abuse treatment programs, probation officer reports, and offender location tracking. Although numerous prediction methods exist for static problems, the methodology for prediction from dynamic data streams is less developed. Consider an example. A prediction model for a new probationer predicts a low risk of reoffense. However, within 1 week of beginning probation, the probationer fails to appear for the first substance abuse treatment session. In the second week, the probationer misses a curfew by 30 minutes. In week three, the probationer finds a part-time job. The assessed risk should adapt to the timing and nature of each of these events. However, this requires new methods of estimating and fielding prediction models. Last, prediction models usually are developed on data that conflate intervention activities with the underlying crime phenomena. For example, before the deployment of any prediction model, probation officers evaluate and monitor their probationers. Even without a sophisticated prediction model, perhaps they can assess accurately the high-risk cases and can monitor in such a way as to prevent probation violations. Perhaps they have a treatment program that is phenomenally effective at suppressing criminal opportunities for high-risk probationers. A prediction model constructed from such cases might learn the rule that genuinely high-risk cases present low risk because the data show that they rarely offend. However, the prediction model has simply learned to predict suppression by the treatment program of high-likelihood crime outcomes, rather than predicting which probationers are at genuine high risk for offending. Fielding a prediction model can exacerbate this conflation by identifying clearly the high-risk cases and situations to which prevention strategies and resources are applied. For example, consider a scenario in which police saturate intersections that the prediction model predicts to be at high risk of violent crime. As a result of the saturation, no violent crime actually occurs at those intersections. An assessment of the prediction model will conclude that no crime occurs in those places the model had predicted crime would occur. Attempts to update the prediction model using new crime data will continue to conflate the previous prediction model, the prevention activities, and the suppressed crime patterns. Interventions based on prediction models will change the system under study. This problem is not easily solved without leaving some cases out of the intervention activities (e.g., continue business as usual for a high-risk probationer or leave high-risk intersections at unsaturated levels). Creating such a set of control cases is uncomfortable for practitioners, especially when they are convinced that their models and prevention efforts are effective. However, without control cases, we cannot be sure that the new prediction model is actually an improvement. Just like in clinical trials, we need to consider the principle of equipoise; when we really do not know whether the new model improves or impairs our performance, we are on solid ethical grounds to withhold an intervention until we have solid evidence. Whereas those deploying prediction models need to be cognizant of these issues, they also need to have prevention strategies at the ready to respond to the predictions. The best prediction models will be useless unless we have effective prevention strategies to implement in response to predictions. Predictive policing research largely has focused on developing algorithms for predicting where and when crime will occur. Responses to those crime predictions almost always involve reallocating police patrol resources to the times and places predicted to have crime. Naturally, these are not the only plausible interventions. For example, changes in the street design, street lighting, bus stop locations, and positioning of school personnel also could be reasonable and perhaps more durable responses to the predictions. I am particularly intrigued by the prospect of prediction models prompting the development of creative prevention models. The Chicago Police Department has created “hot lists” of residents predicted to be at high risk of being victimized. A highly accurate list will only be of value if the department can act on those predictions effectively. The Chicago Tribune reported local police paying visits to those individuals that top the list (Gorner, 2013). This intervention is relatively low cost, although its effectiveness is yet to be seen. Much of the interest surrounds the use of a prediction model to drive the intervention. However, it is possible that the program's success might have more to do with making lists than using prediction models. That is, simply making and acting on lists of residents could be the entirety of the intervention, and machines, detectives, or community members might be equally effective at drawing up the list of high-risk individuals. Other available interventions include extensive surveillance or relocation of those at high risk of victimization. Such interventions are more disruptive and much more expensive. When working with the Los Angeles Police Department (LAPD), I set up a prediction model to help their recruiting efforts (Lim, Matthies, Ridgeway, and Gifford, 2009). Some candidates were very likely to join the police force, such as those with some college and living in Los Angeles County. Other candidates never joined, such as those living outside California and reporting problematic answers on a background questionnaire. Given a candidate predicted to have a high probability of joining the LAPD, numerous candidate strategies were available to prevent these candidates from dropping out of the process. A cost-neutral approach was to prioritize those candidates for interviews and background investigations. Other strategies included additional mentoring and invitations to special events. However, we never resolved whether these intervention strategies further increased the chances of these high-viability candidates joining. So although LAPD had a readily available recruit prediction model with prospective predictive performance that could be measured, the available interventions were untested even if they seemed plausibly effective. As high-quality prediction models become more available, this should continue to prompt the development of new interventions, customized for the individual, time, place, and context of the predicted crime. These intervention strategies need at least as much attention as the prediction model. Although prediction modeling has greatly advanced and those models are prompting innovative responses, substantial barriers lie in the path to linking modern prediction methods with criminal justice interventions. In every field in which statistical prediction methods have made inroads, they face opposition from the “experts” in the field, discomforted by the idea that a black box could compete with their decision making. Guszcza and Lucker (2012: 11) noted, “[t]he biggest challenges of executing on analytics are often found where algorithmic indications should be integrated with human professional judgment.” A classic example is MYCIN (Yu et al., 1979), which is a computerized medical expert system that would take in patient characteristics and recommend treatments. In a well-designed experiment, the system outperformed practitioners in selecting the right antibiotic treatments. If 1979 technology offered such performance gains, surely we should be having even greater performance today. Even though MYCIN is just one in a series of computer-based prediction successes, these systems are not found in doctors’ offices. Heathfield (1999) noted, “theoretical and technical limitations are not the major barriers to the successful implementation ... but rather more complex professional and organizational issues are at stake.” Judges, police officers, correctional officers, probation officers, prosecutors, and other criminal justice actors are accustomed to using their own experiences and instincts in making decisions. As one attorney noted in regard to the Philadelphia Department of Probation and Parole's use of a prediction model to assess offender risk, “I'm comfortable with judges looking at a particular defendant during sentencing, looking that defendant in the eye, asking them questions, gauging their sincerity” (Fiedler, 2013). I would agree if the judge's predictive accuracy was at least as good as alternatives based on statistical risk-assessment tools. We should not be deciding based on comfort but on our criminal justice objectives: public safety effectiveness, efficiency, and fairness. In rare cases, the experts are willing to go head to head against the machine as they did in the MYCIN example. For example, in Ruger, Kim, Martin, and Quinn (2004), a panel of 83 experts, all of whom were prominent Supreme Court scholars, competed against a statistical prediction model to predict how the U.S. Supreme Court would vote on the 2002 docket of 68 cases. They correctly predicted how the Supreme Court would vote on 59% of the cases. The statistical model correctly predicted the majority opinion in 75% of the cases, substantially outperforming the experts. Such examples show that prediction models can be valuable decision support tools in the justice system, but the legacy of MYCIN indicates that acceptance is a major barrier. That is, even with an accurate prediction model and a powerful set of intervention responses, a prediction model will be useless without a decision maker willing to deploy interventions in response to the predictions. As Berk and Bleich (2013) note, Pennsylvania is but one state passing legislation to promote the use of prediction models in criminal justice. The statute says that the instruments should “help,” “may be incorporated,” and “may be an aide” in decisions, leaving room for a professional in the decision-making process. Ultimately the success of these initiatives depends on the prediction model, the prevention strategies, and the interplay of humans and machines in making criminal justice decisions. Greg Ridgeway is the acting director of the National Institute of Justice. His research spans both statistical methodology, including the development of statistical learning methods and open-source implementation of boosting, and empirical analysis in criminal justice, often using modern statistical prediction methods. He is a fellow of the American Statistical Association

Business · Artificial Intelligence in Law · Computer Science · Crime Patterns and Interventions · Legal Education and Practice Innovations

Tools

Open DOISci-HubOpen Access