Direction Augmentation in the Evaluation of Armed Conflict Predictions

Bibliographic Data

ID11895522
AuthorsJohannes Bracher (0000-0002-3777-1410, Karlsruhe Institute of Technology, corresponding author), Lotta Rüter (0009-0001-3865-8161, Karlsruhe Institute of Technology), Fabian Krüger (0000-0002-5112-9037, Karlsruhe Institute of Technology), Sebastian Lerch (0000-0002-3467-4375, Karlsruhe Institute of Technology), Melanie Schienle (0000-0001-6000-5653, Karlsruhe Institute of Technology)
Year2023
Volume49
Issue6
Pages989-1004
Publication date2023-09-19
Peer ReviewedYes
Open AccessNo
TypeARTICLE
VenueInternational Interactions (JOURNAL)
Journal identifiersISSN: 0305-0629 • E-ISSN: 1547-7444
PublisherTaylor & Francis (PUBLISHER • GB)
DOI10.1080/03050629.2023.2255923
OpenAlexW4386849964
LanguageEN
References cited21

In many forecasting settings, there is a specific interest in predicting the sign of an outcome variable correctly in addition to its magnitude. For instance, when forecasting armed conflicts, positive and negative log-changes in monthly fatalities represent escalation and de-escalation, respectively, and have very different implications. In the ViEWS forecasting challenge, a prediction competition on state-based violence, a novel evaluation score called targeted absolute deviation with direction augmentation (TADDA) has therefore been suggested, which accounts for both for the sign and magnitude of log-changes. While it has a straightforward intuitive motivation, the empirical results of the challenge show that a no-change model always predicting a log-change of zero outperforms all submitted forecasting models under the TADDA score. We provide a statistical explanation for this phenomenon. Analyzing the properties of TADDA, we find that in order to achieve good scores, forecasters often have an incentive to predict no or only modest log-changes. In particular, there is often an incentive to report conservative point predictions considerably closer to zero than the forecaster’s actual predictive median or mean. In an empirical application, we demonstrate that a no-change model can be improved upon by tailoring predictions to the particularities of the TADDA score. We conclude by outlining some alternative scoring concepts

Competition (biology · Econometrics · Economics · Incentive · Microeconomics · Order (exchange · Sign (mathematics · Statistics · COVID-19 epidemiological studies · Forecasting Techniques and Applications · Hydrology and Drought Analysis · Mathematics

Tools

Open DOI