The potential of benchmark challenges in the social sciences

Social scientists aim to create explanations of the world. For each social phenomenon, scientists have proposed a myriad of theories to explain its working mechanisms. Traditionally, these theories are tested by generating hypotheses, translating them into a statistical model, and assessing the significance of the model's coefficients. Such an approach, however, often leads to the specification of a large number of (at times contradictory) models, all asserting that they capture the same theory. As things currently stand, there is no framework that allows for a comparison of these models. In this article, we argue that benchmarks can serve as a standard frame of reference that can help to determine which models fit better with empirical observations in a specific context. A benchmark is a standardized validation framework that allows for a direct comparison of the prediction accuracy of various models that address the same research problem. We outline the potential of organizing benchmark challenges in the social sciences and provide recommendations for their utilization

Data science · Empirical research · Management science · Statistics · Computer Science · Ecosystem dynamics and resilience · Innovation, Sustainability, Human-Machine Systems · Mathematics · Species Distribution and Climate Change · Artificial Intelligence

• Antagonistic cooperation of football supporters on a global scale

  Open Access•Chrisitan Brandt, Jimmy Medeiros et al.•International Review for the…•2026

  Incluir item

• The potential of benchmark challenges in the social sciences

  Open Access•Paulina Pankowska, Adrienne Mendrik et al.•Social Science Information•2024

  Incluir item

• Integrating explanation and prediction in computational social science

  Open Access•Jake M Hofman, Duncan J Watts et al.•Nature•2021

  Incluir item

• ImageNet Large Scale Visual Recognition Challenge

  Open Access•Olga Russakovsky, Jia Deng et al.•International Journal of Computer…•2015

  Incluir item

• Prediction and explanation in social systems

  Open Access•Jake M Hofman, Amit Sharma et al.•Science•2017

  Incluir item

• The Fair Guiding Principles for scientific data management and stewardship

  Open Access•Mark D Wilkinson, Michel Dumontier et al.•Scientific Data•2016

  Incluir item

• A Pragmatist’s Guide to Using Prediction in the Social Sciences

  Open Access•Megan Verhagen•Socius Sociological Research for…•2022

  Incluir item

• Fragile Families

  Open Access•Nancy E Reichman, Julien O Teitler et al.•Children and Youth Services Review•2001

  Incluir item

• Should social science be more solution-oriented

  Open Access•Duncan J Watts•Nature Human Behaviour•2017

  Incluir item

• The potential of benchmark challenges in the social sciences

  Open Access•Paulina Pankowska, Adrienne Mendrik et al.•Social Science Information•2024

  Incluir item

• Introduction to the Special Collection on the Fragile Families Challenge

  Open Access•M J Salganik, Ian Lundberg et al.•Socius Sociological Research for…•2019

  Incluir item

• Successes and Struggles with Computational Reproducibility

  Open Access•David M Liu, David Liu et al.•Socius Sociological Research for…•2019

  Incluir item

• Imputing Data for the Fragile Families Challenge

  Open Access•Brian J Goode, Debanjan Datta et al.•Socius Sociological Research for…•2019

  Incluir item

• Improving Metadata Infrastructure for Complex Surveys

  Open Access•Alexander T Kindel, Vineet Bansal et al.•Socius Sociological Research for…•2019

  Incluir item

• Privacy, Ethics, and Data Access

  Open Access•Ian Lundberg, Arvind Narayanan et al.•Socius Sociological Research for…•2019

  Incluir item