What to do instead of significance testing? Calculating the ‘number of counterfactual cases needed to disturb a finding’
Bibliographic Data
| ID | 12595083 |
|---|---|
| Authors | Stephen Gorard (0000-0002-9381-5991, Durham University, corresponding author), Jonathan Gorard (King's College London, London, UK) |
| Year | 2015 |
| Volume | 19 |
| Issue | 4 |
| Pages | 481-490 |
| Publication date | 2015-10-06 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | International Journal of Social Research Methodology (JOURNAL) |
| Journal identifiers | ISSN: 1364-5579 • E-ISSN: 1464-5300 |
| Publisher | Taylor & Francis (PUBLISHER • GB) |
| DOI | 10.1080/13645579.2015.1091235 |
| OpenAlex | W1868623645 |
| Language | EN |
| Citations received | 12 |
| References cited | 18 |
This brief paper introduces a new approach to assessing the trustworthiness of research comparisons when expressed numerically. The ‘number needed to disturb’ a research finding would be the number of counterfactual values that can be added to the smallest arm of any comparison before the difference or ‘effect’ size disappears, minus the number of cases missing key values. This way of presenting the security of findings has several advantages over the use of significance tests, effect sizes and confidence intervals. It is not predicated on random sampling, full response or any specific distribution of data. It bundles together the sample size, magnitude of the finding and the level of attrition in a way that is standardised and therefore comparable between studies
Attrition · Counterfactual thinking · Econometrics · Sample (material · Sample size determination · Sampling (signal processing · Statistical hypothesis testing · Statistics · Trustworthiness · Computer Science · Mathematics · Psychology · Social Psychology · Survey Sampling and Estimation Techniques
Where Are the Costs? Using an Economic Analysis of Educational Interventions Approach to Improve the Evaluation of a Regional School Improvement Programme
Does a dance-based physical education (DBPE) intervention improve year 4 pupils’ reading comprehension attainment? Results from a pilot study in England
Best practice in conducting RCTs
The role of administrative data in the big data revolution in social science research
Can ‘Philosophy for Children’ Improve Primary School Attainment
Explaining the number of counterfactual cases needed to disturb a finding
Do we really need confidence intervals in the new statistics
What works and what fails? Evidence from seven popular literacy ‘catch-up’ schemes for the transition to secondary school in England
Can explicit teaching of knowledge improve reading attainment? An evaluation of the Core Knowledge curriculum
Is Banning Significance Testing the Best Way to Improve Applied Social Science Research? - Questions on
Damaging Real Lives through Obstinacy
Designing Mixed Methods Research by Mixing and Merging Methodologies
Nonparametric statistics for the behavioral sciences
The Relationship Between Sample Sizes and Effect Sizes in Systematic Reviews in Education
The Case Against Statistical Significance Testing
The New Statistics
Significance Tests Die Hard
Sample Attrition Bias in Randomized Experiments
Detecting and Statistically Correcting Sample Selection Bias
Investigating non-response bias in mail surveys
Theory-Testing in Psychology and Physics
| Unique citing works | 12 |
|---|---|
| Citations per year | 1,09 |
| Citation span | 2015 - 2024 (10) |
| Citation velocity | recent |
| Highly cited | No |
| Citation types | Neutral: 12 |