Rigor in Data Synthesis

A Case Study of Reliability in Meta-analysis

Bibliographic Data

ID9730706
AuthorsWilliam A Stock (Arizona State University, corresponding author), Morris A Okun (Arizona State University), Marilyn J Haring (Arizona State University), Wendy Miller (0000-0003-0863-4991, Arizona State University), CLIFFORD KINNEY (Arizona State University), ROBERT W CEURVORST (Market Facts, Inc)
Year1982
Volume11
Issue6
Pages10
Publication date1982-06-01
Peer ReviewedYes
Open AccessNo
TypeARTICLE
VenueEducational Researcher (JOURNAL)
Journal identifiersISSN: 0013-189X • E-ISSN: 1935-102X
PublisherAmerican Educational Research Association (AERA) (PUBLISHER)
DOI10.2307/1174983
OpenAlexW4250017459
LanguageEN
Citations received14
References cited23

The subject of this study was the reliability among persons coding information from primary research sources for use in a metaanalysis. Meta-analysis is a quantitative procedure for synthesizing data in primary research reports (Glass, 1976; 1977). As such, it is one of a number of quantitative procedures proposed as alternatives to the traditional, verbal review of literature. This study rests on the assumption that meta-analysis in particular, and quantitative data synthesis procedures in general, should be examined for sources of invalidity as rigorously as is the process of generating primary research. Pillemer and Light (1980) defined data synthesis as, using formal procedures for combining the results from several experiments (p. 177). Until recently, the traditional review of literature has been the primary formal mechanism for data synthesis. However, in the context of burgeoning numbers of primary research sources, alternate quantitative procedures have been proposed (see Pillemer & Light, 1980, for a review). Among the different methods that have been proposed for data synthesis, meta-analysis has been applied most widely. With regard to primary research, numerous texts call attention to sources of invalidity which investigators must try to avoid (Campbell & Stanley, 1966). However, only recently has it been suggested that quantitative data synthesis also is subject to threats to its validity, and that data synthesis studies should be examined for sources of invalidity as carefully as primary research (Glass, McGaw, & Smith, 1981; Cooper, Note 1). Although few empirical efforts have focused directly on assessing the validity of meta-analysis, critics have expressed several objections. First, they have called attention to the lack of a statistical

Meta-analysis · Reliability (semiconductor · Reliability engineering · Statistics · Computer Science · Engineering · Internal Medicine · Mathematics · Medicine · Meta-analysis and systematic reviews · Psychology

Tools

Open DOISci-Hub