Justifying Aggregation With Consensus-Based Constructs

A Review and Examination of Cutoff Values for Common Aggregation Indices

Bibliographic Data

ID23329680
AuthorsDavid J Woehr (0000-0002-9974-7512, Department of Management, University of North Carolina, Charlotte, NC, USA, corresponding author), Andrew C Loignon (0000-0002-0073-3963, Department of Management, University of North Carolina, Charlotte, NC, USA), Paul B Schmidt (Department of Management, University of North Carolina, Charlotte, NC, USA), Misty L Loughry (0000-0003-3546-7594, Georgia Southern University), Matthew Ohland (0000-0003-4052-1452, School of Engineering Education, Purdue University, West Lafayette, IN, USA), Matthew W Ohland (School of Engineering Education, Purdue University, West Lafayette, IN, USA)
Year2015
Volume18
Issue4
Pages704-737
Publication date2015-10-01
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueOrganizational Research Methods (JOURNAL)
Journal identifiersISSN: 1094-4281 • E-ISSN: 1552-7425
PublisherSAGE Publications (PUBLISHER • US)
DOI10.1177/1094428115582090
OpenAlexW1925128518
LanguageEN
Citations received54
References cited44

Management researchers often use consensus-based composition models to examine the antecedents and effects of higher-level constructs. Typically, researchers present three indices, r wg , ICC(1), and ICC(2), to demonstrate agreement and consistency among lower-level units when justifying aggregation. Nevertheless, researchers debate what values for these indices are sufficient. This study examines the distributional characteristics of ICCs and r wg values from three sources: the multilevel literature, a large multinational sample of student teams, and a large sample of randomly generated “pseudo teams.” Our results support existing cutoff criteria for ICCs but suggest that generally accepted values for r wg may, under certain circumstances, reflect pseudo-agreement (i.e., agreement observed among two raters not attributable to the same target). Thus, when there is minimal between-group variance (i.e., low ICCs), it is difficult to determine whether high r wg values reflect agreement or pseudo-agreement. Based on these findings, we provide recommendations to help researchers interpret aggregation indices.

Consistency (knowledge bases) · Cutoff · Econometrics · Economics · Multilevel model · Multinational corporation · Political science · Sample (material) · Statistics · Variance (accounting) · Accounting · Conflict Management and Negotiation · Experimental Behavioral Economics Studies · Mathematics · Psychology · Social and Intergroup Psychology

Tools

Open DOISci-Hub