Estimation of the Reliability of Ratings
Bibliographic Data
| ID | 23356302 |
|---|---|
| Authors | Robert L Ebel (University of Iowa, corresponding author) |
| Year | 1951 |
| Volume | 16 |
| Issue | 4 |
| Pages | 407-424 |
| Publication date | 1951-12-01 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Psychometrika (JOURNAL) |
| Journal identifiers | ISSN: 0033-3123 • E-ISSN: 1860-0980 |
| Publisher | Cambridge University Press (CUP) (PUBLISHER) |
| DOI | 10.1007/bf02288803 |
| OpenAlex | W2064454539 |
| Language | EN |
| Citations received | 188 |
| References cited | 3 |
A procedure for estimating the reliability of sets of ratings, test scores, or other measures is described and illustrated. This procedure, based upon analysis of variance, may be applied both in the special case where a complete set of ratings from each of k sources is available for each of n subjects, and in the general case where k 1 , k 2 , ..., k n ratings are available for each of the n subjects. It may be used to obtain either a unique estimate or a confidence interval for the reliability of either the component ratings or their averages. The relations of this procedure to others intended to serve the same purpose are considered algebraically and illustrated numerically.
Component (thermodynamics) · Confidence interval · Econometrics · Reliability (semiconductor) · Reliability engineering · Set (abstract data type) · Statistics · Variance (accounting) · Variance components · Computer Science · Engineering · Mathematics · Scientific Measurement and Uncertainty Evaluation
The Evolution and Current Status of the Functional Perspective on Communication in Decision-Making and Problem-Solving Groups
SPSS macros for assessing the reliability and agreement of student evaluations of teaching
Ratings on traits measured by a factored personality inventory
Response selections of neurotic and control subjects as a function of relative familiarity and social desirability of items
Hostility and aversion for expressing hostility in neurotics and controls
The contingency of humor appreciation on the stimulus-confirmation of joke-ending expectations
What Characteristics Can Experts' Reliably Evaluate in the Speech of Black and Mexican-American Children
The Dependent Variable
The Context of Reported Asthma During Psychoanalysis
Intraclass correlations
Rating the ratings
Accurate empathy
Measures of response agreement for qualitative data
Crisis intervention
Assessing similarity between profiles
The meaning and reliability of accurate empathy ratings
Determining the Generalizability of Rating Scales in Clinical Settings
Peer Review of Medical Care
A Study of the Validity of the Task Inventory Method of Job Analysis
Clinical Algorithms Teach Pediatric Decisionmaking More Effectively than Prose
The Concept of Value in Resource Allocation
The Relationship of Selected Observational Variables to the Teaching of Singing
Adversary pretrial procedures and testimonial evidence
Social perception and interpersonal behavior
Cumulative risk, maternal responsiveness, and allostatic load among young adolescents
The Reliability of Observational Measures
An experimental study of the effects of orientation behavior on small group consensus
The Intraclass Correlation Coefficient as a Measure of Reliability
A Method of Estimating the Reliability of Ratings Compared with Certain Methods of Estimating the Reliability of Tests
The Handbook of Research Synthesis and Meta-Analysis
Aggregation bias in estimates of perceptual agreement.
Interrater reliability and agreement of subjective judgments.
Word Frequency and Age of Acquisition as Determiners of Picture-Naming Latency
Psychological climate
The impact of applicant gender compared to qualifications on hiring recommendations
Statistical Inference for Coefficient Alpha
The Approximate Sampling Distribution of Kuder-Richardson Reliability Coefficient Twenty
A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research
The effect of assessment method and respondent population on utilities elicited for Gaucher Disease
Hassles on the job
Changes in Shipley-Hartford Scores with Five Repeated Test Administrations
Inter- and Intra-Rater Reliability of the Brief Psychiatric Rating Scale
Trainee-Centered Explanation for Response to Training in Therapeutic Skills
A Computer Program to Calculate Adjusted and Unadjusted Interrater Reliabilities for Sets and Subsets of Judges
Factor Analyses of Normal and Depressed Patients' Memories of Parental Behavior
Creating Comparability Among Reliability Coefficients
Study of Validity of Ratings
Effectiveness of Crisis Center Volunteers and the Personal Orientation Inventory
Behavioral Referents for Molar Ratings of Heterosocial Skill
Controlling Decremental and Inflationary Effects in Reliability Estimation Resulting from Violations of Assumptions
Male and Female Leaders
Controlling Decremental and Inflationary Effects in Reliability Estimation Resulting From Violations of Assumptions
Creating Comparability among Reliability Coefficients
Contribution of the Education Phase to Stress-Inoculation Training
Profile Stability of a Clinical Rating Scale
Effects of Pretraining Procedures for Clients in Counseling
Need Approval, Modeling and Perception of Models' Adjustment in Behavior during an Interview
Semantic Differential for Third through Fifth Grade Students
Body Concern, Body Image and Nudity
Coefficient Alpha
Psychological Need Relationships in Dyadic Attraction and Rejection
Does A Parent's Right to Parenting Matter in Termination of Parental Rights? An Analysis of Court Cases
The Effect of Individual Difference Variables on Information Sharing in Decision-Making Groups
Evaluating Child-Custody Recommendations of Israeli Arab Social Workers
Between Professional Norms and Professionalism
The Effects of Aggression Replacement Training on Male and Female Antisocial Behavior in a Runaway Shelter
Child’s Religiosity, Ethnic Origin, and Gender
An evaluation of the reliability of probability judgments across response modes and over time
Assessing visibility impairment in class I parks and wilderness areas
Analyzing Dataset Annotation Quality Management in the Wild
An experimental study of the relationship between opinionated leadership and small group consensus
The effects of social interaction on group idea generation
How people disqualify
A descriptive investigation of the possible communication‐based reasons for effective and ineffective group decision making
A laboratory investigation of groupthink
Assessment and application of the revised speech dialect attitudinal scale
Effects of obscene language upon three dimensions of listener attitude
Differences in perceptions created by syntactic‐semantic productions of male and female speakers
Effects of selected American regional dialects upon regional audience members
Examining the role of explicit phonetic instruction in native-like and comprehensible pronunciation development
Claims about the reliability of student evaluations of instruction
The Relationship Between Message Recall and Persuasion
A Computer Program for Assessing the Reliability and Systematic Bias of Individual Measurements1
Estimating the Reliability of Peer Ratings
Estimating the Reliability, Systematic Error and Random Error of Interval Data
Inter-Rater Reliability
Teacher Perceptions of Behaviors Associated With Creative Problem Solving Performance
The Reliability of Letter Grades
A Note on Estimating the Reliability of Categorical Data
Reliability and Validity of a Priori Estimates of Item Characteristics for an Examination of Health Science Information
A Note on the Interpretation of Weighted Kappa and its Relations to Other Rater Agreement Statistics for Metric Scales
Agreement Between Raters
A Generalization of Kuder-Richardson Reliability Formula 21
Rater Agreement Indexes for Performance Assessment
The Efficiency of Various Estimations of Reliability of Rating Scales
The Effects of Scoring Formulas on the Discriminant Validity of Tests of Divergent Thinking
Reliability of College Grades and Grade Point Averages
Minimum Competency Standards Set by Three Divergent Groups of Raters Using Three Judgmental Procedures
Factor-Analyzed Abilities and Success in College Mathematics
Internal-Consistency Reliability Formulas Applied to Randomly Sampled Single-Factor Tests
| Unique citing works | 188 |
|---|---|
| Citations per year | 2,54 |
| Citation span | 1952 - 2024 (73) |
| Citation velocity | recent |
| Highly cited | Yes |
| Citation types | Neutral: 162 |