When Seeing Is Believing
Generalizability and Decision Studies for Observational Data in Evaluation and Research on Teaching
Bibliographic Data
| ID | 12470517 |
|---|---|
| Authors | Timothy J Weston (0000-0003-2982-4874, University of Colorado System, corresponding author), Charles N Hayward (0000-0003-2733-6025, University of Colorado System), Sandra L Laursen (0000-0002-4327-9887, University of Colorado System) |
| Year | 2021 |
| Volume | 42 |
| Issue | 3 |
| Pages | 377-398 |
| Publication date | 2021-07-20 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | American Journal of Evaluation (JOURNAL) |
| Journal identifiers | ISSN: 1098-2140 • E-ISSN: 1557-0878 |
| Publisher | SAGE Publishing (PUBLISHER • US) |
| DOI | 10.1177/1098214020931941 |
| OpenAlex | W3186113926 |
| Language | EN |
| Citations received | 2 |
| References cited | 51 |
Observations are widely used in research and evaluation to characterize teaching and learning activities. Because conducting observations is typically resource intensive, it is important that inferences from observation data are made confidently. While attention focuses on interrater reliability, the reliability of a single-class measure over the course of a semester receives less attention. We examined the use and limitations of observation for evaluating teaching practices, and how many observations are needed during a typical course to make confident inferences about teaching practices. We conducted two studies based on generalizability theory to calculate reliabilities given class-to-class variation in teaching over a semester. Eleven observations of class periods over the length of a semester were needed to achieve a reliable measure, many more than the one to four class periods typically observed in the literature. Findings suggest practitioners may need to devote more resources than anticipated to achieve reliable measures and comparisons
Class (philosophy · Data collection · Data mining · Developmental psychology · Generalizability theory · Inter-rater reliability · Mathematics education · Measure (data warehouse · Observational study · Rating scale · Reliability (semiconductor · Resource (disambiguation · Statistics · Teaching method · Variation (astronomy · Computer Science · Evaluation of Teaching Practices · Mathematics · Online and Blended Learning · Psychology · Teacher Education and Leadership Studies · Artificial Intelligence
Sage Handbook of Mixed Methods in Social & Behavioral Research
Anatomy of STEM teaching in North American universities
The Icap Framework
Where's the evidence that active learning works?
Methodological Issues in the Direct Observation of Parent–Child Interaction
Active learning increases student performance in science, engineering, and mathematics
Developing indicators of classroom practice to evaluate the impact of district mathematics reform initiative
Once is not enough
Making Inferences About Teacher Observation Scores Over Time
Rater calibration when observational assessment occurs at large scale
Conceptualization, Measurement, and Improvement of Classroom Processes
Describing Profiles of Instructional Practice
Building a More Complete Understanding of Teacher Evaluation Using Classroom Observations
When Rater Reliability Is Not Enough
| Citations per year | 0.4 |
|---|---|
| Citation span | 2021 - 2025 (5) |
| Highly cited | No |