Yigal Attali
Biographic Data
| ID | 5730962 |
|---|---|
| NAME | Yigal Attali |
| GIVEN NAMES | Yigal |
| FAMILY NAME | Attali |
| SIGNATURE | ATTALI Y |
| AFFILIATIONS | Educational Testing Service |
| ORCID | 0000-0002-3907-247X |
| VERIFIED | Yes |
| TOTAL WORKS | 13 |
| TOTAL CITATIONS | 2 |
| WORKS AS AUTHOR | 13 |
| WORKS AS EDITOR | 0 |
| FIRST PUBLICATION YEAR | 2005 |
| LATEST PUBLICATION YEAR | 2025 |
| H-INDEX | 1 |
A multi-stage interactive writing task for the assessment of English language writing proficiency
A longstanding criticism of traditional high-stakes writing assessments is their use of static prompts in which test takers compose a single text in response to a prompt. These static prompts do not allow measurement of the writing process. This paper describes the development and validation of an innovative interactive writing task. After the test taker responds to an initial prompt, a large language model automatically analyzes their response t…
Maintaining and monitoring quality of a continuously administered digital assessment
Digital-first assessments are a new generation of high-stakes assessments that can be taken anytime and anywhere in the world. The flexibility, complexity, and high-stakes nature of these assessments pose quality assurance challenges and require continuous data monitoring and the ability to promptly identify, interpret, and correct anomalous results. In this manuscript, we illustrate the development of a quality assurance system for anomaly detec…
Digital‐first assessments
Differential Performance in High Versus Low Stakes Tests
We study how different demographic groups respond to incentives by comparing their performance in ‘high’ and ‘low’ stakes situations. The high stakes situation is the Graduate Record Examination (GRE), and the low stakes situation is a voluntary experimental section of the GRE. We find that males exhibit a larger drop in performance between the high and low stakes examinations than females, and that whites exhibit a larger drop in performance tha…
Effort in Low-Stakes Assessments
Performance of students in low-stakes testing situations has been a concern and focus of recent research. However, researchers who have examined the effect of stakes on performance have not been able to compare low-stakes performance to truly high-stakes performance of the same students. Results of such a comparison are reported in this article. GRE test takers volunteered to take an additional low-stakes test, of either verbal or quantitative re…
Differences in Reaction to Immediate Feedback and Opportunity to Revise Answers for Multiple-Choice and Open-Ended Questions
There are many reasons to believe that open-ended (OE) and multiple-choice (MC) items elicit different cognitive demands of students. However, empirical evidence that supports this view is lacking. In this study, we investigated the reactions of test takers to an interactive assessment with immediate feedback and answer-revision opportunities for the two types of items. Eighth-grade students solved mathematics problems, both MC and OE, with stand…
A comparison of newly-trained and experienced raters on a standardized writing assessment
A short training program for evaluating responses to an essay writing task consisted of scoring 20 training essays with immediate feedback about the correct score. The same scoring session also served as a certification test for trainees. Participants with little or no previous rating experience completed this session and 14 trainees who passed an accuracy threshold proceeded to score other essays. Performance of the newly-trained raters was comp…
A Ranking Method for Evaluating Constructed Responses
This article presents a comparative judgment approach for holistically scored constructed response tasks. In this approach, the grader rank orders (rather than rate) the quality of a small set of responses. A prior automated evaluation of responses guides both set formation and scaling of rankings. Sets are formed to have similar prior scores and subsequent rankings by graders serve to update the prior scores of responses. Final response scores a…
Scoring with the computer
Automated essay scoring can produce reliable scores that are highly correlated with human scores, but is limited in its evaluation of content and other higher-order aspects of writing. The increased use of automated essay scoring in high-stakes testing underscores the need for human scoring that is focused on higher-order aspects of writing. This study experimentally evaluated several alternative procedures for eliciting distinct human scores and…
Sequential Effects in Essay Ratings
Contrary to previous research on sequential ratings of student performance, this study found that professional essay raters of a large-scale standardized testing program produced ratings that were drawn toward previous ratings, creating an assimilation effect. Longer intervals between the two adjacent ratings and higher degree of agreement with other raters were associated with smaller assimilation effects. The effect was also found to create a s…
Immediate Feedback and Opportunity to Revise Answers to Open-Ended Questions
Two experiments examine the psychometric effects of providing immediate feedback on the correctness of answers to open-ended questions, and allowing participants to revise their answers following feedback. Participants answering verbal and math questions are able to correct many of their initial incorrect answers, resulting in higher revised scores. In addition, the reliability of these scores is significantly higher than the reliability of score…
Validity of Scores for a Developmental Writing Scale Based on Automated Scoring
A developmental writing scale for timed essay-writing performance was created on the basis of automatically computed indicators of writing fluency, word choice, and conventions of standard written English. In a large-scale data collection effort that involved a national sample of more than 12,000 students from 4th, 6th, 8th, 10th, and 12th grade, students wrote (in 30-min sessions) up to four essays in two modes of writing on topics selected from…
Scoring and keying multiple choice tests
Differential Performance in High Versus Low Stakes Tests
We study how different demographic groups respond to incentives by comparing their performance in ‘high’ and ‘low’ stakes situations. The high stakes situation is the Graduate Record Examination (GRE), and the low stakes situation is a voluntary experimental section of the GRE. We find that males exhibit a larger drop in performance between the high and low stakes examinations than females, and that whites exhibit a larger drop in performance tha…
Scoring and keying multiple choice tests
Validity of Scores for a Developmental Writing Scale Based on Automated Scoring
A developmental writing scale for timed essay-writing performance was created on the basis of automatically computed indicators of writing fluency, word choice, and conventions of standard written English. In a large-scale data collection effort that involved a national sample of more than 12,000 students from 4th, 6th, 8th, 10th, and 12th grade, students wrote (in 30-min sessions) up to four essays in two modes of writing on topics selected from…
Sequential Effects in Essay Ratings
Contrary to previous research on sequential ratings of student performance, this study found that professional essay raters of a large-scale standardized testing program produced ratings that were drawn toward previous ratings, creating an assimilation effect. Longer intervals between the two adjacent ratings and higher degree of agreement with other raters were associated with smaller assimilation effects. The effect was also found to create a s…
Immediate Feedback and Opportunity to Revise Answers to Open-Ended Questions
Two experiments examine the psychometric effects of providing immediate feedback on the correctness of answers to open-ended questions, and allowing participants to revise their answers following feedback. Participants answering verbal and math questions are able to correct many of their initial incorrect answers, resulting in higher revised scores. In addition, the reliability of these scores is significantly higher than the reliability of score…
Scoring with the computer
Automated essay scoring can produce reliable scores that are highly correlated with human scores, but is limited in its evaluation of content and other higher-order aspects of writing. The increased use of automated essay scoring in high-stakes testing underscores the need for human scoring that is focused on higher-order aspects of writing. This study experimentally evaluated several alternative procedures for eliciting distinct human scores and…
A Ranking Method for Evaluating Constructed Responses
This article presents a comparative judgment approach for holistically scored constructed response tasks. In this approach, the grader rank orders (rather than rate) the quality of a small set of responses. A prior automated evaluation of responses guides both set formation and scaling of rankings. Sets are formed to have similar prior scores and subsequent rankings by graders serve to update the prior scores of responses. Final response scores a…
Effort in Low-Stakes Assessments
Performance of students in low-stakes testing situations has been a concern and focus of recent research. However, researchers who have examined the effect of stakes on performance have not been able to compare low-stakes performance to truly high-stakes performance of the same students. Results of such a comparison are reported in this article. GRE test takers volunteered to take an additional low-stakes test, of either verbal or quantitative re…
Differences in Reaction to Immediate Feedback and Opportunity to Revise Answers for Multiple-Choice and Open-Ended Questions
There are many reasons to believe that open-ended (OE) and multiple-choice (MC) items elicit different cognitive demands of students. However, empirical evidence that supports this view is lacking. In this study, we investigated the reactions of test takers to an interactive assessment with immediate feedback and answer-revision opportunities for the two types of items. Eighth-grade students solved mathematics problems, both MC and OE, with stand…
A comparison of newly-trained and experienced raters on a standardized writing assessment
A short training program for evaluating responses to an essay writing task consisted of scoring 20 training essays with immediate feedback about the correct score. The same scoring session also served as a certification test for trainees. Participants with little or no previous rating experience completed this session and 14 trainees who passed an accuracy threshold proceeded to score other essays. Performance of the newly-trained raters was comp…
Differential Performance in High Versus Low Stakes Tests
We study how different demographic groups respond to incentives by comparing their performance in ‘high’ and ‘low’ stakes situations. The high stakes situation is the Graduate Record Examination (GRE), and the low stakes situation is a voluntary experimental section of the GRE. We find that males exhibit a larger drop in performance between the high and low stakes examinations than females, and that whites exhibit a larger drop in performance tha…
Maintaining and monitoring quality of a continuously administered digital assessment
Digital-first assessments are a new generation of high-stakes assessments that can be taken anytime and anywhere in the world. The flexibility, complexity, and high-stakes nature of these assessments pose quality assurance challenges and require continuous data monitoring and the ability to promptly identify, interpret, and correct anomalous results. In this manuscript, we illustrate the development of a quality assurance system for anomaly detec…
Digital‐first assessments
A multi-stage interactive writing task for the assessment of English language writing proficiency
A longstanding criticism of traditional high-stakes writing assessments is their use of static prompts in which test takers compose a single text in response to a prompt. These static prompts do not allow measurement of the writing process. This paper describes the development and validation of an innovative interactive writing task. After the test taker responds to an initial prompt, a large language model automatically analyzes their response t…
Psychology (11 works) · Computer Science (8 works) · Mathematics education (7 works) · Social Psychology (6 works) · Statistics (6 works) · Mathematics (5 works) · Student Assessment and Feedback (5 works) · Artificial Intelligence (4 works) · Cognitive psychology (4 works) · Developmental psychology (4 works)