The test-takers' choice

An investigation of the effect of topic on language-test performance

Bibliographic Data

ID19380741
AuthorsMartha Jennings (Carleton University, Ontario,), Janna Fox (0000-0003-4528-4513, Carleton University, Ontario), Barbara Graves (Carleton University, Ontario), Barbara Ann Graves (0000-0002-5691-3285, Carleton University), Elana Shohamy (0000-0003-3462-1588, Tel Aviv University)
Year1999
Volume16
Issue4
Pages426-456
Publication date1999-10-01
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueLanguage Testing (JOURNAL)
Journal identifiersISSN: 0265-5322 • E-ISSN: 1477-0946
PublisherSAGE Publications (PUBLISHER • US)
DOI10.1177/026553229901600402
OpenAlexW2097039180
LanguageEN
Citations received17
References cited23

A fundamental issue in validating topic-based tests of language proficiency is the effect of the topic on the test takers’ performance. Topic-based test developers must ensure that test takers are neither advantaged nor disadvantaged in terms of their test results when presented with a given test topic. We have termed this threat a ‘topic effect’ and argue that this topic effect may constitute a source of construct-irrelevant variance (Messick, 1989). We contend that investigating the possibility of a topic effect is a critical step in establishing the validity of all topic-based tests. This research investigates the potential presence of a topic effect for the Canadian Academic English Language (CAEL) Assessment using the mechanism of choice. The principal research question is to determine if test-takers given a choice of topic perform significantly differently than test-takers not given a choice. ESL university applicants ( n = 254) were randomly assigned to one of two conditions: no choice of topic or choice among five topics. Overall Proficiency Level, Reading, Lecture and Essay scores were compared for the two conditions. Ordinal level data were analysed using the Mann Whitney U, Chi-Square and Kolmogorov-Smirnov tests. While the scores for the choice groups were usually higher than the scores for the no-choice groups, the differences were not statistically significant. Despite the lack of significance, we felt that the scores warranted closer examination. For the topic where the difference between the choice and no-choice groups was largest, a textual analysis of the essays was undertaken to look for instances of the use of information not provided in the test. Again, no difference was found between the choice and no-choice groups. The results provide support for the validity of inferences drawn from this test. Because choice is an essential element of the research design, a second focus of the study is to explore the advantages and disadvantages of the use of choice in language testing settings from the perspective of both the tester and the test-taker. The potential value of choice as a testing feature is discussed and a call for further research is made

Cognitive psychology · Disadvantaged · Language assessment · Language proficiency · Linguistics · Mathematics education · Multiple choice · Computer Science · EFL/ESL Teaching and Learning · Psychology · Second Language Learning and Teaching · Social Psychology · Student Assessment and Feedback

Tools

Open DOISci-Hub