Offering a critique of international benchmarking and the impact of computer-based assessment, this paper will make for interesting reading for policy makers and other professionals working with education data.
PISA 2012: How do results for the paper and computer tests compare?
Synopsis
This working paper considers how cross-country comparisons of pupils’ skills differ between paper and computer versions of the PISA mathematics test. The Programme for International Assessment (PISA) is an important international study of 15-year-olds’ academic achievement: it has traditionally been conducted using paper-and-pencil tests but the vast majority of countries used computer-based assessment in 2015.
Using data from PISA 2012, when more than 200,000 children from 32 economies completed both paper and computer versions of the mathematics assessment, the researchers found important differences between the two sets of results, including a substantial drop of more than 50 PISA points in the average performance of pupils from Shanghai.
By considering pupils’ responses to particular test questions, the paper demonstrates how such differences are unlikely to be solely due to the interactive nature of computer test questions, concluding that the move to computer-based testing will be an important change to assessment procedures, potentially influencing how pupils interpret, engage and answer test questions in 2015: a difference that could effect comparisons between 2012 and 2015.