Of interest to those seeking to accurately compare performances in PISA 2015 tests between those taking a paper-based test and those taking a computer-based test.
A digital divide? Randomised evidence on the impact of computer-based assessment in PISA
Synopsis
Between 2000 and 2012, PISA was carried out as a regular paper-based assessment. However, in 2015, pupils in the great majority of countries took the test on a computer instead. Since the change to computer-based assessment could affect pupil performance in itself – in ways that differ between countries – it has the potential to reduce the comparability of PISA test scores across countries and over time.
The researchers had access to data from three countries: Germany, Sweden and Ireland. In all three countries, the results show that pupils completing the computer-based test performed substantially worse than pupils completing the paper-based test. The difference is most pronounced in Germany (up to 26 PISA points), followed by Ireland (up to 18 PISA points) and Sweden (up to 15 PISA points).
Additionally, the researchers find little evidence of systematic gender differences in the impact of computers. The key conclusion is the adjustment made in PISA 2015 does not overcome all the potential challenges of switching to computer-based tests, but that it represents an improvement compared with not making any adjustment at all.
However, the results show that policymakers should take great care when comparing the results obtained via different modes, across and within countries.