Many Shades of Purple - Exploring Comparative Judgement to Assess Writing

Assessment in the new Curriculum for Wales is a hot topic. The practice of making a 'best-fit' judgement against (at best woolly) level descriptions is no more, and many would say good riddance to these and even more so to the additional nonsense that was sub-levels! 

As a result, teachers are now having to grapple with the hugely challenging task of working out exactly what progress along a continuum of Progression Steps looks like, ensuring their curriculum is built to enable this, and deciding how it will be assessed, in all six Areas of Learning (AoLEs) and the disciplines within these. There is a lot of guidance from the Welsh Government on the Hwb platform and in the Progression Code document, but the emphasis is on school and cluster-based agreement on how learners will be enbled to demonstrate their progress.

Against this background, and as the lead for LLC (Languages, Literacy and Communication) in our federation, I jumped at the chance, provided by our regional consortium, to trial comparative judgment as a way of assessing writing. This was facilitated by 'No More Marking' (NMM) using their 'Assessing Primary Writing' project. The 'shades of purple' activity on their 'Demo' page illustrates powerfully how completing a series of comparisons is far more reliable than making absolute judgements. Daisy Christodoulou, the Director of Education at NMM, has written extensively about assessment and the validity and reliability of different methods in her blog. I also attended a webinar with Daisy as an introduction to the project. During this, attendees were invited to take part in a live judging session where we were shown pairs of scripts and had to click left or right, to choose which we thought was the better example. We were encouraged to go with our (professional) 'gut' feeling. In many cases, it was immediately obvious which script was better, and for most others, a decision could be made relatively quickly by reading through the first section of each script. Each script was 'judged' at least 10 times by the group as a whole, and through comparison with different scripts. The whole process took minutes and resulted in the scripts being placed in rank order. We were told that collectively we had achieved a reliability score of 0.93; in other words, there was a high degree of consistency in the judges' decisions. Any score above 0.8 indicates a high degree of reliability. 

Along with a rank order, the NNM comparative judgement engine produces a scale score, and uniquely, a writing age .

Everything I had heard and read persuaded me that this was an approach to assessment that was completely in keeping with the ethos of Curriculum for Wales, whilst improving the reliability of the assessment of writing and having a positive impact on teacher workload. 






Next post:Comparative Judgement - Did it live up to expectations?


Comments

Post a Comment

Popular posts from this blog

Comparative Judgement - What Next?

Image
This is the final post in a series of three in which I have reflected on my experience of trialling comparative judgmement as a way of assessing children's writing. After the second period of remote learning, writing soon emerged as the skill that had taken the biggest 'hit' for a majority of our children. The complexity of writing, and its range of constituent sub-skills meant that to address the issue as a whole, we had to support the development of each part.  As a junior school and a very small primary, most of our children are of junior age. For those who were already in Y3 or above prior to the pandemic, writing stamina and spelling needed to improve, whilst for the younger children, fine motor skills and handwriting were additional areas for our focus. We had already begun to use a new  handwriting scheme  in September 2020, following the first period of remote learning, but after further disruption, we went back to the beginning with all year groups. Teachers in jun
Read more

Comparative Judgement - did it live up to expectations?

Image
My previous post described my introduction to comparative judgement as a way of assessing writing. As a federation, we embarked on our trial of this approach, using No More Marking (NMM), and facilitated by our regional consortium, in September 2022. We were using NMM's 'Assessing Primary Writing' product. For each year group from 1 to 6, there was an assessment 'window' and these were spread across the autumn and spring terms. A set prompt for writing was supplied just before the assessment period began and the children had to write on individual, pre-printed lined sheets. This meant that there was a certain amount of administration on my part, entering names, dates of birth, classes, gender and entitlement to free school meals (the category on the spreadsheet is 'Pupil Premium' as it's designed for the English system), and then printing and distributing the sheets for the children to write on.  The teacher guidance is clear that after explaining the ta
Read more