Caution: Data in use

Data are the lifeblood of analytical research. If it were not for people willing to go out and track the positions of the stars in the sky, the theories of Nicolaus Copernicus would remain unproven and we would be operating on the misconception that the Earth was the centre of the universe. Albert Einstein would just be another clever theorist with some whacky ideas unless people were able to validate his theories by observation and measurement. On a more prosaic level, data matters hugely for modern policy issues as we need some form of benchmark against which to judge whether a policy is working.

As an economist I spend my professional life assessing economic issues on the basis of the evidence in front of me. I may not always be able to foretell the future but I can make a pretty good fist of understanding what is going on based on the data at my disposal. The data may not always be accurate or may be distorted by factors which hide the true message and one of the keys to decision-making is to understand how the evidence is compiled upon which decisions are based. This matters because decisions taken on the basis of faulty data risk making outcomes even worse. Two pieces of evidence released this week highlight the problems inherent in making policy based on uncertain data. 

(i) Covid data

The first instance is the measurement of Covid-19 deaths in England. It is well known that the UK has reported the highest number of deaths in Europe. But what is less well known to the casual observer is the figures assume that once a person is diagnosed as Covid-19 positive, their eventual death will be attributed to Covid-19, irrespective of the manner of their demise and how long after the initial diagnosis. For example, if a person was diagnosed as Covid-19 positive this year but dies a year later as a result of a grand piano landing on their head, they will technically still be classified as a Covid death. The rationale for this approach was that the authorities did not know much about the incubation period of the disease and did not want to be accused of under-reporting mortality figures in the early stages of the pandemic. Incidentally, this explains why, in contrast to many other countries, the UK did not report figures on recoveries. Once you have had the disease, that’s it – you are marked for life.

Obviously this is not particularly sensible. Consequently Public Health England this week imposed a limit of 28 days from diagnosis as the basis for measuring Covid-19 mortality, bringing it into line with Scotland, Wales and Northern Ireland – and indeed the international consensus. This makes a lot of sense: Broadly speaking, if you have not died 28 days after diagnosis, the odds are very much in favour of you remaining alive. This also changes the UK’s position in an international comparison. By my reckoning, the mortality rate is reduced from 70 per 100,000 of population to 62, which is much closer to the figures for Italy, Sweden and Spain. The UK still has the fifth highest global number of deaths but it no longer looks quite the outlier that it did earlier this week. Not that this is to excuse the government’s many failings in the way it handled the Covid pandemic, but by treating the figures on the same basis as other countries we have a more fair picture of how the UK stacks up in an international context.

(ii) Assessing exam data

The second data issue concerns the way in which school leavers’ ‘A’ level results were graded following the cancellation of this year’s exams. For those readers outside the UK, I should point out that the reporting ritual of exam results has become a staple of August news bulletins when there is nothing else for the media to focus on. Until recent years, the cry was that exams were becoming too easy and too many students were achieving top grades. In response to this media outcry, the marking was toughened up. So you can imagine the field day the media have had when it was impossible to hold exams at all.

There was never a satisfactory way to go about assessing what grades students would have achieved in the absence of exams. The starting point was to ask teachers to predict grades, but as Ofqual (The Office of Qualifications and Examinations Regulation) pointed out  teachers tend to offer optimistic predictions of their students’ ability. Teachers were thus also asked to provide “a rank order of students for each grade for each subject” on the basis that empirical evidence suggests people are better at making relative judgements than absolute judgements. Having collated all the evidence, Ofqual concluded that this would “likely to lead to overall national results that were implausibly high.” Accordingly, it had to find a way of normalising the results.

Unfortunately the model it chose is primarily based “on the historical performance of the school or college in that subject.” Even allowing for numerous tweaks, the system contains built-in discrimination against bright students from schools which in the past may not have delivered great exam results and is particularly biased towards independent schools which have a tendency to deliver above-average results. Ofqual argues there is “no evidence that this year’s process of awarding grades has introduced bias.” The data suggest otherwise: The number of A and A* grades awarded to independent schools increased by 4.7% compared to 2019, whereas the increases awarded to sixth form colleges, which tend to be attended by more disadvantaged students, was just 0.3% (chart). The data also show that the most disadvantaged students were more likely to have their predicted grades marked down: The proportion of pupils achieving a C or above fell by 10.4 percentage points among the most deprived third of pupils, compared to 8.3 percentage points among the wealthiest third. For anyone interested in an expert view of the problem, this series of Tweets is worth a read.

For a government that is trying to level up the life chances of those outside the south east of England (if you recall that was the promise in the 2019 election campaign) the 2020 exam process seems to be an odd way to go about it. Moreover, to the extent that young people’s exam results are one of the benchmarks which determine which university they attend, which in turn has a bearing on their future employment prospects, the importance of their exam results matters. Unfortunately, it is difficult to justify the rationale behind the algorithm which predicts exam grades thus devaluing their usefulness as a predictor of student ability. This in turn means that the data generation process becomes more important than the data which it turns out and is an example of why we should not always take data used to feed into policy decisions at face value.

As for the students, it is hard not to feel sympathy for them. They may or may not have gone on to achieve the grades predicted for them. But had they fallen short during the exams, they would at least have done so on their own terms. As it is, they have had grades imposed upon them by what appears to be a flawed data generation process, and judging by the popular outcry, it is not a decision that has played well with the public.