Caution: Data in use

Data are the lifeblood of analytical research. If it were not for people willing to go out and track the positions of the stars in the sky, the theories of Nicolaus Copernicus would remain unproven and we would be operating on the misconception that the Earth was the centre of the universe. Albert Einstein would just be another clever theorist with some whacky ideas unless people were able to validate his theories by observation and measurement. On a more prosaic level, data matters hugely for modern policy issues as we need some form of benchmark against which to judge whether a policy is working.

As an economist I spend my professional life assessing economic issues on the basis of the evidence in front of me. I may not always be able to foretell the future but I can make a pretty good fist of understanding what is going on based on the data at my disposal. The data may not always be accurate or may be distorted by factors which hide the true message and one of the keys to decision-making is to understand how the evidence is compiled upon which decisions are based. This matters because decisions taken on the basis of faulty data risk making outcomes even worse. Two pieces of evidence released this week highlight the problems inherent in making policy based on uncertain data. 

(i) Covid data

The first instance is the measurement of Covid-19 deaths in England. It is well known that the UK has reported the highest number of deaths in Europe. But what is less well known to the casual observer is the figures assume that once a person is diagnosed as Covid-19 positive, their eventual death will be attributed to Covid-19, irrespective of the manner of their demise and how long after the initial diagnosis. For example, if a person was diagnosed as Covid-19 positive this year but dies a year later as a result of a grand piano landing on their head, they will technically still be classified as a Covid death. The rationale for this approach was that the authorities did not know much about the incubation period of the disease and did not want to be accused of under-reporting mortality figures in the early stages of the pandemic. Incidentally, this explains why, in contrast to many other countries, the UK did not report figures on recoveries. Once you have had the disease, that’s it – you are marked for life.

Obviously this is not particularly sensible. Consequently Public Health England this week imposed a limit of 28 days from diagnosis as the basis for measuring Covid-19 mortality, bringing it into line with Scotland, Wales and Northern Ireland – and indeed the international consensus. This makes a lot of sense: Broadly speaking, if you have not died 28 days after diagnosis, the odds are very much in favour of you remaining alive. This also changes the UK’s position in an international comparison. By my reckoning, the mortality rate is reduced from 70 per 100,000 of population to 62, which is much closer to the figures for Italy, Sweden and Spain. The UK still has the fifth highest global number of deaths but it no longer looks quite the outlier that it did earlier this week. Not that this is to excuse the government’s many failings in the way it handled the Covid pandemic, but by treating the figures on the same basis as other countries we have a more fair picture of how the UK stacks up in an international context.

(ii) Assessing exam data

The second data issue concerns the way in which school leavers’ ‘A’ level results were graded following the cancellation of this year’s exams. For those readers outside the UK, I should point out that the reporting ritual of exam results has become a staple of August news bulletins when there is nothing else for the media to focus on. Until recent years, the cry was that exams were becoming too easy and too many students were achieving top grades. In response to this media outcry, the marking was toughened up. So you can imagine the field day the media have had when it was impossible to hold exams at all.

There was never a satisfactory way to go about assessing what grades students would have achieved in the absence of exams. The starting point was to ask teachers to predict grades, but as Ofqual (The Office of Qualifications and Examinations Regulation) pointed out  teachers tend to offer optimistic predictions of their students’ ability. Teachers were thus also asked to provide “a rank order of students for each grade for each subject” on the basis that empirical evidence suggests people are better at making relative judgements than absolute judgements. Having collated all the evidence, Ofqual concluded that this would “likely to lead to overall national results that were implausibly high.” Accordingly, it had to find a way of normalising the results.

Unfortunately the model it chose is primarily based “on the historical performance of the school or college in that subject.” Even allowing for numerous tweaks, the system contains built-in discrimination against bright students from schools which in the past may not have delivered great exam results and is particularly biased towards independent schools which have a tendency to deliver above-average results. Ofqual argues there is “no evidence that this year’s process of awarding grades has introduced bias.” The data suggest otherwise: The number of A and A* grades awarded to independent schools increased by 4.7% compared to 2019, whereas the increases awarded to sixth form colleges, which tend to be attended by more disadvantaged students, was just 0.3% (chart). The data also show that the most disadvantaged students were more likely to have their predicted grades marked down: The proportion of pupils achieving a C or above fell by 10.4 percentage points among the most deprived third of pupils, compared to 8.3 percentage points among the wealthiest third. For anyone interested in an expert view of the problem, this series of Tweets is worth a read.

For a government that is trying to level up the life chances of those outside the south east of England (if you recall that was the promise in the 2019 election campaign) the 2020 exam process seems to be an odd way to go about it. Moreover, to the extent that young people’s exam results are one of the benchmarks which determine which university they attend, which in turn has a bearing on their future employment prospects, the importance of their exam results matters. Unfortunately, it is difficult to justify the rationale behind the algorithm which predicts exam grades thus devaluing their usefulness as a predictor of student ability. This in turn means that the data generation process becomes more important than the data which it turns out and is an example of why we should not always take data used to feed into policy decisions at face value.

As for the students, it is hard not to feel sympathy for them. They may or may not have gone on to achieve the grades predicted for them. But had they fallen short during the exams, they would at least have done so on their own terms. As it is, they have had grades imposed upon them by what appears to be a flawed data generation process, and judging by the popular outcry, it is not a decision that has played well with the public.

Reflections on policy errors

On this of all days, when we remember the centenary of the Armistice that ended World War I, it is worth reflecting on policy errors and their consequences. War is perhaps the ultimate policy error. No rational country chooses to go to war. The decision can usually be linked to a chain of circumstances which, if they had been dealt with differently, could have produced an entirely different outcome.

Unsurprisingly big conflicts generate the biggest headlines which explains why the western world still remembers the two global conflicts that scarred the twentieth century, whilst the US was scarred by the experience of the Vietnam War: For anyone who is interested in the scathing criticism of the errors of that campaign from the people who were there, the film by Ken Burns and Lynn Novik is a must-see (currently available on Netflix). There again, whilst history focuses on the consequences of events that went wrong, there are some instances where things went right that otherwise could gone tragically wrong. For example, history suggests that had the Cuban missile crisis of 1962 not turned out the way it did, we might not be here today to tell the tale.

We tend to remember wars in terms of the human cost. Anywhere between 15 and 19 million were killed during WW I and a further 23 million were wounded. Estimates of global fatalities during WW II are even higher, at over 60 million (3% of the world population in 1940). But there are also significant economic costs associated with wars. According to an authoritative study by Stephen Broadberry and Mark Harrison, World War I cost France and Germany more than 50% of their pre-war national wealth. In Germany’s case, this was primarily the result of the huge reparations bill imposed after the war, which is now perceived to have been a major mistake. Indeed, the economist John Maynard Keynes wrote a best-selling book in 1919, The Economic Consequences of the Peace, in which he warned of the adverse consequences of a punitive settlement.

It is interesting to note the contrasting British and German approaches to dealing with the huge debt incurred as a result of WWI. Both had debt-to-GDP ratios in excess of 100% in 1918. But the German government printed vast quantities of currency, and the resultant collapse in the value of the mark led to hyperinflation and a sharp decline in the value of public debt relative to GDP (see chart). The British approach was to deflate the economy, with the Geddes Axe of the early-1920s designed to slash public spending. It became a byword for how not to conduct fiscal policy, since the huge cuts  were made all at once rather than being phased gradually, with the biggest falls in social security spending, defence and education rather than ‘equal misery’ across all policy areas. It succeeded only in exacerbating the already-sluggish economic performance and the high levels of unemployment it was intended to mitigate, and had no impact on the debt burden.

Fortunately, most policy errors have less grave consequences but the economic costs can still be enormous. The bubble economy that developed post-2002 in the western world was largely the result of policy failures, driven by a laissez-faire attitude towards debt/credit creation that eventually produced the biggest economic crisis in 80 years. It is also notable that many European governments have subsequently adopted a 1920s-style UK approach to fiscal management (including the UK) by tightening fiscal policy at a time when the economy least needs it. In an economic sense, it is as though we have learned nothing from past experience.

They do say that those who cannot remember the past are doomed to repeat it. If nothing else, the day of remembrance across Europe serves to remind us of the futility of war and that it should only ever be the last option. On the whole, generations of Europeans have absorbed the lessons of history well. Whether economic policymakers have been quite as diligent is a matter for debate.