Rationality meets reality

 The rational expectations (RE) revolution which swept through macroeconomics in the 1970s and 1980s has changed the way we think about many aspects of macro. As theories go, it is coherent and persuasive and has allowed us to think differently about many aspects of economics and finance. But there have been rumblings recently from respected professionals in the field expressing doubts about its usefulness. As one who has never fully bought into the idea that this is actually how people form expectations, I am obviously prone to confirmation bias, but clearly I am not the only one who has reservations about one of the key underpinnings of modern macroeconomics.

What are rational expectations?

In very simple terms, RE assumes that economic agents make the best use of all currently available information to make predictions about future events in a logically consistent manner. The upshot is that individuals do not make systematic forecast errors (although they can make random errors). This appears uncontroversial at first glance but it has profound consequences for policymakers. Prior to the work of Robert Lucas in the 1970s, it was assumed that a paradigm used to assess the outcome of a policy change would either remain unchanged in future, or would change only slowly as expectations adapted to new evidence. But Lucas pointed out that as economic agents recognise and internalise the way policy affects the economy, they will change their expectations formation process. As a result, the old paradigm is no longer valid. Applying the same policy options in future would result in different outcomes because agents would anticipate what was likely to happen and act accordingly.

A simple example is the  Phillips curve, which was based on the idea that there is a stable and exploitable inverse trade-off between inflation and unemployment. In this static world, a policymaker wishing to reduce unemployment would be prepared to allow inflation to rise. But in a world where expectations are formed rationally, they can only get away with that once.  Next time round workers push for higher wage claims to offset the erosion of real wages, with the result that employment falls (unemployment rises). As it happened this was pretty much what happened in the 1970s as the inverse relationship between the two broke down.

Lucas led the intellectual charge of New Classical economics which usurped the dominant Keynesian paradigm, and challenged the efficacy of discretionary macroeconomic policies by arguing that if individuals can foresee the consequences of policy changes, attempts to manipulate the economy through fiscal or monetary policy become less effective. The New Keynesian response was to synthesise Keynesian principles with insights from the New Classical revolution. Crucially, however, they did not reject the RE hypothesis.

Finance, too, has been captured by the RE revolution. RE are a key component of the Efficient Markets Hypothesis, according to which asset prices reflect all available information, and market participants form rational expectations about future events. In an efficient market, it is assumed that investors cannot consistently achieve abnormal returns by exploiting past information because prices already incorporate all relevant data. Furthermore, the Capital Asset Pricing Model (CAPM) assumes that investors form homogeneous and internally consistent expectations about returns, which is related to the rational expectations idea that agents' forecasts are consistent with the model they use.

Are expectations formed rationally?

Rational expectations are, to use the jargon, ‘model consistent’. In other words the average predictions across all economic agents match the predictions of an economic model which captures the true structure of the economy. Obviously, we are not solving complex models of the economy to derive views about the future. Instead, we rely on heuristic rules of thumb, public forecasts, market signals or simplified mental models. If these rules generate outcomes in line with how the economy actually works, we can still proceed on the basis that agents form rational expectations. But it is questionable whether such rules actually work, particularly at times of elevated uncertainty such as we are experiencing today. Given the raised prospect of extreme outcomes in the wake of Donald Trump’s election, we have even less certainty about what the world might look like in future. Due to this lack of information, agents can be excused for simply extrapolating forward based on past performance on the basis that this represented “normality” and that risks are evenly distributed around this outcome. It might be a rational way of looking at the world, but now expectations are being formed adaptively rather than in a model-consistent way.

What does the evidence tell us?

Nowhere are RE more central than in financial markets, where asset prices are typically assumed to reflect the rationally expected present value of future cash flows. In an important 1981 paper, Robert Shiller examined this idea by testing whether fluctuations in stock prices could be explained by changes in expectations of future dividends. According to the standard RE-based present value model, most of the variability in prices should come from new information that affects expected future dividends. However, Shiller found that actual stock prices were far more volatile than the relatively stable stream of realised dividends would justify. This result, often described as the “excess volatility puzzle,” called into question whether prices are set purely on the basis of rational expectations, or whether they are also influenced by factors such as changing risk premia, investor sentiment, or other non-fundamental forces.

In an attempt to update the Shiller methodology, I computed the ex post “fair value” of the S&P (in real terms) by discounting actual realised dividends over a five-year horizon, assuming perfect foresight of future payouts. As the dataset extends only to June 2025, the perfect foresight calculation is only feasible up to June 2020; for subsequent periods, I extrapolated using the trailing 12-month average dividend to maintain a continuous valuation series. While this is not exactly comparable, it is an approach often used in the academic literature. The results suggest that around the time of the dot com bubble in the late-1990s, and again around the time of the Lehman’s bust, equities were overvalued relative to fundamentally justified levels. However, these periods pale in comparison to the post-2020 period (methodological differences notwithstanding), suggesting that equities may be experiencing a period of irrational exuberance.

Providing statistical evidence for the existence (or absence) of RE is challenging. One approach is to compare the fundamentally justified price – defined above as the discounted value of future earnings – with the actual observed price. If RE hold, all available information should already be reflected in the price, implying that the difference between the two should not be systematically related to any other variable. Consequently, regressing this difference on another metric should yield a coefficient that is not statistically different from zero (see below).

However, the results suggest that a regression of the difference on observable variables such as the P/E ratio or dividend yield do indeed generate coefficients which are statistically significantly different from zero. The charts (below) plot the forecast error – defined as the difference between the actual return and the expected return – against the observed P/E ratio and dividend yield. Under the RE hypothesis, forecast errors should be purely random: they should not be systematically related to any information known in advance. This should appear as a scatter of points randomly distributed around zero, with the fitted regression line essentially flat. But the plots show a statistically significant upward slope, suggesting that forecast errors are systematically related to both the P/E ratio and dividend yield, which imply that investors could, in principle, have used these variables to improve their forecasts. This provides some evidence against RE, as it implies that prices do not fully incorporate available information at any given time.

You don’t just have to take my word for it. Cliff Asness, one of the most astute and intellectually rigorous portfolio managers out there, wrote an excellent paper in 2024 arguing that markets have become far less efficient since the early 1990s. Asness offers three reasons why markets are now less efficient compared with the pre-1990 period: (i) the rise of indexing has made stock prices more inelastic with respect to new information; (ii) an extended period of low interest rates has distorted investors ability to respond appropriately to changed information and (iii) the rise of social media has amplified trend following and momentum strategies at the expense of rational information processing.

One worrying thought is that if markets, with their access to huge amounts of data, are not processing information in a manner consistent with RE, what is the likelihood that households are doing it? This matters because RE are absolutely central to modern DSGE (Dynamic Stochastic General Equilibrium) models in which agents (households, firms, policymakers) are assumed to form expectations about the future that are model-consistent. If expectations are not formed rationally, forecasts based on such models may be potentially biased, and policy outcomes may be less effective.

Final thoughts

Looking at market movements in recent months, perhaps we can be forgiven for thinking that the past is the only guide we have to future performance. But such reliance on past trends carries its own risks. Adaptive expectations, by definition, anchor forecasts to recent experience, making them slow to incorporate new structural shifts or unprecedented shocks. In the current environment of heightened uncertainty, characterised by a global political realignment and the disruptive potential of technological change, such inertia may lead to systematic forecast errors. Instead of anticipating turning points, markets and policymakers risk being repeatedly surprised by outcomes that fall outside the narrow band of recent history.

Moreover, if everyone leans too heavily on the same backward-looking heuristics, market dynamics themselves can amplify volatility. Herding behaviour may set in, reinforcing bubbles or deepening downturns as agents all update beliefs in the same direction, as Asness implies. In this sense, the process of expectations formation becomes not merely a passive reflection of past data, but an active force that shapes the trajectory of the economy.

Unfortunately, it is extremely difficult to capture the complexity of the expectations formation process, while those which rely solely on historical patterns risk missing the disruptive events that define each economic cycle. Whether expectations can ever be fully rational in the strict sense remains debatable – but recognising the limitations of both model-based and adaptive approaches is a step toward better decision-making in an uncertain world.

 

A rough start with worse to come

Keir Starmer’s Labour government took office a year ago following an overwhelming election victory that consigned an unpopular Conservative government to history. I did, however, warn that Labour’s victory owed less to its own popularity and more to the electorate’s desire for change. As I noted at the time: “This makes it all the more imperative that Starmer’s government gets the big things right quickly. Making voters lives better is the one thing that will raise the chances of a second term in office – a second term that will undoubtedly be required to properly fix many of the things in the economy that require improvement.”

Fiscal challenges and policy U-turns

Measured against this yardstick, the government is failing to meet its objectives and there is common agreement that it does not have a clearly defined philosophy. Economic policy has largely been focused on tackling the UK’s mounting fiscal burden. The Chancellor, Rachel Reeves, claimed in summer 2024 that the situation was worse than Labour imagined before taking office, though as the Institute for Government has noted: “the truth is that most of the tough fiscal choices that this government faced were well known before the election – politicians from both main parties simply chose to ignore them.” Reeves claimed that unfunded spending commitments made by the previous Conservative government resulted in a £22bn “black hole” in public finances (around 0.8% of GDP). While it is true that a Treasury audit pointed to a departmental overspend in fiscal year 2024-25 of £21.9bn, almost half of this arose from the current government’s discretionary decision to accept recommendations for public sector pay awards which were higher than those factored in by the Tories. Nor has the £22bn figure been endorsed by the OBR, which suggested that the previous government’s unannounced policy commitments were worth around £9.5bn.

Hemmed in by its manifesto commitment not to raise taxes on “working people”, the Chancellor was forced to find some measures to show the government’s commitment to fiscal rectitude. But the proposals put forward by Reeves have been poorly presented, with the result that the government has backtracked on policy announcements in the face of opposition. A case in point was the plan to limit the Winter Fuel Allowance to only the poorest pensioners. The payment was originally introduced in 1997 as a universal benefit but the incoming Labour government surprisingly announced in 2024 that it would be converted to a means-tested benefit. This generated a huge wave of criticism and was cited as one of the reasons why Labour performed so poorly in the May local elections and prompted a policy U-turn shortly afterwards. 

For all the political capital which was squandered by the policy, it was only expected to save £1.4bn of fiscal outlays, and the policy reversal, which raises the income threshold for the benefit, will now only save around £0.45bn – a trivial amount in UK fiscal terms (see chart above). Indeed, fiscal data for the first ten months of Labour’s tenure show that borrowing has risen by £33bn (around 1.1% of GDP) versus the corresponding period a year earlier.

Similarly, the government was forced this week to significantly water down its planned reform of the welfare system in the face of fierce opposition from backbench Labour MPs. Here, too, the fiscal savings resulting from tightening access to welfare payments are relatively small in the grand scheme of things, with an estimated saving of £5.5bn by fiscal 2029-30. As was the case with the Winter Fuel Allowance, a huge amount of political capital was risked to achieve a small monetary saving. This has raised questions about the government’s political acumen and has increasingly called the prime minister’s authority into question. While this is the normal response of a commentariat which likes nothing better than to poke holes in the shortcomings of the government of the day, it is increasingly clear that Starmer’s administration has failed to regenerate the feelgood factor. To the extent that the government is going to need a second term to address the deep-seated economic problems facing the economy, the fact that Labour now trails Nigel Farage’s Reform UK party in the polls should act as an urgent wakeup call (see chart below).

Hard to see how taxes cannot rise

Matters might well get worse before they get better. Indeed, there is increasing speculation that the government will be forced to raise taxes in the autumn, despite the manifesto commitment not to do so. The howls of protest from opposition political parties, and perhaps even from backbench Labour MPs, will be predictably loud, but in truth the government is out of fiscal options and has very little headroom to ensure that its fiscal rules can be met over the course of this parliament. Public services are stretched and the electorate has noticed. The NHS remains under huge pressure, and although there has been a small decline in waiting lists for treatment in the past 12 months, public dissatisfaction with the NHS continues to hit new highs (see chart below). Nor has the abolition of NHS England gone down well with medical professionals. The prime minister sold it as a way of increasing efficiency by reducing the degree of centralisation: Insiders see it as a way to cut NHS jobs. Equally importantly, the criminal justice system is operating with no spare capacity, to the point at which the government even experimented with the early release of prisoners to alleviate the strain – a policy which is unpopular with the public.

And then there is defence spending. The government plans to raise it to 2.5% of GDP by 2027 (2024: 2.3%) and has ambitions to increase it to 3% in the next parliament. NATO is seeking agreement from members to raise it to 5% by 2035, comprised of 3.5% under the current core definition with a supplementary 1.5% allocated to critical infrastructure protection. The uplift in 2027 will be financed by a reduction in the overseas aid budget but it is hard to conceive that any further increase can be achieved without raising taxes. A near-doubling of defence spending, to 5% of GDP, would take it to its highest level since the mid-1950s (see chart below). Faced with an ageing population which will stretch the NHS budget, and a sluggish productivity performance which continues to hold back growth, this government – and the next one – will have to make some very hard choices about how to spend their increasingly limited fiscal resources.

The government needs to improve its performance – and fast

Whether Starmer will be the prime minister beyond the next election remains to be seen. He has defied the doubters before, and is not someone who should be underestimated, but his personal popularity ratings are not high. In the wake of the recent policy U-turns, his net approval rating has dropped to -43% with mounting concern that he has responded to the electoral tactics of Reform UK rather than set the agenda on his terms. Starmer’s government has endured a rocky start and needs to outline an agenda which voters can buy into. Failure to do so in the second year of this government will merely raise the risk that Nigel Farage and his band of upstarts could come close to getting their hands on the levers of power in 2029.

As was the case with Labour’s big win in 2024, voters do not have to buy into Reform UK’s policies to reward them at the ballot box. They just need to believe that the status quo is failing, and that it is time to try something different. It may be four years until the next election, and the electorate may eventually forgive Labour for their rocky start, but Starmer and his team cannot afford another year like the last one if they are to win a second term.