Rationality meets reality

 The rational expectations (RE) revolution which swept through macroeconomics in the 1970s and 1980s has changed the way we think about many aspects of macro. As theories go, it is coherent and persuasive and has allowed us to think differently about many aspects of economics and finance. But there have been rumblings recently from respected professionals in the field expressing doubts about its usefulness. As one who has never fully bought into the idea that this is actually how people form expectations, I am obviously prone to confirmation bias, but clearly I am not the only one who has reservations about one of the key underpinnings of modern macroeconomics.

What are rational expectations?

In very simple terms, RE assumes that economic agents make the best use of all currently available information to make predictions about future events in a logically consistent manner. The upshot is that individuals do not make systematic forecast errors (although they can make random errors). This appears uncontroversial at first glance but it has profound consequences for policymakers. Prior to the work of Robert Lucas in the 1970s, it was assumed that a paradigm used to assess the outcome of a policy change would either remain unchanged in future, or would change only slowly as expectations adapted to new evidence. But Lucas pointed out that as economic agents recognise and internalise the way policy affects the economy, they will change their expectations formation process. As a result, the old paradigm is no longer valid. Applying the same policy options in future would result in different outcomes because agents would anticipate what was likely to happen and act accordingly.

A simple example is the  Phillips curve, which was based on the idea that there is a stable and exploitable inverse trade-off between inflation and unemployment. In this static world, a policymaker wishing to reduce unemployment would be prepared to allow inflation to rise. But in a world where expectations are formed rationally, they can only get away with that once.  Next time round workers push for higher wage claims to offset the erosion of real wages, with the result that employment falls (unemployment rises). As it happened this was pretty much what happened in the 1970s as the inverse relationship between the two broke down.

Lucas led the intellectual charge of New Classical economics which usurped the dominant Keynesian paradigm, and challenged the efficacy of discretionary macroeconomic policies by arguing that if individuals can foresee the consequences of policy changes, attempts to manipulate the economy through fiscal or monetary policy become less effective. The New Keynesian response was to synthesise Keynesian principles with insights from the New Classical revolution. Crucially, however, they did not reject the RE hypothesis.

Finance, too, has been captured by the RE revolution. RE are a key component of the Efficient Markets Hypothesis, according to which asset prices reflect all available information, and market participants form rational expectations about future events. In an efficient market, it is assumed that investors cannot consistently achieve abnormal returns by exploiting past information because prices already incorporate all relevant data. Furthermore, the Capital Asset Pricing Model (CAPM) assumes that investors form homogeneous and internally consistent expectations about returns, which is related to the rational expectations idea that agents' forecasts are consistent with the model they use.

Are expectations formed rationally?

Rational expectations are, to use the jargon, ‘model consistent’. In other words the average predictions across all economic agents match the predictions of an economic model which captures the true structure of the economy. Obviously, we are not solving complex models of the economy to derive views about the future. Instead, we rely on heuristic rules of thumb, public forecasts, market signals or simplified mental models. If these rules generate outcomes in line with how the economy actually works, we can still proceed on the basis that agents form rational expectations. But it is questionable whether such rules actually work, particularly at times of elevated uncertainty such as we are experiencing today. Given the raised prospect of extreme outcomes in the wake of Donald Trump’s election, we have even less certainty about what the world might look like in future. Due to this lack of information, agents can be excused for simply extrapolating forward based on past performance on the basis that this represented “normality” and that risks are evenly distributed around this outcome. It might be a rational way of looking at the world, but now expectations are being formed adaptively rather than in a model-consistent way.

What does the evidence tell us?

Nowhere are RE more central than in financial markets, where asset prices are typically assumed to reflect the rationally expected present value of future cash flows. In an important 1981 paper, Robert Shiller examined this idea by testing whether fluctuations in stock prices could be explained by changes in expectations of future dividends. According to the standard RE-based present value model, most of the variability in prices should come from new information that affects expected future dividends. However, Shiller found that actual stock prices were far more volatile than the relatively stable stream of realised dividends would justify. This result, often described as the “excess volatility puzzle,” called into question whether prices are set purely on the basis of rational expectations, or whether they are also influenced by factors such as changing risk premia, investor sentiment, or other non-fundamental forces.

In an attempt to update the Shiller methodology, I computed the ex post “fair value” of the S&P (in real terms) by discounting actual realised dividends over a five-year horizon, assuming perfect foresight of future payouts. As the dataset extends only to June 2025, the perfect foresight calculation is only feasible up to June 2020; for subsequent periods, I extrapolated using the trailing 12-month average dividend to maintain a continuous valuation series. While this is not exactly comparable, it is an approach often used in the academic literature. The results suggest that around the time of the dot com bubble in the late-1990s, and again around the time of the Lehman’s bust, equities were overvalued relative to fundamentally justified levels. However, these periods pale in comparison to the post-2020 period (methodological differences notwithstanding), suggesting that equities may be experiencing a period of irrational exuberance.

Providing statistical evidence for the existence (or absence) of RE is challenging. One approach is to compare the fundamentally justified price – defined above as the discounted value of future earnings – with the actual observed price. If RE hold, all available information should already be reflected in the price, implying that the difference between the two should not be systematically related to any other variable. Consequently, regressing this difference on another metric should yield a coefficient that is not statistically different from zero (see below).

However, the results suggest that a regression of the difference on observable variables such as the P/E ratio or dividend yield do indeed generate coefficients which are statistically significantly different from zero. The charts (below) plot the forecast error – defined as the difference between the actual return and the expected return – against the observed P/E ratio and dividend yield. Under the RE hypothesis, forecast errors should be purely random: they should not be systematically related to any information known in advance. This should appear as a scatter of points randomly distributed around zero, with the fitted regression line essentially flat. But the plots show a statistically significant upward slope, suggesting that forecast errors are systematically related to both the P/E ratio and dividend yield, which imply that investors could, in principle, have used these variables to improve their forecasts. This provides some evidence against RE, as it implies that prices do not fully incorporate available information at any given time.

You don’t just have to take my word for it. Cliff Asness, one of the most astute and intellectually rigorous portfolio managers out there, wrote an excellent paper in 2024 arguing that markets have become far less efficient since the early 1990s. Asness offers three reasons why markets are now less efficient compared with the pre-1990 period: (i) the rise of indexing has made stock prices more inelastic with respect to new information; (ii) an extended period of low interest rates has distorted investors ability to respond appropriately to changed information and (iii) the rise of social media has amplified trend following and momentum strategies at the expense of rational information processing.

One worrying thought is that if markets, with their access to huge amounts of data, are not processing information in a manner consistent with RE, what is the likelihood that households are doing it? This matters because RE are absolutely central to modern DSGE (Dynamic Stochastic General Equilibrium) models in which agents (households, firms, policymakers) are assumed to form expectations about the future that are model-consistent. If expectations are not formed rationally, forecasts based on such models may be potentially biased, and policy outcomes may be less effective.

Final thoughts

Looking at market movements in recent months, perhaps we can be forgiven for thinking that the past is the only guide we have to future performance. But such reliance on past trends carries its own risks. Adaptive expectations, by definition, anchor forecasts to recent experience, making them slow to incorporate new structural shifts or unprecedented shocks. In the current environment of heightened uncertainty, characterised by a global political realignment and the disruptive potential of technological change, such inertia may lead to systematic forecast errors. Instead of anticipating turning points, markets and policymakers risk being repeatedly surprised by outcomes that fall outside the narrow band of recent history.

Moreover, if everyone leans too heavily on the same backward-looking heuristics, market dynamics themselves can amplify volatility. Herding behaviour may set in, reinforcing bubbles or deepening downturns as agents all update beliefs in the same direction, as Asness implies. In this sense, the process of expectations formation becomes not merely a passive reflection of past data, but an active force that shapes the trajectory of the economy.

Unfortunately, it is extremely difficult to capture the complexity of the expectations formation process, while those which rely solely on historical patterns risk missing the disruptive events that define each economic cycle. Whether expectations can ever be fully rational in the strict sense remains debatable – but recognising the limitations of both model-based and adaptive approaches is a step toward better decision-making in an uncertain world.