UMich Transition to Online Surveys Breaks Time Series, Challenging Interpretation

After over 40 years of exclusively relying on telephone surveys, the University of Michigan’s Surveys of Consumers completed its transition to an online survey in July 2024, creating a structural break in the time series and complicating its use for monitoring and forecasting U.S. consumer developments. 

Since its launch in January 2018, Morning Consult's daily online survey has provided high frequency updates on the state of consumer confidence, and it now offers a longer continuous time series than the University of Michigan for tracking consumer sentiment in aggregate and across geographic and demographic cohorts. 

Background

On April 5, 2024, the UMich announced its transition from a telephone survey to an online survey. The transition began in April and was completed in July. The transition to online surveys was motivated by rising data collection costs and falling response rates, both of which posed challenges to the long-run integrity of the survey data. These issues with phone surveys were known as early as 2007, when the then-SCA Director Richard Curtin proposed the transition to online surveys to address the statistical challenges posed by falling response rates 

What happened

In October 2024, economists Ryan Cummings and Ernie Tedeschi published an analysis in Briefing Book that estimated a 8.9 point decrease in the University of Michigan Index of Consumer Sentiment resulting from the transition to online surveys. They found: “Online respondents are resulting in the level of the overall sentiment and current conditions indices being meaningfully lower, making more recent UMich data points inconsistent with pre-April 2024 data points.” The 8.9 point gap is larger than the University of Michigan’s estimate of 6.6 points, which they published in their April announcement. 

Cummings and Tedeschi adopted a two-pronged approach to estimating the impact of the transition online on the UMich ICS. First, they ran regressions on respondents from the April-June 2024 transition period to compare the impact of taking the survey over the phone vs. online. 

In a second step, they compared their regression-adjusted estimates with the Morning Consult U.S. Index of Consumer Sentiment, a continuous survey of 5,000 daily consumers using the same 5 questions as the University of Michigan. Since its launch in January 2018, Morning Consult’s survey has been fielded online, allowing it to serve as a methodologically consistent benchmark for the University of Michigan. As shown in the graph below, the Cummings and Tedeschi adjustment closely aligns with the Morning Consult data, which does not require any adjusting.  

One key takeaway from Cummings and Tedeschi is the conclusion that “(s)witching to online responses has created a structural break in [University of Michigan’s] sentiment.” For non-economists, structural breaks in time series mean that the data before and after a period of time are so different from each other that it would be inappropriate to directly compare the two. In short, it is tantamount to saying that the University of Michigan’s consumer data began in July 2024, with historical data being so different that it does not make sense to try to rely on the time series from 1978 to present. 

The University of Michigan is just the latest institution to transition to online surveys. In May 2021, The Conference Board transitioned from mail-in surveys to online surveys. Following the structural break in the University of Michigan’s data, Morning Consult now offers longer continuous time series for consumer sentiment or confidence than the University of Michigan or The Conference Board: 

• Morning Consult's Index of Consumer Sentiment began January 2018 with a daily target sample size and frequency of 5,000.
• The University of Michigan's Index of Consumer Sentiment began July 2024 with a monthly target sample size and frequency of 900-1,000.
• The Conference Board's Index of Consumer Sentiment began January 2021 with a monthly target sample size and frequency of 3,500.

Why it matters

For forecasters and those looking to monitor the health of the U.S. consumer, the structural break in UMich’s data seriously limits its explanatory or predictive power on a go-forward basis. 

While adding 8.9 points to the University of Michigan ICS provides a quick fix, the issue likely runs deeper than the top-line metric. Consumers across the country and across key demographics are subject to highly heterogeneous economic and financial experiences. For example, during the COVID-19 pandemic, older adults were more exposed to health risks, while lower-income adults grew more exposed to inflation. Resurrecting the usefulness of UMich’s historical data would require the estimation and application of a conversion factor for each and every demographic and geographic cohort. 

Additionally, given UMich’s relatively small monthly sample size, it will likely take many more months if not years to collect enough respondents in each unique cohort combination to be able to begin this exercise in earnest. In short, the transition to online surveys creates tremendous uncertainty regarding the interpretation of the UMich data and requires fairly complex and time-intensive approaches adjusting the data that are unlikely to be stable for subregions and demographics for months to come.

Finally, in addition to the Index of Consumer Sentiment, the UMich Surveys of Consumers releases other statistics on consumers’ experiences and expectations, including their inflation expectations. Investors and Wall Street and policymakers across the Federal Reserve System have closely monitored consumers’ inflation expectations data released by UMich during this most recent period of elevated inflation. However, similar to UMich’s topline ICS, the UMich inflation expectations data is also likely to exhibit structural breaks or empirical anomalies resulting from the transition to online surveys.