Why Do Estimates for COVID-19 Outcomes Keep Changing?

Dear Frustrated With Ever-Changing Curves, We should grant COVID-19 outcome modelers the benefit of the doubt that they are doing the best they can with the information they have on any given day, in much the same way we accept and expect predictions from meteorologists and stock market analysts to be changeable. On any one day, you can skim sources of pandemic information and hear wildly different estimates for outcomes such as deaths, the timing of the peak, hospitalizations, and how long we’ll have to stay home. And, day to day, each source even changes its own estimates! It may seem like total chaos. You might feel like no one knows what is going to happen and all of these estimates are just big fat guesses. You could imagine a brood of chickens pecking at buttons generating these numbers! Well, it’s not quite that hopeless, but it is complex. In the discussion that follows, I hope to illuminate why there are many different estimates and why they change so often. And I’ll hopefully convince you not to call the folks making these predictions incompetent. I’ll share why this phenomenon is nothing to get upset about. This is natural and to be expected from a pandemic caused by a new-to-world pathogen.

Possibilities, probabilities, and predictions, oh my!

When you are given COVID-19 outcome information, first decide if what is being shared is a possibility, a probability, or a prediction.

Some of the confusion from changing forecasts is not actually caused by information that changes, it is caused by the person hearing the news not realizing which of the three things they are hearing. These three terms don’t mean the same thing. A POSSIBILITY is the broadest possible range of everything that could happen. Whenever a new virus emerges there is a range of possibilities for how it might affect humans. In one extreme, the best-case scenario, a new pathogen might not be able to infect humans and zero people get sick. And on the other extreme, the worst-case scenario, if just the right pathogen at just the right time under just the right circumstances emerges, it could infect every single person on earth and cause humans to go extinct. We know a little bit about the properties of the SARS-CoV-2 virus, though, so the range of all possibilities shared by modelers is narrower than the theoretical example for any pathogen I just shared. The lowest and the highest numbers that you have heard are the worst and best-case scenarios created using what we know so far about COVID-19. When you heard that 2 million Americans might die, that was a possibility under the worst-case scenario conditions that nothing at all changed about how we interacted, traveled, and gathered.

Imagine two people are planning a party in a week and need to know what the weather might be like then.

They need to decide if it will be held outside or inside. If it’s going to be nice, they’ll buy hot dogs and hamburgers to grill outside and invite more people- they have small houses but huge yards. If it’s going to rain and be cold, then they'll plan on getting pizza delivered and inviting fewer people. With me so far? OK. We’ll call these two people, Mr. Easy Going and Mr. Hot Temper. They each handle the steps between looking at possible forecasts, making tentative plans, and then experiencing the actual result quite differently. Both party planners look at a few different weather forecasts. The predictions range from saying there is between a 0% and 25% chance of rain, partly sunny to full sun, and a high of 52 to 58 degrees. Both decide to plan for an outdoor party. OK. Fast forward to the parties. The weather is miserable. It never got above 45 degrees and it rained most of the day. Guests are crammed inside tiny houses eating hot dogs cooked on a tiny George Foreman grill. Mr. Easy Going is thinking, “It’s too bad we can’t be outside. But we’ll make do. I saw the forecast changing over this week and I knew yesterday it’d probably rain. I’m not surprised, just disappointed.” Mr. Easy Going understands how modeling based on complex inputs that change over time works. He understands that weather forecasts are somewhat unpredictable but get more accurate as you get closer to the date you were trying to predict. Mr. Easy Going generally believes that meteorologists are doing the best they can. He respects the experts who predict the weather, even when their predictions aren’t perfect. In contrast, Mr. Hot Temper is thinking, “I cannot believe the stupidity of those no-good weather people. Look at this screenshot I took 7 days ago. They lied! They told me it’d be sunny with a low chance of rain. They must be either incompetent or intentionally plotting to ruin my party.” Mr. Hot Temper does not understand how models based on complex inputs that change over time work. He is just looking for someone or something to blame because he is frustrated about the weather that he cannot control. Mr. Hot Temper is biased toward dismissing the value of science and expert-based predictions, thinking the weather people are no smarter than he is. The same logic applies to coronavirus pandemic modeling! People react differently to the changeability and inherent unpredictability of models that change over time. I am hoping that by gaining an understanding of why the outcome predictions are so changeable, it will make it easier to accept that this is the way this situation is and will continue to be, without feelings of anger or frustration. We'll get back to the weather example in a bit. Next, I’ll use some simpler examples to show how three factors affect the likelihood of an accurate prediction: the number of variables, the role of human behavior, and the relative amount of historical knowledge.

If I flip a coin three times in a row, how difficult would it be for someone to guess the number of tails accurately? Very easy.

For each flip of a coin, there are only 2 possibilities, heads or tails. If I flip the coin 3 times in a row, then there are only 4 possible outcome scenarios to this question. The answer has to be that tails show up either three, two, one, or no times.

H, H, H → Heads 3 times (Scenario 1) H, H, T → Heads 2 times, Tails once (Scenario 2) H, T, H→ Heads 2 times, Tails once (Scenario 2) H, T, T → Heads once, Tails 2 times (Scenario 3) T, H, H→ Heads 2 times, Tails once (Scenario 2) T, H, T→ Heads once, Tails 2 times (Scenario 3) T, T, H→ Heads once, Tails 2 times (Scenario 3) T, T, T → Tails 3 times (Scenario 4)

Therefore, there are only 4 possibilities. But, which of these 4 possible scenarios is most probable? Well, if you look at the 8 different combinations of how the coins may fall, you can see that Scenarios 2 and 3 are most likely to happen. Therefore, the probability of the outcome being that you get tails either once or twice is highest. And it is still possible, but not as probable, that you’ll get a tail either never or all three times. Possibilities and probabilities are based on facts and an understanding of the system. We know how coins are designed, they only have 2 variables to worry about- heads and tails. And, we have a lot of knowledge about this. All of us have experience flipping coins. Plus, the rules of this game don’t change. What I think or how I act cannot have an effect on the design of the coin. It is a static fixed object. (this may sound like nonsense right now, but hold tight, because this will evolve in the next examples). So it's fairly easy for us to predict the outcome of a coin toss. There's only a limited range of possibilities, we know exactly what they are and we have a lot of experience with it. All right, simple example. Now, let's make it slightly more challenging.

If I draw one card from a full deck, how difficult would it be for someone who knows how a deck of cards is designed to guess the outcome accurately? A little more difficult than the coin toss, but still easy.

In the coin example, you had a one in four chance of guessing the right prediction because there were only four possible scenario outcomes. But now, we have 52 cards, each unique. So there are 52 possibilities. But which outcome is more probable? They are all equal. The probability of pulling an Ace of spades is equal to the probability of drawing a three of hearts. What if I asked you which is most probable, that I pull an Ace or a Heart? You would say a heart, right? Because there is a 25% chance you will pull a heart and only a 4/52 or ~7.7% chance of drawing an Ace card. Compared to the coin toss example, it’s more difficult to make a prediction in this card draw system because there are more variables, more possible outcomes. Increasing the variables in a system makes it inherently more difficult to make accurate predictions. Ok, now back to the weather analogy.

If I want to know what the weather will be like ten days from now, how difficult would it be for someone who is an expert in weather data to guess the outcome accurately? Somewhat challenging, compared to coin tosses and card draws.

This comes from someone who lives in Ohio and it is April. So, the range of possibilities for the weather in 10 days is huge. It could be sunny and 80 or it could be snowing with no sun. These, literally, are all possible in Ohio in April! The variables that go into weather prediction are many. Unlike a coin toss or a card draw from a deck, zillions of molecules all affected by one another that move around ultimately determine the weather. The more possibilities exist in a system, the more difficult it becomes to figure out what is most probable and to make accurate predictions. You and I can predict coin toss and card draws because the number of variables is something our brains can handle. But, there are too many variables involved in weather prediction, thousands upon thousands of data used, that our brains can no longer do it without help. We need experts that rely on computer algorithms and modeling to do the calculations for us. It starts to become something only experts trained in this with the help of computer programs can do. Meteorologists with the help of computer programs that model using data and assumptions about the weather are still able to give us reasonably accurate weather forecasts, though. I would argue this is easier than predicting the pandemic outcomes because weather forecasters have two advantages over pandemic predictors: humans don’t affect the weather and we have lots of past data. When human behavior can directly change a system, that makes it more difficult to predict. What you and I do today does not change what the weather will be like a week from now. So at least a weather forecaster doesn’t have to try and predict what people are going to do as part of their model. Weather-related data has been collected for decades and can be used to teach computer programs what has happened and what is possible. Compared to pandemic forecasting, weather forecasting benefits from lots of historical knowledge. Even with these advantages, weather forecasts are wrong sometimes. And we all accept and expect them to be wrong sometimes. You and I don’t go back to weather forecasts from 2 weeks ago and scream and holler things like “meteorologists are idiots... They don't know what they're talking about. They are just making stuff up to make me look bad!”, right? I don't do that. We don't do that. We should extend the same reasonable behaviors to the coronavirus pandemic predictions as we do our weather people. Because that's actually even more complicated than predicting the weather.

If I want to know how many people will die in my country from COVID-19, how difficult would it be for someone who is an expert in pandemics to guess the outcome accurately? Even more challenging than predicting the weather.

There are two main reasons why this is so much more difficult than predicting the weather. First, we don’t have experience with human coronavirus pandemics to rely on. Second, our own human behavior changes the outcome. When it first emerged, we didn’t even know what the range of possibilities that the SARS-CoV-2 virus was capable of was. This is a brand new virus that just emerged. It has a cousin, the original or classic SARS-CoV that caused the SARS scare in 2003, but we saw right away that this one behaved differently. The coronaviruses historically have rarely caused problems in humans. So our knowledge about this whole family of viruses is not as well-developed as it is for other types of bacteria and viruses, like influenza viruses that cause the typical flu. And, not only is it a novel virus, but it is also the first time ever that a coronavirus has caused a pandemic. We have a little bit of knowledge of pandemics over history, but not much. Pandemics don’t happen too often. But even the ones we have historical knowledge about were all caused by other types of microbes, not a coronavirus. So we actually have zero knowledge of this exact situation. You have seen and heard many different estimates for how many people will die in the USA before this is over. Some worst-case scenarios had us losing 2 million people if we did nothing. Now they are talking about numbers under 100,000 people, thankfully. Why this big range of probable outcomes? It is because in this system, what each one of us does changes the result. What you are seeing in the news are ranges of probable outcomes that depend upon what we assume people will do and for how long. Let me give you one more example, though. One that makes predicting pandemic outcomes seem easy. I don’t want you to be left off thinking that all the models are useless. They are valuable and they are “right-ish” and over time the pandemic outcomes predicted will become more and more accurate.

If I wanted to know the $ value of IBM stock in 30 days, how difficult would it be for someone who is an expert stock analyst to guess the outcome accurately? Even more challenging than predicting the pandemic!

The stock market price is difficult to predict because there are unlimited variables that can affect it. But, what makes it particularly difficult to predict, is that it is almost exclusively based on human behavior. Even more so than the coronavirus. At least with the virus, we can learn how it physically interacts with human cells, and how it is transmitted, and how well it lives on surfaces. There are some hard science and solid data we can input into our models. With stock price predictions, we have to guess at what people within and outside the company might do. This is nearly impossible. At least, though with stock market prices for companies like IBM that have been around a long time, we do have a lot of historical data we can review about how it was valued over time under different circumstances. If you can appreciate that predicting the weather is not an exact science and if you can appreciate that it is even more difficult to predict the price of a stock in the future, then you should be able to appreciate that predicting this pandemic is at least, if not more, difficult.

Here’s a visual summary of the concepts I just outlined.

Please, when you hear estimates from models about COVID-19, remember to: 1) Appreciate that all models used to guess at the path of a pandemic contain uncertainty and variability that is inherently unavoidable.

2) Understand that looking back in time at past estimates created by models is not useful because they were not predictions, they were merely possibilities based on the available data of that moment. Don’t waste energy or time being part of the groups of people you are pointing fingers at and saying “you were wrong” when looking back at old estimates.

3) Expect that the models will continue to change and that different models developed by different people using different sets of data and assumptions will not agree. It does not mean the people designing it are incompetent.

4) Know that over time the estimates will get more and more accurate. In fact, when all the models start to agree with one another, that will be one sign that this pandemic is more understood and that it is closer to over. I hope this helped you embrace the ambiguity in pandemic model predictions! COVID Carrie