Depending on how you think of it, Bayesian updating is the most — or least — used decision-making technique in business
With today’s post, I am taking a break from Thematiks’ usual focus on peer-reviewed research to highlight a recent article by Management Professor Brian T. McCann (Vanderbilt). The focus of McCann’s article is decision-making under uncertainty; however, unlike most scholarly writing, McCann’s audience in this piece is managers instead of academics.
When faced with questions such as whether to launch a new product or acquire a fast-growing startup, managers often rely on calculations of probability based on past experiences to arrive at a decision. In other words, they calculate the “odds” of a given outcome based on what they have observed in the past. In this scenario, probabilities are typically calculated for events that have been repeated enough to have an outcome history that can serve as the basis for a projection about the future. This approach is common because most decisions that managers make are typically not new; rather, they are the latest instances of choices that have been made many times in the past. Determining odds on the basis of observed past outcomes is known as an objective (or “frequentist”) approach. For example, a manager’s company has launched ten new products a year for ten years, of which 50% succeeded annually, so there is a sound basis for believing that any new product launched in the future has a 50% chance of success. Once the objective probability is set, the manager can multiply it against the outcome’s expected value to calculate the decision’s present value.
The objective perspective on probability, notes McCann, “is likely to be the first, and perhaps only, view [of probability] most managers have been exposed to in their careers.” However, he believes that another less-familiar approach “will help you improve your ability to accurately estimate probabilities across a wide range of contexts, making you a better decision-maker more equipped to deal with the need to make choices in a world of increasing uncertainty.” McCann’s preferred methodology is called Bayesian (or “subjective”) inferencing, and he gives two reasons why it should be as familiar to managers as the more common objective method. First, “the [Bayesian] process results in better probability estimates where ‘better’ means that your estimates are more accurate.” Second, “even if calculating exact probability estimates is not critical to you, being a more explicit Bayesian makes you a clearer thinker in a number of significant ways.”
How Bayesian updating works
Developed by the English statistician Thomas Bayes, the Bayesian approach is related to a process familiar to most people, such as deciding whether to eat at a familiar restaurant. Past positive experiences are the foundation for the expectation of a good meal. A patron might hesitate about returning should a foodie friend give him a different opinion, having dined there the previous evening. For the now-hesitant diner, the calculation about the likelihood of having a good experience is a composite of two things: his personal history (objective probability) and the opinion of his disappointed friend. Indeed, he may even call a few other friends to add additional data, which would expand his (now) subjective analysis even further. The fact that subjective probabilities depend on an individual’s knowledge, notes McCann, “has significant implications; among the most important is that if knowledge changes via encountering new information or evidence, probability estimates should change, too.”
The process by which we continuously update probabilities based on new information is called Bayesian updating, and we can apply it in a business setting as follows. Imagine a CEO debating whether to enter into a new market. Based on prior company experience and recent competitor outcomes, the CEO forms a hypothesis that the move has a 40% chance of working. We label the likelihood of the CEO’s hypothesis being true as p(b), i.e., the “probability of the belief.” To be more confident, the CEO funds an expert focus group to gather more evidence, and the group agrees with her hypothesis. The CEO then asks herself a simple question: if it is a good idea to enter this market, how likely is it that the focus group members would know that today, with a score of 1.0 meaning they would know it absolutely and a score of 0.0 meaning they would never know it. She decides the group would know it 7 times out of ten, and assigns a score of 0.7 to the group’s positive conclusion and a score of 0.3 to the opposite possibility. The former calculation is labeled p(e|b) — the “probability of the belief on the condition that the evidence is right” (the “|” is shorthand for on the condition that). The latter calculation is labeled p(e|~b) — the “probability of the belief on the condition that the evidence is not right,” i.e, wrong. As McCann notes, “these two pieces of information, p(e|b) and p(e|~b), indicate the strength of the evidence generated” by the focus groups.
Once she knows p(e|b) and p(e|~b), the CEO inserts those values into this formula, known as Bayes’ rule:
To see how this formula works, let us return to our CEO. Her p(b) estimate is 0.4, meaning she thinks there is a 40% chance of success (and thus p(~b) = 0.6). Because she knows p(e|b) is 0.7 and p(e|~b) is 0.3, Bayes’s rule indicates that the CEO’s probability of success is 0.61, i.e., ([0.7 x 0.4] / [0.7 x 0.4 + 0.3 x 0.6] = 0.61). In this scenario, the focus group evidence shifts a strategic option from one that is unlikely to succeed (40%) to one that has a somewhat positive probability of success (61%). If we further assume that the CEO commissioned a consulting report that concluded the market expansion had a very good chance of success (scored at 0.8), the CEO then updates her calculations once more with a new result of 0.86, i.e., ([0.8 x 0.6] / [0.8 x 0.6 + 0.2 x 0.4] = 0.86). With two rounds of scored evidence indicating a very positive subjective probability, she decides to approve the market expansion.
In the end, the CEO’s updated (or “posterior”) belief is a combination of her pre-focus group belief p(b), and the strength of the new evidence, represented by p(e|b) and p(e|~b). The beauty of the Bayesian approach, notes McCann, is that “it is simply a mathematical expression of how to quantify learning from experience” that “expresses how a subjective degree of belief should rationally change in light of evidence or data.” Indeed, in the Bayesian mindset, the point of evidence is that both the posterior belief p(b) and carefully analyzed new evidence p(e|b) complement each other. In other words, it is critical to not let either past experience or new evidence alone guide a decision — both must be synthesized in a consistent and systematic way in order to make the best decision with uncertain information. Indeed, digging deeper into the formula, one learns that a very low p(b) will require very high confidence evidence to shift an unlikely outcome to a likely one. This is as it should be, notes McCann who believes that “extraordinary claims require extraordinary evidence.” In other words, to a Bayesian, a low probability belief does not become a high probability one unless there is very strong evidence in favor of that change.
Since the publication of McCann’s paper, a team of researchers pushed back on his suggestion that managers should adopt Bayesian decision-making. In a separate commentary, the team presented two critiques. The first asks whether Bayesian updating is useful — or even possible — when managers cannot assign numerically definite subjective probabilities. As the team notes, there are many decision situations where objective probabilities can’t be set so Bayesian decisions would be no better than those made the simpler objective way. To this critique, McCann replies that “a lack of objective probabilities is definitely an issue in many managerial decisions,” but “it is not clear to me that this precludes the usefulness of Bayesian updating.” In other words, there may be rare occasions when truly novel situations arise, but, as noted earlier, in business that situation is the exception and not the rule — most business choices have some internal or external analog on which to base subjective probabilities.
The second objection is that Bayesian updating is inapplicable in cases where a decision-maker is unable to define all possible outcomes relevant to a decision. In other words, Bayesian updating does not account for “unknown unknowns.” To this objection, McCann notes that the Bayesian approach “does require that decision-makers are open to the possibility of surprise and incorporate that awareness into their probability estimates.” Indeed, one way of doing so is to include a “black swan” [my term] round of evidence that accounts for surprises and scoring that possible evidence as if it existed, i.e., highly unlikely but still possible.
Of course, the simplest critique of Bayesian thinking is the old “garbage in/garbage out” rule. Indeed, as the science writer John Horgan noted in his Scientific American article on the appeal of Bayesian thinking in the sciences: “If you aren’t scrupulous in seeking alternative explanations for your evidence, the evidence will just confirm what you already believe.”
Though unfamiliar to many people, Bayesian techniques have a long track record of success. Bayesian approaches, notes McCann, “have helped firms operating in the U.S. movie industry learn as they enter multiple international markets; it informs decisions of whether to initiate phase three testing of new drug candidates at pharmaceutical company Amgen; Amazon uses machine learning informed by Bayesian techniques to improve customer recommendations and optimize logistics; staffing company Triplebyte appeals to principles of Bayesian updating to motivate its use of a series of coding tests and interviews to screen technical job candidates; and the Pennsylvania Investment Network advocates for a Bayesian approach to improve outcomes in angel investing.” Indeed, perhaps the most ubiquitous application is anti-spam software, which uses Bayes theorem to calculate a probability a certain message is spam and then continuously updates that probability based on user tagging.
As noted earlier, in many ways most managers practice some form of Bayesian updating intuitively. The problem with the intuitive application, in McCann’s view, is that it leads to errors in even expert decision-makers. For example, “people tend to ascribe too much weight to new information causing it to skew probability estimates; they are also more likely to update beliefs after encountering favorable information compared with negative information.” Adopting a more formal Bayesian approach would not only reduce those errors, it has many other benefits, from being more open-minded to new ideas to increased understanding of the real value of new evidence to more accurate communication between leaders and teams when evaluating risk.
Reflecting on his final claim reminded me of a 2018 HBR article that contains the illustration reproduced in Figure 1 below.
Figure 1: How people interpret probabilistic words (Source: HBR)
The graphic highlights how people interpret phrases about probability in many different ways. As the authors note:
Most — but not all — people think “always” means “100% of the time,” for example, but the probability range that most attribute to an event with a “real possibility” of happening spans about 20% to 80%. In general, we found that the word “possible” and its variations have wide ranges and invite confusion.
We also found that men and women see some probabilistic words differently. As the table below shows, women tend to place higher probabilities on ambiguous words and phrases such as “maybe,” “possibly,” and “might happen.” Here again, we see that “possible” and its variations particularly invite misinterpretation.
In McCann’s view, reducing ambiguity in discussions of probability the Bayesian way would help alleviate the problem documented in the HBR piece without a major increase in intellectual effort. Indeed, adopting the Bayesian approach requires just a few basic steps:
Begin by thinking probabilistically, shifting your view of probability to see it as a means to quantify the strength of subjective beliefs. Bayesians think probabilistically rather than in black-and-white terms.
Next, start assigning exact estimates to your degrees of belief. As you encounter new evidence, update those prior estimates and base the magnitude of an update on the strength of the evidence. This step requires you to think about not only how likely it would be to observe that evidence if your beliefs are correct, but also how likely it would be to observe that evidence if your beliefs are incorrect.
Finally, combine prior probabilities and evidence strength estimates using Bayes’s rule.
In the end, the best argument for McCann’s position may be the simplest: since thinking in Bayesian ways is something most managers do intuitively, why not get it right. Indeed, McCann himself, in his rebuttal to his critics, wryly observed that based on their objections he has “slightly downgraded” his probability estimate for the belief that Bayesian updating is useful. One can assume that the updated probability does not invalidate his case for a Bayesian upgrade to how we make decisions with imperfect information.
McCann BT. Using Bayesian Updating to Improve Decisions under Uncertainty. California Management Review. 2020;63(1):26-40. doi:10.1177/0008125620948264