Latest news with #JasonZweig

Harvard Business Review

One Simple Way to Get Better at Reading Data

Edwards Deming famously said, 'In God we trust; all others bring data.' As we've evolved from analytics to data science to AI, the world has never been more data driven. And as a leader, you are expected to make sound decisions backed up by data. However, leaders rarely use raw data directly for decision making. Instead, they are likely to be a consumer of statistics calculated by their direct reports to help them make informed decisions. While data are observed, the presenter decides which statistics are relevant in a particular context. Should the average of the data be presented? Should the standard deviation also be presented? Should the complete distribution of the data be presented? Should differences in the raw data, for example sales, or percentage change in market share be presented? What you need to remember is: Statistics are not data; they are descriptions of data. To make smarter decisions, you need to know how to question the statistics or as The Wall Street Journal columnist Jason Zweig recently wrote, 'learning how to talk back to statistics is your first line of defense.' In our experience, we have noticed one particular basic statistical issue—the use of percentages can be used in confusing ways to influence others. The confusion typically resides with the denominator. Like Zweig, when faced with percentages, we are advocating that leaders need to talk back to, that is, question the statistics. One simple but enlightening question to ask is 'What's the denominator?' Let's look at three cases where asking this question could help avoid misinterpretation and confusion. Percentage Versus Absolute Difference A presenter has the choice to provide an absolute or a percentage change. For example, in his article on stock market volatility, Zweig discusses how financial marketers play to your emotions with online headlines like 'DOW PLUNGES BY MORE THAN 1000 POINTS.' He laments the trick of 'hiding the denominator.' This is a classic example of when you need to ask: What's the denominator? Take a look at the equation below. Here, knowing the denominator lets us convert the change in the value of the Dow to a percentage, which is how we typically think about a change in our investments. If the value of the Dow is 40,000, for example, then we can convert to a percentage change by doing the division and multiplying by 100: Now, read that headline again and ask yourself: Is a drop in value of 2.5% a plunge? That is somewhat subjective, but a headline of 'Dow Plunges by 2.5%' does not seem to generate the same sense of urgency. Hence, how we use certain statistics (or not) and verbiage can be persuasive and mislead decision makers. As a leader, it is prudent to ask why the presenter is choosing to provide raw data versus percentages. For example, if a regional sales manager reports that a new retail outlet increased sales by $100,000 this month, knowing what sales were last month is very relevant. If sales last month were $200,000 that's an impressive 50% increase in sales. If sales were $1,000,000, then it is a less impressive increase of 10%. This same persuasive use occurs when only presenting the percentage change. If the regional sales manager reports 'We had a decline in sales in our Manhattan store this past month, but it is only 2%,' it might be good to know the denominator of this percentage. If the Manhattan store is a very high-performing store, 2% might be a lot of revenue. The bottom line is to be fully informed. You should always expect to receive the percentage and the denominator, the relative and the absolute difference. For example, 'Sales increased by 50%, from $200,000 to $300,000.' Another issue we have seen is what we call the past participle problem. Quite simply, if a percentage triples (or doubles) the absolute amount only triples (or doubles) if the denominator is the same in both cases. If your marketing manager says your market share has tripled in the last year, that is likely to be very good news. But it doesn't mean that revenue has tripled over the same period. In fact, it's possible that revenue decreased. Suppose last year's revenue was $50 million and the market revenue was $1 billion. Your market share was 50/1000 = 5%. If the market shrinks dramatically, say to $200 million and your market share this year is $30 million, your market share has tripled from 5% to 30/200, or 15%, but your revenue dropped by $20 million. Always ask, 'What's the denominator?' In this case the market size in the previous year, and the market size in the current year are the relevant denominators. The Biased Denominator Our second case involves a biased denominator, most often associated with percentages from survey responses. Although somewhat dated, in his column on misapplications of statistics, Arnie Barnett provides an excellent example of this case. In the 1980s, Midway Airlines operated a shuttle between Chicago and New York City. On October 20, 1983, an advertisement in the New York Times stated '84% of frequent business travelers to Chicago prefer Midway Metrolink over American, United, and TWA.' Well, what's the denominator here? Presumably, they surveyed frequent business travelers between New York and Chicago to see which airline they preferred. Of course, one could ask, 'How frequent does one have to fly between New York and Chicago to be counted?' The bias in the denominator in this case is even more blatant. In very small print at the bottom of the ad they provide the answer to 'What's the denominator?' It states, 'Survey conducted among Midway Metrolink passengers between LaGuardia and Chicago.' So, apparently, the denominator only included passengers on their flights! As Barnett indicated, the only conclusion you can really draw from this survey is that 16% of their own customers prefer another airline. As a leader, you will likely track metrics like customer satisfaction and employee engagement. Consider an employee engagement survey which results in 80% of the respondents reporting high job satisfaction. You should ask 'What's the denominator?' For example, if the survey was only sent to non-customer-facing employees, the results would likely be biased. With survey results, you will benefit from knowing the percentage of respondents in each category of response and the raw numbers. In the case of voluntary customer satisfaction surveys, there is always the danger of a bias from only receiving extreme responses (extremely satisfied or extremely unsatisfied customers). Knowing the percentage of customers responding versus the number of surveys distributed, that is, the percentage of customers who respond provides some valuable information on how representative the survey statistics might be. The Flipped Conditional In January 2025, the U.S. Surgeon General Vivek Murthy issued an advisory on alcohol consumption and the risk of cancer. The advisory describes evidence of a causal relationship between alcohol consumption and several different types of cancer. For some types of cancers, the evidence suggests that the risk of cancer increases even for low or moderate consumption of alcohol. One of the courses of action recommended was to expand the warning label on alcohol to include the risk of cancer. A rebuttal to the need for expanding labeling on alcohol followed in The Wall Street Journal and illustrates what we call the flipped conditional. An editorial board member questions the data used by Dr. Murthy and then uses the following argument opposing Dr. Murthy's recommendation: 'the report partially attributes only 17% of these estimated deaths to moderate drinking. Of the 609,820 cancer deaths in 2023, this would mean moderate drinking contributed to 3,400 or about 0.6%.' What's the denominator in this argument? The denominator here is the number of cancer deaths (609,820). The 0.006 is the probability of your cancer being attributed to moderate drinking given that you have cancer. The relevant probability to assess the risk of moderately drinking alcohol is the probability of getting cancer given that you moderately drink alcohol. Think of it this way, how many people have cancer is irrelevant to the risk of cancer from moderately drinking alcohol, precisely because a lot of other things can cause cancer. The Surgeon General's Advisory provides estimated risk of cancer based on gender and the amount of alcohol consumed. These are the relevant statistics one needs to answer questions like 'If I am a male who consumes one alcoholic drink per day, what is my risk of developing cancer?' Suppose your marketing team is reporting on how effective their free trial offer has been and states '75% of our customers who purchased our upgraded premium product have used our free trial!' That sounds very impressive. However, this metric is not relevant for determining the effectiveness of the free trial offer. It is using the wrong denominator. To assess the effectiveness of the free trial offer, you don't need the percentage of premium purchases who used the free trial, you need the percentage of free trial users who wind up purchasing the premium product. To illustrate this, let's imagine a simple scenario. Suppose 1,500 customers took the free trial upgrade, 100 customers purchased the new upgrade and of the 100 who purchased the new upgraded product, 75 had used the free trial. That is, the conversion rate was only 5%. We believe it is always a good idea to question the data. When percentages are used, it is imperative that important information is not masked by the statistics. Ask for percentages and absolutes to both be discussed. Clarity comes by asking 'What's the denominator?' If you want to know how effective something is, it needs to be in the denominator.