An Interesting Christmas Gift

Over the holidays, the New York Times delivered an unusual juxtaposition of headlines and content, and apparent lack of self-awareness, to illicit such a hearty chuckle from its readers as to make the cheerful Old Saint jealous.


[image originally provided by @ddmeyer on Twitter]

To those imbued with the skill of basic high school Algebra 1, the information in the article about Sony’s revenues for the first four days of release of “The Interview” were enough to solve a unit value problem. If we let R = the number of rentals, and S = the number of sales; then,

• R + S = 2 million 
• $6*R + $15*S = $15 million 

With a little quick symbolic manipulation, we see that S = 1/3 million in sales and R = 5/3 million in rentals. That exercise provided just enough mental stimulation and smug self-righteousness to prepare for the day’s sudoku and crossword puzzles. #smug #math

However, not too far into the sudoku puzzle we might realize that a deeper, more instructive problem exists here, a problem that actually permeates all of our daily lives. That problem is related to the precision of the information we have to deal with in planning exercises or, say, garnering market intelligence, etc. A second reading of the article reveals that the sales values, both the total transactions and the total value of them, were reported as approximations. In other words, if the sources at Sony followed some basic rules of rounding, the total number of transactions could range from 1.5 million to 2.4 million, and the total value might range from $14.5 million to $15.4 million. This might not seem like a problem at first consideration. After all, 2 million is in the middleish of its rounding range as is $15 million. Certainly the actual values determined by the simple algebra above point to a good enough approximate answer. Right? Right?

To see if this true, let’s reassign the formulas above in the following way.

• R + S = T 
• $6*R + $15*S = V 

where T = total transactions, and V = total value. Again, with some quick symbolic manipulation, we can get the exactly precise answers for R and T across a range of values for T and V.

• S = 1/9 * V - 2/3 * T 
• R = T - S 

Doing this we now notice something quite at odds with our intuition - the range of variation between the sales and rentals can be quite large as we see in this scatter plot:



[Fig. 1: The distribution of total transaction values for various combinations of rental and direct sales numbers.]

Here we see that the rental numbers could range from about 800 thousand to 2.4 million, while the direct sales could range from nearly 0 to 700 thousand! Maybe more instructive is to consider the range of the ratio of the rentals to direct sales:


[Fig. 2: The distribution of the ratio of rentals to direct sales for various combinations of rental and direct sales numbers.]

If we blithely assume that the reported values of sales were precise enough to support believing that the actual value of rentals and unit sales were close to our initial result, we could be astoundingly wrong. The range of this ratio could run from about 1.11 (for 1.5 million in total transactions; 15.4 million in sales) to 215 (for 2.4 million in total transactions; 14.5 million in sales). If we were trying to glean market intelligence from these numbers on which to base our own operational or marketing activities, we would face quite a conundrum. What’s the best estimate to use?

Fortunately, we can turn to probabilisitic reasoning to help us out. Let’s say we consult a subject matter expert (SME) who gives us a calibrated range and distribution for the sales assumptions such that the range of each distribution stays mostly within the rounding range we specify.

[Fig. 2a, b: The hypothetical distribution of the (a) total sales transactions and (b) total value assessed by our SME.]

Using the sample values underlying these distributions in our last set of formulas, we observe that in all likelihood - an 80th percentile likelihood – the actual ratio of the rentals to sales falls in a much narrower range – the range of 3 to 9, not 1.11 to 215.

[Fig. 3: The 80th percentile prediction interval for the ratio of the rentals to sales falls in the range of 3 to 9.]

Our manager may push back on this by saying that our SME doesn’t really have the credibility to use the distributions assessed above. She asks, "What if we stick with maximal uncertainty within the range?” In other words, what if, instead of assessing a central tendency around the reported values with declining tails on each side, we assume there is a uniform distribution along the range of sales values (i.e., each value is equally probable to all values in the range)?



[Fig. 4a, b: We replace our SME supplied distribution for (a) total sales transactions and (b) total value with one that admits an insufficient reason to suspect that any value in our range is more likely than any other.]

What is the result? Well, we see that even with the assumption of maximal uncertainty, while the most likely range expands by a factor of 2.7 (i.e., the range expanded from 3-9 to 1.7-18), it still remains within a manageable range as the extreme edge cases are ruled out, not as impossible but as fairly unlikely.

[Fig. 5: Replacing our original SME distributions that had peaks with uniform distributions flattens out the distribution of our ratio of rentals to sales, causing the 80th percentile prediction interval to widen. The new range runs from about 1.7 to 18.]

The following graph displays the full range of sales and rental variation that is possible depending on our degrees of belief (as represented by our choice of distribution) about the range of total transactions and total value.

[Fig. 6: A scatter plot that demonstrates the distribution of direct sales and rental combinations as conditioned by our choice of distribution type.]

By focusing on the 80th percentile range of outcomes in the ratio of rentals to sales, we can significantly improve the credible range to estimate the rentals and direct sales from the approximate information we were given.

[Fig. 7: A scatter plot that demonstrates the distribution of direct sales and rental combinations as conditioned by our choice of distribution type, constrained only to those values in the 80th percentile prediction interval.]

Precise? Not within a hair’s breadth, no, but the degree of precision we obtain by employing probabilities (as opposed to relying on just a best guess with no understanding of the implications of the range of the assumptions) into our analysis improves by a factor of 13.1 (assuming maximum uncertainty) to 35.2 (trusting our SME). If our own planning depends on an understanding of this sales ratio, we can exercise more prudence in the effective allocation of the resources required to address it. Now, when our manager asks, “How do you know the actual values aren’t near the edge cases?”, we can respond by saying that we don’t know precisely, but using simple algebra combined with probabilities dictates that the actual values most likely are not.

The Zen of Decision Making

I copied the following nineteen zen-like koans from the website devoted to the Python programming language (don't leave yet...this isn't really going to be about programming!).

• Beautiful is better than ugly.
• Explicit is better than implicit.
• Simple is better than complex.
• Complex is better than complicated.
• Flat is better than nested.
• Sparse is better than dense.
• Readability counts.
• Special cases aren't special enough to break the rules.
• Although practicality beats purity.
• Errors should never pass silently.
• Unless explicitly silenced.
• In the face of ambiguity, refuse the temptation to guess.
• There should be one-- and preferably only one --obvious way to do it.
• Although that way may not be obvious at first unless you're Dutch.
• Now is better than never.
• Although never is often better than *right* now.
• If the implementation is hard to explain, it's a bad idea.
• If the implementation is easy to explain, it may be a good idea.
• Namespaces are one honking great idea -- let's do more of those!

The koans are supposed to communicate the essence of the guiding principles of programming. Their zen-like fashion is intended to motivate reflection and discussion more so than state explicit rules. In fact, there is a twentieth unstated (Or is it? How's that for zen-like clarity?) principle that you must discover for yourself.

Good aphorisms often find meaning beyond their initial intent. That's the way general, somewhat ambiguous guidance works and why some aphorisms last for so long in common parlance. They're malleable to one's circumstances and provide a kind of structure on which to hinge one's thoughts, concerns, and aspirations (I'm pretty sure horoscopes and Myers Briggs work this way). Some of these aphorisms, maybe all of them, struck me as not only useful as guiding principles for programming but also for decision management in general. Seriously. Go back and consider them again, Grasshopper.

So, let me ask you:

• In what way is decision management like programming?
• How would you interpret these principles, if at all, for use in the role of decision making?
• What do you think is the missing principle?

It's Your Move: Creating Valuable Decision Options When You Don't Know What to Do

The followings is the first chapter excerpt from my newly published tutorial.

Business opportunities of moderate to even light complexity often expose decision makers to hundreds, if not tens of thousands, of coordinated decision options that should be considered thoughtfully before making resource commitments. That complexity is just overwhelming! Unfortunately, the typical response is either analysis paralysis or "shooting from the hip," both of which expose decision makers to unnecessary loss of value and risk. This tutorial teaches decision makers how to tame option complexity to develop creative, valuable decision strategies that range from "mild to wild" with three simple thinking tools.

It's Your Move from Robert Brown

---

Read more here.

A Brief Explanation of Expected Value

When helping people analyze the risks they face in complex decisions, I frequently receive requests for an explanation of expected value, as expected value is a measure commonly used to compare the value of alternate risky options. I’ve found that by now most people understand the concept of net present value (NPV) rather well, but they still struggle with the concept of expected value (EV)*. Interestingly enough, and fortunately so, the two concepts share some relationship to each other that makes an explanation a little simpler.

NPV is the means by which we consistently compare cash flows shaped differently in time, assuming that money has a greater meaning to us when we get it or spend it sooner rather than later. For example, NPV would help us understand the relative value of a net cash stream that experienced a small draw down in early periods but paid it back in five years versus a net cash stream that makes a larger draw down in early periods but pays it back in three years.

EV is similar. By it we consistently compare future outcome values that face different probabilities of occurring.

When we do NPV calculations, we don’t anticipate that the final value in our bank account necessarily will equal the NPV calculated. The calculation simply provides a way to make a rational comparison among alternate time-distributed cash streams.

Likewise, when we do EV calculations, we don’t anticipate that the realized value necessarily will equal the EV. In fact, in some cases it would be impossible for that outcome to be the case. EV just simply provides a way to make a rational comparison among alternate probability-distributed outcomes.

Here’s a simple example. Suppose I offer you two gambles to play in order to win some money. (Not really, of course, because the State of Georgia reserves the right to engage in games of chance but prohibits me from doing so.)

In the first game, there are even odds (probability=50%) that you will win either $10 on the outcome of a head or $0 on a tail.

In the second game, which is a little more complicated, I use a biased coin for which the odds are slightly less than even, say, 9:11 (probability=45%), of your winning. If you win, you gain $15; lose, you pay me $5. Which is the better game to play? Believe it or not, the answer depends on how you frame the problem, most notably from your perspective of risk tolerance and how many games you get to play. If you can’t afford to pay $5 if you lose the second game on the first toss, you’re better off to go with the first game because you will lose nothing at least and gain $10 at best. However, if you can afford the possible loss of $5 and you can play the game repeatedly over numerous times, expected value tells us how to compare the two options.

We calculate EV in the following way: EV = prob(H)*(V|H) + prob(T)*(V|T).

For the first game, EV1 = 0.5*($10) + 0.5*(0) = $5.

For the second game, EV2 = 0.45*($15) – 0.55*($5) = $4.

So, since you prefer $5 over $4 (you do, don’t you?), you should play the first game, even though the potential maximum award is alluringly $5 more in game two than one.

But here's the point about the outcomes. At no time in the course of playing either game will you have $5 or $4 in your pocket. Those numbers are simply theoretical values that we use to make a probability-adjusted consistent comparison between two risky options.

In a follow up post, I will describe what your potential winnings could look like if you choose to play either game over many iterations across many parallel universes.

---

*To be honest, I think part of the persistent problem in understanding is contributed by the term "expected" itself. Colloquially, when people use and hear this term, they think "anticipated." In discussions about risk and uncertainty, the technical meaning really refers to a probability weighted average or mean value. Unfortunately, I don't expect that you should wait for us technical types to accommodate common usage. [back]

So, What Is Your Algorithm?

I thought this story about Schwan’s was interesting for this reason: a 3-4% improvement on revenues of ~$3 billion (2010 Annual report) over less than one year didn’t feel that significant to me.  In fact, if you look at it this way, Schwan’s improved sales by 3.5% * $3billion/3million purchasers = $35/purchaser.  That’s two additional entrees, or 5 additional pizzas, per customer over 1 year!

The 3-4% improvement over 1 year doesn’t mean, either, that Schwan’s will maintain annual revenue growth of 3-4%, especially if their customer base doesn’t grow.  In fact, according to their 2011 Annual Report, revenues remained at the ~$3 billion level as 2010. So what I see here, in the limited amount of information in this story, is that Schwan’s system lifted sales to make their fleet incrementally more efficient.

Average inflation for 2011 was reported to be 3.2%. (See "Table of Inflation Rates by Month and Year (1999-2012)")  I would be interested to know if Schwan’s own expenses grew at the inflation rate.  If so, Schwan’s merely kept up with inflation through 2011.

Depending on the cost of the system, the system may have made sense from a stand alone ROI perspective.  Call me skeptical, though, but this doesn’t feel like a sustainable game changing improvement at Schwan’s. I hope the story is different for Schwan's one year down the road from the original publication date of this article.

All that aside, I’m not sure this is a good example of avoiding the kinds of decision/thinking failures that Daniel Kahneman talked about because it seems to me the Schwan’s fleet is simply getting a little bit better information about how to implement an existing strategy, as opposed to Schwan’s avoiding the kinds of biases that make people pick the wrong strategy. Maybe that’s the real story here – Schwan’s let the siren song of advanced technology convince them to continue following a margin sensitive strategy to 1 (!) more decimal place.  They are solving the wrong problem with increasing precision. In fact, this story about Schwan’s isn’t really consistent with the Moneyball story of Billy Beane and the Oakland A’s.  In that example, Billy Beane and Paul Podesta challenged long held beliefs about the value of players’ capabilities, tested their own hypotheses, and bought the resources they needed to win at bargain rates.  They weren’t just squeezing more runs out of superstar players.  They found value where everyone else who esteemed themselves as experts said that it couldn’t be found.  The effect was actually, pardon the pun, game changing for the A’s.  They were no longer building a baseball team or playing baseball the way everyone else said that it had to be managed and played.  They became a uniquely good team versus being a team that tried only to improve within notions of conventional wisdom with declining marginal returns for the effort.  Did they use statistical analysis to do their job? Yes, but I think they really only needed the statistical analysis to indicate the presence of an inefficiency in the baseball marketplace, and then to find the resources they needed. They formulated and pursued a strategy around this idea of exploiting information inefficiencies.

What does this mean for us —those of us who desire to be better at making more valuable and creative decisions or helping others in that endeavor?  First, in the age of Big Data, I think we need to be careful about showcasing Big Data applications as examples of how decision analysis methodologies work.  The story about Schwan’s in this article is an application of data analytics and information technology that gleaned narrow improvements from statistical information, not necessarily good creative decision making.  We need to be careful about the distinction in what some people are calling applications of decision science and what we do with decision analysis and management.  I don’t doubt that the Opera solution is doing advanced analytics. I just doubt that Schwan’s engaged in good decision making. :/

What I see that valuable decision analysis provides is a mindset, a meta-system, to avoid the kind of system 1 (biased intuition) and system 2 failures (intellectual laziness) that Kahneman describes. A thorough application of decision analysis should consider multiple alternate strategies to achieve something more than incremental improvements.  With decision analysis we should seek disconfirming evidence and logic for biased assumptions.  Decision analysis of the kind I think we want to do avoids cognitive inefficiencies that arise from cognitive and motivational biases, information that has been aggregated at too gross of a level, and creative laziness. The questions we ought to help the consumers of our thinking answer are not just whether they need better information systems, but whether, for example, a better information system is the best application of resources to achieve game changing returns.  I think this kind of thinking leads to qualitatively different kinds of question.  I’m not saying that advanced data analysis can’t be powerful. We know that it can be.  But it might not always be the best solution.

Economists Are Overconfident. So Are You

HBR blogger, Justin Fox, provides a great explanation in "Economists Are Overconfident. So Are You" for why you need to report insights with graphs, not just numbers or even no numbers at all. Here's the key take-away for you:

They paid too much attention to the averages, and too little to the uncertainties inherent in them, thereby displaying too much confidence.

Small Business Decision Analysis: A View from the Trenches

Small Business Decision Analysis by Robert Brown