Moar Accuracies

You've probably heard the saying, "It's better to be mostly accurate than precisely wrong." But what does that mean exactly? Aren't accuracy and precision basically the same thing?

Accuracy relates to the likelihood that outcomes fall within a prediction band or measurement tolerance. A prediction/measurement that comprehends, say, 90% of actual outcomes is more accurate than a prediction/measurement that comprehends only 30%. For example, let's say you repeatedly estimate the number of marbles in several Mason jars mostly full of marbles. An estimate of "more than 75 marbles and less than 300 marbles" is probably going to be correct more often than "more than 100 marbles but less than 120 marbles." You might say that's cheating. After all, you can always make your ranges wide enough to comprehend any range of possibilities, and that is true. But the goal of accuracy is just to be more frequently right than not (within reasonable ranges), and wider ranges accomplish that goal. As I'll show you in just a bit, accuracy is very powerful by itself.

Precision relates to the width of the prediction/measurement band relative to the mean of the prediction/measurement. A precision band that varies around a mean by +/- 50% is less precise than one that varies by +/- 10%. When people think about a precise prediction/measurement, they usually think about one that is both accurate and precise. A target pattern usually helps make a distinction between the two concepts.

The canonical target pattern explanation of accuracy and precision.

The problem is that people jump past accuracy before that attempt to be precise, thinking that the two are synonymous. Unfortunately, unrecognized biases can make precise predictions extremely inaccurate, hence the proverbial saying. Jumping ahead of the all too important step of calibrating accuracy is where the "precisely wrong" comes in.

Good accuracy trucks many more miles in most cases than precision, especially when high quality, formal data is sparse. This is because the marginal cost of improving accuracy is usually much less than the marginal costs of improved precision, but the payoff for improved accuracy is usually much greater. To understand this point, take a look again at the target diagram above. The Accurate/Not Precise score is higher than the Not Accurate/Precise score. In practice, a lot of effort is required to create a measurement situation that effectively controls for the sources of noise and contingent factors that swamp efforts to be reasonably more precise. Higher precision usually comes at the cost of tighter control, heightened attention on fine detail, or advanced competence. There are some finer nuances even here in the technical usages of the terms, but these descriptions work well enough for now.

Be careful, though - being more accurate is not just a matter of going with your gut instinct and letting that be good enough. Our gut instinct is frequently the source of the biases that make our predictions look as if we were squiffy when we made them. We usually achieve improved accuracy through the deliberative process of accounting for the causes and sources of the variation (or range of outcome) we might observe in the events we're trying to measure or predict. The ability to do this reflects the depth of expert knowledge we possess about the system we're addressing, the degree of nuances we can bring to bear to explain the causes of variation, and a recognition of the sources of bias that may affect our predictions. In fact, achieving good accuracy usually begins by assessing that we may be biased at all (and we usually are) and why.

Once we've achieved reasonable accuracy about some measurement of concern, it might then make sense to improve our precision of the measurement if the payoff is worth the cost of intensified attention and control. In other words, we only need to improve our precision when it really matters.

[Image from FreeDigitalPhotos.net by Salvatore Vuono.]

Are Your Spreadsheets the Problem?

Mr. Patrick Burns at Burns Statistics (no, not that Mr. Burns) provides an excellent overview for the hidden dangers that lurk in your spreadsheets. Guess what. The problems aren't just programming errors and the potential for their harm, but are errors that are inherent to the spreadsheet software itself. That's right. Before your analysts even make an error, the errors are already built in. Do you know what's lurking in your spreadsheets? Well, do you?

Before you answer that question, ask yourself these:

1. What quality assurance procedures does our organization employ to ensure that our spreadsheets are free of errors of math, units conversion, and logic? 
2. What effort does our organization undertake to make sure that the decision makers and consumers of the spreadsheet analysis comprehend the assumptions, intermediate logic, and results in our spreadsheets? 
3. How do we ensure that spreadsheet templates (or repurposed spreadsheets or previously loved spreadsheets) are actually contextually coherent with the problem framing and subsequent decisions that the spreadsheets are intended to support? 

Each question actually addresses an hierarchically more important level of awareness and intention in our organizations. The first question addresses the simple rules of math and if they are satisfied. The second question addresses the level of agreement that the math/logic coordinates in a meaningful way and is capable of supporting valid and reasonable insights, inferences, or accurate predictions about the system or problem it describes and that everyone understands why. The last question, the most important question, IMHO, addresses whether our analyses point in the right direction of inquiry at all.

My suspicion is that errors of the first level run amok much more than people are willing to admit, but their prevalence is relatively easy to estimate given our knowledge about the rates at which programming errors occur, why they occur, and how they propagate geometrically through spreadsheets. Mr. Burns recommends that the programming language R is a better solution than spreadsheets and easier to adopt than might be currently imagined by your analysts. I agree. I happen to like R a lot, but I love Analytica as a modeling environment more. But the solution to our spreadsheet modeling problems isn't going to be completely resolved by our choice of software and programming mastery of it.

My greater suspicion is that errors of the second and third level are rarely addressed and pose the greatest level of risk to our organizations because we let spreadsheets (which are immediately accessible) drive our thinking instead of letting good thinking determine the structure and use of our spreadsheets. To rid ourselves of the addiction to spreadsheets and their inherent risks, we have to do the hard work first by starting with question 3 and then working our way down to 1. Otherwise, we're being careless at worst and precisely wrong at best.


(Originally published at LinkedIn.)

When A Picture is Worth √1000 Words

This morning @WSJ posted a link to the story about Microsoft’s announcement of its plans to lay off 18,000 employees. This picture (as captured on my iPhone)...

[click image to enlarge]

...accompanied the tweet, which is presumably available through their paywall link.

While I’m really sorry to hear about the Microsoft employees who will be losing their jobs, I am simply outraged at the miscommunication in the pictured graph. (This news appeared to me first on Twitter, and the seemingly typical response on Twitter is hyperbolic outrage.)

Here’s the problem as I see it: the graph communicates one-dimensional information with two-dimensional images. By doing so, it distorts the actual intensity of the information the reporters are supposed to be conveying in an unbiased manner. In fact, it makes the relationships discussed appear much less dramatic than it actually is.

For example, look at Microsoft’s (MSFT) revenue per employee compared to Apple’s (AAPL). WSJ reports MSFT is $786,400/person; APPL, $2,128,400. The former is 37% of the latter. But for some reason, WSJ communicates the intensity with an area, a two-dimensional measure, whereas intensity is one-dimensional. Our eyes are pulled to view the length of the side of the square as a proxy for the measurement being communicated. The sides of the squares are proportionally equal to √(786,400) and √(2,128,400); therefore, the sides of the squares visually communicate the ratio of the productivity of MSFT:AAPL as 61%. In other words, the chart visually overstates the relative productivity of MSFT's employees compared to that of AAPL's by a factor of 1.62.

If the numbers are confusing there, consider this simpler example. The speed of your car as measured by your speedometer is an intensity. It’s one dimensional. It tells you how many miles (or kilometers, if you’re from most anywhere else outside the US) you can cover in one hour if your car maintains a constant speed. Your speedometer aptly uses a needle to point to the current intensity as a single number. It does not use a square area to communicate your speed. If it did, 60 miles per hour would  look 1.41 times faster than 30 miles per hour instead of the actual 2 times faster that it really is. The reason for this is that the the sides of the squares used to display speed would have to be proportional to the square roots of the speed. The square roots of 60 and 30 are 7.75 and 5.48, respectively.

For your own personal edification, I have corrected the WSJ graph here:

[click image to enlarge]

Do you see, now, how much more dramatic the AAPL employees' productivity is over that of MSFT's?

This may not seem like a big deal to you at the moment, but consider how much quantitative information we communicate graphically. The reason is that, as the cliché goes, a picture is figuratively worth a thousand words. I firmly believe graphical displays of information are powerful methods of communication, and a large part of my professional practice revolves around accurately and succinctly communicating complex analysis in a manner that decision makers can easily consume and digest. But I’m also keenly aware of how analyst and reporters often miscommunicate important information via visual displays, either by design, inexperience, or by trying to be too clever. I see these transgressions all the time in the analyses I’m asked to audit.

The way we communicate information is not just a matter of style for business reporters. We often make prodigious decisions based on information. If information is communicated in a way that distorts the underlying relationships involved, we risk making serious misallocations of scarce resources. This affects every aspect of the nature of our wealth - money, time, and quality of life. The way we communicate information bears fiduciary responsibilities.

For discussion sake I ask,

1. How often have you seen, and maybe even been victimized by, graphical information that miscommunicates important underlying relationships and patterns?
2. How often have you possibly incorporated ineffective means of graphically communicating important information? (Pie charts, anyone?)

If you want to learn more about the best ways to communicate through the graphical display of quantitative information, I highly recommend these online resources as a starting point:

• The Work of Edward Tufte and Graphical Press
• Me, myself, and BI

The Value of Knowing What You Do Not Know

The value of_knowing_what_you_don't_know from Robert Brown

How Do You Know That? Funny You Should Ask.

During a recent market development planning exercise, my client recognized that his colleagues were making some rather dubious assumptions regarding the customers they were trying to address (i.e., acceptable price, adoption rate, lifecycle, market size, etc.), the costs of development, and costs of support. Although he frequently asked “How do you know that?”, he seemed to face irritation and mild belligerence in reaction from those he asked to justify their assumptions. So, together we devised a simple little routine to force the recognition that assumed facts might be shakier than previously thought.

After bringing the development team members together, we went around the room and asked for a list of statements that each believed to be true that must be true for the program to succeed. We wrote each down as a succinct, declarative statement. Then, after everyone had the opportunity to reflect on the statements, we converted each to a question simply by converting the periods to question marks.

Before Western explorers proved that the Earth is round, ships used to sail right off the assumed edges.

We then asked the team to supply a statement that answered each question in support of the original statement. Once this was completed, we then appended the dreaded question mark to each of these responses. We repeated this process until no declarative answers could be supplied in response to the questions. The cognitive dissonance among the team members became palpable as they all had to start facing the uncomfortable situation that what they once advocated as fact was largely unsupportable. Many open questions remained. More uncertainty reigned than was previously recognized. The remaining open questions then became the basis for uncertainties in our subsequent modeling efforts in which we examined value tradeoffs in decisions as a function of the quality of information we possessed. You probably won’t be surprised to learn that the team faced even more surprises as the implications of their tenuous assumptions came to light.

I am interested to know how frequently you find yourself participating in planning exercises at work in which key decisions are made on the basis of largely unsupported or untested assumptions. My belief is that such events happen much more often than we care to admit.

I would also be interested to know if the previously described routine works with your colleagues to force awareness of just how tenuous many preconceived notions really are. I outline the steps below for clarity.

1. Write down everything you believe to be true about the issue or subject at hand. 
2. Each statement should be a single declarative statement. 
3. Read each out loud, forcing ownership of the statement.
4. Convert each statement to a question by changing the period to a question mark.
5. Again, read each out loud as a question, opening the door to the tentative nature of the original statement.
6. Supply a statement that you believe to be true that answers each question.
7. Repeat the steps above until you reach a point with each line of statements-questions where you can no longer supply answers.

You might find that using a mind mapping tool such as MindNode or XMind are useful for documenting and displaying the assumptions and branching question/responses. The visual display may serve to help your team see connections among assumptions that were not previously recognized.

Let me know if you try this and how well it works.

Can Modeling a Business Work?

A friend on LinkedIn asks, “Can modeling a business work?” I respond:

For now, or at least until The Singularity occurs, the development of business ideas and plans is a uniquely human enterprise that springs from a combination of intuition, goals, and ambitions. That should not mean, however, that we cannot effectively supplement our intuition and planning with aids to management and decision making. While I think human intuition is a very powerful feature of our species, I’m also convinced it can be led astray or corrupted by biases very quickly, particularly amid the complexities that arise as plans turn into real life execution. This is not a modern realization. The origin of the principles of inventory management, civil engineering, and accounting date back to the antiquities. Think of the seagoing merchants of the Phoenicians and the public works building Babylonians and Egyptians. In fact, historians now believe that the actual founder of Arthur Andersen LLP was none other than the blind Venetian mathematician and priest, Luca Pacioli (ca. 1494). That's right - that musty odor that emanates from accounting books is due to their being more than 500 years old.

Luca Pacioli doodling circles out of sheer boredom after a day of accounting. I made up the part about his being blind.

Business modeling is a tool similar to accounting in that it aids our thinking in a world whose complexity seems often to exceed the grasp of our comprehension. I look at the value of modeling a business as a means to stress test both the business plan logic and the working assumptions that drive the business plan. In regard to the business plan logic, we're asking if the business has the potential ability to produce the value we think it can; and in regard to the working assumptions, we're testing how sensitively important metrics (i.e., payback time, break-even, required resources, shareholder value) of the business plan respond to conditions in the environment and controllable settings to which our business plan will be subjected.

Obtaining such insights from modeling a business, business leaders can modify business plans by changing policies about pricing, products/services offered, costs targeted for reduction or elimination, and contingency or risk mitigation plans that can be adopted, etc. 

However, I recommend awareness of at least three caveats with regard to business modeling:

1. Think of such models as "what-ifs" more so than precise forecasts. Use the "what if" mindset to make a business plan more robust against the things outside your direct control versus using it to justify a belief in guaranteed success. The latter is almost a sure fire approach to failure. 
2. Always compare more than one plan with a model to minimize opportunity costs. Often times, the best business plans derive from hybrids of two models that show how value can be created and retained for at least two different reasons. 
3. Avoid overly complex models as much as, maybe more so than, overly simplistic models. Building a requisite model from an influence diagram first is usually the best way to achieve this happy medium before writing the first formula in a spreadsheet or simulation tool. Richer, more complex models that correspond to the real world with the highest degree of precision are usually not useful for a number of reasons:
• they can be costly to build
• the value frontier of the insights derived decline relative to the cost to achieve them as the degree of complexity increases
• they are difficult to maintain and refactor for other purposes
• they are often used to justify delaying commitment to a decision
• few people will achieve a shared understanding that is useful for collaborating and execution

A requisite model, on the other hand, should deliver clarity and permit making new and interesting testable predictions or reveal insights about, say, uncertainties, that could be made to work in your favor. Admittedly, though, it takes a lot of practice to achieve this third recommendation, but it should be used as a guiding principle.

Double, double toil and trouble; Fire burn, and caldron bubble

This was a great article in The Wall Street Journal today.

For me, the key take away point can be summed up in this quote from Prof. Goetzmann: "Once people buy in, they start to discount evidence that challenges them..." I relate this not only to investing decisions in the market, but also to making organizational decisions--investments in capital projects, new strategies, the next corporate buzz. We've all seen or been apart of the exuberant irrationality that leads organizations into malinvestments.

Let's consider the complementary action--saying "no." Against the tendency toward the irrational "yes, Yes, YES!", learning to say "no" is a very important skill to master. It's probably one of the hardest skills to master when people request something from us that makes us feel important and liked.

I think, however, we always need to be aware that many of our initial reactions are often driven by biases. Reactively saying "no," once we've learned to say it and it becomes easy to do, can emerge from the same biases that urge us unreservedly to say "yes." Both incur their costs: missed opportunity, waste, and rework.

The skill more important to learn than saying "no" is acquiring the skill to consider disconfirming evidence, especially when that evidence challenges our dearest assumptions about what is going to make us rich. Let's not be so quick to say "yes" or smug when we say "no." Rather, let's learn the practice of asking,

• "what information might disabuse me of my favorite assumptions?"
• "what biases are preventing me from seeing clearly?"

Failing to learn these, we all too often find ourselves concocting a witches' brew.