Are Your Business Intelligence Systems Complete?

I've been concerned lately about the state of business intelligence (BI) systems I've seen implemented in the wilds of business.

Currently, BI systems are essentially accounting and reporting systems, oftentimes accompanied with very sophisticated data visualization. That is, they show current and past states of collected information. From this information, trends, correlations, and anomalies can be determined and observed, but they don’t say what they mean to stakeholders. Do the trends and anomalies imply that something important is happening and should have some attention applied to them, or are they simply passing stochastic fluctuations? If they do need attention, what should the action be:

• Correction/mitigation - to bring the system back into control or avoid undesirable outcomes 
• Exploitation - to take advantage of new opportunities to gain competitive advantage?

How do you decide?

Furthermore, there is, I suspect, another risk that lurks within the use of BI systems related to cognitive biases. There is quite a bit of research that shows that there is an optimum to the amount of information executives need to make well-formed decisions. If there is too little, executives are exposed to unfortunate surprises due to lack of critical information. Action based on recent or colorful available information is called, unsurprisingly, recency and availability bias. It’s easy to help people see why this bias is so pernicious.  Just remind them of events they didn’t anticipate, or show them how being too careful in consideration of extreme events or too prejudicial in consideration of emotionally vivid information is likely costing them too much money. The other extreme is having too much information, and it’s difficult to convince people why this is a problem. Here the executive has so much information that they place higher and higher confidence on the validity of their reasoning, while the research shows that their performance doesn’t improve in a commensurate way. It’s often difficult to explain why this is the case because people typically don’t tie the success or failure to their initiatives to the level of information they were using when they made their decisions, and the time lag between when decision are made and when results are measured often extends beyond the memory of participants as well as the presence of the participants. This latter cognitive failure is related to overconfidence bias.  (For a longer list of cognitive biases, go here.)

The effects of the interactions of BI systems within the presence of the various cognitive biases is

• A failure to create a shared understanding of how goals and objectives work together to create value, leading to frustrated ambiguity about the real reasons for taking corrective or exploitative action
• Only the "tangible" costs and benefits are estimated, leaving the fuller range of "intangible" costs and benefits unquantified, treated only in qualitative manner, or disregarded altogether
• The full range of business uncertainty and risk is often overlooked or not understood, leading first to endless discussions about assumptions and forecasts, and finally to unanticipated outcomes and continual rework or unrealized value
• Decision prioritization is based on politics rather than a quantified value to the business 
• Trade-offs between decision timing, optionality, and value are ignored

So, when you do decide take some corrective or exploitative action, how do you know that the actions you take are the most valuable ones and not simply satisficing decisions or, worse, inconsistent and incoherent?

What is missing is an intelligent decision management system that guides decision makers consistently through the thorny issues of what to do in the presence of trends and anomalies reported by their BI systems.  An intelligent decision management system will do at least four things reporting systems coupled with unguided thinking cannot do:

1. Synthesize seemingly unrelated information 
2. Abstract information into requisite models that includes the characterizations of appropriate uncertainties controlled for bias
3. Compare/contrast possible strategies to address trends and anomalies against the stakeholders’ subjective preferences 
4. Interpret results of the analysis into competitive responses 

It does not throw out the BI systems. It complements them. It provides an executive monitoring and feedback loop into the current state and trajectory of an enterprise.