A Decision Analyst's View of Electoral Surprise

I turned off the television last night at 8 PM. Since I had an analytics problem to work on, I didn't want my attention divided, and I knew that clinging to electoral results was more neurotic than helpful. My attention at the moment was not going to change the results. So, I rolled up my sleeves and got to work.

At 12:30 AM, I turned my television back on...

As I watched the polling results roll in and followed the reactions of establishment pundits and the broader hoi polloi (from both sides) in social media, all I could think was, "What is going on here?" Over and over. I mean, Nate Silver was still giving better than 2:1 odds of a Clinton victory just before I turned off the TV. Could the situation really have been that different than assessed? Could things really have changed that quickly? At 4 AM, I finally captured some thoughts that I think should serve as object lessons for all of us, and not just in politics, but in business, too.

1. Never, ever believe your own spin. Humans love narratives that give them comfort. Unfortunately, almost all narratives are constructed from selected evidence that fits a preferred narrative.
2. Always question where your biases are coming from. You are biased. Until you recognize it, you will frequently be rudely embarrassed. 
3. There is no meaningful position in certainty. All beliefs about future events should be treated with degrees of belief. 
4. Even events that happened in the past are open to interpretation. The real issue about the facts of events is not so much whether events have occurred in the past or whether they will occur in the future. The real issue is our epistemic distance from the events. We generally don't know as much as we think we do.
5. We condition our beliefs on the evidence at hand. Thinking that a Clinton victory was highly probable was not a bad position to take. It made sense given much of the evidence. BUT, Prob(Clinton win) > 50% does mean Prob(Clinton win) = 100%! (I'm actually getting tired of explaining this. I'm getting tired of seeing people make this mistake and the effects it has in real life on real people. Probabilities are degrees of belief, not statements of fact.) Always, always, always consider the disconfirming evidence. 
6. Trump never showed an insignificant chance of winning. His victory was always probable. What I see and hear coming from those expressing shocked disappointment about the Clinton loss is that they didn't really explore and consider the edge cases that would lead to a Trump victory. Explore the edge cases. Explore aggressively. Keep exploring. 
7. Informed accuracy trumps false precision (pun intended). Don't be embarrassed to draw your prediction intervals wide. It's more honest, more informative, and will allow you to do a better job preparing contingency plans. When #6 is performed honestly and aggressively, it should lead you to make your prediction intervals even wider. It's better to be humble and recognize how little you know versus being sure and then being rudely surprised.
8. The evolving probability of win curves for this election resemble the curves associated with predicting that a given hypothesis among several is true when there are unaccounted for characteristics at play. Suddenly, a seemingly most likely explanation crashes to be replaced by a previously less likely hypothesis as the unrecognized characteristic manifests itself. This is a long way to say people get caught up in false dichotomies (or n-chotomies) for the possible explanations for what really is the case. It is almost always the case that more explanations are available than the limited set we originally conceived.
9. If something really weird happens and somehow the posted results at 4 AM reverse by the time I wake up, all of the above still applies, maybe more so.

Although Nate Silver was leaning in the wrong direction for predicting the outcome, his odds were actually more realistic and informed than many other pollsters who were giving 19:1 odds or better for a Clinton win.