Business Process Management Foundation

My Breakdown: Correlation vs. Causality

When I look at data science, I see it as more than just gathering information; it’s about understanding the "why" behind the numbers. In this video, I walk through the process of taking data from various sources and building a model to see how different pieces are linked.

I define correlation as simply a relationship or a pattern. If I see a specific outcome in Dataset A and it repeats in Dataset B, I’ve found a link—but I haven't necessarily solved the problem.

The real prize is causality. I think of this as a strict "if-then" scenario. For me to claim causality, I have to prove that if $X$ happens, $Y$ will result, and just as importantly, if $X$ doesn't happen, $Y$ won't either. I emphasize that we have to test these theories relentlessly. I ask questions like: Does it always happen? Is our sample size big enough? Can we repeat this in different models?

I use the example of predicting how a political party member will vote. It’s easy to assume they’ll vote for their own party, but I’d need to look at historical data, conduct interviews, and see how constant that affiliation really is. Without that proof, the data is essentially valueless. My biggest warning is this: assumptions are dangerous. If you jump to conclusions without testing, you’re essentially marching off a cliff. My goal is to make sure that doesn't happen to you by validating findings over and over again.

Reflective Exercises

1. The "Hidden Factor" Challenge

Think about this: Data shows that ice cream sales and shark attacks both increase during the same months. Is this causality? Probably not. I want you to identify the "Variable Z"—the hidden factor—that is actually causing both to rise. How would you design a test to prove that ice cream isn't the cause?

2. Analyzing Your Own Assumptions

Pick a pattern from your own life, like "I’m more productive when I drink coffee." Write this as an "If $X$, then $Y$" statement. Now, look for the "Non-Event": Have you ever been productive without coffee? Have you ever had coffee and stayed unproductive? This helps you see if your belief is a true cause or just a habit-based correlation.

3. The Risk Assessment

Using the political party example from the video, imagine you are a campaign manager. If you assume all party members will vote for you (causality) and spend your entire budget on them, what happens if that relationship was actually just a loose correlation? Describe the "cliff" your campaign might walk off and how you could have used interviews or questionnaires to build a safety net.

In this session, I explore the concept of Power Laws and why our standard expectations of "normal" often fail us. Using the Fukushima reactor's design as a starting point, I discuss the danger of using models of uncertainty to produce a false sense of certainty. While the experts believed a tremor larger than 8.6 was a statistical impossibility, the 2011 earthquake reached 9.1.

We often ignore events that fall outside the "three standard deviations" of a typical bell curve—the 99.7%. However, not all data has a "mean" or average that tells the whole story. In many systems, infrequency does not equal minimal impact. This is the essence of a Power Law: a relationship where "freak events" (the long tail) are responsible for the most significant changes.

I also delve into the Pareto Principle, or the 80/20 Rule. This principle illustrates that in many systems, the majority of effects come from a small minority of causes. By understanding that large catastrophic events often occur for the same reasons small ones do, we can move away from a false sense of security and better respect the interdependencies of the world around us.

Reflective Exercises

• The "Black Swan" Moment: Reflect on a major turning point in your life or career. Was it the result of a slow, predictable trend, or did it happen because of a single, unforeseen "freak event"?

• The 80/20 Audit: Look at your daily habits or the tools you use. Do you find that a small fraction (20%) of your efforts or resources produces the vast majority (80%) of your results? How does that change your perspective on where to focus your energy?

• Beyond the Average: Think of a system you interact with (like traffic, the weather, or social media). When has "the average" completely failed to predict what actually happened? What was the impact of that outlier?

• Interdependency: Consider a "small" event you’ve witnessed that spiraled into something much larger. Looking back, what were the hidden connections that allowed that small change to trigger a massive outcome?