Simulated Data, Real Learnings : Power Analysis
INTRODUCTION
Simulation is a powerful tool in the Data Science tool box. After reading this article, you'll have a good understanding of how simulation can be used to estimate the power of a designed experiment. This is the second part of a multi-part series that discusses how simulation can be useful in data science and machine learning.
Here are the contents that we'll cover:
- Overview of power analysis
- How to calculate power using simulation – example based approach
In this article, I will just give a quick definition of data simulation:
Data simulation is the creation of fictitious data that mimics the properties of the real-world.
In part 1 of this series, I discuss the definition of data simulation much more extensively – you can check it out at the link below:
OVERVIEW OF POWER ANALYSIS
Experimentation is the gold standard for learning about relationships in the world around us. There are many considerations to take into account when planning an experiment. Even though experimentation is the gold standard, a poorly planned experiment can give useless or misleading results. Power analysis is a crucial component of planning good experiments.
Before we get into the details, let's answer this value oriented question: Question : Why is estimating power before running an experiment important? Answer: Because without understanding an experiment's power, we could waste our time and resources running an experiment that cannot detect meaningful results.
In statistical lingo, the definition of power is the probability that the null hypothesis will be correctly rejected. While less precise (and specific to experimental design), I like to define the power of experiments as – the probability that we will pick up on a relationship, given one exists.
Power in experimentation is the probability that we will pick up on a relationship, given one exists
Power is calculated by estimating two competing forces – (1) signal and (2) noise. The relative size of these two variables determines how well we will be able to pick up trends.
When calculating power, we create two distributions – one distribution has a mean of zero (being interpreted as our experimental variable has no relationship with the response variable) and the other distribution has a non-zero mean (interpreted as our variable has a positive relationship with the response). Note that the first distribution corresponds to the null hypothesis and the second distribution corresponds to the alternative hypothesis. Loosely speaking, the power is inversely related to the level of overlap between these two distributions – i.e. more overlap = less power, less overlap = more power.
I pulled the image below from further down in the article. I'll give a quick talk through of the image and then move onto what this article is actually about – the simulation! If you still have a lot of questions about power analysis after this section, don't worry, I'm not the only one on the internet that has written about it!
The green distribution is our null distribution; representing the probability distribution of estimated relationship values we could see if there is no relationship between our experimental and response variables. The blue is an alternative distribution, representing possible relationship estimates if the relationships between experimental and response variables is a positive number. The red line is the cutoff for the top 5% of the null (green) distribution. I'm going to skip a lot of details here and just say that the area in the blue distribution, to the right of the red line is the power. If this doesn't make sense, remember that google is your friend
