How to Ensure the Stability of a Model Using Jackknife Estimation
In many cases, ensuring the robustness of a model is critical for a good consistency and generalization of unseen data. Detecting influential individual data observations can be another crucial reason to avoid inaccurate results.
This process often involves assessing the variability of the model's output and identifying potential Bias, especially when dealing with small datasets. One powerful statistical tool to address these challenges is the Jackknife estimation method.
In this article, we'll deep-dive into the concept of Jackknife estimation, walk through a practical example, and explore step-by-step how it works.
What is Jackknife Estimation?
As Bootstrapping, Jackknique estimation is a resampling statistical technique ** to estimate bias and variance of an estimator. It works by leaving out one observation at a time from a dataset, calculating the estimator on the remaining data, and then using the resulting estimates to compute the overall estimate. To illustrate the usage of this technique, we will explain later a common practical example about churn predictio**n.
The Mathematics of Jackknife Estimation
Being _s={x_1, …, xn} the original sample, we want to estimate the parameter theta, which could be any statistic such as the mean of the sample, the churn rate or even the individual predicted probabilities. This estimator will be called the original estimator of theta.
First generate n samples removing the i-th element on each of them and computing the statistic with the remaining n-1 observations:
Then, aggregate the estimates calculating the mean of the Jackknife samples as:
The Jackknife bias of the estimated parameter is given by:
And the variance estimate as follows:
Finally, the Jackknife estimator of the original parameter can be calculated as:
The bias of this new estimator is 0. In practice, this is not exactly 0, since it actually is a first-order approximation of a Taylor series, but asymptotically will be always smaller than the bias of any given estimator [1]. This means that while the jackknife may not eliminate bias completely in small datasets, it still significantly reduces it compared to the original biased estimator.
Differences with Bootstrap
Bootstrapping is a well-known approach to estimate the distribution of an estimator using resampling. This is a useful method because avoid any assumptions about the inherent distribution of the original data conversly to other common statistical methodologies.
Both are non-parametric techniques based on resampling, useful for the estimation of the bias and variance of a dataset. However, Jackknife is a precedent to Bootstrap being described by Quenouille in 1949 and refined later by Tukey in the '50s.
Nevertheless, one of the main characteristics of the Bootstrap methodology which is sampling with replacement is not followed by Jackknife, that is based on resampling without replacement.
Another curious link between both methods, is that the Jackknife is a linear approximation of the Bootstrap approach. See reference [1] for more detail.
A Practical Example: Churn Prediction
Imagine you build a prediction model to detect churn based on customer data such as subscription duration and other features related with your product. Churn rate normally is pretty small compared with the rest of the population, so it is easy to get false positives or false negatives in your predictions. To avoid this, you would like assess the stability of the model and estimate the sensitivity of the churn predictions for each individual data point.
The plot above shows the relationship between two artificial features and churn probability. Despite is data simulated, the aim was to illustrate the complexity of churn detection over a realistic dataset. Sometimes, there is no specific pattern and the intrinsic variability of the distribution makes difficult to build a robust model to detect churn. We will review the labelled data points later.
To ensure Robustness, after training the prediction model with the entire sample, remove one customer at a time for the dataset. Then retrain the model on the remaining n-1 customers, and record the churn predictions for all customers using the retrained model. These are called Jackknife samples.
Now, use the Jackknife samples to estimate the bias and the variance of the model's predictions and understand why certain predictions deviate from expectations.
Detecting Underestimated or Overestimated Predictions
The Jackknife bias estimates the difference between the prediction calculated on the full dataset and the average of the Jackknife estimates.
Observation 77 is a churn predicted customer with a high negative bias. This indicates that the model is systematically overestimating for that specific observation. In this case the true churn probability is 0.65 and the estimated 0.70. In the other side, point 56 has an original churn probability of 0.31 and it's underestimated to 0.08.
Even the final accuracy metrics could be not highly modified by individual predictions, like in the example, this methodology help us the identify biased predictions, that in some cases could lead to targeting customers incorrectly and to wrong business decisions.
The reason under this is that the model might be too simple to capture the relationships in the data or that the model may have insufficient data to properly represent certain patterns.
Measuring Stability of a Model
The stability of the model can be measured by calculating the variance of predictions across Jackknife iterations. A stable model will show low variance in predictions for all customers and, in the opposite case, a high variance will indicate sensitivity to small changes in the data.
Predictions for customers 15, 77, 41 and 91 fluctuate significantly in the Jackknife samples compared with the remaining observations. As a result, they are probably inaccurate predictions and they are adding complexity to the model.
The presence of influential data points can lead to a unstable model that overreacts to noise. As solution, you can simplify the model by removing the influential observations.
Conclusions
Jackknife is a recommended technique when you want to assess variability and bias in an adjusted model and the sample size is small. For large sample sizes, Bootstrapping will be a preferable option.
These techniques help you to identify whether the model depends on certain observations and allow you to improve confidence in targeting individuals by ensuring the model is stable and reliable.
References
- [1] McIntosh, Avery I. The Jackknife Estimation Method (2016) arXiv: Methodology.
Thanks for reading!✨
Please, feel free to comment! We are all learning, so I would be happy to discuss more about any technical or data topic related!
