How to identify outliers in data with Python
Nassim Taleb writes how "tail" events define a large part of the success (or failure) of a phenomenon in the world.
Everybody knows that you need more prevention than treatment, but few reward acts of prevention. N. Taleb – The Black Swan
A tail event is a rare event, the probability of which is on the tail of the distribution, on the left or right.
According to Taleb, we live our lives focusing primarily on the most plausible events, those that are most likely to happen. By doing this, we are not preparing ourselves to deal with the rare events that might happen.
When rare events happen (especially the negative ones), they take us by surprise and our usual actions that we typically take have no effect.
Just think of our behavior when a rare event occurs, such as the bankruptcy of the FTX cryptocurrency exchange, or a powerful earthquake that disrupts the territory. For those directly involved, the typical reaction is panic.
Anomalies are present everywhere, and when we draw a distribution and its probability function we are actually obtaining useful information to protect ourselves or to implement strategies for these tail events, should they occur.
It is therefore necessary to inform ourselves on how to identify these anomalies, and above all to be ready to act in cases where they are observed.
In this article, we will focus on the methods and techniques used to identify outliers (the mentioned anomalies) in data. In particular, we will explore data visualization techniques and the use of descriptive statistics and statistical testing.
The definition of outlier
An outlier is a value that deviates significantly from the other values in the dataset. This deviation can be numerical or even categorical.
For example, a numeric outlier is when we have one value that is much larger or much smaller than most other values within the dataset.
A categorical outlier, on the other hand, occurs when we have labels known as "other" or "unknown" that represent a much higher proportion of the other labels within the dataset.
Outliers can be caused by measurement errors, input errors, transcription errors or simply by data that does not follow the normal trend of the dataset.
In some cases, outliers can be indicative of broader problems in the dataset or the process that produced the data and can offer important insights to the people who developed the data collection process.
Techniques to help us identify outliers in a dataset
There are several techniques that we can use to identify outliers in our data. These are the ones we will touch upon in this article
- data visualization: which allows you to identify anomalies by looking at the distribution of data by making use of graphs useful for this purpose
- use of descriptive statistics, such as the interquartile range
- use of z-scores
- use of clustering techniques: which allows to identify groups of similar data and to identify any "isolated" or "unclassifiable" data
each of these methods is valid for identifying outliers, and should be chosen based on our data. Let's see them one by one.
Data visualization
One of the most common techniques for finding anomalies is through exploratory data analysis and particularly with data visualization.
Using Python, you can use libraries like Matplotlib or Seaborn to visualize the data in such a way that you can easily spot any anomalies.
For example, you can create a histogram or boxplot to visualize the distribution of your data and spot any values that deviate significantly from the mean.
The anatomy of the boxplot can be understood from this Kaggle post.
https://www.kaggle.com/discussions/general/219871
If you want to read more about how to perform exploratory data analysis (EDA), read this article
