What is an Outlier in Data Mining?
In data analysis, we often come across unusual data values called
outliers. An outlier is a data point that is very different from the rest
of the data in a dataset. It lies far away from the normal pattern or
expected range of values.
Outliers can occur due to measurement errors, data entry mistakes, or
natural variations in the data. During data analysis, it is important to
identify these values because they may affect the accuracy of the results.
In some cases, outliers are removed, while in other cases they are
carefully analyzed because they may provide useful insights.
The concept of outliers was first formally defined by Frank E. Grubbs in
1969.
Difference Between Outliers and Noise
- Noise refers to random errors or unwanted variations in measured data. It usually occurs due to problems in measurement, data collection, or transmission.
- Outliers are extreme data points that significantly differ from the rest of the dataset.
Before detecting outliers, it is usually recommended to remove noise from
the dataset, because noise can make outlier detection more
difficult.
Types of Outliers
Outliers in data mining are generally classified into three types:
- Global (Point) Outliers
- Collective Outliers
- Contextual (Conditional) Outliers
1. Global Outliers (Point Outliers)
Global outliers are the simplest type of outliers.
A global outlier occurs when a single data point is very different from
all other data points in the dataset.
Most outlier detection methods in data mining focus on identifying this
type of outlier.
Example:
If the average exam score of students is between 60 and 80, but one
student scores 10, that value can be considered a global outlier.
2. Collective Outliers
A collective outlier occurs when a group of data points together behaves
abnormally compared to the rest of the dataset.
In this case, individual data points may appear normal, but when
considered as a group, they show unusual behavior.
Example:
In a network intrusion detection system, sending a few data packets from
one computer may be normal. However, if many computers send a large number
of packets at the same time, it may indicate a Denial-of-Service (DoS)
attack. The group of packets together becomes a collective outlier.
3. Contextual Outliers (Conditional Outliers)
A contextual outlier occurs when a data point is considered unusual only
within a specific context or condition.
These outliers depend on two types of attributes:
Contextual attributes – define the context (e.g., time, location)
Behavioral attributes – define the behavior of the data
Example:
A temperature of 45°C may be normal during summer, but it would be
unusual during the rainy season or winter. Therefore, the same value can
be normal or an outlier depending on the context.
Outlier Analysis
The process of identifying and studying unusual data points in a dataset
is called Outlier Analysis or Outlier Mining. It is an important task in
data mining because rare events often provide valuable information.
Although outliers are sometimes removed from datasets, they are very
useful in many real-world
applications.
Applications of Outlier Detection
Outlier detection is widely used in several fields, such as:
- Fraud detection in banking, credit cards, and insurance
- Telecommunication fraud detection
- Medical diagnosis and treatment analysis
- Market analysis to understand unusual customer behavior
- Network intrusion detection systems
- Financial data monitoring
For example, in medical analysis, unusual patient responses to a
treatment can be identified through outlier analysis.
