Classification by Backpropagation in Data Mining
What is Backpropagation?
Backpropagation is a learning algorithm used in neural networks. It works
by sending the error from the output layer back to the input layer and
adjusting the network to improve accuracy.
In simple terms, it:
- Compares predicted output with actual output
- Calculates the error
- Sends the error backward
- Updates the weights to reduce future errors
This method is widely used in data mining tasks like:
- Character recognition
- Signature verification
- Image classification
What is a Neural Network?
A neural network is a computing system inspired by the human brain.
- It consists of artificial neurons
- These neurons are connected like brain cells
- Each neuron receives input, processes it, and passes output
Just like the human brain:
- Neurons communicate through connections
- Each connection has a strength (called weight)
Neural networks are powerful because they can learn complex patterns
from data.
Backpropagation in Neural Networks
Backpropagation is mainly used to train neural networks.
It helps to:
Calculate the loss (error)
Find how much each weight contributes to the error
Adjust weights using methods like:
- Gradient Descent
- Stochastic Gradient Descent
It uses the chain rule to compute gradients layer by layer, starting
from the output layer and moving backward.
Key Features of Backpropagation
- Works with multi-layer neural networks
- Uses gradient-based optimization
- Requires differentiable activation functions
- Efficient in computing errors and updating weights
Stages in Backpropagation Training
There are three main steps:
1. Feed Forward
Input data is passed through the network
Output is generated
2. Error Calculation
Compare predicted output with actual output
Calculate the difference (error)
3. Backward Propagation & Weight Update
Error is sent backward
Weights and biases are updated to reduce error
How Backpropagation Works
- Input data is given to the network
- The network produces an output
- Output is compared with the actual result
- Error is calculated
- Error is propagated backward
- Weights are adjusted
- Process repeats until the error is minimized
Backpropagation Algorithm (History)
First introduced in 1960
Became popular in 1989 through a paper by:
- Rumelhart
- Hinton
- Williams
Neural Network Structure Example
A simple neural network may have:
- Input layer (e.g., 4 neurons)
- Hidden layers (e.g., 4 neurons)
- Output layer (e.g., 1 neuron)
Important Terms
- Activation (a): Output of a neuron after applying activation function
- Weighted input (z): Sum of inputs multiplied by weights
- Weights (W): Strength of connections
- Bias (b): Extra value added to improve learning
Formula idea:
Each layer computes:
Weighted sum → Apply activation function → Output
Matrix Representation (Simplified)
- Inputs are represented as vectors
- Weights are represented as matrices
- Bias is also a vector
Example:
- Weight matrix size = (number of neurons in next layer × number of neurons in current
- layer)
- Input vector size = (number of input neurons × 1)
- This helps perform calculations efficiently.
Need for Backpropagation
Backpropagation is important because:
- It trains neural networks effectively
- Easy to implement
- Works for many applications
- Does not require prior knowledge of the model
Types of Backpropagation
1. Static Backpropagation
Works with fixed input and output
Used for classification problems
Example: Optical Character Recognition (OCR)
2. Recurrent Backpropagation
Used in recurrent networks
Output depends on previous states
Learning continues until stable output is reached
Not instant like static backpropagation
Conclusion
Backpropagation is the core algorithm that makes neural networks
learn. It helps in improving prediction accuracy by continuously
adjusting weights based on errors. Because of its efficiency, it is
widely used in modern data mining and machine learning
applications.
