How do neural networks learn?

Neural networks learn by iteratively adjusting their parameters (weights and biases) to minimize the difference between their predictions and the actual target values. This process involves forward propagation, loss calculation, backward propagation, and parameter updates using optimization techniques. Here's a step-by-step explanation of how neural networks learn:

---

1. Initialization

• Weights and biases are initialized, usually with small random values. Proper initialization methods (e.g., Xavier or He initialization) are crucial to ensure good starting conditions.

---

2. Forward Propagation

• Inputs are passed through the network, layer by layer.
• Each layer computes:
  1. A weighted sum of the inputs: $$z = W \cdot x + b$$ where $W$ are the weights, $b$ is the bias, and $x$ are the inputs.
  2. An activation function is applied to introduce non-linearity: $$a = \text{activation}(z)$$
• The process continues until the output layer produces predictions ($\hat{y}$).

---

3. Loss Calculation

• The network's performance is measured using a loss function, which quantifies the difference between predicted values ($\hat{y}$) and true values ($y$).
  • Example loss functions:
    • Mean Squared Error (MSE) for regression.
    • Cross-Entropy Loss for classification.

---

4. Backward Propagation (Backpropagation)

• The network calculates how much each parameter (weights and biases) contributed to the error by computing the gradient of the loss function with respect to each parameter.
  • Gradients are computed using the chain rule of calculus: $$\frac{\partial L}{\partial W} = \frac{\partial L}{\partial a} \cdot \frac{\partial a}{\partial z} \cdot \frac{\partial z}{\partial W}$$
• Gradients flow backward through the network, layer by layer, starting from the output layer.

---

5. Parameter Updates

• Parameters are updated to reduce the loss using an optimizer, such as Gradient Descent or its variants (e.g., SGD, Adam, RMSProp).
  • Update rule: $$W = W - \eta \cdot \frac{\partial L}{\partial W}$$ $$b = b - \eta \cdot \frac{\partial L}{\partial b}$$ where $\eta$ is the learning rate, controlling the size of the updates.

---

6. Iteration (Epochs and Batches)

• The network is trained over multiple iterations, known as epochs, where the entire dataset is passed through the network.
• For large datasets:
  • Mini-batches are used to compute gradients and update parameters more efficiently.
  • This is called Mini-Batch Gradient Descent.

---

7. Validation

• After training on the training set, the model is evaluated on a separate validation set to monitor its performance and prevent overfitting.
• Techniques like early stopping or regularization can be used to improve generalization.

---

8. Testing

• After training and validation, the network's performance is assessed on a test set, a dataset unseen during training and validation.

---

Key Concepts in Learning:

1. Gradient Descent: The optimization algorithm used to minimize the loss function.
2. Learning Rate ($\eta$): Controls the size of parameter updates; too high can cause divergence, too low can slow learning.
3. Loss Function: Quantifies error and guides the learning process.
4. Backpropagation: Efficiently computes gradients for deep networks.

---

Example Flow:

1. Input image of a cat is fed into the network.
2. Forward pass predicts probabilities for different classes (e.g., cat: 0.7, dog: 0.2, bird: 0.1).
3. Loss function compares the prediction (cat: 0.7) with the actual label (cat: 1.0).
4. Backpropagation computes gradients.
5. Weights are updated to reduce the error on future predictions.

By iterating through these steps, the neural network improves its ability to make accurate predictions.