What is Fully Connected Neural Network?

Fully Connected Neural Network (FCNN)

A Fully Connected Neural Network (FCNN), also known as a Dense Network, is a type of artificial neural network where every neuron in a layer is connected to every neuron in the subsequent layer. These networks are composed of one or more layers of neurons, where each neuron receives input from all neurons of the previous layer, processes the input, and passes the result to the next layer.

---

Key Characteristics of a Fully Connected Neural Network

1. Layers:

   • Input Layer: Receives the input features (e.g., pixels for image data).
   • Hidden Layers: One or more layers where neurons process the input data using weighted connections.
   • Output Layer: Produces the final prediction (e.g., class probabilities for classification tasks).

2. Connections:

   • Fully Connected: Every neuron in a layer is connected to every neuron in the next layer.
   • Each connection has a weight that is learned during training, indicating the strength of the connection between two neurons.

3. Activation Functions:

   • Neurons in each layer apply an activation function (e.g., ReLU, Sigmoid, Tanh) to their weighted inputs to introduce non-linearity.

4. Weights and Biases:

   • Weights: Parameters that control the strength of connections between neurons.
   • Biases: Additional parameters added to the weighted inputs to shift the activation function.

---

How Fully Connected Neural Networks Work

1. Forward Propagation:

   • Input data is passed through the network layer by layer.
   • For each neuron, the weighted sum of its inputs is computed, and the activation function is applied.
   • This process continues through the hidden layers until the output layer is reached.

2. Backpropagation and Training:

   • Loss Function: The difference between the predicted output and the actual target (e.g., mean squared error for regression, cross-entropy for classification).
   • Backpropagation: A process where the error is propagated back through the network to adjust the weights and biases using gradient descent or other optimization algorithms.

---

Mathematics Behind Fully Connected Layers

For a given layer in a fully connected neural network:

• Let $\mathbf{x}$ be the input vector to the layer.
• $\mathbf{W}$ be the weight matrix, and $\mathbf{b}$ the bias vector.
• The output of the layer before applying the activation function is: $$\mathbf{z} = \mathbf{W} \mathbf{x} + \mathbf{b}$$
• After applying the activation function $f(\cdot)$, the output becomes: $$\mathbf{a} = f(\mathbf{z})$$ where $\mathbf{a}$ is the output of the layer.

---

Advantages of Fully Connected Neural Networks

1. Expressive Power:
   • FCNNs can approximate any continuous function given sufficient neurons and layers, thanks to the Universal Approximation Theorem.
2. Simplicity:
   • The structure of FCNNs is straightforward and easy to implement.
3. Flexibility:
   • FCNNs are versatile and can be applied to various tasks, such as regression, classification, and even time series forecasting.

---

Disadvantages of Fully Connected Neural Networks

1. Computational Cost:

   • Due to the dense connectivity, the number of parameters grows quickly as the number of neurons and layers increases, leading to high memory and computational requirements.

2. Overfitting:

   • With a large number of parameters, FCNNs are prone to overfitting, especially with limited training data. Regularization techniques like dropout, weight decay, or early stopping are needed.

3. Inefficiency in Handling Spatial Data:

   • For tasks like image or video processing, FCNNs are less efficient than specialized architectures like Convolutional Neural Networks (CNNs), which take advantage of spatial hierarchies.

---

Example of a Fully Connected Neural Network for Classification

1. Input: A dataset with 784 features (e.g., $28 \times 28$ pixel image flattened into a 1D vector).

2. Network Architecture:

   • Input Layer: 784 neurons (one for each pixel).
   • Hidden Layer 1: 128 neurons, ReLU activation.
   • Hidden Layer 2: 64 neurons, ReLU activation.
   • Output Layer: 10 neurons (for 10 classes in classification), softmax activation for probability distribution.

3. Forward Pass:

   • Input data is passed through each layer, and activations are computed based on the weights and biases.

4. Output: The network outputs a probability distribution over the 10 classes, and the class with the highest probability is selected as the predicted label.

---

Applications of Fully Connected Neural Networks

• Image Classification (when combined with CNNs).
• Time Series Forecasting.
• Speech Recognition.
• Recommendation Systems.
• Natural Language Processing (NLP) (e.g., fully connected layers in a sequence model).

---

Summary

A Fully Connected Neural Network is a simple, versatile type of neural network where each neuron is connected to every other neuron in adjacent layers. They are used for a wide range of applications but can become computationally expensive with larger datasets. While FCNNs are powerful, they are often combined with specialized networks like Convolutional Neural Networks (CNNs) for tasks like image processing.