Introductory description

This module provides an introduction to the theory and implementation of neural networks and an understanding of the important computational neural network architecture and methodology. It aims to give students sufficient knowledge to enable employment or postgraduate study involving neural networks.

Module aims

Outline syllabus

This is an indicative module outline only to give an indication of the sort of topics that may be covered. Actual sessions held may differ.

• Introduction: What Is a Neural Network; AI and Neural Computing; Three Types of Learning Paradigms; Transfer Learning; Pre-Stage: Making Your Own Neural Network from Scratch: The Tools You Will Need; Demo: Creating a Neural Network for Live Image Classification

• Biological Neural Networks: Biological Neuron Structure and Function; Action Potential; Synaptic Transmission; A Brief History of Neural Nets & Deep Learning; Comparison between Biological and Artificial Neural Networks; Basic Principles of Artificial Neural Networks; Different Types of Neural Networks;

• Single-Layer Perceptron: McCulloch-Pitts Neuron; Definition of Perceptron; Weights, Bias, Node, Activation Functions; Perceptron Case Study: Digital Number Recognition; Mimic Logic Operations (AND, OR, NOT); XOR Linear Inseparability Problem; Perceptron Learning Rule; Demo: Implementing MP Neuron using MATLAB

• Multi-Layer Forward Propagation: Representation of Multi-Layer Neural Networks; Predicting Multi-Layer Neural Network Output; Common Nonlinear Activation Functions; How to Choose an Activation Function for Your Model; Forward Propagation for Deep Neural Networks; Vectorisation; Demo: Recognise Seven-Segment Numerals
  Using Perceptron

• Loss Function & Gradient Descent: Different Loss Functions (e.g., MAE, MSE, Cross Entropy Loss); 1D & 2D Forms of Gradient Descent Methods; Variations of Gradient Descent (e.g., Momentum, RMSProp, Adam); Saddle Points; Gradient Descent vs. Newton Method; Stochastic & Mini-Batch Gradient Descent;

• Backpropagation: Computation Graph; Matrix Calculus Revisited; Backpropagation in Logistic Regression; Backpropagation in Multi-Layered NNs; Vanishing Gradient Problems

• Overfitting & Regularization: Bias and Variance; What is Regularization; How does Regularization help in Reducing Overfitting; Different Regularization Techniques (e.g. Lasso & Ridge Regularization, Dropout, Data Augmentation, Early Stopping)

• Recurrent Neural Network: Sequence Model; One-hot Representation; Forward and Backward Propagations; Different Types of Recurrent Neural Networks; Elman and Jordan Networks; Bidirectional Recurrent Neural Network; Vanishing & Exploding Gradient Problems; Gated Recurrent Units (GRU); Long Short-Term Memory Networks (LSTM); Time Series Analysis (optional)

• Graph Neural Networks (GNNs): Basics of Graph Mining; Google PageRank; Pairwise Similarity Model (e.g., SimRank, Penetrating-Rank, RoleSim); Convolutional Neural Network; Graph Embedding; Node2vec (optional); GNN Applications in NLP (optional);

Learning outcomes

By the end of the module, students should be able to:

• Students completing the module should be able to demonstrate: an understanding of the principles of Neural Networks and a knowledge of their main areas of application;
• The ability to design, implement and analyse the behaviour of simple neural networks.

Indicative reading list

1. Michael Taylor. Make Your Own Neural Network: An In-Depth Visual Introduction for Beginners. Amazon Digital Services LLC - Kdp Print Us, 2017, ISBN: 1549869132, 9781549869136

2. Graupe Daniel. Principles of Artificial Neural Networks: Basic Designs to Deep Learning (4th Edition). World Scientific, 2019, ISBN: 9811201242, 9789811201240

3. James Loy. Neural Network Projects with Python: The ultimate guide to using Python to explore the true power of neural networks through six projects, Packt Publishing Ltd, 2019, ISBN: 1789133319, 9781789133318

Subject specific skills

• Understand the basic concepts of biological and artificial neural networks in computer science.
• Apprehend different types of artificial neural networks (e.g. MP neuron, multi-layer perceptron, CNN, RNN, GNN) and their real applications.
• Design and implement simple neural networks and deep learning algorithms.

Transferable skills

• Mathematical analysis of neural computing methods.
• Evaluation of neural network algorithms.
• Programming skills in MATLAB/Python.