Introductory description

The primary aim of this module is to expand upon students' existing understanding of control systems, introducing them to sophisticated methodologies and approaches in nonlinear control and reinforcement learning. The module is designed to equip students with the theoretical foundations and practical skills necessary to analyse, design, and implement control systems that address the complexities and dynamic nature of real-world nonlinear engineering problems. The course aims to foster a comprehensive understanding of how these advanced methods can enhance performance in various sectors, including engineering disciplines such as aerospace, automotive, robotics, and energy systems, as well as AI and data-driven technologies. Through this module, students will be prepared to innovate and apply cutting-edge control solutions to a multitude of engineering challenges, reflecting the evolving demands of contemporary and future technologies.

Module aims

Outline syllabus

This is an indicative module outline only to give an indication of the sort of topics that may be covered. Actual sessions held may differ.

This is an indicative module outline only to give an indication of the sort of topics that may be covered. Actual sessions held may differ.

• Introduction to Model Predictive Control (MPC) Systems, its core principles and formulation
• Objective functions in MPC and designing cost functions for optimal control performance
• MPC without constraints, its theory and applications
• Control horizon and prediction horizon, their roles in MPC performance and computational complexity
• MPC with constraints, incorporating various constraints in MPC
• Overview of optimisation techniques used in MPC (linear programming, quadratic programming)
• Sequential Quadratic Programming (SQP) for solving nonlinear MPC problems
• Implementation challenges and solutions in MPC, scalability and computational aspects
• Review of key probability concepts, and introduction to Reinforcement Learning (RL)
• Exploration vs. exploitation dilemma, value iteration and policy iteration
• Introduction to core algorithms in RL such as Q-learning, State–action–reward–state–action (SARSA)
• Comparing model-free and model-based approaches
• Designing effective reward functions
• Application of RL algorithms to control tasks, differences and synergies between RL and traditional control methods
• Ensuring stability and robustness in RL control systems
• Combining nonlinear control techniques with reinforcement learning for advanced control solutions

Learning outcomes

By the end of the module, students should be able to:

• Understand, identify and describe the key characteristics and challenges of nonlinear systems and their implications for control system design and analysis [M1, M2, M6].
• Apply basic nonlinear control techniques to stabilise and control nonlinear systems, understanding the underlying principles and limitations of these strategies [M1, M2].
• Design and analyse Model Predictive Control (MPC) Systems, both with and without constraints, understanding the formulation, optimisation, and implementation aspects of MPC [M1, M2].
• Understand and explain the core principles of Reinforcement Learning (RL), including the concepts of agents, environments, states, actions, rewards, policies, and value functions [M1, M2].
• Design and implement fundamental Reinforcement Learning algorithms such as Q-learning and policy gradient methods, to solve decision-making problems [M1, M2].
• Evaluate the performance of various reinforcement learning algorithms, understanding how factors like exploration vs. exploitation and discounting affect outcomes [M1, M2].
• Apply advanced software tools such as the Matlab Model Predictive Control Toolbox and Reinforcement Learning Toolbox to solve real-world problems [M3, M12].

Indicative reading list

• Maciejowski, J.M., "Predictive Control with Constraints," Prentice Hall, 2002.
• Wang, L., "Model Predictive Control System Design and Implementation Using MATLAB," Springer, 2009.
• Rawlings, J.B., and Mayne, D.Q., "Model Predictive Control: Theory and Design," Nob Hill Publishing, 2009.
• Sutton, R.S., and Barto, A.G., "Reinforcement Learning: An Introduction," 2nd edition, MIT Press, 2018.
• Szepesvári, C., "Algorithms for Reinforcement Learning," Morgan and Claypool Publishers, 2010.
• Matlab Model Predictive Control Toolbox, User’s Guide, Mathworks
  https://uk.mathworks.com/help/mpc/
• Matlab Reinforcement Learning Toolbox, User’s Guide, Mathworks
  https://uk.mathworks.com/help/reinforcement-learning/
• Franklin, Powell, and Emami-Naeini, Feedback Control of Dynamic Systems (7th Edition), Pearson, 2014.
• Hahn and Valentine, Essential MATLAB for Engineers and Scientists, Academic Press, 2019.

Interdisciplinary

Between Control Engineering and Computer Science (Reinforcement learning)

Subject specific skills

Nonlinear system analysis
Model predictive control design and analysis
Formulation of optimisation problems and integration of constraints in MPC
Simulation and computational tools for model predictive control
Designing, implementing and tuning various Reinforcement Learning algorithms
Policy optimisation in Reinforcement Learning
Simulation and computational tools for reinforcement learning systems.

Transferable skills

Intelligent Control, Adaptive Control, Machine Learning, MATLAB Programming, Technical Report Preparation, Problem Solving, Critical Thinking