Introductory description

An example of stochastic control is AlphaGo, trained as an autonomous agent to play a game like chess. Its goal is to learn a policy to maximise winning chances by making optimal moves in different game states. Through iterative learning, AlphaGo refines its decision-making, demonstrating the application of stochastic control theory in reinforcement learning.

This module is available for students on a course where it is a listed option and as an Unusual Option to students who have completed the prerequisite modules

Prerequisites ST318 Probability Theory and ST333 Applied Stochastic Processes.

Module web page

Module aims

Outline syllabus

This is an indicative module outline only to give an indication of the sort of topics that may be covered. Actual sessions held may differ.

This module will cover the following areas

1. Markov Decision Processes (Markov chains, dynamic programming for controlled Markov chains and optimal stopping)
2. Reinforcement Learning (temporal difference, Q-learning method, actor-critic method, policy gradient)
3. Continuous-Time Deterministic Optimal Control (calculus of variations, dynamic programming and Hamilton-Jacobi equations)
4. Continuous-Time Stochastic Optimal Control (controlled diffusion processes, dynamic programming and Hamilton-Jacobi-Bellman equations)
5. Continuous-Time Reinforcement Learning (entropy-regularized exploratory formulation)

Learning outcomes

By the end of the module, students should be able to:

• Identify and to deal with stochastic control and optimal stopping problems.
• Solve simple Hamilton-Jacobi-Bellman equations both explicitly and numerically.
• Apply the above techniques to mathematical finance and reinforcement learning domains.
• Select and apply appropriate reinforcement learning techniques and its basic algorithms to solve problems.

Indicative reading list

Reading lists can be found in Talis

Specific reading list for the module

Subject specific skills

• Evaluate, select and apply appropriate mathematical and/or probabilist techniques.

• Demonstrate knowledge of and facility with formal probability concepts, both explicitly and by applying them to the solution of problems.

• Create structured and coherent arguments communicating them in written form. 

• Construct logical mathematical arguments with clear identification of assumptions and conclusions.

• Reason critically, carefully, and logically and derive (prove) mathematical results.

Transferable skills

• Problem solving: Use rational and logical reasoning to deduce appropriate and well-reasoned conclusions. Retain an open mind, optimistic of finding solutions, thinking laterally and creatively to look beyond the obvious. Know how to learn from failure.

• Self awareness: Reflect on learning, seeking feedback on and evaluating personal practices, strengths and opportunities for personal growth.

• Communication: Present arguments, knowledge and ideas, in a range of formats.

• Professionalism: Prepared to operate autonomously. Aware of how to be efficient and resilient. Manage priorities and time. Self-motivated, setting and achieving goals, prioritising tasks.