CS416-15 Optimisation Methods

Introductory description

At their core, many real-world problems involve optimisation problems that are complex and challenging, and they require principled mathematical and algorithmic solutions. Companies such as Google and Facebook use powerful algorithmic primitives such as gradient descent to solve a host of challenging problems. These primitives in turn are based on a beautiful mathematical theory developed in the context of convex and discrete optimisation, especially in the era of modern machine learning.

Module aims

Outline syllabus

This is an indicative module outline only to give an indication of the sort of topics that may be covered. Actual sessions held may differ.

Some of the following topics will be discussed. The selection of topics may vary from year to year.

Convex optimisation methods and their applications in machine learning.
Gradient descent, Stochastic Gradient Descent.
Newton's methods, adaptive methods.
Constrained optimization, KKT condition.
Projected gradient descent.
Typical applications in machine learning, e.g., least squares and linear regression, over-parameterization, min-max optimization, edge of stability.

Learning outcomes

By the end of the module, students should be able to:

• Learn mathematical tools from convex optimisation, discrete and combinatorial, and linear algebra.
• Learn optimisation methods that are widely used in applications, especially in machine learning
• Understand the mathematical theory behind these optimisation methods.
• Implement optimisation methods and apply them to real-world problems.

Indicative reading list

Please see Talis Aspire link for most up to date list.

View reading list on Talis Aspire

Subject specific skills

1. Learn basic optimisation methods which are ubiquitous in applications in machine learning, data analysis and scientific computing.
2. Strengthen the mathematical background in analysis and algebra.
3. Learn how to analyse optimisation methods with mathematical rigor.
4. Get programming experience by developing these methods in software and see different scenarios of application

Transferable skills

1. Develop analytical thinking by working with mathematical concepts.
2. Develop research & problem solving skills, particularly for mathematical problems. Students are exposed to some paradigms of optimisation, and later they develop their own approach to solve problems.
3. Communication skills - Learn how to express themselves with mathematical precision.