Introductory description

This module is available for students on a course where it is a listed option and as an Unusual Option to students who have the required backgrounds as covered by the pre-requisite module.

The ideas presented in this module have a vast range of applications, for example routing algorithms in telecommunications (queues), assessment of apparent spatial order in astronomical data (stochastic geometry), description of outbreaks of disease (epidemics). We will only be able to introduce each area - indeed each area could easily be the subject of a course on its own! But the introduction will provide you with a good base to follow up where and when required. (For example: a MORSE graduate found that their firm was asking them to address problems in queuing theory, for which ST333 provided the basis.) We will discuss these and other applications and show how the ideas of stochastic process theory help in formulating and solving relevant questions.

Pre-requisites: ST227 Stochastic Processes.

Anti-requisites: Please check the availability tab and the course handbook.

Module web page

Module aims

Outline syllabus

This is an indicative module outline only to give an indication of the sort of topics that may be covered. Actual sessions held may differ.

1. Continuous time Markov Chains.
   Terms used in the analysis of continuous-time Markov chains: Markov property, transition probability function, standing assumptions, Chapman-Kolmogorov equations, Q-matrix, Kolmogorov forward and backward differential equations, equilibrium distribution. The simplest case: finite state-space Markov chains. The "switcher" example. Exact transition densities for processes on a small number of states. The strong Markov property.

2. Linear Birth-Death processes.
   Poisson (counting) process: construction, ideas of independent increments, superposition, counts and thinning. Pure birth process, linear birth-death process, birth-death-immigration process: construction using "microscopic model", derivation of extinction and equilibrium probabilities. Generalized birth-death processes.

3. Queuing theory.
   The Markov single-server (M/M/1) queue. The concept of detailed balance. Measures of effectiveness. Multiserver (M/M/cl/c2) queues. Erlang's formula. Queues with general service-time distribution (M/G/l) and their embedded Markov chains. Little's formula, Pollaczek-Khintchine formula.

4. Other Markov properties.
   Stopping times. Strong Markov property. Holding theorem.

5. Epidemics.
   Deterministic Epidemic model. Stochastic model without removals. Stochastic model with removals.

Learning outcomes

By the end of the module, students should be able to:

• formulate continuous-time Markov chain models for applied problems.
• use basic theory to gain quick answers to important questions (for example, what is the equilibrium distribution for a specific reversible Markov chain?).
• solve for the transition probabilities for Markov chains on a finite state space.
• understand how to use Markov chains in the modelling and analysis of queues and epidemics.

Indicative reading list

Specific reading list for the module

Subject specific skills

• Demonstrate facility with rigorous probabilistic methods.

• Evaluate, select and apply appropriate mathematical and/or probabilist techniques.

• Demonstrate knowledge of and facility with formal probability concepts, both explicitly and by applying them to the solution of mathematical problems.

• Create structured and coherent arguments communicating them in written form. 

• Construct logical arguments with clear identification of assumptions and conclusions.

• Reason critically, carefully, and logically.

Transferable skills

• Problem solving: Use rational and logical reasoning to deduce appropriate and well-reasoned conclusions. Retain an open mind, optimistic of finding solutions, thinking laterally and creatively to look beyond the obvious. Know how to learn from failure.

• Self awareness: Reflect on learning, seeking feedback on and evaluating personal practices, strengths and opportunities for personal growth.

• Communication: Present arguments, knowledge and ideas, in a range of formats.

• Professionalism: Prepared to operate autonomously. Aware of how to be efficient and resilient. Manage priorities and time. Self-motivated, setting and achieving goals, prioritising tasks.