Optimization of controlled Markov chains with application to dam management

Continuous-time Controlled Markov Chains are a useful model for many processes where it is necessary to alter the future behavior of the chain in a probabilistic way based on the current state. This project examines the techniques required to find the set of optimal controls for each state of the chain given a set of performance criteria, in both the unconstrained and constrained cases. We focus on the control of systems where there is a finite control horizon, the dynamics are non-stationary, and there are no apriori stability conditions. This will be demonstrated with a series of increasingly complex models which describe the management of a single dam or a system of arbitrarily connected dams. In these models a variety of controls are used: a price is imposed on water consumption in order to reduce overall water use, controlled transfers between dams are imposed to maintain system balance and controlled releases are allowed to reduce the chance of catastrophic flood. High performance numerical computing techniques are used for the solution of these problems and we demonstrate that implementable optimal control strategies can be computed.