Graduate Thesis Or Dissertation


Simulator-Defined MDP Planning with Applications in Natural Resource Management Public Deposited

Downloadable Content

Download PDF


Attribute NameValues
  • This work is inspired by problems in natural resource management centered on the challenge of invasive species. Computing optimal management policies for maintaining ecosystem sustainable is challenging. Many ecosystem management problems can be formulated as MDP (Markov Decision Process) planning problems. In a simulator-defined MDP, the Markovian dynamics and rewards are provided in the form of a simulator from which samples can be drawn. Simulators in natural resource management can be very expensive to execute, so that the time required to solve such MDPs is dominated by the number of calls to the simulator. This thesis studies MDP planning algorithms that attempt to minimize the number of simulator calls before terminating and outputting a policy that is approximately optimal with high probability. This thesis addresses three questions on unconstrained MDPs: (a) what confidence interval should be employed to bound the optimality of the policy and (b) how should samples be drawn to shrink the confidence interval as quickly as possible? (c) how can we find the optimal policy with high probability efficiently? Many computational sustainability problems involving MDPs must also be concerned with catastrophic outcomes such as species extinction. These problems can be formulated as constrained MDPs. We define the downside risk as the probability of reaching catastrophic states and then constrain the MDP solution to bound the probability of entering such states. We then develop the first PAC-Safe-RL algorithm for constrained MDPs. We evaluate our algorithms on an invasive species problem as well as on standard reinforcement learning benchmarks.
Resource Type
Date Available
Date Issued
Degree Level
Degree Name
Degree Field
Degree Grantor
Commencement Year
Committee Member
Academic Affiliation
Non-Academic Affiliation
Rights Statement
Peer Reviewed



This work has no parents.

In Collection: