Lecture 20 - Markov Decision Process I

#notes#cs471

Example: Grid World §

Maze-Like Problem: Agent lives in a grid, walls block agent path. Noisy Movement: Actions do not always go as planned

• Example: 80% of the time North goes North
• 10% North goes West, and 10% North goes East. Goal: Maximize sum of rewards

Markov Decision Process §

A MDP is defined by:

• Set of states $s\in S$
• Set of actions $a\in A$
• Transition function $T(s,a,s^{\prime })$
  • Probability that $a$ from $s$ leads to $s^{\prime }$, i.e. $P(s^{\prime }|s,a)$
• Reward function $R(s,a,s^{\prime })$
  • Sometimes just $R(s)$ or $R(s^{\prime })$

What is Markov about MDPs? §

• “Markov” generally means given present state, future and past are independent
• For Markov decision processes, “Markov” means action outcomes depend only on the current state
  • $P(S_{t+1}=s^{\prime }|S_{\mathrm{0..}t}=s_{\mathrm{0..}t},A_{\mathrm{0..}t}=a_{\mathrm{0..}t})$ = $P(S_{t+1}=s^{\prime }|S_t=s_t,A_t=a_t)$

Policies §

• When deterministic, we want a optimal plan from start to goal
• For MDPs, we want a optimal policy ${\pi }^{\ast }:S\to A$
  • Policy $\pi$ gives action for each state
  • Optimal policy maximizes expected utility if followed

Infinite Utilities §

Problem: What if game lasts forever? Solutions:

• Finite Horizon: Similar to depth-limited search.
  • Terminate episodes after fixed T steps (life)
  • Gives nonstationary policies (p depends on time left)
• Discounting: use 0 < y < 1
  • $U([r_0,...,r_{\infty }])={\sum }_{t=0}^{\infty }{y}^t{r}_t\le \frac{R_{max}}{(1-y)}$
• Absorbing State: Guarantee that for every policy, terminal state will be reached.

Optimal Quantities §

The value (utility) of state s:

• $V^{\ast }(s)$ = expected utility starting in $s$ and acting optimall.y The value (utility) of state-action pair: