Lecture 21 - Markov Decision Process - II

#notes#cs471

Recap: Defining MDP §

• Markov Decision Process:
  • Set of states S
  • Start state $s_0$
  • Set of actions A
  • Transitions $P(s^{\prime }|s,a)$
  • Rewards $R(s,a,s^{\prime })$ (and discount $y$)
• Policy $\pi$ = Choice of action for each state
• Utility = Sum of (discounted) rewards

Convergence §

• Claim: Value Iteration Converges
• Notation:
  • $V_k\in {\mathbb{R}}^{|S|}$, a vector of size $|S|$, where $S$ denotes set of states
  • $V_{k+1}=B{V}_k$ where operator B denotes an iteration of Bellman Update
  • $||V_k|{|}_{\infty }=ma{x}_s{V}_k(s)$