Lecture 24 - Reinforcement Learning II - Q Learning

#notes#cs471

Recap §

Direct Evaluation §

• $sample=R(s,\pi (s),s^{\prime })+\underline{\gamma V^{\pi }(s^{\prime })}$

Temporal Difference Learning §

• Big Idea: Learn from every experience using running average
• Factor in result state (running average)
• $sample=R(s,\pi (s),s^{\prime })+\underline{\gamma V^{\pi }(s^{\prime })}$

Active Reinforcement Learning §

Learner has to choose between exploitation and exploration

• Learn utilities for state/actions
• Compute optimal policy for current learned model

How to Explore? §

• Simplest: Random actions ($ϵ$-greedy)
  • Every time step, flip a coin
    • With small probability $ϵ$, act randomly
    • With large probability $1-ϵ$, act on current policy
  • Problem?
    • Will keep acting randomly even once learning is done
    • Can reduce $ϵ$ over time
    • Can use exploration function

Exploration Functions §

When to explore?

• Random: explore a fixed amount
• Better Idea: Explore areas whose values have not been established
• Exploration Function
  • Takes a estimate $u$ and a visit count $n$, and returns a optimistic utility (k > 0), e.g. $f(u,n)=u+k/n$

Q-Learning Properties §

• Q-Learning converges to optimal policy even if acting sub-optimally

Regret §

Even if you learn optimal policy, you will make mistakes along the way Regret: Total mistake cost. Difference between (expected) rewards including youthful suboptimality and optimal (expected) rewards.

Linear Value Functions §

• Can write q function for any state using few weights
• Advantage: Experience summed up in a few powerful numbers
  • $V(s)=w_1{f}_1(s)+w_2{f}_2(s)+..+w_n{f}_n(s)$
• Disadvantage: States may share features but are different in value

Approximate Q-Learning §

• Update w
  • $w_i\leftarrow w_i+a\ast (diff)f_i(s,a)$