\documentclass{article}
\usepackage{cse599s14sp}
\usepackage{url}
\usepackage{algorithm, algorithmicx, algpseudocode}

% Please use these commands where appropriate:
\newcommand{\Regret}{\operatorname{Regret}}
\newcommand{\Loss}{\operatorname{loss}}
\newcommand{\grad}{\triangledown}
\newcommand{\Ex}[2]{\mathop{\mathbb{E}}\displaylimits_{#1}\left
[ #2 \right ]}
\newcommand{\Expect}[1]{\mathop{\mathbb{E}}\left
[ #1 \right ]}
% The following are defined in the style file, please use them as well:
% \newcommand{\field}[1]{\mathbb{#1}}
% \newcommand{\N}{\field{N}} % natural numbers
% \newcommand{\R}{\field{R}} % real numbers
% \newcommand{\Z}{\field{Z}} % integers
% \newcommand{\norm}[1]{\left\| #1\right\|}
% \DeclareMathOperator*{\argmin}{argmin}
% \DeclareMathOperator*{\argmax}{argmax}
% \DeclareMathOperator*{\maximize}{maximize}
% \DeclareMathOperator*{\minimize}{minimize}
%\input{preamble.tex}

%% the format for the lecture environment is
% \begin{lecture}{lecture number}{lecture title}{scribe}{lecture date}
\begin{lecture}{9}{EXP3 Regret Analysis}{Travis Mandel}{Ofer Dekel}{04/29/2014}

\section{Recap: Difference between Experts and Bandits}

Recall that in past lectures we developed algorithms for the Expert problem, where at each round we had full informationabout how the other experts would have performed if we had selected them.  One setting where this can occur in practice is the stock market. In fact the second programming assignment, which is currently being prepared, will involve running experts algorithms on stock market data that was very recently pulled from NASDAQ.

The leading algorithm in the experts setting is Exponentiated Gradient (EG), which in past lectures we showed has $O(\sqrt{T})$ regret.

However, we are now concerned with the bandit problem, in which we only get feedback from the expert (i.e. arm) we pulled.  The question we are investigating in this unit is of the course is can we still do well given that some feedback has been removed?

In the last lecture we showed that we could use a ``blocking" trick to segment time in such a way that separate samples were allocated to exploration and exploitation.  We randomized when these phases occurred, which allowed us to rely on the assumption of an oblivious adversary to achieve a regret bound of  $O(T^{2/3})$.  However, this bound is pretty bad compared to $O(\sqrt{T})$ we have for EG in the experts problem.

In this lecture we want to recover $O(\sqrt{T})$ for the bandit case.  The key idea is that we don't have to allocate separate samples to exploration and exploration, nor do we have to explicitly mix in some amount of uniform exploration.  Instead it turns out that EG, given a small modification to work in the bandit setting, will have sufficient inherent exploration to give us the desired $O(\sqrt{T})$ regret bound.

\section{The EXP3 Algorithm}

EXP3 is a bandit algorithm developed around the year 2000.  Before that, there had been quite a bit of work studying the stochastic multi-armed bandit problem, dating all the way back to Thompson in 1933.  However it wasn't until the year 2000 that people developed good solutions for the \textbf{adversarial} multi-armed bandit problem.

EXP3 works by passing in a modified loss vector $\hat{\ell}_t$ to the experts algorithm EG, which is constructed as follows:


$$\hat{\ell}_t = \left(
\begin{array}{c}
\dots\\
0\\
0\\
\frac{\ell_{t, I_t}}{p_{t, I_t}}\\
0\\
0\\
\dots\\
\end{array}
\right),
$$

where $I_t$ is the arm we pulled at time $t$. Note that $\hat{\ell}_t$ is an unbiased estimator of $\ell_t$, in other words $\Expect{\hat{\ell}_t | p_t} = \ell_t$. This is true since:

\begin{align}
\Expect{\hat{\ell}_{t,i} | p_t} = p_{t,i} \frac{\ell_{t, i}}{p_{t, i}} + (1-p_{t,i})0 = \ell_{t, i}.&
\label{eq:unbiased}
\end{align}

Beyond being an unbiased estimator, $\hat{\ell}_t$ is highly optimistic for arms that it has not pulled, hence it is likely to explore those arms, which intuitively adds a strong exploration component.

EXP3 can be precisely defined as follows for some learning rate $\eta$:

\begin{algorithm}
    \caption{EXP3}
    \label{alg:exp3}
    \begin{algorithmic}
       
				\State $w_1 = (1,\dots,1)$
				\For{$t = 1$ to $T$}
						\State Define $p_t = \frac{w_t}{\norm{w_t}} $
						\State Draw $I_t \sim p_t$
						\State Observe $\ell_{t,I_t} \in  [0,1]$
						\For{$i = 1$ to $d$}
							\If{$i \neq I_t$}
								\State $w_{t+1, i} = w_{t, i}$
							\Else
							  \State $w_{t+1, i} = w_{t, i} e^{-\eta\frac{\ell_{t, i}}{p_{t, i}}}$
							\EndIf
						\EndFor
				\EndFor
		\end{algorithmic}
\end{algorithm}
				
Note that we could have equivalently written  $w_{t+1, i} = w_{t, i} e^{-\eta\frac{\hat{\ell}_{t, i}}{p_{t, i}}}$ for all $i$, since $\hat{\ell}_{t, i} = 0$ if $i \neq I_t$.

\section{Conditional Expectations and Measure Theory}

Before proving the properties of EXP3, it might be useful to review conditional expectations.  In other words, what does a quantity like $\Expect{\hat{\ell}_t | p_t}$ mean?

Assume $Y$ and $X$ are two real-valued random variables. Then $\Expect{Y}$ is just a scalar number.

How about $\Expect{Y | X = 3}$, assuming the event $X = 3$ has positive probability? It is also a scalar, as it is just the expectation of the distribution of $Y$ when  $X = 3$.  Note that we can calculate that distribution as follows: $P(\cdot | X=3) = \frac{P(\cdot)}{P(X=3)}$.

Ok, then what about $\Expect{Y | X}$?  Well, that is a deterministic function $f(x) = \Expect{Y|X=x}$ applied to a random variable $X$.  Therefore this quantity is also a random variable.  One way to think of it is a random variable which averages out $Y$'s randomness but keeps $X$'s.

Now, say we had some $Z = Y + X$ where $X$ and $Y$ are independent.  By linearity of expectation, $\Expect{Z} = \Expect{Y} + \Expect{X}$

Similarly, $\Expect{Z|X} = \Expect{X|X} + \Expect{Y|X}$.  $\Expect{X|X}$ simply returns $X$ since $\Expect{X|X=x} = x$. Since $Y$ and $X$ are assumed independent, $\Expect{Y|X} = \Expect{Y}$.  So we have:

$$\Expect{Z|X} = X + \Expect{Y}.$$

Going back to our online learning setting, note that the random choice we made yesterday determined today's random  distribution.  So $p_t$ is a random variable which depends on $I_1,\dots,I_{t-1}$.  And $\ell_t$ is a random variable which depends on $I_1,\dots,I_t$

Without going too deep into measure theory, measure theorists would say that $\hat{\ell}_t$ is $\mathcal{F}_t$ measurable. Basically this means that in order to calculate $P(\hat{\ell}_t = \alpha)$ for some $\alpha$, we need to expose all $I_1,\dots, I_t$.  We can't reason about $P(\hat{\ell}_t = \alpha)$ without knowing the behavior of some $I_j \in \{I_1,\dots, I_t\}$.

\section{Analysis of EXP3}
First, we can look at the regret bound EG gives us if we plug $\hat{\ell}_t$ into EG.
This will give us, for all $q \in \triangle_d$,

\begin{align}
\Expect{\sum_{t=1}^T \hat{\ell}_t (p_t - q)}.
\label{eq:modregret}
\end{align}

EG plugs in the following regularizer to FTRL:

$$R = \frac{1}{\eta}(\sum_{i=1}^d q_i \log q_i + \log d),$$

which, from the results in past lectures, gives us the following upper bound on equation \eqref{eq:modregret}:

$$R(q) \leq \frac{1}{\eta} \log d + \eta \sum_{t=1}^T \norm{\hat{\ell}_t}_2^2.$$

However, equation \eqref{eq:modregret} is not the actual regret, because it uses $\hat{\ell}_t$ instead of $\ell_t$.  Therefore we need a lemma to relate this to the true regret.

\begin{lemma}
$\Expect{\sum_{t=1}^T \hat{\ell}_t (p_t - q)} = \Expect{\sum_{t=1}^T \ell_t (p_t - q)}$
\label{lem:banditreg}
\end{lemma}

\begin{proof}
\begin{align*}
&\Expect{\sum_{t=1}^T \hat{\ell}_t (p_t - q)} \\
 = &\sum_{t=1}^T \Expect{\hat{\ell}_t (p_t - q)}
\end{align*} 

Which is due to linearity of expectation.  Now, the law of total expectation tell us that for all random variables $Z, X$ $\Expect{Z} = \Expect{\Expect{Z|X}}$. So we can rewrite as:

\begin{align*}
 = &\sum_{t=1}^T \Expect{\Expect{\hat{\ell}_t (p_t - q) | p_t}}.
\end{align*} 

Now, observe that given a fixed $p_t$, the $p_t-q$ term is a constant.  So therefore the random variable $p_t-q$ is conditionally independent of the random variable $\hat{\ell}_t$ given $p_t$.  Therefore we can split apart the multiplication:

\begin{align*}
 = &\sum_{t=1}^T \Expect{\Expect{\hat{\ell}_t |p_t}\Expect{(p_t - q) | p_t}}
\end{align*} 

We have already shown in equation ~\eqref{eq:unbiased} that $\Expect{\hat{\ell}_t |p_t} = \ell_t$.  And then $\Expect{(p_t - q) | p_t} = \Expect{p_t  | p_t} -q = p_t - q$.   Therefore:

\begin{align*}
 = &\sum_{t=1}^T \Expect{\ell_t(p_t - q)} \\
 = &\Expect{\sum_{t=1}^T \ell_t(p_t - q)}
\end{align*} 

\end{proof}


So now we have the following bound on the true regret:

$$R(q) \leq \frac{1}{\eta} \log d + \eta \sum_{t=1}^T \norm{\hat{\ell}_t}_2^2.$$

However, in order to make this bound concrete we need to deal with the $\norm{\hat{\ell}_t}_2^2$ term.  For our EG analysis we could upper bound this by $G$ since we knew $\norm{\ell_t}_2^2 \leq G$.  In this case we do in fact know $\norm{\ell_t}_2^2 \leq 1$, however, this does not bound the 2 norm of $\hat{\ell}_t$.  In fact, $\hat{\ell}_t$ can grow arbitrarily large if the selected arm $I_t$ has low probability $p_{t, I_t}$. 

However, it turns out there is a special property of EG (and almost only EG) that allows us to still get a good regret bound.   To prove it, we need another lemma.

\begin{lemma}
For the EG algorithm,
$$R(T)\leq \frac{\log d}{\eta} + \frac{\eta}{2}\sum_{t=1}^T \sum_{i=1}^d p_{ti} \ell_{ti}^2.$$
\label{lem:betterEG}
\end{lemma}

The intuition for why this is a good choice is that when we plug in $\hat{\ell}_{ti}$ the $p_{ti}$ will cancel out the denominator in the $\hat{\ell}_{ti}$ term.

\begin{proof}
For some $t$, consider

\begin{align}
\frac{1}{\eta} \log \sum_{i=1}^d p_{ti} e^{-\eta\ell_{ti}}
\label{eq:smiley}
\end{align}


For a lower bound, we can plug EG's formula for $p_{ti}$.  EG sets $p_t = \frac{w_t}{\norm{w_t}_1}$, where $w_{ti} = \prod_{\tau=1}^{t} e^{-\eta \ell_{\tau,i}} = e^{-\eta \ell_{1:(t-1),i}}$ giving us a final result of 

$$p_{ti} = \frac{e^{-\eta \ell_{1:(t-1),i}}}{\sum_{i=1}^d e^{-\eta \ell_{1:(t-1),i}}},$$

 which gives us that equation \eqref{eq:smiley} is equal to 

$$\frac{1}{\eta} \log \sum_{i=1}^d \frac{e^{-\eta \ell_{1:(t-1),i}}}{\sum_{i=1}^d e^{-\eta \ell_{1:(t-1),i}}} e^{-\eta\ell_{ti}}$$

$$ = \frac{1}{\eta} \log \sum_{i=1}^d \frac{e^{-\eta \ell_{1:t,i}}}{\sum_{i=1}^d e^{-\eta \ell_{1:(t-1),i}}} $$
$$ = \frac{1}{\eta} \log \sum_{i=1}^d e^{-\eta \ell_{1:t,i}} - \frac{1}{\eta} \log \sum_{i=1}^d e^{-\eta \ell_{1:(t-1),i}} $$

This gives us $\eqref{eq:smiley}$ for one specific $t$, but if we want to sum $\eqref{eq:smiley}$ from $1$ to $T$ this becomes a telescoping sum.  All terms will cancel but the terms for $T$ and $1$, giving us:

$$\sum_{t=1}^T \frac{1}{\eta} \log \sum_{i=1}^d p_{ti} e^{-\eta\ell_{ti}} = \frac{1}{\eta} \log \sum_{i=1}^d e^{-\eta \ell_{1:T,i}} - \frac{1}{\eta} \log \sum_{i=1}^d e^{-\eta \ell_{1:0,i}}. $$

$\ell_{1:0} = 0$, so $\sum_{i=1}^d e^{-\eta \ell_{1:0},i} = d,$ and we have:

\begin{align}
 = \frac{1}{\eta} \log \sum_{i=1}^d e^{-\eta \ell_{1:T,i}} - \frac{1}{\eta} \log d
 = \frac{1}{\eta} \log \sum_{i=1}^d e^{-\eta \ell_{1:T,i}} - \frac{\log d}{\eta}
\label{eq:lowbound}
\end{align}

Next we would like to upper bound equation \eqref{eq:smiley}.  There are a couple of well-known and useful inequalities we will use to do so (they can be proved using the Taylor expansion):

\begin{align}
\label{eq:alpha1}
\forall \alpha \geq 0, \log(\alpha) &\leq \alpha - 1 \\
e^{-\alpha} &\leq 1 - \alpha + \frac{\alpha^2}{2} 
\label{eq:alpha2}
\end{align}

So since equation \eqref{eq:smiley} is:

$$\frac{1}{\eta} \log \sum_{i=1}^d p_{ti} e^{-\eta\ell_{ti}},$$

we can use \eqref{eq:alpha1} to bound it by:

$$\leq \frac{1}{\eta} (\sum_{i=1}^d p_{ti} e^{-\eta\ell_{ti}} - 1).$$

Now we can apply \eqref{eq:alpha2} to bound it by:

$$\leq \frac{1}{\eta} (\sum_{i=1}^d p_{ti} (1-\eta\ell_{ti}+\frac{\eta^2 \ell_{ti}^2}{2}) - 1)$$

$$= \frac{1}{\eta} \sum_{i=1}^d p_{ti} (\frac{\eta^2 \ell_{ti}^2 -\eta\ell_{ti}}{2})$$

$$=  \frac{\eta}{2}\sum_{i=1}^d p_{ti} \ell_{ti}^2  -\sum_{i=1}^d p_{ti}\ell_{ti}$$

So, returning to equation \eqref{eq:smiley} summed over timesteps:

$$\sum_{t=1}^T \frac{1}{\eta} \log \sum_{i=1}^d p_{ti} e^{-\eta\ell_{ti}} \leq  \frac{\eta}{2}\sum_{t=1}^T\sum_{i=1}^d p_{ti} \ell_{ti}^2  - \sum_{t=1}^T \sum_{i=1}^d p_{ti}\ell_{ti}$$

And from equation \eqref{eq:lowbound}:

$$  \sum_{t=1}^T\frac{1}{\eta} \log \sum_{i=1}^d p_{ti} e^{-\eta\ell_{ti}} =  \frac{1}{\eta} \log \sum_{i=1}^d e^{-\eta \ell_{1:T,i}} - \frac{\log d}{\eta}.$$

So: 

\begin{align}
  \frac{1}{\eta} \log \sum_{i=1}^d e^{-\eta \ell_{1:T,i}} -  \frac{\log d}{\eta}   &\leq    \frac{\eta}{2}\sum_{t=1}^T\sum_{i=1}^d p_{ti} \ell_{ti}^2  - \sum_{t=1}^T \sum_{i=1}^d p_{ti}\ell_{ti} \\
  \sum_{t=1}^T \sum_{i=1}^d p_{ti}\ell_{ti} + \frac{1}{\eta} \log \sum_{i=1}^d e^{-\eta \ell_{1:T,i}}    &\leq   \frac{\log d}{\eta} +  \frac{\eta}{2}\sum_{t=1}^T\sum_{i=1}^d p_{ti} \ell_{ti}^2   
	\label{eq:almost}
	\end{align}

Now, observe:

$$ \frac{1}{\eta} \log \sum_{i=1}^d e^{-\eta \ell_{1:T,i}} \geq \frac{1}{\eta} \log  e^{-\eta \min_i \ell_{1:T,i}},$$  

since the sum must be greater than its minimum element.  We can cancel the log to get:

$$= \frac{1}{\eta} -\eta \min_i \ell_{1:T,i}$$
$$\geq -\eta \min_i \ell_{1:T,i}$$

Now, plugging this smaller quantity back in to equation \eqref{eq:almost} we have:

$$  \sum_{t=1}^T \sum_{i=1}^d p_{ti}\ell_{ti}  -\eta \min_i \ell_{1:T,i}    \leq   \frac{\log d}{\eta} +  \frac{\eta}{2}\sum_{t=1}^T\sum_{i=1}^d p_{ti} \ell_{ti}^2.   $$

The equation on the left is just the formula for regret in the experts setting, so we are done.
\end{proof}

\begin{theorem}
EXP3 has regret bound $\sqrt{2Td\log d}$.
\end{theorem}
\begin{proof}
By Lemma \ref{lem:betterEG} EG has a regret bound of 

$$\frac{\log d}{\eta} +  \frac{\eta}{2}\sum_{t=1}^T\sum_{i=1}^d p_{ti} \ell_{ti}^2.  $$

Now if we use EXP3, we have simply changed $\ell_{ti}$ to $\hat{\ell}_{ti}$.  Lemma \ref{lem:banditreg} tells us that

\begin{align*}
\Expect{R(T)} =& \Expect{\frac{\log d}{\eta} +  \frac{\eta}{2}\sum_{t=1}^T\sum_{i=1}^d p_{ti} \hat{\ell}_{ti}^2} \\
=& \frac{\log d}{\eta} +  \frac{\eta}{2}\sum_{t=1}^T\sum_{i=1}^d \Expect{ p_{ti} \hat{\ell}_{ti}^2} \\
=& \frac{\log d}{\eta} +  \frac{\eta}{2}\sum_{t=1}^T\sum_{i=1}^d \Expect{\Expect{p_{ti} \hat{\ell}_{ti}^2 | p_t}} 
\end{align*}

Similar to the proof of Lemma \ref{lem:banditreg}, observe that $\Expect{p_{ti} | p_t}$ is just a constant, so it is conditionally independent of $\hat{\ell}_{ti}^2$ given $p_t$.

\begin{align}
=& \frac{\log d}{\eta} +  \frac{\eta}{2}\sum_{t=1}^T\sum_{i=1}^d \Expect{p_{ti} \Expect{ \hat{\ell}_{ti}^2 | p_t}} 
\label{eq:stopsudden}
\end{align}

Now, observe that, since $p_{ti}$ is the probability of filling in element $i$ of the vector $\hat{\ell}_{ti}$:
$$\Expect{ \hat{\ell}_{ti}^2 | p_t} = p_{ti} \frac{\ell_{ti}^2}{p_{ti}^2}  + (1-p_{ti}) 0 = \frac{\ell_{ti}^2}{p_{ti}}$$. Plugging into \eqref{eq:stopsudden} we have:

\begin{align}
=& \frac{\log d}{\eta} +  \frac{\eta}{2}\sum_{t=1}^T\sum_{i=1}^d \Expect{p_{ti} \frac{\ell_{ti}^2}{p_{ti}}} \\ 
=& \frac{\log d}{\eta} +  \frac{\eta}{2}\sum_{t=1}^T\sum_{i=1}^d \Expect{\ell_{ti}^2}
\end{align}

Next, $\ell_{ti}$ is chosen by the adversary beforehand and so does not involve any randomness once the game starts.  So the expectation is just the fixed value.
\begin{align}
=& \frac{\log d}{\eta} +  \frac{\eta}{2}\sum_{t=1}^T\sum_{i=1}^d \ell_{ti}^2
\end{align}

We know $\ell_{ti} \in [0, 1]$, so $\ell_{ti}^2 \in [0, 1]$.  Therefore $T\sum_{i=1}^d \ell_{ti}^2 \leq Td$, so:

$$R(T) \leq \frac{\log d}{\eta} +  \frac{\eta}{2}Td.$$

All that remains is to choose $\eta$ as a function of $T$ to get the best bound possible.  One good choice is

$$\eta = \sqrt{\frac{2 \log d}{T d}}.$$

Plugging this in, we have:

$$R(T)\leq  \log d\frac{\sqrt{Td}}{\sqrt{2 \log d}} +  \frac{1}{2} \sqrt{\frac{2 \log d}{T d}} Td$$

$$= \frac{1}{\sqrt{2}}\sqrt{Td}\sqrt{\log d} +  \frac{1}{2} \sqrt{2}\sqrt{ \log d }\sqrt{T d}$$

$$R(T) \leq \sqrt{2Td\log d}$$
\end{proof}



\end{lecture}
\theend
