CSE599s, Spring 2014, Online Learning Lecture 99 - 05/08/2014

Adaptive Regret Bound

Lecturer: Brendan McMahan Scribe: Tianqi Chen

1 Recap

A normal regret bound for fixed learning rate is as follows

Regret(u) ≤ \frac{1}{2η} ∥u∥^{2} + η ∑_{t=1}^{T} ∥g_{t}∥_{2}^{2}  (1)

We should note that the regret depends on choice of η. If we do not know T in advance, this bound could be very bad. Even if we know T and set η properly, we need to wait until we get T to get the regret bound. Ideally, we want our bound to hold for any T, this is where we need to introduce adaptive update.

Our goal is prove a bound in the following style.

Regret ≤ B\sqrt{∑_{t=1}^{T} ∥g_{t}∥^{2}} << GB\sqrt{T}  (2)

We want to have a bound that does not need guess and double trick.

There are several class of related algorithms, that we will be discussed in a general framework OGD/Mirro Descent:

ŵ_{t+1} = ŵ_{t} − η_{t}g_{t} = argmin_{w} g_{t}w + \frac{1}{2η} ∥w − w_{t}∥^{2}.  (3)

FTRL-Proximal

w_{t+1} = argmin_{w} f_{1:t}(w) + ∑_{s=1}^{t} \frac{σ_{s}}{2} ∥w − w_{s}∥^{2}.  (4)

Dual-Averaging

w_{t+1} = argmin_{w} f_{1:t}(w) + \frac{σ_{1:t}}{2} ∥w∥^{2}.  (5)

They are equivalent when we have no constraint. FTRL Proximal and dual averaging are equivalent when learning rate is constant.

2 General Framework for Adaptive Update

In this lecture, we will study the update rule in the following form:

w_{t+1} = argmin_{w} f_{1:t}(w) + r_{0:t}(w) = argmin_{w} h_{0:t}(w)  (6)

Note that we have h_{0}(w) = r_{0}(w). Base on this update rule, we have a strong FTRL Lemma as follows

Lemma 1. Strong FTRL Lemma

Regret(u) ≤ r_{0:T} (u) + ∑_{t=1}^{T}[h_{0:t}(w_{t}) − h_{0:t}(w_{t+1}) − r_{t}(w_{t})]  (7)

Proof. The bound can be proved by induction(see previous lecture note)

Before we prove the main theorem, we will need the following lemma

Lemma 2. Let

w_{1} = argmin_{w} ϕ_{1},

w_{2} = argmin_{w} ϕ_{2} = argmin_{w}[ϕ_{1}(w) + ψ(w)],

where ϕ_{1} is 1 strongly convex function with respect to norm ∥.∥, and ψ(w) is convex function. Let b ∈ ∂ψ(w), then we will have

ϕ_{2}(w_{1}) − ϕ_{2}(w_{2}) ≤ \frac{1}{2}∥b∥_{∗},

∥w_{1} − w_{2}∥ ≤ ∥b∥_{∗}.

We can verify that for a special case where ϕ_{1} is quadratic function, and ψ is linear: ϕ_{1}(w) = \frac{1}{2}∥w∥^{2}, ϕ(w) = bw, this bound is tight.

Theorem 3. Assuming r_{t}(w) ≥ 0, r_{t}(w_{t}) = 0, h_{0:t}(w) is 1 strongly convex with respect to ∥.∥_{t}. Then the regret of general framework can be bounded by

Regret(u) ≤ r_{0:T} (u) + \frac{1}{2} ∑_{t=1}^{T} ∥g_{t}∥_{(t,∗)}^{2}.  (8)

Proof. For fixed round t, Let

ϕ_{1}(w) = f_{1:t−1}(w) + r_{1:t−1}(w) + r_{t}(w) = h_{0:t−1}(w) + r_{t}(w_{t})

Note that w_{t} = argmin_{w} r_{t}(w_{t}), we have w_{t} = argmin_{w} ϕ_{1}(w). Let ψ = f_{t}, and b = g_{t}, g_{t} ∈ ∂f_{t}(w). The following inequality holds follows because of Lemma 1.

\begin{aligned}h_{0:t}(w_{t}) − h_{0:t}(w_{t+1}) & = ϕ_{1}(w_{t}) + f_{t}(w_{t}) − ϕ_{1}(w_{t+1}) − f_{t}(w_{t+1}) \\ & = ϕ_{2}(w_{t}) − ϕ_{2}(w_{t+1}) ≤ \frac{1}{2}∥g_{t}∥_{(t,∗)}^{2}. \end{aligned}  (9)

Then the results follows by Lemma 2.

Now we need to make use Theorem 3 to analyze FTRL-Proximal algorithm. A first simple fact is that if r_{t} is σ_{t} strongly convex with respect to ∥.∥, then r_{0:t} is 1 strongly convex with respect to ∥u∥_{t} = \sqrt{σ_{1:t}}∥.∥.

For FTRL-Proximal, we have

• r_{0}(w) = I_{W} (w)

• r_{t} = \frac{σ_{t}}{2} ∥w − w_{t}∥^{2}, note η_{t} = \frac{1}{σ_{1:t}}

• ∥g∥_{t,∗} = \frac{1}{\sqrt{σ_{1:t}}} ∥g∥_{2}

Applying Theorem 3, we can get the following bound for adaptive learning rate.

Regret(u) ≤ \frac{(2B)^{2}}{2η_{T}} + \frac{1}{2} ∑_{t=1}^{T} η_{t}∥g_{t}∥^{2}  (10)

We still need to decide how we can choose η_{t}, an important bound that we will use, is stated by following Lemma Lemma 4. For sequence a_{1}, a_{2}, ⋯, a_{n} , a_{i} ≥ 0 the following inequality holds.

∑_{i=1}^{n} \frac{a_{i}}{\sqrt{∑_{j=1}^{i} a_{j}}} ≤ 2\sqrt{∑_{i=1}^{n} a_{i}}  (11)

Proof. Let x_{i} = ∑_{j=1}^{i} a_{j}, x_{0} = 0, first note the integral equality

∫_{0}^{x_{n}} \frac{1}{\sqrt{z}} dz = 2\sqrt{x_{n}} − 2\sqrt{0}  (12)

This is because 2∂_{x}\sqrt{x} = \frac{1}{\sqrt{x}}. Then we can think how we can “compute” the integral in the left side numerically. We can first discretize the interval into small pieces of length a_{1}, a_{2}, a_{3} ⋯, then take the right end of the function to approximate the function value in that interval. Note that the right end of function \frac{1}{\sqrt{z}} is smaller than the functions in the interval, we can get a lower bound of integral:

∫_{0}^{x_{n}} \frac{1}{\sqrt{z}} dz = ∑_{i=0}^{n−1} ∫_{x_{i+1}}^{x_{i}} \frac{1}{\sqrt{z}} dz ≥ ∑_{i=0}^{n−1} \frac{x_{i+1} − x_{i}}{\sqrt{x_{i+1}}} = ∑_{i=1}^{n} \frac{a_{i}}{\sqrt{∑_{j=1}^{i} a_{j}}}  (13)

As a special case (take a_{i} = 1), we have ∑_{t=1}^{T} \frac{1}{\sqrt{t}} ≤ 2\sqrt{T}. If we choose η_{t} = \frac{\sqrt{2}B}{G\sqrt{t}}, we have

Regret(u) ≤ \frac{(2B)^{2}}{2η_{T}} + \frac{1}{2} ∑_{t=1}^{T} \frac{\sqrt{2}B}{G\sqrt{t}} G^{2} ≤ 2\sqrt{2}GB\sqrt{T}  (14)

We can also let a_{i} = ∥g_{t}∥^{2}, η_{t} = _{\frac{α}{\sqrt{∑_{s=1}^{t} ∥g_{s}∥^{2}}}}

\frac{1}{2} ∑_{t=1}^{T} η_{t}∥g_{t}∥^{2} ≤ α\sqrt{∑_{t=1}^{T} ∥g_{t}∥^{2}}  (15)

The adaptive regret bound is given by

Regret(u) ≤ \frac{(2B)^{2}}{2α} \sqrt{∑_{t=1}^{T} ∥g_{t}∥^{2}} + α\sqrt{∑_{t=1}^{T} ∥g_{t}∥^{2}} = (\frac{2B^{2}}{α} + α) \sqrt{∑_{t=1}^{T} ∥g_{t}∥^{2}}  (16)