\documentclass{article}
\usepackage{cse599s14sp}
\usepackage{url}
% Please use these commands where appropriate
\newcommand{\Regret}{\operatorname{Regret}}
\newcommand{\Loss}{\operatorname{loss}}
\newcommand{\grad}{\triangledown}
\usepackage{natbib}
\usepackage{hyperref}
% The following are defined in the style file, please use them as well:
% \newcommand{\field}[1]{\mathbb{#1}}
% \newcommand{\N}{\field{N}} % natural numbers
% \newcommand{\R}{\field{R}} % real numbers
% \newcommand{\Z}{\field{Z}} % integers
% \newcommand{\norm}[1]{\left\| #1\right\|}
%\DeclareMathOperator*{\argmin}{argmin}
% \DeclareMathOperator*{\argmax}{argmax}
% \DeclareMathOperator*{\maximize}{maximize}
% \DeclareMathOperator*{\minimize}{minimize}

%% the format for the lecture environment is
% \begin{lecture}{lecture number}{lecture title}{scribe}{lecture date}
\begin{lecture}{20}{CTR Predictions and Literature References}{Javad Hosseini}{Brendan McMahan or Ofer Dekel}{6/5/2014}
In the final lecture of the online learning course, we talk about one of the applications of {\it Online Learning} and provide some references on what we have learnt so far and future readings. In section \ref{sec:ctr}, we present a talk in ``ACM SIGKDD international conference on Knowledge discovery and data mining'' on ``Click Through Rates''. We present some ``Literature References'' in section \ref{sec:ref}.
\section{Ad Click Prediction: a view from the trenches} \label{sec:ctr}
Many billions of dollars a year are spent on web advertising. In search engines, after a user enters his/her query, targeted ads are shown in response to the query. Figure \ref{fig:SE} shows an example for Google. Web search ads align incentives. This is because search engines only get paid if the user clicks on the ad. This is not a perfect proxy for relevant ads, but it is good-enough to say that clicked ads are relevant. So search engines like to show relevant ads. On the other hand, users also like relevant ads more. In order to satisfy these, the key ingredient is predicting the probability of a click for a specific ad in response to a specific query. The most important features for this task are search phrases and ad text.  Figure \ref{fig:over} shows the system overview of a prediction system for click probability. The data including ads, features and labels are streaming and we do not know the dimensionality in advance. There are some challenges in systems for training massive models on massive data with minimum resources. These systems should be able to handle 
\begin{itemize}
\item billions of unique features (model coefficients)
\item billions of predictions per day serving live traffic
\item billions of training examples
\end{itemize} 
\begin{figure}[h]
\centering
\includegraphics[scale=.35]{figs/SE.jpg}
\caption{\small Google Adwords ads}
\label{fig:SE}
\end{figure}


\begin{figure}[h]
\centering
\includegraphics[scale=.35]{figs/over.png}
\caption{\small System Overview}
\label{fig:over}
\end{figure}

\subsection{FTRL-Proximal Online Learning Algorithms}
The Follow The Regularized Leader Proximal (FTRL-Proximal) algorithm is similar to stochastic gradient descent, but has much sparser models. It is an online algorithm, therefore can be served for the click prediction task. It is simple, rich and has rich theory. If no regularization is used, this algorithm will be equivalent to Online (Stochastic) Gradient Descent \cite{mcmahan2011follow}. FTRL-Proximal implements regularization in a way similar to RDA \cite{xiao2010dual}, but gives better accuracy in our experiments. The key is re-expressing gradient descent implicitly as:

\begin{equation}
w_{t+1} = \argmin_{w} (g_{1:t}.w+\frac{1}{2}\sum_{s=1}^t\sigma_s \norm{w-w_s}_2^2+\lambda_1\norm{w}_1).
\end{equation}

This algorithm can be implemented as easily as gradient descent with a few tricks.

\subsection{Per-Coordinate Learning Rates}
Instead of having a constant learning rate in stochastic gradient descent, one can think of per-coordinate learning rates. In order to have an intuition why this can be helpful, consider predicting for English queries in the US, which includes 1,000,000 examples. Also, consider learning for Iceland queries in Iceland (100 training examples). If we imagine that the features are disjoint, we will train separate SGD models for each, i.e. generally around $\frac{1}{\# examples}$. On the other hand, if we train a single model by concatenating the feature vectors and interleave training examples, there will be no good learning rate to choose. SGD will either never converge for iceland (with a low learning rate) or will oscillate widely for the US (with a large learning rate). While there may be a middle ground, it will be still sub-optimal.

Theory indicates learning rate 

\begin{equation}
\eta_{t,i} = \frac{\alpha}{\beta+\sqrt{\sum_{s=1}^t g_{s,i}^2}}.
\end{equation}

Implementing this requires storing one extra statistics per coordinate. However, we have a trick for reducing this if you are training multiple similar models. Using per-coordinate learning rates will lead to huge accuracy improvement. This improved AUC by 11.2\% versus a global learning rate while for the advertisement task a 1\% improvement is large \cite{mcmahan2010adaptive, duchi2011adaptive}

\subsection{Techniques for saving memory}
Due to L1 regularization used in FTRL-Proximal, there are some features that are not included in the final model. This motivates seeking techniques for saving memory both while training and testing. While training, we may use probabilistic feature inclusion. Grouping similar documents and randomized rounding are other techniques for saving memory while training. While testing, feature reduction because of L1 regularization leads to saving memory. Also, we may still use randomized rounding. These techniques are discussed later in this section.

Figure \ref{fig:trade} shows the tradeoff frontier for a small ($10^6$ examples) dataset between the fraction of selected features by L1 regularization and AUC for three algorithms including FTRL-Proximal. It can be seen that smaller models will have better accuracy. However, if the fraction of selected features becomes too low, the performance will decrease. Each line varies the L1 parameter for different size/accuracy tradeoffs.

In order to evaluate FTRL-Proximal, we compare it to a baseline algorithm. In baseline, only features that occur at least $K$ times are included. We have tuned the L1 parameter to provide a good sparsity/accuracy tradeoff.	 Then we tuned $K$ to get the same accuracy with the baseline. This resulted in 3  times non-zero features in baseline. We can conclude that L1-regularization selects features well.

\begin{figure}[h]
\centering
\includegraphics[scale=.35]{figs/trade.png}
\caption{\small Tradeoff between and Performance}
\label{fig:trade}
\end{figure}

\subsubsection{Probabilistic Feature Inclusion}
Long-tailed distributions produce many features that only occur a handful of times. In some of our models, half of the unique features occur only once in a training set of billion examples. There generally are not useful for prediction. If we want to use this intuition to reduce dimensionality, it cannot be done directly. Since if we read all the stream of data and then want to prune low-accuracy features, we have already used a lot of memory. We want to distinguish rare features from common features without tracking any state. This can be done by probabilistic techniques. Two probabilistic features that can be used are Poisson inclusion and Bloom-filter inclusion. In Poisson inclusion, when a feature not in the model is seen, it is added to the model with probability $p$. Intuitively, with low $p$, if a feature is seen a lot of times, it will be included. Otherwise, chance of inclusion of a feature is low. In Bloom-filter inclusion, a rolling set of counting Bloom-filters is used to count occurrences. As a result, a feature will be added after $K$ occurrences, but sometimes features with lower than $K$ occurrences will also be added. In Table \ref{tab:inc}, the amount of saved RAM and also detriment in Auc is shown when these techniques are used. while the decrease in performance is low, we see around 50\% decrease in used RAM for different methods. 

\begin{table}[h]
\centering
\includegraphics[scale=.35]{figs/inc.png}
\caption{\small Feature Inclusion Tradeoff}
\label{tab:inc}
\end{table}

\subsubsection{Storing coefficients with fewer bits: Randomized Rounding}
The learning rate tells us how accurately we might know the coefficient. If training on one example might change the coefficient by $\Delta$, one does not know the value with more precision than $\Delta$. Therefore, we do not need to waste memory storing a full-precision floating point value with 32 or 64 bits. In practice, storing a 16-bit fixed point representation is easy and works well. However, we need to be careful about accumulating errors. While an update may not change the value of a coefficient, lots of small updates will be important. We can use randomized rounding trick \cite{golovin2013large}. For training with randomized rounding, usual update is computed at full-precision, then the result is randomly projected to an adjacent value expressible in the fixed-point encoding. It should be noted that rounding should be unbiased, i.e. E[rounded coefficient] = unrounded coefficient. In our experiments, there was no accuracy loss with a 16-bit fixed-point representation, compared to 32 or 64 bit floating point values, which means 50\%-75\% less memory. Figure \ref{tab:round} shows this technique.

\begin{figure}[h]
\centering
\includegraphics[scale=.35]{figs/round.png}
\caption{\small Randomized Rounding}
\label{tab:round}
\end{figure}

\subsubsection{Training many similar models Grouped}
Training similar models causes loss of memory. As an example, assume we train four similar models separately with coefficients stored in a hash table. As it is shown in Figure\ref{fig:ind}, for each feature, we may use 10 bytes for its key, its gradient-squared sum (used to compute learning rate) and the coefficient. The reason that we want to have similar models is that we may want to try different feature variations, learning rates, regularization strengths, etc.

\begin{figure}[h]
\centering
\includegraphics[scale=.35]{figs/ind.png}
\caption{\small Training many similar models: individually}
\label{fig:ind}
\end{figure}

We can train many similar models grouped and store data for all the similar models in one hash table. This technique is shown in Figure\ref{fig:group}. Besides 12 bytes of overhead per coefficient, this technique only uses 2 bytes per coefficient per model. However, we will not have exact per-coordinate learning rates. Instead, we use a good-enough approximation based on the number of observed positive and negative examples. This trick, however, requires using the same learning-rate statistic for all grouped models. The theory-recommended learning rate is 



\begin{equation}
\eta_{t,i} = \frac{\alpha}{\beta+\sqrt{\sum_{s=1}^t g_{s,i}^2}}.
\end{equation} 


\begin{figure}[h]
\centering
\includegraphics[scale=.35]{figs/group.png}
\caption{\small Training many similar models: grouped}
\label{fig:group}
\end{figure}

If we only use counts, we have

\begin{equation}
\sum g_{t,i}^2 = \sum_{\text{positive events}} (1-p_t)^2 + \sum_{\text{negative events}} (p_t)^2\\ \approx P(1-\frac{P}{N+P})^2 + N(\frac{P}{N+P})^2 =\\ \frac{PN}{N+P},
\end{equation} 

where N is the number of positive examples and P is the number of negative examples.

\subsection{High-Dimensional Data Visualization for Model Accuracy}
The aggregate accuracy is not the only thing that matters. In online learning we also care about progressive validation, i.e. computing prediction on each example before training on it. We also want to look at relative changes in accuracy between models and also be able to quickly spot outliers or problem areas. Figure \ref{fig:vis} shows an example of performance monitoring for three models on consecutive data.

\begin{figure}[h]
\centering
\includegraphics[scale=.45]{figs/vis.png}
\caption{\small High-Dimensional Data Visualization for Model Accuracy}
\label{fig:vis}
\end{figure}

\subsection{Automated Feature Management System}
In Google, we need a system to automatically manage features. There are many raw signals such as words-in-user-query, country-of-origin, etc. Many engineers are working on signals and multiple different learning platforms and teams consume them. There is also a metadata index to manage all these signals. It knows what systems are using what signals, what signals are available to different systems, status and version information/deprecation and also whitelist for production readiness/test status. The automated feature management system is used for automatic testing, alerting, notification and cleanup of unused signals.

\subsection{Final remarks about the paper}
In the paper we had also discussed some other topics:
\begin{itemize}
\item Fast approximate training with a single value structure
\item Unbiased training on a biased subsample of training data
\item Assessing model accuracy with progressive validation
\item Cheap confidence estimates
\item Approaches for calibrating predictions
\end{itemize}

There were also some techniques that did not work as well in the paper:
\begin{itemize}
\item Aggressive feature hashing
\item Randomized feature dropout
\item Averaging models trained on different subsets of the features
\item Feature vector normalization
\end{itemize}

\section{References and Other Topics}\label{sec:ref}
We discussed references and some other topics that can be found \href{http://courses.cs.washington.edu/courses/cse599s/14sp/online\_learning\_references.pdf}{here}. Presented references include:

\begin{enumerate}
\item Online Linear and Convex Optimization
\begin{enumerate} 
\item Projected Gradient Descent View
\item Follow-the-Regularized-Leader View
\end{enumerate} 

\item Kalai-Vempala
\item Learning with Structure
\item Log(T) Regret for Strongly Convex f
\item ``Second-Order'' Algorithms (also for Classification in the Mistake Bound Model)
\item ``Second-Order" Algorithms for Linear Functions (aka, AdaGrad)
\begin {enumerate}
\item The per-coordinate gradient descent algorithm
\item General feasible sets
\end{enumerate}
\item The Experts Setting / Entropic Regularization
\begin{enumerate}
\item Experts Setting
\item EG vs GD for Squared Error
\item Game Theory View
\end{enumerate}
\item Minimax Analysis and Unconstrained Linear Optimization
\item K-Armed Bandits (EXP3) and Contextual Bandits (EXP4)
\begin{enumerate}
\item Original EXP3 and EXP4 Analysis
\item Analysis for Losses (No Mixing Needed)
\item Improved EXP4 Analysis
\item High-probability bounds for EXP4
\end{enumerate}
\item Stochastic Approaches to the Contextual Bandits Problem
\begin{enumerate}
\item Stochastic Setting
\item Model
\end{enumerate}
\item Bandit Convex Optimization
\begin{enumerate}
\item General $T^\frac{3}{4}$ Regret
\item Strongly convex functions $T^\frac{2}{3}$ Regret
\item Smooth convex functions, $T^\frac{2}{3}$ Regret
\end{enumerate}
\item Bandit Linear Optimization
\item Online Submodular Minimization
\item Online Kernel Methods with a Budget of Support Vectors
\item Selective Sampling / Online Active Learning / Label Efficient Learning
\item Other Problems
\begin{enumerate}
\item Online PCA
\item Online One-Class Prediction (e.g., outlier detection)
\item Online Ranking
\end{enumerate}

\end{enumerate}



\bibliography{j}
\bibliographystyle{plain}




\end{lecture}
\theend
