Online Learning Survey

Foundations and TrendsR/circlecopyrtin
Machine LearningVol. 4, No. 2 (2011) 107–194
c/circlecopyrt2012 S. Shalev-Shwartz
DOI: 10.1561/2200000018
Online Learning and Online
Convex Optimization
By Shai Shalev-Shwartz
Contents
1 Introduction 108
1.1 Examples 111
1.2 A Gentle Start 1121.3 Organization and Scope 1161.4 Notation and Basic Deﬁnitions 117
2 Online Convex Optimization 119
2.1 Convexiﬁcation 120
2.2 Follow-the-leader 1242.3 Follow-the-Regularized-Leader 1272.4 Online Gradient Descent: Linearization
of Convex Functions 130
2.5 Strongly Convex Regularizers 1342.6 Online Mirror Descent 1412.7 The Language of Duality 1462.8 Bounds with Local Norms 1522.9 Bibliographic Remarks 155
3 Online Classiﬁcation 157
3.1 Finite Hypothesis Class and Experts Advice 158
3.2 Learnability and the Standard Optimal Algorithm 160
3.3 Perceptron and Winnow 168
3.4 Bibliographic Remarks 175
4 Limited Feedback (Bandits) 177
4.1 Online Mirror Descent with Estimated Gradients 178
4.2 The Multi-armed Bandit Problem 1794.3 Gradient Descent Without a Gradient 1824.4 Bibliographic Remarks 185
5 Online-to-Batch Conversions 186
5.1 Bibliographic Remarks 190
Acknowledgments 191References 192
Foundations and TrendsR/circlecopyrtin
Machine LearningVol. 4, No. 2 (2011) 107–194
c/circlecopyrt2012 S. Shalev-Shwartz
DOI: 10.1561/2200000018
Online Learning and Online
Convex Optimization
Shai Shalev-Shwartz
Benin School of Computer Science and Engineering, The Hebrew University
of Jerusalem, Israel, shais@cs.huji.ac.il
Abstract
Online learning is a well established learning paradigm which has both
theoretical and practical appeals. The goal of online learning is tomake a sequence of accurate predictions given knowledge of the cor-rect answer to previous prediction tasks and possibly additional avail-able information. Online learning has been studied in several researchﬁelds including game theory, information theory, and machine learning.It also became of great interest to practitioners due the recent emer-gence of large scale applications such as online advertisement placementand online web ranking. In this survey we provide a modern overviewof online learning. Our goal is to give the reader a sense of some ofthe interesting ideas and in particular to underscore the centrality ofconvexity in deriving eﬃcient online learning algorithms. We do notmean to be comprehensive but rather to give a high-level, rigorous yeteasy to follow, survey.
1
Introduction
Online learning is the process of answering a sequence of questions
given (maybe partial) knowledge of the correct answers to previousquestions and possibly additional available information. The study ofonline learning algorithms is an important domain in machine learn-ing, and one that has interesting theoretical properties and practicalapplications.
Online learning is performed in a sequence of consecutive rounds,
where at round tthe learner is given a question, x
t, taken from an
instance domain X, and is required to provide an answer to this ques-
tion, which we denote by pt. After predicting an answer, the correct
answer, yt, taken from a target domain Y, is revealed and the learner
suﬀers a loss, l(pt,yt), which measures the discrepancy between his
answer and the correct one. While in many cases ptis inY, it is some-
times convenient to allow the learner to pick a prediction from a largerset, which we denote by D.
108
109
Online Learning
fort=1,2,...
receive question xt∈X
predict pt∈D
receive true answer yt∈Y
suﬀer loss l(pt,yt)
The speciﬁc case of yes/no answers and predictions, namely D=
Y={0,1}, is called online classiﬁcation. In this case it is natural to
use the 0–1 loss function: l(pt,yt)=|pt−yt|. That is, l(pt,yt) indicates
ifpt=yt(the prediction is correct) or pt/negationslash=yt(the prediction is wrong).
For example, consider the problem of predicting whether it is going
to rain tomorrow. On day t, the question xtcan be encoded as a vector
of meteorological measurements. Based on these measurements, thelearner should predict if it’s going to rain tomorrow. In the followingday, the learner knows the correct answer.
We can also allow the learner to output a prediction in [0 ,1], which
can be interpreted as the probability of raining tomorrow. This is anexample of an application in which D/negationslash=Y. We can still use the loss
function l(p
t,yt)=|pt−yt|, which can now be interpreted as the prob-
ability to err if predicting that it’s going to rain with probability pt.
The learner’s ultimate goal is to minimize the cumulative loss suf-
fered along its run, which translates to making few prediction mistakesin the classiﬁcation case. The learner tries to deduce information fromprevious rounds so as to improve its predictions on present and futurequestions. Clearly, learning is hopeless if there is no correlation betweenpast and present rounds. Classic statistical theory of sequential predic-tion therefore enforces strong assumptions on the statistical propertiesof the input sequence (e.g., that it is sampled i.i.d. according to someunknown distribution).
In this review we survey methods which make no statistical assump-
tions regarding the origin of the sequence of examples. The sequence isallowed to be deterministic, stochastic, or even adversarially adaptiveto the learner’s own behavior (as in the case of spam email ﬁltering).Naturally, an adversary can make the cumulative loss to our online
110 Introduction
learning algorithm arbitrarily large. For example, the adversary can ask
the same question on each online round, wait for the learner’s answer,and provide the opposite answer as the correct answer. To make non-trivial statements we must further restrict the problem. We considertwo natural restrictions.
The ﬁrst restriction is especially suited to the case of online classi-
ﬁcation. We assume that all the answers are generated by some targetmapping, h
⋆:X→Y . Furthermore, h⋆is taken from a ﬁxed set, called
a hypothesis class and denoted by H, which is known to the learner.
With this restriction on the sequence, which we call the realizable case ,
the learner should make as few mistakes as possible, assuming thatbothh
⋆and the sequence of questions can be chosen by an adversary.
For an online learning algorithm, A, we denote by MA(H) the max-
imal number of mistakes Amight make on a sequence of examples
which is labeled by some h⋆∈H. We emphasize again that both h⋆
and the sequence of questions can be chosen by an adversary. A bound
onMA(H) is called a mistake-bound and we will study how to design
algorithms for which MA(H) is minimal.
Alternatively, the second restriction of the online learning model
we consider is a relaxation of the realizable assumption. We no longerassume that all answers are generated by some h
⋆∈H, but we require
the learner to be competitive with the best ﬁxed predictor from H. This
is captured by the regret of the algorithm, which measures how “sorry”
the learner is, in retrospect, not to have followed the predictions ofsome hypothesis h
⋆∈H. Formally, the regret of the algorithm relative
toh⋆when running on a sequence of Texamples is deﬁned as
RegretT(h⋆)=T/summationdisplay
t=1l(pt,yt)−T/summationdisplay
t=1l(h⋆(xt),yt), (1.1)
and the regret of the algorithm relative to a hypothesis class His
RegretT(H) = max
h⋆∈HRegretT(h⋆). (1.2)
We restate the learner’s goal as having the lowest possible regret
relative to H. We will sometime be satisﬁed with “low regret” algo-
rithms, by which we mean that RegretT(H) grows sub-linearly with
1.1 Examples 111
the number of rounds, T, which implies that the diﬀerence between the
average loss of the learner and the average loss of the best hypothesis
inHtends to zero as Tgoes to inﬁnity.
1.1 Examples
We already mentioned the problem of online classiﬁcation. To make the
discussion more concrete, we list several additional online predictionproblems and possible hypothesis classes.
Online Regression In regression problems, X=R
dwhich cor-
responds to a set of measurements (often called features), andY=D=R. For example, consider the problem of estimating the fetal
weight based on ultrasound measurements of abdominal circumferenceand femur length. Here, each x∈X=R
2is a two-dimensional vector
corresponds to the measurements of the abdominal circumference andthe femur length. Given these measurements the goal is to predict thefetal weight. Common loss functions for regression problems are thesquared loss, /lscript(p,y)=(p−y)
2, and the absolute loss, /lscript(p,y)=|p−y|.
Maybe the simplest hypothesis class for regression is the class of linearpredictors, H={x/mapsto→/summationtext
d
i=1w[i]x[i]:∀i,w[i]∈R}, where w[i]i st h e ith
element of w. The resulting problem is called online linear regression .
Prediction with Expert Advice On each online round the
learner has to choose from the advice of dgiven experts. Therefore,
xt∈X⊂ Rd, where xt[i] is the advice of the ith expert, and
D={1,...,d}. Then, the learner receives the true answer, which is a
vector yt∈Y=[ 0,1]d, where yt[i] is the cost of following the advice of
theith expert. The loss of the learner is the cost of the chosen expert,
/lscript(pt,yt)=yt[pt]. A common hypothesis class for this problem is the
set of constant predictors, H={h1,...,h d}, where hi(x)=ifor all x.
This implies that the regret of the algorithm is measured relative tothe performance of the strategies which always predict according tothe same expert.
Online Ranking On round t, the learner receives a query x
t∈X
and is required to order kelements (e.g., documents) according to
112 Introduction
their relevance to the query. That is, Dis the set of all permuta-
tions of {1,...,k}. Then, the learner receives the true answer yt∈Y=
{1,...,k}, which corresponds to the document which best matches the
query. In web applications, this is the document that the user clickedon. The loss, /lscript(p
t,yt), is the position of ytin the ranked list pt.
1.2 A Gentle Start
We start with studying online classiﬁcation problem, in which Y=D=
{0,1}, and /lscript(p,y)=|p−y|is the 0–1 loss. That is, on each round, the
learner receives xt∈Xand is required to predict pt∈{0,1}. Then, it
receives yt∈{0,1}and pays the loss |pt−yt|. We make the following
simplifying assumption:
•Finite Hypothesis Class: We assume that |H|<∞.
Recall that the goal of the learner is to have a low regret relative to
the hypotheses set, H, where each function in His a mapping from X
to{0,1}, and the regret is deﬁned as
RegretT(H) = max
h∈H/parenleftBiggT/summationdisplay
t=1|pt−yt|−T/summationdisplay
t=1|h(xt)−yt|/parenrightBigg
.
We ﬁrst show that this is an impossible mission — no algorithm
can obtain a sublinear regret bound even if |H|= 2. Indeed, consider
H={h0,h1}, where h0is the function that always returns 0 and h1is
the function that always returns 1. An adversary can make the numberof mistakes of any online algorithm to be equal to T, by simply waiting
for the learner’s prediction and then providing the opposite answer asthe true answer. In contrast, for any sequence of true answers, y
1,...,y T,
letbbe the majority of labels in y1,...,y T, then the number of mistakes
ofhbis at most T/2. Therefore, the regret of any online algorithm
might be at least T−T/2=T/2, which is not a sublinear with T.
This impossibility result is attributed to Cover [13].
To sidestep Cover’s impossibility result, we must further restrict the
power of the adversarial environment. In the following we present twoways to do this.
1.2 A Gentle Start 113
1.2.1 Realizability Assumption
The ﬁrst way to sidestep Cover’s impossibility result is by making one
additional assumption:
•Realizability: We assume that all target labels are generated
by some h⋆∈H, namely, yt=h⋆(xt) for all t. Our goal is to
design an algorithm with an optimal mistake bound. Namely,an algorithm for which M
A(H) is minimal. See deﬁnition of
MA(H) in the prequel.
Next, we describe and analyze online learning algorithms assuming
both a ﬁnite hypothesis class and realizability of the input sequence.The most natural learning rule is to use (at any online round) anyhypothesis which is consistent with all past examples.
Consistent
input: A ﬁnite hypothesis class H
initialize :V1=H
fort=1,2,...
receive xt
choose any h∈Vt
predict pt=h(xt)
receive true answer yt=h⋆(xt)
update Vt+1={h∈Vt:h(xt)=yt}
The Consistent algorithm maintains a set, Vt, of all the hypothe-
ses which are consistent with ( x1,y1),...,(xt−1,yt−1). This set is often
called the version space. It then picks any hypothesis from Vtand
predicts according to this hypothesis.
Obviously, whenever Consistent makes a prediction mistake, at
least one hypothesis is removed from Vt. Therefore, after making M
mistakes we have |Vt|≤| H |− M. Since Vtis always nonempty (by the
realizability assumption it contains h⋆)w eh a v e1 ≤|Vt|≤| H |− M.
114 Introduction
Rearranging, we obtain
Corollary 1.1. LetHbe a ﬁnite hypothesis class. The Consistent
algorithm enjoys the mistake bound MConsistent (H)≤| H |− 1.
It is rather easy to construct a hypothesis class and a sequence of
examples on which Consistent will indeed make |H| − 1 mistakes.
Next, we present a better algorithm in which we choose h∈Vtin a
smarter way. We shall see that this algorithm is guaranteed to makeexponentially fewer mistakes. The idea is to predict according to themajority of hypotheses in V
trather than according to some arbitrary
h∈Vt. That way, whenever we err, we are guaranteed to remove at
least half of the hypotheses from the version space.
Halving
input: A ﬁnite hypothesis class H
initialize :V1=H
fort=1,2,...
receive xt
predict pt= argmaxr∈{0,1}|{h∈Vt:h(xt)=r}|
(in case of a tie predict pt=1 )
receive true answer yt
update Vt+1={h∈Vt:h(xt)=yt}
Theorem 1.2. LetHbe a ﬁnite hypothesis class. The Halving algo-
rithm enjoys the mistake bound MHalving (H)≤log2(|H|).
Proof. We simply note that whenever the algorithm errs we have
|Vt+1|≤|Vt|/2. (Hence the name Halving.) Therefore, if Mis the total
number of mistakes, we have
1≤|VT+1|≤| H | 2−M.
Rearranging the above inequality we conclude our proof.
Of course, Halving ’s mistake bound is much better than
Consistent ’s mistake bound. Is this the best we can do? What is an
1.2 A Gentle Start 115
optimal algorithm for a given hypothesis class (not necessarily ﬁnite)?
We will get back to this question in Section 3.
1.2.2 Randomization
In the previous subsection we sidestepped Cover’s impossibility result
by relying on the realizability assumption. This is a rather strongassumption on the environment. We now present a milder restriction onthe environment and allow the learner to randomize his predictions. Ofcourse, this by itself does not circumvent Cover’s impossibility result asin deriving the impossibility result we assumed nothing on the learner’sstrategy. To make the randomization meaningful, we force the adver-sarial environment to decide on y
twithout knowing the random coins
ﬂipped by the learner on round t. The adversary can still know the
learner’s forecasting strategy and even the random bits of previousrounds, but it doesn’t know the actual value of the random bits usedby the learner on round t. With this (mild) change of game, we analyze
theexpected 0–1 loss of the algorithm, where expectation is with respect
to the learner’s own randomization. That is, if the learner outputs ˆ y
t
where P[ˆyt=1 ]= pt, then the expected loss he pays on round tis
P[ˆyt/negationslash=yt]=|pt−yt|.
Put another way, instead of having the predictions domain being
D={0,1}we allow it to be D=[ 0,1], and interpret pt∈Das the
probability to predict the label 1 on round t. To summarize, we assume:
•Randomized Predictions and Expected Regret : We allow the
predictions domain to be D=[ 0,1] and the loss function is
stilll(pt,yt)=|pt−yt|.
With this assumption it is possible to derive a low regret algorithm
as stated in the following theorem.
Theorem 1.3. LetHbe a ﬁnite hypothesis class. There exists an algo-
rithm for online classiﬁcation, whose predictions come from D=[ 0,1],
116 Introduction
that enjoys the regret bound
T/summationdisplay
t=1|pt−yt|−min
h∈HT/summationdisplay
t=1|h(xt)−yt|≤/radicalbig
0.5ln(|H|)T.
We will provide a constructive proof of the above theorem in the next
section.
To summarize, we have presented two diﬀerent ways to sidestep
Cover’s impossibility result: realizability or randomization. At ﬁrstglance, the two approaches seem to be rather diﬀerent. However, thereis a deep underlying concept that connects them. Indeed, we will showthat both methods can be interpreted as convexiﬁcation techniques.
Convexity is a central theme in deriving online learning algorithms.We study it in the next section.
1.3 Organization and Scope
How to predict rationally is a key issue in various research areas suchas game theory, machine learning, and information theory. The semi-nal book of Cesa-Bianchi and Lugosi [12] thoroughly investigates theconnections between online learning, universal prediction, and repeatedgames. In particular, results from the diﬀerent ﬁelds are uniﬁed usingthe prediction with expert advice framework.
We feel that convexity plays a central role in the derivation of online
learning algorithms, and therefore start the survey with a study of theimportant sub-family of online learning problems, which is called online
convex optimization . In this family, the prediction domain is a convex
set and the loss function is a convex function with respect to its ﬁrstargument. As we will show, many previously proposed algorithms foronline classiﬁcation and other problems can be jointly analyzed basedon the online convex optimization framework. Furthermore, convexityis important because it leads to eﬃcient algorithms.
In Section 3 we get back to the problem of online classiﬁcation.
We characterize a standard optimal algorithm for online classiﬁcation.In addition, we show how online convex optimization can be used forderiving eﬃcient online classiﬁcation algorithms.
In Section 4 we study online learning in a limited feedback model,
when the learner observes the loss value l(p
t,yt) but does not observe
1.4 Notation and Basic Deﬁnitions 117
the actual correct answer yt. We focus on the classic multi-armed ban-
dit problem and derive an algorithm for this problem based on theonline convex optimization algorithmic framework. We also present alow regret algorithm for the general problem of bandit online convexoptimization.
Finally, in Section 5 we discuss several implications of online learn-
ing to batch learning problems, in which we assume that the examplesare sampled i.i.d. from an unknown probability source.
Part of our presentation shares similarities with other surveys on
online prediction problems. In particular, Rakhlin’s lecture notes [34]and Hazan’s book section [22] are good recent surveys on onlineconvex optimization. While part of our presentation shares similari-ties with these surveys, we sometimes emphasize diﬀerent techniques.Furthermore, we connect and relate the new results on online convexoptimization to classic results on online classiﬁcation, thus providinga fresh modern perspective on some classic algorithms. A more classictreatment can be found in Blum’s survey [8].
1.4 Notation and Basic Deﬁnitions
We denote scalars with lower case letters (e.g., xandλ), and vectors
with bold face letters (e.g., xandλ). The ith element of a vector x
is denoted by x[i]. Since online learning is performed in a sequence
of rounds, we denote by xtthetth vector in a sequence of vectors
x1,x2,...,xT. The ith element of xtis denoted by xt[i].
Theinner product between vectors xandwis denoted by /angbracketleftx,w/angbracketright.
Whenever we do not specify the vector space we assume that it is thed-dimensional Euclidean space and then /angbracketleftx,w/angbracketright=/summationtext
d
i=1x[i]w[i]. Sets
are designated by upper case letters (e.g., S). The set of real numbers
is denoted by Rand the set of non-negative real numbers is denoted
byR+. The set of natural numbers is denoted by N. For any k≥1,
the set of integers {1,...,k}is denoted by [ k]. Given a predicate π,w e
use the notation 1[π]to denote the indicator function that outputs 1
ifπholds and 0 otherwise. The hinge function is denoted by [ a]+=
max{0,a}.
The Euclidean (or /lscript2)norm of a vector wis/bardblw/bardbl2=/radicalbig
/angbracketleftw,w/angbracketright.W e
omit the subscript when it is clear from the context. We also use other /lscriptp
118 Introduction
norms, /bardblw/bardblp=(/summationtext
i|w[i]|p)1/p, and in particular /bardblw/bardbl1=/summationtext
i|w[i]|and
/bardblw/bardbl∞= max i|w[i]|. A generic norm of a vector wis denoted by /bardblw/bardbl
and its dual norm is deﬁned as
/bardblx/bardbl⋆= max {/angbracketleftw,x/angbracketright:/bardblw/bardbl≤1}.
The deﬁnition of the dual norm immediately implies the inequality
/angbracketleftw,z/angbracketright≤/bardbl w/bardbl/bardblz/bardbl⋆. (1.3)
For the /lscript2norm (which is dual to itself), this is the well known Cauchy–
Schwartz inequality. For p,q≥1 such that1
p+1
q= 1 we have that
the/lscriptpand/lscriptqnorms are dual, and Equation (1.3) becomes Holder’s
inequality.
A function fis called L-Lipschitz over a set Swith respect to a
norm /bardbl·/bardblif for all u,w∈Swe have |f(u)−f(w)|≤L/bardblu−w/bardbl.
The gradient of a diﬀerentiable function fis denoted by ∇fand
theHessian is denoted by ∇2f.
Throughout the review, we make use of basic notions from convex
analysis. A set Sisconvex if for all w,v∈Sandα∈[0,1] we have
thatαw+( 1−α)v∈Sas well. Similarly, a function f:S→Ris con-
vex if for all w,vandα∈[0,1] we have f(αw+( 1−α)v)≤αf(w)+
(1−α)f(v).
It is convenient to allow convex functions to output the value ∞.
The domain of a function fis the set of points on which fis ﬁnite.
This is convenient, for example, for constraining the solution of an opti-mization problem to be within some set A. Indeed, instead of solving
min
x∈Af(x) we can solve min xf(x)+IA(x), where IAis the function
that outputs 0 if x∈Aand∞ifx/∈A. In the next section we make
use of some additional deﬁnitions and tools from convex analysis. Forclarity, we deﬁne them as per need.
The expected value of a random variable, ψ, is denoted by E[ψ].
In some situations, we have a deterministic function hthat receives a
random variable as input. We denote by E[h(ψ)] the expected value of
the random variable h(ψ). Occasionally, we omit the dependence of h
onψ. In this case, we may clarify the meaning of the expectation by
using the notation E
ψ[h]o r Eψ∼P[h]i fψis distributed according to
some distribution P.
2
Online Convex Optimization
In recent years, the design of many eﬃcient online learning algorithms
has been inﬂuenced by convex optimization tools. Furthermore, it wasobserved that most previously proposed eﬃcient algorithms can bejointly analyzed based on the following elegant model:
Online Convex Optimization (OCO)
input: A convex set S
fort=1,2,...
predict a vector wt∈S
receive a convex loss function ft:S→R
suﬀer loss ft(wt)
In this section we describe algorithms for online convex optimization
and analyze their regret. Recall that the regret of an online algorithmwith respect to a competing hypothesis, which here will be some vectoru, is deﬁned as
Regret
T(u)=T/summationdisplay
t=1ft(wt)−T/summationdisplay
t=1ft(u). (2.1)
119
120 Online Convex Optimization
As before, the regret of the algorithm relative to a set of competing
vectors, U, is deﬁned as
RegretT(U) = max
u∈URegretT(u).
Remark 2.1. ( Uvs.S)In the online convex optimization problem,
the predictions of the learner should come from the set S, while we
analyze the regret with respect to the set U. While in some situations
it makes sense to set U=S, this is not always the case. Whenever we do
not specify the value of Uwe use the default value U=S. Additionally,
our default setting for Swill be S=Rd.
The rest of this section is organized as follows. We start with
convexiﬁcation techniques, showing how to utilize the online convexoptimization framework in nonconvex problems. Next, we startdescribing and analyzing an algorithmic framework for online convexoptimization. First, we describe the Follow-the-Leader approach, inwhich the learner simply picks the vector which performed best onpast rounds. Next we describe a regularized form of Follow-the-Leader,which stabilizes the predictions of the algorithm, and show how stabilityleads to low regret. We proceed with deriving Online Gradient Descentand Online Mirror Descent from Follow-the-Regularized-Leader bya linearization trick. We derive several speciﬁc algorithms from theOnline Mirror Descent framework. Finally, we describe additional prooftechniques and also derive local-norm bounds, that will be used in thenext sections.
2.1 Convexiﬁcation
Some online prediction problems can be seamlessly cast in the onlineconvex optimization framework.
Example 2.1 (Online linear regression). Recall the online linear
regression problem described in Section 1.1. On each online round thelearner ﬁrst receives a vector of features, x
t∈A⊂Rd, and then needs to
predict a scalar, pt. Next, the learner receives the “true” target, yt∈R,
and pays the loss |pt−yt|. The learner should be competitive with the
2.1 Convexiﬁcation 121
set of linear predictors of the form x/mapsto→/angbracketleftw,x/angbracketright. If the predictions of
the learner are also based on linear functions, then we can easily castthis online prediction problem in the online convex optimization frame-work as follows. The learner should decide on a vector w
t, which yields
the prediction pt=/angbracketleftwt,xt/angbracketright. The loss function becomes |pt−yt|=
|/angbracketleftwt,xt/angbracketright−yt|. Therefore, letting ft(w)=|/angbracketleftw,xt/angbracketright−yt|, which is indeed
a convex function, we obtain that ft(wt)=l(pt,yt).
Other online prediction problems do not seem to ﬁt into the online
convex optimization framework. For example, in online classiﬁcationproblems, the predictions domain or the loss functions are not convex.In this section we describe two convexiﬁcation techniques that allowus to utilize the online convex optimization framework in additionalscenarios.
2.1.1 Convexiﬁcation by Randomization
To demonstrate the randomization technique, consider the problem
of prediction with expert advice, where on each online round thelearner has to choose from the advice of dgiven experts. Denote by
p
t∈{1,...,d}the chosen expert. Then, the learner receives a vector
yt∈[0,1]d, where yt[i] is the cost of following the advice of the ith
expert. The learner pays the loss yt[pt]. In this prediction problem, the
decision space is discrete, hence nonconvex.
Furthermore, the problem of online classiﬁcation with a ﬁnite
hypothesis class we encountered in Section 1.2 can be easily cast asa special case of the prediction with expert advice problem. Therefore,Cover’s impossibility result (see again Section 1.2) implies that there isno algorithm that can attain low regret for the prediction with expertadvice problem.
However, as we show below, by allowing the learner to random-
ize his predictions we can cast the problem in the online convex opti-mization framework, and therefore can obtain low regret algorithm forthis problem. Formally, let S={w∈R
d:w≥0∧/bardblw/bardbl1=1}be the
probability simplex, which forms a convex set. At round t, the learner
chooses wt∈Sand based on wtpicks an expert at random according
122 Online Convex Optimization
toP[pt=i]=wt[i]. Then, the cost vector ytis revealed and the learner
pays for his expected cost
E[yt[pt]] =d/summationdisplay
i=1P[pt=i]yt[i]=/angbracketleftwt,yt/angbracketright.
Note that by analyzing the expected cost of the learner we implicitly
restrict the power of the adversarial environment — it cannot base thevector y
ton the random bits the learner employs on round t.
Now we can cast the problem as online convex optimization since
Sis a convex set and the loss function, ft(w)=/angbracketleftw,yt/angbracketright, happens to be
a linear function (hence, convex). Let the set of competing vectors, U,
be the dsingletons, namely the vectors of the form (0 ,...,0,1,0,...,0).
These vectors correspond to always following the advice of a singleexpert. Hence, a regret bound with respect to Uimplies a regret bound
with respect to always predicting the advice of a single expert.
2.1.2 Convexiﬁcation by Surrogate Loss Functions
To explain the second convexiﬁcation technique we again start with
the speciﬁc problem of online classiﬁcation with a ﬁnite hypothesisclass. Recall that one of the techniques we used to sidestep Cover’simpossibility result relied on the realizability assumption. That is, weassumed that there exists h
⋆∈Hsuch that yt=h⋆(xt) for all t. With
this assumption at hand, we described the Halving algorithm and
showed that it makes at most log2(|H|) prediction mistakes.
We now derive a similar guarantee using the language of online
convex optimization. Let us write H={h1,...,h d}and let S={w∈
[0,1]d:/summationtext
iw[i]=1}be the probability simplex. For each online round,
deﬁne vt=(h1(xt),...,h d(xt))∈{0,1}d. Our algorithm will maintain
wt∈Sand will predict the label according to
pt=/braceleftbigg1i f/angbracketleftwt,vt/angbracketright≥1/2
0i f/angbracketleftwt,vt/angbracketright<1/2
LetM={t:pt/negationslash=yt}be the rounds on which our algorithm makes a
prediction mistake. We deﬁne
ft(w)=/braceleftbigg2|/angbracketleftw,vt/angbracketright−yt|ift∈M
0i f t/∈M.
2.1 Convexiﬁcation 123
Note that ftdepends on M, and thus depends on wt. This does not
pose any problem since in the online convex optimization model theenvironment picks the function f
tafter observing wt. The two key
properties of ftare
•ftis a convex function
•ft(wt)≥|pt−yt|, namely, the convex loss upper bounds the
original nonconvex loss.
Hence the name surrogate convex loss . Since Sis a convex set and
ftis a convex function for all twe have obtained an online convex
optimization problem.
In the next sections we will derive algorithms for online convex
optimization problems. In particular, one of these algorithms enjoysthe regret bound
∀u∈S, T/summationdisplay
t=1ft(wt)≤T/summationdisplay
t=1ft(u)+log(d)
η+2ηT/summationdisplay
t=1Lt,
where ηis a parameter, which we will set here to be η=1/4, and Ltis
a Lipschitz parameter of the function ft(with respect to the /lscript1norm).
In our case, Lt=1i f t∈MandLt=0i f t/∈M. Hence,
∀u∈S,T/summationdisplay
t=1ft(wt)≤T/summationdisplay
t=1ft(u) + 4log( d)+1
2|M|.
By the surrogate property of ft, we can lower bound the left-hand side
by|M|. Rearranging, we obtain:
|M| ≤ 2T/summationdisplay
t=1ft(u) + 8log( d).
This type of bound, where the number of mistakes is upper bounded
by the convex surrogate loss of a competing hypothesis, is often calledarelative loss bound .
In the realizable case, we can further simplify the relative loss bound
as follows. Since the bound holds for all u∈Sit holds in particu-
lar for the vector u=( 0,...,0,1,0,...,0), where the 1 is placed in the
124 Online Convex Optimization
coordinate corresponding to the true hypothesis h⋆. By our construc-
tion,ft(u) = 0 for all t, which yields
|M| ≤ 8log(d).
We have obtained a mistake bound of the same order as the Halving ’s
mistake bound.
More generally, the ﬁrst step of the technique involves a re-
parameterization of the problem such that the decision space becomesconvex (instead of maintaing the set V
tinHalving we now maintain
the vector wt∈S). In the second step we construct a function ftof
the predicted parameter that satisﬁes two requirements: It should beconvex and it should upper bound the original loss function. Last, wewould of course like to construct a convex surrogate for which thereexists some u∈Sthat attains a low cumulative loss. Otherwise, the
resulting bound will be meaningless. Typically, this is done by assumingmore on the problem at hand. For example, in the above, the realiz-ability assumption enabled us to construct a surrogate for which therewasu∈Ssuch that f
t(u) = 0 for all t.
2.2 Follow-the-leader
By now, we hope that the reader is convinced that the online convex
optimization framework is an important model, so we turn to derivingalgorithms for online convex optimization.
The most natural learning rule is to use (at any online round) any
vector which has minimal loss on all past rounds. This is in the samespirit of the Consistent algorithm we encountered in Section 1.2 but
in the context of online convex optimization it is usually referred to asFollow-The-Leader .
Follow-The-Leader (FTL)
∀t,wt= argmin
w∈St−1/summationdisplay
i=1fi(w)(break ties arbitrarily)
To analyze FTL, we ﬁrst show that the regret of FTL is upper
bounded by the cumulative diﬀerence between the loss of wtandwt+1.
2.2 Follow-the-leader 125
Lemma 2.1. Letw1,w2,...be the sequence of vectors produced by
FTL. Then, for all u∈Swe have
RegretT(u)=T/summationdisplay
t=1(ft(wt)−ft(u))≤T/summationdisplay
t=1(ft(wt)−ft(wt+1)).
Proof. Subtracting/summationtext
tft(wt) from both sides of the inequality and
rearranging, the desired inequality can be rewritten as
T/summationdisplay
t=1ft(wt+1)≤T/summationdisplay
t=1ft(u).
We prove this inequality by induction. The base case of T= 1 follows
directly from the deﬁnition of wt+1. Assume the inequality holds for
T−1, then for all u∈Swe have
T−1/summationdisplay
t=1ft(wt+1)≤T−1/summationdisplay
t=1ft(u).
Adding fT(wT+1) to both sides we get
T/summationdisplay
t=1ft(wt+1)≤fT(wT+1)+T−1/summationdisplay
t=1ft(u).
The above holds for all uand in particular for u=wT+1. Thus,
T/summationdisplay
t=1ft(wt+1)≤T/summationdisplay
t=1ft(wT+1) = min
u∈ST/summationdisplay
t=1ft(u),
where the last equation follows from the deﬁnition of wT+1. This con-
cludes our inductive argument.
We next use Lemma 2.1 to derive a regret bound for the following
sub-family of online convex optimization.
Deﬁnition 2.1 (Online Quadratic Optimization). This is an
online convex optimization problem where at each round ft(w)=
1
2/bardblw−zt/bardbl2
2for some vector zt.
126 Online Convex Optimization
We further assume that S=Rd. For this case, it is easy to verify that
the FTL rule becomes
wt=1
t−1t−1/summationdisplay
i=1zi,
namely, wtis the average of z1,...,zt−1. Note that we can rewrite
wt+1=1
t(zt+(t−1)wt)=/parenleftbigg
1−1
t/parenrightbigg
wt+1
tzt
which yields
wt+1−zt=/parenleftbigg
1−1
t/parenrightbigg
(wt−zt).
Therefore,
ft(wt)−ft(wt+1)=1
2/bardblwt−zt/bardbl2−1
2/bardblwt+1−zt/bardbl2
=1
2/parenleftBigg
1−/parenleftbigg
1−1
t/parenrightbigg2/parenrightBigg
/bardblwt−zt/bardbl2
≤1
t/bardblwt−zt/bardbl2.
LetL= max t/bardblzt/bardbl. Since wtis the average of z1,...,zt−1we have that
/bardblwt/bardbl≤Land therefore, by the triangle inequality, /bardblwt−zt/bardbl≤2L.W e
have therefore obtained:
T/summationdisplay
t=1(ft(wt)−ft(wt+1))≤(2L)2T/summationdisplay
t=11
t.
Combining the above with Lemma 2.1 and using the inequality/summationtextT
t=11/t≤log(T) + 1 we conclude that
Corollary 2.2. Consider running FTL on an Online Quadratic Opti-
mization problem with S=Rdand let L= max t/bardblzt/bardbl. Then, the regret
of FTL with respect to all vectors u∈Rdis at most 4 L2(log(T) + 1).
While the above result seems promising, we next show that the FTL
rule does not guarantee low regret for another important sub-family.
2.3 Follow-the-Regularized-Leader 127
Deﬁnition 2.2 (Online Linear Optimization). This is an online
convex optimization problem where at each round ft(w)=/angbracketleftw,zt/angbracketrightfor
some vector zt.
Example 2.2 (Failure of FTL). LetS=[−1,1]⊂Rand consider
the sequence of linear functions such that ft(w)=ztwwhere
zt=

−0.5i ft=1
1i f tis even
−1i f t>1∧tis odd
Then, the predictions of FTL will be to set wt= 1 for todd and wt=−1
forteven. The cumulative loss of the FTL algorithm will therefore be T
while the cumulative loss of the ﬁxed solution u=0∈Sis 0. Thus, the
regret of FTL is T!
Intuitively, FTL fails in the above example because its predictions
arenot stable —wtshifts drastically from round to round where we
only added a single loss function to the objective of the optimizationproblem in the deﬁnition of w
t. In contrast, FTL works ﬁne for the
quadratic game since wt+1is “close” to wt. One way to stabilize FTL
is by adding regularization, which is the topic of the next section.
2.3 Follow-the-Regularized-Leader
Follow-the-Regularized-Leader is a natural modiﬁcation of the basicFTL algorithm in which we minimize the loss on all past rounds plusa regularization term. The goal of the regularization term is to stabi-lize the solution. Formally, for a regularization function, R:S→Rwe
deﬁne
Follow-the-Regularized-Leader (FoReL)
∀t,wt= argmin
w∈St−1/summationdisplay
i=1fi(w)+R(w)
128 Online Convex Optimization
Naturally, diﬀerent regularization functions will yield diﬀerent algo-
rithms with diﬀerent regret bounds. We discuss properties of diﬀerentregularization functions later. But, ﬁrst, let us specify FoReL for thecase of linear functions and squared- /lscript
2-norm regularization, which we
often call the Euclidean regularization case.
Example 2.3. Consider the Online Linear Optimization problem
where ft(w)=/angbracketleftw,zt/angbracketrightand let S=Rd. Suppose we run FoReL with
the regularization function R(w)=1
2η/bardblw/bardbl2
2for some positive scalar η.
Then, it is easy to verify that
wt+1=−ηt/summationdisplay
i=1zi=wt−ηzt. (2.2)
Note that ztis the gradient of ftatwt(in fact, at any point). Therefore,
the recursive rule, wt+1=wt−ηzt, can be rewritten as wt+1=wt−
η∇ft(wt). Hence, this rule is often called Online Gradient Descent.
We shall re-visit the Online Gradient Descent rule for general convexfunctions in the next section.
We next turn to the analysis of FoReL. As with the analysis of FTL,
we ﬁrst relate the regret of FoReL to the cumulative diﬀerence betweenthe loss of w
tandwt+1.
Lemma 2.3. Letw1,w2,...be the sequence of vectors produced by
FoReL. Then, for all u∈Swe have
T/summationdisplay
t=1(ft(wt)−ft(u))≤R(u)−R(w1)+T/summationdisplay
t=1(ft(wt)−ft(wt+1)).
Proof. Observe that running FoReL on f1,...,f Tis equivalent to run-
ning FTL on f0,f1,...,f Twhere f0=R. Using Lemma 2.1 we obtain
T/summationdisplay
t=0(ft(wt)−ft(u))≤T/summationdisplay
t=0(ft(wt)−ft(wt+1)).
Rearranging the above and using f0=Rwe conclude our proof.
2.3 Follow-the-Regularized-Leader 129
Based on the above lemma we can easily derive a regret bound for
online linear optimization with the regularizer R(w)=1
2η/bardblw/bardbl2
2.
Theorem 2.4. Consider running FoReL on a sequence of linear func-
tions, ft(w)=/angbracketleftw,zt/angbracketrightfor all t, with S=Rd, and with the regularizer
R(w)=1
2η/bardblw/bardbl2
2, which yields the predictions given in Equation (2.2).
Then, for all uwe have
RegretT(u)≤1
2η/bardblu/bardbl2
2+ηT/summationdisplay
t=1/bardblzt/bardbl2
2.
In particular, consider the set U={u:/bardblu/bardbl≤B}and let Lbe such
that1
T/summationtextT
t=1/bardblzt/bardbl2
2≤L2, then by setting η=B
L√
2Twe obtain
RegretT(U)≤BL√
2T.
Proof. Using Lemma 2.3 and Equation (2.2),
RegretT(u)≤R(u)−R(w1)+T/summationdisplay
t=1(ft(wt)−ft(wt+1))
=1
2η/bardblu/bardbl2
2+T/summationdisplay
t=1/angbracketleftwt−wt+1,zt/angbracketright
=1
2η/bardblu/bardbl2
2+ηT/summationdisplay
t=1/bardblzt/bardbl2
2.
The parameter ηin the above theorem depends on the time hori-
zonT. It is possible to derive a similar result without using the time
horizon. In the next subsection we show a generic way (although notalways optimal) to get rid of the dependence on the time horizon.
We see that the Euclidean regularization function guarantees low
regret for linear functions with bounded /lscript
2-norm because it stabi-
lizes the predictions. We shall later generalize the above result in twoaspects. First, we allow any sequence of Lipschitz functions (ratherthan linear functions with bounded norm). Second, we consider otherregularization functions which guarantee stability in other scenarios.
130 Online Convex Optimization
2.3.1 The Doubling Trick
In Theorem 2.4, the parameter ηdepends on the time horizon T.W e
now show how to get rid of this dependence by a simple trick.
Consider an algorithm that enjoys a regret bound of the form α√
T,
but its parameters require the knowledge of T. The doubling trick,
described below, enables us to convert such an algorithm into an algo-rithm that does not need to know the time horizon. The idea is to dividethe time into periods of increasing size and run the original algorithmon each period.
The Doubling Trick
input: algorithm Awhose parameters depend on the time horizon
form=0,1,2,...
runAon the 2mrounds t=2m,...,2m+1−1
The regret of Aon each period of 2mrounds is at most α√
2m.
Therefore, the total regret is at most
⌈log2(T)⌉/summationdisplay
m=1α√
2m=α⌈log2(T)⌉/summationdisplay
m=1(√
2)m
=α1−√
2⌈log2(T)⌉+1
1−√
2
≤α1−√
2T
1−√
2
≤√
2√
2−1α√
T.
That is, we obtain that the regret is worse by a constant multiplicative
factor.
2.4 Online Gradient Descent: Linearization
of Convex Functions
In the previous section we introduced the FoReL approach and analyzed
it for the case of linear functions, S=Rd, and Euclidean regularization.
2.4 Online Gradient Descent: Linearization of Convex Functions 131
We now generalize this result by deriving a simple reduction from
convex functions to linear functions.
To do so, we use an important property of convex functions, which
is in fact an alternative characterization of convexity, as given by thefollowing lemma.
Lemma 2.5. LetSbe a convex set. A function f:S→Ris convex
iﬀ for all w∈Sthere exists zsuch that
∀u∈S, f (u)≥f(w)+/angbracketleftu−w,z/angbracketright. (2.3)
In words, convexity is characterized by the existence of tangents
that lie below the function. The proof of this lemma can be found inmany convex analysis textbooks (e.g., [9]).
Deﬁnition 2.3 (sub-gradients). A vector zthat satisﬁes Equa-
tion (2.3) is called a sub-gradient offatw. The set of sub-gradients of
fatwis denoted ∂f(w). Furthermore, if fis diﬀerentiable at wthen
∂f(w) contains a single element — the gradient of fatw,∇f(w).
An illustration of sub-gradients is given in Figure 2.1.
Getting back to online convex optimization, for each round t, there
exists ztsuch that for all u,
ft(wt)−ft(u)≤/angbracketleftwt−u,zt/angbracketright.
Fig. 2.1 Left: The right-hand side of Equation (2.3) is the tangent of fatw. For a convex
function, the tangent lower bounds f. Right: Illustration of several sub-gradients of a non-
diﬀerentiable convex function.
132 Online Convex Optimization
It follows that for any sequence of convex functions f1,...,f Tand vec-
tors w1,...,wT, if for all t,zt∈∂ft(wt) (namely, it is a sub-gradient)
then
T/summationdisplay
t=1(ft(wt)−ft(u))≤T/summationdisplay
t=1(/angbracketleftwt,zt/angbracketright−/angbracketleft u,zt/angbracketright). (2.4)
In words, the regret of a procedure that generates the vectors
w1,...,wTfor the sequence of linear functions upper bounds the regret
with respect to the convex functions f1,...,f T.
Note that in this construction, ztdepends on wt. As mentioned pre-
viously, this does not pose any problem since we allow the adversarialenvironment to base its loss function on the vector predicted by thelearner.
Combining the above observation with the FoReL procedure with
Euclidean regularization (see Equation (2.2)) yields the Online Gradi-ent Descent algorithm:
Online Gradient Descent (OGD)
parameter: η>0
initialize: w 1=0
update rule: w t+1=wt−ηztwhere zt∈∂ft(wt)
To analyze OGD, we combine Equation (2.4) with the analysis of
FoReL for linear functions given in Theorem 2.4, to get that
RegretT(u)≤1
2η/bardblu/bardbl2
2+ηT/summationdisplay
t=1/bardblzt/bardbl2
2. (2.5)
This regret bound depends on the norms of the sub-gradients of the
vectors produced by the algorithm, and is therefore not satisfactory.To derive a more concrete bound, we must assure that the norms ofsub-gradients will not be excessively large. One way to do this is byassuming that the functions are Lipschitz. The following lemma relatesnorms of sub-gradients to Lipschitzness of f
t.
2.4 Online Gradient Descent: Linearization of Convex Functions 133
Lemma 2.6. Letf:S→Rbe a convex function. Then, fisL-
Lipschitz over Swith respect to a norm /bardbl·/bardbliﬀ for all w∈Sand
z∈∂f(w) we have that /bardblz/bardbl⋆≤L, where /bardbl·/bardbl⋆is the dual norm.
Proof. Assume that fis Lipschitz. Choose some w∈S,z∈∂f(w).
Let ube such that u−w= argmaxv:/bardblv/bardbl=1/angbracketleftv,z/angbracketright. Therefore, /angbracketleftu−
w,z/angbracketright=/bardblz/bardbl⋆. From the deﬁnition of the sub-gradient,
f(u)−f(w)≥/angbracketleftz,u−w/angbracketright=/bardblz/bardbl⋆.
On the other hand, from the Lipschitzness of fwe have
L=L/bardblu−w/bardbl≥f(u)−f(w).
Combining the above two inequalities we conclude that /bardblz/bardbl⋆≤L.F o r
the other direction, since z∈∂f(w) we also have
f(w)−f(u)≤/angbracketleftz,w−u/angbracketright.
Combining the above with Equation (1.3) we obtain
f(w)−f(u)≤/bardblz/bardbl⋆/bardblw−u/bardbl≤L/bardblw−u/bardbl,
hence fisL-Lipschitz.
Therefore, the term/summationtextT
t=1/bardblzt/bardbl2
2given in Equation (2.5) can be
bounded by/summationtextT
t=1L2
t, where Ltis the Lipschitz constant of ft.W e
conclude:
Corollary 2.7. Assume that OGD is run on a sequence f1,...,f Tof
convex functions. Then, for all uwe have
RegretT(u)≤1
2η/bardblu/bardbl2
2+ηT/summationdisplay
t=1/bardblzt/bardbl2
2.
If we further assume that each ftisLt-Lipschitz with respect to /bardbl·/bardbl2,
and let Lbe such that1
T/summationtextT
t=1L2
t≤L2. Then, for all u, the regret of
OGD satisﬁes
RegretT(u)≤1
2η/bardblu/bardbl2
2+ηTL2.
134 Online Convex Optimization
In particular, if U={u:/bardblu/bardbl2≤B}andη=B
L√
2Tthen
RegretT(U)≤BL√
2T.
Let us now discuss the consequences of Corollary 2.7, starting with
the online linear regression problem (Example 2.1). Recall that for thisexample, f
t(w)=|/angbracketleftw,xt/angbracketright−yt|, where xtcomes from a set A. If the
setAis contained in a ball of /lscript2radius L, then ftisL-Lipschitz with
respect to the /lscript2norm. We therefore obtain a regret bound of BL√
2T
which holds for all competing vectors uwith/bardblu/bardbl2≤B.
For the problem of prediction with expert advice (see Section 2.1.1),
we cannot apply the OGD framework as it does not guarantee that wt
will always be in the probability simplex. In the next section we describe
the FoReL framework with other regularization functions. In particu-lar, we present regularization functions appropriate for the problem ofprediction with expert advice.
2.5 Strongly Convex Regularizers
So far we applied FoReL with the Euclidean regularization function. Asmentioned at the end of the previous section, this regularization can-not be used for the learning with expert advice problem. In this sectionwe consider other regularization functions and underscore strong con-vexity as an important property of regularization functions that yieldsmeaningful regret bounds.
2.5.1 Strong Convexity
Intuitively, a function is strongly convex if it grows faster than a linear
function. To give a precise deﬁnition, recall that for a convex func-tionf, at any point wwe can ﬁnd a linear function (the “tangent”)
which equals to fatwand does not exceed fat any other point (see
Lemma 2.5). A function is strongly convex if fis strictly above the
2.5 Strongly Convex Regularizers 135
tangent, and the diﬀerence can be quantiﬁed as follows:
Deﬁnition 2.4. A function f:S→Risσ-strongly-convex over Swith
respect to a norm /bardbl·/bardblif for any w∈Swe have
∀z∈∂f(w),∀u∈S, f (u)≥f(w)+/angbracketleftz,u−w/angbracketright+σ
2/bardblu−w/bardbl2.
A graphical illustration is given in Figure 2.2.
An important property of strong convexity that we use is the fol-
lowing:
Lemma 2.8. LetSbe a nonempty convex set. Let f:S→Rbe a
σ-strongly-convex function over Swith respect to a norm /bardbl·/bardbl. Let w=
argminv∈Sf(v). Then, for all u∈S
f(u)−f(w)≥σ
2/bardblu−w/bardbl2.
Proof. To give intuition, assume ﬁrst that fis diﬀerentiable and wis
in the interior of S. Then, ∇f(w)=0and therefore, by the deﬁnition
of strong convexity we have
∀u∈S, f(u)−f(w)≥/angbracketleft ∇f(w),u−w/angbracketright+σ
2/bardblu−w/bardbl2=σ
2/bardblu−w/bardbl2,
Fig. 2.2 Illustrating strong-convexity: the distance between fand its tangent at wis at
leastσ
2/bardblu−w/bardbl2.
136 Online Convex Optimization
as required. Even if wis on the boundary of Swe still have that for all
u∈S,/angbracketleft∇f(w),u−w/angbracketright≥0 (otherwise, wwould not have been optimal
since we can make a small step in the direction u−wand decrease the
value of f). So, the desired inequality still holds. Finally, to give the
formal proof for a non-diﬀerentiable f, letg:Rd→R∪ {∞} be such
thatg(w)=f(w)i fw∈Sandg(w)=∞otherwise. We can therefore
rewrite w= argminvg(v). Since gis a proper1convex function we have
that 0∈∂g(w). The inequality follows by using the strong-convexity
ofg.
IfRis twice diﬀerentiable, then it is easy to verify that a suﬃcient
condition for strong convexity of Ris that for all w,x,/angbracketleft∇2R(w)x,x/angbracketright≥
σ/bardblx/bardbl2, where ∇2R(w) is the Hessian matrix of Ratw, namely, the
matrix of second-order partial derivatives of Ratw[39, Lemma 14].
Example 2.4 (Euclidean regularization). The function R(w)=
1
2/bardblw/bardbl2
2is 1-strongly-convex with respect to the /lscript2norm over Rd. To see
this, simply note that the Hessian of Rat any wis the identity matrix.
Example 2.5 (Entropic regularization). The function R(w)=/summationtextd
i=1w[i]log(w[i]) is1
B-strongly-convex with respect to the /lscript1norm
over the set S={w∈Rd:w>0∧/bardblw/bardbl1≤B}. In particular, Ris
1-strongly-convex over the probability simplex, which is the positivevectors whose elements sum to 1.
To see this note that
/angbracketleft∇
2R(w)x,x/angbracketright=/summationdisplay
ix[i]2
w[i]=1
/bardblw/bardbl1/parenleftBigg/summationdisplay
iw[i]/parenrightBigg/parenleftBigg/summationdisplay
ix[i]2
w[i]/parenrightBigg
≥1
/bardblw/bardbl1/parenleftBigg/summationdisplay
i/radicalbig
w[i]|x[i]|/radicalbig
w[i]/parenrightBigg2
=/bardblx/bardbl2
1
/bardblw/bardbl1, (2.6)
where the inequality follows from Cauchy–S chwartz inequality.
1A convex function is proper if it never receives the value −∞and it receives a ﬁnite value
at least once. For such functions, an optimality condition for wbeing a minimum is that
the zero vector is a sub-gradient of the function at w.
2.5 Strongly Convex Regularizers 137
Additional useful properties are given in the following lemma, whose
proof follows directly from the deﬁnition of strong convexity.
Lemma 2.9. IfRis 1-strongly convex over Swith respect to some
norm then σRisσ-strongly-convex over Swith respect to the same
norm. In addition, if S/primeis a convex subset of S, then Ris 1-strongly
convex over S/primeas well.
2.5.2 Analyzing FoReL with Strongly Convex Regularizers
We now analyze FoReL with strongly convex regularizers. Recall the
regret bound given in Lemma 2.3:
T/summationdisplay
t=1(ft(wt)−ft(u))≤R(u)−R(w1)+T/summationdisplay
t=1(ft(wt)−ft(wt+1)).
IfftisL-Lipschitz with respect to a norm /bardbl·/bardblthen
ft(wt)−ft(wt+1)≤L/bardblwt−wt+1/bardbl.
Therefore, we need to ensure that /bardblwt−wt+1/bardblis small. The following
lemma shows that if the regularization function R(w) is strongly convex
with respect to the same norm, then wtwill be close to wt+1.
Lemma 2.10. LetR:S→Rbe aσ-strongly-convex function over S
with respect to a norm /bardbl·/bardbl. Let w1,w2,...be the predictions of the
FoReL algorithm. Then, for all t,i fftisLt-Lipschitz with respect
to/bardbl·/bardblthen
ft(wt)−ft(wt+1)≤Lt/bardblwt−wt+1/bardbl≤L2
t
σ.
Proof. For all tletFt(w)=/summationtextt−1
i=1fi(w)+R(w) and note that the
FoReL rule is wt= argminw∈SFt(w). Note also that Ftisσ-strongly-
convex since the addition of a convex function to a strongly convexfunction keeps the strong convexity property. Therefore, Lemma 2.8implies that:
F
t(wt+1)≥Ft(wt)+σ
2/bardblwt−wt+1/bardbl2.
138 Online Convex Optimization
Repeating the same argument for Ft+1and its minimizer wt+1we get
Ft+1(wt)≥Ft+1(wt+1)+σ
2/bardblwt−wt+1/bardbl2.
Summing the above two inequalities and rearranging we obtain
σ/bardblwt−wt+1/bardbl2≤ft(wt)−ft(wt+1). (2.7)
Next, using the Lipschitzness of ftwe get that
ft(wt)−ft(wt+1)≤Lt/bardblwt−wt+1/bardbl.
Combining with Equation (2.7) and rearranging we get that /bardblwt−
wt+1/bardbl≤L/σand together with the above we conclude our proof.
Combining the above Lemma with Lemma 2.3 we obtain
Theorem 2.11. Letf1,...,f Tbe a sequence of convex functions such
thatftisLt-Lipschitz with respect to some norm /bardbl·/bardbl. Let Lbe such
that1
T/summationtextT
t=1L2
t≤L2. Assume that FoReL is run on the sequence with
a regularization function which is σ-strongly-convex with respect to the
same norm. Then, for all u∈S,
RegretT(u)≤R(u)−min
v∈SR(v)+TL2/σ.
2.5.3 Derived Bounds
We now derive concrete bounds from Theorem 2.11. We start with the
simplest case of Euclidean regularization, which is 1-strongly convexover R
d, hence the following corollary follows.
Corollary 2.12. Letf1,...,f Tbe a sequence of convex functions
such that ftisLt-Lipschitz with respect to /bardbl·/bardbl2. Let Lbe such that
1
T/summationtextT
t=1L2
t≤L2. Assume that FoReL is run on the sequence with the
regularization function R(w)=1
2η/bardblw/bardbl2
2. Then, for all u,
RegretT(u)≤1
2η/bardblu/bardbl2
2+ηTL2.
2.5 Strongly Convex Regularizers 139
In particular, if U={u:/bardblu/bardbl2≤B}andη=B
L√
2Tthen
RegretT(U)≤BL√
2T.
Observe that the bound we obtained is identical to the bound of
Online-Gradient-Descent given in Corollary 2.7.
As mentioned in the previous section, the Euclidean regularization
cannot be applied to the problem of prediction with expert advice sinceit does not enforce w
tto be in the probability simplex. A simple solu-
tion is to enforce the constraint wt∈Sby setting R(w)=∞whenever
w/∈S. In light of Lemma 2.9, the resulting regularization function
remains strongly convex on Sand we obtain the following corollary.
Corollary 2.13. Assume that the conditions of Corollary 2.12 hold.
LetSbe a convex set and consider running FoReL with the regular-
ization function
R(w)=

1
2η/bardblw/bardbl2
2ifw∈S
∞ ifw/∈S
Then, for all u∈S,
RegretT(u)≤1
2η/bardblu/bardbl2
2+ηTL2.
In particular, if B≥max u∈S/bardblu/bardbl2andη=B
L√
2Tthen
RegretT(S)≤BL√
2T.
We can apply the regularization function given in the above corol-
lary to the problem of prediction with expert advice. In this case, Sis
the probability simplex and xt∈[0,1]d. Hence, we can set B= 1 and
L=√
dwhich leads to the regret bound√
2dT. We next show another
regularization function which leads to a regret bound of/radicalbig
2log(d)T.
The improvement is based on the following corollary of Theorem 2.11.
140 Online Convex Optimization
Corollary 2.14. Letf1,...,f Tbe a sequence of convex functions
such that ftisLt-Lipschitz with respect to /bardbl·/bardbl1. Let Lbe such that
1
T/summationtextT
t=1L2
t≤L2. Assume that FoReL is run on the sequence with
the regularization function R(w)=1
η/summationtext
iw[i]log(w[i]) and with the set
S={w:/bardblw/bardbl1=B∧w>0}⊂ Rd. Then,
RegretT(S)≤Blog(d)
η+ηBTL2.
In particular, setting η=√logd
L√
2Tyields
RegretT(S)≤BL/radicalbig
2log(d)T.
The Entropic regularization used in the above corollary is strongly
convex with respect to the /lscript1norm, and therefore the Lipschitzness
requirement of the loss functions is also with respect to the /lscript1-norm.
For linear functions, ft(w)=/angbracketleftw,xt/angbracketright, we have by Holder inequality that,
|ft(w)−ft(u)|=|/angbracketleftw−u,xt/angbracketright |≤/bardbl w−u/bardbl1/bardblxt/bardbl∞.
Therefore, the Lipschitz parameter grows with the /lscript∞norm of xtrather
than the /lscript2norm of xt. Applying this to the problem of prediction with
expert advice (with B= 1 and L= 1), we obtain the regret bound of/radicalbig
2log(d)T.
More generally, it is interesting to compare the two bounds given
in Corollaries 2.13 and 2.14. Apart from the extra log( d) factor
that appears in Corollary 2.14, both bounds look similar. However,the parameters B,L have diﬀerent meanings in the two bounds.
In Corollary 2.12, the parameter Bimposes an /lscript
2constraint on u
and the parameter Lcaptures Lipschitzness of the loss functions with
respect to the /lscript2norm. In contrast, in Corollary 2.14 the parameter B
imposes an /lscript1constraint on u(which is stronger than an /lscript2constraint)
while the parameter Lcaptures Lipschitzness with respect to the /lscript1
norm (which is weaker than Lipschitzness with respect to the /lscript2norm).
Therefore, the choice of the regularization function should depend onthe Lipschitzness of the loss functions and on prior assumptions on
2.6 Online Mirror Descent 141
the set of competing vectors (does a competing vector have a small /lscript1
norm or only a small /lscript2norm). In the prediction with expert advice,
the competing vector would be a singleton, for which both the /lscript2and
/lscript1norms are 1. On the other hand, the gap between Lipschitzness with
respect to /lscript2norm (which was√
d) and Lipschitzness with respect to /lscript1
norm (which was 1) is large. Therefore, we should prefer the Entropic
regularization for the problem of prediction with expert advice.
2.6 Online Mirror Descent
In the previous section we analyzed the FoReL approach in the pres-ence of a general regularization. A possible disadvantage of the FoReLapproach is that it requires solving an optimization problem at eachonline round. In this section we derive and analyze the family of OnlineMirror Descent algorithms from the FoReL framework. We will showthat Online Mirror Descent achieves the same regret bound as FoReLbut the update step is much simpler.
The starting point is to apply FoReL on a sequence of linear func-
tions in which f
t(w)=/angbracketleftw,zt/angbracketrightwith some regularization function R(w).
Throughout, we assume that R(w)=∞forw/negationslash∈S. Using the notation
z1:t=/summationtextt
i=1ziwe can rewrite the prediction of FoReL as follows:
wt+1= argmin
wR(w)+t/summationdisplay
i=1/angbracketleftw,zt/angbracketright
= argmin
wR(w)+/angbracketleftw,z1:t/angbracketright
= argmax
w/angbracketleftw,−z1:t/angbracketright−R(w).
Letting
g(θ) = argmax
w/angbracketleftw,θ/angbracketright−R(w), (2.8)
we can rewrite the FoReL prediction based on the following recursive
update rule:
1.θt+1=θt−zt
2.wt+1=g(θt+1)
142 Online Convex Optimization
Now, if ftis convex but nonlinear, we can use the same technique
we used for deriving the Online Gradient Descent algorithm and usesub-gradients of f
tatwtto linearize the problem. That is, letting zt
be a sub-gradient of ftatwtwe have that for all u
ft(wt)−ft(u)≤/angbracketleftwt,zt/angbracketright−/angbracketleft u,zt/angbracketright.
Summing over twe obtain that the regret with respect to the nonlinear
loss functions is upper bounded by the regret with respect to the linearfunctions. This yields the Online Mirror Descent framework.
Online Mirror Descent (OMD)
parameter: a link function g:Rd→S
initialize: θ1=0
fort=1,2,...
predict wt=g(θt)
update θt+1=θt−ztwhere zt∈∂ft(wt)
Clearly, Online Gradient Descent is a special case of Online Mirror
Descent that is obtained by setting S=Rdandg(θ)=ηθ, for some
η>0. When gis nonlinear we obtain that the vector θis updated
by subtracting the gradient out of it, but the actual prediction is“mirrored” or “linked” to the set Svia the function g. Hence the name
Online Mirror Descent, and this is why gis often referred to as a link
function.
Before we derive concrete algorithms from the OMD framework we
give a generic bound for the OMD family based on our analysis of theFoReL rule.
Theorem 2.15. LetRb ea( 1 /η)-strongly-convex function over Swith
respect to a norm /bardbl·/bardbl. Assume that OMD is run on the sequence with
a link function
g(θ) = argmax
w∈S(/angbracketleftw,θ/angbracketright−R(w)). (2.9)
2.6 Online Mirror Descent 143
Then, for all u∈S,
RegretT(u)≤R(u)−min
v∈SR(v)+ηT/summationdisplay
t=1/bardblzt/bardbl2
⋆,
where /bardbl·/bardbl⋆is the dual norm. Furthermore, if ftisLt-Lipschitz with
respect to /bardbl·/bardbl, then we can further upper bound /bardblzt/bardbl⋆≤Lt.
Proof. As we have shown previously,
T/summationdisplay
t=1(ft(wt)−ft(u))≤T/summationdisplay
t=1/angbracketleftwt−u,zt/angbracketright,
and the OMD algorithm is equivalent to running FoReL on the sequence
of linear functions with the regularization R(w). The theorem now
follows directly from Theorem 2.11 and Lemma 2.6.
2.6.1 Derived Algorithms
We now derive additional algorithms from the OMD framework.
The ﬁrst algorithm we derive is often called normalized Exponen-
tiated Gradient. In this algorithm, S={w:/bardblw/bardbl1=1∧w≥0}is the
probability simplex and g:Rd→Rdis the vector valued function whose
ith component is the function
gi(θ)=eηθ[i]
/summationtext
jeηθ[j]. (2.10)
Therefore,
wt+1[i]=eηθt+1[i]
/summationtext
jeηθt+1[j]=eηθt+1[i]
/summationtext
jeηθt+1[j]·/summationtext
keηθt[k]
/summationtext
keηθt[k]
=eηθt[i]e−ηzt[i]
/summationtext
jeηθt[j]e−ηzt[j]·/summationtext
keηθt[k]
/summationtext
keηθt[k]
=wt[i]e−ηzt[i]
/summationtext
jwt[j]e−ηzt[j].
144 Online Convex Optimization
Normalized Exponentiated Gradient
(normalized-EG)
parameter: η>0
initialize: w 1=( 1/d,..., 1/d)
update rule ∀i, wt+1[i]=wt[i]e−ηzt[i]
/summationtext
jwt[j]e−ηzt[j]where zt∈∂ft(wt)
To analyze the normalized-EG algorithm, we rely on Theorem 2.15.
LetR(w)=1
η/summationtext
iw[i]log(w[i]) be the entropic regularization, let Sbe
the probability simplex, and recall that Ris (1/η)-strongly convex over
Swith respect to the /lscript1norm. Using the technique of Lagrange mul-
tipliers, it is easy to verify that the solution to the optimization prob-lem given in Equation (2.9) is the value of the link function given inEquation (2.10). Therefore, Theorem 2.15 yields:
Corollary 2.16. The normalized EG algorithm enjoys the regret
bound given in Corollary 2.14 (with B= 1).
Next, we derive an algorithm which is called Online Gradient
Descent with Lazy Projections. To derive this algorithm, let Sbe a
convex set and deﬁne
g(θ) = argmin
w∈S/bardblw−ηθ/bardbl2.
That is, g(θ) returns the point in Swhich is closest to ηθ.
Online Gradient Descent with Lazy Projections
parameters: η>0 and a convex set S
initialize: θ1=0
fort=1,2,...,T
wt= argminw∈S/bardblw−ηθt/bardbl2
θt+1=θt−ztwhere zt∈∂ft(wt)
To analyze the above algorithm we consider the Euclidean regular-
ization function R(w)=1
2η/bardblw/bardbl2
2, which is (1 /η)-strongly convex over S
2.6 Online Mirror Descent 145
with respect to the /lscript2norm. We have that
argmax
w∈S(/angbracketleftw,θ/angbracketright−R(w)) = argmax
w∈S(/angbracketleftw,ηθ/angbracketright−1
2/bardblw/bardbl2
2)
= argmin
w∈S1
2/bardblw−ηθ/bardbl2
2= argmin
w∈S/bardblw−ηθ/bardbl2.
We therefore conclude that:
Corollary 2.17. Online Gradient Descent with Lazy Projections
enjoys the same regret bound given in Corollary 2.13.
Finally, we derive the p-norm algorithm, in which S=Rdand
gi(θ)=ηsign(θ[i])|θ[i]|p−1
/bardblθ/bardblp−2
p,
where p≥2 is a parameter and
/bardblθ/bardblp=/parenleftBiggd/summationdisplay
i=1|θ[i]|p/parenrightBigg1/p
.
p-norm
parameters: η>0 and p>2
initialize: θ1=0
fort=1,2,...,T
∀i, wt,i=ηsign(θt[i])|θt[i]|p−1
/bardblθt/bardblp−2
p
θt+1=θt−ztwhere zt∈∂ft(wt)
To analyze the p-norm algorithm consider the regularization func-
tionR(w)=1
2η(q−1)/bardblw/bardbl2
q, where q=p/(p−1). It is possible to show
that if q∈(1,2] then Gis (1/η)-strongly convex over Rdwith respect
to the /lscriptqnorm (see for example [39, Lemma 17]). It is also possible to
verify that g(θ) = argmaxw/angbracketleftw,θ/angbracketright−R(w). We therefore conclude:
Corollary 2.18. Letf1,...,f Tbe a sequence of convex functions such
thatftisLt-Lipschitz over Rdwith respect to /bardbl·/bardblq. Let Lbe such
146 Online Convex Optimization
that1
T/summationtextT
t=1L2
t≤L2. Then, for all u, the regret of the p-norm algorithm
satisﬁes
RegretT(u)≤1
2η(q−1)/bardblw/bardbl2
q+ηTL2.
In particular, if U={u:/bardblu/bardblq≤B}andη=B
L√
2T/(q−1)then
RegretT(U)≤BL/radicalBigg
2T
q−1.
When q= 2 the link function becomes g(θ)=ηθand the p-norm
algorithm boils down to the Online Gradient Descent algorithm.When qis close to 1 it is not very hard to see that the p-norm algorithm
behaves like the Entropic regularization. In particular, when p= log( d)
we can obtain a regret bound similar to the regret bound of the EGalgorithm. Intermediate values of qenables us to interpolate between
the properties of the Entropic and Euclidean regularizations.
2.7 The Language of Duality
In the previous sections we relied on the FoReL framework for deriv-ing online learning algorithms. In this section we present a diﬀerentproof technique that relies on duality. There are several reasons to con-sider this diﬀerent approach. First, in some cases, it is easier to derivetighter bounds based on the duality approach. In particular, we willtighten the regret bounds we derived for the OMD framework by afactor of√
2 and we will also derive tighter bounds that involve the
so-called local norms rather than ﬁxed norms. Second, it may becomeconvenient for developing new algorithms. Last, many previous paperson online learning uses the language of duality, so the reader may ﬁndthis section useful for understanding the previous literature.
We start by giving some background on Fenchel conjugacy and dual-
ity. For more details, see for example [9].
2.7.1 Fenchel Conjugacy
There are two equivalent representations of a convex function. Either
as pairs ( x,f(x)) or as the set of tangents of f, namely pairs of the form
2.7 The Language of Duality 147
Fig. 2.3 Illustrating Fenchel Conjugacy. We can represent a function as a set of points ( left)
or as a set of tangents ( right).
(slope,intersection-with-y-axis). See Figure 2.3 for an illustration. The
function that relates slopes of tangents to their intersection with theyaxis is called the Fenchel conjugate of f, and is formally deﬁned as
f
⋆(θ) = max
u/angbracketleftu,θ/angbracketright−f(u).
It is possible to show that f=(f⋆)⋆if and only if fis a closed2
and convex function (see [9, Theorem 4.2.1]). From now on, we always
assume that our functions are closed and convex.
The deﬁnition of Fenchel conjugacy immediately implies Fenchel–
Young inequality :
∀u,f⋆(θ)≥/angbracketleftu,θ/angbracketright−f(u). (2.11)
It is possible to show that equality holds if uis a sub-gradient of
f⋆atθand in particular, if f⋆is diﬀerentiable, equality holds when
u=∇f⋆(θ).
2fis closed if its epigraph, i.e., the set {(w,y):f(w)≤y}, is a closed set.
148 Online Convex Optimization
Table 2.1. Example of Fenchel conjugate pairs.
f(w) f⋆(θ) Comments
1
2/bardblw/bardbl2
21
2/bardblθ/bardbl2
2
1
2/bardblw/bardbl2
q1
2/bardblθ/bardbl2
p where1
p+1
q=1
/summationtext
iw[i]log(w[i]) +I{v≥0:/bardblv/bardbl1=1}(w) log(/summationtext
ieθ[i]) (normalized Entropy)
/summationtext
iw[i](log(w[i])−1)/summationtext
ieθ[i](un-normalized Entropy)
1
ηg(w)1
ηg⋆(ηθ) where η>0
g(w)+/angbracketleftw,x/angbracketright g⋆(θ−x)
Table 2.1 lists several Fenchel conjugate pairs. Recall that given a
setS, we use the notation
IS(w)=/braceleftbigg0i f w∈S
∞ifw/∈S
2.7.2 Bregman Divergences and the Strong/Smooth
Duality
A diﬀerentiable function Rdeﬁnes a Bregman divergence between two
vectors as follows:
DR(w/bardblu)=R(w)−(R(u)+/angbracketleft∇R(u),w−u/angbracketright). (2.12)
That is, the Bregman divergence is the diﬀerence, at the point w,
between Rand its linearization around u. When Ris convex the Breg-
man divergence is always non-negative. However, it is not a metricmeasure because it is not symmetric and also does not satisfy thetriangle inequality. An illustration is given in Figure 2.4.
When R(w)=
1
2/bardblw/bardbl2
2the Bregman divergence becomes
DR(w/bardblu)=1
2/bardblw−u/bardbl2
2. When R(w)=/summationtext
iw[i]log(w[i]) the Bregman
divergence between two vectors in the probability simplex becomesthe Kullback–Leibler divergence, D
R(w/bardblu)=/summationtext
iw[i]logw[i]
u[i].
Recall the deﬁnition of strong-convexity (Deﬁnition 2.4). If Ris
diﬀerentiable, we can rewrite the σ-strong-convexity requirement as
DR(w/bardblu)≥σ
2/bardblw−u/bardbl2.
A related property is strong-smoothness .
2.7 The Language of Duality 149
Fig. 2.4 Illustrating the Bregman divergence.
Deﬁnition 2.5. A function Risσ-strongly-smooth with respect to a
norm /bardbl·/bardblif it is diﬀerentiable and for all u,wwe have
DR(w/bardblu)≤σ
2/bardblw−u/bardbl2.
Not surprisingly, strong convexity and strong smoothness are dual
properties.
Lemma 2.19. (Strong/Smooth Duality) Assume that Ris a closed
and convex function. Then Risβ-strongly convex with respect to a
norm /bardbl·/bardblif and only if R⋆is1
β-strongly smooth with respect to the
dual norm /bardbl·/bardbl⋆.
The proof can be found, for instance, in [44] (In particular, see
Corollary 3.5.11 on p. 217 and Remark 3.5.3 on p. 218). The abovelemma implies in particular that if Ris strongly convex then R
⋆is
diﬀerentiable. Based on Section 2.7.1, this also implies that
∇R⋆(θ) = argmax
w(/angbracketleftw,θ/angbracketright−R(w)). (2.13)
2.7.3 Analyzing OMD using Duality
Recall that the OMD rule is
wt=g(θt)=g(−z1:t−1),
150 Online Convex Optimization
where the link function gis (see Equation (2.8) in Section 2.6)
g(θ) = argmax
w(/angbracketleftw,θ/angbracketright−R(w)).
Based on Equation (2.13) we can also rewrite g(θ)=∇R⋆(θ).
To analyze OMD, we ﬁrst use the following lemma.
Lemma 2.20. Suppose that OMD is run with a link function g=∇R⋆.
Then, its regret is upper bounded by
T/summationdisplay
t=1/angbracketleftwt−u,zt/angbracketright≤R(u)−R(w1)+T/summationdisplay
t=1DR⋆(−z1:t/bardbl−z1:t−1).
Furthermore, equality holds for the vector uthat minimizes R(u)+/summationtext
t/angbracketleftu,zt/angbracketright.
Proof. First, using Fenchel–Young inequality we have
R(u)+T/summationdisplay
t=1/angbracketleftu,zt/angbracketright=R(u)−/angbracketleftu,−z1:T/angbracketright≥−R⋆(−z1:T),
where equality holds for the vector uthat maximizes /angbracketleftu,−z1:T/angbracketright−R(u)
hence minimizes R(u)+/angbracketleftu,z1:T/angbracketright. Second, using the fact that wt=
∇R⋆(−z1:t−1) and the deﬁnition of the Bregman divergence, we can
rewrite the right-hand side as
−R⋆(−z1:T)=−R⋆(0)−T/summationdisplay
t=1(R⋆(−z1:t)−R⋆(−z1:t−1))
=−R⋆(0)+T/summationdisplay
t=1(/angbracketleftwt,zt/angbracketright−DR⋆(−z1:t/bardbl−z1:t−1)).
(2.14)
Note that R⋆(0) = max w/angbracketleft0,w/angbracketright−R(w)=−min wR(w)=−R(w1).
Combining all the above we conclude our proof.
It is interesting to compare the above lemma to Lemma 2.3, which forlinear functions yields the regret bound
T/summationdisplay
t=1/angbracketleftwt−u,zt/angbracketright≤R(u)−R(w1)+T/summationdisplay
t=1/angbracketleftwt−wt+1,zt/angbracketright.
2.7 The Language of Duality 151
In both bounds we have a stability term but the way we measure it is
diﬀerent.
We can easily derive concrete bounds from Lemma 2.20 if Ris
strongly convex.
Theorem 2.21. LetRb ea( 1 /η)-strongly convex with respect to a
norm /bardbl·/bardbland suppose the OMD algorithm is run with the link function
g=∇R⋆. Then,
T/summationdisplay
t=1/angbracketleftwt−u,zt/angbracketright≤R(u)−R(w1)+η
2T/summationdisplay
t=1/bardblzt/bardbl2
⋆.
Proof. The proof follows directly from Lemma 2.20 and the strong/
smooth duality given in Lemma 2.19.
The reader can easily obtain the bounds we derived in Section 2.6 usingthe above theorem. Consider for example the case in which the regu-larization function is R(w)=
1
2η/bardblw/bardbl2
2, which is (1 /η)-strongly convex
with respect to the Euclidean norm. Since the Euclidean norm is dualto itself, we obtain the bound
T/summationdisplay
t=1/angbracketleftwt−u,zt/angbracketright≤1
2η/bardblu/bardbl2
2+η
2T/summationdisplay
t=1/bardblzt/bardbl2
2. (2.15)
Letting L2=1
T/summationtext
t/bardblzt/bardbl2
2and setting η=B
L√
Tthen for all uwith
/bardblu/bardbl2≤Bwe obtain the bound
T/summationdisplay
t=1/angbracketleftwt−u,zt/angbracketright≤BL√
T.
Comparing this bound to the bound we derived in Corollary 2.7, we
observe that we have obtained a factor of√
2 improvement in the regret
bound.
2.7.4 Other Proof Techniques
In the previous subsection we used Fenchel–Young inequality to derive
bounds for OMD, which is equivalent to FoReL when the loss functions
152 Online Convex Optimization
are linear. It is possible to extend this proof technique, based on Fenchel
duality, and to derive a larger family of online convex optimizationalgorithms. The basic idea is to derive the Fenchel dual of the opti-mization problem min
wR(w)+/summationtext
tft(w) and to construct an online
learning algorithm by incrementally solving the dual problem. We referthe reader to [39] for more details.
Another popular approach is to derive regret bounds by monitoring
the Bregman divergence D
R(wt/bardblu), where uis the competing vec-
tor. A detailed description of this approach can be found in [12, 34]and is therefore omitted from this survey. It is important to note thatthe analysis using the Bregman divergence potential requires that R
will be a Legendre function (see a precise deﬁnition in [12, p. 294]).In particular, Legendre functions guarantee the property that ∇R
and∇R
⋆are inverse mappings. We do not require that Ris Legen-
dre and in particular, the normalized entropy regularization function,R(w)=/summationtext
iw[i]log(w[i]) +IS(w), where Sis the probability simplex,
is not a Legendre function. Therefore, when analyzing the normalizedEG algorithm using Bregman divergences we need to use the unnormal-ized Entropy as a regularization function and to include an additionalBregman projection step on the simplex, which leads to the desirednormalization. To the best of our knowledge, the two proof techniqueslead to the same regret bounds.
2.8 Bounds with Local Norms
Consider running the normalized EG algorithm, namely, runningFoReL on linear loss functions with the normalized entropy R(w)=
1
η/summationtext
iw[i]log(w[i]) +IS(w), where S={w≥0:/bardblw/bardbl1=1}. Previously,
we have derived the regret bound,
T/summationdisplay
t=1/angbracketleftwt−u,zt/angbracketright≤log(d)
η+ηT/summationdisplay
t=1/bardblzt/bardbl2
∞.
We now derive a reﬁned bound for the normalized EG algorithm, in
which each term /bardblzt/bardbl2
∞is replaced by a term/summationtext
iwt[i]zt[i]2. Since wtis
in the probability simplex, we clearly have that/summationtext
iwt[i]zt[i]2≤/bardblzt/bardbl2
∞.
2.8 Bounds with Local Norms 153
In fact, we can rewrite/summationtext
iwt[i]zt[i]2as a local norm /bardblzt/bardbl2
twhere
/bardblz/bardblt=/radicalBigg/summationdisplay
iwt[i]z[i]2.
Note that this is indeed a valid norm. In the next sections we will show
cases in which the reﬁned local-norm bounds lead to much tighter regretbounds.
Theorem 2.22. Assume that the normalized EG algorithm is run on a
sequence of linear loss functions such that for all t,iwe have ηzt[i]≥−1.
Then,
T/summationdisplay
t=1/angbracketleftwt−u,zt/angbracketright≤log(d)
η+ηT/summationdisplay
t=1/summationdisplay
iwt[i]zt[i]2.
Proof. Using Lemma 2.20, it suﬃces to show that
DR⋆(−z1:t/bardbl−z1:t−1)≤η/summationdisplay
iwt[i]zt[i]2,
where, based on Table 2.1, the conjugate function is
R⋆(θ)=1
ηlog/parenleftBigg/summationdisplay
ieηθ[i]/parenrightBigg
.
Indeed,
DR⋆(−z1:t/bardbl−z1:t−1)=R⋆(−z1:t)−R⋆(−z1:t−1)+/angbracketleftwt,zt/angbracketright(2.16)
=1
ηlog/parenleftBigg/summationtext
ie−ηz1:t[i]
/summationtext
ie−ηz1:t−1[i]/parenrightBigg
+/angbracketleftwt,zt/angbracketright (2.17)
=1
ηlog/parenleftBigg/summationdisplay
iwt[i]e−ηzt[i]/parenrightBigg
+/angbracketleftwt,zt/angbracketright.(2.18)
Using the inequality e−a≤1−a+a2which holds for all a≥−1 (and
hence holds by the assumptions of the theorem) we obtain
DR⋆(−z1:t/bardbl−z1:t−1)≤1
ηlog/parenleftBigg/summationdisplay
iwt[i](1−ηzt[i]+η2zt[i]2)/parenrightBigg
+/angbracketleftwt,zt/angbracketright.
154 Online Convex Optimization
Next, we use the fact that/summationtext
iwt[i] = 1 and the inequality log(1 −a)≤
−a, which holds for all a≤1 we obtain
DR⋆(−z1:t/bardbl−z1:t−1)≤1
η/summationdisplay
iwt[i](−ηzt[i]+η2zt[i]2)+/angbracketleftwt,zt/angbracketright
=η/summationdisplay
iwt[i]zt[i]2.
Next, we describe another variant of the EG algorithm in which we
do not normalize the weights to the simplex on each round. We showa similar regret bounds for this variant as well.
Unnormalized Exponentiated Gradient
(unnormalized-EG)
parameters: η,λ > 0
initialize: w 1=(λ,...,λ )
update rule ∀i, wt+1[i]=wt[i]e−ηzt[i]
The following theorem provides a regret bound with local-norms for
the unnormalized EG algorithm.
Theorem 2.23. Assume that the unnormalized EG algorithm is run
on a sequence of linear loss functions such that for all t,iwe have
ηzt[i]≥−1. Then, for all u≥0,
T/summationdisplay
t=1/angbracketleftwt−u,zt/angbracketright≤dλ+/summationtextd
i=1u[i]log(u[i]/(eλ))
η+ηT/summationdisplay
t=1d/summationdisplay
i=1wt[i]zt[i]2.
In particular, setting λ=1/dyields
T/summationdisplay
t=1/angbracketleftwt−u,zt/angbracketright≤1 + (log( d)−1)/bardblu/bardbl1+/summationtextd
i=1u[i]log(u[i])
η
+ηT/summationdisplay
t=1d/summationdisplay
i=1wt[i]zt[i]2.
2.9 Bibliographic Remarks 155
Proof. To analyze the unnormalized-EG algorithm, we ﬁrst note that it
is equivalent to running FoReL on the sequence of linear loss functionswith the following unnormalized entropy regularization function:
R(w)=1
η/summationdisplay
iw[i](log(w[i])−1−log(λ)).
Using Lemma 2.20, it suﬃces to show that
DR⋆(−z1:t/bardbl−z1:t−1)≤η/summationdisplay
iwt[i]zt[i]2,
where, based on Table 2.1, the conjugate function is
R⋆(θ)=λ
η/summationdisplay
ieηθ[i].
We have:
DR⋆(−z1:t/bardbl−z1:t−1)=R⋆(−z1:t)−R⋆(−z1:t−1)+/angbracketleftwt,zt/angbracketright
=λ
η/summationdisplay
ie−ηz1:t−1[i](e−ηzt[i]−1) +/angbracketleftwt,zt/angbracketright
=1
η/summationdisplay
iwt[i](e−ηzt[i]−1) +/angbracketleftwt,zt/angbracketright
≤η/summationdisplay
iwt[i]zt[i]2,
where in the last inequality we used the inequality e−a≤1−a+a2
which holds for all a≥−1 (and hence holds by the assumptions of the
lemma).
2.9 Bibliographic Remarks
The term “online convex programming” was introduced byZinkevich [46] but this setting was introduced some years earlier byGordon [20]. This model is also closely related to the model of relativeloss bounds presented by Kivinen and Warmuth [26, 27, 28]. Ourpresentation of relative mistake bounds follows the works of Little-stone [31], and Kivinen and Warmuth [28].
156 Online Convex Optimization
Zinkevich presented and analyzed the online gradient descent (with
projections or with lazy projections). The name “Follow the Leader” isdue to [25]. Analysis of “Follow the Regularized Leader” was given in[40, 39] using Fenchel duality as part of the analysis of a larger algo-rithmic framework. The more direct analysis we give here is adaptedfrom [34, 22]. The duality based analysis we present is due to [39, 24].Similar ideas in a more limited context have been introduced in [21].
The EG update is due to [26]. See also [6, 28]. The p-norm algo-
rithm was developed by Gentile and Littlestone [19]. Local norms wereintroduced in [2]. See also [34].
3
Online Classiﬁcation
In this section we return to the problem of online classiﬁcation, in
which the target domain is Y={0,1}. We already mentioned this set-
ting in Section 1.2, where we introduced the Consistent andHalving
algorithms. These algorithms rely on two simplifying assumptions: real-izability of the problem and ﬁniteness of H. While trying to relax
the realizability assumption, we presented Cover’s impossibility result,which shows that even if |H|= 2, no algorithm can have low regret
in the unrealizable case. We sidestepped the impossibility result byallowing the learner to randomize his predictions (i.e., the predictionsdomain is allowed to be D=[ 0,1]). As discussed in Section 2.1.1, this
can be thought of as a convexiﬁcation of the problem. Based on this con-vexiﬁcation, we start the section by deriving the Weighted Majority
algorithm for prediction with expert advice. We will see that this isa speciﬁc instance of the normalized EG algorithm discussed in the
previous section. The analysis of Weighted Majority implies onlinelearnability of ﬁnite hypothesis classes.
Next, in Section 3.2, we study the fundamental question of
online learnability, namely, what is an optimal algorithm for a givenhypothesis class. We shall characterize learnability for the general case,namely, without assuming neither realizability nor ﬁniteness of H.
157
158 Online Classiﬁcation
Finally, in Section 3.3, we study the important and practically
relevant hypothesis class of halfspaces with margin and derive the clas-sic Perceptron and Winnow algorithms from the online convex opti-mization framework.
3.1 Finite Hypothesis Class and Experts Advice
Consider the problem of classiﬁcation, where on round tthe learner
receives xt∈X, predicts pt∈[0,1], receives yt∈{0,1}, and pays
|pt−yt|. The goal is to have low regret with respect to a ﬁnite hypoth-
esis class H={h1,...,h d}. As we hinted in Section 2.1.1, this problem
can be reduced to the problem of prediction with expert advice asfollows. Let S={w∈R
d
+:/bardblw/bardbl1=1}be the probability simplex. The
learner will maintain a weight vector wt∈Sand will predict the label
1 with probability pt=/summationtextd
i=1wt[i]hi(xt). The loss he will pay can there-
fore be rewritten as
|pt−yt|=/vextendsingle/vextendsingle/vextendsingle/vextendsingle/vextendsingled/summationdisplay
i=1wt[i]hi(xt)−yt/vextendsingle/vextendsingle/vextendsingle/vextendsingle/vextendsingle=d/summationdisplay
i=1wt[i]|hi(xt)−yt|,
where the last equality follows because both ytandhi(xt) are in {0,1}.
Letting zt=(|h1(xt)−yt|,...,|hd(xt)−yt|) we obtain that the loss is
/angbracketleftwt,zt/angbracketright. This is a special case of the prediction with expert advice
problem, where zt∈[0,1]dis the costs vector of the diﬀerent experts.
Applying the normalized-EG algorithm we obtain an algorithm whichis often called Weighted Majority.
Weighted Majority
parameter: η∈(0,1)
initialize :w1=( 1/d,..., 1/d)
fort=1,2,...
choose i∼wtand predict according to the advice of the i’th
expert receive costs of all experts zt∈[0,1]d
update rule ∀i, wt+1[i]=wt[i]e−ηzt[i]
/summationtext
jwt[j]e−ηzt[j]
3.1 Finite Hypothesis Class and Experts Advice 159
The analysis of Weighted Majority follows directly from the anal-
ysis of the normalized-EG algorithm given in the previous section and
is given in the following theorem.
Theorem 3.1. The Weighted Majority algorithm enjoys the bounds
1.T/summationdisplay
t=1/angbracketleftwt,zt/angbracketright≤min
i∈[d]T/summationdisplay
t=1zt[i]+log(d)
η+ηT
2.T/summationdisplay
t=1/angbracketleftwt,zt/angbracketright≤1
1−η/parenleftBigg
min
i∈[d]T/summationdisplay
t=1zt[i]+log(d)
η/parenrightBigg
In particular, setting η=/radicalbig
log(d)/Tin the ﬁrst bound we obtain
T/summationdisplay
t=1/angbracketleftwt,zt/angbracketright≤min
i∈[d]T/summationdisplay
t=1zt[i]+2/radicalbig
log(d)T,
and setting η=1/2 in the second bound we obtain
T/summationdisplay
t=1/angbracketleftwt,zt/angbracketright≤2min
i∈[d]T/summationdisplay
t=1zt[i] + 4log( d).
Proof. We rely on the analysis of the normalized-EG algorithm given in
Theorem 2.22. Since zt∈[0,1]dandη>0 the conditions of the theorem
holds and we obtain that
T/summationdisplay
t=1/angbracketleftwt−u,zt/angbracketright≤log(d)
η+ηT/summationdisplay
t=1/summationdisplay
iwt[i]zt[i]2.
Since wt∈Sandzt∈[0,1]dthe right-most term is at most ηT. Further-
more, the bound holds for all uin the probability simplex and in partic-
ular for all singletons, i.e., vectors of the form u=( 0,...,0,1,0,...,0).
Rearranging, we obtain,
T/summationdisplay
t=1/angbracketleftwt,zt/angbracketright≤min
i∈[d]T/summationdisplay
t=1zt[i]+log(d)
η+ηT.
160 Online Classiﬁcation
To derive the second bound, observe that wt[i],zt[i]∈[0,1] and there-
forewt[i]zt[i]2≤wt[i]zt[i], which yields
T/summationdisplay
t=1/angbracketleftwt−u,zt/angbracketright≤log(d)
η+ηT/summationdisplay
t=1/angbracketleftwt,zt/angbracketright.
Rearranging, we obtain that
T/summationdisplay
t=1/angbracketleftwt,zt/angbracketright≤1
1−η/parenleftBigg
min
i∈[d]T/summationdisplay
t=1zt[i]+log(d)
η/parenrightBigg
.
In particular, in the realizable case when min i∈[d]/summationtextT
t=1zt[i]=0w e
obtain that Weighted Majority withη=1/2 enjoys the bound
T/summationdisplay
t=1/angbracketleftwt,zt/angbracketright≤4log(d).
This is similar to the bound we derived for Halving in Section 1.2.
3.2 Learnability and the Standard Optimal Algorithm
So far, we focused on ﬁnite hypothesis classes. In this section we take a
more general approach, and aim at characterizing online learnability. Inparticular, we target the following question: what is the optimal onlinelearning algorithm for a given class H?
We start with the realizable case, where we assume that all target
labels are generated by some h
⋆∈H, namely, yt=h⋆(xt) for all t. Later,
in Section 3.2.1 we generalize the results to the unrealizable case. In therealizable case, we study the best achievable mistake bound, formallydeﬁned below.
Deﬁnition 3.1 (Mistake bounds, online learnability). LetHbe
a hypothesis class and let Abe an online learning algorithm. Given
any sequence S=(x1,h⋆(y1)),...,(xT,h⋆(yT)), where Tis any integer
andh⋆∈H, letMA(S) be the number of mistakes Amakes on the
sequence S. We denote by MA(H) the supremum of MA(S) over all
sequences of the above form. A bound of the form MA(H)≤B<∞
is called a mistake bound . We say that a hypothesis class His online
learnable if there exists an algorithm Afor which MA(H)≤B<∞.
3.2 Learnability and the Standard Optimal Algorithm 161
We present a dimension of hypothesis classes that characterizes the
best possible achievable mistake bound. This measure was proposed byNick Littlestone and we therefore refer to it as Ldim( H).
To motivate the deﬁnition of Ldim it is convenient to view the
online learning process as a game between two players: the learnervs. the environment. On round tof the game, the environment picks
an instance x
t, the learner predicts a label pt∈{0,1}, and ﬁnally the
environment outputs the true label, yt∈{0,1}. Suppose that the envi-
ronment wants to make the learner err on the ﬁrst Trounds of the
game. Then, it must output yt=1−pt, and the only question is how
to choose the instances xtin such a way that ensures that for some
h⋆∈Hwe have yt=h⋆(xt) for all t∈[T]={1,...,T }.
It makes sense to assume that the environment should pick xtbased
on the previous predictions of the learner, p1,...,p t−1. Since in our case
we have yt=1−ptwe can also say that xtis a function of y1,...,y t−1.
We can represent this dependence using a complete binary tree ofdepth T(we deﬁne the depth of the tree as the number of edges in a
path from the root to a leaf). We have 2
T−1 nodes in such a tree, and
we attach an instance to each node. Let v1,...,v2T−1be these instances.
We start from the root of the tree, and set x1=v1. At round t,w es e t
xt=vit, where itis the current node. At the end of round t,w eg ot o
the left child of itifyt= 0 or to the right child if yt= 1. That is, it+1=
2it+yt. Unraveling the recursion we obtain it=2t−1+/summationtextt−1
j=1yj2t−1−j.
The above strategy for the environment succeeds only if for any
(y1,...,y T) there exists h∈Hsuch that yt=h(xt) for all t∈[T]. This
leads to the following deﬁnition.
Deﬁnition 3.2 ( HShattered tree). A shattered tree of depth dis
a sequence of instances v1,...,v2d−1inXsuch that for all labeling
(y1,...,y d)∈{0,1}dthere exists h∈Hsuch that for all t∈[d]w eh a v e
h(vit)=yt, where it=2t−1+/summationtextt−1
j=1yj2t−1−j.
An illustration of a shattered tree of depth 2 is given in Figure 3.1.
Deﬁnition 3.3 (Littlestone’s dimension ( Ldim )).Ldim( H)i st h e
maximal integer Tsuch that there exist a shattered tree of depth T.
162 Online Classiﬁcation
Fig. 3.1 An illustration of a shattered tree of depth 2. The dashed blue path corresponds
to the sequence of examples (( v1,1),(v3,0)). The tree is shattered by H={h1,h2,h3,h4},
where the predictions of each hypothesis in Hon the instances v1,v2,v3is given in the
table (the “ ∗” mark means that hj(vi) can be either 1 or 0).
The deﬁnition of Ldim and the discussion above immediately imply
the following:
Lemma 3.2. No algorithm can have a mistake bound strictly smaller
than Ldim( H), namely, ∀A, M A(H)≥Ldim( H).
Proof. LetT= Ldim( H)and let v1,...,v2T−1be a sequence that
satisﬁes the requirements in the deﬁnition of Ldim. If the environmentsetsx
t=vitandyt=1−ptfor all t∈[T], then the learner makes T
mistakes while the deﬁnition of Ldim implies that there exists a hypoth-esish∈Hsuch that y
t=h(xt) for all t.
Let us now give several examples.
Example 3.1. LetHbe a ﬁnite hypothesis class. Clearly, any tree that
is shattered by Hhas depth of at most log2(|H|). Therefore, Ldim( H)≤
log2(|H|). Another way to conclude this inequality is by combining
Lemma 3.2 with Theorem 1.2.
Example 3.2. LetX={1,...,d}and H={h1,...,h d}, where
hd(x)=1i ﬀ x=d. Then, it is easy to show that Ldim( H) = 1 while
|H|=dcan be arbitrarily large.
Example 3.3. LetX=[ 0,1] and H={x/mapsto→1[x>a]:a∈[0,1]}, namely,
His the class of thresholds on the segment [0 ,1]. Then, Ldim( H)=∞.
3.2 Learnability and the Standard Optimal Algorithm 163
To see this, consider the tree for which v1=1
2,v2=1
4,v3=3
4,.... Due
to the density of the reals, this tree is shattered by H.
Example 3.4. LetX={x∈{0,1}∗:/bardblx/bardbl0≤r}and H={x/mapsto→
1[/angbracketleftw,x/angbracketright>0.5]:/bardblw/bardbl0≤k}. The size of His inﬁnite. Nevertheless,
Ldim( H)≤rk. The proof uses the fact that MPerceptron (H)≤rk,
where the Perceptron algorithm will be described in the next section.
Lemma 3.2 states that Ldim( H) lower bounds the mistake bound
of any algorithm. Interestingly, there is a standard algorithm whosemistake bound matches this lower bound. The algorithm is similarto the Halving algorithm. Recall that the prediction of Halving is
according to a majority vote of the hypotheses which are consis-tent with previous examples. We denoted this set by V
t. Put another
way, Halving partition Vtinto two sets: V+
t={h∈Vt:h(xt)=1}and
V−
t={h∈Vt:h(xt)=0}. It then predicts according to the larger of
the two groups. The rationale behind this prediction is that wheneverHalving makes a mistake it ends up with |V
t+1|≤0.5|Vt|.
The optimal algorithm we present below uses the same idea, but
instead of predicting according to the larger class, it predicts accordingto the class with larger Ldim.
Standard Optimal Algorithm (SOA)
input: A hypothesis class H
initialize :V1=H
fort=1,2,...
receive xt
forr∈{0,1}letV(r)
t={h∈Vt:h(xt)=r}
predict pt= argmaxr∈{0,1}Ldim( V(r)
t)
(in case of a tie predict pt=1 )
receive true answer yt
update Vt+1={h∈Vt:h(xt)=yt}
The following lemma formally establishes the optimality of the
above algorithm.
164 Online Classiﬁcation
Lemma 3.3. SOAenjoys the mistake bound MSOA(H)≤Ldim( H).
Proof. It suﬃces to prove that whenever the algorithm makes a predic-
tion mistake we have Ldim( Vt+1)≤Ldim( Vt)−1. We prove this claim
by assuming the contrary, that is, Ldim( Vt+1) = Ldim( Vt). If this holds
true, then the deﬁnition of ptimplies that Ldim( V(r)
t) = Ldim( Vt) for
both r= 1 and r= 0. But, then we can construct a shattered tree
of depth Ldim( Vt) + 1 for the class Vt, which leads to the desired
contradiction.
Combining Lemma 3.3 and Lemma 3.2 we obtain:
Corollary 3.4. LetHbe any hypothesis class. Then, the standard
optimal algorithm enjoys the mistake bound MSOA(H) = Ldim( H) and
no other algorithm can have MA(H)<Ldim( H).
Comparison to VC dimension In the PAC learning model of
Valiant, learnability is characterized by the Vapnik–Chervonenkis (VC)dimension of the class H. To remind the reader, the VC dimension of
a class H, denoted VCdim( H), is the maximal number dsuch that
there are instances x
1,...,xdthat are shattered by H. That is, for any
sequence of labels ( y1,...,y d)∈{0,1}dthere exists a hypothesis h∈H
that gives exactly this sequence of labels. The following theorem relatesthe VC dimension to the Littlestone dimension.
Theorem 3.5. For any class H, VCdim( H)≤Ldim( H), and there are
classes for which strict inequality holds. Furthermore, the gap can bearbitrarily large.
Proof. We ﬁrst prove that VCdim( H)≤Ldim( H). Suppose
VCdim( H)=dand let x1,...,xdbe a shattered set. We now construct
a complete binary tree of instances v1,...,v2d−1, where all nodes at
depth iare set to be xi(see the illustration in Figure 3.2). Now,
3.2 Learnability and the Standard Optimal Algorithm 165
x1
x2
x3 x3x2
x3 x3
Fig. 3.2 How to construct a shattered tree from a shattered sequence x1,..., xd.
the deﬁnition of shattered sample clearly implies that we got a valid
shattered tree of depth d, and we conclude that VCdim( H)≤Ldim( H).
To show that the gap can be arbitrarily large simply note that the classgiven in Example 3.3 has VC dimension of 1 whereas its Littlestonedimension is inﬁnite.
3.2.1 The Unrealizable Case
In the previous section we have shown that Littlestone’s dimension
exactly characterizes the achievable mistake bounds in the realizablecase. We now show that the same dimension characterizes online learn-ability in the unrealizable case as well. Speciﬁcally, we will prove thefollowing.
Theorem 3.6. For any hypothesis class H, there exists an online learn-
ing algorithm such that for any h∈Hand any sequence of Texamples
we have
T/summationdisplay
t=1|pt−yt|−T/summationdisplay
t=1|h(xt)−yt|≤O(/radicalbig
Ldim( H) ln(T)T),
where ptis the learner’s prediction on round t. Furthermore,
no algorithm can achieve an expected regret bound smaller thanΩ(/radicalbig
Ldim( H)T).
Recall that in the unrealizable case, to sidestep Cover’s impossi-
bility result, the learner is allowed to make randomized predictionsand we analyze his expected regret. We will construct a generic onlinealgorithm that has the expected regret bound/radicalbig
Ldim( H) log(T)T.
166 Online Classiﬁcation
Finally, we provide an almost matching lower bound on the achievable
regret.
Our starting point is the Weighted Majority algorithm, whose
regret depends on log( d). In the case of a ﬁnite hypothesis class, we
let each hypothesis be an expert. In the unrealizable case, the mainidea is to construct a set of experts in a more sophisticated way. Thechallenge is how to deﬁne a set of experts that on one hand is notexcessively large while on the other hand contains experts that giveaccurate predictions.
We construct the set of experts so that for each hypothesis h∈H
and every sequence of instances, x
1,x2,...,xT, there exists at least one
expert in the set which behaves exactly as hon these instances. For each
L≤Ldim( H) and each sequence 1 ≤i1<i2<···<iL≤Twe deﬁne
an expert. The expert simulates the game between SOA (presented inthe previous section) and the environment on the sequence of instancesx
1,x2,...,xTassuming that SOA makes a mistake precisely in rounds
i1,i2,...,i L. The expert is deﬁned by the following algorithm.
Expert( i1,i2,...,i L)
input A hypothesis class H; Indices i1<i2<···<iL
initialize :V1=H
fort=1,2,...,T
receive xt
forr∈{0,1}letV(r)
t={h∈Vt:h(xt)=r}
deﬁne ˜ yt= argmaxrLdim/parenleftBig
V(r)
t/parenrightBig
(in case of a tie set ˜ yt=0 )
ift∈{i1,i2,...,i L}
predict ˆ yt=¬˜yt
else
predict ˆ yt=˜yt
update Vt+1=V(ˆyt)
t
The following key lemma shows that, on any sequence of instances,
for each hypothesis h∈Hthere exists an expert with the same
behavior.
3.2 Learnability and the Standard Optimal Algorithm 167
Lemma 3.7. LetHbe any hypothesis class with Ldim( H)<∞. Let
x1,x2,...,xTbe any sequence of instances. For any h∈H, there exists
L≤Ldim( H) and indices 1 ≤i1<i2<···<iL≤Tsuch that when
running Expert( i1,i2,...,i L) on the sequence x1,x2,...,xT, the expert
predicts h(xt) on each online round t=1,2,...,T .
Proof. Fixh∈Hand the sequence x1,x2,...,xT. We must construct
Land the indices i1,i2,...,i L. Consider running SOA on the input
(x1,h(x1)), (x2,h(x2)),...,(xT,h(xT)). SOA makes at most Ldim( H)
mistakes on such input. We deﬁne Lto be the number of mistakes made
by SOA and we deﬁne {i1,i2,...,i L}to be the set of rounds in which
SOA made the mistakes.
Now, consider the Expert( i1,i2,...,i L) running on the
sequence x1,x2,...,xT. By construction, the set Vtmaintained
by Expert( i1,i2,...,i L) equals to the set Vtmaintained by SOA
when running on the sequence ( x1,h(x1)),...,(xT,h(xT)). Since the
predictions of SOA diﬀer from the predictions of hif and only if
the round is in {i1,i2,...,i L}, we conclude that the predictions of
Expert( i1,i2,...,i L) are always the same as the predictions of h.
The above lemma holds in particular for the hypothesis in Hthat
makes the least number of mistakes on the sequence of examples, andwe therefore obtain the following:
Corollary 3.8. Let ( x1,y1),(x2,y2),...,(xT,yT) be a sequence of
examples and let Hbe a hypothesis class with Ldim( H)<∞. There
exists L≤Ldim( H) and indices 1 ≤i1<i2<···<iL≤T, such that
Expert( i1,i2,...,i L) makes at most as many mistakes as the best h∈H
does. Namely,
min
h∈HT/summationdisplay
t=1|h(xt)−yt|
mistakes on the sequence of examples.
168 Online Classiﬁcation
Our generic online learning algorithm is now an application of
theWeighted Majority algorithm with the constructed experts. To
analyze the algorithm we combine Corollary 3.8 with the upper boundon the number of experts,
d=Ldim( H)/summationdisplay
L=0/parenleftbiggT
L/parenrightbigg
≤(eT/Ldim( H))Ldim( H), (3.1)
and with Theorem 3.1. This proves the upper bound part of Theo-
rem 3.6. The proof of the lower bound part can be found in [7].
3.3 Perceptron and Winnow
In this section we describe two classic online learning algorithms forbinary classiﬁcation with the hypothesis class of halfspaces. Through-out this section, it is more convenient to let the labels set be Y={−1,1}
instead of Y={0,1}. Each halfspace hypothesis can be described using
a vector, often called a weight vector. For example, if the vector spaceis the two dimensional Euclidean space (the plane), then instances arepoints in the plane and hypotheses are lines. The weight vector is per-pendicular to the line. The prediction is according to whether the pointfalls on one side of the line or on the other side. See Figure 3.3 for anillustration.
Fig. 3.3 An illustration of linear separators in the plane ( R2). The solid black line separates
the plane into two regions. The circled point represents an input vector which is labeled 1 by
the linear separator. The arrow designates a weight vector that represents the hypothesis.
3.3 Perceptron and Winnow 169
On round t, the learner receives a vector xt∈Rd. The learner main-
tains a weight vector wt∈Rdand predicts pt= sign( /angbracketleftwt,xt/angbracketright). Then, it
receives yt∈Yand pays 1 if pt/negationslash=ytand 0 otherwise.
The goal of the learner is to make as few prediction mistakes as
possible. In the previous section we characterized the optimal algo-rithm and showed that the best achievable regret bound depends onthe Littlestone dimension of the class. In our case, the class of half-spaces is the class H={x/mapsto→sign(/angbracketleftw,x/angbracketright):w∈R
d}. We show below
that if d≥2 then Ldim( H)=∞, which implies that we have no hope
to make few prediction mistakes. Indeed, consider the tree for whichv
1=(1
2,1,0,...,0),v2=(1
4,1,0,...,0),v3=(3
4,1,0,...,0), etc. Due to
the density of the reals, this tree is shattered by the subset of Hwhich
contains all hypotheses that are parametrized by wof the form w=
(−1,a,0,...,0), for a∈[0,1]. We conclude that indeed Ldim( H)=∞.
To sidestep this impossibility result, the Perceptron and Winnow
algorithms rely on the technique of surrogate convex losses we discussed
in Section 2.1.2. We now derive these algorithms.
3.3.1 Perceptron
A weight vector wmakes a mistake on an example ( x,y) whenever the
sign of /angbracketleftw,x/angbracketrightdoes not equal to y. Therefore, we can write the 0–1 loss
function as follows:
/lscript(w,(x,y)) = 1[y/angbracketleftw,x/angbracketright≤0].
On rounds on which the algorithm makes a prediction mistake, we shall
deﬁne the following surrogate convex loss function
ft(w)=[ 1 −yt/angbracketleftw,xt/angbracketright]+,
where [ a]+= max {a,0}is the hinge function. This loss function is often
called the hinge-loss . It satisﬁes the two conditions:
•ftis a convex function
•For all w,ft(w)≥/lscript(w,(xt,yt)). In particular, this holds for
wt.
170 Online Classiﬁcation
On rounds on which the algorithm is correct, we shall deﬁne ft(w)=0 .
Clearly, ftis convex in this case as well. Furthermore, ft(wt)=
/lscript(wt,(xt,yt) )=0 .
Let us now run the Online Gradient Descent (OGD) algorithm
on the sequence of functions. Recall that OGD initializes w1=0and
its update rule
wt+1=wt−ηzt
for some zt∈∂ft(wt). In our case, if yt/angbracketleftwt,xt/angbracketright>0 then ftis the zero
function and we can take zt=0. Otherwise, it is easy to verify that
zt=−ytxtis in∂ft(wt). We therefore obtain the update rule
wt+1=/braceleftbiggwt ifyt/angbracketleftwt,xt/angbracketright>0
wt+ηytxtotherwise
Denote by Mthe set of rounds in which sign( /angbracketleftwt,xt/angbracketright)/negationslash=yt. Note that
on round t, the prediction of the Perceptron can be rewritten as,
pt= sign( /angbracketleftwt,xt/angbracketright) = sign/parenleftBigg/summationdisplay
i∈M:i<tyi/angbracketleftxi,xt/angbracketright/parenrightBigg
.
The above form implies that the predictions of the Perceptron
algorithm and the set Mdo not depend on the actual value of ηas
long as η>0. We have therefore obtained the well known Perceptron
algorithm.
Perceptron
initialize :w1=0
fort=1,2,...,T
receive xt
predict pt= sign( /angbracketleftwt,xt/angbracketright)
ifyt/angbracketleftwt,xt/angbracketright≤0
wt+1=wt+ytxt
else
wt+1=wt
To analyze the Perceptron, we rely on the analysis of OGD given
in the previous section. In particular, we rely on Corollary 2.7 and
3.3 Perceptron and Winnow 171
its improved version is given in Equation (2.15). In our case, the
sub-gradients of ftwe use in the Perceptron are zt=−1[yt/angbracketleftwt,xt/angbracketright≤0]ytxt.
Indeed, the Perceptron’s update is wt+1=wt−zt, and as discussed
before this is equivalent to wt+1=wt−ηztfor any η>0. Therefore,
the bound given in Equation (2.15) tells us that
T/summationdisplay
t=1ft(wt)−T/summationdisplay
t=1ft(u)≤1
2η/bardblu/bardbl2
2+η
2T/summationdisplay
t=1/bardblzt/bardbl2
2.
Since ft(wt) is a surrogate for the 0–1 loss we know that/summationtextT
t=1ft(wt)≥
|M|. Denote R= max t/bardblxt/bardbl, then we obtain
|M| −T/summationdisplay
t=1ft(u)≤1
2η/bardblu/bardbl2
2+η
2|M|R2
Setting η=/bardblu/bardbl
R√
|M|and rearranging, we obtain
|M| − R/bardblu/bardbl/radicalbig
|M| −T/summationdisplay
t=1ft(u)≤0. (3.2)
This inequality implies:
Theorem 3.9. Suppose that the Perceptron algorithm runs on
a sequence ( x1,y1),...,(xT,yT) and let R= max t/bardblxt/bardbl. Let M
be the rounds on which the Perceptron errs and let ft(w)=
1[t∈M][1−yt/angbracketleftw,xt/angbracketright]+. Then, for any u
|M| ≤/summationdisplay
tft(u)+R/bardblu/bardbl/radicalBigg/summationdisplay
tft(u)+R2/bardblu/bardbl2.
In particular, if there exists usuch that yt/angbracketleftu,xt/angbracketright≥1 for all tthen
|M| ≤ R2/bardblu/bardbl2.
Proof. The theorem follows from Equation (3.2) and the following
claim: Given x,b,c∈R+, the inequality x−b√x−c≤0 implies that
x≤c+b2+b√c. The last claim can be easily derived by analyzing
the roots of the convex parabola Q(y)=y2−by−c.
172 Online Classiﬁcation
The last assumption of Theorem 3.9 is called separability with large
margin. That is, there exists uthat not only satisﬁes that the point xt
lies on the correct side of the halfspace, it also guarantees that xtis
not too close to the decision boundary. More speciﬁcally, the distancefrom x
tto the decision boundary is at least γ=1//bardblu/bardbland the bound
becomes ( R/γ)2. This classic mistake bound of the Perceptron appears
in [3, 33].
When the separability assumption does not hold, the bound involves
the terms [1 −yt/angbracketleftu,xt/angbracketright]+which measures how much the separability
with margin requirement is violated.
As a last remark we note that there can be cases in which there
exists some uthat makes zero errors on the sequence but the Per-
ceptron will make many errors. Indeed, this is a direct consequenceof the fact that Ldim( H)=∞. The way we sidestep this impossibility
result is by assuming more on the sequence of examples — the bound inTheorem 3.9 will be meaningful only if/summationtext
tft(u) is not excessively large.
3.3.2 Winnow
Winnow is an online classiﬁcation algorithm originally proposed for
learning the class of kmonotone disjunctive Boolean functions. Namely,
X={0,1}dand a kmonotone disjunction hypothesis takes the form
x[i1]∨...∨x[ik], where {i1,...,i k}⊂[d]. This can be written as a half-
space as follows. Let w∈{0,1}dbe a vector with exactly kelements
that equal 1 (we call these elements the relevant variables). Then, /angbracketleftw,x/angbracketright
will be at least 1 if one of the relevant variables is turned on in x.
Otherwise, /angbracketleftw,x/angbracketrightis 0. So, sign( /angbracketleftw,x/angbracketright−1/2) behaves exactly like a k
monotone disjunction and our hypothesis class is:
H={x/mapsto→sign(/angbracketleftw,x/angbracketright−1/2) :w∈{0,1}d,/bardblw/bardbl1=k}.
In order to learn Hwe need to convexify the learning problem.
The ﬁrst step is to convexify the domain of wby simply enlarging
it to be S=Rd
+. The second step is to construct a surrogate convex
loss function, similarly to the one we used for deriving the Perceptron.A weight vector werrs on ( x,y) if sign( /angbracketleftw,x/angbracketright−1/2)/negationslash=y, or equiva-
lently if y(2/angbracketleftw,x/angbracketright−1)≤0. Therefore, the original 0–1 loss function is
/lscript(w,(x,y)) = 1
[y(2/angbracketleftw,x/angbracketright−1)≤0].
3.3 Perceptron and Winnow 173
On rounds on which the algorithm errs, we deﬁne the hinge-loss
surrogate
ft(w)=[ 1 −yt(2/angbracketleftw,xt/angbracketright−1)]+.
If the algorithm does not err we set ft(w) = 0. This function satisﬁes:
•ftis convex.
•ftis a surrogate: ft(wt)≥/lscript(wt,(xt,yt)).
•In the realizable case, there exists u∈{0,1}dwith/bardblu/bardbl1=k
such that ft(u) = 0 for all t.
Before we derive the Winnow algorithm, we note that we can use
the Perceptron algorithm for learning this problem as follows. Denoteφ(x) to be the vector [2 x,−1], namely, we concatenate the constant
−1 to the vector 2 x. Then, the prediction can be performed according
to sign( /angbracketleftw,φ(x)/angbracketright) and the loss f
t(w) becomes the hinge-loss. In the
realizable case, this yields the mistake bound R2/bardblu/bardbl2
2=4 (d+1 )k.W e
shall see that the mistake bound of Winnow is 8 klog(d). That is, we
obtain an exponential improvement in terms of the dependence on thedimension.
Winnow is a specialization of the unnormalized-EG algorithm, given
in Section 2.8, to the aforementioned surrogate loss. We set the param-eterλof the unnormalized-EG algorithm to be 1 /d.
Winnow
parameter :η>0
initialize :w1=( 1/d,..., 1/d)
fort=1,2,...,T
receive xt
predict pt= sign(2 /angbracketleftwt,xt/angbracketright−1)
ifyt(2/angbracketleftwt,xt/angbracketright−1)≤0
∀i, wt+1[i]=wt[i]e−η2ytxt[i]
else
wt+1=wt
174 Online Classiﬁcation
Theorem 3.10. Suppose that the Winnow algorithm runs with a
parameter η≤1/2 on a sequence ( x1,y1),...,(xT,yT), where xt∈
{0,1}dfor all t. Let Mbe the rounds on which Winnow errs and let
ft(w)=1[t∈M][1−yt(2/angbracketleftw,xt/angbracketright−1)]+. Then, for any u∈{0,1}dsuch
that/bardblu/bardbl1=kit holds that
|M| ≤T/summationdisplay
t=1ft(wt)≤1
1−2η/parenleftBiggT/summationdisplay
t=1ft(u)+klog(d)
η/parenrightBigg
.
In particular, if there exists such ufor which yt(2/angbracketleftu,xt/angbracketright−1)≥1 for
alltthen we can set η=1/4 and obtain
|M| ≤ 8klog(d).
Proof. LetMbe the set of rounds on which Winnow errs, i.e., M={t:
yt(2/angbracketleftwt,xt/angbracketright−1)≤0}. Winnow is derived from the unnormalized-EG
algorithm with
zt=/braceleftbigg2ytxtift∈M
0 ift/∈M
Using Theorem 2.23 (with λ=1/d)w eh a v e
T/summationdisplay
t=1/angbracketleftwt−u,zt/angbracketright≤1+/summationtext
iu[i]log(du[i]/e)
η+ηT/summationdisplay
t=1/summationdisplay
iwt[i]zt[i]2.
In our case, u∈{0,1}dand/bardblu/bardbl1=kso
1+/summationdisplay
iu[i]log(du[i]/e)≤1+klog(d/e)≤klog(d).
Plugging this in the above and using the fact that ztis a sub-gradient
offtatwtwe obtain:
T/summationdisplay
t=1(ft(wt)−ft(u))≤T/summationdisplay
t=1/angbracketleftwt−u,zt/angbracketright≤klog(d)
η+ηT/summationdisplay
t=1/summationdisplay
iwt[i]zt[i]2.
(3.3)
3.4 Bibliographic Remarks 175
We next show that for all t,
/summationdisplay
iwt[i]zt[i]2≤2ft(wt). (3.4)
To do so, it is convenient to consider three cases:
•Ift/∈Mthenft(wt)=0a n d zt=0. Hence, Equation (3.4)
holds.
•Ift∈Mandyt= 1 then the left-hand side of Equation (3.4)
becomes 4 /angbracketleftwt,xt/angbracketright. This is because xt∈{0,1}dsoxt[i]2=
xt[i]. But since t∈Mwe also know that 2 /angbracketleftwt,xt/angbracketright≤1 and so/summationtext
iwt[i]zt[i]2≤2. On the other hand, by the surrogate prop-
erty of ftand the fact that t∈Mwe know that ft(wt)≥1,
which yields Equation (3.4).
•Ift∈M andyt=−1 then the left-hand side of Equa-
tion (3.4) becomes again 4 /angbracketleftwt,xt/angbracketright. However, now we have
thatft(wt)=2/angbracketleftwt,xt/angbracketright, which yields Equation (3.4).
Combining Equation (3.4) with Equation (3.3) and rearranging terms
we obtain
T/summationdisplay
t=1ft(wt)≤1
1−2η/parenleftBiggT/summationdisplay
t=1ft(u)+klog(d)
η/parenrightBigg
.
Combining the above with the surrogate property we conclude our
proof.
3.4 Bibliographic Remarks
The Weighted Majority algorithm is due to [32] and [43]. The StandardOptimal Algorithm was derived by the seminal work of Littlestone [29].A generalization to the nonrealizable case as well as other variants likemargin-based Littlestone’s dimension was derived in [7]. Characteriza-tions of online learnability beyond classiﬁcation has been obtained in[1, 35].
The Perceptron dates back to Rosenblatt [37]. An analysis for the
realizable case (with margin assumptions) appears in [3, 33]. Freundand Schapire [17] presented an analysis for the unrealizable case with a
176 Online Classiﬁcation
squared-hinge-loss based on a reduction to the realizable case. A direct
analysis for the unrealizable case with the hinge-loss was given byGentile [18]. Winnow was invented and analyzed in the realizable caseby Littlestone [29]. An analysis for the unrealizable case was carriedout in [5].
4
Limited Feedback (Bandits)
In Section 2 we studied the general framework of online convex opti-
mization and in particular, derived the family of online mirror descentalgorithms. To apply this algorithm, it is required to ﬁnd a sub-gradientof the loss function at the end of each round. In this section we studyonline learning problems where the learner knows the value of the lossfunction at the predicted vector but he doesn’t know the value of theloss function at other points.
We ﬁrst show that to apply the online mirror descent framework
it suﬃces to know how to calculate an estimate of the gradient. Wenext show how this observation leads to a low regret algorithm for afamous problem called “the multi-armed bandit problem.” This prob-lem is similar to prediction with expert advice but at the end of eachround the learner only observes the cost of the expert he picked anddoes not observe the costs of the rest of the experts.
Finally, we discuss the general problem of online convex optimiza-
tion without gradient information.
177
178 Limited Feedback (Bandits)
4.1 Online Mirror Descent with Estimated Gradients
Recall the Online Mirror Descent (OMD) algorithm we described in
Section 2. Now suppose that instead of setting ztto be a sub-gradient
offtatwt, we shall set ztto be a random vector with E[zt]∈∂ft(wt).
Online Mirror Descent with Estimated
Gradients
parameter: a link function g:Rd→S
initialize: θ1=0
fort=1,2,...
predict wt=g(θt)
pick ztat random such that E[zt|zt−1,...,z1]∈∂ft(wt)
update θt+1=θt−zt
The following theorem tells us how to extend previous regret bounds
we derived for OMD to the case of estimated sub-gradients.
Theorem 4.1. Suppose that the Online Mirror Descent with
Estimated Gradients is run on a sequence of loss functions, f1,...,f T.
Suppose that the estimated sub-gradients are chosen such that withprobability 1 we have
T/summationdisplay
t=1/angbracketleftwt−u,zt/angbracketright≤B(u)+T/summationdisplay
t=1/bardblzt/bardbl2
t,
where Bis some function and for all round tthe norm /bardbl·/bardbltmay depend
onwt. Then,
E/bracketleftBiggT/summationdisplay
t=1(ft(wt)−ft(u))/bracketrightBigg
≤B(u)+T/summationdisplay
t=1E[/bardblzt/bardbl2
t],
where expectation is with respect to the randomness in choosing
z1,...,zT.
Proof. Taking expectation of both sides of the ﬁrst inequality with
respect to the randomness in choosing ztwe obtain that
E/bracketleftBiggT/summationdisplay
t=1/angbracketleftwt−u,zt/angbracketright/bracketrightBigg
≤B(u)+T/summationdisplay
t=1E[/bardblzt/bardbl2
t].
4.2 The Multi-armed Bandit Problem 179
At each round, let vt=E[zt|zt−1,...,z1]. By the law of total probability
we obtain that
E/bracketleftBiggT/summationdisplay
t=1/angbracketleftwt−u,zt/angbracketright/bracketrightBigg
=E/bracketleftBiggT/summationdisplay
t=1/angbracketleftwt−u,vt/angbracketright/bracketrightBigg
.
Since we assume that vt∈∂ft(wt) we know that
/angbracketleftwt−u,vt/angbracketright≥ft(wt)−ft(u).
Combining all the above we conclude our proof.
The above theorem tells us that as long as we can ﬁnd z1,...,zTwhich
on one hand are unbiased estimators of sub-gradients and on the otherhand has bounded norms, we can still obtain a valid regret bound. Inthe next section we demonstrate how to construct such estimates for aspeciﬁc problem and in Section 4.3 we derive a more general approach.
4.2 The Multi-armed Bandit Problem
In the multi-armed bandit problem, there are darms, and on each
online round the learner should choose one of the arms, denoted pt,
where the chosen arm can be a random variable. Then, it receives a costof choosing this arm, y
t[pt]∈[0,1]. The vector yt∈[0,1]dassociates a
cost for each of the arms, but the learner only gets to see the costof the arm it pulls. Nothing is assumed about the sequence of vectorsy
1,y2,...,yT.
This problem is similar to prediction with expert advice. The only
diﬀerence is that the learner does not get to see the cost of expertshe didn’t choose. The goal of the learner is to have low regret for notalways pulling the best arm,
E/bracketleftBigg
T/summationdisplay
t=1yt[pt]/bracketrightBigg
−min
iT/summationdisplay
t=1yt[i],
where the expectation is over the learner’s own randomness.
This problem nicely captures the exploration–exploitation tradeoﬀ.
On one hand, we would like to pull the arm which, based on previousrounds, we believe has the lowest cost. On the other hand, maybe it isbetter to explore the arms and ﬁnd another arm with a smaller cost.
180 Limited Feedback (Bandits)
To approach the multi-armed bandit problem we use the OMD with
estimated gradients method derived in the previous section. As in theWeighted Majority algorithm for prediction with expert advice, we letSbe the probability simplex and the loss functions be f
t(w)=/angbracketleftw,yt/angbracketright.
The learner picks an arm according to P[pt=i]=wt[i] and therefore
ft(wt) is the expected cost of the chosen arm. The gradient of the loss
function is yt. However, we do not know the value of all elements of yt,
we only get to see the value yt[pt]. To estimate the gradient, we deﬁne
the random vector ztas follows:
zt[j]=/braceleftbiggyt[j]/wt[j]i fj=pt
0 else.
To emphasize the dependence of ztonptwe will sometimes use the
notation z(pt)
t. We indeed have that z(pt)
tis an unbiased estimate of the
gradient because
E[z(pt)
t[j]|zt−1,...,z1]=d/summationdisplay
i=1P[pt=i]z(i)
t[j]=wt[j]yt[j]
wt[j]=yt[j].
We update wtusing the update rule of the normalized EG algorithm.
The resulting algorithm is given below.
Multi-Armed Bandit Algorithm
parameter: η∈(0,1)
initialize :w1=( 1/d,..., 1/d)
fort=1,2,...
choose pt∼wtand pull the pt’th arm
receive cost of the arm yt[pt]∈[0,1]
update
˜w[pt]=wt[pt]e−ηyt[pt]/wt[pt]
fori/negationslash=pt,˜w[i]=wt[i]
∀i, wt+1[i]=˜w[i]/summationtext
j˜w[j]
To analyze the algorithm we combining Theorem 4.1 with Theo-
rem 2.22. The conditions of Theorem 2.22 hold here because ztis a
4.2 The Multi-armed Bandit Problem 181
non-negative vector. We therefore obtain that
T/summationdisplay
t=1/angbracketleftwt−u,zt/angbracketright≤log(d)
η+ηT/summationdisplay
t=1/summationdisplay
iwt[i]zt[i]2.
So Theorem 4.1 gives us that
E/bracketleftBiggT/summationdisplay
t=1(ft(wt)−ft(u))/bracketrightBigg
≤log(d)
η+ηT/summationdisplay
t=1E/bracketleftBigg/summationdisplay
iwt[i]zt[i]2/bracketrightBigg
.
The last term can be bounded as follows:
E/bracketleftBigg/summationdisplay
iwt[i]z(pt)
t[i]2|zt−1,...,z1/bracketrightBigg
=/summationdisplay
jP[pt=j]/summationdisplay
iwt[i]z(j)
t[i]2
=/summationdisplay
jwt[j]wt[j](yt[j]/wt[j])2
=/summationdisplay
jyt[j]2≤d.
Overall, we have obtained the following:
Corollary 4.2. The multi-armed bandit algorithm enjoys the bound
E/bracketleftBiggT/summationdisplay
t=1yt[pt]/bracketrightBigg
≤min
iT/summationdisplay
t=1yt[i]+logd
η+ηdT.
In particular, setting η=/radicalbig
log(d)/(dT) we obtain the regret bound of
2/radicalbig
dlog(d)T.
Comparing the above bound to the bound we derived for the
Weighted Majority algorithm we observe an additional factor of d,
which intuitively stems from the fact that here we only receive 1 /d
of the feedback the Weighted Majority algorithm receives. It is possi-ble to rigorously show that the dependence on dis unavoidable and
that the bound we derived is essentially tight.
182 Limited Feedback (Bandits)
4.3 Gradient Descent Without a Gradient
In this section we consider the general online convex optimization
problem, where we only have a black-box access to the loss func-tions, and thus cannot calculate sub-gradients directly. The multi-armed bandit problem discussed in the previous section was a specialcase, where the loss function was linear. However, in the multi-armedbandit problem we had the additional constraint that we can only askthe value of the loss function at singletons vectors. Here we allow thelearner to predict any vector
1but he only receives the evaluation of the
loss function on the vector he picked.
In Section 2 we derived several gradient-based algorithms for online
convex optimization. How can we utilize these algorithms when we canonly receive the evaluation of the loss function on the predicted vector?As in the multi-armed bandit problem, the main idea is to derive aone-shot estimate to the gradient. Below we present a method due toFlaxman, Kalai, and McMahan [16], which derives a one-shot estimateto the gradient.
4.3.1 A One-Shot Gradient Estimate
LetU
bbe the uniform distribution over the unit Euclidean ball and let
Uspbe the uniform distribution over the unit Euclidean sphere. Given
δ>0 we deﬁne a smoothed version of fas follows:
ˆf(w)= E
v∼Ub[f(w+δv)]. (4.1)
As we will show below, the advantage of ˆfis that it is diﬀerentiable
and we can estimate its gradient using a single oracle call to f. But,
before that, we show that ˆfis similar to f.
Lemma 4.3. Letfbe an L-Lipschitz function and let ˆfbe as deﬁned
in Equation (4.1). Then, |ˆf(w)−f(w)|≤Lδ.
1We make another simpliﬁcation, for the sake of simplicity, that the learner does not have
to predict an element from S. In some applications, like the multi-armed bandit problem,
the learner must predict an element from Sand then a more careful analysis is required.
See for example [16].
4.3 Gradient Descent Without a Gradient 183
Proof. By the Lipschitzness, |f(w)−f(w+δv)|≤Lδ/bardblv/bardbl. Using the
fact that /bardblv/bardbl≤1 we obtain the desired result.
We next show that the gradient of ˆfcan be estimated using a single
oracle call to f.
Lemma 4.4. The function ˆfis diﬀerentiable and we have
E
v∼Usp/bracketleftbiggd
δf(w+δv)v/bracketrightbigg
=∇ˆf(w).
Proof. We prove the theorem for the case d= 1. Given a 1-dimensional
function f, letFbe the antiderivative of f, namely, F/prime(w)=f(w). By
the fundmental theorem of calculus we have
/integraldisplayb
af(w)dw=F(b)−F(a).
It follows that
/integraldisplayδ
−δf(w+t)dt=F(w+δ)−F(w−δ).
Note that in the 1-dimensional case we have that vis distributed
uniformly over [ −1,1] and
ˆf(w)= E[f(w+δv)] =/integraltextδ
−δf(w+t)dt
2δ=F(w+δ)−F(w−δ)
2δ.
It follows that
ˆf/prime(w)=f(w+δ)−f(w−δ)
2δ=Ev∼Usp[f(w+δv)v]
δ.
This concludes the proof for the case d= 1. The proof for d>1 follows
similarly using Stoke’s theorem and can be found in [16].
4.3.2 The Resulting Algorithm
The algorithm is based on gradient descent with lazy pro-
jections , we described in Section 2.6.
184 Limited Feedback (Bandits)
Bandit Online Gradient Descent
parameters: η,δ > 0 and a convex set S⊂Rd
initialize: θ1=0
fort=1,2,...,T
letwt= argminw∈S/bardblw−ηθt/bardbl2
pick vt∼Usp
predict wt+δvtand receive ft(wt+δvt)
setzt=d
δft(wt+δvt)vt
update θt+1=θt−zt
Let us now analyze the regret of this algorithm. First, applying
Corollary 2.17 we obtain that for all u∈S,
/summationdisplay
t/angbracketleftzt,wt−u/angbracketright≤1
2η/bardblu/bardbl2
2+ηT/summationdisplay
t=1/bardblzt/bardbl2.
Taking expectation, using Lemma 4.4, and using the fact that vtis in
the unit sphere, we obtain
E/bracketleftBigg/summationdisplay
t(ˆft(wt)−ˆft(u))/bracketrightBigg
≤1
2η/bardblu/bardbl2
2+ηT/summationdisplay
t=1d2
δ2E[ft(wt+δvt)2].(4.2)
LetB= max u∈S/bardblu/bardblandF= max u∈S,tft(u). Then, by the Lipschitz-
ness of ftwe have ft(wt+δvt)≤ft(wt)+Lδ/bardblvt/bardbl≤F+Lδ. Combin-
ing with Equation (4.2) yields
E/bracketleftBigg/summationdisplay
t(ˆft(wt)−ˆft(u))/bracketrightBigg
≤B2
2η+ηTd2(F/δ+L)2. (4.3)
To derive a concrete bound out of the above we need to relate the
regret with respect to ˆftto the regret with respect to ft. We do this
using Lemma 4.3, which implies
ft(wt+δvt)−ft(u)≤ft(wt)−ft(u)+Lδ≤ˆft(wt)−ˆft(u)+3Lδ.
Combining the above with Equation (4.3) yields
Corollary 4.5. Consider running the Bandit Online Convex
Optimization algorithm on a sequence f1,...,f TofL-Lipschitz
4.4 Bibliographic Remarks 185
functions. Let Sbe a convex set and deﬁne B= max u∈S/bardblu/bardbland
F= max u∈S,t∈[T]ft(u). Then, for all u∈Swe have
E/bracketleftBigg/summationdisplay
t(ft(wt+δvt)−ft(u))/bracketrightBigg
≤3LδT+B2
2η+ηTd2(F/δ+L)2.
In particular, setting η=B
d(F/δ+L)√
2T,δ=/radicalBig
BdF
3LT−1/4we obtain that
the regret is bounded by O(√
BdFLT3/4).
Comparing this bound to the full information bound given in Corol-
lary 2.17, we note two main diﬀerence. First, the dependence on Tis
T3/4as opposed to T1/2in the full information case. Second, the regret
depends on the dimensionality. While the dependence on the dimen-sionality is indeed tight, it is not known if the worse dependence on T
is tight.
4.4 Bibliographic Remarks
Multi-armed bandit problems were originally studied in a stochasticsetting [36]. The adversarial multi-armed bandit problem was studiedin [4].
The algorithm we described for bandit online convex optimization
is a variant of the algorithm given in [16]. Better algorithms have beenderived for the speciﬁc case of linear functions. In particular, [15] gave
a non eﬃcient algorithm whose regret is O(poly( d)√
T), and later on,
[2] derived an eﬃcient algorithm with a similar regret bound usingself-concordant regularizers.
5
Online-to-Batch Conversions
In this section we discuss conversions from online learning to stochas-
tic learning. In particular, we consider the following general model ofstochastic learning.
Deﬁnition 5.1.(Vapnik’s General Setting of Learning) LetSbe
a hypothesis class and let Ψ be an example domain set. There is a lossfunction, c:S×Ψ→R, which gets a hypothesis and an example and
returns the cost of using the hypothesis on the example. Let Qbe an
unknown distribution over Ψ and deﬁne
1C(w)= Eψ∼Q[c(w,ψ)]. The
goal of the learner is to approximately minimize CoverS. The learner
does not know Q, but can get independent samples from Q. We denote
byA(ψ1,...,ψ T) the output of the learning algorithm when receiving
Tsamples from Q. We say that a learning algorithm /epsilon1-learns Susing T
1Technically, in the above deﬁnition, for every w∈S, we view the function c(w,·):Ψ→R+
as a random variable, and deﬁne C(w) to be the expected value of this random variable.
For that, we need to require that the function c(w,·) is measurable. Formally, we assume
that there is a σ-algebra of subsets of Ψ, over which the probability Qis deﬁned, and that
the pre-image of every initial segment in R+is in this σ-algebra.
186
187
examples if
E[C(A(ψ1,...,ψ T))]≤min
w∈SC(w)+/epsilon1,
where expectation2is over the random choice of ψ1,...,ψ T.
This model is also closely related to the problem of Stochastic Opti-
mization or more speciﬁcally Stochastic Approximation (for moredetails see for example [41]).
Maybe the most simple algorithm for solving a stochastic learning
problem is by sampling Texamples, ψ
1,...,ψ T, and then returning a
hypothesis which minimizes the average cost over these examples. Thisis called empirical risk minimization (ERM):
ERM
S(ψ1,...,ψ T)∈argmin
w∈S1
TT/summationdisplay
t=1c(w,ψt).
Our goal is to demonstrate how online learning can sometimes yield an
alternative algorithm for solving stochastic learning problems.
The basic idea is as follows. First, we run the online learner on
a sequence of loss functions, where ft(w)=c(w,ψt). This produces a
sequence of predictions w1,...,wT. Second, we produce a single vector
¯wbased on w1,...,wT. There are many ways to produce ¯wand here
we consider two simple approaches: averaging or randomization. Thisyields the following skeleton.
Online-To-Batch Conversion
parameters: a set S; a cost function c(·,·);
an online learning algorithm A
input: ψ1,...,ψ Tare independently sampled from distribution Qover Ψ
fort=1,...,T
letwtbe the prediction of A
provide the loss function ft(w)=c(w,ψt)t oA
output: produce ¯wfrom w1,..., wT. For example:
averaging: ¯w=1
T/summationtextT
t=1wt
randomization: ¯w=wr,
where ris chosen uniformly at random from [ T]
2It is common to require that the inequality will hold with high probability. By a simple
ampliﬁcation argument it is possible to convert a guarantee on the expected value to aguarantee that holds with high probability. Therefore, for the sake of simplicity, we onlyrequire success in expectation.
188 Online-to-Batch Conversions
The following theorem shows that the average online loss of the
online learner upper bounds the cost of ¯w.
Theorem 5.1. Letψ1,...,ψ Tbe a sequence of independent random
variables, each of which is distributed according to a distribution Q
over Ψ. For the online-to-batch conversion with randomiza-
tion, we have that
E[C(¯w)] = E/bracketleftBigg
1
TT/summationdisplay
t=1ft(wt)/bracketrightBigg
,
where expectation is with respect to the random choice of ψ1,...,ψ T
andr. If we further assume that Cis a convex function, then for
online-to-batch conversion with averaging we have that
E[C(¯w)]≤E/bracketleftBigg
1
TT/summationdisplay
t=1ft(wt)/bracketrightBigg
.
Proof. We ﬁrst show that
E/bracketleftBigg
1
TT/summationdisplay
t=1C(wt)/bracketrightBigg
=E/bracketleftBigg
1
TT/summationdisplay
t=1c(wt,ψt)/bracketrightBigg
. (5.1)
Using the linearity of expectation we have,
E/bracketleftBigg
1
TT/summationdisplay
t=1c(wt,ψt)/bracketrightBigg
=1
TT/summationdisplay
t=1E[c(wt,ψt)]. (5.2)
Recall that the law of total expectation implies that for any two random
variables R1,R2, and a function f,ER1[f(R1)] = ER2ER1[f(R1)|R2].
Since wtonly depends on ψ1,...,ψ t−1, we can set R1=ψ1,...,ψ tand
R2=ψ1,...,ψ t−1to get that
E[c(wt,ψt)] = E[C(wt)].
Combining the above with Equation (5.2) yields Equation (5.1). Now,
for the randomization technique we have that C(¯w)= Er[C(wr)] =
1
T/summationtextT
t=1C(wt), which concludes our proof for the case of the
189
randomization technique. For the averaging technique, since we assume
thatCis convex we can apply Jensen’s inequality to get that
C(¯w)=C/parenleftBigg
1
TT/summationdisplay
t=1wt/parenrightBigg
≤1
TT/summationdisplay
t=1C(wt).
Clearly, E[1
T/summationtextT
t=1c(u,ψt)] =C(u). Therefore,
Corollary 5.2. Assume that the conditions of Theorem 5.1 hold and
letube any vector, then
E[C(¯w)]−C(u)≤E/bracketleftBigg
1
TT/summationdisplay
t=1(ft(wt)−ft(u))/bracketrightBigg
.
The right-hand side is the expected regret of the online algorithm
(divided by T). Therefore, if we have an online learning algorithm that
is guaranteed to have a low regret with respect to functions of the formf(w)=c(w,ψ) relative to a set S, we can be assured that the cost of
¯wis close to the optimal cost.
To demonstrate the usage of this approach, we describe the appli-
cability of online mirror descent to stochastic learning of problems inwhich cis convex with respect to its ﬁrst argument.
Stochastic Online Mirror Descent
parameters: a link function g:Rd→S; number of rounds T
input: sampler from distribution Qover Ψ; cost function c(·,·)
initialize: θ1=0
fort=1,...,T
predict wt=g(θt)
pickψt∼Qand let ft(w)=c(w,ψt)
letzt∈∂ft(wt)
update θt+1=θt−zt
output: ¯w=1
T/summationtextT
t=1wt
190 Online-to-Batch Conversions
5.1 Bibliographic Remarks
Littlestone [30] initiated the study of online to batch conversions.
Freund and Schapire [17] demonstrated that using the Perceptron algo-rithm along with the voting online-to-batch technique [23] yields aneﬃcient replacement for Support Vector Machines [14, 38, 42]. Concen-tration bounds on the risk of the ensemble that rely on the online losswere derived by Zhang [45] and Cesa-Bianchi et al. [10, 11].
Acknowledgments
The author would like to thank the anonymous reviewers for their
helpful comments.
191
References
[1] J. Abernethy, P. L. Bartlett, A. Rakhlin, and A. Tewari, “Optimal strategies
and minimax lower bounds for online convex games,” in Procee dings of the
Annual Conference on Computational Learning Theory , 2008.
[2] J. Abernethy, E. Hazan, and A. Rakhlin, “Competing in the dark: An eﬃ-
cient algorithm for bandit linear optimization,” in Procee dings of the Annual
Conference on Learning Theory (COLT) , no. 3, 2008.
[3] S. Agmon, “The relaxation method for linear inequalities,” Canadian Journal
of Mathematics , vol. 6, no. 3, pp. 382–392, 1954.
[4] P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire, “The nonstochastic
multiarmed bandit problem,” SICOMP: SIAM Journal on Computing , vol. 32,
no. 1, pp. 48–77, 2003.
[5] P. Auer and M. Warmuth, “Tracking the best disjunction,” Machine Learning ,
vol. 32, no. 2, pp. 127–150, 1998.
[6] K. Azoury and M. Warmuth, “Relative loss bounds for on-line density estima-
tion with the exponential family of distributions,” Machine Learning , vol. 43,
no. 3, pp. 211–246, 2001.
[7] S. Ben-David, D. Pal, and S. Shalev-Shwartz, “Agnostic online learning,” in
Procee dings of the Annual Conference on Learning Theory (COLT) , 2009.
[8] A. Blum, “On-line algorithms in machine learning,” Online Algorithms ,
pp. 306–325, 1998.
[9] J. Borwein and A. Lewis, Convex Analysis and Nonlinear Optimization .
Springer, 2006.
[10] N. Cesa-Bianchi, A. Conconi, and C. Gentile, “On the generalization ability
of on-line learning algorithms,” IEEE Transactions on Information Theory ,
vol. 50, no. 9, pp. 2050–2057, 2004.
192
References 193
[11] N. Cesa-Bianchi and C. Gentile, “Improved risk tail bounds for on-line algo-
rithms,” Neural Information P roceesing Systems (NIPS) , 2006.
[12] N. Cesa-Bianchi and G. Lugosi, Prediction, Learning, and Games . Cambridge
University Press, 2006.
[13] T. M. Cover, “Behavior of sequential predictors of binary sequences,” in Trans-
actions on Prague Conference on Information Theory Statistical Decision Func-tions, Random P rocesses , pp. 263–272, 1965.
[14] N. Cristianini and J. Shawe-Taylor, An Introduction to Support Vector
Machines . Cambridge University Press, 2000.
[15] V. Dani, T. Hayes, and S. M. Kakade, “The price of bandit information
for online optimization,” Advances in Neural Information P rocessing Systems ,
vol. 20, pp. 345–352, 2008.
[16] A. D. Flaxman, A. T. Kalai, and H. B. McMahan, “Online convex optimization
in the bandit setting: Gradient descent without a gradient,” in Procee dings of
the Annual ACM-SIAM Symposium on Discrete Algorithms , Society for Indus-
trial and Applied Mathematics, pp. 385–394, 2005.
[17] Y. Freund and R. E. Schapire, “Large margin classiﬁcation using the perceptron
algorithm,” Machine Learning , vol. 37, no. 3, pp. 277–296, 1999.
[18] C. Gentile, “The robustness of the p-norm algorithms,” Machine Learning ,
vol. 53, no. 3, pp. 265–299, 2003.
[19] C. Gentile and N. Littlestone, “The robustness of the p-norm algorithms,” in
Procee dings of the Annual Conference on Learning Theory (COLT) , 1999.
[20] G. Gordon, “Regret bounds for prediction problems,” in Procee dings of the
Annual Conference on Learning Theory (COLT) , 1999.
[21] A. J. Grove, N. Littlestone, and D. Schuurmans, “General convergence results
for linear discriminant updates,” Machine Learning , vol. 43, no. 3, pp. 173–210,
2001.
[22] E. Hazan, The Convex Optimization Approach to Regret Minimization . 2009.
[23] D. P. Helmbold and M. Warmuth, “On weak learning,” Journal of Computer
and System Sciences , vol. 50, pp. 551–573, 1995.
[24] S. Kakade, S. Shalev-Shwartz, and A. Tewari, “Regularization techniques for
learning with matrices,” Journal of Machine Learning Research , 2012 (To
appear).
[25] A. Kalai and S. Vempala, “Eﬃcient algorithms for online decision problems,”
Journal of Computer and System Sciences , vol. 71, no. 3, pp. 291–307, 2005.
[26] J. Kivinen and M. Warmuth, “Exponentiated gradient versus gradient descent
for linear predictors,” Information and Computation , vol. 132, no. 1, pp. 1–64,
1997.
[27] J. Kivinen and M. K. Warmuth, “Additive versus exponentiated gradient
updates for linear prediction,” Symposium on Theory of Computing (STOC ),
See also Technical Report UCSC-CRL-94-16, University of California, SantaCruz, Computer Research Laboratory, pp. 209–218, 1995.
[28] J. Kivinen and M. Warmuth, “Relative loss bounds for multidimensional regres-
sion problems,” Machine Learning , vol. 45, no. 3, pp. 301–329, 2001.
[29] N. Littlestone, “Learning quickly when irrelevant attributes abound: A new
linear-threshold algorithm,” Machine Learning , vol. 2, pp. 285–318, 1988.
194 References
[30] N. Littlestone, “From on-line to batch learning,” in Procee dings of the Annual
Conference on Learning Theory (COLT) , pp. 269–284, July 1989.
[31] N. Littlestone, “Mistake bounds and logarithmic linear-threshold learning algo-
rithms,” Ph.D. Thesis, University of California at Santa Cruz, 1990.
[32] N. Littlestone and M. K. Warmuth, “The weighted majority algorithm,” Infor-
mation and Computation , vol. 108, pp. 212–261, 1994.
[33] M. Minsky and S. Papert, Perceptrons: An Introduction to Computational
Geometry . The MIT Press, 1969.
[34] A. Rakhlin, Lecture Notes on Online Learning . draft, 2009.
[35] A. Rakhlin, K. Sridharan, and A. Tewari, “Online learning: Random averages,
combinatorial parameters, and learnability,” in Neural Information P rocessing
Systems (NIPS) , 2010.
[36] H. Robbins, “Some aspects of the sequential design of experiments,” Bulletin
American Mathematical Society , vol. 55, pp. 527–535, 1952.
[37] F. Rosenblatt, “The perceptron: A probabilistic model for information storage
and organization in the brain,” Psychological Review , vol. 65, pp. 386–407, 1958.
(Reprinted in Neurocomputing , (MIT Press, 1988)).
[38] B. Sch¨ olkopf and A. J. Smola, Learning with Kernels: Support Vector Machines,
Regularization, Optimization and Beyond . MIT Press, 2002.
[39] S. Shalev-Shwartz, “Online learning: Theory, algorithms, and applications,”
Ph.D. Thesis, The Hebrew University, 2007.
[40] S. Shalev-Shwartz and Y. Singer, “A primal-dual perspective of online learning
algorithms,” Machine Learning Journal , vol. 69, no. 2, pp. 115–142, 2007.
[41] J. C. Spall, Introduction to Stochastic Search and Optimization: Estimation,
Simulation, and Control , vol. 64. LibreDigital, 2003.
[42] V. N. Vapnik, Statistical Learning Theory . Wiley, 1998.
[43] Volodimir G. Vovk, “Aggregating Strategies,” in Procee dings of the Annual
Conference on Learning Theory (COLT) , pp. 371–383, 1990.
[44] C. Zalinescu, Convex Analysis in General Vector Spaces . River Edge, NJ: World
Scientiﬁc Publishing Co. Inc., 2002.
[45] T. Zhang, “Data dependent concentration bounds for sequential prediction
algorithms,” in Procee dings of the Annual Conference on Learning Theory
(COLT) , pp. 173–187, 2005.
[46] M. Zinkevich, “Online convex programming and generalized inﬁnitesimal
gradient ascent,” in International Conference on Machine Learning , 2003.