Knows What It Knows: A Framework For Self-Aware Learning
Lihong Li lihong@cs.rutgers.edu
Michael L. Littman mlittman@cs.rutgers.edu
Thomas J. Walsh thomaswa@cs.rutgers.edu
Department of Computer Science, Rutgers University, Piscataway, NJ 08854 USA
Abstract
We introduce a learning framework that
combines elements of the well-known PAC
and mistake-bound models. The KWIK
(knows what it knows) framework was de-
signed particularly for its utility in learning
settings where active exploration can impact
the training examples the learner is exposed
to, as is true in reinforcement-learning and
active-learning problems. We catalog several
KWIK-learnable classes and open problems.
1. Motivation
At the core of recent reinforcement-learning algo-
rithms that have polynomial sample complexity guar-
antees (Kearns & Singh, 2002; Kearns & Koller, 1999;
Kakade et al., 2003; Strehl et al., 2007) lies the idea
of distinguishing between instances that have been
learned with suﬃcient accuracy and those whose out-
puts are still unknown.
The Rmax algorithm (Brafman & Tennenholtz, 2002),
for example, estimates transition probabilities for each
state–action–next-state triple of a Markov decision
process (MDP). The estimates are made separately,
as licensed by the Markov property, and the accuracy
of the estimate is bounded using Hoeﬀding bounds.
The algorithm explicitly distinguishes between proba-
bilities that have been estimated accurately (known)
and those for which more experience will be needed
(unknown). By encouraging the agent to gather more
experience in the unknown states, Rmax can guaran-
tee a polynomial bound on the number of timesteps in
which it has a non-near-optimal policy (Kakade, 2003).
In this paper, we make explicit the properties that are
suﬃcient for a learning algorithm to be used in eﬃcient
Appearing in Proceedings of the 25thInternational Confer-
ence on Machine Learning , Helsinki, Finland, 2008. Copy-
right 2008 by the author(s)/owner(s).
[1,1,1] [1,1,1] [1,1,1] [1,1,1][0,0,1] [0,0,1 ]
[1,0,0]
[0,1,1] [0,0,0 ]Figure 1. A cost-vector navigation graph.
exploration algorithms like Rmax. Roughly, the learn-
ing algorithm needs to make only accurate predictions,
although it can opt out of predictions by saying “I
don’t know” ( ⊥). However, there must be a (polyno-
mial) bound on the number of times the algorithm can
respond ⊥. We call such a learning algorithm KWIK
(“know what it knows”).
Section 2 provides a motivating example and sketches
possible uses for KWIK algorithms. Section 3 deﬁnes
the KWIK conditions more precisely and relates them
to established models from learning theory. Sections 4
and 5 survey a set of hypothesis classes for which
KWIK algorithms can be created.
2. A KWIK Example
Consider the simple navigation task in Figure 1. There
is a set of nodes connected by edges, with the node on
the left as the source and the dark one on the right
as the sink. Each edge in the graph is associated with
a binary cost vector of dimension d= 3, indicated in
the ﬁgure. The cost of traversing an edge is the dot
product of its cost vector with a ﬁxed weight vector
w= [1,2,0]. Assume that wis not known to the agent,
but the graph topology and all cost vectors are. In each
episode, the agent starts from the source and moves
along some path to the sink. Each time it crosses an
edge, the agent observes its true cost. Once the sink
is reached, the next episode begins. The learning task
is to take a non-cheapest path in as few episodes as
possible. There are 3 distinct paths in this example.
Given the wabove, the top has a cost of 12, the middle
KWIK Learning Framework
13, and the bottom 15.
A simple approach for this task is for the agent to
assume edge costs are uniform and walk the shortest
(middle) path to collect data. It would gather 4 exam-
ples of [1 ,1,1]→3 and one of [1 ,0,0]→1. Standard
regression algorithms could use this dataset to ﬁnd a
ˆwthat ﬁts this data. Here, ˆ w= [1,1,1] is a natural
choice. The learned weight vector could then be used
to estimate costs for the three paths: 14 for the top,
13 for the middle, 14 for the bottom. Using these es-
timates, an agent would continue to take the middle
path forever, never realizing it is not optimal.
In contrast, consider a learning algorithm that “knows
what it knows”. Instead of creating an approximate
weight vector ˆ w, it reasons about whether the costs
for each edge can be obtained from the available data.
The middle path, since we’ve seen all its edge costs,
is deﬁnitely 13. The last edge of the bottom path has
cost vector [0 ,0,0], so its cost must be zero, but the
penultimate edge of this path has cost vector [0 ,1,1].
This vector is a linear combination of the two observed
cost vectors, so, regardless of w,
w·[0,1,1] =w·([1,1,1]−[1,0,0]) = w·[1,1,1]−w·[1,0,0],
which is just 3 −1 = 2 .Thus, we know the bottom
path’s cost is 14—worse than the middle path.
The vector [0 ,0,1] on the top path is linearly inde-
pendent of the cost vectors we’ve seen, so its cost is
unconstrained. We know we don’t know. A safe thing
to assume provisionally is that it’s zero, encouraging
the agent to try the top path in the second episode.
Now, it observes [0 ,0,1]→0, allowing it to solve for
wand accurately predict the cost for any vector (the
training data spans /Rfracturd). It now knows that it knows all
the costs, and can conﬁdently take the optimal (top)
path.
In general, any algorithm that guesses a weight vec-
tor may never ﬁnd the optimal path. An algorithm
that uses linear algebra to distinguish known from un-
known costs will either take an optimal route ordis-
cover the cost of a linearly independent cost vector on
each episode. Thus, it can never choose suboptimal
paths more than dtimes.
The motivation for studying KWIK learning grew
out of its use in multi-state sequential decision mak-
ing problems like this one. However, other machine-
learning problems could beneﬁt from this perspective
and from the development of eﬃcient algorithms. For
instance, action selection in bandit problems (Fong,
1995) and associative bandit problems (Strehl et al.,
2006) (bandit problems with inputs) can both be ad-
dressed in the KWIK setting by choosing the betterarm when both payoﬀs are known and an unknown
arm otherwise.
KWIK could also be a useful framework for study-
ing active learning (Cohn et al., 1994) and anomaly
detection (Lane & Brodley, 2003), both of which are
machine-learning problems that require some degree of
reasoning about whether a recently presented input is
predictable from previous examples. When mistakes
are costly, as in utility-based data mining (Weiss &
Tian, 2006) or learning robust control (Bagnell et al.,
2001), having explicit predictions of certainty can be
very useful for decision making.
3. Formal Deﬁnition
This section provides a formal deﬁnition of KWIK
learning and its relationship to existing frameworks.
3.1. KWIK Deﬁnition
KWIK is an objective for supervised learning algo-
rithms. In particular, we begin with an input set X
andoutput set Y. The hypothesis class Hconsists of
a set of functions from XtoY:H⊆(X→Y). The
target function h∗∈His the source of training ex-
amples and is unknown to the learner. Note that the
setting is “realizable”, meaning we assume the target
function is in the hypothesis class.
The protocol for a “run” is:
•The hypothesis class Hand accuracy parameters
/epsilon1andδare known to both the learner and envi-
ronment.
•The environment selects a target function h∗∈H
adversarially.
•Repeat:
–The environment selects an input x∈Xad-
versarially and informs the learner.
–The learner predicts an output ˆ y∈Y∪ {⊥} .
–If ˆy/negationslash=⊥, it should be accurate: |ˆy−y| ≤/epsilon1,
where y=h∗(x). Otherwise, the entire run
is considered a failure. The probability of a
failed run must be bounded by δ.
–Over a run, the total number of steps on
which ˆ y=⊥must be bounded by B(/epsilon1, δ),
ideally polynomial in 1 //epsilon1, 1/δ, and parame-
ters deﬁning H. Note that this bound should
hold even if h∗/negationslash∈H, although, obviously, out-
puts need not be accurate in this case.
–If ˆy=⊥, the learner makes an observation
z∈Zof the output, where z=yin the de-
terministic case, z= 1 with probability yand
KWIK Learning Framework
Figure 2. Relationship of KWIK to existing PAC and MB
(mistake bound) frameworks in terms of how labels are
provided for inputs.
0 with probability 1 −yin the Bernoulli case,
orz=y+ηfor zero-mean random variable
ηin the additive noise case.
3.2. Connection to PAC and MB
Figure 2 illustrates the relationship of KWIK
to the similar PAC (probably approximately cor-
rect) (Valiant, 1984) and MB (mistake bound) (Lit-
tlestone, 1987) frameworks. In all three cases, a series
of inputs (instances) is presented to the learner. Each
input is depicted in the ﬁgure by a rectangular box.
In the PAC model, the learner is provided with labels
(correct outputs) for an initial sequence of inputs, de-
picted by cross-hatched boxes. After that point, the
learner is responsible for producing accurate outputs
(empty boxes) for all new inputs. Inputs are drawn
from a ﬁxed distribution.
In the MB model, the learner is expected to produce
an output for every input. Labels are provided to the
learner whenever it makes a mistake (ﬁlled boxes). In-
puts are selected adversarially, so there is no bound
on when the last mistake might be made. However,
MB algorithms guarantee that the total number of
mistakes is small, so the ratio of incorrect to correct
outputs must go to zero asymptotically. Any MB al-
gorithm for a hypothesis class can be used to provide a
PAC algorithm for the same class, but not necessarily
vice versa (Blum, 1994).
The KWIK model has elements of both PAC and MB.
Like PAC, a KWIK algorithm is not allowed to make
mistakes. Like MB, inputs to a KWIK algorithm are
selected adversarially. Instead of bounding mistakes,
a KWIK algorithm must have a bound on the num-
ber of label requests ( ⊥) it can make. By requiring
performance to be independent of the distribution, a
KWIK algorithm can be used in cases in which the in-
put distribution is dependent in complex ways on the
KWIK algorithm’s behavior, as can happen in on-line
or active learning settings. And, like PAC and MB,
the deﬁnition of KWIK algorithms can be naturally
extended to enforce low computational complexity.Note that any KWIK algorithm can be turned into a
MB algorithm with the same bound by simply hav-
ing the algorithm guess an output each time it is not
certain. However, some hypothesis classes are expo-
nentially harder to learn in the KWIK setting than
in the MB setting. An example is conjunctions of n
Boolean variables, in which MB algorithms can guess
“false” when uncertain and learn with n+ 1 mistakes,
but a KWIK algorithm may need Ω(2n/2)⊥s to ac-
quire the negative examples required to capture the
target hypothesis.
3.3. Other Online Learning Models
The notion of allowing the learner to opt out of some
inputs by returning ⊥is not unique to KWIK. Several
other authors have considered related models. For in-
stance, sleeping experts (Freund et al., 1997) can re-
spond ⊥for some inputs, although they need not learn
from these experiences. Learners in the settings of Se-
lective Sampling (SS) (Cesa-Bianchi et al., 2006) and
Label Eﬃcient Prediction (Cesa-Bianchi et al., 2005)
request labels randomly with a changing probability
and achieve bounds on the expected number of mis-
takes and the expected number of label requests for a
ﬁnite number of interactions. These algorithms cannot
be used unmodiﬁed in the KWIK setting because, with
high probability, KWIK algorithms must not make
mistakes at any time. In the MB-like Apple-Tasting
setting (Helmbold et al., 2000), the learner receives
feedback asymmetrically only when it predicts a par-
ticular label (a positive example, say), which conﬂates
the request for a sample with the prediction of a par-
ticular outcome.
Open Problem 1 Is there a way of modifying SS al-
gorithms to satisfy the KWIK criteria?
4. Some KWIK Learnable Classes
This section describes some hypothesis classes for
which KWIK algorithms are available. It is not meant
to be an exhaustive survey, but simply to provide a
ﬂavor for the properties of hypothesis classes KWIK
algorithms can exploit. The complexity of many learn-
ing problems has been characterized by deﬁning the di-
mensionality of hypothesis classes (Angluin, 2004). No
such deﬁnition has been found for the KWIK model, so
we resort to enumerating examples of learnable classes.
Open Problem 2 Is there a way of characterizing
the “dimension” of a hypothesis class in a way that
can be used to derive KWIK bounds?
KWIK Learning Framework
4.1. Memorization and Enumeration
We begin by describing the simplest and most general
KWIK algorithms.
Algorithm 1 The memorization algorithm can learn
any hypothesis class with input space Xwith a KWIK
bound of |X|. This algorithm can be used when the
input space Xis ﬁnite and observations are noise free.
To achieve this bound, the algorithm simply keeps a
mapping ˆhinitialized to ˆh(x) =⊥for all x∈X. When
the environment chooses an input x, the algorithm re-
ports ˆh(x). If ˆh(x) =⊥, the environment will provide
a label yand the algorithm will assign ˆh(x) := y. It
will only report ⊥once for each input, so the KWIK
bound is |X|.
Algorithm 2 The enumeration algorithm can learn
any hypothesis class Hwith a KWIK bound of |H|−1.
This algorithm can be used when the hypothesis class
His ﬁnite and observations are noise free.
The algorithm keeps track of ˆH, the version space, and
initially ˆH=H. Each time the environment provides
input x∈X, the algorithm computes ˆL={h(x)|h∈
ˆH}. That is, it builds the set of all outputs for xfor
all hypotheses that have not yet been ruled out. If
|ˆL|= 0, the version space has been exhausted and
the target hypothesis is not in the hypothesis class
(h∗/negationslash∈H).
If|ˆL|= 1, it means that all hypotheses left in ˆHagree
on the output for this input, and therefore the algo-
rithm knows what the proper output must be. It re-
turns ˆ y∈ˆL. On the other hand, if |ˆL|>1, two
hypotheses in the version space disagree. In this case,
the algorithm returns ⊥and receives the true label y.
It then computes an updated version space
ˆH/prime={h|h∈ˆH∧h(x) =y}.
Because |ˆL|>1, there must be some h∈ˆHsuch
that h(x)/negationslash=y. Therefore, the new version space must
be smaller |ˆH/prime| ≤ | ˆH| −1. Before the next input is
received, the version space is updated ˆH:=ˆH/prime.
If|ˆH|= 1 at any point, |ˆL|= 1, and the algorithm will
no longer return ⊥. Therefore, |H|−1 is the maximum
number of ⊥s the algorithm can return.
Example 1 You own a bar that is frequented by a
group of npatrons P. There is one patron f∈Pwho
is an instigator—whenever a group of patrons is in the
barG⊆P, iff∈G, a ﬁght will break out. However,
there is another patron p∈P, who is a peacemaker.
Figure 3. Schematic of behavior of the planar-distance al-
gorithm after the ﬁrst (a), second (b), and third (c) time
it returns ⊥.
Ifpis in the group, it will prevent a ﬁght, even if fis
present.
You want to predict whether a ﬁght will break out
among a subset of patrons, initially without knowing
the identities of fandp. The input space is X= 2P
and the output space is Y={ﬁght, no ﬁght }.
The memorization algorithm achieves a KWIK bound
of2nfor this problem, since it may have to see each
possible subset of patrons. However, the enumeration
algorithm can KWIK learn this hypothesis class with a
bound of n(n−1)since there is one hypothesis for each
possible assignment of a patron to fandp. Each time
it reports ⊥, it is able to rule out at least one possible
instigator–peacemaker combination.
4.2. Real-valued Functions
The previous two examples exploited the ﬁniteness of
the hypothesis class and input space. KWIK bounds
can also be achieved when these sets are inﬁnite.
Algorithm 3 Deﬁne X=/Rfractur2,Y=/Rfractur, and
H={f|f:X→Y, c∈ /Rfractur2, f(x) =/bardblx−c/bardbl2}.
This is, there is an unknown point and the target func-
tion maps input points to the distance from the un-
known point. The planar-distance algorithm can learn
in this hypothesis class with a KWIK bound of 3.
The algorithm proceeds as follows, illustrated in Fig-
ure 3. First, given initial input x, the algorithm says
⊥and receives output y. Since yis the distance be-
tween xand some unknown point c, we know cmust
lie on the circle illustrated in Figure 3(a). (If y= 0,
then c=x.) Let’s call this input–output pair x1, y1.
The algorithm will return y1for any future input that
matches x1. Otherwise, it will need to return ⊥and
will obtain a new input–output pair x, y, as shown in
Figure 3(b). They become x2andy2.
Now, the algorithm can narrow down the location of c
to the two hatch-marked points. In spite of this ambi-
guity, for any input on the dark diagonal line the algo-
rithm will be able to return the correct distance—all
KWIK Learning Framework
these points are equidistant from the two possibilities.
The two circles must intersect, assuming the target
hypothesis is in H1.
Once an input xis received that is not co-linear with
x1andx2, the algorithm returns ⊥and obtains an-
other x, ypair, as illustrated in Figure 3(c). Finally,
since three circles will intersect at at most one point,
the algorithm can identify the location of cand use
it to correctly answer any future query. Thus, three
⊥s suﬃce for KWIK learning in this setting. The al-
gorithm generalizes to d-dimensional versions of the
setting with a KWIK bound of d+ 1.
Algorithm 3 illustrates a number of important points.
First, since learners have no control over their inputs
in the KWIK setting, they must be robust to degen-
erate inputs such as inputs that lie precisely on a line.
Second, they can often return valid answers for some
inputs even before they have learned the target func-
tion over the entire input space.
4.3. Noisy Observations
Up to this point, observations have been noise free.
Next, we consider the simplest noisy KWIK learning
problem in the Bernoulli case.
Algorithm 4 The coin-learning algorithm can accu-
rately predict the probability that a biased coin will
come up heads given Bernoulli observations with a
KWIK bound of B(/epsilon1, δ) =1
2/epsilon12ln2
δ=O/parenleftbig1
/epsilon12ln1
δ/parenrightbig
.
We have a biased coin whose unknown probability of
heads is p. In the notation of this paper, |X|= 1,
Y= [0,1], and Z={0,1}. We want to learn an
estimate ˆ pthat is accurate ( |ˆp−p| ≤/epsilon1) with high
probability (1 −δ).
If we could observe p, then this problem would be triv-
ial: Say ⊥once, observe p, and let ˆ p=p. The KWIK
bound is thus 1. Now, however, observations are noisy.
Instead of observing p, we see either 1 (with probabil-
ityp) or 0 (with probability 1 −p).
Each time the algorithm says ⊥, it gets an independent
trial that it can use to compute ˆ p=1
T/summationtextT
t=1zt, where
zt∈Zis the tth observation in Ttrials. The number
of trials needed before we are 1 −δcertain our estimate
is within /epsilon1can be computed using a Hoeﬀding bound:
T≥1
2/epsilon12ln2
δ=O/parenleftbigg1
/epsilon12ln1
δ/parenrightbigg
.
1They can also intersect at one point, if the circles are
tangent, in which case the algorithm can identify cunam-
biguously.Algorithm 5 Deﬁne X=/Rfracturd,Y=/Rfractur, and
H={f|f:X→Y, w∈ /Rfracturd, f(x) =w·x}.
That is, His the linear functions on dvariables. Given
additive noise, the noisy linear-regression algorithm
can learn in Hwith a KWIK bound of B(/epsilon1, δ) =
˜O(d3//epsilon14), where ˜O(·)suppresses log factors.
The deterministic case was described in Section 2 with
a bound of d. Here, the algorithm must be cautious
to average over the noisy samples to make predictions
accurately. This problem was solved by Strehl and
Littman (2008). The algorithm uses the least squares
estimate of the weight vector for inputs with high cer-
tainty. Certainty is measured by two terms represent-
ing (1) the number and proximity of previous samples
to the current point and (2) the appropriateness of the
previous samples for making a least squares estimate.
When certainty is low for either measure, the algo-
rithm reports ⊥and observes a noisy sample of the
linear function.
Here, solving a noisy version of a problem resulted
in an increased KWIK bound (from dto essentially
d3). Note that the deterministic Algorithm 3 also has
a bound of d, but no bound has been found for the
stochastic case.
Open Problem 3 Is there a general scheme for tak-
ing a KWIK algorithm for a deterministic class and
updating it to work in the presence of noise?
5. Combining KWIK Learners
This section provides examples of how KWIK learners
can be combined to provide learning guarantees for
more complex hypothesis classes.
Algorithm 6 LetF:X→Ybe the set of functions
mapping input set Xto output set Y. Let H1, . . . , H k
be a set of KWIK learnable hypothesis classes with
bounds of B1(/epsilon1, δ), . . . , B k(/epsilon1, δ)where Hi⊆Ffor all
1≤i≤k. That is, all the hypothesis classes share
the same input/output sets. The union algorithm can
learn the joint hypothesis class H=/uniontext
iHiwith a
KWIK bound of B(/epsilon1, δ) = (1 −k) +/summationtext
iBi(/epsilon1, δ).
The union algorithm is like a higher-level version of
the enumeration algorithm (Algorithm 2) and applies
in the deterministic setting. It maintains a set of
active algorithms ˆA, one for each hypothesis class:
ˆA={1, . . . , k }. Given an input x, the union algorithm
queries each algorithm i∈ˆAto obtain a prediction ˆ yi
from each active algorithm. Let ˆL={ˆyi|i∈ˆA}.
KWIK Learning Framework
If⊥ ∈ ˆL, the union algorithm reports ⊥and obtains
the correct output y. Any algorithm ifor which ˆ y=⊥
is then sent the correct output yto allow it to learn. If
|ˆL|>1, then there is disagreement among the subal-
gorithms. The union algorithm reports ⊥in this case
because at least one of the algorithms has learned the
wrong hypothesis and it needs to know which.
Any algorithm that made a prediction other than yor
⊥is “killed”—removed from consideration. That is,
ˆA/prime={i|i∈ˆA∧(ˆyi=⊥ ∨ˆyi=y)}.
On each input for which the union algorithm reports
⊥, either one of the subalgorithms reported ⊥(at most/summationtext
iBi(/epsilon1, δ) times) or two algorithms disagreed and at
least one was removed from ˆA(at most k−1 times).
The KWIK bound follows from these facts.
Example 2 LetX=Y=/Rfractur. Now, deﬁne H1=
{f|f(x) =|x−c|, c∈ /Rfractur} . That is, each function in
H1maps xto its distance from some unknown point
c. We can learn H1with a KWIK bound of 2 using
a 1-dimensional version of Algorithm 3. Next, deﬁne
H2={f|f(x) =yx+b, m∈ /Rfractur, b∈ /Rfractur} . That is, H2
is the set of lines. We can learn H2with a KWIK
bound of 2 using the regression algorithm in Section 2.
Finally, deﬁne H=H1∪H2, the union of these two
classes. We can use Algorithm 6 to KWIK learn H.
Assume the ﬁrst input is x1= 2. The union algorithm
asks the learners for H1andH2the output for x1and
neither has any idea, so it returns ⊥and receives the
feedback y1= 2, which it passes to the subalgorithms.
The next input is x2= 8. The learners for H1andH2
still don’t have enough information, so it returns ⊥
and sees y2= 4, which it passes to the subalgorithms.
Next, x3= 1. Now, the learner for H1unambiguously
computes c= 4, because that’s the only interpretation
consistent with the ﬁrst two examples ( |2−4|= 2,
|8−4|= 4), so it returns |1−4|= 3. On the other
hand, the learner for H2unambiguously computes m=
1/3andb= 4/3, because that’s the only interpretation
consistent with the ﬁrst two examples ( 2×1/3+4/3 =
2,8×1/3 + 4 /3 = 4 ), so it returns 1×1/3 + 4 /3 =
5/3. Since the two subalgorithms disagree, the union
algorithm returns ⊥one last time and ﬁnds out that
y3= 3. It makes all future predictions (accurately)
using the algorithm for H1.
Next, we consider a variant of Algorithm 1 that com-
bines learners across disjoint input spaces.
Algorithm 7 LetX1, . . . , X kbe a set of disjoint in-
put spaces ( Xi∩Xj=∅ifi/negationslash=j) and Ybe an out-
put space. Let H1, . . . , H kbe a set of KWIK learnablehypothesis classes with bounds of B1(/epsilon1, δ), . . . , B k(/epsilon1, δ)
where Hi∈(Xi→Y). The input-partition algorithm
can learn the hypothesis class H∈(X1∪· · ·∪ Xk→Y)
with a KWIK bound of B(/epsilon1, δ) =/summationtext
iBi(/epsilon1, δ/k ).
The input-partition algorithm runs the learning algo-
rithm for each subclass Hi. When it receives an input
x∈Xi, it queries the learning algorithm for class Hi
and returns its response, which is /epsilon1accurate, by re-
quest. To achieve 1 −δcertainty, it insists on 1 −δ/k
certainty from each of the subalgorithms. By the union
bound, the overall failure probability must be less than
the sum of the failure probabilities for the subalgo-
rithms.
Example 3 An MDP consists of nstates and mac-
tions. For each combination of state and action and
next state, the transition function returns a probability.
As the reinforcement-learning agent moves around in
the state space, it observes state–action–state transi-
tions and must predict the probabilities for transitions
it has not yet observed. In the model-based setting,
an algorithm learns a mapping from the size n2min-
put space of state–action–state combinations to prob-
abilities via Bernoulli observations. Thus, the prob-
lem can be solved via the input-partition algorithm
(Algorithm 7) over a set of individual probabilities
learned via Algorithm 4. The resulting KWIK bound
isB(/epsilon1, δ) =O/parenleftBig
n2m
/epsilon12lnnm
δ/parenrightBig
.
Note that this approach is precisely what is found in
most eﬃcient RL algorithms in the literature (Kearns
& Singh, 2002; Brafman & Tennenholtz, 2002).
Algorithm 7 combines hypotheses by partitioning the
input space. In contrast, the next example concerns
combinations in input and output space.
Algorithm 8 LetX1, . . . , X kandY1, . . . , Y kbe a set
of input and output spaces and H1, . . . , H kbe a set
of KWIK learnable hypothesis classes with bounds of
B1(/epsilon1, δ), . . . , B k(/epsilon1, δ)on these spaces. That is, Hi∈
(Xi→Yi). The cross-product algorithm can learn the
hypothesis class H∈((X1×· · ·× Xk)→(Y1×· · ·× Yk))
with a KWIK bound of B(/epsilon1, δ) =/summationtext
iBi(/epsilon1, δ/k ).
Here, each input consists of a vector of inputs from
each of the spaces X1, . . . , X kand outputs are vectors
of outputs from Y1, . . . , Y k. Like Algorithm 7, each
component of this vector can be learned independently
via the corresponding algorithm. Each is learned to
within an accuracy of /epsilon1and conﬁdence 1 −δ/k. Any
time any component returns ⊥, the cross-product algo-
rithm returns ⊥. Since each ⊥returned can be traced
to one of the subalgorithms, the total is bounded as
KWIK Learning Framework
described above. By the union bound, total failure
probability is no more than k×δ/k=δ.
Example 4 Transitions in factored-state MDP can be
thought of as mappings from vectors to vectors. Given
known dependencies, the cross-product algorithm can
be used to learn each component of the transition func-
tion. Each component is, itself, an instance of Algo-
rithm 7 applied to the coin-learning algorithm. This
three-level KWIK algorithm provides an approach to
learn the transition function of a factored-state MDP
with a polynomial KWIK bound. This insight can be
used to derive the factored-state-MDP learning algo-
rithm used by Kearns and Koller (1999).
The previous two algorithms apply to both determin-
istic and noisy observations. We next provide a pow-
erful algorithm that generalizes the union algorithm
(Algorithm 6) to work with noisy observations as well.
Algorithm 9 LetF:X→Ybe the set of functions
mapping input set Xto output set Y= [0,1]. Let Z=
{0,1}be a binary observation set. Let H1, . . . , H kbe a
set of KWIK learnable hypothesis classes with bounds
ofB1(/epsilon1, δ), . . . , B k(/epsilon1, δ)where Hi⊆Ffor all 1≤i≤
k. That is, all the hypothesis classes share the same in-
put/output sets. The noisy union algorithm can learn
the joint hypothesis class H=/uniontext
iHiwith a KWIK
bound of B(/epsilon1, δ) =O/parenleftbigk
/epsilon12lnk
δ/parenrightbig
+/summationtextk
i=1Bi(/epsilon1
4,δ
k+1).
For simplicity, we sketch the special case of k= 2.
The general case will be brieﬂy discussed at the end.
The noisy union algorithm is similar to the union al-
gorithm (Algorithm 6), except that it has to deal with
noisy observations. The algorithm proceeds by run-
ning the KWIK algorithms, using parameters ( /epsilon10, δ0),
as subalgorithms for each of the Hihypothesis classes,
where /epsilon10=/epsilon1
4andδ0=δ
3. Given an input xtin trial t,
it queries each algorithm ito obtain a prediction ˆ yti.
LetˆLtbe the set of responses.
If⊥ ∈ ˆLt, the noisy union algorithm reports ⊥, ob-
tains an observation zt∈Z, and sends it to all subal-
gorithms iwith ˆ yti=⊥to allow them to learn. In the
following, we focus on the other case where ⊥/∈ˆLt.
If|ˆyt1−ˆyt2| ≤4/epsilon10, then these two predictions are suf-
ﬁciently consistent, and we claim that, with high prob-
ability, the prediction ˆ pt= (ˆyt1+ ˆyt2)/2 is/epsilon1-close to
yt= Pr( zt= 1). This claim follows because, by as-
sumption, one of the predictions, say ˆ yt1, deviates from
ytby at most /epsilon10with probability at least 1 −δ/3, and
hence |ˆpt−yt|=|ˆpt−ˆyt1+ ˆyt1−yt| ≤ | ˆpt−ˆyt1|+
|ˆyt1−ˆyt|=|ˆyt1−ˆyt2|/2 +|ˆyt1−ˆyt| ≤2/epsilon10+/epsilon10< /epsilon1.If|ˆyt1−ˆyt2|>4/epsilon10, then the individual predictions are
not consistent enough for the noisy union algorithm to
make an /epsilon1-accurate prediction. Thus, the noisy union
algorithm reports ⊥and needs to know which subal-
gorithm provided an inaccurate response. But, since
the observations are noisy in this problem, it cannot
eliminate hion the basis of a single observation. In-
stead, it maintains the total squared prediction error
for every subalgorithm i:/lscripti=/summationtext
t∈I(ˆyti−zt)2, where
I={t| |ˆyt1−ˆyt2|>4/epsilon10}is the set of trials in which
the subalgorithms gave inconsistent predictions. We
observe that |I|is the number of ⊥s returned by the
noisy union algorithm alone (not counting those re-
turned by the subalgorithms). Our last step is to show
/lscriptiprovides a robust measure for eliminating invalid
predictors when |I|is suﬃciently large.
Applying the Hoeﬀding bound and some algebra, we
ﬁnd Pr ( /lscript1> /lscript2)≤
exp/parenleftBigg
−/summationtext
t∈I|ˆyt1−ˆyt2|2
8/parenrightBigg
≤exp/parenleftbig
−2/epsilon12
0|I|/parenrightbig
.
Setting the righthand side to be δ/3 and solving for
|I|, we have |I|=1
2/epsilon12
0ln3
δ=O/parenleftbig1
/epsilon12ln1
δ/parenrightbig
.
Since each hisucceeds with probability 1 −δ
3, and the
comparison of /lscript1and/lscript2also succeeds with probability
1−δ
3, a union bound implies that the noisy union algo-
rithm succeeds with probability at least 1 −δ. All⊥s
are either from a subalgorithm (at most/summationtext
iBi(/epsilon10, δ0))
or from the noisy union algorithm ( O/parenleftbig1
/epsilon12ln1
δ/parenrightbig
).
The general case where k >2 can be reduced to the
k= 2 case by pairing the klearners and running the
noisy union algorithm described above on each pair.
Here, each subalgorithm is run with parameter/epsilon1
4and
δ
k+1. Although there are/parenleftbigk
2/parenrightbig
=O(k2) pairs, a slightly
improved reduction and analysis can reduce the de-
pendence of |I|onkfrom quadratic to linearithmic,
leading to the bound given in the statement.
Example 5 Without known dependencies, learning
a factored-state MDP is more challenging. Strehl
et al. (2007) showed that each possible dependence
structure can be viewed as a separate hypothesis and
provided an algorithm for learning the dependencies
in a factored-state MDP while learning the transition
probabilities. The algorithm can be viewed as a four-
level KWIK algorithm with a noisy union algorithm
at the top (to discover the dependence structure), a
cross-product algorithm beneath it (to decompose the
transitions for the separate components of the factored-
state representation), an input-partition algorithm be-
low that (to handle the diﬀerent combinations of state
component and action), and a coin-learning algorithm
KWIK Learning Framework
at the very bottom (to learn the transition probabili-
ties themselves). Note that Algorithm 9 is conceptu-
ally simpler, signiﬁcantly more eﬃcient ( klogkvs.k2
dependence on k), and more generally applicable than
the one due to Strehl et al. (2007).
6. Conclusion and Future Work
We described the KWIK (“knows what it knows”)
model of supervised learning, which identiﬁes and gen-
eralizes a key step common to a class of algorithms
for eﬃcient exploration. We provided algorithms for a
set of basic hypothesis classes given deterministic and
noisy observations as well as methods for composing
hypothesis classes to create more complex algorithms.
One example algorithm consisted of a four-level de-
composition of an existing learning algorithm from the
reinforcement-learning literature.
By providing a set of example algorithms and compo-
sition rules, we hope to encourage the use of KWIK
algorithms as a component in machine-learning appli-
cations as well as spur the development of novel algo-
rithms. One concern of particular interest in applying
the KWIK framework to real-life data we leave as an
open problem.
Open Problem 4 How can KWIK be adapted to ap-
ply in the unrealizable setting in which the target hy-
pothesis can be chosen from outside the hypothesis
class H?
References
Angluin, D. (2004). Queries revisited. Theoretical Com-
puter Science ,313, :175–194.
Bagnell, J., Ng, A. Y., & Schneider, J. (2001). Solving
uncertain Markov decision problems (Technical Report
CMU-RI-TR-01-25). Robotics Institute, Carnegie Mel-
lon University, Pittsburgh, PA.
Blum, A. (1994). Separating distribution-free and mistake-
bound learning models over the Boolean domain. SIAM
Journal on Computing ,23, 990–1000.
Brafman, R. I., & Tennenholtz, M. (2002). R-MAX—a
general polynomial time algorithm for near-optimal re-
inforcement learning. Journal of Machine Learning Re-
search ,3, 213–231.
Cesa-Bianchi, N., Gentile, C., & Zaniboni, L. (2006).
Worst-case analysis of selective sampling for linear clas-
siﬁcation. Journal of Machine Learning Research ,7,
1205–1230.
Cesa-Bianchi, N., Lugosi, G., & Stoltz, G. (2005). Minimiz-
ing regret with label eﬃcient prediction. IEEE Transac-
tions on Information Theory ,51, 2152–2162.Cohn, D. A., Atlas, L., & Ladner, R. E. (1994). Improving
generalization with active learning. Machine Learning ,
15, 201–221.
Fong, P. W. L. (1995). A quantitative study of hypothesis
selection. Proceedings of the Twelfth International Con-
ference on Machine Learning (ICML-95) (pp. 226–234).
Freund, Y., Schapire, R. E., Singer, Y., & Warmuth, M. K.
(1997). Using and combining predictors that specialize.
STOC ’97: Proceedings of the twenty-ninth annual ACM
symposium on Theory of computing (pp. 334–343).
Helmbold, D. P., Littlestone, N., & Long, P. M. (2000).
Apple tasting. Information and Computation ,161, 85–
139.
Kakade, S., Kearns, M., & Langford, J. (2003). Explo-
ration in metric state spaces. Proceedings of the 20th
International Conference on Machine Learning .
Kakade, S. M. (2003). On the sample complexity of rein-
forcement learning . Doctoral dissertation, Gatsby Com-
putational Neuroscience Unit, University College Lon-
don.
Kearns, M. J., & Koller, D. (1999). Eﬃcient reinforce-
ment learning in factored MDPs. Proceedings of the 16th
International Joint Conference on Artiﬁcial Intelligence
(IJCAI) (pp. 740–747).
Kearns, M. J., & Singh, S. P. (2002). Near-optimal rein-
forcement learning in polynomial time. Machine Learn-
ing,49, 209–232.
Lane, T., & Brodley, C. E. (2003). An empirical study of
two approaches to sequence learning for anomaly detec-
tion. Machine Learning ,51, 73–107.
Littlestone, N. (1987). Learning quickly when irrelevant
attributes abound: A new linear-threshold algorithm.
Machine Learning ,2, 285–318.
Strehl, A. L., Diuk, C., & Littman, M. L. (2007). Eﬃcient
structure learning in factored-state MDPs. Proceedings
of the Twenty-Second National Conference on Artiﬁcial
Intelligence (AAAI-07) .
Strehl, A. L., & Littman, M. L. (2008). Online linear re-
gression and its application to model-based reinforce-
ment learning. Advances in Neural Information Process-
ing Systems 20 .
Strehl, A. L., Mesterharm, C., Littman, M. L., & Hirsh,
H. (2006). Experience-eﬃcient learning in associative
bandit problems. Proceedings of the Twenty-third Inter-
national Conference on Machine Learning (ICML-06) .
Valiant, L. G. (1984). A theory of the learnable. Commu-
nications of the ACM ,27, 1134–1142.
Weiss, G. M., & Tian, Y. (2006). Maximizing classiﬁer util-
ity when training data is costly. SIGKDD Explorations ,
8, 31–38.
Acknowledgments
Support provided by NSF IIS-0325281 and DARPA TL.

Lihong Li
Michael L. Littman
Thomas J. Walsh