ImageNet Classification with Deep Convolutional Neural Networks

ImageNet Classiﬁcation with Deep Convolutional
Neural Networks
Alex Krizhevsky
University of Toronto
kriz@cs.utoronto.caIlya Sutskever
University of Toronto
ilya@cs.utoronto.caGeoffrey E. Hinton
University of Toronto
hinton@cs.utoronto.ca
Abstract
We trained a large, deep convolutional neural network to classify the 1.2 million
high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 dif-
ferent classes. On the test data, we achieved top-1 and top-5 error rates of 37.5%
and 17.0% which is considerably better than the previous state-of-the-art. The
neural network, which has 60 million parameters and 650,000 neurons, consists
of ﬁve convolutional layers, some of which are followed by max-pooling layers,
and three fully-connected layers with a ﬁnal 1000-way softmax. To make train-
ing faster, we used non-saturating neurons and a very efﬁcient GPU implemen-
tation of the convolution operation. To reduce overﬁtting in the fully-connected
layers we employed a recently-developed regularization method called “dropout”
that proved to be very effective. We also entered a variant of this model in the
ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%,
compared to 26.2% achieved by the second-best entry.
1 Introduction
Current approaches to object recognition make essential use of machine learning methods. To im-
prove their performance, we can collect larger datasets, learn more powerful models, and use bet-
ter techniques for preventing overﬁtting. Until recently, datasets of labeled images were relatively
small — on the order of tens of thousands of images (e.g., NORB [16], Caltech-101/256 [8, 9], and
CIFAR-10/100 [12]). Simple recognition tasks can be solved quite well with datasets of this size,
especially if they are augmented with label-preserving transformations. For example, the current-
best error rate on the MNIST digit-recognition task (<0.3%) approaches human performance [4].
But objects in realistic settings exhibit considerable variability, so to learn to recognize them it is
necessary to use much larger training sets. And indeed, the shortcomings of small image datasets
have been widely recognized (e.g., Pinto et al. [21]), but it has only recently become possible to col-
lect labeled datasets with millions of images. The new larger datasets include LabelMe [23], which
consists of hundreds of thousands of fully-segmented images, and ImageNet [6], which consists of
over 15 million labeled high-resolution images in over 22,000 categories.
To learn about thousands of objects from millions of images, we need a model with a large learning
capacity. However, the immense complexity of the object recognition task means that this prob-
lem cannot be speciﬁed even by a dataset as large as ImageNet, so our model should also have lots
of prior knowledge to compensate for all the data we don’t have. Convolutional neural networks
(CNNs) constitute one such class of models [16, 11, 13, 18, 15, 22, 26]. Their capacity can be con-
trolled by varying their depth and breadth, and they also make strong and mostly correct assumptions
about the nature of images (namely, stationarity of statistics and locality of pixel dependencies).
Thus, compared to standard feedforward neural networks with similarly-sized layers, CNNs have
much fewer connections and parameters and so they are easier to train, while their theoretically-best
performance is likely to be only slightly worse.
1
Despite the attractive qualities of CNNs, and despite the relative efﬁciency of their local architecture,
they have still been prohibitively expensive to apply in large scale to high-resolution images. Luck-
ily, current GPUs, paired with a highly-optimized implementation of 2D convolution, are powerful
enough to facilitate the training of interestingly-large CNNs, and recent datasets such as ImageNet
contain enough labeled examples to train such models without severe overﬁtting.
The speciﬁc contributions of this paper are as follows: we trained one of the largest convolutional
neural networks to date on the subsets of ImageNet used in the ILSVRC-2010 and ILSVRC-2012
competitions [2] and achieved by far the best results ever reported on these datasets. We wrote a
highly-optimized GPU implementation of 2D convolution and all the other operations inherent in
training convolutional neural networks, which we make available publicly1. Our network contains
a number of new and unusual features which improve its performance and reduce its training time,
which are detailed in Section 3. The size of our network made overﬁtting a signiﬁcant problem, even
with 1.2 million labeled training examples, so we used several effective techniques for preventing
overﬁtting, which are described in Section 4. Our ﬁnal network contains ﬁve convolutional and
three fully-connected layers, and this depth seems to be important: we found that removing any
convolutional layer (each of which contains no more than 1% of the model’s parameters) resulted in
inferior performance.
In the end, the network’s size is limited mainly by the amount of memory available on current GPUs
and by the amount of training time that we are willing to tolerate. Our network takes between ﬁve
and six days to train on two GTX 580 3GB GPUs. All of our experiments suggest that our results
can be improved simply by waiting for faster GPUs and bigger datasets to become available.
2 The Dataset
ImageNet is a dataset of over 15 million labeled high-resolution images belonging to roughly 22,000
categories. The images were collected from the web and labeled by human labelers using Ama-
zon’s Mechanical Turk crowd-sourcing tool. Starting in 2010, as part of the Pascal Visual Object
Challenge, an annual competition called the ImageNet Large-Scale Visual Recognition Challenge
(ILSVRC) has been held. ILSVRC uses a subset of ImageNet with roughly 1000 images in each of
1000 categories. In all, there are roughly 1.2 million training images, 50,000 validation images, and
150,000 testing images.
ILSVRC-2010 is the only version of ILSVRC for which the test set labels are available, so this is
the version on which we performed most of our experiments. Since we also entered our model in
the ILSVRC-2012 competition, in Section 6 we report our results on this version of the dataset as
well, for which test set labels are unavailable. On ImageNet, it is customary to report two error rates:
top-1 and top-5, where the top-5 error rate is the fraction of test images for which the correct label
is not among the ﬁve labels considered most probable by the model.
ImageNet consists of variable-resolution images, while our system requires a constant input dimen-
sionality. Therefore, we down-sampled the images to a ﬁxed resolution of 256256. Given a
rectangular image, we ﬁrst rescaled the image such that the shorter side was of length 256, and then
cropped out the central 256256patch from the resulting image. We did not pre-process the images
in any other way, except for subtracting the mean activity over the training set from each pixel. So
we trained our network on the (centered) raw RGB values of the pixels.
3 The Architecture
The architecture of our network is summarized in Figure 2. It contains eight learned layers —
ﬁve convolutional and three fully-connected. Below, we describe some of the novel or unusual
features of our network’s architecture. Sections 3.1-3.4 are sorted according to our estimation of
their importance, with the most important ﬁrst.
1http://code.google.com/p/cuda-convnet/
2
3.1 ReLU Nonlinearity
Figure 1: A four-layer convolutional neural
network with ReLUs (solid line) reaches a 25%
training error rate on CIFAR-10 six times faster
than an equivalent network with tanh neurons
(dashed line) . The learning rates for each net-
work were chosen independently to make train-
ing as fast as possible. No regularization of
any kind was employed. The magnitude of the
effect demonstrated here varies with network
architecture, but networks with ReLUs consis-
tently learn several times faster than equivalents
with saturating neurons.The standard way to model a neuron’s output fas
a function of its input xis withf(x) = tanh(x)
orf(x) = (1 +ex)1. In terms of training time
with gradient descent, these saturating nonlinearities
are much slower than the non-saturating nonlinearity
f(x) = max(0;x). Following Nair and Hinton [20],
we refer to neurons with this nonlinearity as Rectiﬁed
Linear Units (ReLUs). Deep convolutional neural net-
works with ReLUs train several times faster than their
equivalents with tanh units. This is demonstrated in
Figure 1, which shows the number of iterations re-
quired to reach 25% training error on the CIFAR-10
dataset for a particular four-layer convolutional net-
work. This plot shows that we would not have been
able to experiment with such large neural networks for
this work if we had used traditional saturating neuron
models.
We are not the ﬁrst to consider alternatives to tradi-
tional neuron models in CNNs. For example, Jarrett
et al. [11] claim that the nonlinearity f(x) =jtanh(x)j
works particularly well with their type of contrast nor-
malization followed by local average pooling on the
Caltech-101 dataset. However, on this dataset the pri-
mary concern is preventing overﬁtting, so the effect
they are observing is different from the accelerated
ability to ﬁt the training set which we report when us-
ing ReLUs. Faster learning has a great inﬂuence on the
performance of large models trained on large datasets.
3.2 Training on Multiple GPUs
A single GTX 580 GPU has only 3GB of memory, which limits the maximum size of the networks
that can be trained on it. It turns out that 1.2 million training examples are enough to train networks
which are too big to ﬁt on one GPU. Therefore we spread the net across two GPUs. Current GPUs
are particularly well-suited to cross-GPU parallelization, as they are able to read from and write to
one another’s memory directly, without going through host machine memory. The parallelization
scheme that we employ essentially puts half of the kernels (or neurons) on each GPU, with one
additional trick: the GPUs communicate only in certain layers. This means that, for example, the
kernels of layer 3 take input from all kernel maps in layer 2. However, kernels in layer 4 take input
only from those kernel maps in layer 3 which reside on the same GPU. Choosing the pattern of
connectivity is a problem for cross-validation, but this allows us to precisely tune the amount of
communication until it is an acceptable fraction of the amount of computation.
The resultant architecture is somewhat similar to that of the “columnar” CNN employed by Cire¸ san
et al. [5], except that our columns are not independent (see Figure 2). This scheme reduces our top-1
and top-5 error rates by 1.7% and 1.2%, respectively, as compared with a net with half as many
kernels in each convolutional layer trained on one GPU. The two-GPU net takes slightly less time
to train than the one-GPU net2.
2The one-GPU net actually has the same number of kernels as the two-GPU net in the ﬁnal convolutional
layer. This is because most of the net’s parameters are in the ﬁrst fully-connected layer, which takes the last
convolutional layer as input. So to make the two nets have approximately the same number of parameters, we
did not halve the size of the ﬁnal convolutional layer (nor the fully-conneced layers which follow). Therefore
this comparison is biased in favor of the one-GPU net, since it is bigger than “half the size” of the two-GPU
net.
3
3.3 Local Response Normalization
ReLUs have the desirable property that they do not require input normalization to prevent them
from saturating. If at least some training examples produce a positive input to a ReLU, learning will
happen in that neuron. However, we still ﬁnd that the following local normalization scheme aids
generalization. Denoting by ai
x;ythe activity of a neuron computed by applying kernel iat position
(x;y)and then applying the ReLU nonlinearity, the response-normalized activity bi
x;yis given by
the expression
bi
x;y=ai
x;y=0
@k+min(N1;i+n=2)X
j=max(0;in=2)(aj
x;y)21
A
where the sum runs over n“adjacent” kernel maps at the same spatial position, and Nis the total
number of kernels in the layer. The ordering of the kernel maps is of course arbitrary and determined
before training begins. This sort of response normalization implements a form of lateral inhibition
inspired by the type found in real neurons, creating competition for big activities amongst neuron
outputs computed using different kernels. The constants k;n; , andare hyper-parameters whose
values are determined using a validation set; we used k= 2,n= 5,= 104, and= 0:75. We
applied this normalization after applying the ReLU nonlinearity in certain layers (see Section 3.5).
This scheme bears some resemblance to the local contrast normalization scheme of Jarrett et al. [11],
but ours would be more correctly termed “brightness normalization”, since we do not subtract the
mean activity. Response normalization reduces our top-1 and top-5 error rates by 1.4% and 1.2%,
respectively. We also veriﬁed the effectiveness of this scheme on the CIFAR-10 dataset: a four-layer
CNN achieved a 13% test error rate without normalization and 11% with normalization3.
3.4 Overlapping Pooling
Pooling layers in CNNs summarize the outputs of neighboring groups of neurons in the same kernel
map. Traditionally, the neighborhoods summarized by adjacent pooling units do not overlap (e.g.,
[17, 11, 4]). To be more precise, a pooling layer can be thought of as consisting of a grid of pooling
units spaced spixels apart, each summarizing a neighborhood of size zzcentered at the location
of the pooling unit. If we set s=z, we obtain traditional local pooling as commonly employed
in CNNs. If we set s < z , we obtain overlapping pooling. This is what we use throughout our
network, with s= 2 andz= 3. This scheme reduces the top-1 and top-5 error rates by 0.4% and
0.3%, respectively, as compared with the non-overlapping scheme s= 2;z= 2, which produces
output of equivalent dimensions. We generally observe during training that models with overlapping
pooling ﬁnd it slightly more difﬁcult to overﬁt.
3.5 Overall Architecture
Now we are ready to describe the overall architecture of our CNN. As depicted in Figure 2, the net
contains eight layers with weights; the ﬁrst ﬁve are convolutional and the remaining three are fully-
connected. The output of the last fully-connected layer is fed to a 1000-way softmax which produces
a distribution over the 1000 class labels. Our network maximizes the multinomial logistic regression
objective, which is equivalent to maximizing the average across training cases of the log-probability
of the correct label under the prediction distribution.
The kernels of the second, fourth, and ﬁfth convolutional layers are connected only to those kernel
maps in the previous layer which reside on the same GPU (see Figure 2). The kernels of the third
convolutional layer are connected to all kernel maps in the second layer. The neurons in the fully-
connected layers are connected to all neurons in the previous layer. Response-normalization layers
follow the ﬁrst and second convolutional layers. Max-pooling layers, of the kind described in Section
3.4, follow both response-normalization layers as well as the ﬁfth convolutional layer. The ReLU
non-linearity is applied to the output of every convolutional and fully-connected layer.
The ﬁrst convolutional layer ﬁlters the 2242243input image with 96 kernels of size 11113
with a stride of 4 pixels (this is the distance between the receptive ﬁeld centers of neighboring
3We cannot describe this network in detail due to space constraints, but it is speciﬁed precisely by the code
and parameter ﬁles provided here: http://code.google.com/p/cuda-convnet/.
4
Figure 2: An illustration of the architecture of our CNN, explicitly showing the delineation of responsibilities
between the two GPUs. One GPU runs the layer-parts at the top of the ﬁgure while the other runs the layer-parts
at the bottom. The GPUs communicate only at certain layers. The network’s input is 150,528-dimensional, and
the number of neurons in the network’s remaining layers is given by 253,440–186,624–64,896–64,896–43,264–
4096–4096–1000.
neurons in a kernel map). The second convolutional layer takes as input the (response-normalized
and pooled) output of the ﬁrst convolutional layer and ﬁlters it with 256 kernels of size 5548.
The third, fourth, and ﬁfth convolutional layers are connected to one another without any intervening
pooling or normalization layers. The third convolutional layer has 384 kernels of size 33
256connected to the (normalized, pooled) outputs of the second convolutional layer. The fourth
convolutional layer has 384 kernels of size 33192, and the ﬁfth convolutional layer has 256
kernels of size 33192. The fully-connected layers have 4096 neurons each.
4 Reducing Overﬁtting
Our neural network architecture has 60 million parameters. Although the 1000 classes of ILSVRC
make each training example impose 10 bits of constraint on the mapping from image to label, this
turns out to be insufﬁcient to learn so many parameters without considerable overﬁtting. Below, we
describe the two primary ways in which we combat overﬁtting.
4.1 Data Augmentation
The easiest and most common method to reduce overﬁtting on image data is to artiﬁcially enlarge
the dataset using label-preserving transformations (e.g., [25, 4, 5]). We employ two distinct forms
of data augmentation, both of which allow transformed images to be produced from the original
images with very little computation, so the transformed images do not need to be stored on disk.
In our implementation, the transformed images are generated in Python code on the CPU while the
GPU is training on the previous batch of images. So these data augmentation schemes are, in effect,
computationally free.
The ﬁrst form of data augmentation consists of generating image translations and horizontal reﬂec-
tions. We do this by extracting random 224224patches (and their horizontal reﬂections) from the
256256images and training our network on these extracted patches4. This increases the size of our
training set by a factor of 2048, though the resulting training examples are, of course, highly inter-
dependent. Without this scheme, our network suffers from substantial overﬁtting, which would have
forced us to use much smaller networks. At test time, the network makes a prediction by extracting
ﬁve224224patches (the four corner patches and the center patch) as well as their horizontal
reﬂections (hence ten patches in all), and averaging the predictions made by the network’s softmax
layer on the ten patches.
The second form of data augmentation consists of altering the intensities of the RGB channels in
training images. Speciﬁcally, we perform PCA on the set of RGB pixel values throughout the
ImageNet training set. To each training image, we add multiples of the found principal components,
4This is the reason why the input images in Figure 2 are 2242243-dimensional.
5
with magnitudes proportional to the corresponding eigenvalues times a random variable drawn from
a Gaussian with mean zero and standard deviation 0.1. Therefore to each RGB image pixel Ixy=
[IR
xy;IG
xy;IB
xy]Twe add the following quantity:
[p1;p2;p3][11;22;33]T
where piandiareith eigenvector and eigenvalue of the 33covariance matrix of RGB pixel
values, respectively, and iis the aforementioned random variable. Each iis drawn only once
for all the pixels of a particular training image until that image is used for training again, at which
point it is re-drawn. This scheme approximately captures an important property of natural images,
namely, that object identity is invariant to changes in the intensity and color of the illumination. This
scheme reduces the top-1 error rate by over 1%.
4.2 Dropout
Combining the predictions of many different models is a very successful way to reduce test errors
[1, 3], but it appears to be too expensive for big neural networks that already take several days
to train. There is, however, a very efﬁcient version of model combination that only costs about a
factor of two during training. The recently-introduced technique, called “dropout” [10], consists
of setting to zero the output of each hidden neuron with probability 0.5. The neurons which are
“dropped out” in this way do not contribute to the forward pass and do not participate in back-
propagation. So every time an input is presented, the neural network samples a different architecture,
but all these architectures share weights. This technique reduces complex co-adaptations of neurons,
since a neuron cannot rely on the presence of particular other neurons. It is, therefore, forced to
learn more robust features that are useful in conjunction with many different random subsets of the
other neurons. At test time, we use all the neurons but multiply their outputs by 0.5, which is a
reasonable approximation to taking the geometric mean of the predictive distributions produced by
the exponentially-many dropout networks.
We use dropout in the ﬁrst two fully-connected layers of Figure 2. Without dropout, our network ex-
hibits substantial overﬁtting. Dropout roughly doubles the number of iterations required to converge.
Figure 3: 96 convolutional kernels of size
11113learned by the ﬁrst convolutional
layer on the 2242243input images. The
top 48 kernels were learned on GPU 1 while
the bottom 48 kernels were learned on GPU
2. See Section 6.1 for details.5 Details of learning
We trained our models using stochastic gradient descent
with a batch size of 128 examples, momentum of 0.9, and
weight decay of 0.0005. We found that this small amount
of weight decay was important for the model to learn. In
other words, weight decay here is not merely a regularizer:
it reduces the model’s training error. The update rule for
weightwwas
vi+1 := 0:9vi0:0005wi@L
@w
wi
Di
wi+1 :=wi+vi+1
whereiis the iteration index, vis the momentum variable, is the learning rate, andD
@L
@w
wiE
Diis
the average over the ith batchDiof the derivative of the objective with respect to w, evaluated at
wi.
We initialized the weights in each layer from a zero-mean Gaussian distribution with standard de-
viation 0.01. We initialized the neuron biases in the second, fourth, and ﬁfth convolutional layers,
as well as in the fully-connected hidden layers, with the constant 1. This initialization accelerates
the early stages of learning by providing the ReLUs with positive inputs. We initialized the neuron
biases in the remaining layers with the constant 0.
We used an equal learning rate for all layers, which we adjusted manually throughout training.
The heuristic which we followed was to divide the learning rate by 10 when the validation error
rate stopped improving with the current learning rate. The learning rate was initialized at 0.01 and
6
reduced three times prior to termination. We trained the network for roughly 90 cycles through the
training set of 1.2 million images, which took ﬁve to six days on two NVIDIA GTX 580 3GB GPUs.
6 Results
Our results on ILSVRC-2010 are summarized in Table 1. Our network achieves top-1 and top-5
test set error rates of 37.5% and17.0%5. The best performance achieved during the ILSVRC-
2010 competition was 47.1% and 28.2% with an approach that averages the predictions produced
from six sparse-coding models trained on different features [2], and since then the best pub-
lished results are 45.7% and 25.7% with an approach that averages the predictions of two classi-
ﬁers trained on Fisher Vectors (FVs) computed from two types of densely-sampled features [24].
Model Top-1 Top-5
Sparse coding [2] 47.1% 28.2%
SIFT + FVs [24] 45.7% 25.7%
CNN 37.5% 17.0%
Table 1: Comparison of results on ILSVRC-
2010 test set. In italics are best results
achieved by others.We also entered our model in the ILSVRC-2012 com-
petition and report our results in Table 2. Since the
ILSVRC-2012 test set labels are not publicly available,
we cannot report test error rates for all the models that
we tried. In the remainder of this paragraph, we use
validation and test error rates interchangeably because
in our experience they do not differ by more than 0.1%
(see Table 2). The CNN described in this paper achieves
a top-5 error rate of 18.2%. Averaging the predictions
of ﬁve similar CNNs gives an error rate of 16.4%. Training one CNN, with an extra sixth con-
volutional layer over the last pooling layer, to classify the entire ImageNet Fall 2011 release
(15M images, 22K categories), and then “ﬁne-tuning” it on ILSVRC-2012 gives an error rate of
16.6%. Averaging the predictions of two CNNs that were pre-trained on the entire Fall 2011 re-
lease with the aforementioned ﬁve CNNs gives an error rate of 15.3% . The second-best con-
test entry achieved an error rate of 26.2% with an approach that averages the predictions of sev-
eral classiﬁers trained on FVs computed from different types of densely-sampled features [7].
Model Top-1 (val) Top-5 (val) Top-5 (test)
SIFT + FVs [7] — — 26.2%
1 CNN 40.7% 18.2% —
5 CNNs 38.1% 16.4% 16.4%
1 CNN* 39.0% 16.6% —
7 CNNs* 36.7% 15.4% 15.3%
Table 2: Comparison of error rates on ILSVRC-2012 validation and
test sets. In italics are best results achieved by others. Models with an
asterisk* were “pre-trained” to classify the entire ImageNet 2011 Fall
release. See Section 6 for details.Finally, we also report our error
rates on the Fall 2009 version of
ImageNet with 10,184 categories
and 8.9 million images. On this
dataset we follow the convention
in the literature of using half of
the images for training and half
for testing. Since there is no es-
tablished test set, our split neces-
sarily differs from the splits used
by previous authors, but this does
not affect the results appreciably.
Our top-1 and top-5 error rates
on this dataset are 67.4% and
40.9% , attained by the net described above but with an additional, sixth convolutional layer over the
last pooling layer. The best published results on this dataset are 78.1% and 60.9% [19].
6.1 Qualitative Evaluations
Figure 3 shows the convolutional kernels learned by the network’s two data-connected layers. The
network has learned a variety of frequency- and orientation-selective kernels, as well as various col-
ored blobs. Notice the specialization exhibited by the two GPUs, a result of the restricted connec-
tivity described in Section 3.5. The kernels on GPU 1 are largely color-agnostic, while the kernels
on on GPU 2 are largely color-speciﬁc. This kind of specialization occurs during every run and is
independent of any particular random weight initialization (modulo a renumbering of the GPUs).
5The error rates without averaging predictions over ten patches as described in Section 4.1 are 39.0% and
18.3%.
7
Figure 4: (Left) Eight ILSVRC-2010 test images and the ﬁve labels considered most probable by our model.
The correct label is written under each image, and the probability assigned to the correct label is also shown
with a red bar (if it happens to be in the top 5). (Right) Five ILSVRC-2010 test images in the ﬁrst column. The
remaining columns show the six training images that produce feature vectors in the last hidden layer with the
smallest Euclidean distance from the feature vector for the test image.
In the left panel of Figure 4 we qualitatively assess what the network has learned by computing its
top-5 predictions on eight test images. Notice that even off-center objects, such as the mite in the
top-left, can be recognized by the net. Most of the top-5 labels appear reasonable. For example,
only other types of cat are considered plausible labels for the leopard. In some cases (grille, cherry)
there is genuine ambiguity about the intended focus of the photograph.
Another way to probe the network’s visual knowledge is to consider the feature activations induced
by an image at the last, 4096-dimensional hidden layer. If two images produce feature activation
vectors with a small Euclidean separation, we can say that the higher levels of the neural network
consider them to be similar. Figure 4 shows ﬁve images from the test set and the six images from
the training set that are most similar to each of them according to this measure. Notice that at the
pixel level, the retrieved training images are generally not close in L2 to the query images in the ﬁrst
column. For example, the retrieved dogs and elephants appear in a variety of poses. We present the
results for many more test images in the supplementary material.
Computing similarity by using Euclidean distance between two 4096-dimensional, real-valued vec-
tors is inefﬁcient, but it could be made efﬁcient by training an auto-encoder to compress these vectors
to short binary codes. This should produce a much better image retrieval method than applying auto-
encoders to the raw pixels [14], which does not make use of image labels and hence has a tendency
to retrieve images with similar patterns of edges, whether or not they are semantically similar.
7 Discussion
Our results show that a large, deep convolutional neural network is capable of achieving record-
breaking results on a highly challenging dataset using purely supervised learning. It is notable
that our network’s performance degrades if a single convolutional layer is removed. For example,
removing any of the middle layers results in a loss of about 2% for the top-1 performance of the
network. So the depth really is important for achieving our results.
To simplify our experiments, we did not use any unsupervised pre-training even though we expect
that it will help, especially if we obtain enough computational power to signiﬁcantly increase the
size of the network without obtaining a corresponding increase in the amount of labeled data. Thus
far, our results have improved as we have made our network larger and trained it longer but we still
have many orders of magnitude to go in order to match the infero-temporal pathway of the human
visual system. Ultimately we would like to use very large and deep convolutional nets on video
sequences where the temporal structure provides very helpful information that is missing or far less
obvious in static images.
8
References
[1] R.M. Bell and Y . Koren. Lessons from the netﬂix prize challenge. ACM SIGKDD Explorations Newsletter ,
9(2):75–79, 2007.
[2] A. Berg, J. Deng, and L. Fei-Fei. Large scale visual recognition challenge 2010. www.image-
net.org/challenges. 2010.
[3] L. Breiman. Random forests. Machine learning , 45(1):5–32, 2001.
[4] D. Cire¸ san, U. Meier, and J. Schmidhuber. Multi-column deep neural networks for image classiﬁcation.
Arxiv preprint arXiv:1202.2745 , 2012.
[5] D.C. Cire¸ san, U. Meier, J. Masci, L.M. Gambardella, and J. Schmidhuber. High-performance neural
networks for visual object classiﬁcation. Arxiv preprint arXiv:1102.0183 , 2011.
[6] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical
Image Database. In CVPR09 , 2009.
[7] J. Deng, A. Berg, S. Satheesh, H. Su, A. Khosla, and L. Fei-Fei. ILSVRC-2012 , 2012. URL
http://www.image-net.org/challenges/LSVRC/2012/ .
[8] L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An
incremental bayesian approach tested on 101 object categories. Computer Vision and Image Understand-
ing, 106(1):59–70, 2007.
[9] G. Grifﬁn, A. Holub, and P. Perona. Caltech-256 object category dataset. Technical Report 7694, Cali-
fornia Institute of Technology, 2007. URL http://authors.library.caltech.edu/7694 .
[10] G.E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R.R. Salakhutdinov. Improving neural net-
works by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580 , 2012.
[11] K. Jarrett, K. Kavukcuoglu, M. A. Ranzato, and Y . LeCun. What is the best multi-stage architecture for
object recognition? In International Conference on Computer Vision , pages 2146–2153. IEEE, 2009.
[12] A. Krizhevsky. Learning multiple layers of features from tiny images. Master’s thesis, Department of
Computer Science, University of Toronto, 2009.
[13] A. Krizhevsky. Convolutional deep belief networks on cifar-10. Unpublished manuscript , 2010.
[14] A. Krizhevsky and G.E. Hinton. Using very deep autoencoders for content-based image retrieval. In
ESANN , 2011.
[15] Y . Le Cun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, L.D. Jackel, et al. Hand-
written digit recognition with a back-propagation network. In Advances in neural information processing
systems , 1990.
[16] Y . LeCun, F.J. Huang, and L. Bottou. Learning methods for generic object recognition with invariance to
pose and lighting. In Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the
2004 IEEE Computer Society Conference on , volume 2, pages II–97. IEEE, 2004.
[17] Y . LeCun, K. Kavukcuoglu, and C. Farabet. Convolutional networks and applications in vision. In
Circuits and Systems (ISCAS), Proceedings of 2010 IEEE International Symposium on , pages 253–256.
IEEE, 2010.
[18] H. Lee, R. Grosse, R. Ranganath, and A.Y . Ng. Convolutional deep belief networks for scalable unsuper-
vised learning of hierarchical representations. In Proceedings of the 26th Annual International Conference
on Machine Learning , pages 609–616. ACM, 2009.
[19] T. Mensink, J. Verbeek, F. Perronnin, and G. Csurka. Metric Learning for Large Scale Image Classiﬁ-
cation: Generalizing to New Classes at Near-Zero Cost. In ECCV - European Conference on Computer
Vision , Florence, Italy, October 2012.
[20] V . Nair and G. E. Hinton. Rectiﬁed linear units improve restricted boltzmann machines. In Proc. 27th
International Conference on Machine Learning , 2010.
[21] N. Pinto, D.D. Cox, and J.J. DiCarlo. Why is real-world visual object recognition hard? PLoS computa-
tional biology , 4(1):e27, 2008.
[22] N. Pinto, D. Doukhan, J.J. DiCarlo, and D.D. Cox. A high-throughput screening approach to discovering
good forms of biologically inspired visual representation. PLoS computational biology , 5(11):e1000579,
2009.
[23] B.C. Russell, A. Torralba, K.P. Murphy, and W.T. Freeman. Labelme: a database and web-based tool for
image annotation. International journal of computer vision , 77(1):157–173, 2008.
[24] J. Sánchez and F. Perronnin. High-dimensional signature compression for large-scale image classiﬁcation.
InComputer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on , pages 1665–1672. IEEE,
2011.
[25] P.Y . Simard, D. Steinkraus, and J.C. Platt. Best practices for convolutional neural networks applied to
visual document analysis. In Proceedings of the Seventh International Conference on Document Analysis
and Recognition , volume 2, pages 958–962, 2003.
[26] S.C. Turaga, J.F. Murray, V . Jain, F. Roth, M. Helmstaedter, K. Briggman, W. Denk, and H.S. Seung. Con-
volutional networks can learn to generate afﬁnity graphs for image segmentation. Neural Computation ,
22(2):511–538, 2010.
9

Alex Krizevsky
Ilya Sutskever
Geoff Hinton


Human-level control through deep reinforcement learning

### Subjects

* [Computer science](/subjects/computer-science)

Abstract
--------

The theory of reinforcement learning provides a normative account[1](/articles/nature14236#ref-CR1 "Sutton, R. & Barto, A. Reinforcement Learning: An Introduction (MIT Press, 1998)"), deeply rooted in psychological[2](/articles/nature14236#ref-CR2 "Thorndike, E. L. Animal Intelligence: Experimental studies (Macmillan, 1911)") and neuroscientific[3](/articles/nature14236#ref-CR3 "Schultz, W., Dayan, P. & Montague, P. R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997)") perspectives on animal behaviour, of how agents may optimize their control of an environment. To use reinforcement learning successfully in situations approaching real-world complexity, however, agents are confronted with a difficult task: they must derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experience to new situations. Remarkably, humans and other animals seem to solve this problem through a harmonious combination of reinforcement learning and hierarchical sensory processing systems[4](/articles/nature14236#ref-CR4 "Serre, T., Wolf, L. & Poggio, T. Object recognition with features inspired by visual cortex. Proc. IEEE. Comput. Soc. Conf. Comput. Vis. Pattern. Recognit. 994–1000 (2005)"),[5](/articles/nature14236#ref-CR5 "Fukushima, K. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol. Cybern. 36, 193–202 (1980)"), the former evidenced by a wealth of neural data revealing notable parallels between the phasic signals emitted by dopaminergic neurons and temporal difference reinforcement learning algorithms[3](/articles/nature14236#ref-CR3 "Schultz, W., Dayan, P. & Montague, P. R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997)"). While reinforcement learning agents have achieved some successes in a variety of domains[6](/articles/nature14236#ref-CR6 "Tesauro, G. Temporal difference learning and TD-Gammon. Commun. ACM 38, 58–68 (1995)"),[7](/articles/nature14236#ref-CR7 "Riedmiller, M., Gabel, T., Hafner, R. & Lange, S. Reinforcement learning for robot soccer. Auton. Robots 27, 55–73 (2009)"),[8](/articles/nature14236#ref-CR8 "Diuk, C., Cohen, A. & Littman, M. L. An object-oriented representation for efficient reinforcement learning. Proc. Int. Conf. Mach. Learn. 240–247 (2008)"), their applicability has previously been limited to domains in which useful features can be handcrafted, or to domains with fully observed, low-dimensional state spaces. Here we use recent advances in training deep neural networks[9](/articles/nature14236#ref-CR9 "Bengio, Y. Learning deep architectures for AI. Foundations and Trends in Machine Learning 2, 1–127 (2009)"),[10](/articles/nature14236#ref-CR10 "Krizhevsky, A., Sutskever, I. & Hinton, G. ImageNet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25, 1106–1114 (2012)"),[11](/articles/nature14236#ref-CR11 "Hinton, G. E. & Salakhutdinov, R. R. Reducing the dimensionality of data with neural networks. Science 313, 504–507 (2006)") to develop a novel artificial agent, termed a deep Q-network, that can learn successful policies directly from high-dimensional sensory inputs using end-to-end reinforcement learning. We tested this agent on the challenging domain of classic Atari 2600 games[12](/articles/nature14236#ref-CR12 "Bellemare, M. G., Naddaf, Y., Veness, J. & Bowling, M. The arcade learning environment: An evaluation platform for general agents. J. Artif. Intell. Res. 47, 253–279 (2013)"). We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters. This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks.

[Access through your institution](https://wayf.springernature.com?redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fnature14236)

[Buy or subscribe](#access-options)

This is a preview of subscription content, [access via your institution](https://wayf.springernature.com?redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fnature14236)

 if (window.dataLayer) {
 window.dataLayer.push({
 content: { article: { relevantArticlesCount: 3 }}
 })
 }

Relevant articles
-----------------

Open Access articles citing this article.

* ###
[Task offloading optimization mechanism based on deep neural network in edge-cloud environment](https://doi.org/10.1186/s13677-023-00450-6)

	+ Lingkang Meng
	+ , Yingjie Wang
	+ … Zhipeng Cai
*Journal of Cloud Computing*
Open Access
11 May 2023
* ###
[A dosing strategy model of deep deterministic policy gradient algorithm for sepsis patients](https://doi.org/10.1186/s12911-023-02175-7)

	+ Tianlai Lin
	+ , Xinjue Zhang
	+ … Junhui Gao
*BMC Medical Informatics and Decision Making*
Open Access
04 May 2023
* ###
[Governance and sustainability of distributed continuum systems: a big data approach](https://doi.org/10.1186/s40537-023-00737-0)

	+ Praveen Kumar Donta
	+ , Boris Sedlak
	+ … Schahram Dustdar
*Journal of Big Data*
Open Access
28 April 2023

Access options
--------------

[Access through your institution](https://wayf.springernature.com?redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fnature14236)

[Access through your institution](https://wayf.springernature.com?redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fnature14236)

[Change institution](https://wayf.springernature.com?redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fnature14236)

[Buy or subscribe](#access-options)

/\* style specs start \*/
style{display:none!important}.LiveAreaSection-193358632 \*{align-content:stretch;align-items:stretch;align-self:auto;animation-delay:0s;animation-direction:normal;animation-duration:0s;animation-fill-mode:none;animation-iteration-count:1;animation-name:none;animation-play-state:running;animation-timing-function:ease;azimuth:center;backface-visibility:visible;background-attachment:scroll;background-blend-mode:normal;background-clip:borderBox;background-color:transparent;background-image:none;background-origin:paddingBox;background-position:0 0;background-repeat:repeat;background-size:auto auto;block-size:auto;border-block-end-color:currentcolor;border-block-end-style:none;border-block-end-width:medium;border-block-start-color:currentcolor;border-block-start-style:none;border-block-start-width:medium;border-bottom-color:currentcolor;border-bottom-left-radius:0;border-bottom-right-radius:0;border-bottom-style:none;border-bottom-width:medium;border-collapse:separate;border-image-outset:0s;border-image-repeat:stretch;border-image-slice:100%;border-image-source:none;border-image-width:1;border-inline-end-color:currentcolor;border-inline-end-style:none;border-inline-end-width:medium;border-inline-start-color:currentcolor;border-inline-start-style:none;border-inline-start-width:medium;border-left-color:currentcolor;border-left-style:none;border-left-width:medium;border-right-color:currentcolor;border-right-style:none;border-right-width:medium;border-spacing:0;border-top-color:currentcolor;border-top-left-radius:0;border-top-right-radius:0;border-top-style:none;border-top-width:medium;bottom:auto;box-decoration-break:slice;box-shadow:none;box-sizing:border-box;break-after:auto;break-before:auto;break-inside:auto;caption-side:top;caret-color:auto;clear:none;clip:auto;clip-path:none;color:initial;column-count:auto;column-fill:balance;column-gap:normal;column-rule-color:currentcolor;column-rule-style:none;column-rule-width:medium;column-span:none;column-width:auto;content:normal;counter-increment:none;counter-reset:none;cursor:auto;display:inline;empty-cells:show;filter:none;flex-basis:auto;flex-direction:row;flex-grow:0;flex-shrink:1;flex-wrap:nowrap;float:none;font-family:initial;font-feature-settings:normal;font-kerning:auto;font-language-override:normal;font-size:medium;font-size-adjust:none;font-stretch:normal;font-style:normal;font-synthesis:weight style;font-variant:normal;font-variant-alternates:normal;font-variant-caps:normal;font-variant-east-asian:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-position:normal;font-weight:400;grid-auto-columns:auto;grid-auto-flow:row;grid-auto-rows:auto;grid-column-end:auto;grid-column-gap:0;grid-column-start:auto;grid-row-end:auto;grid-row-gap:0;grid-row-start:auto;grid-template-areas:none;grid-template-columns:none;grid-template-rows:none;height:auto;hyphens:manual;image-orientation:0deg;image-rendering:auto;image-resolution:1dppx;ime-mode:auto;inline-size:auto;isolation:auto;justify-content:flexStart;left:auto;letter-spacing:normal;line-break:auto;line-height:normal;list-style-image:none;list-style-position:outside;list-style-type:disc;margin-block-end:0;margin-block-start:0;margin-bottom:0;margin-inline-end:0;margin-inline-start:0;margin-left:0;margin-right:0;margin-top:0;mask-clip:borderBox;mask-composite:add;mask-image:none;mask-mode:matchSource;mask-origin:borderBox;mask-position:0 0;mask-repeat:repeat;mask-size:auto;mask-type:luminance;max-height:none;max-width:none;min-block-size:0;min-height:0;min-inline-size:0;min-width:0;mix-blend-mode:normal;object-fit:fill;object-position:50% 50%;offset-block-end:auto;offset-block-start:auto;offset-inline-end:auto;offset-inline-start:auto;opacity:1;order:0;orphans:2;outline-color:initial;outline-offset:0;outline-style:none;outline-width:medium;overflow:visible;overflow-wrap:normal;overflow-x:visible;overflow-y:visible;padding-block-end:0;padding-block-start:0;padding-bottom:0;padding-inline-end:0;padding-inline-start:0;padding-left:0;padding-right:0;padding-top:0;page-break-after:auto;page-break-before:auto;page-break-inside:auto;perspective:none;perspective-origin:50% 50%;pointer-events:auto;position:static;quotes:initial;resize:none;right:auto;ruby-align:spaceAround;ruby-merge:separate;ruby-position:over;scroll-behavior:auto;scroll-snap-coordinate:none;scroll-snap-destination:0 0;scroll-snap-points-x:none;scroll-snap-points-y:none;scroll-snap-type:none;shape-image-threshold:0;shape-margin:0;shape-outside:none;tab-size:8;table-layout:auto;text-align:initial;text-align-last:auto;text-combine-upright:none;text-decoration-color:currentcolor;text-decoration-line:none;text-decoration-style:solid;text-emphasis-color:currentcolor;text-emphasis-position:over right;text-emphasis-style:none;text-indent:0;text-justify:auto;text-orientation:mixed;text-overflow:clip;text-rendering:auto;text-shadow:none;text-transform:none;text-underline-position:auto;top:auto;touch-action:auto;transform:none;transform-box:borderBox;transform-origin:50% 50%0;transform-style:flat;transition-delay:0s;transition-duration:0s;transition-property:all;transition-timing-function:ease;vertical-align:baseline;visibility:visible;white-space:normal;widows:2;width:auto;will-change:auto;word-break:normal;word-spacing:normal;word-wrap:normal;writing-mode:horizontalTb;z-index:auto;-webkit-appearance:none;-moz-appearance:none;-ms-appearance:none;appearance:none;margin:0}.LiveAreaSection-193358632{width:100%}.LiveAreaSection-193358632 .login-option-buybox{display:block;width:100%;font-size:17px;line-height:30px;color:#222;padding-top:30px;font-family:Harding,Palatino,serif}.LiveAreaSection-193358632 .additional-access-options{display:block;font-weight:700;font-size:17px;line-height:30px;color:#222;font-family:Harding,Palatino,serif}.LiveAreaSection-193358632 .additional-login>li:not(:first-child)::before{transform:translateY(-50%);content:"";height:1rem;position:absolute;top:50%;left:0;border-left:2px solid #999}.LiveAreaSection-193358632 .additional-login>li:not(:first-child){padding-left:10px}.LiveAreaSection-193358632 .additional-login>li{display:inline-block;position:relative;vertical-align:middle;padding-right:10px}.BuyBoxSection-683559780{display:flex;flex-wrap:wrap;flex:1;flex-direction:row-reverse;margin:-30px -15px 0}.BuyBoxSection-683559780 .box-inner{width:100%;height:100%}.BuyBoxSection-683559780 .readcube-buybox{background-color:#f3f3f3;flex-shrink:1;flex-grow:1;flex-basis:255px;background-clip:content-box;padding:0 15px;margin-top:30px}.BuyBoxSection-683559780 .subscribe-buybox{background-color:#f3f3f3;flex-shrink:1;flex-grow:4;flex-basis:300px;background-clip:content-box;padding:0 15px;margin-top:30px}.BuyBoxSection-683559780 .subscribe-buybox-nature-plus{background-color:#f3f3f3;flex-shrink:1;flex-grow:4;flex-basis:100%;background-clip:content-box;padding:0 15px;margin-top:30px}.BuyBoxSection-683559780 .title-readcube,.BuyBoxSection-683559780 .title-buybox{display:block;margin:0;margin-right:10%;margin-left:10%;font-size:24px;line-height:32px;color:#222;padding-top:30px;text-align:center;font-family:Harding,Palatino,serif}.BuyBoxSection-683559780 .title-asia-buybox{display:block;margin:0;margin-right:5%;margin-left:5%;font-size:24px;line-height:32px;color:#222;padding-top:30px;text-align:center;font-family:Harding,Palatino,serif}.BuyBoxSection-683559780 .asia-link{color:#069;cursor:pointer;text-decoration:none;font-size:1.05em;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:1.05em6}.BuyBoxSection-683559780 .access-readcube{display:block;margin:0;margin-right:10%;margin-left:10%;font-size:14px;color:#222;padding-top:10px;text-align:center;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:20px}.BuyBoxSection-683559780 .access-asia-buybox{display:block;margin:0;margin-right:5%;margin-left:5%;font-size:14px;color:#222;padding-top:10px;text-align:center;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:20px}.BuyBoxSection-683559780 .access-buybox{display:block;margin:0;margin-right:10%;margin-left:10%;font-size:14px;color:#222;opacity:.8px;padding-top:10px;text-align:center;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:20px}.BuyBoxSection-683559780 .price-buybox{display:block;font-size:30px;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;padding-top:30px;text-align:center}.BuyBoxSection-683559780 .price-buybox-to{display:block;font-size:30px;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;text-align:center}.BuyBoxSection-683559780 .price-info-text{font-size:16px;padding-right:10px;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif}.BuyBoxSection-683559780 .price-value{font-size:30px;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif}.BuyBoxSection-683559780 .price-per-period{font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif}.BuyBoxSection-683559780 .price-from{font-size:14px;padding-right:10px;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:20px}.BuyBoxSection-683559780 .issue-buybox{display:block;font-size:13px;text-align:center;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:19px}.BuyBoxSection-683559780 .no-price-buybox{display:block;font-size:13px;line-height:18px;text-align:center;padding-right:10%;padding-left:10%;padding-bottom:20px;padding-top:30px;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif}.BuyBoxSection-683559780 .vat-buybox{display:block;margin-top:5px;margin-right:20%;margin-left:20%;font-size:11px;color:#222;padding-top:10px;padding-bottom:15px;text-align:center;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:17px}.BuyBoxSection-683559780 .tax-buybox{display:block;width:100%;color:#222;padding:20px 16px;text-align:center;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:NaNpx}.BuyBoxSection-683559780 .button-container{display:flex;padding-right:20px;padding-left:20px;justify-content:center}.BuyBoxSection-683559780 .button-container>\*{flex:1px}.BuyBoxSection-683559780 .button-container>a:hover,.Button-1078489254:hover,.Button-2496381730:hover{text-decoration:none}.BuyBoxSection-683559780 .readcube-button{background:#fff;margin-top:30px}.BuyBoxSection-683559780 .button-asia{background:#069;border:1px solid #069;border-radius:0;cursor:pointer;display:block;padding:9px;outline:0;text-align:center;text-decoration:none;min-width:80px;margin-top:75px}.BuyBoxSection-683559780 .button-label-asia,.ButtonLabel-3296148077,.ButtonLabel-1651148777{display:block;color:#fff;font-size:17px;line-height:20px;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;text-align:center;text-decoration:none;cursor:pointer}.Button-1078489254,.Button-2496381730{background:#069;border:1px solid #069;border-radius:0;cursor:pointer;display:block;padding:9px;outline:0;text-align:center;text-decoration:none;min-width:80px;max-width:320px;margin-top:10px}.Button-1078489254 .readcube-label,.Button-2496381730 .readcube-label{color:#069}
/\* style specs end \*/Subscribe to this journal

Receive 51 print issues and online access

$199.00 per year

only $3.90 per issue

[Learn more](/nature/subscribe)Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

[Learn more](//www.nature.com/articles/nature14236.epdf?no_publisher_access=1&r3_referer=nature)Prices may be subject to local taxes which are calculated during checkout

### Additional access options:

* [Log in](https://idp.nature.com/authorize/natureuser?client_id=grover&redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fnature14236)
* [Learn about institutional subscriptions](https://www.springernature.com/gp/librarians/licensing/license-options)
* [Read our FAQs](https://support.nature.com/en/support/home)
* [Contact customer support](https://www.springernature.com/gp/contact)

**Figure 1: Schematic illustration of the convolutional neural network.**![](//media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fnature14236/MediaObjects/41586_2015_BFnature14236_Fig1_HTML.jpg)**Figure 2: Training curves tracking the agent’s average score and average predicted action-value.**![](//media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fnature14236/MediaObjects/41586_2015_BFnature14236_Fig2_HTML.jpg)**Figure 3: Comparison of the DQN agent with the best reinforcement learning methods[15](/articles/nature14236#ref-CR15 "Bellemare, M. G., Veness, J. & Bowling, M. Investigating contingency awareness using Atari 2600 games. Proc. Conf. AAAI. Artif. Intell. 864–871 (2012)") in the literature.**![](//media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fnature14236/MediaObjects/41586_2015_BFnature14236_Fig3_HTML.jpg)**Figure 4: Two-dimensional t-SNE embedding of the representations in the last hidden layer assigned by DQN to game states experienced while playing Space Invaders.**![](//media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fnature14236/MediaObjects/41586_2015_BFnature14236_Fig4_HTML.jpg)

References
----------

1. Sutton, R. & Barto, A. *Reinforcement Learning: An Introduction* (MIT Press, 1998)

[MATH](http://www.emis.de/MATH-item?1407.68009)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Reinforcement%20Learning%3A%20An%20Introduction&publication_year=1998&author=Sutton%2CR&author=Barto%2CA)
2. Thorndike, E. L. *Animal Intelligence: Experimental studies* (Macmillan, 1911)

[Book](https://doi.org/10.5962%2Fbhl.title.55072)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Animal%20Intelligence%3A%20Experimental%20studies&doi=10.5962%2Fbhl.title.55072&publication_year=1911&author=Thorndike%2CEL)
3. Schultz, W., Dayan, P. & Montague, P. R. A neural substrate of prediction and reward. *Science* **275**, 1593–1599 (1997)

[Article](https://doi.org/10.1126%2Fscience.275.5306.1593)
 [CAS](/articles/cas-redirect/1:CAS:528:DyaK2sXhvFSntro%3D)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=A%20neural%20substrate%20of%20prediction%20and%20reward&journal=Science&doi=10.1126%2Fscience.275.5306.1593&volume=275&pages=1593-1599&publication_year=1997&author=Schultz%2CW&author=Dayan%2CP&author=Montague%2CPR)
4. Serre, T., Wolf, L. & Poggio, T. Object recognition with features inspired by visual cortex. *Proc. IEEE. Comput. Soc. Conf. Comput. Vis. Pattern. Recognit.* 994–1000 (2005)

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Object%20recognition%20with%20features%20inspired%20by%20visual%20cortex&pages=994-1000&publication_year=2005&author=Serre%2CT&author=Wolf%2CL&author=Poggio%2CT)
5. Fukushima, K. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. *Biol. Cybern.* **36**, 193–202 (1980)

[Article](https://doi.org/10.1007%2FBF00344251)
 [CAS](/articles/cas-redirect/1:STN:280:DyaL3c7nsFKntw%3D%3D)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Neocognitron%3A%20A%20self-organizing%20neural%20network%20model%20for%20a%20mechanism%20of%20pattern%20recognition%20unaffected%20by%20shift%20in%20position&journal=Biol.%20Cybern.&doi=10.1007%2FBF00344251&volume=36&pages=193-202&publication_year=1980&author=Fukushima%2CK)
6. Tesauro, G. Temporal difference learning and TD-Gammon. *Commun. ACM* **38**, 58–68 (1995)

[Article](https://doi.org/10.1145%2F203330.203343)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Temporal%20difference%20learning%20and%20TD-Gammon&journal=Commun.%20ACM&doi=10.1145%2F203330.203343&volume=38&pages=58-68&publication_year=1995&author=Tesauro%2CG)
7. Riedmiller, M., Gabel, T., Hafner, R. & Lange, S. Reinforcement learning for robot soccer. *Auton. Robots* **27**, 55–73 (2009)

[Article](https://doi.org/10.1007%2Fs10514-009-9120-4)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Reinforcement%20learning%20for%20robot%20soccer&journal=Auton.%20Robots&doi=10.1007%2Fs10514-009-9120-4&volume=27&pages=55-73&publication_year=2009&author=Riedmiller%2CM&author=Gabel%2CT&author=Hafner%2CR&author=Lange%2CS)
8. Diuk, C., Cohen, A. & Littman, M. L. An object-oriented representation for efficient reinforcement learning. *Proc. Int. Conf. Mach. Learn.* 240–247 (2008)

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=An%20object-oriented%20representation%20for%20efficient%20reinforcement%20learning&pages=240-247&publication_year=2008&author=Diuk%2CC&author=Cohen%2CA&author=Littman%2CML)
9. Bengio, Y. Learning deep architectures for AI. *Foundations and Trends in Machine Learning* **2**, 1–127 (2009)

[Article](https://doi.org/10.1561%2F2200000006)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Learning%20deep%20architectures%20for%20AI&journal=Foundations%20and%20Trends%20in%20Machine%20Learning&doi=10.1561%2F2200000006&volume=2&pages=1-127&publication_year=2009&author=Bengio%2CY)
10. Krizhevsky, A., Sutskever, I. & Hinton, G. ImageNet classification with deep convolutional neural networks. *Adv. Neural Inf. Process. Syst.* **25**, 1106–1114 (2012)

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=ImageNet%20classification%20with%20deep%20convolutional%20neural%20networks&journal=Adv.%20Neural%20Inf.%20Process.%20Syst.&volume=25&pages=1106-1114&publication_year=2012&author=Krizhevsky%2CA&author=Sutskever%2CI&author=Hinton%2CG)
11. Hinton, G. E. & Salakhutdinov, R. R. Reducing the dimensionality of data with neural networks. *Science* **313**, 504–507 (2006)

[Article](https://doi.org/10.1126%2Fscience.1127647)
 [ADS](http://adsabs.harvard.edu/cgi-bin/nph-data_query?link_type=ABSTRACT&bibcode=2006Sci...313..504H)
 [MathSciNet](http://www.ams.org/mathscinet-getitem?mr=2242509)
 [CAS](/articles/cas-redirect/1:CAS:528:DC%2BD28Xnt1KntrY%3D)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Reducing%20the%20dimensionality%20of%20data%20with%20neural%20networks&journal=Science&doi=10.1126%2Fscience.1127647&volume=313&pages=504-507&publication_year=2006&author=Hinton%2CGE&author=Salakhutdinov%2CRR)
12. Bellemare, M. G., Naddaf, Y., Veness, J. & Bowling, M. The arcade learning environment: An evaluation platform for general agents. *J. Artif. Intell. Res.* **47**, 253–279 (2013)

[Article](https://doi.org/10.1613%2Fjair.3912)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=The%20arcade%20learning%20environment%3A%20An%20evaluation%20platform%20for%20general%20agents&journal=J.%20Artif.%20Intell.%20Res.&doi=10.1613%2Fjair.3912&volume=47&pages=253-279&publication_year=2013&author=Bellemare%2CMG&author=Naddaf%2CY&author=Veness%2CJ&author=Bowling%2CM)
13. Legg, S. & Hutter, M. Universal Intelligence: a definition of machine intelligence. *Minds Mach.* **17**, 391–444 (2007)

[Article](https://doi.org/10.1007%2Fs11023-007-9079-x)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Universal%20Intelligence%3A%20a%20definition%20of%20machine%20intelligence&journal=Minds%20Mach.&doi=10.1007%2Fs11023-007-9079-x&volume=17&pages=391-444&publication_year=2007&author=Legg%2CS&author=Hutter%2CM)
14. Genesereth, M., Love, N. & Pell, B. General game playing: overview of the AAAI competition. *AI Mag.* **26**, 62–72 (2005)

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=General%20game%20playing%3A%20overview%20of%20the%20AAAI%20competition&journal=AI%20Mag.&volume=26&pages=62-72&publication_year=2005&author=Genesereth%2CM&author=Love%2CN&author=Pell%2CB)
15. Bellemare, M. G., Veness, J. & Bowling, M. Investigating contingency awareness using Atari 2600 games. *Proc. Conf. AAAI. Artif. Intell.* 864–871 (2012)

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Investigating%20contingency%20awareness%20using%20Atari%202600%20games&pages=864-871&publication_year=2012&author=Bellemare%2CMG&author=Veness%2CJ&author=Bowling%2CM)
16. McClelland, J. L., Rumelhart, D. E. & Group, T. P. R. *Parallel Distributed Processing: Explorations in the Microstructure of Cognition* (MIT Press, 1986)

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Parallel%20Distributed%20Processing%3A%20Explorations%20in%20the%20Microstructure%20of%20Cognition&publication_year=1986&author=McClelland%2CJL&author=Rumelhart%2CDE&author=Group%2CTPR)
17. LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P. Gradient-based learning applied to document recognition. *Proc. IEEE* **86**, 2278–2324 (1998)

[Article](https://doi.org/10.1109%2F5.726791)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Gradient-based%20learning%20applied%20to%20document%20recognition&journal=Proc.%20IEEE&doi=10.1109%2F5.726791&volume=86&pages=2278-2324&publication_year=1998&author=LeCun%2CY&author=Bottou%2CL&author=Bengio%2CY&author=Haffner%2CP)
18. Hubel, D. H. & Wiesel, T. N. Shape and arrangement of columns in cat’s striate cortex. *J. Physiol.* **165**, 559–568 (1963)

[Article](https://doi.org/10.1113%2Fjphysiol.1963.sp007079)
 [CAS](/articles/cas-redirect/1:STN:280:DyaF387kslWitA%3D%3D)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Shape%20and%20arrangement%20of%20columns%20in%20cat%E2%80%99s%20striate%20cortex&journal=J.%20Physiol.&doi=10.1113%2Fjphysiol.1963.sp007079&volume=165&pages=559-568&publication_year=1963&author=Hubel%2CDH&author=Wiesel%2CTN)
19. Watkins, C. J. & Dayan, P. Q-learning. *Mach. Learn.* **8**, 279–292 (1992)

[MATH](http://www.emis.de/MATH-item?0773.68062)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Q-learning&journal=Mach.%20Learn.&volume=8&pages=279-292&publication_year=1992&author=Watkins%2CCJ&author=Dayan%2CP)
20. Tsitsiklis, J. & Roy, B. V. An analysis of temporal-difference learning with function approximation. *IEEE Trans. Automat. Contr.* **42**, 674–690 (1997)

[Article](https://doi.org/10.1109%2F9.580874)
 [MathSciNet](http://www.ams.org/mathscinet-getitem?mr=1454208)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=An%20analysis%20of%20temporal-difference%20learning%20with%20function%20approximation&journal=IEEE%20Trans.%20Automat.%20Contr.&doi=10.1109%2F9.580874&volume=42&pages=674-690&publication_year=1997&author=Tsitsiklis%2CJ&author=Roy%2CBV)
21. McClelland, J. L., McNaughton, B. L. & O’Reilly, R. C. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. *Psychol. Rev.* **102**, 419–457 (1995)

[Article](https://doi.org/10.1037%2F0033-295X.102.3.419)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Why%20there%20are%20complementary%20learning%20systems%20in%20the%20hippocampus%20and%20neocortex%3A%20insights%20from%20the%20successes%20and%20failures%20of%20connectionist%20models%20of%20learning%20and%20memory&journal=Psychol.%20Rev.&doi=10.1037%2F0033-295X.102.3.419&volume=102&pages=419-457&publication_year=1995&author=McClelland%2CJL&author=McNaughton%2CBL&author=O%E2%80%99Reilly%2CRC)
22. O’Neill, J., Pleydell-Bouverie, B., Dupret, D. & Csicsvari, J. Play it again: reactivation of waking experience and memory. *Trends Neurosci.* **33**, 220–229 (2010)

[Article](https://doi.org/10.1016%2Fj.tins.2010.01.006)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Play%20it%20again%3A%20reactivation%20of%20waking%20experience%20and%20memory&journal=Trends%20Neurosci.&doi=10.1016%2Fj.tins.2010.01.006&volume=33&pages=220-229&publication_year=2010&author=O%E2%80%99Neill%2CJ&author=Pleydell-Bouverie%2CB&author=Dupret%2CD&author=Csicsvari%2CJ)
23. Lin, L.-J. Reinforcement learning for robots using neural networks. Technical Report, DTIC Document. (1993)
24. Riedmiller, M. Neural fitted Q iteration - first experiences with a data efficient neural reinforcement learning method. *Mach. Learn.: ECML* **3720**, 317–328 (Springer, 2005)

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Neural%20fitted%20Q%20iteration%20-%20first%20experiences%20with%20a%20data%20efficient%20neural%20reinforcement%20learning%20method&journal=Mach.%20Learn.%3A%20ECML&volume=3720&pages=317-328&publication_year=2005&author=Riedmiller%2CM)
25. Van der Maaten, L. J. P. & Hinton, G. E. Visualizing high-dimensional data using t-SNE. *J. Mach. Learn. Res.* **9**, 2579–2605 (2008)

[MATH](http://www.emis.de/MATH-item?1225.68219)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Visualizing%20high-dimensional%20data%20using%20t-SNE&journal=J.%20Mach.%20Learn.%20Res.&volume=9&pages=2579-2605&publication_year=2008&author=Van%20der%20Maaten%2CLJP&author=Hinton%2CGE)
26. Lange, S. & Riedmiller, M. Deep auto-encoder neural networks in reinforcement learning. *Proc. Int. Jt. Conf. Neural. Netw.* 1–8 (2010)

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Deep%20auto-encoder%20neural%20networks%20in%20reinforcement%20learning&pages=1-8&publication_year=2010&author=Lange%2CS&author=Riedmiller%2CM)
27. Law, C.-T. & Gold, J. I. Reinforcement learning can account for associative and perceptual learning on a visual decision task. *Nature Neurosci.* **12**, 655 (2009)

[Article](https://doi.org/10.1038%2Fnn.2304)
 [CAS](/articles/cas-redirect/1:CAS:528:DC%2BD1MXks12ktbw%3D)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Reinforcement%20learning%20can%20account%20for%20associative%20and%20perceptual%20learning%20on%20a%20visual%20decision%20task&journal=Nature%20Neurosci.&doi=10.1038%2Fnn.2304&volume=12&publication_year=2009&author=Law%2CC-T&author=Gold%2CJI)
28. Sigala, N. & Logothetis, N. K. Visual categorization shapes feature selectivity in the primate temporal cortex. *Nature* **415**, 318–320 (2002)

[Article](https://doi.org/10.1038%2F415318a)
 [ADS](http://adsabs.harvard.edu/cgi-bin/nph-data_query?link_type=ABSTRACT&bibcode=2002Natur.415..318S)
 [CAS](/articles/cas-redirect/1:CAS:528:DC%2BD38Xptlajsg%3D%3D)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Visual%20categorization%20shapes%20feature%20selectivity%20in%20the%20primate%20temporal%20cortex&journal=Nature&doi=10.1038%2F415318a&volume=415&pages=318-320&publication_year=2002&author=Sigala%2CN&author=Logothetis%2CNK)
29. Bendor, D. & Wilson, M. A. Biasing the content of hippocampal replay during sleep. *Nature Neurosci.* **15**, 1439–1444 (2012)

[Article](https://doi.org/10.1038%2Fnn.3203)
 [CAS](/articles/cas-redirect/1:CAS:528:DC%2BC38Xht12js7bO)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Biasing%20the%20content%20of%20hippocampal%20replay%20during%20sleep&journal=Nature%20Neurosci.&doi=10.1038%2Fnn.3203&volume=15&pages=1439-1444&publication_year=2012&author=Bendor%2CD&author=Wilson%2CMA)
30. Moore, A. & Atkeson, C. Prioritized sweeping: reinforcement learning with less data and less real time. *Mach. Learn.* **13**, 103–130 (1993)

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Prioritized%20sweeping%3A%20reinforcement%20learning%20with%20less%20data%20and%20less%20real%20time&journal=Mach.%20Learn.&volume=13&pages=103-130&publication_year=1993&author=Moore%2CA&author=Atkeson%2CC)
31. Jarrett, K., Kavukcuoglu, K., Ranzato, M. A. & LeCun, Y. What is the best multi-stage architecture for object recognition? *Proc. IEEE. Int. Conf. Comput. Vis.* 2146–2153 (2009)

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=What%20is%20the%20best%20multi-stage%20architecture%20for%20object%20recognition&pages=2146-2153&publication_year=2009&author=Jarrett%2CK&author=Kavukcuoglu%2CK&author=Ranzato%2CMA&author=LeCun%2CY)
32. Nair, V. & Hinton, G. E. Rectified linear units improve restricted Boltzmann machines. *Proc. Int. Conf. Mach. Learn.* 807–814 (2010)
33. Kaelbling, L. P., Littman, M. L. & Cassandra, A. R. Planning and acting in partially observable stochastic domains. *Artificial Intelligence* **101**, 99–134 (1994)

[Article](https://doi.org/10.1016%2FS0004-3702%2898%2900023-X)
 [MathSciNet](http://www.ams.org/mathscinet-getitem?mr=1641530)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Planning%20and%20acting%20in%20partially%20observable%20stochastic%20domains&journal=Artificial%20Intelligence&doi=10.1016%2FS0004-3702%2898%2900023-X&volume=101&pages=99-134&publication_year=1994&author=Kaelbling%2CLP&author=Littman%2CML&author=Cassandra%2CAR)

[Download references](https://citation-needed.springer.com/v2/references/10.1038/nature14236?format=refman&flavour=references)

Acknowledgements
----------------

We thank G. Hinton, P. Dayan and M. Bowling for discussions, A. Cain and J. Keene for work on the visuals, K. Keller and P. Rogers for help with the visuals, G. Wayne for comments on an earlier version of the manuscript, and the rest of the DeepMind team for their support, ideas and encouragement.

Author information
------------------

Author notes1. Volodymyr Mnih, Koray Kavukcuoglu and David Silver: These authors contributed equally to this work.

### Authors and Affiliations

1. Google DeepMind, 5 New Street Square, London EC4A 3TW, UK,

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg & Demis Hassabis

Authors1. Volodymyr Mnih[View author publications](/search?author=Volodymyr%20Mnih)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Volodymyr%20Mnih) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Volodymyr%20Mnih%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
2. Koray Kavukcuoglu[View author publications](/search?author=Koray%20Kavukcuoglu)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Koray%20Kavukcuoglu) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Koray%20Kavukcuoglu%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
3. David Silver[View author publications](/search?author=David%20Silver)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=David%20Silver) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22David%20Silver%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
4. Andrei A. Rusu[View author publications](/search?author=Andrei%20A.%20Rusu)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Andrei%20A.%20Rusu) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Andrei%20A.%20Rusu%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
5. Joel Veness[View author publications](/search?author=Joel%20Veness)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Joel%20Veness) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Joel%20Veness%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
6. Marc G. Bellemare[View author publications](/search?author=Marc%20G.%20Bellemare)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Marc%20G.%20Bellemare) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Marc%20G.%20Bellemare%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
7. Alex Graves[View author publications](/search?author=Alex%20Graves)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Alex%20Graves) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Alex%20Graves%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
8. Martin Riedmiller[View author publications](/search?author=Martin%20Riedmiller)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Martin%20Riedmiller) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Martin%20Riedmiller%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
9. Andreas K. Fidjeland[View author publications](/search?author=Andreas%20K.%20Fidjeland)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Andreas%20K.%20Fidjeland) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Andreas%20K.%20Fidjeland%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
10. Georg Ostrovski[View author publications](/search?author=Georg%20Ostrovski)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Georg%20Ostrovski) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Georg%20Ostrovski%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
11. Stig Petersen[View author publications](/search?author=Stig%20Petersen)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Stig%20Petersen) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Stig%20Petersen%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
12. Charles Beattie[View author publications](/search?author=Charles%20Beattie)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Charles%20Beattie) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Charles%20Beattie%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
13. Amir Sadik[View author publications](/search?author=Amir%20Sadik)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Amir%20Sadik) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Amir%20Sadik%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
14. Ioannis Antonoglou[View author publications](/search?author=Ioannis%20Antonoglou)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Ioannis%20Antonoglou) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Ioannis%20Antonoglou%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
15. Helen King[View author publications](/search?author=Helen%20King)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Helen%20King) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Helen%20King%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
16. Dharshan Kumaran[View author publications](/search?author=Dharshan%20Kumaran)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Dharshan%20Kumaran) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Dharshan%20Kumaran%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
17. Daan Wierstra[View author publications](/search?author=Daan%20Wierstra)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Daan%20Wierstra) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Daan%20Wierstra%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
18. Shane Legg[View author publications](/search?author=Shane%20Legg)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Shane%20Legg) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Shane%20Legg%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
19. Demis Hassabis[View author publications](/search?author=Demis%20Hassabis)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Demis%20Hassabis) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Demis%20Hassabis%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
### Contributions

V.M., K.K., D.S., J.V., M.G.B., M.R., A.G., D.W., S.L. and D.H. conceptualized the problem and the technical framework. V.M., K.K., A.A.R. and D.S. developed and tested the algorithms. J.V., S.P., C.B., A.A.R., M.G.B., I.A., A.K.F., G.O. and A.S. created the testing platform. K.K., H.K., S.L. and D.H. managed the project. K.K., D.K., D.H., V.M., D.S., A.G., A.A.R., J.V. and M.G.B. wrote the paper.

### Corresponding authors

Correspondence to
 [Koray Kavukcuoglu](mailto:korayk@google.com) or [Demis Hassabis](mailto:demishassabis@google.com).

Ethics declarations
-------------------

### Competing interests

The authors declare no competing financial interests.

Extended data figures and tables
--------------------------------

### [Extended Data Figure 1 Two-dimensional t-SNE embedding of the representations in the last hidden layer assigned by DQN to game states experienced during a combination of human and agent play in Space Invaders.](/articles/nature14236/figures/5)

The plot was generated by running the t-SNE algorithm[25](/articles/nature14236#ref-CR25 "Van der Maaten, L. J. P. & Hinton, G. E. Visualizing high-dimensional data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008)") on the last hidden layer representation assigned by DQN to game states experienced during a combination of human (30 min) and agent (2 h) play. The fact that there is similar structure in the two-dimensional embeddings corresponding to the DQN representation of states experienced during human play (orange points) and DQN play (blue points) suggests that the representations learned by DQN do indeed generalize to data generated from policies other than its own. The presence in the t-SNE embedding of overlapping clusters of points corresponding to the network representation of states experienced during human and agent play shows that the DQN agent also follows sequences of states similar to those found in human play. Screenshots corresponding to selected states are shown (human: orange border; DQN: blue border).

### [Extended Data Figure 2 Visualization of learned value functions on two games, Breakout and Pong.](/articles/nature14236/figures/6)

**a**, A visualization of the learned value function on the game Breakout. At time points 1 and 2, the state value is predicted to be ∼17 and the agent is clearing the bricks at the lowest level. Each of the peaks in the value function curve corresponds to a reward obtained by clearing a brick. At time point 3, the agent is about to break through to the top level of bricks and the value increases to ∼21 in anticipation of breaking out and clearing a large set of bricks. At point 4, the value is above 23 and the agent has broken through. After this point, the ball will bounce at the upper part of the bricks clearing many of them by itself. **b**, A visualization of the learned action-value function on the game Pong. At time point 1, the ball is moving towards the paddle controlled by the agent on the right side of the screen and the values of all actions are around 0.7, reflecting the expected value of this state based on previous experience. At time point 2, the agent starts moving the paddle towards the ball and the value of the ‘up’ action stays high while the value of the ‘down’ action falls to −0.9. This reflects the fact that pressing ‘down’ would lead to the agent losing the ball and incurring a reward of −1. At time point 3, the agent hits the ball by pressing ‘up’ and the expected reward keeps increasing until time point 4, when the ball reaches the left edge of the screen and the value of all actions reflects that the agent is about to receive a reward of 1. Note, the dashed line shows the past trajectory of the ball purely for illustrative purposes (that is, not shown during the game). With permission from Atari Interactive, Inc.

**Extended Data Table 1 List of hyperparameters and their values**[Full size table](/articles/nature14236/tables/1)**Extended Data Table 2 Comparison of games scores obtained by DQN agents with methods from the literature[12](/articles/nature14236#ref-CR12 "Bellemare, M. G., Naddaf, Y., Veness, J. & Bowling, M. The arcade learning environment: An evaluation platform for general agents. J. Artif. Intell. Res. 47, 253–279 (2013)"),[15](/articles/nature14236#ref-CR15 "Bellemare, M. G., Veness, J. & Bowling, M. Investigating contingency awareness using Atari 2600 games. Proc. Conf. AAAI. Artif. Intell. 864–871 (2012)") and a professional human games tester**[Full size table](/articles/nature14236/tables/2)**Extended Data Table 3 The effects of replay and separating the target Q-network**[Full size table](/articles/nature14236/tables/3)**Extended Data Table 4 Comparison of DQN performance with linear function approximator**[Full size table](/articles/nature14236/tables/4)Supplementary information
-------------------------

### [Supplementary Information](https://static-content.springer.com/esm/art%3A10.1038%2Fnature14236/MediaObjects/41586_2015_BFnature14236_MOESM122_ESM.pdf)

This file contains a Supplementary Discussion. (PDF 110 kb)

### [Performance of DQN in the Game Space Invaders](https://static-content.springer.com/esm/art%3A10.1038%2Fnature14236/MediaObjects/41586_2015_BFnature14236_MOESM123_ESM.mov)

This video shows the performance of the DQN agent while playing the game of Space Invaders. The DQN agent successfully clears the enemy ships on the screen while the enemy ships move down and sideways with gradually increasing speed. (MOV 5106 kb)

### [Demonstration of Learning Progress in the Game Breakout](https://static-content.springer.com/esm/art%3A10.1038%2Fnature14236/MediaObjects/41586_2015_BFnature14236_MOESM124_ESM.mov)

This video shows the improvement in the performance of DQN over training (i.e. after 100, 200, 400 and 600 episodes). After 600 episodes DQN finds and exploits the optimal strategy in this game, which is to make a tunnel around the side, and then allow the ball to hit blocks by bouncing behind the wall. Note: the score is displayed at the top left of the screen (maximum for clearing one screen is 448 points), number of lives remaining is shown in the middle (starting with 5 lives), and the “1” on the top right indicates this is a 1-player game. (MOV 1500 kb)

PowerPoint slides
-----------------

### [PowerPoint slide for Fig. 1](https://static-content.springer.com/esm/art%3A10.1038%2Fnature14236/MediaObjects/41586_2015_BFnature14236_MOESM118_ESM.ppt)

### [PowerPoint slide for Fig. 2](https://static-content.springer.com/esm/art%3A10.1038%2Fnature14236/MediaObjects/41586_2015_BFnature14236_MOESM119_ESM.ppt)

### [PowerPoint slide for Fig. 3](https://static-content.springer.com/esm/art%3A10.1038%2Fnature14236/MediaObjects/41586_2015_BFnature14236_MOESM120_ESM.ppt)

### [PowerPoint slide for Fig. 4](https://static-content.springer.com/esm/art%3A10.1038%2Fnature14236/MediaObjects/41586_2015_BFnature14236_MOESM121_ESM.ppt)

Rights and permissions
----------------------

[Reprints and Permissions](https://s100.copyright.com/AppDispatchServlet?title=Human-level%20control%20through%20deep%20reinforcement%20learning&author=Volodymyr%20Mnih%20et%20al&contentID=10.1038%2Fnature14236&copyright=Springer%20Nature%20Limited&publication=0028-0836&publicationDate=2015-02-25&publisherName=SpringerNature&orderBeanReset=true)

This article is cited by
------------------------

* ###
[Task offloading optimization mechanism based on deep neural network in edge-cloud environment](https://doi.org/10.1186/s13677-023-00450-6)

	+ Lingkang Meng
	+ Yingjie Wang
	+ Zhipeng Cai*Journal of Cloud Computing* (2023)
* ###
[Governance and sustainability of distributed continuum systems: a big data approach](https://doi.org/10.1186/s40537-023-00737-0)

	+ Praveen Kumar Donta
	+ Boris Sedlak
	+ Schahram Dustdar*Journal of Big Data* (2023)
* ###
[Deep reinforcement learning for data-efficient weakly supervised business process anomaly detection](https://doi.org/10.1186/s40537-023-00708-5)

	+ Eman Abd Elaziz
	+ Radwa Fathalla
	+ Mohamed Shaheen*Journal of Big Data* (2023)
* ###
[A dosing strategy model of deep deterministic policy gradient algorithm for sepsis patients](https://doi.org/10.1186/s12911-023-02175-7)

	+ Tianlai Lin
	+ Xinjue Zhang
	+ Junhui Gao*BMC Medical Informatics and Decision Making* (2023)
* ###
[A deep reinforcement learning model for resilient road network recovery under earthquake or flooding hazards](https://doi.org/10.1186/s43065-023-00072-x)

	+ Xudong Fan
	+ Xijin Zhang
	+ Xiong Yu*Journal of Infrastructure Preservation and Resilience* (2023)

Comments
--------

By submitting a comment you agree to abide by our [Terms](/info/tandc.html) and [Community Guidelines](/info/community-guidelines.html). If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Volodymyr Mnih
Koray Kavukcuoglu
David Silver
Andrei A. Rusu
Joel Veness
Marc G. Bellemare
Alex Graves
Martin Riedmiller
Andreas K. Fidjeland
Georg Ostrovski
Stig Petersen
Charles Beattie
Amir Sadik
Ioannis Antonoglou
Helen King
Dharshan Kumaran
Daan Wierstra
Shane Legg
Demis Hassabis