Visualizing Data Using t-SNE

JournalofMachineLearning Research 9(2008)2579-2605 Submitted 5/08;Revised9/08;Published 11/08
Visualizing Datausingt-SNE
LaurensvanderMaaten LVDMAATEN@GMAIL.COM
TiCC
TilburgUniversity
P.O.Box90153,5000LETilburg,TheNetherlands
GeoffreyHinton HINTON@CS.TORONTO.EDU
Department ofComputer Science
UniversityofToronto
6King'sCollegeRoad,M5S3G4Toronto,ON,Canada
Editor: Yoshua Bengio
Abstract
Wepresent anewtechnique called “t-SNE” thatvisualizes high-dimensional data bygiving each
datapoint alocation inatwoorthree-dimensional map. Thetechnique isavariation ofStochastic
Neighbor Embedding (Hinton andRoweis, 2002) thatismuch easier tooptimize, andproduces
signiﬁcantly better visualizations byreducing thetendenc ytocrowdpoints together inthecenter
ofthemap. t-SNE isbetter than existing techniques atcreating asingle map thatrevealsstructure
atmanydifferent scales. This isparticularly important forhigh-dimensional data thatlieonseveral
different, butrelated, low-dimensional manifolds, such asimages ofobjects from multiple classes
seen from multiple viewpoints. Forvisualizing thestructure ofverylargedata sets, weshowhow
t-SNE canuserandom walks onneighborhood graphs toallowtheimplicit structure ofallofthe
datatoinﬂuence thewayinwhich asubset ofthedataisdisplayed. Weillustrate theperformance of
t-SNE onawide variety ofdata setsandcompare itwith manyother non-parametric visualization
techniques, including Sammon mapping, Isomap, andLocally Linear Embedding. Thevisualiza-
tions produced byt-SNE aresigniﬁcantly better than those produced bytheother techniques on
almost allofthedata sets.
Keywords: visualization, dimensionality reduction, manifold learning, embedding algorithms,
multidimensional scaling
1.Introduction
Visualization ofhigh-dimensional data isanimportant problem inmanydifferent domains, and
deals with data ofwidely varying dimensionality .Cell nuclei thatarerelevanttobreast cancer ,
forexample, aredescribed byapproximately 30variables (Street etal.,1993), whereas thepixel
intensity vectors used torepresent images ortheword-count vectors used torepresent documents
typically havethousands ofdimensions. Overthelastfewdecades, avariety oftechniques for
thevisualization ofsuch high-dimensional data havebeen proposed, manyofwhich arereviewed
bydeOliveiraandLevkowitz (2003). Important techniques include iconographic displays such as
Chernof ffaces (Chernof f,1973), pixel-based techniques (Keim, 2000), andtechniques thatrepre-
sentthedimensions inthedata asvertices inagraph (Battista etal.,1994). Most ofthese techniques
simply provide tools todisplay more than twodata dimensions, andleavetheinterpretation ofthe
c
2008LaurensvanderMaatenandGeoffreyHinton.
VANDERMAATENANDHINTON
data tothehuman observ er.This severely limits theapplicability ofthese techniques toreal-w orld
data setsthatcontain thousands ofhigh-dimensional datapoints.
Incontrast tothevisualization techniques discussed above,dimensionality reduction methods
convertthehigh-dimensional data setX=fx1;x2;:::;xngintotwoorthree-dimensional data Y=
fy1;y2;:::;yngthatcanbedisplayed inascatterplot. Inthepaper ,werefer tothelow-dimensional
data representation Yasamap, andtothelow-dimensional representations yiofindividual da-
tapoints asmap points. The aimofdimensionality reduction istopreserv easmuch ofthesig-
niﬁcant structure ofthehigh-dimensional data aspossible inthelow-dimensional map. Various
techniques forthisproblem havebeen proposed thatdifferinthetype ofstructure theypreserv e.
Traditional dimensionality reduction techniques such asPrincipal Components Analysis (PCA;
Hotelling, 1933) andclassical multidimensional scaling (MDS; Torgerson, 1952) arelinear tech-
niques thatfocus onkeeping thelow-dimensional representations ofdissimilar datapoints farapart.
Forhigh-dimensional data thatliesonornear alow-dimensional, non-linear manifold itisusu-
allymore important tokeepthelow-dimensional representations ofverysimilar datapoints close
together ,which istypically notpossible with alinear mapping.
Alargenumber ofnonlinear dimensionality reduction techniques thataimtopreserv ethelocal
structure ofdata havebeen proposed, manyofwhich arereviewed byLeeandVerleysen (2007).
Inparticular ,wemention thefollowing seventechniques: (1)Sammon mapping (Sammon, 1969),
(2)curvilinear components analysis (CCA; Demartines andH´erault, 1997), (3)Stochastic Neighbor
Embedding (SNE; Hinton andRoweis, 2002), (4)Isomap (Tenenbaum etal.,2000), (5)Maximum
Variance Unfolding (MVU; Weinber geretal.,2004), (6)Locally Linear Embedding (LLE; Roweis
andSaul, 2000), and(7)Laplacian Eigenmaps (Belkin andNiyogi, 2002). Despite thestrong per-
formance ofthese techniques onartiﬁcial data sets, theyareoften notverysuccessful atvisualizing
real, high-dimensional data. Inparticular ,most ofthetechniques arenotcapable ofretaining both
thelocal andtheglobal structure ofthedata inasingle map. Forinstance, arecent study reveals
thatevenasemi-supervised variant ofMVU isnotcapable ofseparating handwritten digits into
their natural clusters (Song etal.,2007).
Inthispaper ,wedescribe awayofconverting ahigh-dimensional data setintoamatrix ofpair-
wise similarities andweintroduce anewtechnique, called “t-SNE”, forvisualizing theresulting
similarity data. t-SNE iscapable ofcapturing much ofthelocal structure ofthehigh-dimensional
dataverywell, while alsorevealing global structure such asthepresence ofclusters atseveralscales.
Weillustrate theperformance oft-SNE bycomparing ittothesevendimensionality reduction tech-
niques mentioned aboveonﬁvedata setsfrom avariety ofdomains. Because ofspace limitations,
most ofthe(7+1)5=40maps arepresented inthesupplemental material, butthemaps thatwe
present inthepaper aresufﬁcient todemonstrate thesuperiority oft-SNE.
Theoutline ofthepaper isasfollows.InSection 2,weoutline SNE aspresented byHinton and
Roweis (2002), which forms thebasis fort-SNE. InSection 3,wepresent t-SNE, which hastwo
important differences from SNE. InSection 4,wedescribe theexperimental setup andtheresults
ofourexperiments. Subsequently ,Section 5showshowt-SNE canbemodiﬁed tovisualize real-
worlddata setsthatcontain manymore than 10;000datapoints. Theresults ofourexperiments are
discussed inmore detail inSection 6.Ourconclusions andsuggestions forfuture workarepresented
inSection 7.
2580
VISUALIZINGDATAUSINGT-SNE
2.Stochastic Neighbor Embedding
Stochastic Neighbor Embedding (SNE) starts byconverting thehigh-dimensional Euclidean dis-
tances between datapoints intoconditional probabilities thatrepresent similarities.1Thesimilarity
ofdatapoint xjtodatapoint xiistheconditional probability ,pjji,thatxiwould pick xjasitsneighbor
ifneighbors were pickedinproportion totheir probability density under aGaussian centered atxi.
Fornearby datapoints, pjjiisrelati velyhigh, whereas forwidely separated datapoints, pjjiwillbe
almost inﬁnitesimal (forreasonable values ofthevariance oftheGaussian, σi).Mathematically ,the
conditional probability pjjiisgivenby
pjji=exp
kxixjk2=2σ2
i
∑k6=iexp
kxixkk2=2σ2
i; (1)
whereσiisthevariance oftheGaussian thatiscentered ondatapoint xi.Themethod fordetermining
thevalueofσiispresented later inthissection. Because weareonly interested inmodeling pairwise
similarities, wesetthevalue ofpijitozero. Forthelow-dimensional counterparts yiandyjofthe
high-dimensional datapoints xiandxj,itispossible tocompute asimilar conditional probability ,
which wedenote byqjji.Weset2thevariance oftheGaussian thatisemplo yedinthecomputation
oftheconditional probabilities qjjito1p
2.Hence, wemodel thesimilarity ofmap point yjtomap
point yiby
qjji=exp
kyiyjk2
∑k6=iexp(kyiykk2):
Again,since weareonly interested inmodeling pairwise similarities, wesetqiji=0.
Ifthemap points yiandyjcorrectly model thesimilarity between thehigh-dimensional data-
points xiandxj,theconditional probabilities pjjiandqjjiwillbeequal. Moti vated bythisobserv a-
tion, SNE aims toﬁndalow-dimensional data representation thatminimizes themismatch between
pjjiandqjji.Anatural measure ofthefaithfulness with which qjjimodels pjjiistheKullback-
Leibler divergence (which isinthiscase equal tothecross-entrop yuptoanadditi veconstant). SNE
minimizes thesum ofKullback-Leibler divergences overalldatapoints using agradient descent
method. Thecostfunction Cisgivenby
C=∑
iKL(PijjQi)=∑
i∑
jpjjilogpjji
qjji; (2)
inwhich Pirepresents theconditional probability distrib ution overallother datapoints givendata-
point xi,andQirepresents theconditional probability distrib ution overallother map points given
map point yi.Because theKullback-Leibler divergence isnotsymmetric, different types oferror
inthepairwise distances inthelow-dimensional map arenotweighted equally .Inparticular ,there
isalargecost forusing widely separated map points torepresent nearby datapoints (i.e., forusing
1.SNE canalsobeapplied todata setsthatconsist ofpairwise similarities between objects rather than high-dimensional
vector representations ofeach object, provided these simiarities canbeinterpreted asconditional probabilities. For
example, human wordassociation data consists oftheprobability ofproducing each possible wordinresponse toa
givenword,asaresult ofwhich itisalready intheform required bySNE.
2.Setting thevariance inthelow-dimensional Gaussians toanother value only results inarescaled version oftheﬁnal
map. Note thatbyusing thesame variance foreverydatapoint inthelow-dimensional map, welose theproperty
thatthedata isaperfect model ofitself ifweembed itinaspace ofthesame dimensionality ,because inthehigh-
dimensional space, weused adifferent variance σiineach Gaussian.
2581
VANDERMAATENANDHINTON
asmall qjjitomodel alargepjji),butthere isonly asmall cost forusing nearby map points to
represent widely separated datapoints. This small costcomes from wasting some oftheprobability
mass intherelevantQdistrib utions. Inother words, theSNE costfunction focuses onretaining the
local structure ofthedata inthemap (forreasonable values ofthevariance oftheGaussian inthe
high-dimensional space,σi).
Theremaining parameter tobeselected isthevariance σioftheGaussian thatiscentered over
each high-dimensional datapoint, xi.Itisnotlikelythatthere isasingle value ofσithatisoptimal
foralldatapoints inthedata setbecause thedensity ofthedata islikelytovary.Indense regions,
asmaller value ofσiisusually more appropriate than insparser regions. Anyparticular value of
σiinduces aprobability distrib ution, Pi,overalloftheother datapoints. This distrib ution hasan
entrop ywhich increases asσiincreases. SNE performs abinary search forthevalue ofσithat
produces aPiwith aﬁxedperple xitythatisspeciﬁed bytheuser.3Theperple xityisdeﬁned as
Perp(Pi)=2H(Pi);
where H(Pi)istheShannon entrop yofPimeasured inbits
H(Pi)=∑
jpjjilog2pjji:
Theperple xitycanbeinterpreted asasmooth measure oftheeffectivenumber ofneighbors. The
performance ofSNE isfairly robusttochanges intheperple xity,andtypical values arebetween 5
and50.
The minimization ofthecost function inEquation 2isperformed using agradient descent
method. Thegradient hasasurprisingly simple form
δC
δyi=2∑
j(pjjiqjji+pijjqijj)(yiyj):
Physically ,thegradient may beinterpreted astheresultant force created byasetofsprings between
themap point yiandallother map points yj.Allsprings exertaforce along thedirection (yiyj).
Thespring between yiandyjrepels orattracts themap points depending onwhether thedistance
between thetwointhemap istoosmall ortoolargetorepresent thesimilarities between thetwo
high-dimensional datapoints. Theforce exerted bythespring between yiandyjisproportional toits
length, andalsoproportional toitsstiffness, which isthemismatch (pjjiqjji+pijjqijj)between
thepairwise similarities ofthedata points andthemap points.
Thegradient descent isinitialized bysampling map points randomly from anisotropic Gaussian
with small variance thatiscentered around theorigin. Inorder tospeed uptheoptimization andto
avoidpoor local minima, arelati velylargemomentum term isadded tothegradient. Inother words,
thecurrent gradient isadded toanexponentially decaying sum ofprevious gradients inorder to
determine thechanges inthecoordinates ofthemap points ateach iteration ofthegradient search.
Mathematically ,thegradient update with amomentum term isgivenby
Y(t)=Y(t1)+ηδC
δY+α(t)
Y(t1)Y(t2)
;
3.Note thattheperple xityincreases monotonically with thevariance σi.
2582
VISUALIZINGDATAUSINGT-SNE
where Y(t)indicates thesolution atiteration t,ηindicates thelearning rate, andα(t)represents the
momentum atiteration t.
Inaddition, intheearly stages oftheoptimization, Gaussian noise isadded tothemap points
after each iteration. Gradually reducing thevariance ofthisnoise performs atype ofsimulated
annealing thathelps theoptimization toescape from poor local minima inthecost function. Ifthe
variance ofthenoise changes veryslowlyatthecritical point atwhich theglobal structure ofthe
map starts toform, SNE tends toﬁndmaps with abetter global organization. Unfortunately ,this
requires sensible choices oftheinitial amount ofGaussian noise andtherateatwhich itdecays.
Moreo ver,these choices interact with theamount ofmomentum andthestepsizethatareemplo yed
inthegradient descent. Itistherefore common toruntheoptimization severaltimes onadata set
toﬁndappropriate values fortheparameters.4Inthisrespect, SNE isinferior tomethods thatallow
convexoptimization anditwould beuseful toﬁndanoptimization method thatgivesgood results
without requiring theextra computation time andparameter choices introduced bythesimulated
annealing.
3.t-DistributedStochastic Neighbor Embedding
Section 2discussed SNE asitwaspresented byHinton andRoweis (2002). Although SNE con-
structs reasonably good visualizations, itishampered byacostfunction thatisdifﬁcult tooptimize
andbyaproblem werefer toasthe“crowding problem”. Inthissection, wepresent anewtechnique
called “t-Distrib uted Stochastic Neighbor Embedding” or“t-SNE” thataims toalleviate these prob-
lems. Thecostfunction used byt-SNE differsfrom theoneused bySNE intwoways: (1)ituses a
symmetrized version oftheSNE costfunction with simpler gradients thatwasbrieﬂy introduced by
Cook etal.(2007) and(2)ituses aStudent-t distrib ution rather than aGaussian tocompute thesim-
ilarity between twopoints inthelow-dimensional space .t-SNE emplo ysaheavy-tailed distrib ution
inthelow-dimensional space toalleviate both thecrowding problem andtheoptimization problems
ofSNE.
Inthissection, weﬁrstdiscuss thesymmetric version ofSNE (Section 3.1). Subsequently ,we
discuss thecrowding problem (Section 3.2), andtheuseofheavy-tailed distrib utions toaddress this
problem (Section 3.3). Weconclude thesection bydescribing ourapproach totheoptimization of
thet-SNE costfunction (Section 3.4).
3.1Symmetric SNE
Asanalternati vetominimizing thesum oftheKullback-Leibler divergences between thecondi-
tional probabilities pjjiandqjji,itisalsopossible tominimize asingle Kullback-Leibler divergence
between ajoint probability distrib ution, P,inthehigh-dimensional space andajoint probability
distrib ution, Q,inthelow-dimensional space:
C=KL(PjjQ)=∑
i∑
jpijlogpij
qij:
where again,wesetpiiandqiitozero. Werefer tothistype ofSNE assymmetric SNE, because it
hastheproperty thatpij=pjiandqij=qjifor8i;j.Insymmetric SNE, thepairwise similarities in
4.Picking thebestmap after severalruns asavisualization ofthedata isnotnearly asproblematic aspicking themodel
thatdoes best onatestsetduring supervised learning. Invisualization, theaimistoseethestructure inthetraining
data, nottogeneralize toheld outtestdata.
2583
VANDERMAATENANDHINTON
thelow-dimensional map qijaregivenby
qij=exp
kyiyjk2
∑k6=lexp(kykylk2); (3)
Theobvious waytodeﬁne thepairwise similarities inthehigh-dimensional space pijis
pij=exp
kxixjk2=2σ2
∑k6=lexp(kxkxlk2=2σ2);
butthiscauses problems when ahigh-dimensional datapoint xiisanoutlier (i.e., allpairwise dis-
tances kxixjk2arelargeforxi).Forsuch anoutlier ,thevalues ofpijareextremely small for
allj,sothelocation ofitslow-dimensional map point yihasverylittle effectonthecost function.
Asaresult, theposition ofthemap point isnotwell determined bythepositions oftheother map
points. Wecircumv entthisproblem bydeﬁning thejoint probabilities pijinthehigh-dimensional
space tobethesymmetrized conditional probabilities, thatis,wesetpij=pjji+pijj
2n.This ensures that
∑jpij>1
2nforalldatapoints xi,asaresult ofwhich each datapoint ximakesasigniﬁcant contri-
bution tothecostfunction. Inthelow-dimensional space, symmetric SNE simply uses Equation 3.
Themain advantage ofthesymmetric version ofSNE isthesimpler form ofitsgradient, which is
faster tocompute. Thegradient ofsymmetric SNE isfairly similar tothatofasymmetric SNE, and
isgivenby
δC
δyi=4∑
j(pijqij)(yiyj):
Inpreliminary experiments, weobserv edthatsymmetric SNE seems toproduce maps thatarejust
asgood asasymmetric SNE, andsometimes evenalittle better .
3.2TheCrowdingProblem
Consider asetofdatapoints thatlieonatwo-dimensional curvedmanifold which isapproximately
linear onasmall scale, andwhich isembedded within ahigher -dimensional space. Itispossible to
model thesmall pairwise distances between datapoints fairly well inatwo-dimensional map, which
isoften illustrated ontoyexamples such asthe“Swiss roll” data set.Nowsuppose thatthemani-
fold hastenintrinsic dimensions5andisembedded within aspace ofmuch higher dimensionality .
There areseveralreasons whythepairwise distances inatwo-dimensional map cannot faithfully
model distances between points ontheten-dimensional manifold. Forinstance, intendimensions,
itispossible tohave11datapoints thataremutually equidistant andthere isnowaytomodel this
faithfully inatwo-dimensional map. Arelated problem istheverydifferent distrib ution ofpairwise
distances inthetwospaces. Thevolume ofasphere centered ondatapoint iscales asrm,where ris
theradius andmthedimensionality ofthesphere. Soifthedatapoints areapproximately uniformly
distrib uted intheregion around iontheten-dimensional manifold, andwetrytomodel thedis-
tances from itotheother datapoints inthetwo-dimensional map, wegetthefollowing “crowding
problem”: thearea ofthetwo-dimensional map thatisavailable toaccommodate moderately distant
datapoints willnotbenearly largeenough compared with thearea available toaccommodate nearby
datapoints. Hence, ifwewanttomodel thesmall distances accurately inthemap, most ofthepoints
5.This isapproximately correct fortheimages ofhandwritten digits weuseinourexperiments inSection 4.
2584
VISUALIZINGDATAUSINGT-SNE
thatareatamoderate distance from datapoint iwill havetobeplaced much toofarawayinthe
two-dimensional map. InSNE, thespring connecting datapoint itoeach ofthese too-distant map
points willthus exertaverysmall attracti veforce. Although these attracti veforces areverysmall,
theverylargenumber ofsuch forces crushes together thepoints inthecenter ofthemap, which
prevents gapsfrom forming between thenatural clusters. Note thatthecrowding problem isnot
speciﬁc toSNE, butthatitalso occurs inother local techniques formultidimensional scaling such
asSammon mapping.
Anattempt toaddress thecrowding problem byadding aslight repulsion toallsprings waspre-
sented byCook etal.(2007). Theslight repulsion iscreated byintroducing auniform background
model with asmall mixing proportion, ρ.Sohoweverfarapart twomap points are,qijcanneverfall
below2ρ
n(n1)(because theuniform background distrib ution isovern(n1)=2pairs). Asaresult, for
datapoints thatarefarapart inthehigh-dimensional space, qijwillalwaysbelargerthan pij,leading
toaslight repulsion. This technique iscalled UNI-SNE andalthough itusually outperforms stan-
dard SNE, theoptimization oftheUNI-SNE costfunction istedious. Thebestoptimization method
knownistostart bysetting thebackground mixing proportion tozero (i.e., byperforming standard
SNE). Once theSNE cost function hasbeen optimized using simulated annealing, thebackground
mixing proportion canbeincreased toallowsome gapstoform between natural clusters asshown
byCook etal.(2007). Optimizing theUNI-SNE cost function directly does notworkbecause two
map points thatarefarapart willgetalmost alloftheir qijfrom theuniform background. Soeven
iftheir pijislarge,there willbenoattracti veforce between them, because asmall change intheir
separation willhaveavanishingly small proportional effectonqij.This means thatiftwoparts of
acluster getseparated early onintheoptimization, there isnoforce topullthem back together .
3.3Mismatched TailscanCompensate forMismatched Dimensionalities
Since symmetric SNE isactually matching thejoint probabilities ofpairs ofdatapoints inthehigh-
dimensional andthelow-dimensional spaces rather than their distances, wehaveanatural way
ofalleviating thecrowding problem that works asfollows. Inthehigh-dimensional space, we
convertdistances intoprobabilities using aGaussian distrib ution. Inthelow-dimensional map, we
canuseaprobability distrib ution thathasmuch heaviertails than aGaussian toconvertdistances
intoprobabilities. This allowsamoderate distance inthehigh-dimensional space tobefaithfully
modeled byamuch largerdistance inthemap and, asaresult, iteliminates theunwanted attracti ve
forces between map points thatrepresent moderately dissimilar datapoints.
Int-SNE, weemplo yaStudent t-distrib ution with onedegree offreedom (which isthesame
asaCauch ydistrib ution) astheheavy-tailed distrib ution inthelow-dimensional map. Using this
distrib ution, thejoint probabilities qijaredeﬁned as
qij=
1+kyiyjk21
∑k6=l(1+kykylk2)1: (4)
WeuseaStudent t-distrib ution with asingle degree offreedom, because ithastheparticularly
nice property that
1+kyiyjk21approaches aninversesquare lawforlargepairwise distances
kyiyjkinthelow-dimensional map. This makesthemap’ srepresentation ofjoint probabilities
(almost) invariant tochanges inthescale ofthemap formap points thatarefarapart. Italsomeans
thatlargeclusters ofpoints thatarefarapart interact injustthesame wayasindividual points, sothe
optimization operates inthesame wayatallbuttheﬁnest scales. Atheoretical justiﬁcation forour
2585
VANDERMAATENANDHINTON
High−dimensional distance >Low−dimensional distance >

024681012141618
(a)Gradient ofSNE.
High−dimensional distance >Low−dimensional distance >

−4−202468101214
(b)Gradient ofUNI-SNE.
High−dimensional distance >Low−dimensional distance >

−1−0.500.51
(c)Gradient oft-SNE.
Figure 1:Gradients ofthree types ofSNE asafunction ofthepairwise Euclidean distance between
twopoints inthehigh-dimensional andthepairwise distance between thepoints inthe
low-dimensional data representation.
selection oftheStudent t-distrib ution isthatitisclosely related totheGaussian distrib ution, asthe
Student t-distrib ution isaninﬁnite mixture ofGaussians. Acomputationally convenient property
isthatitismuch faster toevaluate thedensity ofapoint under aStudent t-distrib ution than under
aGaussian because itdoes notinvolveanexponential, eventhough theStudent t-distrib ution is
equivalent toaninﬁnite mixture ofGaussians with different variances.
Thegradient oftheKullback-Leibler divergence between PandtheStudent-t based joint prob-
ability distrib ution Q(computed using Equation 4)isderivedinAppendix A,andisgivenby
δC
δyi=4∑
j(pijqij)(yiyj)
1+kyiyjk21: (5)
InFigure 1(a) to1(c), weshowthegradients between twolow-dimensional datapoints yiandyjas
afunction oftheir pairwise Euclidean distances inthehigh-dimensional andthelow-dimensional
space (i.e., asafunction ofkxixjkandkyiyjk)forthesymmetric versions ofSNE, UNI-SNE,
andt-SNE. Intheﬁgures, positi vevalues ofthegradient represent anattraction between thelow-
dimensional datapoints yiandyj,whereas negativevalues represent arepulsion between thetwo
datapoints. From theﬁgures, weobserv etwomain advantages ofthet-SNE gradient overthe
gradients ofSNE andUNI-SNE.
First, thet-SNE gradient strongly repels dissimilar datapoints thataremodeled byasmall pair-
wise distance inthelow-dimensional representation. SNE hassuch arepulsion aswell, butitseffect
isminimal compared tothestrong attractions elsewhere inthegradient (thelargestattraction inour
graphical representation ofthegradient isapproximately 19,whereas thelargestrepulsion isapprox-
imately 1).InUNI-SNE, theamount ofrepulsion between dissimilar datapoints isslightly larger,
however,thisrepulsion isonly strong when thepairwise distance between thepoints inthelow-
dimensional representation isalready large(which isoften notthecase, since thelow-dimensional
representation isinitialized bysampling from aGaussian with averysmall variance thatiscentered
around theorigin).
Second, although t-SNE introduces strong repulsions between dissimilar datapoints that are
modeled bysmall pairwise distances, these repulsions donotgotoinﬁnity .Inthisrespect, t-SNE
differsfrom UNI-SNE, inwhich thestrength oftherepulsion between verydissimilar datapoints
2586
VISUALIZINGDATAUSINGT-SNE
Algorithm 1:Simple version oft-Distrib uted Stochastic Neighbor Embedding.
Data :data setX=fx1;x2;:::;xng,
costfunction parameters: perple xityPerp,
optimization parameters: number ofiterations T,learning rateη,momentum α(t).
Result :low-dimensional data representation Y(T)=fy1;y2;:::;yng.
begin
compute pairwise afﬁnities pjjiwith perple xityPerp(using Equation 1)
setpij=pjji+pijj
2n
sample initial solution Y(0)=fy1;y2;:::;yngfrom N(0;104I)
fort=1toTdo
compute low-dimensional afﬁnities qij(using Equation 4)
compute gradientδC
δY(using Equation 5)
setY(t)=Y(t1)+ηδC
δY+α(t)Y(t1)Y(t2)
end
end
isproportional totheir pairwise distance inthelow-dimensional map, which may cause dissimilar
datapoints tomovemuch toofarawayfrom each other .
Takentogether ,t-SNE puts emphasis on(1)modeling dissimilar datapoints bymeans oflarge
pairwise distances, and(2)modeling similar datapoints bymeans ofsmall pairwise distances. More-
over,asaresult ofthese characteristics ofthet-SNE costfunction (and asaresult oftheapproximate
scale invariance oftheStudent t-distrib ution), theoptimization ofthet-SNE cost function ismuch
easier than theoptimization ofthecost functions ofSNE andUNI-SNE. Speciﬁcally ,t-SNE in-
troduces long-range forces inthelow-dimensional map thatcanpull back together two(clusters
of)similar points thatgetseparated early onintheoptimization. SNE andUNI-SNE donothave
such long-range forces, asaresult ofwhich SNE andUNI-SNE need tousesimulated annealing to
obtain reasonable solutions. Instead, thelong-range forces int-SNE facilitate theidentiﬁcation of
good local optima without resorting tosimulated annealing.
3.4Optimization Methodsfort-SNE
Westart bypresenting arelati velysimple, gradient descent procedure foroptimizing thet-SNE cost
function. This simple procedure uses amomentum term toreduce thenumber ofiterations required
anditworks bestifthemomentum term issmall until themap points havebecome moderately well
organized. Pseudocode forthissimple algorithm ispresented inAlgorithm 1.Thesimple algorithm
canbesped upusing theadapti velearning ratescheme thatisdescribed byJacobs (1988), which
gradually increases thelearning rateindirections inwhich thegradient isstable.
Although thesimple algorithm produces visualizations thatareoften much better than those
produced byother non-parametric dimensionality reduction techniques, theresults canbeimpro ved
further byusing either oftwotricks. Theﬁrsttrick, which wecall“early compression”, istoforce
themap points tostay close together atthestart oftheoptimization. When thedistances between
map points aresmall, itiseasy forclusters tomovethrough oneanother soitismuch easier to
explore thespace ofpossible global organizations ofthedata. Early compression isimplemented
byadding anadditional L2-penalty tothecost function thatisproportional tothesum ofsquared
2587
VANDERMAATENANDHINTON
distances ofthemap points from theorigin. Themagnitude ofthispenalty term andtheiteration at
which itisremo vedaresetbyhand, butthebehavior isfairly robustacross variations inthese two
additional optimization parameters.
Aless obvious waytoimpro vetheoptimization, which wecall“early exaggeration”, isto
multiply allofthepij’sby,forexample, 4,intheinitial stages oftheoptimization. This means that
almost alloftheqij’s,which stilladdupto1,aremuch toosmall tomodel their corresponding pij’s.
Asaresult, theoptimization isencouraged tofocus onmodeling thelargepij’sbyfairly largeqij’s.
Theeffectisthatthenatural clusters inthedata tend toform tight widely separated clusters inthe
map. This creates alotofrelati velyempty space inthemap, which makesitmuch easier forthe
clusters tomovearound relati vetooneanother inorder toﬁndagood global organization.
Inallthevisualizations presented inthispaper andinthesupporting material, weused exactly
thesame optimization procedure. Weused theearly exaggeration method with anexaggeration
of4fortheﬁrst 50iterations (note thatearly exaggeration isnotincluded inthepseudocode in
Algorithm 1).Thenumber ofgradient descent iterations Twasset1000, andthemomentum term
wassettoα(t)=0:5fort<250andα(t)=0:8fort250. Thelearning rateηisinitially setto100
anditisupdated after everyiteration bymeans oftheadapti velearning ratescheme described by
Jacobs (1988). AMatlab implementation oftheresulting algorithm isavailable athttp://ticc.
uvt.nl/˜lvdrmaaten/tsne .
4.Experiments
Toevaluate t-SNE, wepresent experiments inwhich t-SNE iscompared tosevenother non-parametric
techniques fordimensionality reduction. Because ofspace limitations, inthepaper ,weonly com-
pare t-SNE with: (1)Sammon mapping, (2)Isomap, and(3)LLE. Inthesupporting material, we
also compare t-SNE with: (4)CCA, (5)SNE, (6)MVU, and(7)Laplacian Eigenmaps. Weper-
formed experiments onﬁvedata setsthatrepresent avariety ofapplication domains. Againbecause
ofspace limitations, werestrict ourselv estothree data setsinthepaper .Theresults ofourexperi-
ments ontheremaining twodata setsarepresented inthesupplemental material.
InSection 4.1,thedata setsthatweemplo yedinourexperiments areintroduced. Thesetup of
theexperiments ispresented inSection 4.2.InSection 4.3,wepresent theresults ofourexperiments.
4.1DataSets
The ﬁvedata setsweemplo yedinourexperiments are: (1)theMNIST data set,(2)theOlivetti
faces data set,(3)theCOIL-20 data set,(4)theword-features data set,and(5)theNetﬂix data set.
Weonly present results ontheﬁrstthree data setsinthissection. Theresults ontheremaining two
data setsarepresented inthesupporting material. Theﬁrstthree data setsareintroduced below.
TheMNIST data set6contains 60,000 grayscale images ofhandwritten digits. Forourexperi-
ments, werandomly selected 6,000 oftheimages forcomputational reasons. Thedigit images have
2828=784pixels(i.e., dimensions). TheOlivettifaces data set7consists ofimages of40indi-
viduals with small variations inviewpoint, largevariations inexpression, andoccasional addition
ofglasses. Thedata setconsists of400images (10perindividual) ofsize92112=10;304pixels,
andislabeled according toidentity .TheCOIL-20 data set(Nene etal.,1996) contains images of20
6.TheMNIST data setispublicly available fromhttp://yann.lecun.com/exdb/mnist/index.html .
7.TheOlivettifaces data setispublicly available fromhttp://mambo.ucsc.edu/psl/olivetti.html .
2588
VISUALIZINGDATAUSINGT-SNE
different objects viewed from 72equally spaced orientations, yielding atotal of1,440 images. The
images contain 3232=1;024pixels.
4.2Experimental Setup
Inallofourexperiments, westart byusing PCA toreduce thedimensionality ofthedata to30.
This speeds upthecomputation ofpairwise distances between thedatapoints andsuppresses some
noise without severely distorting theinterpoint distances. Wethen useeach ofthedimensionality
reduction techniques toconvertthe30-dimensional representation toatwo-dimensional map and
weshowtheresulting map asascatterplot. Forallofthedata sets, there isinformation about the
class ofeach datapoint, buttheclass information isonly used toselect acolor and/or symbol for
themap points. Theclass information isnotused todetermine thespatial coordinates ofthemap
points. Thecoloring thus provides awayofevaluating howwell themap preserv esthesimilarities
within each class.
The cost function parameter settings weemplo yedinourexperiments arelisted inTable 1.
Inthetable, Perprepresents theperple xity oftheconditional probability distrib ution induced by
aGaussian kernel andkrepresents thenumber ofnearest neighbors emplo yedinaneighborhood
graph. Intheexperiments with Isomap andLLE, weonly visualize datapoints thatcorrespond to
vertices inthelargestconnected component oftheneighborhood graph.8FortheSammon mapping
optimization, weperformed Newton’ smethod for500iterations.
Technique Cost function parameter s
t-SNE Perp=40
Sammon mapping none
Isomap k=12
LLE k=12
Table 1:Cost function parameter settings fortheexperiments.
4.3Results
InFigures 2and3,weshowtheresults ofourexperiments with t-SNE, Sammon mapping, Isomap,
andLLE ontheMNIST data set.Theresults revealthestrong performance oft-SNE compared to
theother techniques. Inparticular ,Sammon mapping constructs a“ball” inwhich only three classes
(representing thedigits 0,1,and7)aresome what separated from theother classes. Isomap and
LLE produce solutions inwhich there arelargeoverlaps between thedigit classes. Incontrast, t-
SNE constructs amap inwhich theseparation between thedigit classes isalmost perfect. Moreo ver,
detailed inspection ofthet-SNE map revealsthatmuch ofthelocal structure ofthedata (such as
theorientation oftheones) iscaptured aswell. This isillustrated inmore detail inSection 5(see
Figure 7).The map produced byt-SNE contains some points thatareclustered with thewrong
class, butmost ofthese points correspond todistorted digits manyofwhich aredifﬁcult toidentify .
Figure 4showstheresults ofapplying t-SNE, Sammon mapping, Isomap, andLLE totheOlivetti
faces data set.Again,Isomap andLLE produce solutions thatprovide little insight intotheclass
8.Isomap andLLE require data thatgivesrisetoaneighborhood graph thatisconnected.
2589
VANDERMAATENANDHINTON
  0
1
2
3
4
5
6
7
8
9
(a)Visualization byt-SNE.

(b)Visualization bySammon mapping.
Figure 2:Visualizations of6,000 handwritten digits from theMNIST data set.
2590
VISUALIZINGDATAUSINGT-SNE

(a)Visualization byIsomap.

(b)Visualization byLLE.
Figure 3:Visualizations of6,000 handwritten digits from theMNIST data set.
2591
VANDERMAATENANDHINTON

(a)Visualization byt-SNE.
(b)Visualization bySammon mapping.

(c)Visualization byIsomap.
(d)Visualization byLLE.
Figure 4:Visualizations oftheOlivettifaces data set.
structure ofthedata. The map constructed bySammon mapping issigniﬁcantly better ,since it
models manyofthemembers ofeach class fairly close together ,butnone oftheclasses areclearly
separated intheSammon map. Incontrast, t-SNE does amuch better jobofrevealing thenatural
classes inthedata. Some individuals havetheir tenimages split intotwoclusters, usually because a
subset oftheimages havethehead facing inasigniﬁcantly different direction, orbecause theyhave
averydifferent expression orglasses. Forthese individuals, itisnotclear thattheir tenimages form
anatural class when using Euclidean distance inpixelspace.
Figure 5showstheresults ofapplying t-SNE, Sammon mapping, Isomap, andLLE totheCOIL-
20data set.Formanyofthe20objects, t-SNE accurately represents theone-dimensional manifold
ofviewpoints asaclosed loop. Forobjects which look similar from thefront andtheback, t-SNE
distorts theloop sothattheimages offront andback aremapped tonearby points. Forthefour
types oftoycarintheCOIL-20 data set(the four aligned “sausages” inthebottom-left ofthet-
SNE map), thefour rotation manifolds arealigned bytheorientation ofthecarstocapture thehigh
2592
VISUALIZINGDATAUSINGT-SNE

(a)Visualization byt-SNE.
(b)Visualization bySammon mapping.

(c)Visualization byIsomap.
(d)Visualization byLLE.
Figure 5:Visualizations oftheCOIL-20 data set.
similarity between different cars atthesame orientation. This prevents t-SNE from keeping the
four manifolds clearly separate. Figure 5also revealsthattheother three techniques arenotnearly
asgood atcleanly separating themanifolds thatcorrespond toverydifferent objects. Inaddition,
Isomap andLLE only visualize asmall number ofclasses from theCOIL-20 data set,because the
datasetcomprises alargenumber ofwidely separated submanifolds thatgiverisetosmall connected
components intheneighborhood graph.
5.Applyingt-SNEtoLargeDataSets
Likemanyother visualization techniques, t-SNE hasacomputational andmemory comple xitythat
isquadratic inthenumber ofdatapoints. This makesitinfeasible toapply thestandard version of
t-SNE todata setsthatcontain manymore than, say,10,000 points. Obviously ,itispossible to
pick arandom subset ofthedatapoints anddisplay them using t-SNE, butsuch anapproach failsto
2593
VANDERMAATENANDHINTON
Figure 6:Anillustration oftheadvantage oftherandom walkversion oft-SNE overastandard
landmark approach. Theshaded points A,B,andCarethree (almost) equidistant land-
mark points, whereas thenon-shaded datapoints arenon-landmark points. The arrows
represent adirected neighborhood graph where k=3.Inastandard landmark approach,
thepairwise afﬁnity between AandBisapproximately equal tothepairwise afﬁnity be-
tween AandC.Intherandom walkversion oft-SNE, thepairwise afﬁnity between A
andBismuch largerthan thepairwise afﬁnity between AandC,andtherefore, itreﬂects
thestructure ofthedata much better .
makeuseoftheinformation thattheundisplayed datapoints provide about theunderlying manifolds.
Suppose, forexample, thatA,B,andCareallequidistant inthehigh-dimensional space. Ifthere
aremanyundisplayed datapoints between AandBandnone between AandC,itismuch more
likelythatAandBarepartofthesame cluster than AandC.This isillustrated inFigure 6.Inthis
section, weshowhowt-SNE canbemodiﬁed todisplay arandom subset ofthedatapoints (so-called
landmark points) inawaythatuses information from theentire (possibly verylarge)data set.
Westart bychoosing adesired number ofneighbors andcreating aneighborhood graph forall
ofthedatapoints. Although thisiscomputationally intensi ve,itisonly done once. Then, foreach
ofthelandmark points, wedeﬁne arandom walkstarting atthatlandmark point andterminating
assoon asitlands onanother landmark point. During arandom walk,theprobability ofchoosing
anedge emanating from node xitonode xjisproportional toekxixjk2.Wedeﬁne pjjitobethe
fraction ofrandom walksstarting atlandmark point xithatterminate atlandmark point xj.This has
some resemblance tothewayIsomap measures pairwise distances between points. However,asin
diffusion maps (Lafon andLee, 2006; Nadler etal.,2006), rather than looking fortheshortest path
through theneighborhood graph, therandom walk-based afﬁnity measure integrates overallpaths
through theneighborhood graph. Asaresult, therandom walk-based afﬁnity measure ismuch less
sensiti veto“short-circuits” (Lee andVerleysen, 2005), inwhich asingle noisy datapoint provides
abridge between tworegions ofdataspace thatshould befarapart inthemap. Similar approaches
using random walks havealso been successfully applied to,forexample, semi-supervised learning
(Szummer andJaakk ola,2001; Zhuetal.,2003) andimage segmentation (Grady, 2006).
2594
VISUALIZINGDATAUSINGT-SNE
Themost obvious waytocompute therandom walk-based similarities pjjiistoexplicitly per-
form therandom walks ontheneighborhood graph, which works verywell inpractice, giventhat
onecaneasily perform onemillion random walkspersecond. Alternati vely,Grady (2006) presents
ananalytical solution tocompute thepairwise similarities pjjithatinvolvessolving asparse linear
system. Theanalytical solution tocompute thesimilarities pjjiissketched inAppendix B.Inpre-
liminary experiments, wedidnotﬁndsigniﬁcant differences between performing therandom walks
explicitly andtheanalytical solution. Intheexperiment wepresent below,weexplicitly performed
therandom walksbecause thisiscomputationally lessexpensi ve.However,forverylargedata sets
inwhich thelandmark points areverysparse, theanalytical solution may bemore appropriate.
Figure 7showstheresults ofanexperiment, inwhich weapplied therandom walkversion
oft-SNE to6,000 randomly selected digits from theMNIST data set,using all60,000 digits to
compute thepairwise afﬁnities pjji.Intheexperiment, weused aneighborhood graph thatwas
constructed using avalue ofk=20nearest neighbors.9The inset oftheﬁgure showsthesame
visualization asascatterplot inwhich thecolors represent thelabels ofthedigits. Inthet-SNE
map, allclasses areclearly separated andthe“continental” sevensform asmall separate cluster .
Moreo ver,t-SNE revealsthemain dimensions ofvariation within each class, such astheorientation
oftheones, fours, sevens, andnines, orthe“loopiness” ofthetwos.The strong performance of
t-SNE isalso reﬂected inthegeneralization error ofnearest neighbor classiﬁers thataretrained on
thelow-dimensional representation. Whereas thegeneralization error (measured using 10-fold cross
validation) ofa1-nearest neighbor classiﬁer trained ontheoriginal 784-dimensional datapoints is
5.75%, thegeneralization error ofa1-nearest neighbor classiﬁer trained onthetwo-dimensional
data representation produced byt-SNE isonly 5.13%. Thecomputational requirements ofrandom
walkt-SNE arereasonable: ittook only onehour ofCPU time toconstruct themap inFigure 7.
6.Discussion
Theresults intheprevious twosections (and those inthesupplemental material) demonstrate the
performance oft-SNE onawide variety ofdata sets. Inthissection, wediscuss thedifferences
between t-SNE andother non-parametric techniques (Section 6.1), andwealsodiscuss anumber of
weaknesses andpossible impro vements oft-SNE (Section 6.2).
6.1Comparison withRelatedTechniques
Classical scaling (Torgerson, 1952), which isclosely related toPCA (Mardia etal.,1979; Williams,
2002), ﬁnds alinear transformation ofthedatathatminimizes thesumofthesquared errors between
high-dimensional pairwise distances andtheir low-dimensional representati ves.Alinear method
such asclassical scaling isnotgood atmodeling curvedmanifolds anditfocuses onpreserving
thedistances between widely separated datapoints rather than onpreserving thedistances between
nearby datapoints. Animportant approach thatattempts toaddress theproblems ofclassical scaling
istheSammon mapping (Sammon, 1969) which alters thecost function ofclassical scaling by
dividing thesquared error intherepresentation ofeach pairwise Euclidean distance bytheoriginal
Euclidean distance inthehigh-dimensional space. Theresulting costfunction isgivenby
C=1
∑ijkxixjk∑
i6=j(kxixjkkyiyjk)2
kxixjk;
9.Inpreliminary experiments, wefound theperformance ofrandom walkt-SNE tobeveryrobustunder changes ofk.
2595
VANDERMAATENANDHINTON
Figure 7:Visualization of6,000 digits from theMNIST data setproduced bytherandom walk
version oft-SNE (emplo ying all60,000 digit images).
2596
VISUALIZINGDATAUSINGT-SNE
where theconstant outside ofthesum isadded inorder tosimplify thederivation ofthegradient.
Themain weakness oftheSammon costfunction isthattheimportance ofretaining small pairwise
distances inthemap islargely dependent onsmall differences inthese pairwise distances. Inpar-
ticular ,asmall error inthemodel oftwohigh-dimensional points thatareextremely close together
results inalargecontrib ution tothecost function. Since allsmall pairwise distances constitute the
local structure ofthedata, itseems more appropriate toaimtoassign approximately equal impor -
tance toallsmall pairwise distances.
Incontrast toSammon mapping, theGaussian kernel emplo yedinthehigh-dimensional space
byt-SNE deﬁnes asoft border between thelocal andglobal structure ofthedata andforpairs
ofdatapoints thatareclose together relati vetothestandard deviation oftheGaussian, theimpor -
tance ofmodeling their separations isalmost independent ofthemagnitudes ofthose separations.
Moreo ver,t-SNE determines thelocal neighborhood sizeforeach datapoint separately based onthe
local density ofthedata (byforcing each conditional probability distrib ution Pitohavethesame
perple xity).
Thestrong performance oft-SNE compared toIsomap ispartly explained byIsomap’ ssuscep-
tibility to“short-circuiting”. Also, Isomap mainly focuses onmodeling largegeodesic distances
rather than small ones.
Thestrong performance oft-SNE compared toLLE ismainly duetoabasic weakness ofLLE:
theonly thing thatprevents alldatapoints from collapsing onto asingle point isaconstraint onthe
covariance ofthelow-dimensional representation. Inpractice, thisconstraint isoften satisﬁed by
placing most ofthemap points near thecenter ofthemap andusing afewwidely scattered points
tocreate largecovariance (see Figure 3(b) and4(d)). Forneighborhood graphs that arealmost
disconnected, thecovariance constraint canalsobesatisﬁed bya“curdled” map inwhich there are
afewwidely separated, collapsed subsets corresponding tothealmost disconnected components.
Furthermore, neighborhood-graph based techniques (such asIsomap andLLE) arenotcapable of
visualizing data thatconsists oftwoormore widely separated submanifolds, because such data
does notgiverisetoaconnected neighborhood graph. Itispossible toproduce aseparate map for
each connected component, butthisloses information about therelati vesimilarities oftheseparate
components.
LikeIsomap andLLE, therandom walkversion oft-SNE emplo ysneighborhood graphs, butit
does notsufferfrom short-circuiting problems because thepairwise similarities between thehigh-
dimensional datapoints arecomputed byintegrating overallpaths through theneighborhood graph.
Because ofthediffusion-based interpretation oftheconditional probabilities underlying therandom
walkversion oft-SNE, itisuseful tocompare t-SNE todiffusion maps. Diffusion maps deﬁne a
“diffusion distance” onthehigh-dimensional datapoints thatisgivenby
D(t)(xi;xj)=vuuut∑
k
p(t)
ikp(t)
jk2
ψ(xk)(0);
where p(t)
ijrepresents theprobability ofaparticle traveling from xitoxjinttimesteps through a
graph onthedata with Gaussian emission probabilities. Thetermψ(xk)(0)isameasure forthelocal
density ofthepoints, andservesasimilar purpose totheﬁxedperple xityGaussian kernel thatisem-
ployedinSNE. Thediffusion map isformed bytheprincipal non-tri vialeigen vectors oftheMark ov
matrix oftherandom walksoflength t.Itcanbeshownthatwhen all(n1)non-tri vialeigen vec-
2597
VANDERMAATENANDHINTON
torsareemplo yed, theEuclidean distances inthediffusion map areequal tothediffusion distances
inthehigh-dimensional data representation (Lafon andLee, 2006). Mathematically ,diffusion maps
minimize
C=∑
i∑
j
D(t)(xi;xj)kyiyjk2
:
Asaresult, diffusion maps aresusceptible tothesame problems asclassical scaling: theyassign
much higher importance tomodeling thelargepairwise diffusion distances than thesmall ones and
asaresult, theyarenotgood atretaining thelocal structure ofthedata. Moreo ver,incontrast tothe
random walkversion oft-SNE, diffusion maps donothaveanatural wayofselecting thelength, t,
oftherandom walks.
Inthesupplemental material, wepresent results thatrevealthatt-SNE outperforms CCA (De-
martines andH´erault, 1997), MVU (Weinber geretal.,2004), andLaplacian Eigenmaps (Belkin and
Niyogi, 2002) aswell. ForCCA andtheclosely related CDA(Lee etal.,2000), these results can
bepartially explained bythehard border λthatthese techniques deﬁne between local andglobal
structure, asopposed tothesoftborder oft-SNE. Moreo ver,within therangeλ,CCA suffersfrom
thesame weakness asSammon mapping: itassigns extremely high importance tomodeling the
distance between twodatapoints thatareextremely close.
Liket-SNE, MVU (Weinber geretal.,2004) tries tomodel allofthesmall separations well but
MVU insists onmodeling them perfectly (i.e., ittreats them asconstraints) andasingle erroneous
constraint may severely affecttheperformance ofMVU. This canoccur when there isashort-circuit
between twoparts ofacurvedmanifold thatarefarapart intheintrinsic manifold coordinates. Also,
MVU makesnoattempt tomodel longer range structure: Itsimply pulls themap points asfarapart
aspossible subject tothehard constraints so,unlik et-SNE, itcannot beexpected toproduce sensible
large-scale structure inthemap.
ForLaplacian Eigenmaps, thepoor results relati vetot-SNE may beexplained bythefactthat
Laplacian Eigenmaps havethesame covariance constraint asLLE, anditiseasy tocheat onthis
constraint.
6.2Weaknesses
Although wehaveshownthatt-SNE compares favorably toother techniques fordatavisualization, t-
SNE hasthree potential weaknesses: (1)itisunclear howt-SNE performs ongeneral dimensionality
reduction tasks, (2)therelati velylocal nature oft-SNE makesitsensiti vetothecurse oftheintrinsic
dimensionality ofthedata, and(3)t-SNE isnotguaranteed toconvergetoaglobal optimum ofits
costfunction. Below,wediscuss thethree weaknesses inmore detail.
1)Dimensionality reduction forother purposes. Itisnotobvious howt-SNE willperform on
themore general task ofdimensionality reduction (i.e., when thedimensionality ofthedata isnot
reduced totwoorthree, buttod>3dimensions). Tosimplify evaluation issues, thispaper only
considers theuseoft-SNE fordata visualization. Thebehavioroft-SNE when reducing data totwo
orthree dimensions cannot readily beextrapolated tod>3dimensions because oftheheavytails
oftheStudent-t distrib ution. Inhigh-dimensional spaces, theheavytails comprise arelati velylarge
portion oftheprobability mass under theStudent-t distrib ution, which might lead tod-dimensional
data representations thatdonotpreserv ethelocal structure ofthedata aswell. Hence, fortasks
2598
VISUALIZINGDATAUSINGT-SNE
inwhich thedimensionality ofthedata needs tobereduced toadimensionality higher than three,
Student t-distrib utions with more than onedegree offreedom10arelikelytobemore appropriate.
2)Curseofintrinsic dimensionality .t-SNE reduces thedimensionality ofdata mainly based on
local properties ofthedata, which makest-SNE sensiti vetothecurse oftheintrinsic dimensional-
ityofthedata (Bengio, 2007). Indata setswith ahigh intrinsic dimensionality andanunderlying
manifold thatishighly varying, thelocal linearity assumption onthemanifold thatt-SNE implicitly
makes(byemplo ying Euclidean distances between near neighbors) may beviolated. Asaresult,
t-SNE might belesssuccessful ifitisapplied ondata setswith averyhigh intrinsic dimensional-
ity(forinstance, arecent study byMeytlis andSirovich (2007) estimates thespace ofimages of
faces tobeconstituted ofapproximately 100dimensions). Manifold learners such asIsomap and
LLE sufferfrom exactly thesame problems (see, e.g., Bengio, 2007; vanderMaaten etal.,2008).
Apossible wayto(partially) address thisissue isbyperforming t-SNE onadata representation
obtained from amodel thatrepresents thehighly varying data manifold efﬁciently inanumber of
nonlinear layers such asanautoencoder (Hinton andSalakhutdino v,2006). Such deep-layer archi-
tectures canrepresent comple xnonlinear functions inamuch simpler way,andasaresult, require
fewerdatapoints tolearn anappropriate solution (asisillustrated forad-bits parity taskbyBengio
2007). Performing t-SNE onadata representation produced by,forexample, anautoencoder is
likelytoimpro vethequality oftheconstructed visualizations, because autoencoders canidentify
highly-v arying manifolds better than alocal method such ast-SNE. However,thereader should note
thatitisbydeﬁnition impossible tofully represent thestructure ofintrinsically high-dimensional
data intwoorthree dimensions.
3)Non-con vexityofthet-SNE cost function. Anice property ofmost state-of-the-art dimen-
sionality reduction techniques (such asclassical scaling, Isomap, LLE, anddiffusion maps) isthe
convexityoftheir costfunctions. Amajor weakness oft-SNE isthatthecostfunction isnotconvex,
asaresult ofwhich severaloptimization parameters need tobechosen. Theconstructed solutions
depend onthese choices ofoptimization parameters andmay bedifferent each time t-SNE isrun
from aninitial random conﬁguration ofmap points. Wehavedemonstrated thatthesame choice of
optimization parameters canbeused foravariety ofdifferent visualization tasks, andwefound that
thequality oftheoptima does notvarymuch from runtorun. Therefore, wethink thattheweakness
oftheoptimization method isinsuf ﬁcient reason toreject t-SNE infavorofmethods thatleadtocon-
vexoptimization problems butproduce noticeably worse visualizations. Alocal optimum ofacost
function thataccurately captures what wewantinavisualization isoften preferable totheglobal
optimum ofacost function thatfailstocapture important aspects ofwhat wewant.Moreo ver,the
convexityofcost functions canbemisleading, because their optimization isoften computationally
infeasible forlargereal-w orld data sets, prompting theuseofapproximation techniques (deSilva
andTenenbaum, 2003; Weinber geretal.,2007). EvenforLLE andLaplacian Eigenmaps, theopti-
mization isperformed using iterati veArnoldi (Arnoldi, 1951) orJacobi-Da vidson (Fokkema etal.,
1999) methods, which may failtoﬁndtheglobal optimum duetoconvergence problems.
7.Conclusions
Thepaper presents anewtechnique forthevisualization ofsimilarity datathatiscapable ofretaining
thelocal structure ofthedata while alsorevealing some important global structure (such asclusters
10.Increasing thedegrees offreedom ofaStudent-t distrib ution makesthetails ofthedistrib ution lighter .Withinﬁnite
degrees offreedom, theStudent-t distrib ution isequal totheGaussian distrib ution.
2599
VANDERMAATENANDHINTON
atmultiple scales). Both thecomputational andthememory comple xity oft-SNE areO(n2),but
wepresent alandmark approach thatmakesitpossible tosuccessfully visualize largereal-w orld
data sets with limited computational demands. Our experiments onavariety ofdata sets show
thatt-SNE outperforms existing state-of-the-art techniques forvisualizing avariety ofreal-w orld
data sets. Matlab implementations ofboth thenormal andtherandom walkversion oft-SNE are
available fordownload athttp://ticc.uvt.nl/ ˜lvdrmaaten/tsne .
Infuture workweplan toinvestig atetheoptimization ofthenumber ofdegrees offreedom of
theStudent-t distrib ution used int-SNE. This may behelpful fordimensionality reduction when
thelow-dimensional representation hasmanydimensions. Wewillalsoinvestig atetheextension of
t-SNE tomodels inwhich each high-dimensional datapoint ismodeled byseverallow-dimensional
map points asinCook etal.(2007). Also, weaimtodevelop aparametric version oft-SNE that
allowsforgeneralization toheld-out testdata byusing thet-SNE objecti vefunction totrain amul-
tilayer neural netw orkthatprovides anexplicit mapping tothelow-dimensional space.
Acknowledgments
Theauthors thank Sam Roweis formanyhelpful discussions, Andriy Mnih forsupplying theword-
features data set,Ruslan Salakhutdino vforhelp with theNetﬂix data set(results forthese data sets
arepresented inthesupplemental material), andGuido deCroon forpointing ustotheanalytical
solution oftherandom walkprobabilities.
Laurens vanderMaaten issupported bytheCATCH-programme oftheDutch Scientiﬁc Orga-
nization (NW O),project RICH (grant 640.002.401), andcooperates with RACM. Geof freyHinton
isafellowoftheCanadian Institute forAdvanced Research, andisalso supported bygrants from
NSERC andCFIandgifts from Google andMicrosoft.
AppendixA.Derivationofthet-SNEgradient
t-SNE minimizes theKullback-Leibler divergence between thejoint probabilities pijinthehigh-
dimensional space andthejoint probabilities qijinthelow-dimensional space. Thevalues ofpijare
deﬁned tobethesymmetrized conditional probabilities, whereas thevalues ofqijareobtained by
means ofaStudent-t distrib ution with onedegree offreedom
pij=pjji+pijj
2n;
qij=
1+kyiyjk21
∑k6=l(1+kykylk2)1;
where pjjiandpijjareeither obtained from Equation 1orfrom therandom walkprocedure described
inSection 5.Thevalues ofpiiandqiiaresettozero. TheKullback-Leibler divergence between the
twojoint probability distrib utions PandQisgivenby
C=KL(PjjQ)=∑
i∑
jpijlogpij
qij
=∑
i∑
jpijlogpijpijlogqij: (6)
2600
VISUALIZINGDATAUSINGT-SNE
Inorder tomakethederivation lesscluttered, wedeﬁne twoauxiliary variables dijandZasfollows
dij=kyiyjk;
Z=∑
k6=l(1+d2
kl)1:
Note thatifyichanges, theonly pairwise distances thatchange aredijanddjifor8j.Hence, the
gradient ofthecostfunction Cwith respect toyiisgivenby
δC
δyi=∑
jδC
δdij+δC
δdji
(yiyj)
=2∑
jδC
δdij(yiyj): (7)
ThegradientδC
δdijiscomputed from thedeﬁnition oftheKullback-Leibler divergence inEquation 6
(note thattheﬁrstpartofthisequation isaconstant).
δC
δdij=∑
k6=lpklδ(logqkl)
δdij
=∑
k6=lpklδ(logqklZlogZ)
δdij
=∑
k6=lpkl1
qklZδ((1+d2
kl)1)
δdij1
ZδZ
δdij
Thegradientδ((1+d2
kl)1)
δdijisonly nonzero when k=iandl=j.Hence, thegradientδC
δdijisgivenby
δC
δdij=2pij
qijZ(1+d2
ij)22∑
k6=lpkl(1+d2
ij)2
Z:
Noting that∑k6=lpkl=1,weseethatthegradient simpliﬁes to
δC
δdij=2pij(1+d2
ij)12qij(1+d2
ij)1
=2(pijqij)(1+d2
ij)1:
Substituting thisterm intoEquation 7,weobtain thegradient
δC
δyi=4∑
j(pijqij)(1+kyiyjk2)1(yiyj):
AppendixB.Analytical SolutiontoRandomWalkProbabilities
Below,wedescribe theanalytical solution totherandom walkprobabilities thatareemplo yedinthe
random walkversion oft-SNE (see Section 5).Thesolution isdescribed inmore detail byGrady
(2006).
2601
VANDERMAATENANDHINTON
Itcanbeshownthatcomputing theprobability thatarandom walkinitiated from anon-landmark
point (onagraph thatisspeciﬁed byadjacenc ymatrix W)ﬁrstreaches aspeciﬁc landmark point
isequal tocomputing thesolution tothecombinatorial Dirichlet problem inwhich theboundary
conditions areatthelocations ofthelandmark points, theconsidered landmark point isﬁxedto
unity ,andtheother landmarks points aresettozero (Kakutani, 1945; DoyleandSnell, 1984).
Inpractice, thesolution canthus beobtained byminimizing thecombinatorial formulation ofthe
Dirichlet integral
D[x]=1
2xTLx;
where Lrepresents thegraph Laplacian. Mathematically ,thegraph Laplacian isgivenbyL=
DW,where D=diag
∑jw1j;∑jw2j;:::;∑jwnj
.Without lossofgenerality ,wemay reorder the
landmark points such thatthelandmark points come ﬁrst. Asaresult, thecombinatorial Dirichlet
integraldecomposes into
D[xN]=1
2
xT
LxT
NLLB
BTLNxL
xN
=1
2
xT
LLLxL+2xT
NBTxM+xT
NLNxN
;
where weusethesubscript Ltoindicate thelandmark points, andthesubscript Ntoindicate the
non-landmark points. Differentiating D[xN]with respect toxNandﬁnding itscritical points amounts
tosolving thelinear systems
LNxN=BT: (8)
Please note thatinthislinear system, BTisamatrix containing thecolumns from thegraph Lapla-
cian Lthat correspond tothelandmark points (excluding therowsthat correspond tolandmark
points). After normalization ofthesolutions tothesystems XN,thecolumn vectors ofXNcontain
theprobability thatarandom walkinitiated from anon-landmark point terminates inalandmark
point. One should note thatthelinear system inEquation 8isonly nonsingular ifthegraph iscom-
pletely connected, orifeach connected component inthegraph contains atleast onelandmark point
(Biggs, 1974).
Because weareinterested intheprobability ofarandom walkinitiated from alandmark point
terminating atanother landmark point, weduplicate alllandmark points intheneighborhood graph,
andinitiate therandom walks from theduplicate landmarks. Because ofmemory constraints, itis
notpossible tostore theentire matrix XNintomemory (note thatweareonly interested inasmall
number ofrowsfrom thismatrix, viz., intherowscorresponding totheduplicate landmark points).
Hence, wesolvethelinear systems deﬁned bythecolumns ofBTone-by-one, andstore only the
parts ofthesolutions thatcorrespond totheduplicate landmark points. Forcomputational reasons,
weﬁrstperform aCholesk yfactorization ofLN,such thatLN=CCT,where Cisanupper -triangular
matrix. Subsequently ,thesolution tothelinear system inEquation 8isobtained bysolving the
linear systems Cy=BTandCxN=yusing afastbacksubstitution method.
References
W.E.Arnoldi. Theprinciple ofminimized iteration inthesolution ofthematrix eigen valueproblem.
Quarterly ofApplied Mathematics ,9:17–25, 1951.
2602
VISUALIZINGDATAUSINGT-SNE
G.D. Battista, P.Eades, R.Tamassia, andI.G. Tollis. Annotated bibliograph yongraph drawing.
Computational Geometry: Theory andApplications ,4:235–282, 1994.
M.Belkin andP.Niyogi. Laplacian Eigenmaps andspectral techniques forembedding andcluster -
ing. InAdvances inNeur alInformation Processing Systems ,volume 14,pages 585–591, Cam-
bridge, MA, USA, 2002. TheMIT Press.
Y.Bengio. Learning deep architectures forAI. Technical Report 1312, Universit´edeMontr ´eal,
2007.
N.Biggs. Algebraic graph theory .InCambridg eTracts inMathematics ,volume 67.Cambridge
University Press, 1974.
H.Chernof f.Theuseoffaces torepresent points ink-dimensional space graphically .Journal ofthe
American Statistical Association ,68:361–368, 1973.
J.A. Cook, I.Sutsk ever,A.Mnih, andG.E. Hinton. Visualizing similarity data with amixture of
maps. InProceedings ofthe11thInternational Confer ence onArtiﬁcial Intellig ence andStatistics ,
volume 2,pages 67–74, 2007.
M.C. Ferreira deOliveiraandH.Levkowitz. From visual data exploration tovisual data mining: A
survey.IEEE Transactions onVisualization andComputer Graphics ,9(3):378–394, 2003.
V.deSilvaandJ.B.Tenenbaum. Global versus local methods innonlinear dimensionality reduction.
InAdvances inNeur alInformation Processing Systems ,volume 15,pages 721–728, Cambridge,
MA, USA, 2003. TheMIT Press.
P.Demartines andJ.H´erault. Curvilinear component analysis: Aself-or ganizing neural netw orkfor
nonlinear mapping ofdata sets. IEEE Transactions onNeur alNetworks ,8(1):148–154, 1997.
P.DoyleandL.Snell. Random walks andelectric netw orks. InCarus mathematical mono graphs ,
volume 22.Mathematical Association ofAmerica, 1984.
D.R. Fokkema, G.L.G. Sleijpen, andH.A. vanderVorst. Jacobi–Da vidson style QRandQZalgo-
rithms forthereduction ofmatrix pencils. SIAM Journal onScientiﬁc Computing ,20(1):94–125,
1999.
L.Grady .Random walks forimage segmentation. IEEE Transactions onPattern Analysis and
Machine Intellig ence,28(11):1768–1783, 2006.
G.E. Hinton andS.T.Roweis. Stochastic Neighbor Embedding. InAdvances inNeur alInformation
Processing Systems ,volume 15,pages 833–840, Cambridge, MA, USA, 2002. TheMIT Press.
G.E. Hinton andR.R. Salakhutdino v.Reducing thedimensionality ofdata with neural netw orks.
Science ,313(5786):504–507, 2006.
H.Hotelling. Analysis ofacomple xofstatistical variables intoprincipal components. Journal of
Educational Psychology,24:417–441, 1933.
R.A. Jacobs. Increased rates ofconvergence through learning rateadaptation. Neur alNetworks ,1:
295–307, 1988.
2603
VANDERMAATENANDHINTON
S.Kakutani. Mark ovprocesses andtheDirichlet problem. Proceedings oftheJapan Academy ,21:
227–233, 1945.
D.A. Keim. Designing pixel-oriented visualization techniques: Theory andapplications. IEEE
Transactions onVisualization andComputer Graphics ,6(1):59–78, 2000.
S.Lafon andA.B. Lee. Diffusion maps andcoarse-graining: Auniﬁed frame workfordimension-
ality reduction, graph partitioning, anddata setparameterization. IEEE Transactions onPattern
Analysis andMachine Intellig ence,28(9):1393–1403, 2006.
J.A. LeeandM.Verleysen. Nonlinear dimensionality reduction ofdata manifolds with essential
loops. Neur ocomputing ,67:29–53, 2005.
J.A. LeeandM.Verleysen. Nonlinear dimensionality reduction .Springer ,NewYork, NY,USA,
2007.
J.A. Lee, A.Lendasse, N.Donck ers,andM.Verleysen. Arobustnonlinear projection method. In
Proceedings ofthe8thEuropean Symposium onArtiﬁcial Neur alNetworks ,pages 13–20, 2000.
K.V.Mardia, J.T.Kent,andJ.M. Bibby .Multivariate Analysis .Academic Press, 1979.
M.Meytlis andL.Sirovich. Onthedimensionality offacespace. IEEE Transactions ofPattern
Analysis andMachine Intellig ence,29(7):1262–1267, 2007.
B.Nadler ,S.Lafon, R.R. Coifman, andI.G.Kevrekidis. Diffusion maps, spectral clustering and
thereaction coordinates ofdynamical systems. Applied andComputational Harmonic Analysis:
Special Issue onDiffusion Maps andWavelets ,21:113–127, 2006.
S.A. Nene, S.K. Nayar ,andH.Murase. Columbia Object Image Library (COIL-20). Technical
Report CUCS-005-96, Columbia University ,1996.
S.T.Roweis andL.K. Saul. Nonlinear dimensionality reduction byLocally Linear Embedding.
Science ,290(5500):2323–2326, 2000.
J.W.Sammon. Anonlinear mapping fordata structure analysis. IEEE Transactions onComputer s,
18(5):401–409, 1969.
L.Song, A.J. Smola, K.Borgwardt, andA.Gretton. Colored Maximum Variance Unfolding. In
Advances inNeur alInformation Processing Systems ,volume 21(inpress), 2007.
W.N.Street, W.H.Wolber g,andO.L. Mang asarian. Nuclear feature extraction forbreast tumor
diagnosis. InProceedings oftheIS&T/SPIE International Symposium onElectr onic Imaging:
Science andTechnolo gy,volume 1905, pages 861–870, 1993.
M.Szummer andT.Jaakk ola. Partially labeled classiﬁcation with Mark ovrandom walks. InAd-
vances inNeur alInformation Processing Systems ,volume 14,pages 945–952, 2001.
J.B. Tenenbaum, V.deSilva,andJ.C. Langford. Aglobal geometric frame workfornonlinear
dimensionality reduction. Science ,290(5500):2319–2323, 2000.
2604
VISUALIZINGDATAUSINGT-SNE
W.S.Torgerson. Multidimensional scaling I:Theory andmethod. Psychometrika ,17:401–419,
1952.
L.J.P .vanderMaaten, E.O. Postma, andH.J.vandenHerik. Dimensionality reduction: Acompar -
ativereview.Online Preprint, 2008.
K.Q. Weinber ger,F.Sha, andL.K. Saul. Learning akernel matrix fornonlinear dimensionality
reduction. InProceedings ofthe21stInternational Confernence onMachine Learning ,2004.
K.Q. Weinber ger,F.Sha, Q.Zhu, andL.K. Saul. Graph Laplacian regularization forlarge-scale
semideﬁnite programming. InAdvances inNeur alInformation Processing Systems ,volume 19,
2007.
C.K.I. Williams. Onaconnection between Kernel PCA andmetric multidimensional scaling. Ma-
chine Learning ,46(1-3):11–19, 2002.
X.Zhu, Z.Ghahramani, andJ.Lafferty.Semi-supervised learning using Gaussian ﬁelds andhar-
monic functions. InProceedings ofthe20thInternational Confer ence onMachine Learning ,
pages 912–919, 2003.
2605