A Regression Approach for Modeling Games with Many Symmetric Players.

A Regression Approach for Modeling Games with Many Symmetric Players
Bryce Wiedenbeck
Swarthmore College
bwieden1@swarthmore.eduFengjun Yang
Swarthmore College
fyang1@swarthmore.eduMichael P. Wellman
University of Michigan
wellman@umich.edu
Abstract
We exploit player symmetry to formulate the representa-
tion of large normal-form games as a regression task. This
formulation allows arbitrary regression methods to be em-
ployed in in estimating utility functions from a small subset of
the game’s outcomes. We demonstrate the applicability both
neural networks and Gaussian process regression, but focus
on the latter. Once utility functions are learned, computing
Nash equilibria requires estimating expected payoffs of pure-
strategy deviations from mixed-strategy proﬁles. Computing
these expected values exactly requires an infeasible sum over
the full payoff matrix, so we propose and test several approxi-
mation methods. Three of these are simple and generic, appli-
cable to any regression method and games with any number
of player roles. However, the best performance is achieved
by a continuous integral that approximates the summation,
which we formulate for the speciﬁc case of fully-symmetric
games learned by Gaussian process regression with a radial
basis function kernel. We demonstrate experimentally that the
combination of learned utility functions and expected payoff
estimation allows us to efﬁciently identify approximate equi-
libria of large games using sparse payoff data.
The study of intelligent agents naturally gives rise to ques-
tions of how multiple agents will interact. Game-theoretic
models offer powerful methods for reasoning about such in-
teractions and have therefore become a key component of
the AI toolkit. Like many core AI methods, game-theoretic
analysis has beneﬁted from recent advances in machine
learning algorithms and from the availability of ever-larger
data sets. Most notably, automated playing and solving of
extremely large extensive-form games has advanced con-
siderably with the aid of learning algorithms that general-
ize across game states. In this paper, we consider extremely
large normal-form games, and show that learning algorithms
generalizing across symmetric players can help to compactly
represent and efﬁciently solve such games.
In both normal and extensive form games, the chief ob-
stacle to computing equilibria is the enormous size of the
standard input representation (Papadimitriou and Roughgar-
den 2005; Bowling et al. 2015). In games like poker and
go, the extensive form representation is far too big to be
constructed explicitly, but observations of particular states
Copyright c
2018, Association for the Advancement of Artiﬁcial
Intelligence (www.aaai.org). All rights reserved.of the game can be straightforwardly generated by simula-
tion. Using simulated data from a small subset of the game’s
states, learning techniques have succeeded in extracting gen-
eral principles that apply to never-before-seen parts of the
game. Agents employing such learning have achieved high-
level play (Silver et al. 2016; Morav ˇc´ık et al. 2017) and iden-
tiﬁed approximate equilibria (Heinrich, Lanctot, and Silver
2015).
In normal-form games, the representational complexity
derives principally from the number of players. A standard
payoff matrix representation of a game with jPjplayers and
jSjstrategies per player records a vector of payoff values
in each ofjSjjPjcells. In many AI applications, symme-
tries among agents permit some economy of representation,
but even a fully symmetric game must record payoffs forjPj+jSj1
jPj
distinct outcomes (Cheng et al. 2004). As a re-
sult, analysts are often restricted to studying games with
small numbers of players and strategies, or with special-
purpose compact representations.
In the present work, we use data about a small subset of
a game’s outcomes as input to learning methods that seek
to extract general principles of the game’s utility function.
Data of this form is common in the ﬁeld of empirical game-
theoretic analysis (Wellman 2006). EGTA is used to study
multi-agent interactions where a closed-form game model is
unavailable or intractable. In such settings, an agent-based
model can often capture key features of the interaction and
provide data about agent incentives. Generally, a single run
of an agent-based simulation generates a noisy sample of
each payoff value in one cell of the payoff matrix. Set-
tings where EGTA has been employed include identifying
bidding strategies in continuous double auctions (Phelps,
Marcinkiewicz, and Parsons 2006), designing network rout-
ing protocols (Wellman, Kim, and Duong 2013), under-
standing credit provision in the absence of a central currency
(Dandekar et al. 2015), and ﬁnding ways to mitigate the ef-
fects of high-frequency trading (Wah and Wellman 2016).
Because of the combinatorial growth of the payoff matrix,
ﬁlling every cell by simulation quickly becomes intractable
as the number of players and strategies grows. Early EGTA
studies were often restricted to small numbers of players and
strategies (Phelps, Marcinkiewicz, and Parsons 2006; Walsh
et al. 2002). More recently, growth in the strategy space
has been kept in check by iterative exploration approaches,
and games with many players have been approximated using
player reduction methods (Wellman, Kim, and Duong 2013;
Wah, Hurd, and Wellman 2015). Our work presents an alter-
native to player reduction, where the use of machine learn-
ing allows for better generalization from more ﬂexible data
sets, resulting in better approximations of large symmetric
games.
The key to learning that generalizes across outcomes is
that games with large numbers of players generally exhibit
substantial structure beyond player symmetries. With many
players, we often expect that no single opponent can unilat-
erally exert an outsized inﬂuence on a player’s payoff. For
example, in a 100-player game, the difference in payoff be-
tween 36 or 37 opponents choosing a particular strategy is
likely to be small. A related notion of bounded inﬂuence
is formalized and studied by Kearns and Mansour (2002).
Further, when a game represents interactions among com-
putational agents, and the payoff matrix is too big to be rep-
resented, we should expect that the agents themselves are
not reasoning about the full game, but rather some more-
compact summarization. As a result, in many large games of
interest, payoff functions will exhibit smoothness and sim-
plicity that make them amenable to machine learning.
Related Work
Much previous work in machine learning for game analysis
has focused on extensive-form games like poker (Heinrich,
Lanctot, and Silver 2015; Morav ˇc´ık et al. 2017). In these
settings, the game being studied is precisely deﬁned, but too
large to be analyzed directly, so learning is used to express
strategies compactly and estimate expected payoffs in var-
ious game states without performing an exhaustive search.
Our work is motivated by empirical settings where the game
model, in addition to being extremely large, is initially un-
known and must be induced from data. As is common in
such settings, we treat the set of strategies as ﬁxed and ex-
ogenously speciﬁed. Further, our methods treat strategies as
categorical, requiring no relationship among the strategies to
make generalizations.
We are interested analyzing in games with a sufﬁciently
large number of players to pose representational challenges
even if the set of strategies can be fully enumerated. A
common approach to analyzing normal-form games with
many players is to employ game models with compact rep-
resentations that can be analyzed directly. Examples in-
clude potential games (Monderer and Shapley 1996), graph-
ical games (Kearns 2007), and action-graph games (Jiang,
Leyton-Brown, and Bhat 2011). In an empirical setting,
where the game model is not known in advance, such repre-
sentations are difﬁcult to apply, but some researchers have
investigated learning compact representations from data.
Duong et al. (2009) developed a method for detecting graph-
ical structures representing independences among players’
payoffs. Honorio and Ortiz (2015) likewise learn graphical
game models, in their case from observations of play rather
than payoffs, using assumptions about the structure of utility
functions and the way that play is generated conditional on
the actual payoff function. Neither of these approaches canbe applied to symmetric games, because graphical games
derive their compactness from player independence, which
cannot arise in non-trivial symmetric games.
An alternative approach that has been used when EGTA
environments simulate a large number of symmetric play-
ers is called player reduction . Player reductions deﬁne a
reduced game with a small number of players, and ﬁll in
the reduced game’s payoff matrix using data from the full
game by aggregating the decisions of several symmetric
players. Equilibria are then computed in the reduced game
and treated as approximate equilibria of the full game. Sev-
eral player reduction methods have been proposed, varying
on the choice of full-game proﬁles to simulate and how they
map them to payoffs in the reduced game.
The ﬁrst player-reduction method to see widespread use
was hierarchical reduction (Wellman et al. 2005), which
treats each reduced-game player as controlling an equal
fraction of the full-game agents. Hierarchical reduction
has been largely supplanted by a more recent technique
called deviation-preserving reduction (DPR) (Wiedenbeck
and Wellman 2012). DPR treats each player in the reduced
game as controlling the strategy choice of a single agent
in the full game, but each player views its opponents as an
aggregation of the remaining full-game agents. This means
that payoff differences resulting from a single reduced-game
player switching strategies reﬂect payoff changes from a
single full-game agent deviating, making DPR equilibria
more reﬂective of full-game equilibria. We employ DPR as a
benchmark against which to compare our learning methods.
Relative to DPR or other player reductions, our methods
take advantage of greater ﬂexibility in allowable input data.
Player reduction methods prescribe a ﬁxed set of proﬁles to
simulate, and to ensure accurate estimates of reduced game
payoffs, users often end up simulating the same proﬁle many
times. Our learned models do not require a ﬁxed set of full-
game proﬁles and can therefore spread sampling effort over
a wider variety of proﬁles. DPR also ignores some freely
available data: simulating proﬁles with many strategies but
using the payoff data for only one strategy. Our regressions
can always make use of any data that is available.
Most closely related to the present work, V orobeychik,
Wellman, and Singh (2007) demonstrated the use of regres-
sion methods to learn payoff functions over continuous strat-
egy spaces. The present paper can be viewed as extend-
ing their work to the domain of categorical strategies. Their
learning methods relied on strategy sets that were fully de-
scribed by varying continuous parameters, such as bids in
a single-unit auction. By contrast, our methods can handle
arbitrary sets of strategies, relying instead on the game hav-
ing a large number of symmetric players, which is typical of
environments deﬁned by agent-based simulations.
Background and Notation
Game Theory
We focus on games represented in normal form that have
signiﬁcant symmetry among players. In the following pre-
sentation, we focus on fully-symmetric games, where all
players have the same set of strategies and face the same
set of incentives. However, most of our models generalize
straightforwardly to role-symmetric games, where players
are partitioned into some number of roles (such as buyers
and sellers), and players are symmetric within, but not across
roles. Formally, a symmetric game consists of:
a set of players P
a set of strategies S
a utility function u:S~S!R
Aproﬁle~ sis an assignment of one strategy to every
player. Because players are symmetrical, we can represent
a proﬁle by a vector of the number of players choosing each
strategy. We denote the set of all proﬁles ~S. The utility func-
tionu(s;~ s)maps a proﬁle ~ sand a strategy sto the utility
of a player choosing strategy swhen players jointly choose
proﬁle~ s. A mixed strategy speciﬁes a probability distri-
bution over a player’s strategies. A symmetric mixture ~ is
a common mixed strategy played by all agents.
A player selecting pure strategy swhen other players
jointly play according to symmetric mixture ~ receives an
expected payoff:
u(s;~ ) =X
~ s2~SPr[~ sj~ ]u(s;~ s) (1)
A player from playing according to a symmetric mixture ~
receives expected payoff:
u(~ ) =X
s2S~ (s)u(s;~ )
The regret of a symmetric mixture is the maximum
amount that any player could gain by deviating to a pure
strategy:
regret (~ ) = max
s2Su(s;~ )u(~ )
A symmetric Nash equilibrium is a symmetric mixture
withregret (~ ) = 0 . It follows from a proof by Nash (1951)
that a (role-) symmetric game must have a (role-) symmetric
Nash equilibrium, but ﬁnding a Nash equilibrium is com-
putationally hard. Typical analysis of normal form games
seeks an-Nash equilibrium, a proﬁle ~ withregret (~ ).
In this paper, we measure our success in approximating
large games in two ways. First, we compare expected pay-
offsu(s;~ )estimated by our model to ground truth ex-
pected payoffs in the game being learned, averaged over a
wide range of symmetric mixtures. Second, we identify -
Nash equilibria in our learned models and compute their re-
gret in the ground-truth game. In the experiments presented
here, we compute symmetric mixed-strategy equilibria using
replicator dynamics (Gintis 2009), but we see similar results
when computing equilibria with ﬁctitious play.
Gaussian Process Regression
Gaussian process regression (GPR) is a ﬂexible method for
supervised learning (Rasmussen and Williams 2006) that
learns a mapping from input ~ xito outputyi. The set ofnin-
put points of dimension dcan be collected into an ndma-
trixX, and the corresponding targets into the n-dimensionalcolumn vector ~ y. GPR estimates the value at a new point ~ x
ask(~ x)K1~ y, wherekandKare deﬁned as:
k(~ x)[k(~ x1;~ x);k(~ x2;~ x);:::;k (~ xn;~ x)]
K2
664k(~ x1;~ x1)k(~ x1;~ x2)k(~ x1;~ xn)
k(~ x2;~ x1)k(~ x2;~ x2)k(~ x2;~ xn)
............
k(~ xn;~ x1)k(~ xn;~ x2)k(~ xn;~ xn)3
775
The kernel function k(;)must be speciﬁed by the user;
we use the radial basis function (RBF) kernel:
k
~ a;~b
=cexp
1
2l2

~ a~b

2
2
(2)
The RBF kernel has an important hyperparameter l, the
length-scale over which the function is expected to vary.
This can be estimated by MLE, but in our experiments, we
found it important to constrain lto the range [ 1;jPj], and
that a length scale close to these bounds was sometimes ev-
idence of a poor ﬁt.
Distributions
We denote a multivariate Gaussian distribution with mean
vector~ and covariance matrix asN(j~ ;). The prod-
uct of two Gaussians can be rewritten in the following way
(Petersen and Pedersen 2008):
N(~ xj~ 1;1)N(~ xj~ 2;2) =
N(~ 1j~ 2;1+2)N(~ xj~ 3;3)(3)
where~ 3and3are deﬁned as
~ 33(11~ 1+21~ 2)
3(11+21)1
We denote the multinomial distribution of ndraws from
discrete distribution ~ pasM(jn;~ p). Whennis large and~ p
is not near the edge of the domain, a multinomial distribution
is well-approximated by the following Gaussian distribution
(Severini 2005):
M(n;p)N(np;M ) (4)
whereMis deﬁned as
M2
664p1p2
1p1p2 p1pn
p2p1p2p2
2 p2pn
............
pnp1pnp2pnp2
n3
775
Methods
Our method for approximating normal-form games relies on
two key steps. First, we use regression to learn a mapping
from pure-strategy proﬁles to payoffs. This mapping allows
us to generalize from a small data set to functions that can
be efﬁciently queried for arbitrary proﬁles. Second, we use
queries to these utility functions to estimate expected pay-
offs of playing each pure strategy against a symmetric mix-
ture. These expected payoff estimates enable us to compute
symmetric mixed-strategy -Nash equilibria.
(a) Generic methods.
0:1 0:5 0:9
probability of strategy 1expected payotrue game
point
neighbor
sample
(b) GP-speciﬁc integration method.
0:1 0:5 0:9
probability of strategy 1expected payotrue game
integration
Figure 1: Comparing methods for estimating the expected
payoff to strategy 1 in a 100-player, 2-strategy game learned
from complete data with GPR. Zoom recommended.
Payoff Learning
The key insight that enables us to learn payoffs from player
symmetry is that strategy proﬁles in a (role-) symmetric
game can be encoded as a vector of strategy counts. For
each strategy s, an entry in this vector encodes the number
of players selecting strategy s. By representing each strategy
as a separate dimension of the regression input, this method
does not emphasize generalizing across strategies. Instead,
it allows us to learn general effects caused by many play-
ers selecting the same strategy. Such effects are common
in the literature, appearing in congestion games, local ef-
fect games, and more-specialized compact representations
of large games
Further emphasizing the categorical nature of normal-
form game strategies, we run a separate regression for each
strategy’s utility function. This serves two purposes: ﬁrst, it
lowers the overall runtime of the regression, and second, it
allows us to specialize the data set. Given a collection of pro-
ﬁles and corresponding payoff values, we construct the data
set for strategy sby selecting all proﬁles (and corresponding
payoffs) where at least one player chooses s. Any regres-
sion method can be run on this data set; we focus our testing
on Gaussian process regression, but have also run proof-of-
concept tests using neural networks.
Data Selection In initial testing, we found that our meth-
ods produced low average error in estimating expected util-
ities, but surprisingly poor results in identifying Nash equi-
libria. The root cause of this problem was inaccurate regres-sion estimates near the edges of the proﬁle space (zero or
one players selecting a strategy). Such proﬁles were rare in
randomly-generated data sets, but are extremely important
in computing equilibria, because in most equilibria, only a
small number of strategies are played with non-zero prob-
ability (Porter, Nudelman, and Shoham 2008). As a result,
the data set for all experiments that follow include a large
over-representation of proﬁles in which zero or one players
play various strategies.
Expected Payoff Estimation
Given a learned payoff model for a symmetric game, we
want to identify symmetric mixed-strategy -Nash equilib-
ria. The critical input to computing equilibria is the expected
payoff of playing pure strategy sagainst opponents jointly
following a symmetric mixture ~ , given by equation 1. Com-
puting this expectation exactly requires summing over all
proﬁles in the game, and is therefore infeasible in large
games. We propose and evaluate several methods for esti-
mating expected payoffs without computing the full sum.
Generic Methods We consider three methods for estimat-
ing expected payoffs that are applicable regardless what re-
gression method was used to learn utility functions. The ﬁrst
method, sampling , selectskrandom proﬁles according the
distribution~ siPr[~ sj~ ], and computes the average payoff:
u(s;~ )1
kkX
i=1^u(s;~ si)
Where ^uis the regression estimate. The second method,
point , queries the learned function only at the modal proﬁle:
u(s;~ )^u(s;jPj~ )
The third method neighbor computes a weighted sum over
just the proﬁles within ddeviations of the maximum-
likelihood proﬁle. Letting sbe the maximum-likelihood pro-
ﬁle, we deﬁne the set N=f~ sjk~ ssk1dg. The neigh-
bor estimate is then:
u(s;~ )1P
~ s2NPr[~ sj~ ]X
~ s2NPr[~ sj~ ]^u(s;~ s)
All three generic methods have strengths and weaknesses.
Sampling provides an unbiased estimator, and is correct in
the limit as k! 1 . However, it provides unstable es-
timates that are unsuitable for most algorithms that com-
pute Nash equilibria. Point estimation is fast, requiring far
fewer queries to the regression model than any other method.
It also provides smoothly-varying estimates (subject to the
smoothness of the regression model) that are correct in
the limit asjPj ! 1 . However, its payoff estimates ex-
hibit bias that can interfere with equilibrium computation or
other analysis. Neighbor estimation provides a sort of mid-
dle ground, in that avoids the randomness of sampling and
has lower bias than point estimation. When d= 0neighbor
approximates point, and when d=jPj, neighbor computes
the exact expected payoff, so dcan be chosen to trade off ac-
curacy with computation time. Unfortunately, neighbor suf-
fers from discrete-steps in its estimates as the maximum-
likelihood proﬁle changes. This problem, illustrated in ﬁg-
ure 1a, can occasionally prevent iterative methods for equi-
librium computation from converging.
Continuous Approximation A natural approach to the
problem of estimating a sum over a ver large number of low-
probability terms is to approximate the summation with an
integral. In this section we show how this can be done for
a fully-symmetric game learned using Gaussian process re-
gression with a radial basis function kernel. In the case of a
one-role game, the probability of a proﬁle can be expressed
as a multinomial, Pr[~ sj~ ] =M(~ sjn1;), so we can
re-write equation 1 as:
u(s;~ ) =X
~ s2~SM(~ sjn1;)u(~ s;s)
X
~ s2~SM(~ sjn1;)k(~ s)K1~ y (5)
=~kK1~ y (6)
Equation 5 employs the GPR payoff estimate for strategy
s. We reach equation 6 by noting that K1~ ydoes not depend
on~ s, so it can be pulled out of the sum, and by deﬁning ~kP
~ s2~SM(~ sjn1;)k(~ s). We next consider component
iof this vector, ~ki. DeﬁningC1in terms of the mixture :
C12
66412
112 1n
2122
2 2n
............
n1n2n2
n3
775
lets us use equation 4 to approximate ~kiwith a Gaussian.
Since we now have a continuous distribution, the summa-
tion can be approximated by an integration over the proﬁle
simplex.
~kiX
~ s2~SN~ s(n~ ;C 1)k(~ s;~ xi)
~kiZ
~SN~ s(n~ ;C 1)k(~ s;~ xi)d~ s (7)
Figure 2: Comparing input data for regression in a 100-
player, 3-strategy congestion game. Simplex coordinates
specify a mixture, and color speciﬁes error from true ex-
pected payoffs. Left: payoffs are estimated poorly near the
corners of the simplex. Right: over-sampling edge proﬁles
reduces this error.We can further simplify equation 7 by re-writing the RBF
kernel from equation 2 in the form of a Gaussian:
k(~ s;~ xi) =cexp
1
2l2(~ s~ xi)TC21(~ s~ xi)
=c
(2)jSjdet (C2)1
2N(~ sj~ xi;C2)
where we deﬁne C2by:
C212
6642=l21=l2 1=l2
1=l22=l2 1=l2
............
1=l21=l2 2=l23
775
This gives us:
~kiZ
~SN(~ sjn~ ;C 1)c
(2)jSjdet(C2)1
2N(~ sj~ xi;C2)d~ s
=c((2)jSjdet(C2))1
2Z
~SN~ s(n~ ;C 1)N~ s(~ xi;C2)d~ s
=c((2)jSjdet(C2))1
2Z
~SN(n~ j~ xi;C1+C2)
N(~ sj0;C0)d~ s (8)
=c((2)jSjdet(C2))1
2N(n~ j~ xi;C1+C2)Z
~SN(~ sj0;C0)d~ s (9)
Equation 8 makes use of the product-of-Gaussians iden-
tity from equation 3. This leaves us with the integration of
a Gaussian distribution over the full proﬁle simplex, which
we can approximate by 1, simplifying equation 9 to:
~kic((2)jSjdet(C2))1
2N(n~ j~ xi;C1+C2)
=c(2)jSjdet(C2)
(2)jSjdet(C1+C2)1
2

exp
(n~ ~ xi)T(C1+C2)1(n~ ~ xi)
=cdet(C2)
det(C1+C2)1
2

exp
(n~ ~ xi)T(C1+C2)1(n~ ~ xi)
(10)
Equation 10 shows how we can approximate each element
of the vector ~k, and therefore each component of equation 6
computationally. In equation 10, C2depends only on the
length-scale lof the RBF kernel, so det(C2)can be com-
puted in advance and re-used for every expected payoff com-
putation.C1changes with each mixture being evaluated, but
given a mixture, it is the same for all pure strategies and each
~ki. This means that each mixture considered in an equilib-
rium computation algorithm requires one inversion and one
determinant calculation on an nnmatrix.
In principle, this approximation decays near the edges of
the mixed-strategy simplex, because the Gaussian approxi-
mation to the multivariate distribution and the approxima-
tion of the full-simplex integral by 1 should both perform
Figure 3: Comparing accuracy of expected payoff estimates
by GPR and neural networks. Simplex coordinates specify
a mixture, and color speciﬁes error from true expected pay-
offs. Left: GPR. Right: 60-node neural network.
less well. In practice, however, we have found this decay to
be small relative to the inherent inaccuracy of learning from
small data sets. As shown in ﬁgure 1b, when the Gaussian
process regression learns accurate payoffs, the integration
methods estimates expected payoffs extremely accurately.
Experiments
In all of our experiments, we generate random large games,
which we represent compactly as action-graph games with
additive function nodes (Jiang, Leyton-Brown, and Bhat
2011). This ensures that we can compare the results of var-
ious approximation methods to a ground truth, checking the
expected payoff of mixed strategies, and the regret of ap-
proximate equilibria. Previous work in approximating large
games has used similar random data sets, but focused on
much smaller games; we believe that our experiments on
games with 100 or more players provide strong evidence that
our methods can scale effectively.
In our ﬁrst experiment, we isolate the effect of expected
payoff estimation methods by constructing a 100-player, 2-
strategy game, and providing exact payoffs for all 101 pro-
ﬁles as inputs to GPR. Figure 1 compares all three generic
methods and the GPR-speciﬁc integration method for ex-
pected payoff estimation. Figure 1a shows that sampling
(withk= 100 ) to be quite accurate, but noisy. Point estima-
tion is smooth, but overshoots the movements of the true ex-
pected payoff. Neighbor estimation (with d= 5) falls some-
where in between, exhibiting moderate bias and less noise.
However, note the stepped appearance of the neighbor es-
timates; we found that this occasionally prevented equilib-
rium computation from converging. Figure 1b shows excel-
lent performance for the integration method, which is borne
out through the remainder of our results.
In our second experiment, we compare different choices
of input data in a 100-player, 3-strategy congestion game.
In Figure 2, the simplex coordinates specify a mixed strat-
egy: the corners correspond to all players choosing the same
strategy with probability 1, and the center is the uniform dis-
tribution. Color plotted in the simplex gives the error rela-
tive to true-game expected payoffs: blue indicates low er-
ror, green indicates moderate error. expected payoffs come
from GPR and point estimation in both plots. In the left-hand
plot, input proﬁles have been spaced evenly throughout the(a) Error vs. approximation method and game size:
DPR IGP PGP NGP0:000:010:020:030:040:050:060:07expected payo error33 players
65 players
129 players
(b) Error vs. approximation method and game type:
DPR IGP PGP NGP0:000:020:040:060:080:100:12expected payo errorCongestion
Local Eect
Polynomial
Sin
Figure 4: Learning methods compared to deviation-
preserving reduction (DPR). IGP GPR + integration, PGP
GPR + point, NGP GPR + neighbor. IGP performs best
across all game sizes and most game types.
proﬁle space ~S. This results in insufﬁcient data for accurate
regression estimates near the extremes of the simplex. Be-
cause Nash equilibria often occur near edges of the simplex,
especially in higher dimensions (more strategies), this can
cause large errors in equilibrium estimates. In the right-hand
plot, proﬁles have been re-allocated to the edges of the sim-
plex, over-representing proﬁles with 1 or 0 players choosing
a given strategy. This helps the regression to develop better
estimates of the extreme proﬁles.
Our third experiment demonstrates that other regression
methods can also be used. Figre 3 again shows errors in ex-
pected payoff estimates plotted on the probability simplex
for a 100-player, 3-strategy congestion game. The left sim-
plex of Figure 3 was created with the same settings as the
right simplex of Figure 2, but a different randomly generated
congestion game. The right simplex shows the accuracy of
neural network learning with point-estimate expected pay-
offs on the same game. The neural network used hidden lay-
ers of 32, 16, 8, and 4 nodes, with sigmoid activation func-
tions, 0.2 dropout probability and the Adam optimizer. The
average error is comparable across the two methods, but the
distribution of mistakes differs signiﬁcantly. The neural net-
work hyperparameters may not be sufﬁciently optimized.
Our fourth experiment compares our regression method
against the best existing method for approximating large
symmetric games, deviation-preserving reduction (Wieden-
beck and Wellman 2012). Figure 4 shows average results on
a data set of 120 randomly-generated games. The games in-
clude 10 instances of each combination of parameters from
(33, 65, or 129 players) and (congestion game, local effect
game, action-graph game with polynomial function nodes,
or action-graph game with sinusoidal function nodes). All
methods were given the same amount of data, chosen to
suit DPR. The number of players in the random games were
also chosen to be optimal for DPR. The graph shows av-
erage error in estimating the expected payoff u(s;~ ), for
a large set of symmetric mixtures, including a grid spaced
evenly across the space of possible mixtures and a num-
ber of randomly generated mixtures. Despite having many
parameters chosen advantageously, DPR was outperformed
by the regression methods on all game sizes and nearly
all game types. Among the regression methods, estimating
expected payoffs by the continuous integral approximation
was clearly superior. Neighbor estimation does not consis-
tently out-perform point estimation, which suggests that it is
probably not worth the extra computational burden of query-
ing many more points. We also computed equilibria in these
games, and while DPR closes the gap slightly in terms of
measured true-game regret, GPR with integration remains
the clear winner.
Our ﬁfth experiment compares GPR with integration to
DPR as a function of the number of proﬁles used as input,
with noisy observations. For this experiment, we constructed
a data set of 15 action-graph games with polynomial func-
tion nodes, and used rejection sampling to ensure that all
games in the data set had only mixed symmetric equilibria.
In the preceding experiments, all methods received correct
payoff values for proﬁles in the data set. Here each data point
has had normally-distributed noise added. In the presence of
noise, it is common, before performing player reduction, to
simulate the same proﬁle multiple times for a better esti-
mate of its payoffs. Our experiment shows that this can be a
good use of simulation resources, as increasing the number
of samples per proﬁle reduces error and regret as a function
of the total number of simulations over the range of 1–20
samples per proﬁle. This effect tapers off eventually, and by
100 samples per proﬁle, it would be better to sample more
proﬁles fewer times.
Because regression is inherently robust to noisy inputs,
our method has less need to resample the same proﬁle re-
peatedly, and can sample a larger variety of proﬁles at the
same simulation cost. As shown in Figure 5a, our method
signiﬁcantly outperforms all variants of DPR in terms of av-
erage error of expected payoff estimates. Our most impor-
tant experimental result is shown in Figure 5b. This graph
demonstrates that -Nash equilibria computed by replicator
dynamics have signiﬁcantly lower true-game regret under
our method than under DPR.
Conclusions
We have demonstrated a new method for computing ap-
proximate Nash equilibria in games with a large number(a) expected payoff estimation error vs. data set size:
0 5000 10000 15000 20000 25000
proles0:000:020:040:060:080:100:12expected payo error
dpr 1 sample
dpr 2 samples
dpr 20 samples
dpr 100 samples
gp (integration)
(b) Regret of computed equilibria vs. data set size:
0 5000 10000 15000 20000 25000
proles0:000:010:020:030:040:050:060:070:08regret
dpr 1 sample
dpr 2 samples
dpr 20 samples
dpr 100 samples
gp (integration)
Figure 5: Learning compared to deviation-preserving reduc-
tion (DPR) with noisy payoff samples. IGP performs better
in terms of error and regret whether DPR samples proﬁles
repeatedly, or constructs a larger reduced game.
of players. Our method uses player symmetries as the ba-
sis for regression-learning of pure-strategy payoffs. Using
these regression estimates, we have shown that expected
payoffs of mixed strategies can be estimated effectively, al-
lowing low-regret symmetric mixtures to be identiﬁed. We
provided strong experimental evidence that our methods out-
perform previous techniques for working with large normal-
form games.
Future work on this topic should include extending the
continuous regression approximation to games with multiple
roles and/or to other regression methods. We would also like
to combine our methods for generalizing over players with
existing methods for generalizing over strategies (V orobey-
chik, Wellman, and Singh 2007). The ability to use data
about arbitrary proﬁles as input to regression opens the prob-
lem of choosing an appropriate set of proﬁles to simulate; it
may be possible to interleave equilibrium computation and
sample collection in useful ways. Finally, we hope to inves-
tigate the possibility of learning expected payoffs directly;
if feasible, this could dramatically improve computational
efﬁciency, and/or approximation performance.
References
Bowling, M.; Burch, N.; Johanson, M.; and Tammelin, O.
2015. Heads-up limit hold’em poker is solved. Science
347(6218):145–149.
Cheng, S.-F.; Reeves, D. M.; V orobeychik, Y .; and Wellman,
M. P. 2004. Notes on equilibria in symmetric games. In Sixth
International Workshop on Game-Theoretic and Decision-
Theoretic Agents .
Dandekar, P.; Goel, A.; Wellman, M. P.; and Wiedenbeck, B.
2015. Strategic formation of credit networks. ACM Trans-
actions on Internet Technology 15(1):3:1–41.
Duong, Q.; V orobeychik, Y .; Singh, S.; and Wellman, M. P.
2009. Learning graphical game models. In 21st Interna-
tional Joint Conference on Artiﬁcial Intelligence , 116–121.
Gintis, H. 2009. Game Theory Evolving: A problem-
centered introduction to modeling strategic behavior .
Princeton university press, second edition.
Heinrich, J.; Lanctot, M.; and Silver, D. 2015. Fictitious
self-play in extensive-form games. In 32nd International
Conference on Machine Learning , 805–813.
Honorio, J., and Ortiz, L. 2015. Learning the structure and
parameters of large-population graphical games from behav-
ioral data. Journal of Machine Learning Research 16:1157–
1210.
Jiang, A. X.; Leyton-Brown, K.; and Bhat, N. A. R. 2011.
Action-graph games. Games and Economic Behavior
71(1):141–173.
Kearns, M., and Mansour, Y . 2002. Efﬁcient Nash com-
putation in large population games with bounded inﬂuence.
In18th Conference on Uncertainty in Artiﬁcial Intelligence ,
259–266.
Kearns, M. 2007. Graphical games. In Vazirani, V .;
Nisan, N.; Roughgarden, T.; and Tardos, E., eds., Algorith-
mic Game Theory . Cambridge University Press. chapter 7,
159–180.
Monderer, D., and Shapley, L. S. 1996. Potential games.
Games and Economic Behavior 14(1):124–143.
Morav ˇc´ık, M.; Schmid, M.; Burch, N.; Lis `y, V .; Morrill, D.;
Bard, N.; Davis, T.; Waugh, K.; Johanson, M.; and Bowling,
M. 2017. Deepstack: Expert-level artiﬁcial intelligence in
heads-up no-limit poker. Science 356(6337):508–513.
Nash, J. 1951. Non-cooperative games. Annals of Mathe-
matics 286–295.
Papadimitriou, C. H., and Roughgarden, T. 2005. Comput-
ing equilibria in multi-player games. In 16th Annual ACM-
SIAM Symposium on Discrete Algorithms , 82–91.
Petersen, K. B., and Pedersen, M. S. 2008. The matrix cook-
book. Technical report, Technical University of Denmark.
Phelps, S.; Marcinkiewicz, M.; and Parsons, S. 2006. A
novel method for automatic strategy acquisition in n-player
non-zero-sum games. In 5th International Joint Conference
on Autonomous Agents and Multiagent Systems , 705–712.
Porter, R.; Nudelman, E.; and Shoham, Y . 2008. Simple
search methods for ﬁnding a Nash equilibrium. Games and
Economic Behavior 63(2):642–662.Rasmussen, C. E., and Williams, C. K. 2006. Gaussian
Processes for Machine Learning , volume 1. MIT Press.
Severini, T. 2005. Elements of Distribution Theory . Cam-
bridge Series in Statistica. Cambridge University Press.
Silver, D.; Huang, A.; Maddison, C. J.; Guez, A.; Sifre, L.;
Van Den Driessche, G.; Schrittwieser, J.; Antonoglou, I.;
Panneershelvam, V .; Lanctot, M.; et al. 2016. Mastering
the game of go with deep neural networks and tree search.
Nature 529(7587):484–489.
V orobeychik, Y .; Wellman, M. P.; and Singh, S. 2007. Learn-
ing payoff functions in inﬁnite games. Machine Learning
67(1-2):145–168.
Wah, E., and Wellman, M. P. 2016. Latency arbitrage in
fragmented markets: A strategic agent-based analysis. Al-
gorithmic Finance 5:69–93.
Wah, E.; Hurd, D.; and Wellman, M. P. 2015. Strategic
market choice: Frequent call markets vs. continuous double
auctions for fast and slow traders. In Third EAI Conference
on Auctions, Market Mechanisms, and Their Applications .
Walsh, W. E.; Das, R.; Tesauro, G.; and Kephart, J. O.
2002. Analyzing complex strategic interactions in multi-
agent systems. In AAAI-02 Workshop on Game-Theoretic
and Decision-Theoretic Agents .
Wellman, M. P.; Reeves, D. M.; Lochner, K. M.; Cheng, S.-
F.; and Suri, R. 2005. Approximate strategic reasoning
through hierarchical reduction of large symmetric games.
In20th National Conference on Artiﬁcial Intelligence , 502–
508.
Wellman, M. P.; Kim, T. H.; and Duong, Q. 2013. Analyzing
incentives for protocol compliance in complex domains: A
case study of introduction-based routing. In 12th Workshop
on the Economics of Information Security .
Wellman, M. P. 2006. Methods for empirical game-theoretic
analysis (extended abstract). In 21st National Conference on
Artiﬁcial Intelligence , 1152–1155.
Wiedenbeck, B., and Wellman, M. P. 2012. Scal-
ing simulation-based game analysis through deviation-
preserving reduction. In 11th International Conference on
Autonomous Agents and Multiagent Systems , 931–938.

Bryce Wiedenbeck
Fengjun Yang
Michael P
Wellman