Who owns artificial intelligence? A preliminary analysis of corporate intellectual property strategies and why they matter.

WORKING PAPER
Who owns artificial intelligence? A preliminary analysis
of corporate intellectual property strategies and why
they matter

Nathan Calvin and Jade Leung1
Centre for the Governance of AI
Future of Humanity Institute, University of Oxford

February 2020

1 For comments and ideas we thank Markus Anderljung, Miles Brundage, Allan Dafoe, Jeffrey Ding, Carrick Flynn,
Rose Hadshar, Carin Ism, Saif Khan, Brian Tse, Baobao Zhang, and Remco Zwetsloot. Special thanks to Cullen
O’Keefe, who provided crucial input on several drafts, Emefa Agawu, for substantial support improving the
structure and content of the report, and Emmie Hines, for proofreading a nd copyediting support. This work was
funded by the Berkeley Existential Risk Initiative. All errors are ours alone.

 Summary and Key Takeaways
This working paper is a preliminary analysis of the legal rules, norms, and strategies governing artificial
intelligence (AI) -related intellectual property (IP). We analyze the existing AI -related IP practices of select
companies and governments, and provi de some tentative predictions for how these strategies and dynamics
may continue to evolve in the future. In summary:
● AI developers use a mix of patents, trade secrets, and open -source licensing agreements to protect
their AI -related IP.
● Many AI companies are pursuing what may seem like a counterintuitive IP strategy: aggressively
patenting AI technologies while sharing them freely. They experience competitive pressure to patent
in order to present the threat of a countersuit if another company sues them for IP infringement.
However, they also experience pressure to open -source their work in order to attract top talent an d
entice consumers to use their platforms.
● Governments broadly have two goals related to IP policy for AI that are at times in conflict with the
goals of researchers and/or companies: to ensure that AI -related inventions can be patented, and to
ensure that national -security -relevant AI inventions are restricted for government use and/or kept
secret.
● Significant uncertainty exists regarding how AI patentability, open -sourcing, and infringement
litigation will evolve in the future.
● There is an opportunity for patent pools to be used to facilitate pro -social behavior and ethical
norms among AI developers. Existing patent pools and practices by international standards
organisations represent possible models to replicate.

 Table of Contents
1. Introduction 1
2. Understanding Corporate AI Developer IP Strategies 2
Pressure to Patent 4
Pressure to Open -Source 5
A Hybrid Strategy: The Best of Both Worlds? 6
3. Governments’ Pursuit of AI Innovation and National Security Through IP Law 7
Box 1: The International Patent System and the World Intellectual Property Organization 7
Making AI Patentable 7
Box 2: Chall enges Comparing Patent Filings: The US and China 9
Regulating National -Security -Related IP Access 9
4. Three Scenarios for the Future of AI Intellectual Property Strategies 11
Path #1: Open Research Continu es 11
Path #2: Patent Lawsuits and Secrecy 12
Path #3: Expansion of Patent Pools 13
5. Conclusion 15
References 16

1 1. Introduction
Artificial intelligence (AI) is increasingly a focal point of competition between leading firms, and between
states. This paper focuses on a key, often under -examined component of the competitive strategies being
employed by both corporate AI developers and national governments: the protection of their intellectual
property.

Intellectual property (IP) is a broad and flexible concept, referring to creations of the mind that are eligible
for protection through law.2 Today, companies are using a mix of patents, trade secrets, and open -source
licensing agreements to protect their AI -related IP. Simultaneously, government patent offices, judiciaries,
and national security apparatuses are deciding which aspects of AI sho uld be patentable, and whether certain
inventions should be restricted for military purposes.

The IP policy choices that governments and corporations make can have profound implications for the
development trajectory of a technology. For example, in the 1990s, the biotechnology industry was
transformed after court decisions in the US enabled a broader range of biological compounds and processes
to be protected by patent law. This development spurred additional private investment, but critically also
allow ed companies to claim ownership over what previously would have been considered basic academic
research. This, in turn, encouraged higher levels of secrecy to protect valuable intellectual property.3 More
recently, one of the most prominent advances in bi otechnology, the CRISPR gene editing mechanism
(originally derived from a naturally occurring process in bacteria), has been subject to a protracted legal battle
over overlapping patent claims in the US.4

Changes in IP law and strategy may have a simila rly large impact on the trajectory of AI development. What
these impacts could be, however, have received little study. This paper aims to provide a preliminary analysis
of the goals and strategies of corporations and governments focused on AI development, and what the
implications of these strategies may be. First, we explain how corporate AI developers currently protect their
AI-related intellectual property and why they choose the methods that they utilize. Second, we describe how
governments use intelle ctual property law to pursue national goals related to AI. Finally, we describe three
plausible scenarios for how IP strategies in AI may evolve.

2 World International Property Organization
3 World International Property Organization
4 Cohen, 2019

2 2. Understanding Corporate AI Developer IP Strategies
Corporate AI developers face two key decisions arou nd how to protect their AI -related intellectual property:
whether or not to patent AI techniques and systems, and whether to open -source models or keep them
private as trade secrets.

A prevalent strategy among top AI developers today involves accumulatin g patents while simultaneously
sharing research with the open -source community.5 For example, Microsoft holds the most number of
machine learning patents in the US (see Figure 1),6 but is also an active participant in the open -source
community,7 sharing source code for machine learning methods and under certain circumstances providing
free licenses for their patents. Microsoft’s strategy is not an anomaly. Amazon, Google, IBM, Facebook,
Baidu, Tencent, and several other companies are prolific pat ent holders in AI (see Figures 1 and 2) while also
open -sourcing substantial portions of their systems and sharing their work at academic conferences such as
ICML and NeurIPS.8

Figure 1: Top Machine Learning Patent Holders, 2000 –2015

5 There are, however, exceptions. AI developers that specialize in work with national security and more hardware
centric applications rely on trade secrets more and open -source less. Our analysis does not explicitly investigate
trade secrets, as this inform ation is particularly difficult to obtain.
6 Webb et al., 2018
7 Microsoft
8 Amazon Web Services; Cai, 2018; IBM 2018; Baidu

3
Figure 2: Top Ne ural Network Patent Holders, 2000 –2015
Notably, and perhaps unintuitively, some of the largest software patent holders in the world (Google,
Amazon, and Facebook) signed an amicus brief to the Supreme Court in 2014 advocating that the court make
it more challenging to patent abstract software patents, which includes a considerable percentage of more
theoretical AI and ML patents :
“Abstract software patents have become a plague on computer -related industries. The
software industry developed and flourished without abstract patents before changes in the
Federal Circuit’s jurisprudence led to a flood of them. Far from promoting innovati on,
abstract software patents have impaire d it by granting exclusive rights over high level ideas
and thereby blocking others from undertaking the truly innovative task of developing specific
applications. ”9
These observations beg the question: why do so many of the top AI developers grow AI -related patent
portfolios while simultaneously sharing their research at academic conferences, open -sourcing machine
learning models, and advocating for the legal dissolution of many AI -related software patents?
In this section we argue that these corporate IP strategies help to manage a variety of objectives that
companies wish to pursue. AI developers apply for patents because of competitive pressures to do so. These
same developers also often open -source AI mod els in order to build their reputation, attract talent, and
incentivize customers to use paid products. AI developers can also use selective open -source licensing
agreements as a hybrid strategy, enabling companies to participate in the open -source communi ty while
maintaining the legal threat of their patents. We discuss these incentives for both patenting and open -
sourcing in turn, along with limitations and drawbacks of each approach.

9 American Bar Association, 2014

4 Pressure to Patent
Patents have several uses beyond simply enabling t he patent holder to sue for patent infringement. In the case
of big technology firms, there is a strong incentive to engage in “defensive patenting;” that is, patenting without
the intention to offensively litigate for infringement, but rather to present a credible threat of counter -lawsuit
to another company. Google senior patent counsel Suzanne Michel explained this defensive motivation for
building up a large patent portfolio during a 2013 symposium at American University:
“If everybody else is running a fter every trivial patent and building a big portfolio, you have
to too. ... It is called mutually assured destruction. That is the dynamic. … You cannot
opt out of the patent system and decide ‘I am an open -source company and anyone can use
my stuff.’ You have to have a massive portfolio of your own and that is really expensive
and it is what it is.”10
As the “mutually assured destruction” analogy makes clear, patent litigation is extremely costly for all
involved due to substantial legal fees and the sti gma for investors of working with a company whose products
are in legal purgatory. At the height of smartphone -related litigation in 2011, Apple and Google each spent
more money on patent litigation (primarily in suits and countersuits against one another) than they did on
research and development, a sum in the billions of dollars.11 In that same year, Google spent $12 billion
acquiring Motorola, which market analysts evaluated as being primarily for Motorola's substantial smartphone
patent portfolio.12 Perhaps if Google had acquired Motorola’s patents before litigation began with Apple, the
threat of a more substantial retaliation could have prevented the lawsuits.

This defensive rationale is also the stated reason for Google’s new AI patent filings in ma chine learning and
neural networks. When asked about Google’s new filings, spokespeople for Google and DeepMind stated
that they “hold patents defensively, not with the intent to start fights with others.”13 This dynamic can also
help explain why Google ha s advocated for narrowing the influence of software patents while simultaneously
growing its own patent portfolio; while Google may prefer a world without expensive software patent
litigation, its defensive patenting strategy is shaped by threats in the ex isting patent litigation regime.

Beyond defensive patenting, corporate AI developers could also be incentivized to hold patents in order to
gain leverage in other settings. For example, Google’s patent sharing arrangement with the Chinese tech giant
Tencent is paving the way for Google’s entry into the Chinese market.14 Google also allows other companies
to enter into its “Open Patent Pledge” and make use of patents in a pool as long as they commit not to
engage in patent litigation against Google.15

However, building and maintaining a large patent portfolio has its drawbacks. First, in the US, patent holders
must pay substantial upkeep fees to keep their patents active; up to thousands of dollars a year depending on

10 Quinn, 2013
11 Duhigg and Lohr, 2012
12 Hardy, 2011
13 Simonite, 2018
14 Cadell, 2018
15 Google

5 the size of the patent holding en tity and age of the patent (see Table 1).16 For large companies like Google —
which has more than 50,000 active patents —these costs can be in the tens of millions of dollars.17 Second, for
companies that place a high premium on secrecy, the disclosure requir ements and public nature of patent
filings may be onerous. Finally, some in the AI research community are philosophically opposed to the idea
of patenting AI concepts and techniques, particularly broad theoretical methods that are seen as mathematical
truths rather than human inventions.18 Companies that do choose to patent regardless may face pushback
from those opposed, which may have flow -on effects on their ability to, for example, attract and retain
research talent.

Patent Fee Schedu le
(per patent)19 Large Entity Fee Small Entity Fee Micro Entity Fee
Patent Maintenance fee
due 3.5 years after patent
is issued $1,600 $800 $400
Patent Maintenance fee
due at 7.5 years $3,600 $1,800 $900
Patent Maintenance fee
due at 11.5 years $7,400 $3,700 $1,800

Table 1: USPTO patent upkeep costs, per patent
Pressure to Open -Source
Incentives for corporate open -sourcing are also complex, typically extending beyond an altruistic or
philosophical belief in open science. For example, open -sourcing can be used to build a firm’s reputation,
generate goodwill among the research community, and encourage customers to use paid pr oducts.

Apple’s recent trend towards open -source and sharing more AI research demonstrates some of these
incentives at work. Apple has a notorious culture of secrecy, with numerous internal mechanisms in place to
prevent leaks and sequester information.20 Apple has benefited from this culture of secrecy in consumer
hardware design, a world where preventing leaks and copycat designs are critically important. However, this
culture was also a barrier for Apple to recruit top ML researchers, who typically str ongly value being able to
publish and share their work at conferences. Notably, several of Apple’s rival firms enabled researchers to do
so.21 In 2017, Apple changed its approach, launching a machine learning journal and enabling its researchers
to publicl y share findings at top ML conferences, including NeurIPS and ICML.22

16 US Patent and Trademark Office
17 Regalado, 2013
18 For an example of this view, see Mark Riedl’s quote in Simonite, 2018
19 US Patent and Trademark Office
20 Stone and Vance, 2009
21 Clark, 2015
22 Lewsing, 2017

6
Open -sourcing can also be used as a tool to generate more paying customers. For example, companies with
substantial cloud computing businesses often offer free machine learning tools to encourage customers to
design an application using the open -source tool. Then, thes e customers go on to pay for these compute -
intensive machine learning processes to be implemented on that same firm’s cloud service. In some cases, this
occurs through explicit lock -in. For example, Amazon’s image recognition software “Rekognition” is only
available on Amazon Web Services.23 In other cases, companies aim to retain customers through brand
loyalty; Google hopes that customers will use its ML open -source platform Tensorflow on Google Cloud,
though they could also use it on Amazon Web Services or Microsoft’s Azure. This proves to be a strong
incentive for open -sourcing given that cloud services appear to be incredibly lucrative. In 2018, Amazon,
Microsoft, and Google earned $25 billion, $23 billion, and $4 billion in revenue from their cloud bus inesses,
respectively.24 IBM’s $34 billion acquisition of open -source cloud computing provider RedHat (which
reportedly was also in acquisition talks with Google, Microsoft, and Amazon before selling to IBM25) and
Microsoft’s $11 billion JEDI cloud computi ng contract with the Pentagon26 further show how cloud business
is a priority for large AI developers.

The major downside of open -sourcing is the opportunity cost: open -sourcing in its purest form
means forgoing licensing fees from users. It also means sharing what would otherwise be
competitive secrets with rival companies. In the next section we discuss some ways that companies
manage to avoid these drawbacks.
A Hybrid Strategy: The Best of Both Worlds?
Despite the apparent conflict between building a large patent portfolio and participating in the open -source
community, selective licensing rights can en able a hybrid strategy where companies participate in the open -
source community while maintaining their patents. Companies can and do create selective licensing terms for
the use of their patents in open -source projects. These agreements can include defini ng permitted usage in a
way that allows some users to utilise the code while disincentivizing competitors to include that code in a
product.27

AI developers have also used selective open -source licensing agreements to achieve other ends. In 20 19,
Microsoft offered Azure cloud users access to a substantial portion of their patents as an incentive for users
to join the platform.28 Facebook attempted to add a licensing stipulation to its popular open -source platform
React, which would have caused users to retroactively lose their licenses if they ever engaged in patent
litigation against Facebook.29

23 Amazon Web Services
24 Griswold, 2019; Microsoft, 2018; Novet, 2018
25 Peterson, 2018
26 New York Times, 2019
27 E.g. The GNU Operating System GPL3 Open -Source license disincentiviz es commercial use.
28 Microsoft
29 The plan was a bandoned after developer backlash. See Wolff, 2017

7 3. Governments’ Pursuit of AI Innovation and National
Security Through IP Law
Governments around the world are grappling with how t o best take advantage of the recent wave of advances
in AI, with several nations releasing national plans on how their country intends to incentivize and capitalize
on AI innovation.30 These plans include ensuring that effective IP law regimes exist for AI and ML. This
tends to break down into two objectives: making AI patentable, and regulating access to national security
related IP. (For additional context on how national patent systems interact at the international level, see Box
1.)

Box 1: The Internat ional Patent System and the World Intellectual Property
Organization
Patent systems are primarily domestic in nature rather than international. Each country has its own patent
office and companies interested in seeking a patent for an invention must appl y separately in all
jurisdictions in which they wish to be awarded a patent.31 A patent awarded in one country cannot be used
to litigate infringement in another, though that patent does count as a form of “prior art” which can be
used to prevent the award of a patent for that invention in another country. The World Intellectual
Property Organization helps harmonize this process by assisting inventors to file their inventions in several
jurisdictions at once. However, the decision to award a patent will ult imately fall to individual countries.

While patent treaties such as “The Agreement on Trade -Related Aspects of Intellectual Property Rights”
(TRIPS32) have taken steps to harmonize intellectual property law across member nations, differences
persist at e very level of the process: what inventions are patentable, the level of scrutiny applied before a
patent is granted, how patents are enforced and reviewed, the length for which patents are valid, and
upkeep costs required to maintain the patent.

Making AI Patentable
A prominent element of several AI national plans is to ensure that AI -related inventions can receive patents
in a timely fashion. The goal of this policy is to encourage research and development (R&D) investment in AI
by rewarding tha t investment with a potentially lucrative patent. This strategy functions to both encourage
domestic companies to invest in AI -related research, and entice corporations choosing between different IP
systems to set up shop in their country rather than elsew here.

For example, US Patent and Trademark Office Chief Andrei Iancu recently expressed in a Senate hearing that
the US needs to make sure that its IP rules adequately protect and incentivize innovation in AI.33 In China’s

30 Dutton, 2018
31 World International Property Organization, 2017
32 Agreement on Trade -Related Aspects of Intellectual Property Rights, 1995
33 Simonite, 2018

8 state council plan, which decla red the nation’s intention to be the world leader in AI by 2030, one section
advised that policy makers in China must “[s]trengthen the protection of intellectual property in the field of
AI.”34 The European Patent Office also recently released specific gu idance on how to successfully patent
inventions in AI and machine learning,35 and Singapore is allowing AI patents to be “fast tracked” for review
through its patent system.36 Proposals for increasing Britain’s competitiveness in AI have also highlighted i ts
patent system’s challenges in protecting AI -related inventions as a liability.37

In some ways, the question of how to create patent protections for AI is not a new one. AI patents mostly fall
into the existing category of software patents, and countries have struggled for years to find regulatory
structures that incentivize innovation without allowing individual companies to control overly broad, abstract,
or obvious ideas.38 In fact, despite a push to allow more patenting and offer more stringe nt protections for
inventions, two recent major changes in IP law within the United States —the 2011 America Invents Act39
and the 2014 Supreme Court decision Alice Corp. v. CLS Bank International40—made it more difficult to claim
and enforce broad software patents. It will be difficult to change patenting rules in AI without also
implicating these existing decisions on software patents.

Furthermore, it is unclear whether expanding the range of patentable AI -relevant inventions would effectively
incentiviz e innovation. For one, AI and ML commercial activity has experienced massive growth and
international investment even while the patentability of innovations remains uncertain, suggesting that the
ability to patent AI is not necessary for innovation. In add ition, more patents increases the likelihood of
litigation, which could act to disincentivize innovation and slow down industry growth. This is particularly a
concern for some software patents due to their broad and abstract nature.41

34 Wang, 2018
35 European Patent Office, 2019
36 Spruson and Ferguson, 2019
37 Clark et al. 2019
38 Lee, 2013
39 Leahy -Smith America Invents Act, 2011
40 Alice Corp. v. CLS Bank International, 2014
41 Bessen, 2013

9 Box 2: Challenges Comparing Patent Filings: The US and China
China has outpaced the US in new patent applications related to the machine learning subfield of deep
learning.42 While some observers have interpreted this information as evidence of China’s fa st approaching
superiority over the US in new AI innovation, three key pieces of information about China and the US’s
patent systems should make us view these statistics in a different light.

First, patents in the US and China have very different standar ds, requirements and protections. The
majority of technology patents in China are filed as “utility model” patents , a category of patent in China
not extant in the US.43 Utility patents require a smaller inventive step, are subject to less rigorous
examination upon filing, and last only 10 years (in comparison to 21 years for American patents).
Consequently, this has led to filers taking advantage of lax inspections and review. In a 2018 report, the
Chinese state -owned Xinhua News Agency accused Chi na’s IP system of being characterized by “weak IP,
fake demands and some companies fervent on phony innovation,” according to a translation by Bloomberg
News.44 Chinese “invention patents” have requirements more similar to those in the US and last for 20
years rather than 10. However, they only comprise 23% of patent holdings in China.

Second, in China, there are strong government financial incentives for researchers to file patents, regardless
of the underlying patents’ merit.45 This is particularly true in AI, where the Chinese central government has
committed substantial public funds to encourage inventions.

Finally, it is important to look at the “discard rate” for Chinese patents —i.e. how quickly a patent is
discarded by its holder —and how they comp are to patents in the United States. A substantial percentage of
Chinese patent holders allow their patents to expire before the patent’s lifespan is complete —61% for
utility patents and 37% for invention patents —in comparison to a discard rate of 15% in t he US.46 When
viewed in context with the previous points, it becomes clear that there are many Chinese researchers who
file for AI patents in order to claim government incentives without genuinely believing their invention is
notable.47

Regulating National -Security -Related IP Access
Countries’ patent regimes for AI are not only shaped by economic motivations, but also by national security
interests. National governments typically pursue two primary interests on this front: to ensure that their own
national security apparatuses have access to state -of-the-art technology, and to withhold that access from
perceived rivals.

On the goal of ensuring access, a US court decision in 2015 held that the federal government can use
patented inventio ns without the permission of the patent holder and cannot be forced to cease usage of a

42 Huang, 2018
43 Chen, 2018
44 Ibid.
45 Ibid.
46 Ibid.
47 Ibid.

10 patented invention; the only remedy is for the patent holder to request damages assessed at market rate
(which amounts to compulsory licensing).48 This means if a paten t holder does not wish for their patent to be
used by the government (e.g. a patent that has potential surveillance applications) their only recourse after
suing for infringement is to force the government to pay for a reasonably costed license. This proce ss is quite
distinct from what happens when a private entity infringes on a patent, where the private entity can be
enjoined to cease usage of the patent or be assigned additional damages.

Additionally, over the last few years the Chinese government has passed broad laws on national security and
cybersecurity with implications for access to intellectual property.49 One of these laws mandates that network
operators (broadly defined) provide “technical support and assistance” to national security relevant
government offices.50 The exact legal authority of the Chinese government to force cooperation with Chinese
companies is difficult to know. The Center for a New American Security’s Ashley Feng reports that “U.S.
government officials, including at the FBI, interpreted this vagu e language to mean that all Chinese
companies, including Huawei, are subject to the direct orders of the Chinese government.”51 However, The
Diplomat’s Jack Wagner reports that the main purpose of the law is to mandate additional data localization
and stor age on Chinese mainland servers and to set standards around cybersecurity.52 Additional concerns
around Chinese corporations acting as extensions of the state are conceivable, but more speculative in nature.

On the goal of withholding access from perceiv ed rivals, the US and the UK have long had government
statutes that empower their patent offices to prevent public disclosure and bar the award of patents that have
national security implications, regardless of their other merits.53 In the US in 2018, 5,79 2 patent applications
were covered within these so -called “secrecy orders,” higher than at any point since the Cold War.54

48 Astornet Technologies Inc. v. BAE Systems, Inc, 2015
49 Feng, 2019
50 Ibid.
51 Ibid.
52 Wag ner, 2017
53 Schulz, 2013; Marks, 2010
54 Patent and Trademark Office, 2018

11 4. Three Scenarios for the Future of AI Intellectual Property
Strategies
Given the observed corporate AI developer IP strategies and growing government interest in IP law as a lever
for influencing AI development, how could the dynamics of AI intellectual property protection evolve? What
would these dynamics then mean for the f uture of the AI industry, and in particular, on the competitive
strategies employed by firms and states?

Here we present three plausible scenarios which focus on how the IP strategies of corporate AI developers
could evolve in the near future:
(1) Open Rese arch Continues: The status quo persists: open research alongside defensive patents
remains the norm within the ML industry.
(2) Patent Lawsuits and Secrecy: Offensive patent litigation breaks out within the ML community and
prompts additional secrecy among de velopers.
(3) Expansion of Patent Pools: In response to the threat of litigation, AI developers enter into additional
patent sharing agreements.
In the following section, we describe each scenario and present evidence to support its plausibleness. These
scenarios are intended to be illustrative rather than predictive; indeed, there are several alternative and hybrid
scenarios that could warrant further investigation as well.
Path #1: Open Research Continues
Scenario:
Each of the major AI developers weighs the costs of engaging in patent litigation against their competitors,
and decides that the threat of a countersuit and the symmetrical legal costs make litigation a poor choice.
Each developer continues to file for pa tents in order to maintain a credible response, but analogous to a
nuclear standoff, this “mutually assured destruction” framework holds.

This equilibrium is bolstered by competition for researchers who want to work at companies that prioritize
openness and cooperation with their peers. Some patent trolls —firms that profit from licensing and litigating
on patents without producing any products of their own —may gain control of patents and engage in
litigation without fear of reprisal, but these lawsuits r emain relatively insignificant.

Evidence in Favor:
● Despite the recent flurry of activity on the subject, the machine learning community currently shows
little sign of changing its open and non -litigious culture. Academics and individuals around the wor ld
can use cutting edge machine learning techniques from open -source platforms free of charge. There
has been some litigation over trade secret theft in autonomous vehicles (Waymo vs. Uber,55 Baidu vs.
JingChi56), but no large -scale patent litigation over broad concepts in machine learning.

55 Korosec, 2018
56 Borak, 2017

12 ● Current patent rules in the US make AI -related software patents less threatening for the purposes of
litigation than they were before the Supreme Court’s 2014 decision in Alice Corp. v. CLS Bank
International , which mad e software patents more likely to be classified as “abstract ideas” and thus
unpatentable. Given this decision, it is likely that many existing AI patents, particularly those
covering broad mathematical concepts, will be rejected during litigation or at th e Patent Trial and
Appeal Board.
● While some AI -related patents in the US are being granted, the majority are not. Data from patent
filings shows that in recent years, over 90% of AI -related patent applications in the US were initially
rejected, many for being merely “abstract ideas” that are not eligible for patentability.57 By
comparison, the overall rejection rate for patents in the US is 48%.58 Fewer AI -related patents mean
fewer opportunities for companies to litigate over infringement, thus bolsterin g the incentives for
open research.
● If large tech companies choose to litigate with one another over AI -related IP, they do not just have
to contend with a defendant's AI -related patents, but also with all of the other patents that would
likely be used in a countersuit.59 Google, Microsoft, Amazon, and IBM have expansive business
operations across several verticals; this makes patent aggression with other large companies less
attractive.
Path #2: Patent Lawsuits and Secrecy
Scenario:
Major patent litigations break out between AI developers. While the previous open equilibrium may be
preferable for the collective interest of major private AI developers, it may only take one large company
deciding it is in its interest to pursue active litigation for this state of affairs to deteriorate. For instance, if
IBM, with its trove of AI -related patents and its struggling core enterprise business,60 chose to litigate against
its rivals, it could provoke additional litigation. So -called “patent trolls” (entiti es that accumulate patents while
not producing products of their own) could also threaten to disrupt the mutually assured destruction
equilibrium and engage in lawsuits without fear of reprisal. This litigation could bleed over and affect
academic research . While the EU has a research exemption that protects academic use from being deemed
infringement, the US has no such exemption, and university researchers could thus potentially find
themselves on the wrong end of litigation.61

As litigation escalates, t here is a substantial incentive for companies to ensure that potentially patentable
inventions are kept secret from rivals. In a 2017 paper, Nick Bostrom discusses how the pursuit of patents in
AI could cause companies to share research less often in order to prevent other entities from using

57 Decker, 2019
58 Carley et al. 2015
59 For example, if Google were to engage in litigation against another company over machine learning patents,
they would have to contend with its opposition’s patents in E -commerce, search, drones, self -driving cars,
smartphone hardware and software and telecommunications, because each of these are areas in which Google
operates in and is thus capable of infringem ent.
60 Imbert, 2018
61 Miller, 2002

13 intermediary research to obtain a patent first.62 Public research could also be used as evidence in litigation to
prove infringement (e.g. releasing a model using a patented method), further dissuading companies in a
litigious environment from engaging in open -source communities.

Evidence in Favor:
● As discussed previously, governments in the US, China, EU, and elsewhere are pushing to broaden
the scope of patentable material in AI and encourage filings. If these effo rts translate into more
ostensibly defensible AI patents being filed successfully, this could increase incentives for litigation
between patent -holders.
● While Google reported that it is holding new AI patents on a defensive basis, other developers have
been non -committal about their future intention to litigate with their patents.63 When asked about
the issue, a Facebook spokesperson said that “its filings shouldn’t be read to indicate current or
future plans.’64 IBM’s patent counsel released a statement that said its large AI patent portfolio
“reflects its commitment to fundamental research.”65
Path #3: Expansion of Patent Pools
Scenario:
AI developers agree to cross -license their patents to one another in a “ pool” to reduce the risk of litigation.
We previously discussed patent sharing agreements in the context of companies like Google and Tencent
using these deals to gain market entry into new countries. However, patent sharing agreements need not only
be bet ween two actors. There are several historical examples of large technology companies pooling their
patents to protect against litigation and create advantageous licensing dynamics. The DVD6C Licensing
Group was comprised of eight of the most high profile p atent holders in DVD technology (including
Samsung and Toshiba, among others).66 Third -party manufacturers interested in using their technology could
approach the group as a one -stop clearinghouse in order to obtain licenses instead of approaching each
mem ber individually. Similar arrangements would enable companies within the pool to share trade secrets and
research and development, though too much coordination would come under the ire of antitrust
enforcement.

In this scenario, patent pools could be use d not only to curtail litigation risks, but also to limit or promote
certain applications of AI. As previously mentioned, companies are currently able to individually place
stipulations on patent licensing. If a company decided that it wanted to refuse to license its patents to
manufacturers of, for example, autonomous weapons, on moral grounds, it could certainly do so.

Extending this to a patent pool, corporate AI developers could group together to share intellectual property
and establish shared standa rds for how they wish to have their intellectual property used, perhaps based on
certain ethical principles. These standards could then be implemented via, for example, selective licensing

62 Bostrom, 2017
63 Simonite, 2018
64 Ibid.
65 Ibid.
66 DVS6C Licensing Group

14 agreements which restrict use of the pool’s IP to actors who commit to abiding by those standards.
Membership of the patent pool could also be made conditional on abiding by the pool’s standards. Indeed,
patent pools used for licensing agreements by standard setting bodies such as the International Standards
Organization are precedent for similar structures’ success. It is worth noting, however, that such multilateral
“refusals to deal” would need to be implemented with caution and appropriate due diligence in order to avoid
potential infringements of antitrust law.67

Evidence in Favor:
● Existing software patent pools show the demand for and utility of this type of coordination. Google
and several other tech companies participate in the Open Invention Network and Android
Networked Cross -License Agreement in order to protec t Linux and Android developers from
infringement litigation.68 As an alternative, the MPEG LA group is an example of a software patent
pool that operates as a profitable licensing association for its members.69 Facebook and Google’s
aforementioned existing patent non -aggression agreements could also be a path towards greater
cooperation.
● Increasing returns to scale in AI (meaning that more data improves AI systems and platforms, which
attracts more users, which in turn generates more data) could increase the odds of industry
centralization. A smaller number of relevant actors improves the feasibility of this kind of
coordination.

67 Department of Justice
68 Open Invention Network, 2017
69 MPEG LA

15 5. Conclusion
AI is poised to be a critically impactful technology, and its development will be deeply affected by existing
social and legal institutions. This paper has preliminarily explored an under -exami ned aspect of this
infrastructure: the legal rules, norms, and strategies governing AI -related intellectual property.

Leading corporate AI developers today have employed a dynamic and at times unintuitive IP strategy that
allows them to respond to the sh ifting competitive landscape surrounding them. Governments are also
seeking to shape IP systems that incentivize innovation around AI while also protecting their national security
interests. How each actor balances its varying objectives in relation to AI and how they choose to wield IP
strategies to achieve these objectives remains to be seen.

This preliminary analysis scratches the surface of what may be an important element of the strategic
landscape shaping competition and cooperation among AI firms a nd prominent national governments.
Further investigation in this direction could be fruitful for better understanding the goals and strategies of
actors seeking to protect AI -related intellectual property, and how these strategies have flow -on implications
on the competitive dynamics that arise between AI developers. This, in turn, could shed light on questions
related to the prospects for cooperation between these actors to achieve prosocial outcomes with respect to
AI ethics and safety, and more broadly, the approp riate governance of AI going forward.

16 References
Agreement on Trade -Related Aspects of Intellectual Property Rights. 1 Jan. 1995,
https://www.wto.org/english/docs\_e/legal \_e/27 -trips.pdf . Accessed 12 Apr. 2019.

Alice Corp v. CLS Bank International. 19 June 2014, https://www.supr emecourt.gov/opinions/13pdf/13 -
298\_7lh8.pdf . Accessed 12 Apr. 2019.

Amazon Web Services. “Amazon Rekognition.”
https://aws.amazon.com/rekognition/ . Accessed 7 Nov. 2019.

Amazon Web Services. “Ope n Source at AWS.” https://aws.amazon.com/opensource/ . Accessed 12 Apr.
2019.

American Bar Association . “Brief of Google Inc., et al as Amici Curiae in support of respondents”
https://www.americanbar.org/content/dam/aba/publications/supreme\_court\_preview/b riefs-v3/13 -
298\_resp\_amcu\_google -etal.authcheckdam.pdf Accessed 5 April 2020

Apple. “Legal Contact/Patent.” https://www.apple.com/legal/contact/patent.html . Accessed 12 Apr. 2019.

Astornet Technologies Inc. v. BAE Systems, Inc. 19 Sept. 2015,
http://www.cafc.uscourts.gov/sites/default/files/opinions -orders/14 -1854. Opinion.9 -15-2015.1.PDF .
Accessed 12 Apr. 2019.

Baidu. “Github/OpenEdge.” https://github.com/baidu/openedge . Accessed 12 Apr. 2019.

Bessen , James. “The patent troll crisis is really a software patent crisis.” Washington Post , 3 Sept. 2013,
https://www.washi ngtonpost.com/news/the -switch/wp/2013/09/03/the -patent -troll-crisis -is-really -a-
software -patent -crisis/ . Accessed 12 Apr. 2019.

Borak, Masha. “Baidu sues its former SVP for stealing self -driving trade -secrets and using them in his US -
based startup.” TechNode , 22 Dec. 2017, https://technode.com/2017/12/22/baidu -sues-former -svp-stealing -
self-driving -trade -secrets -using -us-based -startup/ . Accessed 12 Apr. 2019.

Bostrom, Nick. “Strategic Implications of Openness in AI Development.” Global Policy , 2017,
https://nickbostrom.com/papers/openness.pdf . Accessed 12 Apr. 2 019.

Cadell, Cate. “Google announces patent agreement with Tencent amid China push.” Reuters , 18 Jan. 2018,
https://www.reuters.com/article/us -china -google -tencent/google -announces -patent -agreement -with-
tencent -amid -china -push-idUSKBN1F80DF . Accessed 12 Apr. 2019.

Cai, Fangyu. “Tencent Open -Sources its Massive Multi -Labeled Image Dataset.” 10 Sept. 2018,
https://medium.com/syncedreview/tencent -open -sources -its-massive -multi -labeled -image -dataset -
7b0b3dd5373f . Accessed 12 Apr. 2019.

Carley, Michael, Deepak Hegde, and Alan Marco. “What is the probability of receiving a US patent?” Yale
Journal of Law and Technology , 2015.

Chen, Lulu YiLun. “China Claims More Patents Than Any Country –Most Aare Worthless.” Bloomberg , 26
Sept. 2018,

17 https://www.bloomberg.com/news/articles/2018 -09-26/china -claims -more -patents -than-any-country -most -
are-worthless . Accessed 12 April 2019.

Clark, Alex, Ella Hollowood, Imogen Harper, and Alexi Mostrous. “AI in the UK.” Tortoise Media , 4 Dec.
2019, https://members.tortoisemedia.com/2 019/12/04/ai -in-the-uk/content.html . Accessed 16 Jan. 2020.

Clark, Jack. “Apple’s Deep Learning Curve.” Bloomberg , 29 Oct. 2015,
https: //www.bloomberg.com/news/articles/2015 -10-29/apple -s-secrecy -hurts -its-ai-software -development.
Accessed 12 Apr. 2019.

Cohen, Jon. “CRISPR Patent Fight Revived.” Science , 5 July 2019,
https://science.sciencemag.org/content/365/6448/15.2 . Accessed 22 Sept. 2019.

Conger, Kate, David E. Sanger, and Scott Shane. “Microsoft Wins Pentagon’s $10 Billion JEDI Contract,
Thwarting Amazon.” New York Times , 25 Oct. 2019,
https://www.nytimes.com/2019/10/25/technology/dod -jedi-contract.html . Accessed 7 Nov. 2019.

Decker, Susan. “Who Profits from AI? It’s Getting Harder to Find Out.” Bloomberg , 23 Jan. 2019,
https://www.bloomberg.com/news/articles/2019 -01-23/who -profits -from -ai-uncertain -as-u-s-patent -office -
gets-pickier . Acce ssed 26 Aug. 2019.

Department of Justice. “The Strategic Use of Licensing: Unilateral Refusals to License PatentsDeal.”
https://www.justice.gov/atr/chapter -1-strategic -use-licensing -unilateral -refusals -license -patents . Accessed 12
Apr. 2019.

Dignan, Larry. “All of Amazon’s 2017 operating income comes from AWS.” ZDNet , 1 Feb. 2018,
https://www.zdnet.com/article/all -of-amazons -2017-operating -income -comes -from -aws/. Accessed 12 Aug.
2019.

Duhigg, Charles and Steve Lohr. “The Patent, Used as a Sword.” New York Times , 7 Oct. 2012,
https://www.nytimes.com/2012/10/08/technology/patent -wars-among -tech-giants -can-stifle-
competition.html . Accessed 12 Apr. 2019.

Dutton, Tim. “An Overview of National AI Strategies.” 28 Jun. 2018, https://medium.com/politics -ai/an -
overview -of-national -ai-strategies -2a70ec6edfd . Accessed 12 Apr. 2019.

DVD6C Licensing Group. “DVD6C Patent Pool.” http://www.dvd6cla.com/ . Accessed 12 Apr. 2019.

European Patent Office. “Guidelines for Examination.” https://www.e po.org/law -practice/legal -
texts/html/guidelines2018/e/g\_ii\_3\_3\_1.htm . Accessed 12 Apr. 2019.

Feng, Ashley. “We Can’t Tell if Chinese Firms Work for the Party.” CNAS , 7 Feb. 2019,
https://www.cnas.org/publications/commentary/we -cant-tell-if-chinese -firms -work -for-the-party . Accessed
12 Apr. 2019.

Ford, Martin R. Architects of Intelligence: the truth about AI from the people building it . Pack t, 2018.

Google. “Patents in the Service of Open Source.”
https://www.google.com/patents/opnpledge/ . Accessed 12 Apr. 2019.

18 Google. “Introducing PAX: the Android Networked Cross -License Agreement.” 3 Apr. 2017,
https://blog.google/topics/public -policy/introducing -pax-android -networked -cross -license -agreement/ .
Accessed 7 Nov. 2019.

Griswold, Alison. “Amazon Web Services brought in more money than McDonald’s in 2018.” Quartz , 1 Feb.
2019,
https://qz.com/1539546/amazon -web-services -brought -in-more -money -than-mcdonalds -in-2018/ .
Accessed 7 Nov. 2019.

Gurman, Mark. “Apple Warns Employees to Stop Leaking Information to the Media.” Bloomberg , 13 Apr.
2013,
https://www.bloomberg.com/news/articles/2018 -04-13/apple -warns -employees -to-stop-leaking -
information -to-media . Accessed 12 Apr. 2 019.

Hardy, Quentin. “Google Buys Motorola For Patent Parts.” Forbes , 15 Aug. 2011,
https://www.forbes.com/sites/quentinhardy/2011/08/15/google -buys-motorola -for-patent -
parts/#17555b562fff . Accessed 12 Apr. 2019 .

Huang, Echo. “China has shot far ahead of the US on deep -earning patents.” Quartz , 2 Mar. 2018,
https://qz.com/1217798/china -has-shot-far-ahead -of-the-us-on-ai-patents/ . Accessed 26 Aug. 2019.

IBM. “IBM and NVIDIA Collaborate to Expand Open Source Machine Learning Tools for Data Scientists.”
10 Oct. 2018, https://newsroom.ibm.com/2018 -10-10-IBM-and-NVIDIA -Collaborate -to-Expand -Open -
Source -Machine -Learning -Tools -for-Data-Scientists . Accessed 12 Apr. 2019.

Imbert, Fred. “Sell IBM shares because its profits are in an ‘ irreversible structural decline,’ analyst says.”
CNBC , 4 Oct. 2018,
https://www.cnbc.com/2018/10/04/ibm -profits -are-in-an-irreversible -structural -decline -analyst -says.html .
Accessed 22 Sept. 2019.

Korosec, Kirsten. “Waymo V. Uber: What You Need to Know About the High -Stakes Self -Driving Tech
Trial.” Fortune , 5 Feb. 2018, http://fortune.com/2018/02/05/waymo -v-uber-what-you-need-to-know -about -
the-high-stakes -self-driving -tech-trial/ . Accessed 12 Apr. 2019.

Leahy -Smith America Invents Act. 16 Sept. 2011,
https://www.uspto.gov/sites/default/files/aia\_implementation/20110916 -pub-l112-29.pdf . Accessed 12
Apr. 2019.

Lee, Timothy B. “Here’s why economists hate software patents.” Washington Post , 31 July 2013,
https://www.washingtonpost.com/news/the -switch/wp/2013/07/31/heres -why-economists -hate-software -
patents/ . Accessed 7 Nov. 2019.

Leswing, Kif. “Ap ple is now publishing a ‘journal’ with its cutting -edge machine learning research.” Business
Insider , 19 July 2017, https://www.businessinsider.com/apple -announces -apple -machine -learning -journal -
2017-7. Accessed 12 Apr. 2019.

Marks, Paul. “UK keeps three times as many patents secret as the US.” New Scientist , 23 Mar. 2010,
https://www.newscientist.com/article/dn18691 -uk-keeps -three -times -as-many -patents -secret -as-the-us/.
Accessed 12 Apr. 2019.

Microsoft. “Annual Report 2018.” 16 Oct. 2018.
https://www.microsoft.com/en -us/annualreports/ar2018/annualreport . Accessed 7 Nov. 2019.

19
Microsoft. “Azure IP Advantage.”
https://azur e.microsoft.com/en -us/overview/azure -ip-advantage/ . Accessed 12 Apr. 2019.

Microsoft. “Discover more options with open source on Azure.” https://azure.microsoft.com/en -
us/overview/choose -azure -opensource/ .

Microsoft. “Microsoft to acquire Github for $7.5 Billion.” 4 June 2018,
https://news.microsoft.com/2018/06/04/microsoft -to-acquire -github -for-7-5-billion/ . Accessed 12 Apr.
2019.

Miller, Jennifer. “Sealing the Coffin on the E xperimental Use Exception.” 2002,
https://scholarship.law.duke.edu/cgi/viewcontent.cgi?article=1081&context=dltr . Accessed 12 Apr. 2019.

MPEG LA. https://www.mpegla.com/ . Accessed 7 Nov. 2019.

Novet, Jordan. “Google says its cloud now brings in $1 billion per quarter.” CNBC , 1 Feb. 2018,
https://www.cnbc.com/2018/02/01/google -cloud -revenue -passes -1-billion -per-quarter.html . Accessed 7
Nov. 2019.

Open Invention Network. https://www.openinventionnetwork.com/ . Accessed 7 Nov. 2019.

Patent and Trademark Office. “Invention Secrecy Activity.”
https://fas.org/sgp/othergov/invention/stats.html . Accessed 12 Apr. 2019.

Peterson, Becky. “IBM's $34 billion Red Hat acquisition came after deal talks with Microsoft, Google, and
Amazon, sources say.” Business Insider , 16 Dec. 2018,
https://www.businessinsider.com/red -hat-deal-talks-with-amazon -microsoft -google -before -34-billion -ibm-
acquisition -2018-12. Accessed 12 Apr. 2019.

Quinn, Gene. “Google: We Don’t Sell to Patent Trolls.” IPWatchdog , 2 May 2013,
http://www.ipwatchdog.com/2013/05/02/google -we-dont-sell-to-patent -trolls/id=39927/ . Accessed 12
Apr. 2019.

Regalado, Anthony. “Google’s Growing Patent Stockpile.” MIT Technology Review , 29 Nov 2013,
https://www.technologyreview.com/s/521946/googles -growing -patent -stockpile/ . Accessed 26 Aug. 2019.

Schulz, G.W. “Government Secrecy Order on Patents Have Stifled More Than 5,000 Inventions.” Wired , 16
Apr. 2013, https://www.wired.com/2013/04/gov -secrecy -orders -on-patents/ . Accessed 12 Apr. 2019.

Simonite, Tom. “Despite Pledging Openness, Companies Rush to Patent AI Tech.” Wired , 31 July 2018,
https://www.wired.com/story/d espite -pledging -openness -companies -rush-to-patent -ai-tech/ . Accessed 12
Apr. 2019.

Spruson and Ferguson, “Singapore Update: IPOS Grants First Accelerated Patent Under Fintech Fast Track
Initiative.” 14 Dec. 2018, https://www.spruson.com/fintech/singapore -update -ipos-grants -first-accelerated -
patent -fintech -fast-track -initiative/ . Accessed 12 Apr. 2019.

Stone, Brad and Ashlee Vance. “Apple’s Obsession With Secrecy Grows Stronger.” New York Times , 22 June
2009, https://www.nytimes.com/2009/06/23/technology/23apple.html .

TensorFlow. “An end -to-end open -source machine learning platform.”

20 https://www.tensorflow.org/ . Accessed 12 Apr. 2019.

United States Patent and Trademark Office. “USPTO Fee Schedule.”
https://www.uspto.gov/learning -and-resources/fees -and-payment/uspto -fee-
schedule#Patent%20Maintenance%20Fee . Accessed 26 Aug. 2019.

Wagner, Jack. “China’s Cybersecurity Law: What You Need to Know.” The Diplomat , 1 June 2017,
https://thediplomat.com/2017/06/chinas -cybersecurity -law-what-you-need-to-know/ .

Wang, Qian. “Global Perspectives: AI and IP Policies Around the World.” Artificial Intelligence: Intellectual
Property Considerations , 31 Jan. 2018. Presentation.

Webb, Michael, Nicholas Bloom, Nick Short, and Josh Lerner. “Some Facts of High Tech Patenti ng.”
Harvard Business School , July 2018, https://www.hbs.edu/faculty/Publication%20Files/19 -014\_cf3bccb7 -1ab7-
4f43-b822 -2d7c9b04a2c0.pdf . Acc essed 12 Apr. 2019.

Wolff, Adam. “Relicensing React, Jest, Flow, and Immutable.js.” Facebook , 22 Sept. 2017,
https://code.fb.com/web/relicensing -react-jest-flow-and-immutable -js/. Accessed 12 Apr. 2019.

World Economic Forum. “Artificial Intelligence Collides with Patent Law.” Apr. 2018,
http://www3.weforum.org/docs/ WEF\_48540\_WP\_End\_of\_Innovation\_Protecting\_Patent\_Law.pdf .
Accessed 26 Aug. 2019.

World International Property Organization. “Protecting yYour Inventions Abroad: Frequently Asked
Questions About the Patent Cooperation Treaty.” Oct. 2017, https://www.wipo.int/pct/en/faqs/faqs.html .
Accessed 12 Apr. 2019.

World International Property Organization. “What is Intellectual Property?” https://www.wipo.int/about -
ip/en/ . Accessed 7 Nov. 2019.


Oﬀ-policy Monte Carlo agents with variable behaviour policies

O-policy Monte Carlo agents with variable
behaviour policies
Stuart Armstrong1;2
1The Future of Humanity Institute
Faculty of Philosophy
University of Oxford
Suite 1, Littlegate House
16/17 St Ebbes Street
Oxford, OX1 1PT UK
stuart.armstrong@philosophy.ox.ac.uk
2Machine Intelligence Research Institute
2030 Addison Street #300
Berkeley, CA 94704
Abstract. This paper looks at the convergence property of o-policy
Monte Carlo agents with variable behaviour policies. It presents results
about convergence and lack of convergence. Even if the agent generates
every possible episode history innitely often, the algorithm can fail to
converge on the correct Q-values. On the other hand, it can converge
on the correct Q-values under certain conditions. For instance, if, during
then-th episode, the agent has an independent probability of 1 =log(n)
of following the original policy at any given state, then it will converge
on the right Q-values for that policy.
Keywords: Monte Carlo agents, o-policy, model-free, reinforcement
learning
1 O-policy Monte Carlo
The Monte Carlo agent is a model-free reinforcement learning agent [3]. These
operate when the environment is a Markov decision process (MDP). In an MDP,
the next observation depends only on the current observation { the state { and
the current action. The full set of state action pairs is designated by SA .
In each state, an agent chooses its actions according to a (possibly stochas-
tic) policy , that is assumed to be Markov. Monte Carlo agents operate by
computing the Q-values of a policy for each state-action pair ( s;a), which is
the expected return if the agent choose ainsand subsequently follows . It
is episodic, exploring the same MDP repeatedly to compute Q-values. Episodic
MDP's have initial state (where the agent starts an episode) and terminal states
(where the agent ends an episode). This paper focuses on computing Q-values,
not on updating policy choice in consequence.
Assumption 1 In the following, these are always assumed, required for conver-
gence results:
(a) The MDP is nite,
(b) Whatever policy the agent uses, its expected time until reaching a terminal
state is nite.
(c) The rewards after each step have nite expectation (consequently the total
reward has nite expectation).
If an agent is following one policy ( e.g.,) but wishes to compute the Q-
values of another ( e.g.,), it can use the o-policy Monte Carlo algorithm [3].
In this case is the behaviour policy , andis the evaluation policy .
The algorithm requires that for all state-action pairs ( s;a):
(ajs)>0)(ajs)>0: (1)
Because of this requirement, (ajs) is zero whenever (ajs) is, and the ratio
(ajs)=(ajs) can be dened as 0 in these cases. The full algorithm is then given
in box 1 { there are two variants, ordinary importance sampling usingN(s;a)
(the number of episodes in which the agent has encountered the state-action pair
(s;a)) as the denominator, and weighted importance sampling usingW(s;a) (the
sum of the weights) instead [4, 1].
1. For alls2Sanda2As, initialiseN(s;a),W(s;a) andWR(s;a) to zero.
2. For alln1:
(a) Generate an episode history hnby following policy .
(b) The episode data will consist of the state, action chosen in the state, the
immediate reward experienced by the agent.
(c) For each state-action pair ( s;a) appearing in the episode history:
i. Lettbe the rst appearance of ( s;a) in the history.
ii. LetRn(s;a) be the total subsequent reward.
iii. Dene
wn(s;a) =Y
k>t(akjsk)
(akjsk): (2)
iv.Nn(s;a) =Nn1(s;a) + 1.
v.Wn(s;a) =Wn1(s;a) +wn(s;a).
vi.WRn(s;a) =WRn1(s;a) +wn(s;a)R(s;a).
vii.EitherQn(s;a) =WRn(s;a)
Nn(s;a),orQn(s;a) =WRn(s;a)
Wn(s;a).
(d) Discard the episode data.
Table 1. O-policy Monte Carlo algorithm
Note thatW(s;a) andN(s;a) have the same expectation: let H(s;a)be the
set of possible histories in the MDP subsequent to ( s;a). Forh2H (s;a), relabel
the indexes so that hstarts in step zero at ( s;a). Useandto denote the
probabilities of certain events conditional on the agent following those policies,
and Eand Esimilarly. Then if In(s;a) denotes the event that ( s;a) appears
in then-the episode,
E(w(s;a)jIn(s;a)) =X
h2H(s;a)(hjIn(s;a))Y
k>0(hakjhsk)
(hakjhsk)
=X
h2H(s;a)(hjIn(s;a))(hjIn(s;a))
(hjIn(s;a))
=X
h2H(s;a)(hjIn(s;a))
=(H(s;a)jIn(s;a)) = 1:(3)
wherehakdenotes the k-th action of history h, andhskdenotes the k-th state
of historyh.
Then simply note that the value of Nn(s;a) is simply the sumPn
j=1Ij
(s;a),
to see that Nn(s;a) andWn(s;a) =Pn
j=1Ij
(s;a)wn(s;a) have same expectation.
Ifis xed, then by sampling only those histories including ( s;a), the strong
law of large numbers implies that the ratio of Nn(s;a) andWn(s;a) converges
to 1 almost surely.
The same argument as in equation (3) shows that
E(wn(s;a)Rn(s;a)jIn(s;a)) =X
h2H(s;a)Rh(s;a)(hjIn(s;a))Y
k>0(hakjhsk)
(hakjhsk)
= E(R(s;a)jIn(s;a));
(4)
whereRh(s;a) is the reward along the history hsubsequent to the rst ( s;a).
Thus the expected weighted reward from following policy , is the expected
reward from following policy .
Assume that agents following would almost surely explore every state-
action pair innitely often (equivalently, that Nn(s;a)!1 almost surely). Then
the convergence of ordinary importance sampling is simply a consequence of the
law of large numbers applied to every episode that visits ( s;a). Convergence of
weighted importance sampling is a consequence of Corollary 1 below.
2 Varying the behaviour policy
The previous proof, however, assumes that policy is xed. But what if it varies
from episode to episode?
Ifand0are two policies, any p2[0;1] denes the mixed policy:
(1p)+p0:
This is the policy that, in each state, independently chooses whether to follow
, with probability 1 p, and0, with probability p. Because the decisions are
independent, the mixed policy is Markov if and0are.
A general form for all behaviour policies is then:
Proposition 1. Ifandobey the restriction in equation (1), then there exists
a2[0;1)and a policy 0such that
= (1)+0:
Proof. Letbe the maximum across SA of the ratio (ajs)=(ajs), in all
cases where (ajs)6= 0. Then dene (1 ) = 1=. Since(ajs)=(ajs),
(1) = 1=(ajs)=(ajs) and hence (ajs)(1)(ajs).
Then dene 0as
0(ajs) =1

(ajs)(1)(ajs)
:
To check that this quantity is less than 1, note that (ajs) = 1P
b6=a(bjs)
1P
b6=a(1)(bjs) hence that (ajs)(1)(ajs)1P
b(1)(bjs)
1(1).
Then, by construction, (ajs) = (1)(ajs) + ()0(ajs). u t
Fornbeing the episode number, dene nby
n= (1n)+n0
n; (5)
for some0
nandn2[0;1). For convenience, further dene the set S0S, which
is the set of states ssuch thatn(s)6=(s).
Then neither o-policy Monte Carlo algorithm need converge:
Theorem 1. Neither o-policy Monte Carlo algorithm following policy nneed
converge to the correct Q-values for , even if the agent generates every possible
episode history innitely often.
The proof is given in Section 3. But that convergence failure need not happen
for all such n. A sucient condition for convergence is:
Theorem 2. Letn= 1=(1n). Assume that nis eventually non-decreasing,
and that there exists a >0such that, for large enough n,(n)p
log(n)<n1.
Assume that visits each state-action pair with non-zero probability.
Then both o-policy Monte Carlo algorithms following policy nwill almost
surely converge on the correct Q-values for .
This will be proved in Section 4. Note that the requirement for visiting all
(s;a) pairs is for the evaluation policy { the behaviour policy will also do so,
as a consequence of the conditions above. One immediate corollary of theorem
2 is:
Corollary 1. Ifnis bounded above by  < 1, both o-policy Monte Carlo
algorithms using nwill converge on the correct Q-values for , ifvisits each
state-action pair with non-zero probability.
Proof. This kind of result has been proved before [5]. The nare bounded above
by 1=(1), and thus, for any , are less than n1for suciently large n. Then
Theorem 2 implies the result. u t
A second corollary is:
Corollary 2. Ifn= 11
log(n), both o-policy Monte Carlo algorithms using
nwill converge on the correct Q-values for , ifvisits each state-action pair
with non-zero probability.
Proof. Set= 1=2, note that n= log(n), and that
log
(n)p
log(n)
=p
log(n) log(log(n))
 r (s;a)whenever(ajs)>0, for constants
r(s;a)>0. Then both o-policy Monte Carlo algorithms using nwill converge
on the correct Q-values for , ifvisits each state-action pair with non-zero
probability.
Proof. Letrbe the minimum value of r(s;a)=(ajs), across all ( s;a) where
(ajs)>0. Then0
ncan be rewritten as:
0
n=r+ (1r)00
n;
for some00
n. Similarlyncan be rewritten as
n= (1(1r)n)+ (n)(1r)00
n:
Then note that ( n)(1r) is bounded above by 1 r, and the result follows by
corollary 1. u t
The rest of this paper will be dedicated to proving theorems 1 and 2.
3 Failure to converge
This Section aims to prove Theorem 1, by constructing a counter-example using
the MDP of gure 1.
Set(ajs1) = 1,(ajs2) =(bjs2) = 1=2,S0=fs2g,0
n(ajs2) = 0,0
n(bjs2) =
1, and dene nas
n= 11=(nlog(n)):
s3 s2 s1a,0 b,3a,1
Fig. 1. An MDP where the o-policy Monte Carlo algorithms can fail to compute the
correct Q-values.
The probability of the n-th episode history being hb= (s1;a;0;s2;b;3;s4)
is greater or equal to 1 =2: this history therefore almost surely gets generated
innitely often.
Now consider the episode history ha= (s1;a;0;s2;a;1;s4). It will get gen-
erated during episode nwith probability
1
2(1n) =1
2nlog(n):
Each episode is independent andP
n1=(2nlog(n)) =1, so by the converse
Borel-Cantelli lemma, the episode is generated innitely often.
Then note that if the history hais generated during the n-th episode, it is
generated with weight 1 =(1n) =nlog(n), while the history hbis generated
with weight 1 =(1 +n) = 1=(21=(nlog(n))). Split the weight total Wn(s1;a)
asWa
n(s1;a) +Wb
n(s1;a) (the weight totals due to the histories haandhbre-
spectively). Then ordinary importance sampling will give Qnas:
Qn(s1;a) =Wa
n(s1;a)3Wb
n(s1;a)
Nn(s1;a); (6)
while weighted importance sampling gives:
Qn(s1;a) =Wa
n(s1;a)3Wb
n(s1;a)
Wan(s1;a) +Wbn(s1;a): (7)
In states1, both actions are equiprobable and independent, so the law of
large numbers implies Nn(s1;a)=n!1 almost surely. The Wb
n(s1;a) is the sum
of weights less than 1, so Wb
n(s1;a)Nn(s1;a). The probability of hbincreases
to 1, and the weights for that history are larger than 1 =2 forn1. Thus, almost
surely, for suciently large n,n=3 and 2nare lower and upper bounds for both
Nn(s1;a) andWb
n(s1;a).
Now pick an nwhere episode history hais generated, which must happen
innitely often. The weight of this history is nlog(n), soWa
n(s1;a)n. In
the limit, this must dominate the n-bounded contributions of Nn(s1;a) and
Wb
n(s1;a).
Thus, for a large enough nwherehais generated, equation (6) then gives the
upper bound
Qn(s1;a)Clog(n);
for some constant C. Conversely, equation (7) gives an upper bound:
Qn(s1;a)1C=log(n);
for someC.
The correct Q-value for Q(s1;a) underis clearly (31)=2 =2, so neither
algorithm can converge to the correct values. Ordinary importance sampling
clearly cannot nd the optimal policy (of choosing ain states2) either. Ifbhad
a reward of 0 instead of a reward of 2, then weighted importance sampling
would fail to nd the optimal policy on that MDP, so both algorithms can fail
to nd optimal policies.
Remark 1. The lack of convergence can also be proved for n= 11=n, but the
proofs are more involved.
4 Proof of convergence
This Section aims to prove3Theorem 2.
4.1 Innite variance
The reason that such mathematical machinery is needed is because, in many
cases, the variance of the reward become innite. Consider the MDP of gure
2. Two actions are available in s2: actionawhich, with pprobability will return
the agent to state 1 with a reward of 1, and otherwise send them to state s3with
no reward. And b, which goes straight to s3with no reward. Set (ajs1) = 1,
0
n(bjs2) = 1, and n= 1q. The weights are powers of 1 =q,n(ajs2) =q,
so the expected weighted reward is the correctP1
l=0(l=ql)(pq)l, which is nite.
Since for any random variable X, Var(X) = E(X2)E(X)2, the variance of the
weighted reward is nite i the expected squared weighted reward is nite.
Then the expected squared weighted reward is:
1X
l=0(l=ql)2(pq)l=1X
l=0l2(p=q)l:
And this sum diverges for qp.
4.2 Proving convergence
This proof will use ordinary importance sampling, then generalise to weighted
importance sampling.
Fix any pair ( s;a). Sincemust visit that pair with nite probability, there
is anm0 such that an agent following would
3The proof will closely mirror the standard proofs of the strong law of large num-
bers, see for instance https://terrytao.wordpress.com/2008/06/18/the-strong-law-
of-large-numbers/
s3 s2 s1a,0
a,0,1qb,0
a,1,q
Fig. 2. An MDP where the variance of the reward can become innite within a single
episode.
reach (s;a) inmsteps with probability . During episode n, an agent following
has a probability at least (1n)mof reaching ( s;a). For large enough n,p
log(n)>m. Then, by the assumptions of the Theorem, for large enough n,
(n)m<(n)p
log(n)<n1=n , soP1(1n)m=
1. Then since each episode is independent, the converse Borel-Cantelli lemma
implies that an agent following will almost surely visit ( s;a) innitely often.
How regular will these visits be? The expected number of visits during the
episodes up to the n-th is simply
nX
j=1(1j)m:
For large enough n,n<n1p
log(n)and so eventually
(1n)m>nm1p
log(n)>n0;
for any0>0. Therefore, for large enough n, the number of visits to ( s;a) among
thenepisodes must be almost surely greater thanPn
j=1j0=O(n10). Thus,
for large enough n,Nn(s;a)>n10almost surely.
LetQbe the true Q-value of the MDP under , and letHbe the set of
possible episode histories, ignoring rewards. Let Hlbe the set of histories of
lengthl. Write(hn) forh2H to designate the probability that episode nhas
historyhif the agent follows the policy . Note that since is xed,(hn) is
independent of n(unliken(hn)).
Letbe the policy designed to maximise the expected time the agent spends
in the MDP. This means that is a Markov policy, as the policy that maximises
the time spent if the agent is in state ( s0;a0) does not depend on the agent's prior
history. Combined with the MDP, describes a Markov chain, with absorbing
nal states. Its transition matrix is of the form:
P0R
0Id
;
forP0a transition matrix on the non-absorbing state-action pairs, and Idthe
identity matrix on the absorbing ones. Since any episode must terminate with
probability 1, the matrix P0has a single maximal real eigenvalue 0<1 [2]. For
largen, the probability that the matrix will not have terminated by the n-th
episode is bounded by ( 0)n. Sinceis the policy that maximises the expected
time spent in the MDP, an agent following cannot expect to stay in MDP
longer than that, so there exists a C0such that
(Hl)C0(0)l:
Fix any0<< 1, then because l2(0)lmust eventually be less than l, there
exists aCsuch that
l(Hl)l2(Hl)Cl:
LetEbe the maximal expected reward the agent can generate from a single
state-action pair. Let Sbe the maximal expected squared reward the agent
can generate from a single state-action pair. Then if Rhis the random variable
denoting the reward generated along history h2Hl,
E (Rhjh)lE
Var (Rhjh)E
R2
hjh
l2S:
LetWRl
ndenote the random variable that returns 0 if the length of the n-th
episode is not l, and the (weighted) reward otherwise, under the assumption that
the agent visits (s;a)during episode n. Therefore:
WRl
n=X
h2Hlw(hn)Rh(hn):
Note thatw(hn) =(hn)=(hn)(n)ih, whereihlis the number of times
that the agent goes through a state in S0alongh. Then
En
WRl
n
=X
h2Hlw(hn)E (Rhjh)n(hn)
=X
h2HlE (Rhjh)(hn);
which is the expected reward from episode histories of length lfrom an agent
following. ThereforeP1
l=1En
WRl
n
=P
hE (Rhjh)(hn) =Q.
The expectation and variance of WRl
ncan be bounded as:
En
WRl
n
=X
h2HlE (Rhjh)(hn)
lE(Hl)
ECl
Varn
WRl
n
En
WRl
n2
X
h2HlE
(w(hn)Rh)2jh
(hn)
X
h2Hl(n)ihE
(Rh)2jh
(hn)
(Hl)(n)ll2S
SC(n)l:(8)
RedeneCas max(EC;SC ) so that these bounds are ClandC(n)lrespec-
tively.
Then dene WR1; so, redening Cif needed to cover the nitely many smaller values
ofn, the second bound is C(n)l:
En
WRl
n
Cl
Varn
WRn10for any0>0 and large enough n. Dene
Ql
n=1
Nn(s;a)nX
j=1IjWRl
j
Q0 and for large
enoughn:
En
Ql
n
1
Nn(s;a)nX
j=1IjClPn
j=1Ij
Nn(s;a)ClCl
Varn
Q1such thatmj+1=mj>cfor suciently large j. ThenQmjconverges
toQalmost surely as j!1 .
Proof. Fix any > 0, and consider the subsequence nj=depje. Then since
fmjgis eventually an exponentially growing sequence, mj> njfor suciently
largej.
Setlj=b4pjcand consider Qmj=Qljmj+Q 0.
Fix0==2. Hence for large enough j,C
(mj)1=2(mj)lj<1
(mj)=2. Since
m=2
j>n=2
je(=2)pj>Cj2=2for large enough j, eventually
Var
Q> 0, dene the lacunary sequence mj=d(1 +)je. Formjd(1 +)je,
Qnmust be within 3 (Q+ 1) ofQ. Sincewas arbitrary, this proves that Qn
converges to Qalmost surely as n!1 .
This completes the proof for all MDP's with positive rewards. Note that the
same proof works for MDP's with negative rewards. Then the general proof is
established for by dividing the rewards into positive and negative parts, noting
their separate convergence, and noting that the Q-values update process is linear
in rewards.
Since this proves the convergence of Qn(s;a) toQ(s;a) for any (s;a), and
there are nitely many ( s;a) pairs, this proves the almost sure convergence of
QntoQin general.
It is now necessary to extend the result to weighted importance sampling. To
do that, it will suce to show that, under the conditions above, Wn(s;a)=Nn(s;a)!
1 almost surely.
To see this, change the MDP by setting all rewards to 0, except for the nal
reward when the agent reaches a terminal state, where they will get 1. This
means that the reward along each history is 1, and the weighted reward is just
the weight. Thus the new Q0
nfor ordinary importance sampling is Q0
n(s;a) =
Wn(s;a)=Nn(s;a). By the result we've just proved, these must converge almost
surely to the correct Q-values for the modied MDP, i.e., to 1. This proves that
the ratios converge almost surely to 1, as required.
A Note on the independence assumption
The policy n, as dened in equation (5), assumes that the agent choose inde-
pendently between and0
nfor eachs2S. If this independence is dropped, the
situation can get even worse for convergence { the agent may fail to converge to
the right values4even for xed <1.
Consider the MDP in gure 3. Dene the policy as choosing randomly
amongst the two actions at s1, and choosing aotherwise. Dene S0=fs2;s3g
and0
nas choosing bfrom all states in S0.
For, the probabilities of choosing and0
ninSare equal (and the same
from episode to episode), 2=3= 0:5, but they are strictly anti-correlated
within a given episode.
s4 s3 s2
s1b,0a,0a,1b,0
b,0
a,1
Fig. 3. An MDP on which the agent has non-Markov policy choices.
Notice that the episode history ( s1;a;0;s2;a;1;s3;a;1;s4)never appears .
This is because the uses of 0
nats2ands3are anti-correlated, so the agent
cannot avoid both of them. Therefore the agent never experiences a total re-
ward of 2; moreover, the only episodes with rewards are the (equiprobable)
4For non-Markov policies like this one, the o-policy algorithm has to be adjusted to
consider ratios (h)=(h) for entire histories h(rather the product of state-action
pair probabilities), but this is not a large change.
(s1;a;0;s2;a;1;s3;b;0;s4) and (s1;b;s 3;a;1;s4), both with reward 1. Therefore
the agent will compute Q(s2;a) andQ(s3;a) as having the same values. However,
it is clear that if following ,Q(s2;a) = 3=2 andQ(s3;a) = 1.
References
1. Precup, D., Sutton, R.S., Dasgupta, S.: O-policy temporal-dierence learning with
function approximation. Proceedings of the 18th International Conference on Ma-
chine Learning (2001)
2. Sandberg, A., Snyder-Beattie, A., Armstrong, S.: Nuclear war near misses and an-
thropic shadows. In preparation (2015)
3. Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. MIT Press, Cam-
bridge, MA (1998)
4. Sutton, R., Barto, A.: Reinforcement Learning: An Introduction, Second edition,
Draft. Cambridge, MA (2015)
5. Thomas, P.S.: Safe reinforcement learning. Thesis presented to the Graduate School
of the University of Massachusetts Amherst (2015)


Learning the Preferences of Bounded Agents
Owain Evans
University of OxfordAndreas Stuhlm ¨uller
Stanford UniversityNoah D. Goodman
Stanford University
Introduction
A range of work in applied machine learning, psychology, and social science involves inferring a
person’s preferences and beliefs from their choices or decisions. This includes work in economics
onStructural Estimation , which has been used to infer beliefs about the rewards of education from
observed work and education choices [1] and preferences for health outcomes from smoking behav-
ior [2]. In machine learning, Inverse Reinforcement Learning has been applied to diverse planning
and decision tasks to learn preferences and task-speciﬁc strategies [3, 4]. Large-scale systems in
industry also learn preferences from behavior: for example, people’s behavior on social networking
sites is used to infer what movies, articles, and photos they will like [5].
Existing approaches to inferring human beliefs and preferences typically assume that human behav-
ior is optimal up to unstructured “random noise” [6, 7]. However, human behavior may deviate from
optimality in systematic ways. This can be due to biases such as time inconsistency and framing
effects [8, 9] or due to planning or inference being a (perhaps resource-rational) approximation to
optimality [10, 11]. If such deviations from optimality are not modeled, we risk mistaken inferences
about preferences and beliefs. Consider someone who smokes every day while wishing to quit and
regretting their behavior. A model that presupposes optimality up to noise will infer a preference
for smoking. Similarly, consider someone who always eats at the same restaurant and never tries
anywhere new. One explanation is that the person has strong prior beliefs about the (low) quality of
other restaurants; however, another explanation is that the person fails to fully take into account the
information-value of trying a new restaurant.
This paper explicitly models structured deviations from optimality when inferring preferences and
beliefs. We use models of bounded and biased cognition as part of a generative model for human
choices in decision problems, and infer preferences by inverting this model [12]. We are not primar-
ily interested in investigating models of bounded cognition for their own sake, but instead in using
such models to help accurately infer beliefs and preferences. We aim to highlight the potential value
of models of bounded cognition as tools to aid inference in major applications in social science and
in machine learning.
This paper focuses on inferring preferences (rather than beliefs) and bases inferences on observed
choices in sequential decision problems (MDPs and POMDPs) [13]. We model the following types
of agents:
1.Time-inconsistent (hyperbolic-discounting) agents , which are based on standard models
of temptation, procrastination, and precommitment in psychology [8, 14, 15].
2.Agents that use Monte Carlo approximations of expected utility , which are related to
sampling-based approximate inference and planning [16].
3.Myopic agents , which have short planning horizons.
4.Bounded Value-of-Information agents , which approximate value-of-information compu-
tations and have been used to investigate human performance in POMDPs [17].
We perform Bayesian inference over the space of such agents to jointly infer an agent’s preferences,
beliefs, and their bounds or biases from observed choices.
1
Computational Framework
This section describes the generative models of bounded and unbounded agents that we use to infer
preferences from observed choices. We ﬁrst describe the structure that is common to all of our
agents. We then show that varying different parts of this structure lets us construct agents with
different constraints. Finally, we describe how to use these models to infer preferences.
Agent structure
We treat an agent as a stochastic function that returns an action a2A, denoted by C:?!A
(“choice function”). The structure of the agent is given by a mutual recursion between Cand an
expected utility function, EU:A!R. Both functions are parameterized by the agent’s state, ,
which has a form that varies depending on agent type. To refer to the probability that Creturnsa,
we writeC(a;).
Agents choose actions in proportion to exponentiated expected utility ( softmax ):
C(a;)/eEU(a;)(1)
The noise parameter modulates between uniformly random choice ( = 0) and perfect maxi-
mization (as !1 ). Expected utility depends on current and future utility; future utility in turn
depends on the agent’s simulation of its future internal states and choices:
EU (a;) =U(;a) + E
0M(;a)
a0C(h(0))[EU (a0;0)] (2)
To fully deﬁne an agent type, we have to specify how the agent computes immediate utility ( U), how
it predicts its future internal state for simulation of future choices and expected utility ( M), and how
it modiﬁes the state when it gets passed to the choice function ( h).
Agent types
The two optimal agents we consider as a basis for bounded agents are the standard ones used in
MDP and POMDP settings (Figure 1). In the MDP setting, the agent’s internal state corresponds
Agent type Type of  U (;a )M(;a ) h()
MDP s2S U (s;a) T(s;a)
MDP + myopia (s;d)2SNU(s;a) ifd(1 +).
4
increases we infer a higher Ut(as Tall is chosen despite increasing risk of disaster). The MC model
infers consistently lower values for Ut. It partially explains away John’s choice by positing small N,
which will sometimes lead to overestimates of the expected value of climbing Tall.
3. Failure to explore (myopic and Bounded VOI agents)
Human performance on bandit problems has been studied extensively [20]. For example, Zhang
and Yu [17] show that human performance in a low-dimensional bandit problem is sub-optimal
and is better captured by the Knowledge-Gradient algorithm than by optimal play. The Knowledge-
Gradient algorithm is analogous to our Bounded VOI agent with one level of lookahead. This work
suggests that assuming optimality may lead to inaccurate inferences even for low-dimensional bandit
problems. For higher-dimensional problems, optimal play is intractable and so any realistic agent
will use approximations. Consider the following bandit-style problem:
You get out of a meeting and choose between two nearby restaurants, AandB.
You know the utility UAexactly (e.g. Ais a chain) but are uncertain about UB. For
the nextTmonths, you have meetings in the same place and will face the same
choice between AandB.
The goal is to infer UA, the agent’s utility for A. We run inference repeatedly for increasing values of
T(i.e. we increase the expected value of exploration). For each T, we condition on the observation
that the agent chooses restaurant Afor each of the Tmonths. That is, the agent never explores, even
asTgrows.
The “optimal” inference model assumes that the agent solves the POMDP perfectly given their
prior onUB. On the “potentially myopic” model, the agent can either plan optimally or else plan
myopically with the time horizon set to one trial. As Tincreases, exploration becomes more valuable
to the optimal agent. Assuming optimality therefore leads to progressively higher inferred values for
UA. In contrast, a myopic agent will not explore more as Tincreases, resulting in a ﬂatter curve in
Figure 2c.
The Bounded VOI model with kvoi>0behaves optimally for this bandit-style problem. But con-
sider the following elaboration:
You are choosing somewhere to eat, but there are no restaurants nearby. You can
head to restaurant A(which again has known utility UA) or try somewhere new.
Most restaurants are worse than A, but some might be better. Before trying a
restaurant you ask for advice in two distinct steps. First you ask which neighbor-
hood has the best restaurants and later you ask a local of that neighborhood which
restaurant is best. When you try a restaurant you learn its utility. There are no
costs to getting advice and your distance traveled is not a constraint. As above,
this choice is repeated for each of Tmonths.
The goal for inference is again to infer UA. We ﬁx the problem parameters2and varyT. The ob-
servation we condition on is the agent choosing Aevery time (same as before). The Bounded VOI
agent withkvoi= 1 deviates from optimal on this problem. This agent models itself as updating
on the ﬁrst question (“which neighborhood?”) but neither on the second question nor the direct ob-
servation of restaurant quality. It can fail to explore even when Tis high (without having a strong
preference for A). The Bounded VOI model only matches optimal behavior when kvoi>2. Figure
2d compares a “potentially bounded VOI” model (which includes the optimal model as a special
case) with the optimal model.
2We assume two neighborhoods with two restaurants each. The agent has a prior over the utility of unknown
restaurants such that most are bad. Our inference prior on UAis such that we think it likely that the agent expects
the best unknown restaurant to be better than A.
5
do nothing!u = 0
promise!!u = -
do !work!u = -1
help friend!u = +R
Figure 3: Transition graph for Example 1 (procrastination). Nodes represent states; uis the utility
the agent receives for every day spent at the state. Arcs indicate possible (uni-directional) determin-
istic transitions between states. The agent takes T= 8 actions, one for each day. We condition on
observing the agent moving directly to “Promise” and staying there for the remaining 7 days.
var agent =function (state, delay, timeLeft){
return Marginal( function (){
var action = uniformDraw(actions)
var eu = expUtility (state, action, delay, timeLeft)
factor(alpha \*eu)
return action
})
}
var expUtility =function (state, action, delay, timeLeft){
var u = discountedUtility(state, action, delay, K)
if(timeLeft == 1){
return u
}else {
return u + expectation(INFER\_EU( function (){
var nextState = transition(state, action)
var nextAction = sample( agent (nextState, delay+1, timeLeft-1))
return expUtility (nextState, nextAction, delay+1, timeLeft-1)
}))
}
}
var simulate = function (startState, totalTime){
var sampleStateSequence = function (state, timeLeft, history){
if(timeLeft==0){
return history
}else {
var delay = 0
var action = sample(agent(state, delay, PLANNING\_HORIZON))
var nextState = transition(state, action)
return sampleStateSequence(nextState, timeLeft-1, update(history, nextState))
}
}
return Marginal( function (){
return sampleStateSequence(startState, totalTime, initHistory(startState))
})
}
Figure 4: Implementation of a generative model for agents in the MDP setting. The language is
WebPPL (with minor implementation details omitted) [21]. Note the mutual recursion between
agent andexpUtility : the agent’s reasoning about future expected utility includes a (po-
tentially biased) model of its own decision-making. The function Marginal computes exact
distributions over the output of its function argument. The factor statement implements soft
conditioning—it is used here for softmax “planning-as-inference” [22]. To generate agent behav-
ior, we specify a decision problem by providing implementations for transition andutility .
We then call simulate(startState, totalTime) . For exact planning, we set INFER\_EU
toMarginal . For the Monte Carlo agent, we set INFER\_EU to a function that computes sampling-
based approximations. If the constant Kis set to zero, the agent does not discount (and so is optimal);
otherwise, the agent performs Naive hyperbolic discounting.
References
[1] Victor Aguirregabiria and Pedro Mira. Dynamic discrete choice structural models: A survey. Journal of
Econometrics , 156(1):38–67, 2010.
6
[2] Michael Darden et al. Smoking, expectations, and health: a dynamic stochastic model of lifetime smoking
behavior. Health Econometrics and Data Group-University of York, Working Paper , 10:28, 2010.
[3] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Pro-
ceedings of the Twenty-First International Conference on Machine Learning , page 1. ACM, 2004.
[4] Stefano Ermon, Yexiang Xue, Russell Toth, Bistra Dilkina, Richard Bernstein, Theodoros Damoulas,
Patrick Clark, Steve DeGloria, Andrew Mude, Christopher Barrett, et al. Learning large-scale dynamic
discrete choice models of spatio-temporal preferences with application to migratory pastoralism in east
africa. In Meeting Abstract , 2014.
[5] Dietmar Jannach, Markus Zanker, Alexander Felfernig, and Gerhard Friedrich. Recommender Systems:
an Introduction . Cambridge University Press, 2010.
[6] Dongho Kim, Catherine Breslin, Pirros Tsiakoulis, Milica Ga ˇsic, Matthew Henderson, and Steve Young.
Inverse reinforcement learning for micro-turn management. In Proceedings of the Annual Conference
of the International Speech Communication Association, INTERSPEECH , pages 328–332. International
Speech and Communication Association, 2014.
[7] Jiangchuan Zheng, Siyuan Liu, and Lionel M Ni. Robust bayesian inverse reinforcement learning with
sparse behavior noise. In Twenty-Eighth AAAI Conference on Artiﬁcial Intelligence , 2014.
[8] George Ainslie. Breakdown of Will . Cambridge University Press, 2001.
[9] Daniel Kahneman and Amos Tversky. Prospect theory: an analysis of decision under risk. Econometrica:
Journal of the Econometric Society , pages 263–291, 1979.
[10] Edward Vul, Noah Goodman, Thomas L Grifﬁths, and Joshua B Tenenbaum. One and done? optimal
decisions from very few samples. Cognitive Science , 38(4):599–637, 2014.
[11] Thomas L Grifﬁths, Falk Lieder, and Noah D Goodman. Rational use of cognitive resources: Levels
of analysis between the computational and the algorithmic. Topics in Cognitive Science , 7(2):217–229,
2015.
[12] Joshua B Tenenbaum, Charles Kemp, Thomas L Grifﬁths, and Noah D Goodman. How to grow a mind:
Statistics, structure, and abstraction. Science , 331(6022):1279–1285, 2011.
[13] Alessandro Panella and Piotr Gmytrasiewicz. Learning policies of agents in partially observable domains
using bayesian nonparametric methods. In AAMAS Workshop on Multiagent Sequential Decision Making
Under Uncertainty , 2014.
[14] David Laibson. Golden eggs and hyperbolic discounting. The Quarterly Journal of Economics , pages
443–477, 1997.
[15] Owain Evans, Andreas Stuhlm ¨uller, and Noah D Goodman. Learning the preferences of ignorant, incon-
sistent agents. In press.
[16] Christian Robert and George Casella. Monte Carlo Statistical Methods . Springer Science & Business
Media, 2013.
[17] Shunan Zhang and J Yu Angela. Forgetful bayes and myopic planning: Human learning and decision-
making in a bandit setting. In Advances in Neural Information Processing Systems , pages 2607–2615,
2013.
[18] Jon Kleinberg and Sigal Oren. Time-inconsistent planning: a computational problem in behavioral eco-
nomics. In Proceedings of the Fifteenth ACM Conference on Economics and Computation , pages 547–
564. ACM, 2014.
[19] George A Akerlof. Procrastination and obedience. The American Economic Review , pages 1–19, 1991.
[20] Mark Steyvers, Michael D Lee, and Eric-Jan Wagenmakers. A bayesian analysis of human decision-
making on bandit problems. Journal of Mathematical Psychology , 53(3):168–179, 2009.
[21] Noah D Goodman and Andreas Stuhlm ¨uller. The Design and Implementation of Probabilistic Program-
ming Languages . 2014.
[22] Matthew Botvinick and Marc Toussaint. Planning as inference. Trends in Cognitive Sciences , 16(10):485–
488, 2012.
[23] Richard A Posner. Catastrophe: Risk and Response . Oxford University Press, 2004.
7