PERSPECTIVE ARTICLE
Aligning AI Optimization to Community Well-Being
Jonathan Stray1
Received: 7 February 2020 / Accepted: 9 October 2020 / Published online: 4 November 2020
#Springer Nature Switzerland AG 2020
Abstract
This paper investigates incorporating community well-being metrics into the objectives
of optimization algorithms and the teams that build them. It documents two cases where
a large platform appears to have modified their system to this end. Facebook incorpo-
rated“well-being ”metrics in 2017, while YouTube began integrating “user satisfac-
tion”metrics around 2015. Metrics tied to community well-being outcomes could also
be used in many other systems, such as a news recommendation system that tries to
increase exposure to diverse views, or a product recommendation system that
opstimizes for the carbon footprint of purchased products. Generalizing from these
examples and incorporating insights from participatory design and AI governance leads
to a proposed process for integrating community well-being into commercial AI
systems: identify and involve the affected community, choose a useful metric, use this
metric as a managerial performance measure and/or an algorithmic objective, and
evaluate and adapt to outcomes. Important open questions include the best approach
to community participation and the uncertain business effects of this process.
Keywords Artificial intelligence .AI ethics .Community well-being .Optimization .
Corporate social responsibility
Introduction
This paper is an extended analysis of a simple idea: large-scale commercial optimizing
systems may be able to manage harmful side effects on communities by monitoring
established well-being metrics. It sketches a theory that ties together quantitative
measures of well-being, contemporary metrics-driven management practice, the objec-
tive function of optimization algorithms, participatory and multi-stakeholder gover-
nance of algorithmic systems, and the protection or promotion of community well-
being. Detailed analyses of recent efforts by Facebook and YouTube are used toInternational Journal of Community Well-Being (2020) 3:443 –463
https://doi.org/10.1007/s42413-020-00086-3
\*Jonathan Stray
jonathan@partnershiponai.org
1Partnership on AI, San Francisco, CA, USA
illustrate the challenges and unknowns of this approach, which generalizes to a variety
of different types of artificial intelligence (AI) systems. The core contribution of thisarticle is a proposed process for the use of community well-being metrics withincommercial AI systems.
Well-being encompasses “people ’s living conditions and quality of life today
(current well-being), as well as the resources that will help to sustain people ’s
well-being over time (natural, ec onomic, human and social capital) ”(OECD
2019b , p. 2). Community well-being attempts to evaluate well-being at the level
of a community defined “in geographic terms, such as a neighborhood or town …
or in social terms, such as a group of people sharing common chat rooms on theInternet, a national professional association or a labor union ”(Phillips and Pittman
2015 , p. 3). The measurement of well-being is now a well-established field with a
long history, and is increasingly used i n policy-making (Exton and Shinwell
2018 ).
Large AI systems can have both positive and harmful side effects on communities,
through effects on employment and inequality (Korinek and Stiglitz 2017 ), privacy and
safety (OECD 2019a ), addictive behavior (Andreassen 2015 ), fairness and discrimina-
tion (Barocas et al. 2018 ), human rights (Donahoe and Metzger 2019 ), polarization,
extremism, and conflict (Ledwich and Zaitsev 2020 ; Stoica and Chaintreau 2019 ), and
potentially many other areas (Kulynych et al. 2020 ). Importantly, AI systems can affect
non-users too, as with environmental externalities.
Most AI is built around optimization “in which the aim is to find the best state
according to an objective function ”(Russell and Norvig 2010 ,p .1 2 1 )w h e r ea n
objective function is some method for quantitatively evaluating the desirability of
an outcome (Dantzig 1982 ). Standard management practice also increasingly
involves the maximization of qua ntitative metrics (Parmenter 2020 ), which can
be considered an optimization process. T his paper is concerned with optimizing
systems composed of people and algorithm s which affect communities, where the
choice of objective might have significant societal influence. Examples includesystems used to allocate resources or assign work, choose what news people see,
recommend products to buy, or implement government policy. Many of these
systems would be considered AI, but perhaps the phrase “autonomous and intel-
ligent systems ”(Schiff et al. 2020 , p. 1) which appears in certain standards efforts
would be better, because an automated system does not have to be very smart tocause harm. Rather, the unifying feature is optimization –both the cause of many
problems and an opportunity for a response.
The central idea of this paper is to incorporate community well-being metrics into
the optimization process at both the mana gerial and technical level. This is a
sociotechnical approach to systems design (Baxter and Sommerville 2011 )t h a tc o n -
siders the role of both people and technology. There are many technical interventionsthat could be undertaken aside from the modification of an algorithmic objectivefunction; for example, a social media product team could choose to show a simplechronological list of posts rather than using algorithmic content personalization. How-ever, if product managers are evaluated on community well-being outcomes, they maychoose to make such a change based on the expected effects on users. The integrationof the managerial and the technical in an optimization framework can motivate many
possible product design changes.444 International Journal of Community Well-Being (2020) 3:443 –463
Background
This paper responds most directly to recent calls for research into well-being and AI. It
proposes specific “improvements to product design ”(Schiff et al. 2019 ,p .3 )a n di ti s
interdisciplinary, systems-based, and community-oriented (Musikanski et al. 2020 ). It
draws on and contributes to the emerging field of recommender alignment, the practice
of building algorithms for content ranking and personalization that enact human values
(Stray et al. 2020 ).
The goal of the process proposed in this paper is the governance of large-scale
commercial algorithmic systems. Rahwan ( 2018 ) calls this society-in-the-loop control,
defined as “embedding the values of society, as a whole, in the algorithmic governance
of societal outcomes ”(p. 3). In this sense community participation is a key element of
the proposed framework, and this paper draws on approaches as diverse as participatorydesign (Simonsen and Robertson 2012 ) and corporate stakeholder engagement
(Manetti 2011 ).
Community Well-BeingAt the individual level well-being is usually studied as an experiential state, and there is
now a wealth of research on the definition and reliable measurement of subjective well-being (Diener et al. 2018 ). Although well-being is a rich, multidimensional construct,
even single questions can reveal substantial information, such as overall, how satisfied
are you with life as a whole these days? answered on a 0 –10 scale. This well-studied
measure has several advantages: it correlates with how people make major life deci-sions, gives a similarly reliable result across cultures, and is by itself informativeenough to be used in quantitative evaluations of policy choices (O ’Donnell et al. 2014 ).
Community well-being “embraces a wide range of economic, social, environmental,
political, cultural dimensions, and can be thought of as how well functions of commu-nity are governed and operating ”(Sung and Phillips 2018 , p. 64). In practice, commu-
nity well-being is assessed using a variety of metrics across many domains. Often both
subjective and objective indicators are needed to get a full picture (Musikanski et al.
2019 ). A survey of local and national well-being indicator frameworks in use in the
United Kingdom gives an overview of the substance and range of such metrics(Bagnall et al. 2017 ). Community well-being frameworks can originate from consid-
eration of geographic communities, or communities of interest (Phillips and Pittman2015 )w h i c hm a yb ep a r t i c u l a r l yr e l e v a n tt oo n l i n ep l a t f o r m s .
As an example community well-being framework, the OECD Better Life Index
(Durand 2015 ) aims to measure “both current material conditions and quality of life ”
(p. 1) across countries through the metrics shown in Table 1. This framework includes
the life satisfaction measure above, as well as statistical indicators around health,education, employment, etc. in conjunction with subjective indicators such as whetherone feels safe walking alone at night.
Technologists and scholars have begun to appreciate the significance of well-being
measures in the design and operation of AI systems (Musikanski et al. 2020 ). The IEEE
7010 Recommended Practice Standard for Assessing the Impact of Autonomous andIntelligent Systems on Human Well-Being collects pre-existing measures from sources
such as the OECD Better Life Index, the UN Sustainable Development Indicators, theInternational Journal of Community Well-Being (2020) 3:443 –463 445
Human Development Index, the World Health Organization, the World Values Survey,
Freedom House, and others (Schiff et al. 2020 ). From the point of view of a technol-
ogist who is concerned about the societal effects of their work, established well-beingmetrics have the advantage of representing extensive deliberation by domain experts.
OptimizationOptimization is used extensively in AI to guide training and learning. A problem to be
solved is expressed as a scalar function –a method to calculate a single number that
expresses the desirability of any given hypothetical solution. Solving the problemmeans finding a solution that maximizes this function. The encapsulation of concernsinto a single function was a major conceptual advance that enabled the creation ofgeneric optimization algorithms (Dantzig 1982 ). Conceptually, any problem that has
some set of best solutions can be expressed as optimization with a single objectiveTable 1 Indicators from the OECD Better Life Index (Durand 2015 ). Each of these has a specific statistical
definition and has been collected across OECD countries since 2011
Domain Indicators
Housing Dwellings without basic facilities
Housing expenditure
Rooms per person
Income Household net adjusted disposable income
Household net wealth
Jobs Labor market insecurity
Employment rate
Long term unemployment rate
Community Quality of support network
Education Educational attainment
Student skillsYears in education
Environment Air pollution
Water quality
Civic engagement Stakeholder engagement for developing regulations
Voter turnout
Health Life expectancy
Self-reported health
Life Satisfaction Life satisfaction
Safety Feeling safe walking alone at night
Homicide rate
Work-life balance Employees working very long hours
Time devoted to leisure and personal care446 International Journal of Community Well-Being (2020) 3:443 –463
function, though practical problem-solving often involves the optimization of multiple
sub-goals.
A supervised machine learning algorithm that attempts to identify objects from
images would usually be trained through a loss function that penalizes incorrectanswers. A reinforcement learning approach to playing a video game might use thegame score as a reward function. There are also value functions, cost functions, fitness
functions, energy functions and more, all of which operate on similar principles
(Russell and Norvig 2010 ). For simplicity, in this paper I refer to all of the scalar
functions used to drive AI behavior as objective functions .
In this paper I refer to an optimizing system as if there were one optimizer and one
objective. In practice such systems, especially those at platform scale, may includedozens or hundreds of optimizing components (numerous trained sub-models, forexample). There isn ’t one objective function that can be altered, but many. Nonetheless,
there are usually a few high-level goals concerned with the system ’sm a i no u t p u t s .T h i s
is the case at Groupon with many interacting models and a master objective function
that aligns to company goals (Delgado et al. 2019 ).
Quantitative metrics analogous to objective functions are also used in corporate
management. Modern management practice includes concepts such as key performance
indicators (Parmenter 2020 )a n d objectives and key results (Doerr 2017 ), both of which
involve quantitative indicators of progress. Economic theory frequently models thecorporation as a profit optimizer (e.g. Samuelson and Marks 2014 ). More sophisticated
descriptions try to account for the creation of various types of long-term value, such as
thebalanced scorecard (Kaplan 2009 ) and sustainability accounting (Richardson
2013 ), both of which describe various non-financial metrics that are intended to be
optimized.
Case Studies of Platform Interventions
This section presents two examples where large technology companies seem to have
optimized for well-being, or a similar concept. These cases have been reconstructed
through documentary evidence such as public posts, previously published interviews,financial reports, and research articles by employees.
Facebook ’s Well-Being Optimization
In late 2017 and early 2018, Facebook made a number of changes to their product
explicitly designed to promote well-being. Facebook researchers Ginsberg and Burke
(2017 ) wrote in a public post in December 2017:
What Do Academics Say? Is Social Media Good or Bad for Well-
Being? A c c o r d i n gt ot h er e s e a r c h ,i tr e a l l yc o m e sd o w nt o how you use
the technology. For example, on social media, you can passively scrollthrough posts, much like watching T V, or actively interact with friends —
m e s s a g i n ga n dc o m m e n t i n go ne a c ho t h e r ’s posts. Just like in person,
interacting with people you care abou t can be beneficial, while simply
watching others from the sidelines may make you feel worse. (para. 7).International Journal of Community Well-Being (2020) 3:443 –463 447
This post cites a number of peer-reviewed studies on the well-being effects of social
media, some of which were collaborations between Facebook researchers andacademics. Ginsberg and Burke ( 2017 ) cite Verduyn et al. ’s(2017 ) review paper on
the effects of social media on well-being, which has an obvious resonance withFacebook ’sf r a m i n g :
passively using social network sites provokes social comparisons and envy,
which have negative downstream consequences for subjective well-being. Incontrast, when active usage of social network sites predicts subjective well-being,it seems to do so by creating social capital and stimulating feelings of socialconnectedness. (Verduyn et al. 2017 ,p .2 7 4 )
A close reading of posts around this time shows that Facebook developed a well-being
proxy metric. A January 2018 post by Facebook ’s Chief Executive Officer notes that
“research shows that strengthening our relationships improves our well-being and
happiness ”(Zuckerberg 2018 , para. 2) and mentions well-being twice more, then
switches to the phrase “meaningful social interactions: ”
I'm changing the goal I give our product teams from focusing on helping you find
relevant content to helping you have more meaningful social interactions.(Zuckerberg 2018 ,p a r a .7 )
Relevance is a term of art in recommender systems, referring to user preferences as
expressed through item clicks or ratings, and is increasingly understood as a simplistic
objective (Jannach and Adomavicius 2016 ). The algorithmic change away from rele-
vance was described by the head of the News Feed product:
Today we use signals like how many people react to, comment on or share posts
to determine how high they appear in News Feed. With this update, we will alsoprioritize posts that spark conversations and meaningful interactions betweenpeople. To do this, we will predict which posts you might want to interact withyour friends about, and show these posts higher in feed (Mosseri 2018 , para. 3).
Facebook created a well-being metric and assigned it as a goal to a product team, which
incorporated it into an existing algorithmic objective function. This objective functionwas augmented by creating a model that uses existing data such as past user behaviorand post content to predict whether a user will have a meaningful social interaction if
shown any particular post. There is little public documentation of how meaningful
social interactions are measured. The most detailed description is from the transcript of
a call where Facebook reported earnings to investors, which explains that meaningful
social interactions are measured through user surveys:
So the thing that we're going to be measuring is basically, the number of
interactions that people have on the platform and off because of what they'reseeing that they report to us as meaningful …the way that we've done this for448 International Journal of Community Well-Being (2020) 3:443 –463
years is we've had a panel, a survey, of thousands of people who basically we
asked, what's the most meaningful content that they had seen in the platform orthey have seen off the platform. (Facebook 2018 ,p .1 3 )
The resulting system is reconstructed in Fig. 1.
While there is no public account of the effects of the incorporation of the meaningful
social interactions prediction model on the meaningful social interactions metric as
measured by Facebook through user surveys, Facebook has reported reduced engage-ment on at least one product, suggesting that the meaningful social interactions
objective was weighted strongly enough to cause significant changes in which itemsare presented to users:
video is just a passive experience. To shift that balance, I said that we were going
to focus on videos that encourage meaningful social interactions. And in Q4, we
updated our video recommendations and made other quality changes to reflectthese values. We estimate these updates decreased time spent on Facebook byroughly 5% in the fourth quarter. To put that another way: we made changes thatreduced time spent on Facebook by an estimated 50 million hours every day tomake sure that people's time is well spent. (Facebook 2018 ,p .2 ) . . .
YouTube ’s User Satisfaction Metrics
John Doerr ’sMeasure What Matters (2017) documents YouTube ’sm u l t i - y e a re f f o r tt o
reach one billion hours of daily user watch time through interviews with SusanWojcicki, Chief Executive Officer and Cristos Goodrow, Vice President of Engineeringat YouTube (Doerr 2017 , pp. 154 –172). Goodrow describes the inception of
Fig. 1 A reconstruction of Facebook ’su s eo f meaningful social interactions circa 2018. Well-being effects are
unobserved because they happen outside of user interactions with FacebookInternational Journal of Community Well-Being (2020) 3:443 –463 449
YouTube ’s recommendation system in 2011, and how he advocated to optimize for
watch time instead of video views as:
On a dedicated team named Sibyl, Jim McFadden was building a system for
selecting “watch next ”recommendations, aka related videos or “suggestions. ”It
had tremendous potential to boost our overall views. But were views what we
really wanted to boost?...
I sent a provocative email to my boss and the YouTube leadership team. Subject line:
“Watch time, and only watch time. ”It was a call to rethink how we measured success:
“All other things being equal, our goal is to increase [video] watch time. ”...
Our job was to keep people engaged and hanging out with us. By definition, viewers are
happier watching seven minutes of a ten-minute video (or even two minutes of a ten-minute video) than all of a one-minute video. And when they ’re happier, we are, too.
(Goodrow quoted in Doerr 2017 ,p .1 6 2 ) . . .
Goodrow ’s retelling includes user happiness andsatisfaction as goals along with the
more business-oriented engagement . For the purposes of this paper, I assume user
happiness andsatisfaction are analogous to well-being, but unlike the Facebook case,
YouTube ’s public statements have not mentioned well-being. In accordance with the
unified treatment of managerial and technical optimization proposed in this paper,Goodrow confirms that a team-level metric drove engineering decisions:
Reaching one billion hours was a game of inches; our engineers were hunting for
changes that might yield as little as 0.2 percent more watch time. In 2016 alone,they would find around 150 of those tiny advances. We ’d need nearly all of them
to reach our objective. (Goodrow quoted in Doerr 2017 ,p .1 6 9 )
Yet watch time was not the only objective, and YouTube incorporated other changes to
improve the quality of the product and the effects on users:
In fact, we ’d commit to some watch-time-negative decisions for the benefit of our
users. For example, we made it a policy to stop recommending trashy, tabloid-style videos —like“World ’s Worst Parents, ”where the thumbnail showed a baby
in a pot on the stove. Three weeks in, the move proved negative for watch time by
half a percent. We stood by our decision because it was better for the viewerexperience, cut down on click bait, and reflected our principle of growingresponsibly. Three months in, watch time in this group had bounced back andactually increased. Once the gruesome stuff became less accessible, peoplesought out more satisfying content. (Goodrow quoted in Doerr 2017 ,p .1 6 4 )
This was the beginning of a move away from strict maximization of time spent. Starting
in 2015 YouTube began to incorporate user satisfaction metrics (Doerr 2017 ,p .1 7 0 ) .
As in the Facebook case, these are derived from surveys:450 International Journal of Community Well-Being (2020) 3:443 –463
we learned that just because a user might be watching content longer does not
mean that they are having a positive experience. So we introduced surveys to askusers if they were satisfied with particular recommendations. With this directfeedback, we started fine-tuning and improving these systems based on this high-fidelity notion of satisfaction. (Google 2019 ,p .2 1 )
These user satisfaction survey results were incorporated directly into the objectives of
the YouTube recommendation system, as discussed in a recent YouTube technicalpaper:
we first group our multiple objectives into two categories: 1) engagement objec-
tives, such as user clicks, and degree of engagement with recommended videos;2) satisfaction objectives, such as user liking a video on YouTube, and leaving arating on the recommendation. (Zhao et al. 2019 ,p .4 3 )
Analysis of Facebook and YouTube Cases
The Facebook and YouTube cases are significant because they are examples of a major
platform operator explicitly saying that they have decided to monitor and optimize for awell-being proxy, operationalized at both the management and algorithmic levels.
Facebook has provided a public justification for its meaningful social interaction
metric in terms of prior research which suggests that active use of social mediaimproves well-being while passive use decreases it. While this is far from a holisticmeasure of well-being, let alone community well-being, at least it connects to previous
work in a clear way. Public statements from YouTube have not mentioned well-being,
instead focusing on “responsibility ”(Wojcicki 2019 , para. 2) and user satisfaction as
assessed through surveys.
Explicit user surveys are an improvement on YouTube ’s previous identification of
watch time with user happiness. Researchers report a negative correlation between TVwatching and well-being that suggests there is something like an addiction mechanisminvolved: “individuals with incomplete control over, and foresight into, their own
behavior watch more TV than they consider optimal for themselves and their well-
being is lower than what could be achieved ”(Frey et al. 2007 ,p .2 8 3 ) .S i m i l a re f f e c t s
have been observed in social media use where addicted users “have typically attempted
to cut down on social networking without success ”(Andreassen 2015 , p. 176). Google
now publicly recognizes that maximizing watch time does not optimize for “positive ”
outcomes (Google 2019 ,p .2 1 ) .
A more systematic conception of well-being would articulate what aspects of well-
being matter to YouTube and why user satisfaction is a good proxy. Of course, well-
being outcomes depend enormously on who a user is and what they watch. A user
might learn valuable and fulfilling skills from how-to videos, become more politically
engaged, consume worthwhile art, or they might be radicalized into violence (Ledwichand Zaitsev 2020 ).
Another issue is that both companies are optimizing for individual outcomes: well-
being but not necessarily community well-being. Community well-being “is more thanInternational Journal of Community Well-Being (2020) 3:443 –463 451
an aggregate of individuals ’satisfaction ”(Sung and Phillips 2018 , p. 65) and cannot be
assessed simply by adding up the well-being of all individuals in the community. Thisis analogous to the classic problem of aggregating utilities in welfare economics (Fosterand Sen 1997 , p. 16). Conversely, optimizing for each person individually will not
necessarily promote community well-being due to problems of externalities, collectiveaction, and conflicting preferences (Baum 2020 ;M i l a n oe ta l . 2019b ). Attention to
aggregates may also miss local problems, such as negative effects in a particular city or
for a particular subgroup, or run into Simpsons ’paradox issues where the sign of the
effect depends on the granularity of the groups studied (Kievit et al. 2013 ). For all these
reasons, clarity on the definition of community or communities matters greatly.
Perhaps the biggest weakness of these cases is that there is no record of consultation
with the putative beneficiaries of these algorithmic changes, and no public evaluation ofthe results. Hopefully algorithmic interventions of this magnitude were informed byuser research or some sort of consultative process, but none was reported. Presumably
meaningful social interactions anduser satisfaction were increased, but there has been
no disclosure of how much. Absent also is any report of effects on any other compo-nents of well-being, such as feelings of social connectedness or life satisfaction, or evenobjective indicators like employment status. It ’s similarly unclear how these changes
affected not just individual well-being but community well-being for different com-munities; there may even have been negative effects on certain types of users. Infor-mation about outcomes is especially important because the link between Facebook ’s
meaningful interactions and well-being is theoretical, deduced from previous research
into active and passive social media use, while YouTube has said their user satisfaction
surveys are included in a “responsibility ”metric (Bergen 2019 , para. 10) and that they
aim for “positive ”experiences (Google 2019 , p. 21) without providing any further
explanation of their goals or results. Determining the actual effect of these large-scaleinterventions is itself a significant social science research effort, and if Facebook orYouTube have these answers, they have not been shared. This is algorithmic manage-ment, but not yet the algorithmic governance that the society-in-the-loop model envi-
sions (Rahwan 2018 ).
The reported business outcomes are also instructive, as both the Facebook and
YouTube changes resulted in at least temporary reductions in engagement metrics.Facebook reports that the incorporation of a meaningful social interactions metric into
their video product caused a 5% reduction in time spent, which was consideredsignificant enough to be discussed with investors (Facebook 2018 )b u tt h el o n g e r -
term effects are unclear. YouTube described changes that reduced watch time but alsoreports that watch time recovered over a time span of months as users changed theirbehavior. This demonstrates both that major corporations are willing to accept reduc-
tions in engagement to pursue social ends, and that the long-term business effects of
incorporating well-being metrics are not necessarily negative.
Generalization to Other Domains
The Facebook and YouTube cases suggest the possibility of a general method formanaging the well-being outcomes of commercial optimizing systems, which is the
core contribution of this article. This section begins by arguing that some type of452 International Journal of Community Well-Being (2020) 3:443 –463
metric-driven community well-being optimization is not only useful but likely neces-
sary for any AI system with broad social impacts, because individual user control willnot be sufficient. It then shows how this general method could apply to diverse domainsby working through potential applications to news recommendation and online shop-ping. These hypothetical applications demonstrate the generality of a metrics-drivenapproach and illuminate further possibilities and challenges that shape the recommen-
dations in this paper.
User Control is not Sufficient for Community Well-BeingThis article recommends participatory processes to involve users and other stakeholders
in metric-driven optimization for community well-being. A potential alternative is toprovide increased user control directly, so that people can choose what is best forthemselves. Many authors have pointed to the central role of user agency in the ethics
of AI systems (Floridi and Cowls 2019 ) and in the important context of content ranking
Paraschakis ( 2017 ) has proposed “controls [that] enable users to adjust the recom-
mender system to their individual moral standards ”(p. 6). However, increasing user
agency will not by itself solve the problem of ensuring good outcomes at the commu-nity level because many users will not customize the systems they use, and becauseindividually good choices do not necessarily produce socially good outcomes.
Any set of controls must necessarily be few enough to be humanly manageable. This
restricts the number of dimensions that can be controlled and will make it difficult to
express nuanced conceptions of well-being. Natural language interfaces e.g. Yu et al.
(2019 ) may allow the expression of more complicated concepts. Nonetheless users will
probably leave most parameters at default settings, which means that the defaults mustpromote well-being.
Even if all users in fact succeeded in directing an AI system to do exactly as desired
this would not necessarily result in the best community outcomes. As Ostrom ( 2000 )
has articulated, individual action does not succeed in producing social goods withoutthe concurrent evolution of social norms. These challenges of collective action have
been explored in the context of AI systems from the perspective of social choice theory
(Baum 2020 ) and multi-stakeholder recommendation systems (Milano et al. 2019a ).
Further, existing societal inequalities can constrain users ’ability to exploit algorithmi-
cally provided choices (Robertson and Salehi 2020 ), for example due to a lack of
information or the cost burden of choosing the “best”option.
User control is essential, perhaps even necessary for community well-being, but it is
not sufficient. Collective algorithmic governance is needed for much the same reasonssocietal governance is needed, and appropriate well-being metrics are useful in algo-
rithmic governance just as they are in public policy.
Diverse News RecommendationsNews recommenders are the algorithms that choose, order, and present journalism
content to users. The potential application of community well-being metrics to thesesystems illustrates the challenges around defining a community and choosing metrics.News recommendation algorithms can have societal consequences (Helberger 2019 )
but it is not clear how to manage such algorithms for community well-being. To beginInternational Journal of Community Well-Being (2020) 3:443 –463 453
with, there is no single community that consumes news, but many overlapping
communities organized around different geographic regions and different topics(Reader and Hatcher 2011 , p. 3). Each of these communities may have different
concerns at any given moment. Incorporating social network analysis or country-specific data can improve the performance of recommender systems as measured bytraditional relevance metrics (Chen et al. 2018 ; Roitero et al. 2020 ) but the question of
how a recommender system impacts pre-existing communities, e.g. a city, has not been
explored. Conversely, existing community well-being indicators have not been de-signed to capture the consequences of news recommender systems.
One well-developed concern with news recommenders is exposure diversity, mean-
ing the range of sources, topics, and viewpoints that each person is algorithmicallypresented (Bernstein et al. 2020 ). Taking political theory as a starting point Helberger
et al. ( 2018 ) identify liberal, deliberative, and radical approaches to the design of
diverse news recommenders. Consider the problem of designing a national news
recommender that supports a deliberative view of diversity, one in which:
exposure to diverse viewpoints is considered valuable because it helps citizens
develop more informed opinions and less polarized, more tolerant attitudestowards those with whom they disagree …it is conceivable to design metrics
that would focus, for example, on user engagement with opposing political views,cross-ideological references in public debates or social media connections be-tween people who represent different ideological positions. (Helberger et al.
2018 ,p .1 9 5 )
Diversity metrics could be constructed from algorithmic methods to estimate the
ideological position of users or posts (Budak et al. 2016 ; Garimella and Weber
2017 ). These give a measure of distance between any two items, which could then
be used to define the diversity of a set of recommended items according to variousstandard formulas such as the average distance between any pair (Kunaver and Po žrl
2017 ).
Such a metric would capture the output of the system, not its effects on users.
Facebook and YouTube use user surveys to tie algorithmic changes to human out-comes. It may be possible to establish a causal connection from news diversity metricsto existing well-being metrics such as voter turnout, and Facebook has already dem-onstrated a substantial effect on voter turnout by presenting users with personalizedmessages (Bond et al. 2012 ). It would be better to direct the optimization process
towards more closely related outcomes like polarization or tolerance that are notincluded in current well-being frameworks. Directly measuring these outcomes is
crucial because exposure to diverse opinions can actually increase polarization (Bail
et al. 2018 ). Polarization and tolerance outcomes are also explicitly relational, and thus
indicate aspects of community well-being not captured in individual-level metrics.
Low Carbon ShoppingLarge-scale product recommender systems have profound influence over what is
purchased. One reason for this is that it is not possible to navigate millions of possible
products without them. Rolnick et al. ( 2019 ) have proposed using these systems to454 International Journal of Community Well-Being (2020) 3:443 –463
direct consumers to lower-carbon alternatives. This possibility highlights two problems
that may arise in the course of modifying AI objective functions: obtaining the dataneeded to evaluate a metric and understanding the business impacts of such a change.
Climate change is a key issue for many communities (Fazey et al. 2018 ) and carbon
emissions appear in a number community well-being frameworks (Bagnall et al. 2017 ).
Carbon emissions from recommended products are also a key example of AI system
side effects on non-users. From a technical point of view, carbon footprint can be
incorporated using multi-stakeholder recommendation algorithms that explicitly con-sider the effect on parties other than the user (Abdollahpouri et al. 2020 ).
This is possible only if the carbon footprint of each product is available. There are
now established methods to estimate product carbon footprints (BSI 2011 ;I S O 2018 )
but there are no product carbon footprint (PCF) databases comprehensive enough tocover the millions of different products sold by a large online retailer. However, it maybe possible to use machine learning methods to estimate the PCF values of an entire
product portfolio starting from a comparatively small database of examples
(Meinrenken et al. 2012 ). Robust, scalable product carbon footprint estimation could
be a key enabling technology for low-carbon commerce and, ultimately, long-termcommunity well-being.
A commercial operator will want to know the business effects before any such
system is implemented, and it is tempting to evaluate the potential revenue effect ofincorporating a carbon term into the objective function by testing against historicalpurchase data. Such back-testing will show that optimizing for anything other than
profit must drive the system away from a profit maximum, but offline estimates will not
give the full story because both consumer and producer behavior may change if carbonfootprint starts to affect product ranking. Users might appreciate being informed oflow-carbon alternatives and buy more from that retailer or pay a premium for lowercarbon items, while producers will have an incentive to sell lower carbon products. Thecase of organic food demonstrates the existence of such market dynamics, as it is 22 –
35% more profitable globally than conventional alternatives even though it is typicallymore expensive to produce (Crowder and Reganold 2015 ).
Recommendations
The incorporation of community well-being metrics into both managerial and algorith-
mic optimization is a very general method for managing the effects of commercialoptimizing systems, yet good management is only part of good governance. Thissection synthesizes the analysis and discussion above with previous work on algorith-
mic governance, participatory design, best use of metrics, and corporate stakeholder
engagement to make recommendations for fostering community well-being in AIsystems in ways that are both effective and accepted as legitimate. It also identifiesgaps and unknowns where future research would be valuable.
Identifying and Involving CommunitiesAn attempt to optimize for community well-being is an attempt to benefit a particular
group of people, who need to have a say in what is done on their behalf. In some casesInternational Journal of Community Well-Being (2020) 3:443 –463 455
it would be reasonable to say that every user of the system (potentially billions of
people) is a member of the community, but that would preclude the management oflocal outcomes such as a system ’s effects on the residents of a particular city, or on
people of a certain age, or workers in particular professions. Non-users can be affectedas well, as in environmental externalities or a navigation system that routes cars to aformerly quiet street. Each view of community is a choice about who counts, and this
choice should be made explicit before any intervention begins.
Once a community is identified, there are many approaches to try to integrate its
members into the process of selecting and using metrics. Participatory design is anorientation and a set of practices that attempts to actively involve all stakeholders in asystem design process (Simonsen and Robertson 2012 ). It is a promising framework for
algorithmic governance. The WeBuildAI method (Lee et al. 2019 ) demonstrates what
participatory design of metrics might look like. Researchers worked with a food-delivery non-profit to design an algorithm to match donated food with volunteer drivers
and local food distribution charities. Stakeholders from each of these groups worked
with researchers to build quantitative models of their preferred trade-offs betweenfactors such as driver travel time, time since last donation, neighborhood poverty level,etc. At run time this system ranks the possible matches for each donation according tothe models representing the preferences of each stakeholder, with the final result chosenthrough a ranked-choice voting rule. Future work could investigate participatory metricdesign in the context of a large commercial platform.
There are both instrumental and political goals when attempting to integrate com-
munities into the selection and use of metrics. Without engaging the community, it is
not possible to know which aspects of well-being matter most to them and how seriousthese issues are, and therefore how to make tradeoffs. Engagement is also necessary forcredibility. When choosing community indicators, “most communities consider input
by its residents and others to be vital; it builds support for the use of indicators as wellas help vest those most impacted by subsequent actions in decision-making processes ”
(Sung and Phillips 2018 , p. 73). In the context of commercial systems it will also be
important to draw on the experience of corporate stakeholder engagement efforts such
as those found in sustainability reporting (GSSB 2016 ;M a n e t t i 2011 ).
Choosing MetricsAside from the well-known issues with using metrics in a management context
generally (Jackson 2005 ) metrics pose a problem for AI systems in particular because
most AI algorithms are based on strongly optimizing a narrow objective (Thomas andUminsky 2020 ). Poor use of metrics can result in a damaging emphasis on short term
outcomes, manipulation and gaming, and unwanted side effects (Jackson 2005 ;
Thomas and Uminsky 2020 ). Even a successful metric cannot remain static, as the
structure of the world it measures is constantly changing. In addition, there are manydomains without a clear consensus on well-being goals, necessitating a process ofnormative deliberation before metrics can be chosen. The following issues should beconsidered in choice of metrics:
Deciding What to Measure In many cases existing well-being metrics will not be
directly usable because they are too expensive to collect at scale or don ’t readily apply456 International Journal of Community Well-Being (2020) 3:443 –463
in the company ’s domain. These issues drove Facebook ’ss u b s t i t u t i o no f meaningful
social interactions for more general measures of user well-being. Creating a custom
metric is challenging because community well-being is a theoretical construct, not anobservable property, and there may be misalignment between the designer ’si n t e n t i o n s
and what is actually measured. For example, decreasing polarization measures may justindicate that minority voices have been effectively silenced. The particular well-being
aspect of interest must first be “operationalized ”and tested for reliability and validity
(Jacobs and Wallach 2019 ).
Long-Term Outcomes If a metric is evaluated only over the short term it may lead to
poor longer-term outcomes. As the YouTube case demonstrates, a video platform thattries to maximize user watch time may encourage binging behavior where userseventually regret the time they spend. While effective AI optimization requires frequentfeedback, it is critical to pick shorter-term metrics that are thought to drive longer-term
outcomes (Lalmas and Hong 2018 ).
Gaming Any measure that becomes a target will change meaning as people change
their behavior, a very general problem that is sometimes known as Goodhart ’sl a w
(Manheim and Garrabrant 2018 ). This is particularly relevant to large platforms that
must defeat adversarial efforts to gain exposure for financial or political ends. Whilethere are emerging methods to use causal inference to design metrics that resist gaming(Miller et al. 2019 ), a more robust solution is to continuously monitor and change the
metrics in use.
Dynamism The metrics employed need to be able to change and adapt, a property that
Jackson ( 2005 ) names dynamism . This is necessary because of gaming and other
behavior change in response to metrics, but more importantly the world can and doeschange; at the onset of the COVID-19 pandemic many existing machine learningmodels stopped working (Heaven 2020 ). Dynamism also avoids the serious problems
that can arise from over-optimization for a single objective, such as a robot which
injures humans in an attempt to fetch a coffee more quickly (Russell 2019 ). In the
context of contemporary commercial optimization, there are always humans supervis-ing and operating the AI system, and they are free to change the objective function asneeded.
Normative Uncertainty Catalogs such as IEEE 7010 (Schiff et al. 2020 ) provide a long
list of consensus metrics but not all of them will correspond to community needs, andnot all AI systems can be effectively evaluated using metrics originally designed for
public policy use. In short, many systems will face a lack of consensus around what a
“good”outcome would be. Appropriate values for AI systems cannot be derived from
first principles but must be the result of societal deliberation (Gabriel 2020 ), which
again underscores the necessity for participatory processes.International Journal of Community Well-Being (2020) 3:443 –463 457
Evaluating Outcomes
It may be very challenging to determine the actual well-being effects of incorporating a
metric into an optimization process. Facebook uses ongoing user panels to countmeaningful social interactions, but this is a narrow facet of user well-being, let alone
community well-being. They could use broader well-being instruments such as a life
satisfaction survey question, but it would be difficult to assess the causal contribution of
Facebook use to any changes. In other cases, such as the diverse news recommender,pre-existing well-being indicators would not apply so assessing societal impact wouldrequire the creation and validation of new community well-being metrics.
Outcome evaluation at scale is essentially corporate social science. The IEEE 7010
Recommended Practice Standard for Assessing the Impact of Autonomous and Intel-ligent Systems on Human Well-Being proposes what amounts to a difference-in-
differences design between users and non-users before and after an algorithmic change
(Schiff et al. 2020 ). This is a promising approach, but there do not seem to be any
published examples.
Business ImplicationsFor commercial AI systems, metrics-driven changes must also integrate legitimate
business concerns such as the cost of implementation and the effects on businessoutcomes. Although a naïve analysis of multi-objective optimization suggests that
considering anything other than revenue can only reduce revenue, this assumes every-
thing else is equal. In reality there are complex secondary effects, such as changes inuser and supplier behavior. YouTube ’s experience demoting clickbait videos is a
documented case where doing the responsible thing led to a short-term decrease in
the primary watch time metric, but then a long-term increase. It is difficult to predict thefinancial effects of incorporating well-being into optimization. Business and socialobjectives may turn out to be aligned, but this cannot be expected to be true as a rule.While ethical outcomes can sometimes be achieved through changes to optimization
goals, there are also situations that could conceivably require avoiding features,
products, or business models altogether (Barocas et al. 2020 ).
Case studies are one promising avenue for progress on the problem of uncertain
business outcomes. If companies are already in corporating well-being metrics into their
management and algorithms then documenting these cases will let others learn from theirexperiences, develop the field, and normalize the idea that companies should proactivelymanage the effects of their optimizers. This un derscores the need for t ransparency around
work that is explicitly designed to impro ve the lives of great numbers of people.
Conclusion
This paper has explored the integratio n of community well-being metrics into
commercially-operated optimizing systems. Community well-being is an attractive goalbecause it is well-developed in public policy contexts and practically measurable. Atleast two large technology companies, Facebook and YouTub e, have explicitly458 International Journal of Community Well-Being (2020) 3:443 –463
modified their objective functions in pursuit of well-being, demonstrating the practi-
cality of this approach.
There are still a number of weaknesses in the interventions that Facebook and
YouTube have undertaken, at least in terms of what has been reported publicly.The community that these interventions are intended to serve has not been welldefined; rather, these metrics and interven tions are oriented towards the individual
level and do not account for existing communities such as cities or discussion
groups. It is not clear if or how users were engaged in selecting the meaningful
social interactions anduser satisfaction metrics; there is no report of the outcomes
either in terms of these metrics or with res pect to broader well-being metrics; and
although both companies reported reduced short term engagement, the broaderbusiness effects have not been discussed.
However incomplete, the Facebook and YouTube cases suggest that the optimiza-
tion of community well-being metrics may be a powerful general method for managing
the societal outcomes of commercial AI systems. The same methods could be applied
to many other types of systems, such as a news recommender system that incorporatesmeasures of content diversity in an attempt to increase tolerance and reduce polariza-tion, or an online shopping platform that uses product-level estimates of carbonfootprint to steer users toward more environmentally friendly purchases. Althoughmany scholars and critics have stressed the importance of increased user control overAI systems, no amount of user control can replace appropriate well-being metrics dueto issues of collective action and the need for reasonable defaults.
An analysis of the above cases suggests that the following multi-step process may be
effective:
Identify a community to define the scope of action. In online settings this may be a
challenging decision.
Select a well-being metric, perhaps from existing frameworks. This stage frames
the problem to be solved in concrete terms, so it may be where community
involvement matters most.
Use this metric as a performance measure for the team building and operating the
system.
Directly translate the metric into code as a modification to an algorithmic objective
function or use these measured outcomes to evaluate more general design changes.
Evaluate the results, in terms of actual human outcomes, and adjust accordingly.
This may require adjusting the chosen metric in response to changing conditions,or if it is found to be causing side effects of its own.
Require transparency throughout to make participation possible and to hold
companies accountable to the communities who are meant to be served by thisprocess.International Journal of Community Well-Being (2020) 3:443 –463 459
Funding The author is an employee of Partnership on AI. Partnership on AI is supported by donations from
companies and philanthropies, including Facebook and Google. The author did not receive funding fromFacebook or Google for the creation of this article.
Data Availability N/ACode AvailabilityN/A
Compliance with Ethical Standards
Conflict of Interest The author declares that they have no conflicts of interest.
Ethics Approval This paper does not include any studies with human participants or animals performed by
the author.
References
Abdollahpouri, H., Adomavicius, G., Burke, R., Guy, I., Jannach, D., Kamishima, T., Krasnodebski, J., &
Pizzato, L. (2020). Multistakeholder recommendation: survey and research directions. User Modeling and
User-Adapted Interaction, 30 (1), 127 –158. https://doi.org/10.1007/s11257-019-09256-1 .
Andreassen, C. S. (2015). Online social network site addiction: a comprehensive review. Current Addiction
Reports, 2 (2), 175 –184. https://doi.org/10.1007/s40429-015-0056-9 .
Bagnall, A., South, J., Mitchell, B., Pilkington, G., & Newton, R. (2017). Systematic scoping review of
indicators of community wellbeing in the UK .1–71.http://eprints.leedsbeckett.ac.uk/5238/1/community-
wellbeing-indicators-scoping-review-v1-2-aug2017.pdf .
Bail, C. A., Argyle, L. P., Brown, T. W., Bumpus, J. P., Chen, H., Fallin Hunzaker, M. B., Lee, J., Mann, M.,
Merhout, F., & Volfovsky, A. (2018). Exposure to opposing views on social media can increase politicalpolarization. Proceedingss of the National Academy of Sciences of the United States of America, 115 (37),
9216–9221. https://doi.org/10.1073/pnas.1804840115 .
Barocas, S., Hardt, M., & Narayanan, A. (2018). Fairness and Machine Learning .http://fairmlbook.org
Barocas, S., Biega, A. J., Fish, B., Niklas, J., & Stark, L. (2020). When not to design, build, or deploy.
Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , 695–695. https://
doi.org/10.1145/3351095.3375691 .
Baum, S. D. (2020). Social choice ethics in artificial intelligence. AI and Society, 35 (1), 165 –176. https://doi.
org/10.1007/s00146-017-0760-1 .
Baxter, G., & Sommerville, I. (2011). Socio-technical systems: from design methods to systems engineering.
Interacting with Computers, 1 ,4–17.https://doi.org/10.1016/j.intcom.2010.07.003 .
Bergen, M. (2019). YouTube executives ignored warnings, Let Toxic Videos Run Rampant. Bloomberg .
https://www.bloomberg.com/news/features/2019-04-02/youtube-executives-ignored-warnings-letting-toxic-videos-run-rampant .
Bernstein, A., de Vreese, C., Helberger, N., Schulz, W., Zweig, K., Baden, C., Beam, M. A., Hauer, M. P.,
Heitz, L., Jürgens, P., Katzenbach, C., Kille, B., Klimkiewicz, B., Loosen, W., Moeller, J., Radanovic, G.,Shani, G., Tintarev, N., Tolmeijer, S., …Zueger, T. (2020). Diversity in News Recommendations .http://
arxiv.org/abs/2005.09495 .
Bond, R. M., Fariss, C. J., Jones, J. J., Kramer, A. D. I., Marlow, C., Settle, J. E., & Fowler, J. H. (2012). A 61-
million-person experiment in social influence and political mobilization. Nature, 489 (7415), 295 –298.
https://doi.org/10.1038/nature11421 .
BSI. (2011). PAS 2050:2011 Specification for the assessment of the life cycle greenhouse gas emissions of
goods and services .https://shop.bsigroup.com/en/Browse-By-Subject/Environmental-Management-and-
Sustainability/PAS-2050/ .
Budak, C., Goel, S., & Rao, J. M. (2016). Fair and balanced? Quantifying media bias through crowdsourced
content analysis. Public Opinion Quarterly, 80 (Specialissue1), 250 –271. https://doi.org/10.1093/poq/
nfw007 .460 International Journal of Community Well-Being (2020) 3:443 –463
Chen, R., Hua, Q., Chang, Y. S., Wang, B., Zhang, L., & Kong, X. (2018). A survey of collaborative filtering-
based recommender systems: from traditional methods to hybrid methods based on social networks. IEEE
Access, 6 , 64301 –64320. https://doi.org/10.1109/ACCESS.2018.2877208 .
Crowder, D. W., & Reganold, J. P. (2015). Financial competitiveness of organic agriculture on a global scale.
Proceedings of the National Academy of Sciences of the United States of America, 112 (24), 7611 –7616.
https://doi.org/10.1073/pnas.1423674112 .
Dantzig, G. B. (1982). Reminiscences about the origins of linear programming. Operations Research Letters,
1(2), 43 –48.https://doi.org/10.1016/0167-6377(82)90043-8 .
Delgado, J., Lind, S., Radecke, C., & Konijeti, S. (2019). Simple objectives work better. Workshop on
Recommendation in Multi-stakeholder Environments, RecSys. http://ceur-ws.org/Vol-2440/paper5.pdf .
Diener, B. E., Oishi, S., & Tay, L. (2018). Handbook of well-being . DEF Publishers. nobascholar.com .
Doerr, J. E. (2017). Measure what matters: How Google, Bono, and the Gates Foundation rock the world with
OKRs . Portfolio Penguin.
Donahoe, E., & Metzger, M. M. (2019). Artificial intelligence and human rights. Journal of Democracy,
30(2), 115 –126. https://doi.org/10.1353/jod.2019.0029 .
Durand, M. (2015). The OECD better life initiative: How ’s life? And the measurement of well-being. Review
of Income and Wealth, 61 (1), 4–17.https://doi.org/10.1111/roiw.12156 .
Exton, C., & Shinwell, M. (2018). Policy use of well-being metrics: describing countries ’experiences. OECD
Statistics Working Papers, 33 (94). https://doi.org/10.1787/d98eb8ed-en .
Facebook. (2018). Facebook, Inc. (FB) Fourth Quarter and Full Year 2017 Results Conference Call .https://
investor.fb.com/ .
Fazey, I., Carmen, E., Chapin, F. S., Ross, H., Rao-Williams, J., Lyon, C., Connon, I. L. C., Searle, B. A., &
Knox, K. (2018). Community resilience for a 1.5 °C world. Current Opinion in Environmental
Sustainability, 31 ,3 0–40.https://doi.org/10.1016/j.cosust.2017.12.006 .
Floridi, L., & Cowls, J. (2019). A unified framework of five principles for AI in society. Harvard Data
Science Review, 1 (1).https://doi.org/10.1162/99608f92.8cd550d1 .
Foster, J., & Sen, A. (1997). On economic inequality . Clarendon Press.
Frey, B. S., Benesch, C., & Stutzer, A. (2007). Does watching TV make us happy? J o u r n a lo fE c o n o m i c
Psychology, 28 (3), 283 –313. https://doi.org/10.1016/j.joep.2007.02.001 .
Gabriel, I. (2020). Artificial intelligence, values, and alignment .https://arxiv.org/abs/2001.09768 .
Garimella, V. R. K., & Weber, I. (2017). A long-term analysis of polarization on twitter. Proceedings of the
11th international conference on web and social media, ICWSM 2017, 528–531. https://www.aaai.org/
ocs/index.php/ICWSM/ICWSM17/paper/view/15592 .
Ginsberg, D., & Burke, M. (2017). Hard questions: Is spending time on social media bad for us? Facebook.
https://about.fb.com/news/2017/12/hard-questions-is-spending-time-on-social-media-bad-for-us/ .
Google. (2019). How Google Fights Disinformation .https://www.blog.google/documents/37/How\_Google\_
Fights\_Disinformation.pdf .
GSSB. (2016). Global Reporting Inititative 101: Foundation .https://www.globalreporting.org/standards/gri-
standards-download-center/gri-101-foundation-containing-standard-interpretation-1/ .
Heaven, W. D. (2020). Our weird behavior during the pandemic is messing with AI models . MIT Technology
Review. https://www.technologyreview.com/2020/05/11/1001563/covid-pandemic-broken-ai-machine-
learning-amazon-retail-fraud-humans-in-the-loop/ .
Helberger, N. (2019). On the democratic role of news recommenders. Digital Journalism, 7 (8), 993 –1012.
https://doi.org/10.1080/21670811.2019.1623700 .
Helberger, N., Karppinen, K., & D ’Acunto, L. (2018). Exposure diversity as a design principle for recom-
mender systems. Information Communication and Society, 21 (2), 191 –207. https://doi.org/10.1080/
1369118X.2016.1271900 .
ISO. (2018). Greenhouse gases —Carbon footprint of products —Requirements and guidelines for
quantification (ISO 14067:2018) . International Organization for Standardization. https://www.iso.org/
standard/71206.html .
Jackson, A. (2005). Falling from a great height: principles of good practice in performance measurement and
the perils of top down determination of performance indicators. Local Government Studies, 31 (1), 21 –38.
https://doi.org/10.1080/0300393042000332837 .
Jacobs, A. Z., & Wallach, H. (2019). Measurement and Fairness .http://arxiv.org/abs/1912.05511 .
Jannach, D., & Adomavicius, G. (2016). Recommendations with a purpose. RecSys 2016 - Proceedings of the
10th ACM Conference on Recommender Systems ,7–10.https://doi.org/10.1145/2959100.2959186 .
Kaplan, R. S. (2009). Conceptual foundations of the balanced scorecard. Handbook of Management
Accounting Research, 3 , 1253 –1269. https://doi.org/10.1016/S1751-3243(07)03003-9 .International Journal of Community Well-Being (2020) 3:443 –463 461
Kievit, R. A., Frankenhuis, W. E., Waldorp, L. J., & Borsboom, D. (2013). Simpson ’s paradox in psycho-
logical science: a practical guide. Frontiers in Psychology, 4 (August), 1 –14.https://doi.org/10.3389/
fpsyg.2013.00513 .
Korinek, A., & Stiglitz, J. E. (2017). Artificial intelligence and its implications for income distribution and
unemployment. In National Bureau of Economic Research .https://doi.org/10.7208/chicago/
9780226613475.003.0014 .
Kulynych, B., Overdorf, R., Troncoso, C., & Gürses, S. (2020). POTs: Protective Optimization Technologies.
FAT\* ‘20: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency .https://
doi.org/10.1145/3351095.3372853 .
Kunaver, M., & Po žrl, T. (2017). Diversity in recommender systems –A survey. Knowledge-Based Systems,
123,1 5 4–162. https://doi.org/10.1016/j.knosys.2017.02.009 .
Lalmas, M., & Hong, L. (2018). Tutorial on metrics of user engagement: Applications to news, search and E-
commerce. WSDM 2018 - Proceedings of the 11th ACM International Conference on Web Search and
Data Mining, 3, 781 –782. https://doi.org/10.1145/3159652.3162010 .
Ledwich, M., & Zaitsev, A. (2020). Algorithmic extremism: Examining YouTube ’s rabbit hole of radicaliza-
tion.https://doi.org/10.5210/fm.v25i3.10419 .
Lee, M. K., Kusbit, D., Kahng, A., Kim, J. T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas,
A., & Procaccia, A. D. (2019). Webuildai: Participatory framework for algorithmic governance.
Proceedings of the ACM on Human-Computer Interaction .https://doi.org/10.1145/3359283 .
Manetti, G. (2011). The quality of stakeholder engagement in sustainability reporting: empirical evidence and
critical points. Corporate Social Responsibility and Environmental Management, 18 ,1 1 0–122. https://
doi.org/10.1002/csr.255 .
Manheim, D., & Garrabrant, S. (2018). Categorizing Variants of Goodhart ’sL a w .1–10.http://arxiv.org/abs/
1803.04585 .
Meinrenken, C. J., Kaufman, S. M., Ramesh, S., & Lackner, K. S. (2012). Fast carbon footprinting for large
product portfolios. Journal of Industrial Ecology, 16 (5), 669 –679. https://doi.org/10.1111/j.1530-9290.
2012.00463.x .
Milano, S., Taddeo, M., & Floridi, L. (2019a). Ethical aspects of multi-stakeholder recommendation systems .
https://ssrn.com/abstract=3493202 .
Milano, S., Taddeo, M., & Floridi, L. (2019b). Recommender systems and their ethical challenges .https://
ssrn.com/abstract=3378581 .
Miller, J., Milli, S., & Hardt, M. (2019). Strategic classification is causal modeling in disguise .http://arxiv.
org/abs/1910.10362 .
Mosseri, A. (2018). Bringing people closer together .F a c e b o o k . https://about.fb.com/news/2018/01/news-
feed-fyi-bringing-people-closer-together/ .
Musikanski, L., Phillips, R., & Jean Crowder. (2019). The happiness policy handbook: How to make
happiness and well-being the purpose of your government . Gabriola: New Society Publishers.
Musikanski, L., Rakova, B., Bradbury, J., Phillips, R., & Manson, M. (2020). Artificial intelligence and
community well-being: a proposal for an emerging area of research. International Journal of Community
Well-Being, 3 ,3 9–55.https://doi.org/10.1007/s42413-019-00054-6 .
O’Donnell, G., Deaton, A., Durand, M., Halpern, D., & Layard, R. (2014). Wellbeing and Policy .L e g a t u m
Institute. https://li.com/reports/the-commission-on-wellbeing-and-policy/ .
OECD. (2019a). Artificial intelligence in society . OECD Publishing. https://doi.org/10.1787/eedfee77-en .
OECD. (2019b). Measuring well-being and progress .https://www.oecd.org/sdd/OECD-Better-Life-Initiative.
pdf.
Ostrom, E. (2000). Collective action and the evolution of social norms. Journal of Economic Perspectives,
14(3), 137 –158. https://doi.org/10.1257/jep.14.3.137 .
Paraschakis, D. (2017). Towards an ethical recommendation framework. Proceedings of the International
Conference on Research Challenges in Information Science 2017 ,2 1 1–220. https://doi.org/10.1109/
RCIS.2017.7956539 .
Parmenter, D. (2020). Key performance indicators: Developing, implementing, and using winning KPIs (4th
ed.). Wiley.
Phillips, R., & Pittman, R. H. (Eds.). (2015). An introduction to community development .R o u t l e d g e .
Rahwan, I. (2018). Society-in-the-loop: programming the algorithmic social contract. Ethics and Information
Technology, 20 (1), 5–14.https://doi.org/10.1007/s10676-017-9430-8 .
Reader, B., & Hatcher, J. A. (Eds.). (2011). Foundations of community journalism . SAGE Publications Inc.
Richardson, J. (2013). Accounting for sustainability. In A. Henriques & J. Richardson (Eds.), The triple
bottom line - Does it all add up? (pp. 34 –44). Routledge.462 International Journal of Community Well-Being (2020) 3:443 –463
Robertson, S., & Salehi, N. (2020). What if I don ’t like any of the choices? The limits of preference elicitation
for participatory algorithm design. Participatory Approaches to Machine Learning Workshop, ICML
2020 .http://arxiv.org/abs/2007.06718 .
Roitero, K., Carterette, B., Mehrotra, R., & Lalmas, M. (2020). Leveraging behavioral heterogeneity across
markets for cross-market training of recommender systems .6 9 4–702. https://doi.org/10.1145/3366424.
3384362 .
Rolnick, D., Donti, P. L., Kaack, L. H., Kochanski, K., Lacoste, A., Sankaran, K., Ross, A. S., Milojevic-
Dupont, N., Jaques, N., Waldman-Brown, A., Luccioni, A., Maharaj, T., Sherwin, E. D., Mukkavilli, S.K . ,K o r d i n g ,K .P . ,G o m e s ,C . ,N g ,A .Y . ,H a s s a b i s ,D . ,P l a t t ,J .C . , …Bengio, Y. (2019). Tackling
climate change with machine learning .http://arxiv.org/abs/1906.05433 .
Russell, S. (2019). Human compatible: Artificial intelligence and the problem of control .V i k i n g .
Russell, S., & Norvig, P. (2010). Artificial intelligence: A modern approach 3rd edition . Prentice Hall.
Samuelson, W. F., & Marks, S. G. (2014). Managerial economics (8th ed.). Wiley.
Schiff, D., Murahwi, Z., Musikanski, L., & Havens, J. C. (2019). A New Paradigm for Autonomous and
Intelligent Systems Development : Why Well-being Measurement Matters. Workshop on Designing
Digital Wellbeing, CHI 2019 .https://digitalwellbeingworkshop.files.wordpress.com/2019/04/02-
wellbeing\_measurement\_schiff\_murahwi\_musikanski\_havens.pdf .
Schiff, D., Ayesh, A., Musikanski, L., & Havens, J. C. (2020). IEEE 7010: A New Standard for Assessing the
Well-Being Implications of Artificial Intelligence .http://arxiv.org/abs/2005.06620 .
Simonsen, J., & Robertson, T. (2012). Routledge international handbook of participatory design. In Routledge
international handbook of participatory design (1st Editio). Routledge. https://doi.org/10.4324/
9780203108543 .
Stoica, A.A., & Chaintreau, A. (2019). Hegemony in social media and the effect of recommendations. The
Web Conference 2019 ,2 ,5 7 5 –580. https://doi.org/10.1145/3308560.3317589 .
Stray, J., Adler, S., & Hadfield-Menell, D. (2020). What are you optimizing for ? Aligning Recommender
Systems with Human Values. Participatory Approaches to Machine Learning Workshop, ICML 2020 .
https://participatoryml.github.io/papers/2020/42.pdf .
Sung, H., & Phillips, R. G. (2018). Indicators and community well-being: exploring a relational framework.
International Journal of Community Well-Being, 1 (1), 63 –79.https://doi.org/10.1007/s42413-018-0006-0 .
Thomas, R. L., & Uminsky, D. (2020). Reliance on metrics is a fundamental challenge for AI. Ethics of Data
Science Conference .https://arxiv.org/abs/2002.08512 .
Verduyn, P., Ybarra, O., Résibois, M., Jonides, J., & Kross, E. (2017). Do social network sites enhance or
undermine subjective well-being? A critical review. Social Issues and Policy Review, 11 (1), 274 –302.
https://doi.org/10.1111/sipr.12033 .
Wojcicki, S. (2019). Preserving openness through responsibility . Inside YouTube Blog. https://blog.youtube/
inside-youtube/preserving-openness-through-responsibility .
Yu, T., Shen, Y., & Jin, H. (2019). A visual dialog augmented interactive recommender system. Proceedings
of the ACM SIGKDD international conference on knowledge discovery and data mining
, 157–165.
https://doi.org/10.1145/3292500.3330991 .
Zhao, Z., Chi, E., Hong, L., Wei, L., Chen, J., Nath, A., Andrews, S., Kumthekar, A., Sathiamoorthy, M., &
Yi, X. (2019). Recommending what video to watch next: a multitask ranking system. Proceedings of the
13th ACM Conference on Recommender Systems (RecSys ‘19),4 3–51.https://doi.org/10.1145/3298689.
3346997 .
Zuckerberg, M. (2018). No Title .https://www.facebook.com/zuck/posts/10104413015393571 .
Publisher ’sN o t e Springer Nature remains neutral with regard to jurisdictional claims in published maps and
institutional affiliations.International Journal of Community Well-Being (2020) 3:443 –463 463


Feasibility of Training an AGI using Deep RL:
A Very Rough Estimate
Baeo Maltinsky Jack Gallagher Jessica Taylor
March 24, 2019
1 Introduction
Several months ago, we were presented with a scenario for how artiﬁcial general intelligence
(AGI) may be achieved in the near future. We found the approach surprising, so we attempted
to produce a rough model to investigate its feasibility. This document presents the model and its
conclusions.
The usual cliches about the folly of trying to predict the future go without saying and this
shouldn’t be treated as a rigorous estimate, but hopefully it can give a loose, rough sense of some
of the relevant quantities involved. The notebook and the data used for it can be found in the
Median Group numbers GitHub repo if the reader is interested in using different quantities or
changing the structure of the model. If the reader is interested in a more general approach based
on the rate of theoretical progress, see our Insights model.
To brieﬂy explain the proposal:
•Create a simulated world somewhat similar to our world (i.e, containing space-time, mate-
rial, programming, multiple agents, etc).
•Simulate many agents, each with an amount of computing power equal to the human brain.
These agents interact in a manner similar to that of evolution; the most ﬁt survive and re-
produce.
•At the end, a collection of the most ﬁt individuals are given data about our world and asked
to solve problems for rewards (this is really not a good way to do AI alignment but can
plausibly produce misaligned AGI).
This does not take into account:
•serial vs. parallel training
•FLOPS vs. other considerations (memory, TEPS, latency)
•breakdown of exponential price performance improvement
•costs associated with things other than a subset of the training hardware (there are many
others)
•time required to develop software
•difﬁculty of creating and running a simulated world
•paradigm shifts in AI (e.g. no longer doing reinforcement learning) that would make this
scenario irrelevant
•whether “difﬁculty ratio” relative to Go makes sense (it probably doesn’t except in the loos-
est sense)
1
1.1 A Note about Uncertainty and Distributions
Much of the challenge associated with modelling this scenario derives from the difﬁculty of esti-
mating (or even bounding) many of the parameters. When possible, we use distributions or values
corresponding to commonly accepted quantities. In the cases where there are no such quantities,
we attempt to construct reasonable distributions with very wide bounds to reﬂect the uncertainty.
If reasonable conﬁdence bounds can be established (these terms are meant loosely), we will
use a generally log-uniform distribution.
In [ 51]:%matplotlib inline
import matplotlib
import numpy asnp
import pandas aspd
from matplotlib import pyplot asplt
import seaborn assns
from IPython .display import display, HTML
import warnings
# Suppress Pandas plotting error
warnings .filterwarnings( "ignore ")
# plot aesthetics
sns.set()
# number of samples for Monte Carlo simulation
n=50000
# summary table of distribution
def summary (data, unit =''):
"""
Returns Pandas dataframe summarizing distribution of data with percentiles
Mostly for convenience
Paramaeters
-----------
data : array\_like
Input array or array-like object
unit : str
Units associated with data
Returns
-------
out : pandas.DataFrame
"""
percentiles =[1,5,25,50,75,95,99]
values =np.percentile(data, percentiles)
2
df=pd.DataFrame(data ={"Percentile ": percentiles, unit: values})
# df = df.concat(["Mean", np.mean(data)])
return df[[ "Percentile ", unit]]
# log-uniform distribution
def loguniform (low =0, high =1, size =None ):
"""
Returns loguniform distribution bounded by low and high orders of magnitude.
Parameters
----------
low : float, optional
Order-of-magnitude lower boundary of the output interval.
All values generated will be greater than or equal to low.
The default value is 0.
high : float, optional
Order-of-magnitude upper boundary of the output interval.
All values generated will be less than high.
The default value is 1.
size : int or tuple of ints, optional
Output shape
Returns
-------
out : ndarray or scalar
samples from the loguniform distribution
"""
return 10\*\*np.random .uniform(low, high, size)
1.2 Computational Performance of the Brain
The computational performance of the brain will be estimated in operations per second. This is a
convenient but incredibly ﬂawed approach .Sandberg and Bostrom (2009) contains a summary of
estimates from the literature:
•Tuszynski (2006): 1028FLOPS
•Kurzweil (1999): 21016FLOPS
•Thagard (2002): 1023FLOPS
•Bostrom (1998): 1017operations per second
•Merkle (1989): 21015operations per second
•Merkle (1989): 1013operations per second
•Dix (2005): 1016synaptic operations per second
•Seitz (2007): 21016synaptic operations per second (note: the Bostrom paper incorrectly reports
this as 21012, see the original )
•Malickas (1996): 1016synaptic operations per second
Martins et al. 2012 offers a few more estimates:
•Moravec (1998): 1014FLOPS
3
•Sarpeshka (1998): 3.61015synaptic operations per second
The estimates from Tuszynski (2006) and Thagard (2002) are derived from speculations about
quantum computation occurring in the brain. As this is usually regarded as biologically implau-
sible, they will be excluded from the data set.
Note: the smaller estimate by Merkle (1989) seems to over-estimate both the capacitance of
axons and how completely that capacitance is discharged when conducting an action potential,
but it will be included anyway.
In [ 33]:brain\_flops =np.log10([ 2e16 ,1e17 ,1e13 ,2e15 ,1e16 ,2e16 ,1e14 ,3.6e15 ])
plt.hist(brain\_flops, density =True );
plt.xlabel( "FLOPS (log-scale) ");
plt.ylabel( "Frequency ");
These estimates are typically generated by multiplying uncertain quantities, so we expect the
resulting distribution to resemble a mixture of Gaussians in log-space. We will use a kernel density
estimator to produce a distribution from the estimates, with the bandwidth hyperparameter tuned
empirically using cross-validation.
In [ 34]:from sklearn .neighbors import KernelDensity
from sklearn .model\_selection import GridSearchCV
from sklearn .model\_selection import LeaveOneOut
x=np.linspace( 10,20).reshape( -1,1)
brain\_flops =brain\_flops .reshape( -1,1)
4
bandwidths =10\*\*np.linspace( -1,1)
grid =GridSearchCV(KernelDensity(kernel ='gaussian '),
{'bandwidth ': bandwidths},
cv=LeaveOneOut())
grid .fit(brain\_flops);
bw=grid .best\_params\_[ 'bandwidth ']
kde =KernelDensity(bandwidth =bw, kernel ='gaussian ')
kde.fit(brain\_flops)
brain\_compute =10\*\*kde.sample(n)[:, 0]
plt.hist(brain\_flops,
bins =5,
histtype ='stepfilled ',
density =True ,
label ="Estimates ");
plt.plot(x, np .exp(kde .score\_samples(x)), label ="KDE");
plt.xlabel( "Operations per second (log) ");
plt.ylabel( "Frequency ");
plt.legend()
summary(brain\_compute, "Operations per second ")
Out[ 34]: Percentile Operations per second
0 1 2.926178e+11
1 5 3.113016e+12
2 25 2.307572e+14
3 50 4.015672e+15
4 75 3.945256e+16
5 95 7.642728e+17
6 99 5.129267e+18
5
1.3 Subjective-Lifespan of AI
How long does each AI have to live (in subjective time) in order to become generally competent
enough to take signiﬁcant strategic action in the world, and then actually take this action? This
could be comparable to human lifetimes or substantially shorter. We will err towards the side of
“substantially shorter”.
In [ 35]:from scipy .stats import lognorm
# longevity in days
longevity =loguniform(high =2, low =-.5, size =n)
plt.hist(longevity, bins =50, log =True );
plt.xlabel( 'Lifetime (years) ');
plt.ylabel( 'Frequency (log-plot) ');
summary(longevity, 'Years ')
Out[ 35]: Percentile Years
0 1 0.334516
1 5 0.417490
2 25 1.313593
3 50 5.474537
4 75 23.136432
5 95 74.352380
6 99 93.907044
6
2 Comparisons to AlphaGo Zero
2.0.1 AlphaGo Zero Training Time
This model uses DeepMind’s AlphaGo Zero as a benchmark, which beat the superhuman Go AI
AlphaGo Lee 36 hours into its initial three-day training period. AlphaGo Zero played ~4.9 million
games during this period, so AlphaGo Zero obtained superhuman performance after roughly 2.5
million games.
In [ 36]:# number of games played by AlphaGo Zero
agz\_games =2.5e6
2.0.2 Sample Efﬁciency of General Intelligence Compared to Go
A ratio of one would mean that, in the future, AI researchers discover an AI architecture that is
as efﬁcient ag learning general action in the world as AlphaZero is at learning Go. A ratio of
1000 would mean it takes 1000 times as many lifetimes per unit of learning compared with Alp-
haZero. It is plausible that useful architectures could be learned from neuroscience, although this
would require large advances in the neuroscience ﬁeld (see Could a Neuroscientist Understand a
Microprocessor? ). We can think of this as a difﬁculty ratio .
This parameter is essentially impossible to estimate in a principled way, so we’ll be a bit sloppy.
To begin thinking about this, consider that AlphaGo Zero takes as input a 19-by-19 Go board (and
the previous 16 board states). Each square can take one of three states (black, white, or empty)
and there can be at most 19192=722moves in a game, so each game provides at most
7
722log2(31919)4.1105bits of information. By comparison, the human optic nerve carries
6106bits per second ( see Raichle (2010) ), or51011bits in a 24-hour period. Natural
imagery and Go boards are both highly compressible, but a difference of this size suggests that we
should probably be very surprised if the difﬁculty ratio were small.
It’s hard to tell whether or not our particular distribution is reasonable, but it’s worth noting
that simple neural networks have been around a very long time. Estimates derived from molecular
clock studies on homologous genes suggest that sponges, jellyﬁsh, and cone jellies diverged in the
Mesoproterozoic over one billion years ago. Quite a lot of computation can happen over long
geological time scales.
In [ 37]:difficulty =loguniform(low =5, high =37, size =n)
plt.hist(difficulty, bins =50, log =True );
plt.xlabel( 'Difficulty Ratio ');
plt.ylabel( 'Frequency (log-scale) ');
summary(difficulty, 'Difficulty Ratio ')
Out[ 37]: Percentile Difficulty Ratio
0 1 2.054450e+05
1 5 4.017578e+06
2 25 1.029188e+13
3 50 1.008063e+21
4 75 1.171638e+29
5 95 2.907911e+35
6 99 5.188879e+36
8
2.0.3 Number of Hyperparameter Tuning Attempts
The AlphaGo Zero paper implies that a few different variations were trained in parallel during the
initial training period, as is common practice for large deep learning projects. We should expect
something similar in the scenario under consideration.
In [ 38]:hyperparameters =loguniform(high =1.5, low =0, size =n)
plt.hist(hyperparameters, bins =50);
plt.xlabel( "Number of Tuning Attempts ");
plt.ylabel( "Frequency ");
summary(hyperparameters, 'Tuning Attempts ')
Out[ 38]: Percentile Tuning Attempts
0 1 1.033877
1 5 1.184992
2 25 2.368868
3 50 5.646292
4 75 13.410038
5 95 26.650292
6 99 30.568390
2.1 Future Hardware Improvements
2.1.1 Trends in Computing Costs
While machine learning ASICs like TPUs are likely the future, the recent deep learning boom
was powered by GPUs. Commonly mentioned ﬁgures for the price-performance generalization
9
of Moore’s Law suggest that price-performance doubles roughly every two years, but it’s worth
checking this ﬁgure against the observed price-performance changes in GPUs. The ﬁle gpu.csv
contains the model name, launch date, single precision performance in GFLOPS, and release price
in non-inﬂation-adjusted US dollars for 223 Nvidia and AMD GPUs (scraped from Wikipedia).
The dataset covers almost two decades, so we will need to adjust the prices for inﬂation using the
Consumer Price Index .
This notably neglects several important features:
•Much of the work that has been done in the past several years has been focused on reducing
energy usage, which we don’t attempt to capture with our focus on naive price-performance.
•The large proportion of the cost of running a data center today comes from electricity usage,
not ﬁxed capital like computing hardware.
•We are not considering the costs of other components, or attempting to project changes in
their price performance.
•This does not consider the labor and engineering costs for such a project, which will almost
certainly be very substantial.
However, it’s an easy quantity to work with and provides some insight.
In [ 39]:gpu =pd.read\_csv( "gpu.csv ")
gpu[ 'Date ']=pd.to\_datetime(gpu[ 'Date '], infer\_datetime\_format =True )
cpi =pd.read\_csv( "cpi.csv ")
cpi[ 'Date ']=pd.to\_datetime(cpi[ 'Date '], infer\_datetime\_format =True )
# Price multiplier to convert to 2018$
cpi[ 'Price Multiplier ']=cpi[ 'CPI'].iloc[ -1]/cpi[ 'CPI']
gpu =pd.merge\_asof(
gpu, cpi .filter(
items =['Date ','Price Multiplier ']),
on='Date ',
direction ='nearest ')
# Adjust prices for inflation
gpu[ 'Adjusted Price (2018$) ']=gpu[ 'Release Price (USD) ']\*gpu[ 'Price Multiplier ']
# Inflation adjusted performance cost
gpu[ 'FLOPS per 2018$ ']=gpu[ 'GFLOPS ']/gpu[ 'Adjusted Price (2018$) ']\*10\*\*9
# Convert dates to unix time
time, y =gpu[ 'Date '].astype(np .int64) /10\*\*9, gpu[ 'FLOPS per 2018$ ']
# Linear regression in log-space
#rate, constant = np.polyfit(time, np.log(y), 1, w=np.sqrt(y))
rate, constant =np.polyfit(time, np .log(y), 1)
gpu[ 'Fit Exponential ']=pd.Series(np .exp(constant) \*np.exp(rate \*time))
gpu.plot(x ='Date ', y=['FLOPS per 2018$ ','Fit Exponential '], style =['.','-'])
#plt.savefig( 'gpu\_full.png ')
10
# Pearson 's r (correlation coefficient)
print ('r = {:.3f} '.format(gpu[ 'FLOPS per 2018$ '].corr(gpu[ 'Fit Exponential '])))
r = 0.716
This implies a doubling time of ~1.5 years.
In [ 40]:doubling\_time =1/(np.log2(np .e)\*rate \*3600 \*24\*365)
print ("Doubling time: {:.2f} years ".format(doubling\_time))
Doubling time: 1.48 years
It should be noted that this is somewhat misleading because the price-performance curve
isn’t a clean exponential. An exponential curve appears linear in log-space, whereas the price-
performance curve is sub-linear. Inspecting a log-plot suggests that price-performance has been
in a distinctly slower growth regime since around 2012.
In [ 41]:gpu.plot(x ='Date ', y='FLOPS per 2018$ ', style ='.', logy =True );
#plt.savefig( 'gpu\_log.png ')
11
Fitting a curve to data from 2012 onward yields a doubling time of ~3.9 years.
In [ 42]:gpu =gpu[gpu[ 'Date ']>pd.Timestamp( '2012-01-01 ')]
# Convert dates to unix time
time, y =gpu[ 'Date '].astype(np .int64) /10\*\*9, gpu[ 'FLOPS per 2018$ ']
# Linear regression in log-space
rate, constant =np.polyfit(time, np .log(y), 1, w=np.sqrt(y))
gpu[ 'Fit Exponential ']=pd.Series(np .exp(constant) \*np.exp(rate \*time))
# Pearson 's r (correlation coefficient)
gpu.plot(x ='Date ', y=['FLOPS per 2018$ ','Fit Exponential '], style =['.','-'])
#plt.savefig( 'gpu\_recent.png ')
print ('r = {:.3f} '.format(gpu[ 'FLOPS per 2018$ '].corr(gpu[ 'Fit Exponential '])))
doubling\_time =1/(np.log2(np .e)\*rate \*3600 \*24\*365)
print ("Doubling time: {:.2f} years ".format(doubling\_time))
r = 0.673
Doubling time: 3.93 years
12
It is unclear how long even this reduced rate of growth can continue, and these improvements
are expected to break down in the coming decades, with more than an additional 10-12 doublings
being very unlikely within the current paradigm. The question of whether and how the trend will
continue, perhaps through breakthroughs in spintronics or further improvements in 3D integrated
circuits , is beyond the scope of this document.
2.1.2 Architecture-Speciﬁc Improvements
So far, TPUs have been one generation behind the cutting-edge of semiconductor fabrication. Pri-
vate communication with experts in the ﬁeld suggests that a further one to two orders of magni-
tude of improvement in price-performance from technological progress and economies of scale is
likely, with more not being out of the question.
In [ 43]:tpu\_improvement =loguniform(low =1, high =2.5, size =n)
plt.hist(tpu\_improvement, bins =50, log =True );
plt.xlabel( "Factor of Improvement ");
plt.ylabel( "Frequency (log-scale) ");
summary(tpu\_improvement, 'Factor of Improvement ')
Out[ 43]: Percentile Factor of Improvement
0 1 10.364245
1 5 11.898779
2 25 23.586700
3 50 55.881355
4 75 132.640638
13
5 95 264.910053
6 99 304.889214
2.2 Computational Requirements
The estimated computational requirements are summarized below.
In [ 44]:# Number of arithmetic operations to simulate a lifetime
lifetime\_compute =brain\_compute \*longevity \*3600 \*24
# Number of simulated lifetimes needed to train an AGI
lifetimes =hyperparameters \*difficulty \*agz\_games
# Number of arithmetic operations to train an AGI
agi\_compute =lifetime\_compute \*lifetimes
plt.hist(agi\_compute, bins =100, log =True );
plt.xlabel( "Number of Operations ");
plt.ylabel( "Frequency (log-scale) ");
summary(agi\_compute, 'Operations ')
Out[ 44]: Percentile Operations
0 1 1.199615e+32
1 5 4.659938e+34
2 25 1.806512e+41
3 50 1.754430e+49
14
4 75 2.146480e+57
5 95 9.705500e+63
6 99 2.164436e+66
2.3 Economic Considerations
2.3.1 Monetary resources available for AGI (in inﬂation-adjusted 2018 dollars)
The total revenue of the semi-conductor industry in 2017 was a bit over 400 billion USD and has
been increasing linearly as a function of time since the mid-1980s. This is remarkable given that
this period saw the rise of personal computing, the birth of the web, and much of the information
revolution. Extrapolating this trend suggests that the revenue won’t reach one trillion USD for
more than a century. Given that we are considering costs of speciﬁc hardware components and
the time and expense associated with building new semiconductor fabs, this suggests that physical
production may impose a hard limit on how much can be spent.
In [ 45]:semiconductors =pd.read\_csv( "semiconductor.csv ")
semiconductors[ "Date "]=pd.to\_datetime(semiconductors[ 'Date ']+1,format ="%Y")
# adjust for inflation, convert to USD
semiconductors =pd.merge\_asof(
semiconductors,
cpi.filter(items =['Date ','Price Multiplier ']),
on='Date ',
direction ='nearest ')
15
semiconductors[ 'Adjusted Sales (2018$) ']=semiconductors[ 'Sales (Thousands of USD) ']\*semiconductors[ 'Price Multiplier ']\*1000
# unix time
time =semiconductors[ 'Date '].astype(np .int64) /10\*\*9
y=semiconductors[ 'Adjusted Sales (2018$) ']
# linear regression
slope, intercept =np.polyfit(time, y, 1)
semiconductors[ 'Fit Line ']=pd.Series(intercept +slope \*time)
semiconductors .plot(x ='Date ',
y=['Adjusted Sales (2018$) ','Fit Line '],
style =['.','-'])
print ('r = {:.3f} '.format(
semiconductors[ 'Adjusted Sales (2018$) '].corr(
semiconductors[ 'Fit Line '])))
print ("Years to reach one trillion USD: {:.2f} ".format(
(1e12 -intercept) /slope /(3600 \*24\*365)))
r = 0.957
Years to reach one trillion USD: 107.62
16
Also worth noting is that both the Apollo Program and the Manhattan Project each cost just
under half of one percent Gross World Product , and were regarded as hugely expensive and un-
sustainable efforts.
In [ 46]:projects =pd.DataFrame(data ={"Project ":["Manhattan Project ",
"Apollo Program "],
"Cost ":[2e9,25.4e9 ],
"Date ":["1940-01-01 ","1973-01-01 "],
"GWP (1990$) ":[4502584e6 ,16022888e6 ]})
projects[ 'Date ']=pd.to\_datetime(projects[ 'Date '],
infer\_datetime\_format =True )
projects =pd.merge\_asof(projects,
cpi.filter(items =['Date ','Price Multiplier ']),
on='Date ',
direction ='nearest ')
projects[ 'Cost ']=projects[ 'Cost ']\*projects[ 'Price Multiplier ']
projects[ 'GWP (2018$) ']=(projects[ 'GWP (1990$) ']\*
float (cpi[cpi[ 'Date ']==
'1990-01-01 ']['Price Multiplier ']))
projects[ 'Percent of GWP ']=projects[ 'Cost ']/projects[ 'GWP (2018$) ']\*100
projects[[ 'Project ','Percent of GWP ']]
Out[ 46]: Project Percent of GWP
0 Manhattan Project 0.407120
1 Apollo Program 0.474081
Gross World Product is currently estimated to be around $80 trillion and projected to be $38
trillion in 2060. Taking these numbers as a guide, it seems unlikely that expenditures would exceed
\$1 trillion before the late 21st century, and in all likelihood not even that.
In [ 47]:#budget = lognorm.rvs(s=1, scale=1e11, size=n)
budget =loguniform(low =10, high =12, size =n)
plt.hist(budget, bins =50, log =True );
plt.xlabel( "AGI Budget (2018$) ");
plt.ylabel( "Frequency (log-scale) ");
summary(budget, 'AGI Budget (2018$) ')
Out[ 47]: Percentile AGI Budget (2018$)
0 1 1.047156e+10
1 5 1.266899e+10
2 25 3.209692e+10
3 50 1.006332e+11
4 75 3.202036e+11
5 95 7.982820e+11
6 99 9.526886e+11
17
2.3.2 Hardware Price-Performance
At the time of this writing, a TPU performing 180 FLOPS can be rented at the rate of 4.5 USD per
hour .
In [ 48]:# Cost per hour of TPU
tpu\_cost =4.5
# Computational performance of TPU
tpu\_compute =180e12
# Current price-performance of ML hardware
price\_performance =tpu\_compute \*3600 /tpu\_cost
2.3.3 Number of Doublings Required
We can now calculate the number of doublings in price-performance required to train an AGI
given our budgetary constraints.
In [ 49]:# Required cost efficiency of ML hardware
required\_performance =agi\_compute /budget
# Number of doublings required for AGI to be economically feasible
doublings\_required =np.log2(required\_performance
/(price\_performance \*tpu\_improvement))
plt.hist(doublings\_required,
bins =100,
18
histtype ="stepfilled ",
cumulative =True );
plt.xlabel( "Number of Doublings Required ");
plt.ylabel( "Frequency ");
#plt.savefig("doublings.pdf")
summary(doublings\_required, 'Doublings ')
Out[ 49]: Percentile Doublings
0 1 6.612339
1 5 15.637536
2 25 37.673011
3 50 64.201534
4 75 91.083337
5 95 113.442426
6 99 121.385305
2.4 Projected Timeline
As was noted above, there is not currently an obvious path to exponential improvements in price-
performance after ~10-12 further doublings. The conclusion seems to be that this approach is
simply not viable, but if we assume that these trends (somehow) continue, we can estimate the
timeline.
In [ 50]:timeline =doublings\_required \*doubling\_time
plt.hist(timeline,
19
bins =1000 ,
cumulative =True ,
histtype ="stepfilled ",
density =True );
plt.xlabel( "Years from Present ");
plt.ylabel( "Frequency ");
#plt.savefig("timeline.pdf")
summary(timeline, 'Years ')
Out[ 50]: Percentile Years
0 1 25.982804
1 5 61.446788
2 25 148.033909
3 50 252.276198
4 75 357.906678
5 95 445.765419
6 99 476.976503
20