Exploring AI Safety in Degrees:
Generality, Capability and Control
John Burden
University of York
jjb531@york.ac.ukJos´e Hern ´andez-Orallo
Universitat Polit `ecnica de Val `encia, Spain
Leverhulme Centre for the Future of Intelligence, Cambridge, UK
jorallo@upv.es
Abstract
The landscape of AI safety is frequently explored differently
by contrasting specialised AI versus general AI (or AGI), by
analysing the short-term hazards of systems with limited ca-
pabilities against those more long-term risks posed by ‘su-
perintelligence’, and by conceptualising sophisticated ways
of bounding control an AI system has over its environment
and itself (impact, harm to humans, self-harm, containment,
etc.). In this position paper we reconsider these three aspects
of AI safety as quantitative factors –generality, capability and
control–, suggesting that by deﬁning metrics for these dimen-
sions, AI risks can be characterised and analysed more pre-
cisely. As an example, we illustrate how to deﬁne these met-
rics and their values for some simple agents in a toy scenario
within a reinforcement learning setting.
Introduction
Despite the impressive advances in AI in recent years, AI
systems remain narrow. They typically solve or perform well
at one task, or one type of task. These systems lack general-
ity and perform poorly outside of their target domain. Gener-
ality is intrinsic to the notion of “intelligence” and Artiﬁcial
General Intelligence (AGI) in particular.
We clearly want AI (general or not) to be safe and ben-
eﬁcial to humanity. It is not that narrow AI systems do
not pose risks, ranging from mistakes made by improperly
validated systems, through to the misuse of research and
technology, but AGI seems to pose unique safety issues.
Bostrom describes many scenarios in which a hypothetical
superintelligent AGI could present existential risk to human-
ity (Bostrom 2014) due to its enhanced capabilities over a
wide range of problem domains. The environmental con-
trol wielded by AGI systems could further increase safety
risks from the systems. The ability of a system to manipu-
late its environment could overcome previous system safety
guarantees as well as cause harm to other entities sharing
Copyright c⃝2020, Held by the authors. All rights reserved.the environment. Various strategies for controlling intelli-
gent systems have been proposed (Armstrong, Sandberg,
and Bostrom 2012), although none are entirely satisfactory.
However, there are several problems with this view of the
risk posed by very powerful systems. First, while Bostrom’s
Orthogonality Thesis (Bostrom 2012) posits that any goal or
value system is compatible with any level of intelligence,
it is unclear how this ‘level’ affects the hazards. It is as-
sumed that some risks are more likely as AI systems be-
come more intelligent, but the intelligence ‘level’ is never
fully characterised, beyond the notion of superhuman intelli-
gence. These omissions are not necessarily an oversight, but
may simply originate from major unsolved problems to char-
acterise the behaviour of intelligent systems in a richer, more
predictable way. Recent approaches, such as (Armstrong and
Levinstein 2017) and (Drexler 2019), and the overview by
(Amodei et al. 2016), introduce frameworks that go beyond
a monolithic view of intelligence, but do not aim at charac-
terising, and measuring, different dimensions of agent be-
haviour.
We believe many views of intelligence conﬂate different
levels of generality, capability and control, and disentan-
gling them can allow for a richer understanding, especially
as safety is concerned. For instance, can we have very capa-
ble but narrow systems? And very general systems with low
control over their environment?
In this position paper we analyse three separate factors
that are frequently integrated, but may affect risk differently.
We disentangle them using agent characteristic curves and
analyse how they relate to AI risk. Finally, we introduce a
toy scenario in which we precisely deﬁne metrics of gener-
ality, capability and control, and assess agents and situations
with them. With the aid of these unambiguous deﬁnitions,
the scenario highlights possible disagreements about percep-
tions of these three factors. This should encourage fruitful
discussions about how to deﬁne or generalise these metrics
for more complex situations, in a way that can be used to
explore a wide range of AI systems that have different levelsCopyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

of capability, generality and control over their environment.
Disentangling the Factors
Before we can analyse how the factors of capability, gener-
ality and control contribute to AI risk, we need some under-
standing of what those terms mean in the literature, and dis-
entangle them from terms such as AGI or superintelligence.
Capability: Capability is how good a system is at solv-
ing or performing in the domains for which it was designed.
Evaluating a system’s capability at a speciﬁc domain is rel-
atively straightforward if the domain has some sort of per-
formance metric, which would preferably be linked to the
resources available to the system, such as the available com-
putational power, memory and time (Mart ´ınez-Plumed et al.
2018). One key issue of capability is its scale. Ignoring this,
forces us to speak of capability in less than ideal terms —
either comparing capabilities vaguely or only with regard to
very narrow AI systems across very similar domains. An-
other issue appears when the task has inﬁnitely many in-
stances, and agents are only able to solve a ﬁnite number
of them, precisely because of resources. The percentage of
success would be 0, and hence a useless metric.
One ingenious solution for this problem comes from Psy-
chometrics, and Item Response Theory (IRT) (Embretson
and Reise 2000), in particular. IRT tries to provide a well-
founded statistical method for scoring abilities of test-takers.
IRT focuses not just on the ability of the test-taker when
scoring an individual, but also the difﬁculty of the ques-
tion or “item”. IRT then can produce an “Item Characteristic
Curve” (ICC) for each item, approximating the probability
an individual scores correctly on the item in question based
on their ability. From the ICCs, the ability of a test-taker can
be evaluated through the use of a maximum likelihood esti-
mate of the test-taker’s item responses. These allow the cre-
ation of “Agent Characteristic Curves” (ACC) for each test-
taker, mapping item difﬁculty to the probability of a correct
response. Figure 1 shows an ACC.
●
0 2 4 6 80.0 0.2 0.4 0.6 0.8 1.0
hψMean: 0.28
Variance: 0.09
Capability: 3.32
Figure 1: An ACC showing performance Ψof a Q-learning
agent over environments of increasing difﬁculty ( h, on the
x-axis). Data from (Insa-Cabrera et al. 2011).
In IRT, the ACC is usually sigmoidal, and ability is just
thex-axis position at 0.5. For a non-parametric curve, we
just consider capability as the area (3.32 in the example). In
this way, if we introduce inﬁnitely many extremely difﬁcult
problem instances in the pool that the agent cannot solve, the
capability does not change. Note that this does not happenwhen we calculate the average expected score over the set of
problem instances (in the ﬁgure, this is 0.28).
Generality: Generality is a measure of the number and
types of domains for which a system is able to handle and
perform well in. Generality is not always easy to evaluate
or quantify, especially when we do not have a clear deﬁni-
tion of the types of domain, but just a wide set of tasks. One
novel solution to this problem comes again by looking at
ACCs. As the item difﬁculty increases we typically expect
to see a performance drop. The rate at which this happens
captures generality over the problem space, which can be
calculated as some kind of proxy to the slope of the ACC. A
test-taker with a slow, gradual decline in performance over
the problem-space is much less general than an agent cover-
ing a consistent level of item difﬁculty with a sharp decline
in performance after this point. This can seem quite counter-
intuitive, but it is key to understand that we usually make the
comparison of systems with similar area under the ACC —
for a sharper decline, the decline must begin later and thus
the agent retains its high performance for a higher level of
difﬁculty. This approach frames generality as system per-
formance on as many low-difﬁculty tasks as possible, with-
out capturing the notion of systems performing tasks over
wildly different areas of problem-space. But this notion of
generality ensures that this breakout does not happen at least
until a level of difﬁculty.
Ultimately, unless we give an indication of the types of
domains for which a system performs well in, we would be
forced to speak of generality in aggregated terms. For the
purposes of this paper we will use generality to refer to the
gradient of an ACC.
Control: Another factor of a system that is of interest from
a safety perspective is control. By agent control we mean the
the reliability and deliberate intent of an agent’s actions and
decisions. We often wish to measure control with respect to
speciﬁc behavioural properties such as completing a goal or
avoiding “risky” behaviour.
This idea of control is also related to that of “affordances”
in the science of perception (Gibson 1979), where Gibson
describes an affordance as actions resulting from the rela-
tionship between agents and their environment. Affordances
have made their way into AI research as Nye and Silverman
(2012) make clear in their literature review of the subject.
For the purpose of our paper, we will refer to control as
opposite to the expected entropy of visited states conditioned
over the behaviour for which we wish to measure control.
Control is hence opposite to variability. If we look at an
ACC, control is related to the dispersion along the ACC.
Interaction between Risk and the Factors
Now we discuss the ways in which the factors may be re-
lated to the risk of an AI system. First, we need to under-
stand what it is exactly we are trying to assess. Risk is usu-
ally described as exposure to danger of some kind. While
danger to life or well-being is ultimately what we are trying
to prevent, in the context of AI safety a few speciﬁc scenar-
ios present themselves as potentially enabling this indirectly
and thus we consider those scenarios risky too. These types
of scenarios or behaviours include the system misinterpret-
ing goals or goals being poorly deﬁned. This is known as
the Value Learning problem (Soares 2015). Another risk we
are concerned with is the system resisting external efforts to
alter the system by its designers; ensuring the system allows
this is known as corrigibility (Soares et al. 2015). There are
countless other ways risk can manifest in AI (Amodei et al.
2016). Again, there are no wholly satisfactory solutions for
quantifying risk, but within one environment we can often
quantify speciﬁc risks as probabilities or as expected penal-
ties. It is then straightforward to compare –numerically– the
effect our factors have on risk within that environment.
Now let us explore the interaction at an abstract level. In-
tuitively there is potentially more risk associated with higher
capability and generality, and lower control. However, there
are more nuances on this than originally expected. Some of
these nuances come from the fact that very incompetent sys-
tems (low capability) are dangerous, as they do not fulﬁll
their duties, which is a clear safety concern. But these can
be categorised as known unknowns. As capability increases
in a given domain it becomes more difﬁcult to identify which
states the system may traverse through and what sort of side
effects may be caused as a result. These are unknown un-
knowns. Once an agent becomes more and more capable, we
encounter Vingean uncertainty — that is, if the agent is more
capable in a domain than we are, then we cannot completely
predict what the agent will do in that domain, otherwise we
would have the same domain capabilities. Guaranteeing safe
behaviour is much more difﬁcult in this scenario. Note that
the view of ACC, where difﬁculty is on the x-axis helps us
understand this. As tasks become more complex, solutions
can be achieved in ways we are not able to anticipate, espe-
cially if we ourselves do not reach that capability level.
Similarly, as generality increases, we can intuit that the
risk posed by the system also increases. This is because the
system becomes more adept at a wide range of tasks and
this makes constructing safety guarantees more time con-
suming. Further, if we use the generality notion from the
ACC, with an unchanging capability, we have worse perfor-
mance on high-difﬁculty tasks. These high difﬁculty tasks
may present more danger than their low-difﬁculty counter-
parts, especially if we do not understand them. But gener-
ality makes the system more expectable conditioned to dif-
ﬁculty. For instance, it gives reassurance to know that an
automated assistant is going to work well for all easy tasks.
The risks of a high capability and generality system are
further exacerbated by Omohundro’s “Basic AI Drives”
(Omohundro 2008), which hypothesises that such a system
may develop self-preservation instincts in order to ensure
its goals are achieved, as well as preemptively acquiring re-
sources in aid of these goals. Even seemingly benign goals
may pose a signiﬁcant risk.
Finally, as a system’s control over its environment in-
creases, the related risk the system poses may actually de-
crease. With higher control comes an agent acting more
deliberately and hence more predictably. When a system’sgoals are aligned with our safety standards, this deliberate
action performance can reduce the chance of violating safety
properties. For instance, a personal assistant that sometimes
fails at some easy tasks possibly because it solves them very
stochastically may become a hazard. The view of control as
reliability (or reduction of entropy) is aligned with this per-
spective. Of course, this requires a notion of safety align-
ment for the system’s actions. Deliberately unaligned ac-
tions obviously make no safety guarantees.
However, the system may also exert excessive control
over the environment and reduce the freedom of other agents
in it. This has been studied in AI-safety previously, often
from the perspective of enforcing safety by minimising or
reducing environmental control over certain factors. Exam-
ples of this include penalising actions that cannot be un-
done (Krakovna et al. 2018), or by minimising the impact
a system has on its environment (Armstrong and Levinstein
2017). Both of these approaches attempt to make exerting
certain types of control over the environment undesirable.
Capability, generality and environmental control can also
in some ways inﬂuence each other, in turn further increasing
the risk posed by the system. Whilst these factors are heavily
decoupled from each other, they are not entirely orthogonal.
Figure 2 shows the inter-connectivity between these factors
more visually.
Risk
Generality CapabilityControl
Figure 2: Relations between properties of an AI system and
the risk it poses.
An example of this entanglement is that as environmen-
tal control increases, new affordances become available to
the system. We already know that this can increase the risk
posed by the system, but these new affordances can allow
new behaviour, which may increase the system’s capability
across domains, or even the types of domains that the system
can perform well in.
The reverse of this relation can also be considered. The
idea of an AI breakout posits that as the system becomes
more capable and general, it may be able to detect and ex-
ploit vulnerabilities in the environment it is situated in, ul-
timately gaining control over the environment and utilising
that control to achieve its goals, behaving in a manner the
designers deemed unacceptable.
It is also possible that generality and capability can affect
each other in a system. It may be the case that the further a
system specialises into speciﬁc problem domains, the fewer
computational resources are left for other task types.
Overall, we can see that the factors of capability, general-
ity and control can all seemingly affect each other. What is
not so clear is the extent to which they do this, or the exact
relationship between them — these factors are clearly not
orthogonal nor directly correlated.
Scenario: Exploration in a RL Setting
While the relations shown in Figure 2 are hard to determine
in an abstract way, especially if we do not specify the partic-
ular risk we want to analyse, the analysis can be done for par-
ticular scenarios. Figure 3 shows a simple grid environment
in which an agent (designated as a smaller orange square)
must navigate to the goal (designated by a green square) in
the allotted number of steps.
Different properties of the agent can inﬂuence its capabil-
ity, generality and control over tasks from this environment
type. Such properties can include the agent’s method of ob-
servation (see the vision cone in Figure 3 as an example) and
algorithm employed for learning. Comparing the expected
success rate of different agents can give valuable insight into
how capability, generality and control are related to risk.
We now give a more formal description of the environ-
ment and measures of capability, generality and control. It
is important to note that the formal descriptions given here
for risk and the factors are domain speciﬁc. We consider a
bounded grid of mˆncells, where the agent πis located in
one cell and can move in the four cardinal directions at each
step. A goal is also located in one cell and yields a positive
rewardrgą0when the agent reaches the cell. Reaching the
goal terminates the episode. There may be one or more pits:
cells that if the agent reaches them, it can never go out for
the rest of the episode. Episodes have a length of ﬁxed Te
steps. Rewards are always 0 unless stated otherwise above.
Environments µare generated with a distribution ppµqover
the location of agent, goal and pits.
Letspπ,µqdenote that agent πwas successful on environ-
mentµ(reaching the goal before the episode terminates).
Now we deﬁne a simple notion of difﬁculty ℏfor each
environment as ℏpµq “1´Ppspπrnd,µqqwhereπrndis a
random-walk agent. In other words, the difﬁculty of an envi-
ronment is 1 minus the probability of success of a random-
walk agent. Now, we build an ACC as follows. For each
difﬁculty h, we calculate the expected success of agent π
only for the environments of that difﬁculty, i.e., we condi-
tionponh. We denote this success rate per difﬁculty as:
Shpπ,pq “Ppspπ,µq |ℏpµq “hq. By plotting Shfor the
range of difﬁculties we have a “curve”, the agent character-
istic curve for π, very much like the example in Figure 1.
Thecapability (Ψpπ,pq) of an agent is the area under its
curve. The generality (Γpπ,pq) of an agent is (a proxy of)
the steepness of this curve.
As examples of agents we could have agents that solve
all situations (very capable and general), agents that solve
the task only when the goal is nearby (general but not very
capable) or speciﬁc agents that solve the task only when the
goal is in the upper half of the grid (e.g., imagine that it has
been trained on a distribution where the goal was usually in
the upper half).
The control of an agent in environment µis deﬁned as
cpπ,µq “Hmax ´Hpµ,πq, whereHis the entropy of the
statesπis expected to visit in µ, andHmax is the maximum
Figure 3: Possible domain to test risks related to control
entropy. Then an agent’s control over all environments is:
Cpπ,pq “Eµ„p,Ppspπ,µqqě1´δrcpπ,µqs
i.e., the expected control for the environments where the
agent is expected to be successful1. The failure probability
threshold δP p0,1qcan be selected as appropriate for the
environment and agent.
For instance, a random-walk agent is expected to have
medium generality, low capability and very low control. An
agent that goes optimally to the goal is expected to have high
generality, high capability and high control.
The risk of an agent πwithin environment µisυpπ,µq
is the probability that πwill fall into a pit during an episode
within environment µ. The risk of an agent over a whole task
distribution is Υpπ,pq “Eµ„prυpπ,µqs. This risk is usually
referred to as safe exploration.
It is worth noting that in the test environment it is indeed
possible to have agents of high capability and low control
and vice versa. Although in many cases control will correlate
well with capability. Similarly, many capable and general
agents within our setting may have a high risk level.
We claim that an increase in an agent’s control will lead
to a decrease in the risk of the agent. That is, a higher level
of control yields safer exploration. As an agent’s control
Cpπ,pqincreases, the expected control of tasks drawn from
pwhich we expect πto succeed in also increases — this is
just the deﬁnition of agent control. Subsisting in the deﬁni-
tion for agent control within a speciﬁc environment tells us
thatEµ„p,Ppspπ,µqě1´δqrHmax´Hpµ,πqsincreases. Since
Hmaxis ﬁxed, the expected entropy in successful environ-
mentsHpµ,πqmust be decreasing. In a successful environ-
ment, we expect the agent to reach the goal within the time
limitTe. If the expected entropy decreases, the agent is less
likely to select moves that are not on the successful path.
This reduces the chance of entering a pit cell and thus re-
duces the agent’s risk.
Discussion
Characterising AI agents in terms of average performance
is simplistic, as there are many different ways in which the
same performance can be obtained. Much active research in
AI safety at the moment is characterised by efforts to go be-
yond this narrow view The perspective of robustness in AI
1Note that the deﬁnition of control here seems related to the
observability the agent has over the environment — an agent with
perfect “sight” and policy will have much greater control than an
agent with partial observability due to needing to explore to spot
the goal or build up belief states.
safety is identiﬁed here by the assurances that certain difﬁ-
culty is achieved (capability), that up to this level of difﬁ-
culty no pocket of problems is ignored by the agent (gen-
erality) and that the variability in results and strategies for
successful policies is low (control). In a way, we are limiting
unpredictability conditioned to a bounded difﬁculty, which
can be used to deﬁne a safe area for the system.
Although the main goal of the paper was to raise aware-
ness of the necessity and relevance of disentangling perfor-
mance into more reﬁned factors and the opportunities of
analysing risk according to them, as future work we plan
to explore these ideas ourselves for different classes of en-
vironments. We also encourage AI safety researchers to ob-
tain theoretical and experimental results under the factors
introduced here, such as the degree of correlation between
capability and control. New formulations may be needed.
For instance, as problems become harder the maximum en-
tropy also increases, and relating or normalising control to
difﬁculty may be more appropriate.
Acknowledgements
We thank the anonymous reviewers for their comments. This
work was funded by the Future of Life Institute, FLI, under
grant RFP2-152, and also supported by the EU (FEDER)
and Spanish MINECO under RTI2018-094403-B-C32, and
Generalitat Valenciana under PROMETEO/2019/098.
References
Amodei, D.; Olah, C.; Steinhardt, J.; Christiano, P.; Schulman,
J.; and Man ´e, D. 2016. Concrete problems in AI safety. arXiv
preprint arXiv:1606.06565 .
Armstrong, S., and Levinstein, B. 2017. Low impact artiﬁcial
intelligences. arXiv preprint arXiv:1705.10720 .
Armstrong, S.; Sandberg, A.; and Bostrom, N. 2012. Thinking
inside the box: Controlling and using an oracle ai. Minds and
Machines 22(4):299–324.
Bostrom, N. 2012. The superintelligent will: Motivation and
instrumental rationality in advanced artiﬁcial agents. Minds and
Machines 22(2):71–85.
Bostrom, N. 2014. Superintelligence: Paths, dangers, strate-
gies. Oxford University Press.
Drexler, K. E. 2019. Reframing superintelligence: Comprehen-
sive ai services as general intelligence.
Embretson, S. E., and Reise, S. P. 2000. Item response theory
for psychologists . L. Erlbaum.
Gibson, J. J. 1979. The Ecological Approach to Visual Percep-
tion. Houghton Mifﬂin.
Insa-Cabrera, J.; Dowe, D. L.; Espa ˜na-Cubillo, S.; Hern ´andez-
Lloreda, M. V .; and Hern ´andez-Orallo, J. 2011. Comparing
humans and AI agents. In International Conference on Artiﬁ-
cial General Intelligence , 122–132. Springer.
Krakovna, V .; Orseau, L.; Martic, M.; and Legg, S. 2018.
Measuring and avoiding side effects using relative reachability.
CoRR abs/1806.01186.
Mart ´ınez-Plumed, F.; Avin, S.; Brundage, M.; Dafoe, A.;
h´Eigeartaigh, S. ´O.; and Hern ´andez-Orallo, J. 2018. Account-
ing for the neglected dimensions of AI progress. arXiv preprint
arXiv:1806.00610 .Nye, B. D., and Silverman, B. G. 2012. Affordances in AI .
Boston, MA: Springer US. 183–187.
Omohundro, S. M. 2008. The basic AI drives. Artiﬁcial Gen-
eral Intelligence 171:483–493.
Soares, N.; Fallenstein, B.; Armstrong, S.; and Yudkowsky, E.
2015. Corrigibility. In Workshops at the Twenty-Ninth AAAI
Conference on Artiﬁcial Intelligence .
Soares, N. 2015. The value learning problem. Ethics for Arti-
ﬁcial Intelligence Workshop at 25th International Joint Confer-
ence on Artiﬁcial Intelligence .Powered by TCPDF (www.tcpdf.org)


Antitrust-Compliant AI Industry Self-Regulation

A n titr u s t- C o m p l i a n t   A I    I n d u s tr y
S e l f - R e g u l a tio n

Cullen   O’Keefe
Research   Scientist   (Policy),   OpenAI
Research   Affiliate,   Centre   for   the   Governance   of   AI,   Future   of   Humanity   Institute
Research   Affiliate,   Legal   Priorities   Project

July   7,   2020
S um m a r y
The   touchstone   of   antitrust   compliance   is   competition.   To   be   legally   permissible,   any   industrial
restraint   on   trade   must   have   sufficient   countervailing   procompetitive   justifications.   Usually ,
anticompetitive   horizontal   agreements   like   boycotts   (including   a   refusal   to   produce   certain
products)   are    per   se    illegal.

The   “learned   professions,”   including   engineers,   frequently   engage   in   somewhat   anticompetitive
self-regulation   through   professional   standards.   These   standards   are   not   exempt   from   antitrust
scrutiny .   However ,   some   Supreme   Court   opinions   have   nevertheless   held   that   some   forms   of
professional   self-regulation   that   would   otherwise   receive    per   se    condemnation   could   receive
more   preferential   antitrust   analysis   under   the   “Rule   of   Reason.”   This   Rule   weighs
procompetitive   and   anticompetitive   impacts   to   determine   legality .   To   receive   the   rule-of-reason
review ,   such   professional   self-regulation   would   need   to:

1.Be   promulgated   by   a   professional   body;
2.Not   directly   affect   price   or   output   level;   and
3.Seek   to   correct   some   market   failure,   such   as   information   asymmetry   between
professionals   and   their   clients.

Professional   ethical   standards   promulgated   by   a   professional   body   (i.e.,   comparable   to   the
American   Medical   Association   or   American   Bar   Association)   that   prohibit   members   from
building   unsafe   AI   could   plausibly   meet   all   of   these   requirements.

This   paper   does    not    argue   that   this   would   clearly   win   in   court,   or   that   such   an   agreement   would
be   legal.   Nor   does   it   argue   that   it   would   survive   rule-of-reason   review .   It   merely   says   that   there  1
exists   a   colorable   argument   for   analyzing   such   an   agreement   under   the   Rule   of   Reason,   rather
than   a    per   se    rule.   Thus,   this   could   be   a   plausible   route   to   an   antitrust-compliant   horizontal
agreement   to   not   engineer   AI   unsafely .
L eg a l    B a c k g r o un d:   P r o f es s i o n a l    S el f - R eg u l a t i o n    a n d
A n t i t r us t
Antitrust   laws   apply   to   the   “learned   professions”:   e.g.,   law,   medicine,   and   engineering.   Thus,  2 3
anticompetitive   practices   by   professionals   can   violate   the   antitrust   laws.
1  However,   it   is   very   difficult   for   plaintiffs   to   prevail   in   rule-of-reason   cases.   According   to   one   empirical   study,   “[i]n
221   of   222   [rule-of-reason]   cases   [that   did   not   settle]   .   .   .   ,   the   defendant   won.”   Michael   A.   Carrier,    The   Rule   of
Reason:   An   Empirical   Update   for   the   21st   Century ,   16   G EO .   M ASON     L .   R EV .   827,   830   (2009)   (footnote   omitted).
2   See   Goldfarb   v.   Virginia   State   Bar ,   421   U.S.   773,   787   (1975).
3   See    William   Blumenthal,    A   Primer   on   the   Application   of   Antitrust   Law   to   the   Professions   in   the   United   States ,   14
A NDREWS    A NTITRUST    L ITIG .   R EP .   13,   at   *1   (2007).

The   case   that   established   the   applicability   of   antitrust   laws   to   professionals   was    Goldfarb   v.
Virginia   State   Bar .   There,   plaintif fs   challenged   a   minimum   fee   schedule   for   legal   work   set   by  4
the   state   bar   association.   The   Supreme   Court   held   that   this   minimum   fee   schedule   was   illegal  5
price-fixing.   The   Court   rejected   defendants’   argument   that   the   practice   of   law,   as   a   profession,  6
fell   outside   the   antitrust   laws.   In   a   famous   footnote   (“Footnote   17”),   however ,   the   Court   opined:  7

The  fact  that  a  restraint  operates  upon  a  profession  as  distinguished  from  a
business  is,  of  course,  relevant  in  determining  whether  that  particular  restraint
violates  the  Sherman  Act.  It  would  be  unrealistic  to  view  the  practice  of
professions  as  interchangeable  with  other  business  activities,  and  automatically  to
apply  to  the  professions  antitrust  concepts  which  originated  in  other  areas.  The
public  service  aspect,  and  other  features  of  the  professions,  may  require  that  a
particular  practice,  which  could  properly  be  viewed  as  a  violation  of  the  Sherman
Act  in  another  context,  be  treated  differently .  We  intimate  no  view  on  any  other
situation   than   the   one   with   which   we   are   confronted   today .  8

The   next   major   Supreme   Court   case   on-point   was    National   Society   of   Professional   Engineers   v.
United   States .   There,   the   US   sued   over   an   ethical   canon   that   prohibited   engineers   from   engaging  9
4  421   U.S.   773   (1975).
5   See   id.    at   776–78.
6   See   id.    at   781–83.
7   See   id.    at   787–88.
8   Id.    at   788   n.17.
9  435   U.S.   679   (1978).
in   “competitive   bidding.”   Defendants   asserted   that   the   canon   served   the   public   interest   because  10
price   competition   “would   lead   to   deceptively   low   bids,   and   would   thereby   tempt   individual
engineers   to   do   inferior   work   with   consequent   risk   to   public   safety   and   health.”  11

The   Court   held   that   the   canon   nevertheless   constituted   illegal   price-fixing.   The   Court  12
thoroughly   rejected   defendants’   public   welfare   arguments,   holding   that,   “[e]ven   assuming
occasional   exceptions   to   the   presumed   [good]   consequences   of   competition,   the   statutory   policy
precludes   inquiry   into   the   question   whether   competition   is   good   or   bad”:  13

[R]estraint  on  competition  .  .  .  on  the  basis  of  the  potential  threat  that  competition
poses  to  the  public  safety  .  .  .  is  nothing  less  than  a  frontal  assault  on  the  basic
policy  of  the  Sherman  Act.  .  .  .  Exceptions  to  the  Sherman  Act  for  potentially
dangerous   goods   and   services   would   be   tantamount   to   a   repeal   of   the   statute.  14

The   Court   also   found   Footnote   17   inapplicable   to   the   defendants’   canon:

[T]he  cautionary  footnote  in Goldfarb  cannot  be  read  as  fashioning  a  broad
exemption  under  the  Rule  of  Reason  for  learned  professions.  We  adhere  to  the
10   See   id.    at   682–84.
11   Id.    at   693.   Note   the   close   analogy   to   arguments   in   favor   of   non-racing   agreements   in   AI:   competition   in   AI
capabilities   development    would   lead   to   suboptimal   investment   in   safety,   to   the   risk   of   public   safety.    Cf.    N ICK
B OSTROM ,   S UPERINTELLIGENCE    246–54   (2014);   Stuart   Armstrong   et   al.,    Racing   to   the   Precipice:   A   Model   of   Artificial
Intelligence   Development    2   (Future   of   Humanity   Inst.,   Technical   Report   No.   2013-1),    https://perma.cc/S56B-Z5UZ .
12   See    Prof’l   Engineers,   435   U.S.   at   695.
13   Id. ;    see   also     id.    at   696   (“[T]he   Rule   of   Reason   does   not   support   a   defense   based   on   the   assumption   that
competition   itself   is   unreasonable.”).
14   Id.    at   695.   Thanks   to   Carrick   Flynn   for   suggesting   the   inclusion   of   this   quotation.
view  expressed  in Goldfarb  that,  by  their  nature,  professional  services  may  differ
significantly  from  other  business  services,  and,  accordingly ,  the  nature  of  the
competition  in  such  services  may  vary. Ethical  norms  may  serve  to  regulate  and
promote  this  competition,  and  thus  fall  within  the  Rule  of  Reason .  But  the
Society’ s  argument  in  this  case  is  a  far  cry  from  such  a  position.  We  are  faced
with  a  contention  that  a  total  ban  on  competitive  bidding  is  necessary  because
otherwise  engineers  will  be  tempted  to  submit  deceptively  low  bids.  Certainly ,  the
problem  of  professional  deception  is  a  proper  subject  of  an  ethical  canon.  But,
once  again,  the  equation  of  competition  with  deception,  like  the  similar  equation
with  safety  hazards,  is  simply  too  broad;  we  may  assume  that  competition  is  not
entirely  conducive  to  ethical  behavior ,  but  that  is  not  a  reason,  cognizable  under
the   Sherman   Act,   for   doing   away   with   competition.  15

Thus,   the    Professional   Engineers    Court   left   open   the   door   to   some   narrow   forms   of   professional
self-regulation   without   establishing   a   clear   rule   therefor .

Next   came    FTC   v.   Indiana   Federation   of   Dentists .   There,   defendants   attempted   to   limit   insurer  16
second-guessing   of   the   dental   necessity   of   patient   care   by   agreeing   not   to   submit   x-rays   to
insurers   that   requested   them.   One   of   the   proffered   defenses   of   this   policy   was   that   it   protected  17
15   Id.    at   696   (emphasis   added)   (footnote   omitted)   (citation   omitted).
16  476   U.S.   447   (1986).
17   See   id.    at   449–50.
the   quality   of   patient   care.   The   Court   rejected   the   availability   of   this   defense   under   the   rule   of  18
Professional   Engineers .  19

Allied   Tube   &   Conduit   Corp.   v.   Indian   Head,   Inc.   did   not   involve   the   learned   professions   but  20
contained   the   following   relevant   dicta:

Agreement  on  a  product  standard  is  .  .  .  implicitly  an  agreement  not  to
manufacture,  distribute,  or  purchase  certain  types  of  products.  Accordingly ,
private  standard-setting  associations  have  traditionally  been  objects  of  antitrust
scrutiny .  When,  however ,  private  associations  promulgate  safety  standards  based
on  the  merits  of  objective  expert  judgments  and  through  procedures  that  prevent
the  standard-setting  process  from  being  biased  by  members  with  economic
interests  in  stifling  product  competition,  those  private  standards  can  have
significant  procompetitive  advantages.  It  is  this  potential  for  procompetitive
benefits  that  has  led  most  lower  courts  to  apply  rule-of-reason  analysis  to  product
standard-setting   by   private   associations.  21
18   See   id.    at   462.
19   See   id.    at   463–64.   Einer   Elhauge   and   Damien   Geradin   offer   the   following   analysis   of   this   point:

On  principle,  the  [quality-of-care]  justification  was  rejected  because  it  involved  a  claim  that
informed  consumers  would  make  bad  choices.  Factually,  it  was  rejected  on  the  grounds  that
insurers  represented  consumer  interests  because  insurers  had  to  compete  for  consumer  patronage.
The  latter  seems  more  debatable  because  agency  costs  give  insurers  incentives  to  increase  profits
by  misusing  x-ray  information  to  deny  desirable  claims.  Perhaps  the  case  would  come  out
differently  if  the  insurers  did  not  use  dentists  to  make  the  denial  decisions,  because  then  there
would   have   been   more   fear   the   decisions   were   not   being   based   on   sound   medical   judgment.

E INER    E LHAUGE    &   D AMIEN    G ERADIN ,   G LOBAL    A NTITRUST    L AW     AND    E CONOMICS    212   (3d.   ed.   2018).
20   486   U.S.   492   (1988).
21   Id.    at   500–01   (citations   omitted).

The   final   relevant   case   is    California   Dental   Association   v.   FTC .   There,   defendants’   code   of  22
ethics   prohibited   advertising   containing   certain   promises   of   low   prices.   The   Court   held   that   the  23
prohibition   should   receive   full   rule-of-reason   review   because   it   “fail[ed]   to   present   a   situation  24
in   which   the   likelihood   of   anticompetitive   effects   is   comparably   obvious.”   The   Court   reasoned  25
that   the   restrictions   “might   plausibly   be   thought   to   have   a   net   procompetitive   effect,   or   possibly
no   effect   at   all   on   competition”:  26

The  restrictions  on  both  .  .  .  advertising  are,  at  least  on  their  face,  designed  to
avoid  false  or  deceptive  advertising  in  a  market  characterized  by  striking
disparities  between  the  information  available  to  the  professional  and  the  patient.
In  a  market  for  professional  services,  in  which  advertising  is  relatively  rare  and
the  comparability  of  service  packages  not  easily  established,  the  difficulty  for
customers  or  potential  competitors  to  get  and  verify  information  about  the  price
and  availability  of  services  magnifies  the  dangers  to  competition  associated  with
misleading  advertising.  What  is  more,  the  quality  of  professional  services  tends  to
resist  either  calibration  or  monitoring  by  individual  patients  or  clients,  partly
because  of  the  specialized  knowledge  required  to  evaluate  the  services,  and  partly
because  of  the  difficulty  in  determining  whether ,  and  the  degree  to  which,  an
22  526   U.S.   756   (1999).
23   See   id.    at   760,   760   n.1.
24  As   opposed   to   “quick   look   analysis,”   which   is   appropriate   “when   the   great   likelihood   of   anticompetitive   effects
can   easily   be   ascertained.”    See   id.    at   770.
25   See   id.    at   771.
26   Id.
outcome  is  attributable  to  the  quality  of  services  (like  a  poor  job  of  tooth  filling)
or  to  something  else  (like  a  very  tough  walnut).  Patients’  attachments  to  particular
professionals,  the  rationality  of  which  is  difficult  to  assess,  complicate  the  picture
even  further . The  existence  of  such  significant  challenges  to  informed
decisionmaking  by  the  customer  for  professional  services  immediately  suggests
that  advertising  restrictions  arguably  protecting  patients  from  misleading  or
irrelevant  advertising  call  for  more  than  cursory  treatment  as  obviously
comparable   to   classic   horizontal   agreements   to   limit   output   or   price   competition.  27

Note   that   this   applies   even   though   the   dentists   were   financially   interested   in   the   restraint.   28

Taking   the   above   cases   together ,   “[t]he   resulting   doctrine   thus   seems   to   be   that,   unlike   ordinary
businesses,   professionals   can   impose   .   .   .   restraints   to   try   to   regulate   market   failures,   subject   to
rule-of-reason   review ,   as   long   as   they   do   not   involve   direct   restraints   on   price   or   [level   of]   output
.   .   .   .” ,   Uninterested   self-regulations   are   likely   to   receive   more   favorable   treatment.  2930 31
27   Id.    at   771–73   (emphasis   added)   (footnotes   omitted)   (citations   omitted).
28   See    E LHAUGE    &   G ERADIN ,    supra    note   19,   at   229.
29   Id. ;    see   also    Koefoot   v.   Am.   Coll.   of   Surgeons,   652   F.   Supp.   882,   888   (N.D.   Ill.   1986)   (“[R]ule   of   reason   analysis
is   appropriate   when   facially   legitimate   ethical   canons   are   challenged   under   the   Sherman   Act.   The   Court   defines
facially   legitimate   ethical   canons   as   being   rules   of   professional   practice   which,   on   their   face,   establish   professional
standards   of   care   without   reference   to   the   economic   interests   of   the   professionals.”).
30  Note   that   “level   of   output”   means   a   nonzero   level.   Refusing   to   produce   something   in   some   sense   restrains   output
of   that   thing,   but   it   does   not   fix   prices   to   the   benefit   of   the   producers,   which   is   the   worry   with   direct   output
restraints.    Cf.    E LHAUGE    &   G ERADIN ,    supra    note   19,   at   211.
31   See    Koefoot,   652   F.   Supp.   at   888.
A p p l i c a t i o n    t o    A I
S c h e m a
Of   interest   here   is   a   hypothetical   and   potentially   desirable   horizontal   agreement   (“Agreement”)
between   AI   engineers   (or   those   working   for   labs   aiming   to   produce   AGI)   not   to   produce   unsafe
A(G)I.  32
P r i m a    F a c i e    A p p l i c a t i o n    o f    D o c t r i n e
E n g i n e e r s    a r e    a    P r o f e s s i o n
“Engineering   is   a[]   .   .   .   learned   profession.”   Thus,   “computer   engineers”   or   “AI   engineers”   or  33
even   “AGI   engineers”   could   theoretically   avail   themselves   of   the   protections   of   this   doctrine.
The   Agreement   could   be   part   of   the   code   of   ethics   of   a   new   or   existing   professional   organization
of   engineers.
An    Ag r e e m e n t    W o u l d    N o t    D i r e ct l y    A e c t    P r i c e    o r    Ou t pu t    L e v e l
The   Agreement   prohibits   engineers   from   producing   some   defective   product.   It   does   not   directly
affect   price   or   output   level.
32  The   exact   meaning   of   “unsafe   AI”   is   not   of   interest   here.
33  Prof'l   Eng’rs,   435   U.S.   at   681.
An    Ag r e e m e n t    C o r r e ct s    M a r k e t    F a i l u r e
There   is   a   colorable   argument   that   an   Agreement   would   correct   a   market   failure   (namely ,
information   asymmetry).   AI   engineers   are   much   better -positioned   than   their   clients   to   know
whether   AI   they   produce   is   safe.   Thus,   an   Agreement   would   be   analogous   to   the   ethical   canon
at-issue   in    California   Dental :   it   would   regulate   informational   asymmetries   in   the   professionals’
market.

Note   that   the   market   failure   is    not    the   externalized   risk   to   the   public   from   unsafe   AI.   Under
California   Dental ,   the   procompetitive   benefits   of   the   Agreement   must   accrue   to   the    consumers .  34

If   the   above   arguments   are   correct,   then    California   Dental    dictates   that   the   Agreement   would   be
subject   to   rule-of-reason   review .   The   defendants   would   then   have   to   show   that   its   procompetitive
effects   outweigh   its   anticompetitive   effects.   The   fact   that   the   engineers   arguably   have   no
financial   interest   in   the   Agreement   militates   against   a   finding   of   anticompetitive   effects.   35 36
34   Cf.    California   Dental,   526   U.S.   at   773;   Clark   C.   Havinghurst   &   Nancy   M.   P.   King,    Private   Credentialing   of
Health   Care   Personnel:   An   Antitrust   Perspective:   Part   One ,   9   A M .   J.L.   &   M ED .   131,   170   n.124   (1983).   Furthermore,
fixing   market   failures    generally    is   not   a   sufficient   reason   to   engage   in   anticompetitive   behavior.    Cf.    Fashion
Originators’   Guild   of   Am.   v.   Fed.   Trade   Comm’n,   312   U.S.   457,   467–68   (1941)   (holding   that   dress   manufacturer
industry   group   plan   to   boycott   retailers   that   bought   pirated   dress   designs   was   violation   of   antitrust   laws).
35   Cf.    E LHAUGE    &   G ERADIN ,    supra    note   19,   at   212.
36   See   id.    (citing   Allied   Tube,   486   U.S.   at   501);    see   also   id.    at   211.
A n a l o g y    t o    M e d i c a l    E t h i c a l    C a n o n s
The   Agreement   is   most   analogous   to   existing   canons   of   medical   ethics   that   prohibit   doctors   from
administering   unscientific   treatments.   The   commonality   of   such   canons   will   likely   make   a   Court
hesitate   to   find   them    per   se    illegal.   37

One   circuit   court   case   held   that   such   canons   should   receive   rule-of-reason   review:    Wilk   v.
American   Medical   Association .   There,   defendants   (i.e.,   the   AMA)   maintained   the   following  38
ethical   canon:   “‘A   physician   should   practice   a   method   of   healing   founded   on   a   scientific   basis;
and   he   should   not   voluntarily   professionally   associate   with   anyone   who   violates   this   principle.’”
  They   interpreted   this   last   clause   to   include   chiropractors.  39 40

“Plaintif fs   alleged   that   defendants   engaged   in   a   combination   and   conspiracy   to   eliminate   the
chiropractic   profession   through   refusing   to   deal   with   plaintif fs   and   other   chiropractors.”   The  41
AMA   said   it   promulgated   the   canon   “because   defendants   respected   scientific   method   as   the   basis
for   diagnosis   and   treatment   and   were   unwilling   to   risk   the   health   and   lives   of   their   patients   by
associating   professionally   in   the   care   of   patients   with   persons   who   (so   defendants   thought)   do   not
37   See   id.    at   211–12.
38  719   F.2d   207   (7th   Cir.   1983).
39   Id.    at   213.
40   See   id.
41   Id.    at   211.
share   respect   for   scientific   method.”   The   Court   held   that   this   saved   the   boycott   from    per   se  42
illegality .  43

This   suggests   that   the   best   hope   for   the   survival   of   an   Agreement   is   to   analogize   an   Agreement   to
such   uncontroversial   canons   from   medical   ethics.
A n a l o g y    t o    E n g i n e e r i n g    E t h i c a l    C a n o n s
The   National   Society   of   Professional   Engineers   (“NSPE”)   also   maintains   a   Code   of   Ethics.  44
Safety-related   canons   include:

Engineers,   in   the   fulfillment   of   their   professional   duties,   shall:
1.Hold   paramount   the   safety ,   health,   and   welfare   of   the   public.
2.Perform   services   only   in   areas   of   their   competence.  45
Engineers   shall   hold   paramount   the   safety ,   health,   and   welfare   of   the   public.
a.If  engineers’  judgment  is  overruled  under  circumstances  that
endanger  life  or  property ,  they  shall  notify  their  employer  or  client
and   such   other   authority   as   may   be   appropriate.
b.Engineers  shall  approve  only  those  engineering  documents  that  are
in   conformity   with   applicable   standards.  46
42   Id.    at   219.
43   See   id.    at   222;    see   also    Johnson   v.   Blue   Cross/Blue   Shield   of   New   Mexico,   677   F.   Supp.   1112,   1118   (D.N.M.
1987)   (applying    Wilk    on   similar   facts).
44   See   NSPE   Code   of   Ethics   for   Engineers ,   N AT ’ L    S OC ’ Y    P ROF ’ L    E NG ’ RS    (July   2018),    https://perma.cc/9C2R-U29D .
45   Id.    § I.
46   Id.    § II(1).
Engineers   shall   perform   services   only   in   the   areas   of   their   competence.
a.Engineers  shall  undertake  assignments  only  when  qualified  by
education   or   experience   in   the   specific   technical   fields   involved.
b.Engineers  shall  not  affix  their  signatures  to  any  plans  or  documents
dealing  with  subject  matter  in  which  they  lack  competence,  nor  to
any  plan  or  document  not  prepared  under  their  direction  and
control.  47
Engineers  shall  not  complete,  sign,  or  seal  plans  and/or  specifications  that  are  not
in  conformity  with  applicable  engineering  standards.  If  the  client  or  employer
insists  on  such  unprofessional  conduct,  they  shall  notify  the  proper  authorities  and
withdraw   from   further   service   on   the   project.  48

The   NSPE   has   a   Board   of   Ethical   Review   which   publishes   advisory   opinions   interpreting   and
applying   the   Code   of   Ethics.   Some   of   these   opinions   seemingly   restrain   trade   in   the   name   of  49
either   safety   or   professional   competence   (which   is   presumably   at   least   partially   a   proxy   for
safety).   For   example:

●Case   No.   98-8,   holding   that   a   civil   engineer   was   incompetent   to   certify   that   arms   storage
room   satisfied   Army   regulations   and   citing   public   safety   reasons.  50
47   Id.    §   II(2).
48   Id.    § III(2)(b).
49   See   generally     Board   of   Ethical   Review   Cases ,   N AT ’ L    S OC ’ Y    P ROF ’ L    E NG ’ RS    (2017),
https://www.nspe.org/resources/ethics/ethics-resources/nonmember/board-of-ethical-review-cases .   This   somewhat
resembles   a   horizontal   division   of   markets,   which   is   illegal   under   antitrust.
50  NSPE   Bd.   of   Ethical   Rev.,   Case   No.   98-8   (1998),    https://perma.cc/FG9X-XUCQ .
●Case   No.   08-12,   holding   that   an   engineer   could   not   remove   emer gency   exit   lights   from
building   specifications   to   save   cost   at   supervisor ’s   command.  51
●Case   No.   15-4,   holding   that   a   computer   engineer   could   not   develop   a   computer   program
that   would   understate   actual   pollution   emitted   by   power   generators   using   that   program.  52
●Case   No.   15-11,   holding   that   an   engineer   could   not   continue   a   public   works   plan   wherein
the   employer   requests   traffic   lanes   narrower   than   those   specified   by   current   engineering
standards.  53
●Case   No.   16-5,   holding   that   an   engineer   developing   driverless   car   software   had   “an
obligation   to   state   that   the   prime   ethical   obligation   of   the   vehicle   operation   is   to   minimize
harm   to   affect   the   least   number   of   persons”   in   a   hypothetical   unavoidable   crash.  54

51  NSPE   Bd.   of   Ethical   Rev.,   Case   No.   08-12   (2008),    https://perma.cc/8Z2V-AXV8 .
52  NSPE   Bd.   of   Ethical   Rev.,   Case   No.   15-4   (2015),    https://perma.cc/4ATV-QDKV .   The   fact   that   such
underreporting   might   be   illegal   was   part   of   the   Board’s   reasoning,   but   does   not   seem   decisive:

[T]his  case  .  .  .  involve[s]  a  clear  conflict  between  the  obligation  of  the  engineer  to  the  public
health   and   safety   and   the   obligation   to   be   a   faithful   agent   and   trustee   to   an   employer   or   client.

Balancing  the  ethical  considerations,  it  is  the  Board’s  view  that  Engineer  A’s  proposed  coding
would  be  inconsistent  with  his  obligation  to  the  public  health  and  safety.  The  Board  further  notes
that  such  coding  would  also  run  a  risk  that  Engineer  A,  Company  X,  and  its  power  generation
clients  could  find  themselves  in  violation  of  state  and  federal  pollution  control  laws  and
regulations.  While  the  power  generation  clients  may  see  some  benefit  in  an  equipment  feature  that
will  extend  the  life  of  the  equipment  and  provide  better  value  to  them  and  other  power  generating
companies  that  will  purchase  the  equipment,  the  effect  on  accurate  pollution  monitoring  is  clear
and   unmistakable   and   wholly   inconsistent   with   a   reading   of   the   NSPE   Code   of   Ethics.

Id.    at   3.
53  NSPE   Bd.   of   Ethical   Rev.,   Case   No.   15-11   (2015),    https://perma.cc/J3BY-Z94L .
54  NSPE   Bd.   of   Ethical   Rev.,   Case   No.   16-5   (2016),    https://perma.cc/W37P-XY7E .   Admittedly,   this   is   arguably   not
anticompetitive   at   all   because   the   obligation   could   only   require   that   the   engineer   “state”   an   opinion,   but   not
necessarily   refuse   to   engineer   a   system   that   would   behave   contrarily.
These   canons   might   also   provide   solid   bases   for   an   Agreement   of   the   type   contemplated   here.   To
my   knowledge,   with   the   exception   of    Professional   Engineers ,   no   plaintif f   has   alleged   that   canons
of   engineering   ethics   violate   antitrust   laws.