AI Services: Introduction v1.3

Sharing status:

\* This document is currently written with long-termist audience in mind. As such, please do not advertise it where the missing context could cause idea inoculation.
\* To prompt the creation of a general-audience version, you can [email me](mailto:vojta.kovarik@gmail.com).
\* Public version (with commenting disabled) is [here](https://www.google.com/url?q=https://docs.google.com/document/d/1Jk2GgJnF9pLIQqg9hgX0Tvdukq6MR2qmXW3FyvqiiUg/edit&sa=D&source=editors&ust=1689706828301217&usg=AOvVaw0e21wGJdz78ETOjBy0ndrY).

---

Acknowledgment: This document has been written by myself (Vojta Kovarik), based on ideas I collected from a collaboration with Cara Selvarajah, Chris van Merwijk, Francisco Carvalho, Jan Kulveit, and Tushant Jha during the 2019/2020 [AI Safety Research Program](https://www.google.com/url?q=https://aisrp.org/&sa=D&source=editors&ust=1689706828301973&usg=AOvVaw3GHqESQsmHFVo1Crihrcw5). While they gave a lot of feedback to the text and many of the ideas are originally theirs, they might not necessarily agree with all arguments and framing presented here. All mistakes are mine. Additionally, I would also like to thank Michael Dennis for discussing modelling of service systems, and Misha Yagudin, Viliam Lisý, Anna Gajdová, Tomáš Gavenčiak, and others who gave feedback on the draft of this text.

---

Systems of Services
as a Paradigm for AI Alignment

Vojta Kovarik

31.3. 2020

This document aims to serve as an introduction for researchers who want to study the long-term impact of AI through the lens of AI services. It introduces basic concepts related to these systems and gives initial observations to enhance their initial study. It points to several relevant research fields that could be leveraged to study AI services, mentions a number of problems that seem specific to this setting, and makes suggestions for future work.

| |
| --- |
| Contents[1. Introduction](#h.mv6h35shwd1v)[2. Basic Concepts](#h.ff9s7afwzb5w)[Tasks and Services](#h.17es1kmxqfk4)[Systems of Services](#h.4g7gdavf65u7)[3. Modelling Approaches](#h.2ukea95tbs03)[Models for Specific Purposes](#h.r8bncl4zhh79)[A Simple Abstract Model of Service Systems](#h.mohtu6txrpn9)[Universal Models](#h.6x54xfsy9zua)[4. Research Questions](#h.4frz7qhnosty)[Classifying Research Questions](#h.ayilk6i3g47b)[List of Research Questions](#h.q2x0ng9anneh)[Problems Introduced by Presence of Multiple Services](#h.y5k859jsfpo7)[Problems Related to Presence of Multiple Users](#h.rfgurt70o28g)[Problems Related to Changes in Environment, System, or Users](#h.x2sjbt8tyvyw)[Problems Related to Presence of Human-Implemented  Services](#h.43z24lqwhti4)[Problems Related to Advanced Capability of Services](#h.yrf1m4lxobz9)[Problems Related to Future Technologies](#h.temt75qcxi2i)[Increasing Effectivity of Research on Service Systems](#h.b53rqxxij9bl)[5. Related Fields of Study](#h.iob8y6oskvcj)[6. Research Suggestions](#h.bmw99ws36gi7)[Getting Familiar with Further Relevant Ideas](#h.c1vq480g8vs)[Promising Topics for Investigation](#h.x50mgy7c9yih)[References](#h.q95xdkejigv5) |

1. Introduction
===============

AI has become more capable over time and it has been hypothesized that it could pose risks by becoming a highly capable agent. However, an alternative is that AI continues to operate within services, as it currently does, but that these services become sufficiently capable that their use comes to pose a risk. The possibility of this scenario (highlighted in Reframing Superintelligence [1]) suggests that studying systems of AI services could be helpful for mitigating AI risk.

To appreciate the usefulness of the service-systems framework, note that this paradigm is likely to be neither fully equivalent nor completely incompatible with the agent-like[[1]](#ftnt1) view. To the extent that the two are different, understanding service systems could expose new problems and help us determine which type of AI to expect in the long term. Even if we ultimately end up with an agent-like AI, such AI will likely emerge from and be shaped by a world with powerful and wide-spread AI services. As a result, increased understanding service systems might have beneficial consequences even for agent-like AI. To the extent that the two paradigms are equivalent, service-systems could provide a fresh angle on problems already studied in AI alignment. Moreover, both experts and non-experts must rely to some extent on their intuitions, and the current world already contains many AI services but nothing that resembles a highly-capable AI agents. Our intuitions about AI services are thus likely to be better calibrated than those about AI agents (at least up to a point). In particular, the service-system paradigm might be more accessible to experts from relevant less-technical fields (e.g., AI governance) and more suited for communication with general public.

Which types of systems should we consider when studying AI risk? First, it is critical to realize that AI-related existential risk does not come only from futuristic versions of AI that vastly exceed our abilities. Indeed, an artificial intelligence could cause existential risk as soon as it is radically transformative [5], a point we might encounter much earlier than AGI. The implication for systems of AI services is that our study should include systems whose offer of AI services is still far from comprehensive.

Secondly, viewing the world as a huge network of services[[2]](#ftnt2) and users highlights an important fact: As long as a non-trivial portion of services is implemented by humans, the AI services are likely to be closely interlinked with humans. We thus believe that even if we only care to study AI services, we cannot fully understand their impact without understanding the whole system of services - both human and AI. Consequently, we focus on hybrid systems of services, i.e., ones where services can be implemented by both AI and humans and that might contain large sectors of human-only services and institutions. And while many dynamics can (and should) be studied in the context of a much smaller system, our default example of a system of services is going to be “all the services that exist in the world”. Note that this is an important distinction between our text and Reframing Superintelligence [1], which primarily focuses on comprehensive AI services (thus having less of a need to discuss human services).

The purpose of this document is somewhat different from that of [1]. Indeed, [1] introduces an important new paradigm and provides many deep insights about the nature of AI services. However, as far as we are aware, neither of these have yet led to a follow-up technical work by other researchers. We believe this is because it is not yet sufficiently clear how to use this paradigm to study potential problems with AI, or indeed what problems in this paradigm are. Rather than focusing on object-level research, this document therefore aims to provide the initial directions for researchers interested in studying AI risk in the framework of service systems.

We start out by describing the basic terminology, examples, and modelling approaches relevant to service systems (Sections 2 and 3). We then introduce a way of classifying potential issues with service systems and describe a number of relevant research problems (Section 4). To reduce potential duplication of effort, we highlight several existing research fields that can be utilized to address these problems (Section 6). We conclude with suggestions for future research (Section 7).

2. Basic Concepts
=================

The concept of a service is central to our topic. However, this term is very general and different people might understand it very differently. To preempt possible misunderstandings, this section discusses services, tasks, and service systems. To bootstrap intuitions one might have about service systems, we also provide many examples of services and service systems. Additionally, we discuss several concepts - generality, capability, task-coverage, granularity, and degree of automatization - which allow for having more nuanced conversations about service systems.

Tasks and Services
------------------

To talk about services, we first need the concept of a task. Informally, we can assume that some description of the world is given by context and view a task as something specified by an initial state and an end state (or states) - accomplishing the task amounts to causing a transformation from the starting state to one of the desired end states. Since this concept has been frequently used and formalised in classical AI - in particular in planning [30] - we will not go into further details here. We should, however, point out that the critical problem here is finding a suitable world description. A related challenge is the [frame problem](https://www.google.com/url?q=https://plato.stanford.edu/entries/frame-problem/&sa=D&source=editors&ust=1689706828314253&usg=AOvVaw0ZCl0PFMurGN2-EuBWEC2T) - identifying which consequences of accomplishing a task are important, and which can be ignored.

Apart from chaining tasks together, composing them into higher-level tasks, or decomposing them into lower-level tasks, we can group similar tasks into domains. While a domain can, by default, be just an arbitrary collection of tasks, oftentimes it will be endowed with some partial ordering over task solutions (i.e., some way of evaluating performance). For example, potatoes cooked just right are clearly a better culinary output than overcooked ones, but our preference between potatoes and rice might depend on other variables such as the main dish. This additional structure will allow us to talk about capability of services.

We use the word “service” to refer to anything that can be used to perform tasks[[3]](#ftnt3), such as

\* a horse, a plane, a train, a self-driving car, (hypothetically) a portable teleportation device,
\* an encyclopedia, Google search engine, (hypothetically) an automated digital secretary,
\* a trained AlphaGo algorithm, human Go player, a Go-playing algorithm that takes random moves,
\* a cleaner, a restaurant attendant, a Roomba robot,
\* a mathematician, a calculator, an automated theorem-prover.[[4]](#ftnt4)

Typically, a service will be implicitly associated with a specific set of tasks - its domain. However, sometimes it makes sense to restrict a service to some subdomain (e.g., when evaluating the cleaning skills of a restaurant attendant), or consider whether the service can be used outside of its default domain (e.g., when attempting to get a mathematician in cleaning tasks). Correspondingly, one way of comparing services is in terms of generality:

Definition (Service generality): When the domain of service A is a subset of the domain of service B, we say that B is more general than A.

For any domain D, the partial ordering between solutions of tasks in D allows us to define a corresponding partial ordering on services that operate in D.

Definition (Service capability): Let A and B be two services whose domains are at least D. We say that a service A is more capable[[5]](#ftnt5) at D than service B if it performs each task from D at least as well as B. If D is clear from context, we simply say that A is more capable than B.

As an example to illustrate these definitions, we can see that, among transportation services, walking is more general than taking a train, but not more capable, and a hypothetical teleporter device that you could carry in your pocket would be more capable than both. A mathematician is more general than a calculator, but the calculator will be more capable at routine arithmetics (where what matters is speed and accuracy). Naturally, these results do not hold in all abstractions of reality - if you want to exercise in addition to moving from A to B, teleport is no longer strictly better than walking. Similarly, a mathematician might[[6]](#ftnt6) be preferable to a calculator if you also need the answer explained.

Depending on the context, we might want to specify additional properties of a service such as resource consumption (fuel consumed by a plane), cost (price of a plane ticket), access rights (you can only fly to a foreign country if you have a valid passport[[7]](#ftnt7)), and inspection and modification rights (I am not allowed to view and rewrite the source code of Google’s software).

Systems of Services
-------------------

By a system of services, we will typically mean some set of services, together with a (possibly implicit) description of how different services relate to each other, which environment they operate in, and how the system interacts with end-users. Examples of service systems include:

\* (i) All the restaurants, shops, and the movie theatre at the local shopping mall.
\* (ii) My smartphone with all the apps currently installed on it. The operating system running on my laptop, together with all programs installed on it.
\* (iii) MS Office. Google’s browser-based services.
\* (iv) All the services available before the industrial revolution, connected as they were at the time. Same thing after the revolution.
\* (v) All the services available in the current world (performed by humans, machines, computers, animals, environment).

To get a better sense for how systems of services could look like, we can also consider some hypothetical[[8]](#ftnt8) scenarios:

\* (a) A hypothetical future version of the current world, where we were able to automate most of the manual labour and tasks like driving, hairdressing, and cooking (say, using devices and humanoid robots).
\* (b) A hypothetical future version of the current world, where we automated 95% of the tasks that humans are currently being paid for doing (incl. things like programming and doing research).
\* (c) A hypothetical version of any system, where the services are being centrally controlled by a single authority (e.g., a government, an Agent-like AGI, or a single person who owns the majority of the system).

We see that some important dimensions along which service systems can vary are task-coverage, capability (of offered services), “granularity” (i.e., how narrow vs general are the individual services)[[9]](#ftnt9), and degree of automatization. Moreover, variants of the last three examples illustrate that “a more advanced system” is not necessarily the same as “a more beneficial system”. Indeed, a “better” system could fail to be beneficial for example if the system from (c) is used to support a totalitarian regime, if we become reliant on the system from (b) to the point of enfeeblement, or if the resulting economic incentives make living within the system from (a) unenjoyable for many people (e.g., due mismanaged massive unemployment).

Finally, we highlight (non-exhaustively) several classes of services that seem relevant for the study of service systems and their interaction with human society.

\* Essential services such as: Fruit-bearing trees, grocery stores, supermarkets. Caves (for hunter-gatherers), houses. Shamans, herbalists, hospitals.
\* Entertainment such as: Books, e-books. Theaters, movie theatres, TV. Board games, computer games, virtual-reality games.
\* Service Catalogues. Ways of identifying which service is relevant for my requests and desires. Some examples include asking friends, Yellow pages, or Google search.
\* Research & Development. Various activities related to identifying new technologies and products (and hence services), their study and development, application, monitoring, and improvement. For examples, we can look at most parts of the academia and industry.
\* Security and Policing. Ensuring that the system behaves according to its specification and does not fall prey to an adversarial attack. Some examples include password protection on a computer, an antivirus software, a police department, or a procedure for determining whether a new product (or service) can be safely added to the market.
\* Governance and System Design. Institutions which monitor the system and attempt to set it up in a way that benefits the users. Typical examples include, unsurprisingly, actual governments and governance-related research, while a less obvious example could be the choice of rules for an online discussion forum.
\* Long-Term Planning. Long-term planning should keep track of the big-picture strategy, identify long-term challenges, and devise counter-measures that need to be taken for the system to remain beneficial to its users. Some real-world examples are finding ways to deal with dangerous asteroids, mitigate global warming, and address unemployment from growing automatization of labour.[[10]](#ftnt10)
\* Infrastructure. In some sense, both physical infrastructure (the post office and cargo trains) and digital infrastructure (mobile phones networks, wi-fi, or TCP-protocols) can be viewed as a particular case of services that enhance (or enable) communication between other services.

As we can see, some of the current services can be viewed as “upscaled” versions of earlier services (e.g., physical calendar → Google calendar with shareable events or encyclopedias → Wikipedia). Similarly, we can hypothesize that some of the current services will get upgraded further yet. For example, we can imagine a calendar that automatically tracks my appointments without me having to input them or an automatically maintained database of humanity’s knowledge that presents its contents in a manner that is personalized based on each reader’s knowledge. Similarly, we might eventually see a world where a large portion of the security or R&D sector becomes AI-powered.

3. Modelling Approaches
=======================

In the ideal world, there would already be a common-knowledge understanding of what is our object of study, what are the key problems that need to be solved, and how these problems are stated - the “only” thing left to do would be to find solutions of those problems. However, with systems of (AI) services, not a single one of these aspects is understood sufficiently. As such, we might be interested in “modelling” service systems from several different reasons[[11]](#ftnt11):

1. enabling more effective discussions (and thinking) by grounding key concepts,
2. building basic intuitions and calibrating the existing ones,
3. identifying technical problems,
4. describing and solving technical problems,
5. deriving specialized models in which technical problems are easier to tackle.

Naturally, we should expect different models to be suitable for different purposes. In this section, we will briefly describe three types of models and discuss their uses.

Models for Specific Purposes
----------------------------

In classical AI, many specific problems already have models that are particularly suitable for addressing them. For example, Atari games can be effectively tackled using Markov decision processes, while chess and Go can be solved with the help of game-theoretical trees. Ultimately, we would similarly like to have specialized models that are particularly effective at addressing problems such as service-caused value drift, system stability in the presence of malicious actors, and other problems that might arise in systems of services. While some existing models can be used or repurposed towards this goal (see the “Relevant Research Fields” section below), many of the hypothesized technical problems with service systems are not yet associated with suitable specific models. For now, we instead describe two types of more general models that might simplify finding the more-specific models later.

A Simple Abstract Model of Service Systems
------------------------------------------

When thinking about some class of objects, it is good to have a readily-available simple abstract model that we can cheaply query to obtain basic intuitions about the behaviour of such objects. For example, whenever we think about how humans behave, we can instead ask “How would I behave?”. While this procedure is bound to lead to mistakes, we can mitigate them by asking “What are the important aspects in which other people are different from me?” and “How does my model of myself diverge from reality?”. More generally, we can reason about how a cheap model fails to be accurate, which allows us to determine whether those failures are likely to manifest in the situation at hand, and possibly derive a more suitable model.

![](https://docs.google.com/drawings/d/s416PUwmqIAszO0pDf5NS1g/image?parent=1SYgvWBe1ruDl9dQnxmjll-8COUHPycGOlLvTI68xtLA&rev=22&drawingRevisionAccessToken=i0GMC74Mptd0gw&h=315&w=292∾=1)

To give one more example, a simple abstract model of an Agent-like AGI is that of a “box” that, whenever deciding its next action, always selects the option that is optimal for some pre-specified goal. Such a model is highly useful, as long as we remember that not all hypothetical Agent-like AGIs have to be optimizing for some goal (and that even if they were, they wouldn’t be able to achieve optimality).

As a first shot at a simple abstract model for service systems, we can imagine services as input-output boxes for passing information. For simplicity, we can assume these boxes are arranged into a network, let’s say a directed one with no cycles, that has designated sensors (where messages originate) and actuators (where messages turn into actions that affect the real-world). To complement this description, we can imagine that the world further contains humans and the environment. The system interacts with these by receiving requests from the humans (one for each human) and observations from the environment. These get propagated through the network, resulting in a collection of actions (one for each actuator) which change the state of the environment. Finally, this generates some utilities for the humans.

If we want to add a notion of optimality into the model, we can associate each of the previously-arbitrary input-output boxes with some loss function, and assume the box operates optimally with respect to it. These losses can take various forms - they can depend only on the input-output pair, but also on outputs of other services. This choice will then influence whether the services compete with each other in a game-theoretical sense or whether they behave as more straightforward optimizers.

This model makes many unrealistic assumptions such as service optimality, the particular shape of the reward (resp. loss) assignment process, all requests taking the same time to be processed, humans having utility functions, and, critically, the decoupling of humans, the environment, and the system. However, it is a good starting point for discussing those simplifying assumptions, removing them, and thus obtaining more realistic models.

Regarding the comparison between Agent-like AGI and service-systems, note that in their simple abstract model, Agent-like AGI is associated with optimal agents while service systems are associated with systems consisting of optimal sub-agents. In terms of [Dennett’s three stances](https://www.google.com/url?q=https://en.wikipedia.org/wiki/Intentional\_stance%23Dennett's\_three\_levels&sa=D&source=editors&ust=1689706828327031&usg=AOvVaw28CUrgoegQ8qMAAMI8ZlIX), Agent-like AGI is closer to the intentional stance (it “wants to achieve its goal”) while service systems are closer to the design stance (services are set up to perform a particular task).

Universal Models
----------------

In a sense, quantum physics can be viewed as a suitable “underlying model” for chemistry. Similarly, rational numbers Q can be viewed as a suitable underlying model for elementary-school calculations, calculus for determining the volume of solids. More generally, for a collection C of problems that one might wish to study, we might hypothesize there is some suitable “universal” model U such that every problem P from C can be accurately described and solved using U. However, a class of problems might have multiple corresponding universal models. For example, instead of using U = Q for C = elementary-school math, we could in principle do all relevant calculations using raw set theory notation (U’). Since we want models that are not just universal but also actually useful, we can restrict our attention to “reasonable” universal models (i.e., those that aren’t “needlessly complicated for C”).

No reasonable universal model model is currently known for service systems. However, we believe that coming up with one would be a useful enterprise. Since C = “problems with service systems” might get dangerously close to subsuming “all things that could happen with the economy and the physics of Earth”, we would also like to have an “[80/20](https://www.google.com/url?q=https://en.wikipedia.org/wiki/Pareto\_principle&sa=D&source=editors&ust=1689706828329433&usg=AOvVaw2LjLL0Bp5HysZPrqbGHDrZ)” version of such model, in which a compromise is made between expressiveness and complexity. Unfortunately, even reasonable universal models will generally be too “low-level” and thus impractical for solving any specific problem P ϵ C. However, such models are nonetheless useful from several reasons:

\* Universal models can be used to “ground” and formalize all more advanced concepts, objects, and problems relevant to C.
\* When working with a subclass C’ ⊂ C, we can make additional simplifying assumptions, and use U to derive a simpler model U’ universal for C’. Similarly, we can derive a specialized model for a particular problem P ϵ C.
\* If we want to study the interaction of problems P1 and P2 (for which we perhaps have specialized models), the knowledge of a universal model can make it easier to find a model suitable for addressing both P1 and P2.
\* Finally, the universal model is useful even if we never formally use it to derive arguments and models. Indeed, whenever our results seem to be wrong, the fact that the formalization should in principle be possible allows us to use the heuristic “the mistake is more likely to occur in the hard-to-formalize step”.

4. Research Questions
=====================

While there is currently no exhaustive list of research that needs to be done in regarding service systems (or of potential problems with them), the generality of the topic suggests that any such list is going to be “rather extensive”. To improve the efficiency of discussing these problems and reasoning about them, we first introduce an informal taxonomy for research questions and identify topics which are already being sufficiently tackled in classical AI and AI Alignment. In the second part of the section, we use the taxonomy to generate and organize a list of research areas for service systems. While this section could also contain a discussion of which tools are likely to be useful for tackling which problems, we defer such analysis into the “Related Fields of Study” section.

Classifying Research Questions
------------------------------

We now introduce several axes along which we can carve up the service-system research space. As we show in the next subsection, these axes can be used to classify and brainstorming relevant research questions: For example, were traffic congestion a particularly important problem, we could put it into the category of “problems related to having multiple users” and generate further problems by asking what else falls under this category. The axes that seem the most relevant are the following:

\* Single human vs multiple humans. For each potential problem with the system, we can ask whether it is crucial that the system interacts with multiple users. Some problems can thus be studied in the single-user setting, while others require the (potentially more complicated) general case.
\* The number of services. We can ask whether the core of the problem is mostly about a single service or a collection of a small number of services or whether it can only be explained on the level of the whole system. While this trichotomy is somewhat false, it nonetheless gives useful hints about which models and tools we should use to attack the problem.
\* Synchronic vs diachronic. Similarly, we can ask whether a problem already occurs in settings that are closer to being stationary (synchronic), or whether it is crucial that the system (or its users, or the environment) develops over time. For example, algorithmic bias is a synchronic problem while value drift is a diachronic one.
\* Problem solving vs theory building. Finally, some important research activities are less about trying to fix something in a specific service system and more about finding ways that will enable us to think about service-systems more effectively. Some examples that fall mostly towards the “theory building” side are Sutton’s [Bitter Lesson](https://www.google.com/url?q=http://www.incompleteideas.net/IncIdeas/BitterLesson.html&sa=D&source=editors&ust=1689706828334045&usg=AOvVaw3OQ4J3erKogJJGKq7zvZT-), the paper Concrete Problems in AI Safety [26], the act of coining a new definition, and the present text. In contrast, the AlphaZero algorithm [27] and most applications of deep learning are more on the “problem-solving” side.

There are other axes worth considering, such as

\* how many services are performed by humans,
\* how capable are the AI services (much stronger than the current ones, or not?),
\* whether the problem at hand is closely tied to hypothetical future technologies that haven’t yet been developed (or fully developed; e.g., quantum computers, nano-technology),
\* and how centralized is the system (by default, we assume a situation similar to the current world, where the system is neither fully centralized nor fully decentralized).

Since many of the problems with service systems can already be studied in the framework of classical AI or AI safety in the frame of utility maximization, we use the remainder of this section to briefly sketch how these fields relate to the above categories (and leave problems that “primarily” concern systems of services for the next section).

Current AI (and machine learning in particular) predominantly[[12]](#ftnt12) uses the “baseline” assumptions and thus focuses on single-human single-service synchronic scenarios with AI and technologies that do not qualitatively differ from the existing ones. Examples of such problems include:

\* natural language processing, inverse reinforcement learning,
\* general “capabilities” research,
\* robustness, interpretability, verification,
\* avoiding side-effects, corrigibility.

Another problem worth highlighting is that of competitiveness: to prevent a system that is known to be safe from being replaced by less safe alternatives, we need to ensure that its capability doesn’t fall too far behind its competitors.

Some problems in “AGI safety” [25] go beyond the baseline assumptions, for example by additionally considering

\* diachronic problems such as value drift in humans or ensuring that recursively self-improving AI remains beneficial,
\* theory building such as Bostrom’s classification of ways in which we could create smarter-than-human systems [2] or MIRI’s reconciliation of probabilistic reasoning with uncertainty about logical facts [28],
\* potential problems with more advanced AIs such as mesa-optimization [3] (i.e., the accidental emergence of “agency” as a consequence of optimization pressures), or “treacherous turn” whereby an advanced AI acts against our interest unexpectedly (sometimes in conjunction with the use of novel technology) [2].

List of Research Questions
--------------------------

We now use the tools from the previous subsection to lay out research tasks relevant to services of systems. In particular, we list some problems that are “related to” the service-system setting - that is, those that are either introduced by this setting or for which the interaction with it is sufficiently “game-changing”. The structure of this section is based on the observation that each problem has some “critical assumptions” without which its study would be meaningless (as with the absurdity of investigating congestion in a system with only one vehicle). In this spirit, we go through the (arguably) “non-default” assumptions from the previous section. For each assumption A, we list some of the problems with service-systems for which A is critical in the above sense. When reading the list, keep in mind that it is not meant to be complete and that some problems might fall under multiple categories. We conclude by giving several avenues for increasing the overall effectivity of research in this area.

### Problems Introduced by Presence of Multiple Services

First, it matters that the system consists not of a single service, but of multiple services that interact with each other. While many consequential questions are already studied by game theory, economy, or complexity science, the fact that many of the participants are AI algorithms gives these problems a different flavour. Indeed, unlike humans, AIs might be able to act near-optimally, make credible commitments, or even inspect each other’s source code. This multi-agent setting also highlights several other problems:

\* Undesired appearance of agency. Often, the stability of the system might rely on services behaving predominantly as tools with no goals of their own. However, it could potentially happen that an agent-like behaviour would arise in the system. Apart from a single service becoming an agent (as a result of mesa-optimization or unsafe-but-granted user-request), this could hypothetically be caused by a symbiosis of several narrow-purpose services[[13]](#ftnt13). Alternatively, agency could “emerge” in some yet-unforeseen manner on the level of the whole system, perhaps akin to how a brain is a general intelligence implemented on neurons.
  A related hypothetical issue is that of “[ascended economy](https://www.google.com/url?q=https://slatestarcodex.com/2016/05/30/ascended-economy/&sa=D&source=editors&ust=1689706828338740&usg=AOvVaw1HbKTexdbfcS1W9muxRfYB)”, wherein the market could gradually get decoupled from humanity’s interests while at the same time being difficult or impossible to shut down.
\* Error propagation and correlated failures. Inevitably, some parts of the system will sometimes exhibit errors. Such errors could cascade from service to service or appear in many places at once. Indeed, the latter case is likely if a similar solution is adopted in many different places in the system. The system needs to be able to contain such malfunctions and continue to function well, even in multiple simultaneous failures.
\* Communication along illegal channels. Services might sometimes be required to communicate using a specific protocol. However, they might develop ways of bypassing these rules (e.g., using unrelated events to coordinate action or AI services unexpectedly communicating through non-digital means). Since the desirable properties of the system might depend on these protocols being followed, we need to develop methods of preventing (or disincentivizing) such communication.

Moreover, there are multi-service and system-wide variants of many of the single-service problems. Indeed, some examples of such problems include understanding the system (interpretability, verification), preventing unintended consequences of running the system (avoiding side-effects), “turning off” parts of the system safely if they start behaving undesirably (corrigibility), and the system being replaced by a less-understood Agent-like AGI (competitiveness).

### Problems Related to Presence of Multiple Users

Secondly, an important aspect specific to service-systems is that the multi-user scenario becomes the default. This raises topics such as the following:

\* Magnifying human conflict. In a decentralized system, the growing availability of powerful services will likely raise the stakes in existing human conflicts, but will not necessarily offer tools for their mitigation. As the example of social media and social bubbles suggests, services might also cause new conflicts and deepen the existing ones. We will require methods for improving the offense-defense balance [29] and diffusing conflicts between humans.
\* Resistance to blackmail. Despite the best efforts of the security services, actors might appear who will attempt to exploit the system. Indeed, such actors could be powerful users, come from outside of the system, or arise within it (for example via a random mutation of services). Apart from the risks listed elsewhere, the system should be prepared to deal with the possibility that such actors could attempt to blackmail users or services.
\* Vastly increased importance of cyber-security. As automated services become more comprehensive and ubiquitous, any security holes in the system could have far greater consequences. A particular case of this issue is the need for careful management of “access rights” in future systems - determining whether a given user (or service) should be granted a particular request or not.

### Problems Related to Changes in Environment, System, or Users

Thirdly, systems of services offer a new perspective on the diachronic problems of ensuring that the system, the environment, and humanity develop desirably. Indeed, in an Agent-like AGI scenario, we can transform the problem into ensuring that the entity with a decisive strategic advantage is sufficiently capable and its intent is aligned with our goals. In contrast, sufficiently complicated systems of services will likely lack a single actor (or service) that can oversee the whole system. The problem thus changes to designing the system such that it remains beneficial even as it, its environment, or its users change over time.[[14]](#ftnt14) Some problems in this vein include:

\* Safe R&D services. As new services are developed, either to satisfy a direct request of some user or as part of the system improving itself, we need to prevent the addition of services that are directly harmful or that would break the system in some other way.
\* Maintenance of the system. The system needs to be designed in such a way that it keeps itself functional even as new services get added, without ever posing a catastrophic risk.
\* Long-term planning. If we are to safely navigate through potential future problems, the system needs to contain procedures that identify such problems and plans accordingly. For example, it needs to be able to deal with detecting dangerous asteroids, global warming, overpopulation, human enfeeblement, and other issues. Moreover, there also needs to be a process of turning these plans into action, even if they require large-scale coordination. (Notably, this is something that humanity has not yet mastered.) Whether this can be achieved without invoking some form of “agency” is an important open question.
\* Value drift and addictive or manipulative services. One side effect of new technologies is that our interaction with them can cause lasting changes to our values. For example, not only have we have gotten worse at handwriting as computers became commonplace, we are now much less likely to consider handwriting a valuable skill. Moreover, as witnessed by the issue of fake news and, historically, the ultimate fate of fire-water consuming native Americans[[15]](#ftnt15), the side-effects of introduction of novel attractive services can sometimes be undesirable or even fatal. And while this problem isn’t specific to AI services, it their inclusion makes it significantly more threatening. Indeed, it is plausible that unless this issue is explicitly dealt with, superintelligent variants of present-day services could drastically reshape society in a matter of days.
\* System stability. Unless the system is sufficiently secure, a powerful service or a user with high access level could gradually co-opt the majority of services towards its goals, leading to an undesirable state for most users. A similar outcome could happen as a result of a powerful company (i.e., a group of services with a human component) gaining dominance over the system through economic means.

### Problems Related to Presence of Human-Implemented  Services

Fourth, it is critical that the present-day services are still mostly performed by humans, and the hypothetical transition to fully- or mostly-automated systems is going to proceed in multiple (or rather, many) steps[[16]](#ftnt16). This consideration highlights several concerns:

\* Labour displacement. New work opportunities will need to be created for those people whose jobs have been replaced by automated services.
\* Eventual unemployability. It is likely that many people will eventually become unable to provide enough value to sustain themselves. As an upper bound, this will naturally happen once a majority of tasks get automated (with the remaining ones requiring an exceptional amount of skill or intelligence). However, this problem is likely to arise much earlier, once the rate of automation and change surpasses people’s ability to adapt and requalify to a new task. It is prudent to provide all affected people with an alternative source of income. Additionally, the “inability to contribute” might have undesirable psychological consequences for many people. These should be dealt with or, preferably, prevented.
\* The loss of dignity. As a twist on the problem of unemployability, it might happen that before running out of economically viable tasks for less qualified or adaptable (e.g., due to age) people, we will run out of tasks that are compatible with their happiness or dignity. (As an example, consider the scenario where the only job available to a former high-school teacher is “providing training data for ML algorithms”, which consists of repetitively performing trivial manual tasks on camera.) While such situations might have been unavoidable in the past (or perhaps they still are at the present day), we might encounter them even at a point where not having anybody perform those tasks will only constitute a negligible economic cost. In such situations, we should make sure to remove economic pressures that might effectively force people into performing such tasks.
\* Adaptability of the humans in the system. If the rate of progress keeps increasing, it could happen that people in key (and irreplaceable) positions - e.g., law-makers - would become unable to keep up with the system’s changes. This effect could become particularly pronounced with economies of scale and discontinuous automatization, as technologies go from “just below economically feasible” to “just above net positive and adopted everywhere“. We need to develop methods for mitigating these effects as much as possible.

### Problems Related to Advanced Capability of Services

Fifth, the services in future systems might become significantly more advanced than their current analogies. For example, AI might be able to create perfect fake video recordings, or get so good at targeted advertisement that the current democratic system becomes obsolete (unless countermeasures are devised in time). It is currently unclear to us which problems this implies. However, one particular consequence of this effect is that we should take care to aim either for “future-proof” solutions (to the extent this is possible), or at least for near-term solutions that will put us in a better position once it comes to solving the longer-term issues. To see why, note that adopting “whatever does the job” solutions now might cause the entrenchment of a poor solution as more and more other tools are built on top of it. Once firmly in place, replacing such a broken solution might often be much harder than devising a brand new one from scratch. Indeed, an excellent example of this phenomenon is the null-pointer that is present in many programming languages despite being the cause of many security risks[[17]](#ftnt17).

### Problems Related to Future Technologies

Sixth, the solutions we devise should count with the technologies beyond what we currently have, and with the possibility of encountering new technologies that significantly change the dynamics of the system. Some problems related to this are:

\* Physical security. With further progress in areas such as robotics, automated services will become able to significantly affect the physical world. Apart from creating laws to govern the digital realm, we should thus ensure that services do not circumvent these laws by physical means. A related problem is that services that aren’t allowed to communicate with each other could bypass this restriction through the real-world (say, by sending postcards or using unrelated real-world events as a basis for a learned coordination mechanism).
\* Preparedness for game-changing technologies. The introduction of significantly novel technologies could break solutions that have worked until now - an example of a potential concern is the impact of quantum computing on cryptography [23]. Moreover, novel technologies could significantly shift the demand for existing services and thus pull the system into a configuration far from the one for which it has been optimized. We should ensure that the system remains beneficial (and stable as much as this is possible) despite events such as these.

### Increasing Effectivity of Research on Service Systems

Finally, we consider the “theory-building” (and field-building) issues around increasing our understanding of services systems and directing future research more effectively. Some of the topics here are:

\* Mapping out potential problems. We need better tools for identifying potential downsides of service systems and talking about them. In particular, it would be useful to have models and terminology that allows us to have better discussions about this topic.
\* Identifying truly novel problems vs connections to existing results. To better leverage the existing knowledge on the one hand and effectively pursue novel research directions on the other, we should map out which problems can and cannot be tackled using existing methods.
\* Clarifying specific problems. To solve the problems we identify effectively, we need to find ways to talk about them in as precise language as possible. In particular, any problem should have a corresponding model which describes it well without incurring significant overhead by talking about aspects irrelevant to the issue.
\* Forecasting future development. To focus research, governance, and strategic efforts on problems that will actually matter, we should reduce our uncertainty about the hypothetical development of service systems as much as possible. To this end, we should develop better methods for forecasting AI (and service systems in general).
\* Devising terminology accessible to the target audience. Various problems in service systems are either directly interdisciplinary or at least spread over different research fields (ranging from theoretical computer science to psychology and governance). As a result, it is important to formulate problems and communicate ideas in ways that allow and incentivize the participation of people with the appropriate expertise.
  This problem is particularly important to address since the paradigm of systems of services is, arguably, more accessible to the communities around AI ethics, strategy, governance, and forecasting than the paradigm of superintelligent agent-like AIs. Indeed, unlike the issues discussed, for example, in Bostrom’s Superintelligence, many potential problems with powerful service systems already manifest, in diminished forms, with present-day technologies. Such problems are thus more suited for discussion (and analysis) by more general audiences.

5. Related Fields of Study
==========================

AI services are a broad and important phenomenon, and there will likely be many different fields outside of AI that have something to contribute to their study. To prevent a duplication of effort, this section suggests some research fields that may contribute to the study of systems of services. Additionally, it presents an initial commentary on why these fields might be relevant and why their interaction with systems of (AI) services is meaningful. However, please note that the list is not meant to be exhaustive, and not all parts have been run through experts in the corresponding fields.

For readers who already have expertise in one of these fields, a worthwhile future work would be to look in more detail at the connection between their field and service systems. This effort could have two specific outputs: First, one could create, e.g., a summary of relevant existing results to make those results more accessible to researchers interested in service-systems. Second, if one can explain the importance of some service-systems problem in terms of a different research field, the work on that problem can be accelerated by leveraging the community that already exists around that field. (These fields are not listed in any particular order.)

AI strategy and policy research. An interaction that seems particularly relevant is the one between AI and governance of AI [21] (and fields related to it). In one direction, laws, standards, and other norms directly influence interactions between services, thus influencing the dynamics of the service-system. There are also indirect effects, such as funding strategies differentially affecting which research gets done, or relationships between different nations changing how service-systems are applied (e.g., whether more emphasis is placed on military power or public good). In the opposite direction, the paradigm of AI services should be much more accessible and credible to a more general public than that of Agent-like AI (since AI currently takes the form of services). As a result, it seems essential to frame potential problems with advanced AI (also) in terms of AI services, such that they can be effectively communicated to more general audiences (once this becomes desirable).

AI ethics. The emerging field of AI ethics focuses on ensuring that applications of AI algorithms do not lead to morally-undesirable outcomes. At the moment, the field’s primary interest lies in near-term issues such as algorithmic bias and morally-relevant decisions made by self-driving cars. On their own, these topics are undoubtedly important. However, we believe the field’s potential is far greater. Indeed, the key insight is that many present-day issues are in fact “scaled-down” variants of more general problems. For example, the expected labour displacement caused by the arrival of self-driving cars foreshadows the general issue of what to do once human work becomes economically unprofitable. Once these interactions between near- and long-term issues become clearer, we can prioritize solving near-term problems in ways that put us into a better position to tackle the long-term challenges. Conversely, the community’s failure to acknowledge these connections could lead to the entrenchment of near-term solutions that make it much more difficult to adopt good solutions in the future. Since the AI ethics community is plausibly larger than the long-termist community, communication with this field seems to be of particularly high importance. We believe that the service-systems framework is superior to the Agent-like AGI for this communication (due to its potentially higher accessibility and credibility).

Neuroscience, predictive processing, and, to some extent, psychology and sociology. To determine whether the impact of the service system is beneficial, we should improve our understanding of what humans “want” (in the many different meanings of the word). Moreover, humans have many cognitive biases and other limitations that sometimes make it impossible for them to take actions that are in their interest. By default, services tend to exploit these weaknesses whenever possible - not out of malice, but as a result of competitive pressures within the system. To prevent such exploitation by progressively stronger services, we need to understand human limitations and design appropriate safeguards. While many of our historical attempts to understand human wants and limitations come from the fields of psychology and sociology, we hope that a grounded understanding can come from neuroscience. Relatedly, some researchers believe that the predictive processing model is particularly relevant in this mission.

Economics. To understand the interactions between different services (and groups of services), we can draw inspiration from economics. To some extent, economics could also be useful directly, for modelling service systems; however, it is possible that the specifics of AI service systems will be sufficiently different to prevent a direct application. For example, AI might revolve about currencies that behave differently from money (compute, access rights) and be subject to more regulation (or a very different one) than classical markets. However, we might nonetheless benefit a lot from the field’s tacit know-how for distilling real-world situations into models that can be analyzed formally and from its insight into topics such as the principal-agent problem or advertising.

Behavioural economics. An important limitation of economics is that humans are often sub-optimally rational, and behave nothing like the idealized “homo economicus”. Indeed, it is likely that it is fundamentally flawed to think of humans as even trying to optimize some utility function. Since the field of behavioural economics deals with precisely these complications, we believe we can draw upon it to gain a more realistic picture of interactions between the system of services and its human users.

Game theory, multiagent systems, mechanism design. Once we have formally described situations of interest within the system, we can use tools from game theory and multiagent systems [20] to predict the likely outcomes. Inversely, we can use the knowledge of mechanism design to set up the system in a way that leads to beneficial outcomes. However, we should keep in mind that the main historical motivation behind these fields (perhaps except for multiagent systems) is the study of interactions between human actors. As a result, it might be necessary to ask slightly different questions from those that have been asked before and rethink the assumptions made when addressing them. To give one example, while humans often have trouble trusting each other’s promises (and justifiably so), AI services might be capable of  making their commitment credible. When widespread, this new assumption would significantly shift the questions of interest in bargaining scenarios.

Design of operating systems. Understanding operating systems, and their design seems relevant to service systems for two reasons. Firstly, since traditional computers, smartphones, and other similar devices are likely to remain a part of the global system of services for some time, knowledge of operating systems is likely to be relevant directly. Secondly, we can draw insight from analogies between large-scale service-systems and operating systems. This might help us identify service-system analogies of issues (and possible remedies) such as those with the management of access rights for different users, resource sharing, and security tools like firewall and antivirus. Moreover, operating systems can be used as a conceptual test-bed for proposed solutions to some problems with service-systems (e.g., those related to security services).

Cybersecurity. As service systems get progressively more automated, it is likely that cybersecurity will accordingly grow in importance. Similarly to the design of operating systems, we should expect cybersecurity to be relevant both through direct application and through analogies between present-day cybersecurity concepts and issues with more advanced systems.

Process algebras. Similarly to how lambda calculus can be used as an underlying formalism for computable functions and functional programming, process algebras [19] can be used to model processes that run in parallel. Consequently, they might be useful for formalizing systems of services that operate in the digital domain (i.e., many of them). A limitation of this approach is that the resulting description might be too “low-level” to be practically useful, and might not apply to services that aren’t fully automated.

Complex systems. The analysis of complex systems [7] investigates how relationships between a system's components give rise to its collective behaviors and how the system interacts and forms relationships with its environment. Since networks of interacting AI services might exhibit properties that would not be apparent from observation of their isolated components, they are amenable to this type of analysis. Complexity science might thus inform our decisions when designing the system, by predicting how the system’s parameters influence its structure and properties (such as resource expenditure or robustness to failures).

AI forecasting. To ensure that research on AI risk focuses on areas that end up being relevant, we need to know which scenarios are likely to occur. Providing this clarity is one of the goals of the emerging field of AI forecasting [22]. To see how this field might interact with service-systems, note that AI is currently affecting the world through collections of AI services (rather than agent-like AIs). One way in which AI forecasting could be helpful is determining how long this situation is likely to continue. Moreover, if AI keeps the form of services for a substantial amount of time, their interaction with AI forecasting might grow in importance. In such a world, we should make more service-centered forecasts (to better inform our decisions) and improve our understanding of service-systems (to enable better forecasts).

Non-exhaustive list of other related topics. First, while organizational theory traditionally studies human institutions, it could likely offer a lot of insight into systems that include AI services.[[18]](#ftnt18) To understand how systems of AI services get developed, behave, and get maintained, we can look to the extensive knowledge accumulated in software engineering.[[19]](#ftnt19) Another problem where we expect to find pre-existing results is service specification - that is, describing the intended behaviour of a service such that no matter how it ends up being implemented, it will robustly and beneficially perform its intended purpose. This is likely relevant to formal verification (see, e.g., the [DeepSpec](https://www.google.com/url?q=https://deepspec.org/main&sa=D&source=editors&ust=1689706828357039&usg=AOvVaw1578z7o5Qr55hn5OdirleN) initiative). Note also that in some sense, service specification is precisely the goal of contracts created by law. As a result, we can look to this field for inspiration and intuitions for which problems to expect. Another area with a notable amount of existing literature are “service ecosystems” [8-18]. While the area’s usage of the term “service (eco)systems” is somewhat different from ours, some of the literature might be relevant. Finally, advances in computer science and AI give us access to tools such as blockchain, smart contracts, and zero-knowledge proofs. These methods could transform dynamics of service systems, for example by serving as cooperation-enhancement tools.

6. Research Suggestions
=======================

In this section, we present our subjective ideas (Vojta’s in particular) on which steps might be useful to take towards making progress in the area of service systems. Most of the items below can also be viewed as bookmarks for “all the team-members from AISRP (Cara, Chris, Francisco, Jan, Tushant, Vojta) have some understanding of these ideas, and will probably be happy to discuss them”. Note that this doesn’t necessarily mean that they agree with arguments, conclusions, or framing of the presented ideas.

Getting Familiar with Further Relevant Ideas
--------------------------------------------

Since the present text cannot capture all the existing relevant ideas, we start by giving a few pointers to ideas that can make further investigation of AI services more effective.

Comprehensive AI services. An important resource is Drexler’s extensive technical report [1] on reframing superintelligence in terms of AI services. As we mentioned, our text describes systems of varying degree of automation while [1] focuses on systems that automate the whole R&D process, possibly soon resulting in comprehensive coverage of the task-space. We believe that studying this report can be useful for two reasons: Firstly, taken as a whole, the report paints a coherent picture of how advanced AI-service systems could look in the future. It also lists some risks associated with such systems (see, e.g., Section 14). Second, the overall text identifies and informally states many important hypotheses about the nature of the AI systems, such as the claim that there will be no compelling incentives to replace comprehensive narrow services by an Agent-like AGI (Section 12). We believe it would be valuable to (a) map out the different hypotheses and assumptions made in the report and (b) formalize specific hypotheses and explore them further.

Agency as an accidental by-product of optimization. Apart from discussions about the economic feasibility of narrow AI services (vs. general AI agents), a recent paper [3] by Hubinger et al. raises the concern of “mesa-optimization”[[20]](#ftnt20), i.e. that agency might also appear as a result of optimization pressures, somewhat akin to how human intelligence is a result of evolutionary pressures. This issue seems particularly relevant in the context of AI services, and we recommend reading the paper for more details.

Radically-transformative AI. While artificial general intelligence certainly has the potential to have an extreme impact on humanity, even much narrower systems could already result in existential risk and other high-impact events. To get more clarity on this topic, we have found it helpful to get familiar with the terminology (and discussion) presented in [5].

Undesirable stable equilibria. With agent-like superintelligent AI, the main concern is that the goal of such AI would conflict with humanity’s interests. The situation is different in the context of systems of AI services since services might often not have goals in any meaningful sense - they might “just do things”. However, the service-system paradigm has its own nemesis in the form of economic (or other) pressures perpetuating situations that nobody is happy with, but nobody can escape. An example of such a dynamic is the “[race to the bottom](https://www.google.com/url?q=https://en.wikipedia.org/wiki/Race\_to\_the\_bottom&sa=D&source=editors&ust=1689706828360804&usg=AOvVaw1kM1yhDnyukh9HnTyHfjij)” between states as they underbid each other on environmental policies to become more attractive to outside investors. To get an impression of a range (and the overwhelming magnitude) of other issues of this type, we refer the reader to a post [Meditations on Moloch](https://www.google.com/url?q=https://slatestarcodex.com/2014/07/30/meditations-on-moloch/&sa=D&source=editors&ust=1689706828361271&usg=AOvVaw3SoXVwO3NKwhBm3xQ90H9q) by Scott Alexander, or Yudkowsky’s (short and accessible) book Inadequate Equilibria. Somewhat related to this topic is the recent research agenda on [Cooperation, Conflict, and Transformative AI](https://www.google.com/url?q=https://www.alignmentforum.org/s/p947tK8CoBbdpPtyK&sa=D&source=editors&ust=1689706828361577&usg=AOvVaw1GUuRQ-mAQQyaSKMjAy7Wv).

Mathematical modelling. When attempting to come up with useful mathematical models for systems of AI services, we kept going back to questions such as “What purpose is the model supposed to serve?” and “How should a good model even look like?”. While there are likely further resources unknown to us, the following series of posts on Artem Kaznatcheev’s blog was quite illuminating: [Methods and morals for mathematical modeling](https://www.google.com/url?q=https://egtheory.wordpress.com/2018/10/06/metamodel-linkdex/&sa=D&source=editors&ust=1689706828362206&usg=AOvVaw04Pk3pCfSA2my47h5MRdKN).

Promising Topics for Investigation
----------------------------------

In this subsection, we list some of the directions for future research that seem particularly important to us. (For a more extensive list of useful topics, recall that: (i) the “Problems” section contains many potential issues with service systems, (ii) the “Related Fields of Study” section lists many areas whose connection to service-system should be understood better, and (iii) the extensive technical report [1] should provide fertile ground for further investigation.)

Tools or agents? Many people seem to believe that while Agent-like AGI and “fully-automated comprehensive system of services” might have similar capabilities, there is some fundamental difference between the two types of AI. At the same time, there seems to be a general confusion around this topic. Some relevant questions are:

\* Can we find a framework in which these similarities and differences could be explained or dissolved?
\* In particular, does there perhaps exist a formalization of “agency” that can differentiate between the two?
\* If there are meaningful distinctions, how do they translate into what it means to “align” each type of AI?
\* How does the effectivity of narrow-purpose algorithms differ from the effectivity of general-purpose algorithms? Should we expect economic pressures towards generality (and maybe even Agent-like AGI)?

Clarifying the connection between near-term AI issues and X-risk. We believe that the link between near- and long-term AI issues is currently insufficiently understood and, as a result, underappreciated by the AI alignment community. For example, we would like to have clearer answers to questions such as: Which AI risks are associated with artificial general intelligence, and which are already present with advanced narrow systems? Which problems scale from the present-day AI systems to the superintelligent ones? For which of these problems is it the case that solving the near-term issue in the wrong way will make us much worse off when we face the superintelligent version later (due to, e.g., a lock-in of the bad approach)? Which “problems with AI” are in fact “merely” general problems with our society, magnified by the power of AI? Understanding these questions should make our discussions more effective, enable us to focus on problems that matter the most, and - we predict - diminish the tension between long-termist views and the more general AI community. (We recommend the recent paper [24] which clarifies some key concepts relevant to this topic and argues against viewing the near-term vs long-term axis as a “dichotomy”.)

Utilizing the field of AI ethics. It is plausible that the research community around AI ethics is currently larger than the long-termist community in AI. As a result, it might be having a more substantial impact on shaping the trajectory of the whole AI field, potentially in ways that are sub-optimal from the long-termist perspective. At the same time, the stated goals of both communities are similar: ensuring that AI has a beneficial impact on society. We believe that a potentially high-impact action would be to clearly lay out the connections between AI alignment and AI ethics and bring them to the attention of the AI ethics community in the right way. However, if unsuccessful, the “high impact” of this endeavour could easily turn out to be a highly-negative one (due to effects such as the [unilateralist’s curse](https://www.google.com/url?q=https://concepts.effectivealtruism.org/concepts/unilateralists-curse/&sa=D&source=editors&ust=1689706828365189&usg=AOvVaw2m5N\_Av\_NYSJy4GUP\_8H8M) and [idea inoculation](https://www.google.com/url?q=https://en.wikipedia.org/wiki/Inoculation\_theory&sa=D&source=editors&ust=1689706828365578&usg=AOvVaw2jG2LYPbFTPRb6NUAVh1Cd)). As such, we also think that this task should only be attempted by people who are experienced at communicating ideas on this level and well-positioned to do so.

References
==========

1. Drexler, K. Eric. "Reframing Superintelligence." (2019).
2. Bostrom, Nick. Superintelligence. Dunod, 2017.
3. Hubinger, Evan, et al. "Risks from Learned Optimization in Advanced Machine Learning Systems." arXiv preprint arXiv:1906.01820 (2019).
4. Yudkowsky, Eliezer. Inadequate Equilibria: Where and How Civilizations Get Stuck. Machine Intelligence Research Institute, 2017.
5. Gruetzemacher, Ross, and Jess Whittlestone. "Defining and Unpacking Transformative AI." arXiv preprint arXiv:1912.00747 (2019).
6. Simler, Kevin, and Robin Hanson. The elephant in the brain: Hidden motives in everyday life. Oxford University Press, 2017.
7. Barabási, Albert-László. Network science. Cambridge university press, 2016.
8. [Considerations on Modeling Service Ecosystems](https://www.google.com/url?q=https://www.ceeol.com/search/article-detail?id%3D668918&sa=D&source=editors&ust=1689706828367456&usg=AOvVaw1ZLOiuT9tiH4wZkAwV34mj) (2018)
9. [The Application of a Service Ecosystems Lens to Public Policy Analysis and Design: Exploring the Frontiers](https://www.google.com/url?q=https://journals.sagepub.com/doi/10.1177/0743915618818566&sa=D&source=editors&ust=1689706828367909&usg=AOvVaw1\_5Ne2\_NzPp3p4NecBrORX) (2018)
10. [Business modeling for service ecosystems](https://www.google.com/url?q=https://www.researchgate.net/publication/220884279\_Business\_modeling\_for\_service\_ecosystems&sa=D&source=editors&ust=1689706828368344&usg=AOvVaw1yYESdveffS1bkye7q5uMa) (2010)
11. [Modeling Service Ecosystems Innovation](https://www.google.com/url?q=https://www.researchgate.net/publication/282495905\_Modeling\_Service\_Ecosystems\_Innovation&sa=D&source=editors&ust=1689706828368781&usg=AOvVaw0goz1oPYBayAwEOG9M3jY\_) (2015)
12. [Handbook of Research on Service-Oriented Systems and Non-functional Properties: Future Directions](https://www.google.com/url?q=https://www.amazon.com/Handbook-Research-Service-Oriented-Non-functional-Properties/dp/1613504322?SubscriptionId%3DAKIAILSHYYTFIVPWUY6Q%26tag%3Dduckduckgo-brave-20%26linkCode%3Dxm2%26camp%3D2025%26creative%3D165953%26creativeASIN%3D1613504322&sa=D&source=editors&ust=1689706828369364&usg=AOvVaw1pSKxBlDiWP8O9rmMeIPPm) (2011)
13. [The User Perspective on Service Ecosystems: Key Concepts and Models](https://www.google.com/url?q=https://link.springer.com/chapter/10.1007/978-3-319-65151-4\_34&sa=D&source=editors&ust=1689706828369797&usg=AOvVaw1D\_zvIGERhIY\_4zfTgup5X) (2017)
14. [A Service Description Method for Service Ecosystems - Meta Models, Modeling Notations, and Model Transformations](https://www.google.com/url?q=https://fis.uni-bamberg.de/handle/uniba/286&sa=D&source=editors&ust=1689706828370190&usg=AOvVaw0Yp4B7x5VuKcfSEsb8SvEd) (2011)
15. [Business Modeling for Service Descriptions: A Meta Model and a UML Profile](https://www.google.com/url?q=https://www.researchgate.net/publication/221592291\_Business\_Modeling\_for\_Service\_Descriptions\_A\_Meta\_Model\_and\_a\_UML\_Profile&sa=D&source=editors&ust=1689706828370697&usg=AOvVaw0Qt\_63IHeaUrjr\_serc4By) (2010)
16. [IBM - Service-oriented modeling and architecture](https://www.google.com/url?q=https://www.ibm.com/developerworks/library/ws-soa-design1/index.html&sa=D&source=editors&ust=1689706828371116&usg=AOvVaw1TfXLKCM1E9u2qpX0CgIyj) (2004)
17. [Service Value Properties for Service Ecosystems: A Reference Model and a Modeling Guideline](https://www.google.com/url?q=https://www.researchgate.net/publication/247927245\_Service\_Value\_Properties\_for\_Service\_Ecosystems\_A\_Reference\_Model\_and\_a\_Modeling\_Guideline&sa=D&source=editors&ust=1689706828371643&usg=AOvVaw0CCuojPR1xEPfbxL4BO2Rr) (2009)
18. [Reflexive and Evolutional Digital Service Ecosystems with Models at Runtime](https://www.google.com/url?q=http://ceur-ws.org/Vol-2019/mrt\_2.pdf&sa=D&source=editors&ust=1689706828372029&usg=AOvVaw0eg5z52x18Blr0r6Ck8JWL) (2017+)
19. Introduction to Process Algebra, Baeten-Beek-Rooda ([link](https://www.google.com/url?q=http://mate.tue.nl/mate/pdfs/8509.pdf&sa=D&source=editors&ust=1689706828372428&usg=AOvVaw1tQzXDhxv5I17ulmymv3Y8))
20. Shoham, Yoav, and Kevin Leyton-Brown. Multiagent systems: Algorithmic, game-theoretic, and logical foundations. Cambridge University Press, 2008.
21. Dafoe, Allan. "AI governance: A research agenda." Governance of AI Program, Future of Humanity Institute, University of Oxford: Oxford, UK (2018).
22. Gruetzemacher, Ross. "A Holistic Framework for Forecasting Transformative AI." Big Data and Cognitive Computing 3.3 (2019): 35.
23. De Wolf, Ronald. "The potential impact of quantum computers on society." Ethics and Information Technology 19.4 (2017): 271-276.
24. Prunkl, Carina, and Jess Whittlestone. "Beyond Near-and Long-Term: Towards a Clearer Account of Research Priorities in AI Ethics and Society." arXiv preprint arXiv:2001.04335 (2020).
25. Everitt, Tom, Gary Lea, and Marcus Hutter. "AGI safety literature review." arXiv preprint arXiv:1805.01109 (2018).
26. Amodei, Dario, et al. "Concrete problems in AI safety." arXiv preprint arXiv:1606.06565 (2016).
27. Silver, David, et al. "A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play." Science 362.6419 (2018): 1140-1144.
28. Garrabrant, Scott, et al. "Logical induction." arXiv preprint arXiv:1609.03543 (2016).
29. Garfinkel, Ben, and Allan Dafoe. "How does the offense-defense balance scale?." Journal of Strategic Studies 42.6 (2019): 736-763.
30. Intelligence, Artificial. "Rich E., Knight K." (1991).

---

[[1]](#ftnt\_ref1)  We frequently need to distinguish between (i) AI that takes the form of a collection of AI services and (ii) monolithic AI associated with utility functions, instrumental goals, etc. While this distinction might ultimately turn out to be too simplistic, we believe it is nevertheless useful as a first approximation. We use the terms “system of AI services” and “agent-like” AI to point to the corresponding clusters of AI.

[[2]](#ftnt\_ref2) Apart from services, an important role in our society is played by institutions and companies. For the purpose of this document, we view these as “clusters” of services connected in a particular manner.

[[3]](#ftnt\_ref3) Suggestions for a more suitable definition are welcome.

[[4]](#ftnt\_ref4) It is crucial to note that many services have important functions that are distinct from their “official” tasks. For example, if a part of my reason for visiting the coffee shop is to socialize with the waiter, I might stop coming if the waiter is replaced by a robot. As Simler and Hanson point out in [6], these “secondary” functions might be particularly difficult to notice when acknowledging them would put us into a bad light. Indeed, I might be reluctant to admit that I only frequent the coffee shop to flirt with the waiter. Similarly employers might be uncomfortable admitting that one (among many) benefit of having formal education is a hard-to-fake certificate of being willing to perform routine tasks that one considers mostly pointless. Ultimately, whether we approve of these secondary functions or not, we might soon see a large degree of automation. As a result, it seems unwise to ignore these functions or, worse yet, remain blind to them.

[[5]](#ftnt\_ref5) The above definition can be viewed as solving an important general problem of defining “a capability of an algorithm”. Arguably, the definition does not so much solve the problem as transforms it into a different problem to be solved elsewhere (“define a partial ordering on task-solutions in the given domain”). However, in this particular case, this “delegation trick” is extremely useful since most domains do naturally come with some means of comparing solutions.

[[6]](#ftnt\_ref6) Then again, maybe they will not.

[[7]](#ftnt\_ref7) And if the borders aren’t closed...

[[8]](#ftnt\_ref8) In the author’s opinion, it is a useful “mental move” to sometimes consider scenarios that are far-off or unlikely, or even impossible. One of the benefits is that this allows us to identify hidden assumptions and intuitions. For example, my intuitions might say to not expect global chaos ensue as AI technology becomes more capable and wide-spread. At the same time, consider the thought-experiment of immediately turning everybody into a wizard from J.K.Rowling’s books (as impossible as it is given our laws of physics). My intuitions say that in this scenario, the world probably would erupt into chaos. Noting this, I can ask why the two versions of “increased capabilities” are different, and start examining my intuitions about the former to see whether they are justified.

[[9]](#ftnt\_ref9) In this terminology, both the stereotypical images of monolithic AGI and comprehensive AI services (CAIS) of [1] can be viewed as fully-automated systems that encompass most of the tasks currently performed by humans. The distinction is that CAIS consists of many narrow services, while the monolithic AGI consists of a single general-purpose service.

[[10]](#ftnt\_ref10) Technically, long-term planning should fall under governance and system design. We list it separately since it is quite possible to deal with the more immediate concerns of policy-making and yet fail at the long-term challenges (“Let’s have the guys after us deal with the pension reform.”).

[[11]](#ftnt\_ref11) This list isn’t meant to be complete, mutually exclusive, nor final. For example, the item (4) should be further split depending on whether we are more interested in formal proofs or simulation-based arguments, the suitability of a model for (1) will vary depending on the target audience, etc.

[[12]](#ftnt\_ref12) There are certainly exceptions, some of which include game theory, multiagent systems, and cybersecurity.

[[13]](#ftnt\_ref13) In a vague analogy to how cells are made of smaller components, some of which previously existed independently, a world-model could hypothetically join up with a planner, a learning module, and some sensors and actuators, and thus create an agent.

[[14]](#ftnt\_ref14) We only mean to imply that each of the settings is naturally associated with a specific [stance](https://www.google.com/url?q=https://en.wikipedia.org/wiki/Intentional\_stance&sa=D&source=editors&ust=1689706828379284&usg=AOvVaw14STF91CCK7QA\_z6gW9vRf) (intentional vs design), not that any of the settings makes the problem easier to solve than the other.

[[15]](#ftnt\_ref15) The case of native Americans is meant to illustrate the impact that change of environment can have on society. We do not mean to suggest that alcohol was the primary problem here.

[[16]](#ftnt\_ref16) This does not contradict the possibility for those steps eventually becoming bigger or happening in a rapid succession, nor the possibility of the creation of “monolithic” AGI in the middle of this process.

[[17]](#ftnt\_ref17) Hoare, Tony (25 August 2009). "Null References: The Billion Dollar Mistake". InfoQ.com.

[[18]](#ftnt\_ref18) Organizational theory offers many relevant concepts such as authority and delegation, monitoring outputs and audits, and hiring employees and terminating contracts.

[[19]](#ftnt\_ref19) Software engineering deals with many relevant topics such as architecture choice, communication, and bug detection and removal.

[[20]](#ftnt\_ref20) It originally came as a surprise to the author of this text that instead of being a typo, the prefix “mesa-” comes from greek, and broadly has the [opposite meaning to “meta”](https://www.google.com/url?q=https://www.gwiznlp.com/wp-content/uploads/2014/08/Whats-the-opposite-of-meta.pdf&sa=D&source=editors&ust=1689706828376863&usg=AOvVaw3ZyhOapYzg7xeoiKv8tStG).


The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation (2018). Brundage and Avin et al.

[![Logo Google](//ssl.gstatic.com/docs/common/product/docs\_lockup2.png)](//support.google.com/docs/)Żądany plik nie istnieje.

Sprawdź, czy URL jest poprawny i czy plik istnieje.

\*\*Dysk Google – zrobisz wszystko w jednym miejscu\*\*

Aplikacje Dysku Google ułatwiają tworzenie, przechowywanie i udostępnianie różnego rodzaju dokumentów, w tym arkuszy kalkulacyjnych i prezentacji.

Dowiedz się więcej na [drive.google.com/start/apps](https://drive.google.com/start/apps).


![](http://gcrinstitute.org/wp-content/uploads/2021/11/Plant-Protection-Drone-Dji-Uav-Farmland-Agriculture-4204798-1024x609.jpg)
[View the paper “The Ethics of Sustainability for Artificial Intelligence”](https://gcrinstitute.org/papers/060\_sustainability-ai.pdf)

[Access the data used in the paper. ](https://gcrinstitute.org/papers/060\_sustainability-ai-data.xlsx)

AI technology can have significant effects on domains associated with sustainability, such as certain aspects of human society and the natural environment. Sustainability itself is widely regarded as a good thing, including in recent initiatives on AI and sustainability. There is therefore a role for ethical analysis to clarify what is meant by sustainability and the ways in which sustainability in the context of AI might or might not be good. This paper provides a foundational ethical analysis of sustainability for AI, describes the ethical basis of the existing body of work on AI and sustainability, and presents an argument for a specific ethical view on AI and sustainability. The paper is part of the conference [AI for People: Towards Sustainable AI, CAIP’21](https://aiforpeople.org/conference).

As the paper explains, sustainability is not an inherently ethical concept. “Sustainability” simply refers to the ability of something to continue over time; the thing to be sustained can be good, bad, or neutral. Common usage of the term “sustainability” assumes that the thing to be sustained is some combination of social and ecological systems. The term is sometimes also used in other ways, such as to refer to the sustainability of a business or organization, or the sustainability of an AI system. The paper argues that usage of the term “sustainability” should address three ethics questions. First, what should be able to be sustained, and why? Second, for how long should it be able to be sustained? Third, how much effort should be made for sustainability?

The paper further distinguishes between sustainability and optimization. Making something sustainable means giving it the potential to continue existing in at least some minimal form. In contrast, optimizing something means putting it in the best form that it can have. Therefore, sustainability may be considered a basic minimum standard of conduct toward future time periods, whereas optimization may be considered a more substantial goal. In common usage, sustainability is treated as a good thing, but it may be better understood as a not-terrible thing. If human civilization has to focus on sustaining itself rather than on loftier goals like optimization, then it is in a very bad situation.

With this theoretical perspective in place, the paper surveys prior work on AI and sustainability. It examines published sets of AI ethics principles and academic research on AI and sustainability. The paper finds that most work on AI and sustainability focuses on common conceptions of environmental sustainability, although some work has been done on the sustainability of AI systems and other things. Additionally, most work is ultimately oriented toward sustaining human populations, with AI and the environment having value insofar as they support human populations. Finally, most work lacks well-specified the ethical foundations, with no clear answers to the three questions listed above.

The paper then provides its own answers to the three questions. First, it argues for sustaining both humans and nonhumans. Second, it argues for sustainability over long time scales, including the astronomically distant future. Third, it argues for a large amount of effort toward sustainability. It additionally calls for emphasizing optimization over sustainability in cases where the two diverge.

Finally, the paper presents implications for AI. One is that AI should be used to improve long-term sustainability and optimization, such as by reducing global catastrophic risk. Another is that attention should be paid to long-term forms of AI, which could be particularly consequential for long-term sustainability and optimization. These AI topics only partial overlap with what is typically considered within the realm of AI and sustainability, but the paper argues that these topics are a more appropriate focus for work on AI and sustainability.

The paper extends GCRI’s research on AI ethics, especially the papers [Moral consideration of nonhumans in the ethics of artificial intelligence](https://gcrinstitute.org/moral-consideration-of-nonhumans-in-the-ethics-of-artificial-intelligence) and [Reconciliation between factions focused on near-term and long-term artificial intelligence](https://gcrinstitute.org/reconciliation-between-factions-focused-on-near-term-and-long-term-artificial-intelligence). It additionally builds on GCRI’s research on sustainability and environmental risks, especially [Integrating the planetary boundaries and global catastrophic risk paradigms](https://gcrinstitute.org/integrating-the-planetary-boundaries-and-global-catastrophic-risk-paradigms).

This paper has also been [summarized](https://montrealethics.ai/the-ethics-of-sustainability-for-artificial-intelligence/) in the [AI Ethics Brief #85](https://brief.montrealethics.ai/p/queer-china-sustainability-unesco-ai-ethics) of the Montreal AI Ethics Institute and is included in the 2022 [The State of AI Ethics Report](https://montrealethics.ai/volume6/). The paper is also discussed in the MEDIUM article [“Is 2022 the Year that AI Ethics Takes Sustainability Seriously?”](https://josh-gellers.medium.com/is-2022-the-year-that-ai-ethics-takes-sustainability-seriously-8a10953105e9).

Academic citation:
Owe, Andrea and Seth D. Baum, 2021. [The ethics of sustainability for artificial intelligence](https://gcrinstitute.org/papers/060\_sustainability-ai.pdf). In Philipp Wicke, Marta Ziosi, João Miguel Cunha, and Angelo Trotta (Editors), \*Proceedings of the 1st International Conference on AI for People: Towards Sustainable AI (CAIP 2021),\*Bologna, pages 1-17, [DOI 10.4108/eai.20-11-2021.2314105](http://dx.doi.org/10.4108/eai.20-11-2021.2314105).

\*Image credit:\* [\*Max Pixel\*](https://www.maxpixel.net/Plant-Protection-Drone-Dji-Uav-Farmland-Agriculture-4204798)

 Tagged with [artificial intelligence](https://gcrinstitute.org/tag/artificial-intelligence/), [ethics](https://gcrinstitute.org/tag/ethics/)