Chapter 4
How Change Agencies Can Affect Our
Path Towards a Singularity
Ping Zheng and Mohammed-Asif Akhmad
4.1 Introduction
This chapter adopts a change agency perspective to help us understand how tech-
nological change may mutually interact with human agency and socioeconomicforces. More than in any previous time, controversial speculations about the sin-gularity, i.e. an arti ﬁcial intelligence becoming self-aware and self-evolving leading
to an unprecedented rapid technological change in human civilization (Eden et al.2013 ), have intrigued growing interest and posed hypotheses. These attempt to
deﬁne the human future and determine what it means to be human through the
innovation of technologies (see Chap. 2Risks). Most discussions of the techno-
logical singularity have emphasised a perspective on how the emergence of arti ﬁcial
intelligent machines outperforming human brain capability could lead to plausibleextinction of the human race (Miller 2012 ; Kurzweil 2009 ; Bostrom 2002 ). The
focus of such techno-human future visions is mainly eyed on narrow AI or AGI(Artiﬁcial General Intelligence), a single technological variable leading to an
intelligence explosion. However, pre-singularity is a long-term process in whichmany variables can affect the timescale and sequence of pre-singularity components
and thus in ﬂuence the consequences of post-singularity. It is possible that tech-
nological change could be managed as a rational or planned process with a trans-parent agenda. We would be able to navigate the changing process and eventuallycreate a desirable outcome of the singularity.
P. Zheng ( ✉)
Senior Lecturer in Business and Management, Canterbury Christ Church University,
Canterbury, UK
e-mail: ping.zheng@canterbury.ac.uk
M.-A. Akhmad
Principal Scientist, Applied Intelligence Laboratories, BAE Systems plc, Chelmsford, UKe-mail: optonet@gmail.com
© Springer-Verlag GmbH Germany 2017
V. Callaghan et al. (eds.), The Technological Singularity ,
The Frontiers Collection, DOI 10.1007/978-3-662-54033-6\_487
Technological change is an economic and social phenomenon taking place at
different levels of temporal aggregation in economic and social settings, involvingindividual consumers, businesses, markets, science, technology, formal and infor-mal institutions and culture at wider levels of society (Suarez-Villa 2009 ; Compagni
et al. 2015 ). These variables contain a variety of technological, organisational,
social, national and institutional aspects, which become agents of change thatengage and interact in the technology transformation (see Fig. 1). Technologicalinnovations in robotics, biotech, nanotech and cognitive science, and so on, cancause a paradigm shift that may subsequently affect how AI is eventually used,designed and integrated. The basic concept is summarised in Fig. 1. This chapterattempts to provide an explanation of how change agencies interact within theinstitutional environment in the pre-singularity process. The two primary outcomes
are either a desirable ‘controlled ’or undesirable ‘uncontrolled ’singularity event
arising. We also propose the condition for ‘no singularity ’occurring, which is
explained in Sect. 4(Anti-singularity Postulate).
4.2 Pre-singularity: The Dynamic Process
of Technological Change
4.2.1 Paradigm Shifts
Paradigm Shifts are formed from a series of nudges, especially in the case of
technology research, which is driven by solving present problems whilst building afoundation for future problem solving. When it comes to how we push thedevelopment of technology, how do we create paradigm shifts that bene ﬁts us all?
Our fundamental way of thinking towards technological progress is a key aspect to
determine a path towards a singularity event. For example, if an extra-terrestrialalien philanthropist wanted to consider investing in Earth and the potential of thehuman race, what would the “human race plc. ”sales brochure state as its mission
statement?
…..the vehement urge to survive and overcome all obstacles ….…… to grow our knowledge
and experience through ingenious and courageous exploration of our reality …..
This fundamental motivation and philosophy of how our civilisation perpetuates
forward is inexorably linked to how we embrace and utilise paradigm shiftingtechnologies. The explosion of singularity aspects may become less likely if thedevelopments of these technologies are carried out in-line with singularity controland monitoring mechanisms such as regulatory technology review boards (seeChap. 2Risks 3.3.1). Relationships between the use of key technologies (in
speciﬁed time frames) and possible pre-singularity scenarios can be predicted and
considered. Key technology innovations and management of those technologies
that will cause the pre-singularity environment can be carefully monitored and88 P. Zheng and M.-A. Akhmad
explored for possible prevention of an unexpected singularity event from occurring.
Similar to ‘The Observer Effect ’in physics, i.e. the act of observation can change
the phenomena being observed (Heisenberg 1958 ), the momentum of analysis and
thought regarding our future and potential technological singularity is increasing
and is changing how the human race will navigate its path to the singularity and
beyond. The realisation of the emergence of advanced technologies and theircombinations and how they could have effect on a singularity scenario are beingimplemented into technology roadmaps spanning the next 50+ years. Although thisparadigm shift is expected, it is a spectrum of technology milestones, respectivedecisions from change agents and dynamic market conditions that will de ﬁne the
events leading to the singularity point being recognised and accepted.
4.2.2 Technological Change and Innovation Adoption
Studies of the social dynamics of technology development, including perspectiveson the social construction of technology, the actor-network, human agency, havelooked into the interactive effect of technological change with the socioeconomicforces (Boudreau and Robey 2005 ; Orlikowski 2000 ). Leonardi ( 2009 ) emphasises
not only that social groups change a technology but also how technological and
organisational changes mutually constitute one another in contexts where tech-nology development and use occurs within a single organisation or across multipleorganisations that are highly interdependent. The impact of technology can vary ondifferent users depending on their attitude and approach of adoption. The appre-ciation of technology features and capability to integrate into the organisation ’s
structure will lead to the recreation of ‘sociomaterial ’structure where it accom-
modates the use and full application of new technology. For example, Compagni
et al’study of the diffusion of robotic surgery in Italy revealed that there were
non-adopters (hospitals) deliberately refusing to adopt the innovation, with theirjustiﬁcation pertaining to the uncertainty surrounding the technical bene ﬁts and
economic sustainability of medical robotic systems (Compagni et al. 2015 :251).
Waarts et al. ( 2002) point out that the most important stimulating factors for tech-
nology adopters are a combination of internal drives, such as the ﬁrm’s attitude towards
innovation and the strategic im portance of innovation for the ﬁrm with external forces
like the industry competitiveness. Early adopters in particular run a higher risk by
adopting an innovative product/system (e.g. Enterprise Resource Planning) that is
unproven in the long term. The ability to bear risk is a function of suf ﬁcient resources,
the size and the ﬁnancial strength of the company (Webster 1969;W a a r t se ta l . 2002).
The non-adopters or late adopters are more concerned with practical aspects of rapidchanges in technology, costs and requisite knowledge (Robertson 1986). When busi-
nesses consider the costs and risks involved, the most common strategy is to postponethe adoption of technological innovation. Thus, new technology suppliers may possiblyincrease the likelihood of adoption by offer i n gag r a d u a lp a t ho fi m p l e m e n t a t i o na sa n4 How Change Agencies Can Affect Our Path … 89
alternative to the ‘big-bang ’approach. Apart from this, there are multiple factors
affecting the diffusion and adoption of technological innovations, such as; legal regu-lation, institutional arrangements, gov ernment policies, market competition and
organisational resources.
4.2.3 The Change Agency Perspective
The role of agency in the diversi ﬁcation of socioeconomic change emphasises the
development of institutional environments in terms of socioeconomic frameworksof rules, routines, conventions and normative pressures (Scott 1987 ; Oliver 1996 ;
Hoffman 1999 ). The concept of “agency ”originates from the ﬁeld of sociology,
which has mainly focused on the interaction between human/organisational agen-cies and institutional orders in shaping social orders (see Lukes 1973 ; Coleman
1986 ; Emirbayer and Mische 1998 ; Battilana and Casciaro 2012 ). Actors are able to
intentionally create new institutions and also effect societal-level changes (Garudet al. 2002 ; Seo and Creed 2002 ; Battilana 2006 ). The interplay of key actors within
institutional orders is central to a better understanding of the forces of economic andsocial change (Lamberg and Pajunen 2010 ; Hodgson 2007 ).
To some extent, national governments can play a critical role of socioeconomic
change. Take China for example; its transition to a market economy is based on theliberalisation of market relations by a major agency; the Communist Party. This haschosen to cultivate market socialism through developing a number of key agenciesfor social and economic transformation. It has focused upon developing technologyinnovation by sponsoring three types of key agency; restructured state-ownedenterprises, the encouragement of foreign-owned joint ventures and the develop-ment of privately-owned entrepreneurial enterprises (Zheng and Scase 2012 ). The
capital accumulation process as shaped by the actions and behaviour of economic
agents —that is, institutions and individuals —is conditioned by a variety of
socio-economic and political settings and constraints which, themselves, are theoutcome of historical legacies, cultures and con ﬂicts (North 1990 ; Redding and
Witt 2009 ). Seo and Creed ( 2002 ) argue that those who arise as change agents are
likely to be the groups that are dissatis ﬁed with how their interests are accommo-
dated within institutions and, as such, become potential forces for initiating struc-tural change. Garud et al. ( 2002 ,2007 ) also suggest that organisational ﬁelds are
characterised by dominant and peripheral actors, who can act as institutional change
agents because they collectively experience tensions arising from contradictions ina given socio-historical context. This body of literature focuses almost entirely onentrepreneurs as the main agent of change agents that are capable of mobilisingresources to create new institutions or transforming existing ones. Other dominanteconomic agents in different organisational ﬁelds also impact upon structural
changes and interplay within this evolving process of societal transformation. Theactors of different structural environments interactively respond to the emergent90 P. Zheng and M.-A. Akhmad
problems posed by changing historical, political, social and economic situations.
These orientations and actions towards emergent situations —whether they are
‘deliberate ’or‘improvised ’(Baker et al. 2003 )—are an important force of struc-
tural change. In applying the interactive role of different actors and agencies, we
may gain insights into the dynamics of the ‘socio-technological ’change process
that is interactive with and mutually conditioned by the changing forces. Wehighlight the discursive process that agents of change actively engage in the dif-fusion of innovations and social transformation.
4.2.3.1 Business Organisations as Agents of Change in Innovation
Practice
Multinational and international organisations are major employers in the global
market and as a result, their employment policies and practices have a strongbearing on reshaping the pool of human resources and the experience of work forinternational business practice. They are facilitating the reproduction of skills andknowledge required for the creation and implementation of new technologies. Theyare often regarded as a ‘best practice ’benchmark. Furthermore, they are the
dominant force of innovation practices because their capabilities are based upon a
combination of local and global skills and knowledge with R&D technologies and
advanced management practices. They are proactive in responding to marketdemands because of their creative and deliberative attempts to be at the forefront ofmarket threads. Just to name a few; Google, Apple, IBM, Samsung, Huawei aremarket leaders in their ﬁelds of technology innovation. Google ’s robot dog, Apple ’s
intelligent Safari browser, IBM ’s Watson smart computer, Samsung ’s smart TV,
Sony’s dancing robots and Huawei ’s FusionCloud computing solution are good
examples of their leading innovation products. These global giants have powerful
inﬂuence over technological development. Under technology-driven capitalism, the
corporate strategic focus has been shifted from production to research (Suarez-Villa2009 ). Perhaps their key role as agents of change lies in their own pursuits of
maintaining market leadership in innovation and isomorphic pressure on fastresearch outcomes. They are in the race to create new technologies, inventions andinnovations against their rival forces.
4.2.3.2 Social Networks as Agents of Change
The innovation model for knowledge economies is shifting from in-house and
‘closed ’style to ‘open’model that heavily depends on external collaboration and
elusive resources (Chesbrough 2006 ). In the era of the Internet, organisations have
become socially connected and widely embedded in an open platform within thesocioeconomic structure. For example, when Apple ﬁrst introduced iOS, it exer-
cised complete control over preinstalled, native applications to ensure the value4 How Change Agencies Can Affect Our Path … 91
expected from the platform. Reluctantly but inevitably, Apple later embraced and
integrated third party developers ’creative products into its product platform that
offered a sharing community introducing user-installed, unof ﬁcial applications
created by individuals and businesses worldwide, which resulted in the present day
Apple App Store (Yoo et al. 2012 ). Growing areas, such as robotics, nanotech-
nology, bioinformatics and so on, demand creativity more than many otherresources. Their future advancement rides on the quality and capabilities of thesocial networks to reproduce it (Suarez-Villa. 2009 ).
Social media and social networks are taking precedence in product marketing
communications which draw attention to the social construction processes thatsurround the diffusion of technological innovations. Individuals and businesses aregrowing more interdependent on each other through electronically connected
socioeconomic infrastructure in a global context. Such external networks thus
become the new resources providing for creativity and skills that innovation andentrepreneurship requires. For example, crowd-funding networks such as‘Kick-Starter ’are social platforms allowing entrepreneurs or inventors to demon-
strate their new ideas (mainly technology-based product ideas) to the public in anattempt at obtaining individual funds/donations for start up capital. Crowd fundinghas solved the seed capital problem for small businesses and creative individualsthat often struggle to obtain funds from ﬁnancial institutions. Crowd funding social
platforms not only encourages new technology based inventions from individual
actors rather than organisational actors, but also it functions as a market assessmentmechanism, approving good quality proposals with market demand.
Social networks can also act as mediators. It dilutes organisational hierarchical
control over the technological innovations and creates such network mechanisms asconsensual decision-making, easy access, participatory discussion and open debate(Suarez-Villa 2009 ). These measures help sustain a network ’s control apparatus. In
this way, networks are agents of change. The human agency, the sharing of data and
processes with digital tools have altered conventional norms of ownership, roles,
and rules, often triggering new con ﬁgurations of relationships among actors
involved in innovation networks (Yoo et al. 2012 ).
4.2.3.3 The In ﬂuence of Entrepreneurs as Agents of Change
Entrepreneurial ﬁrms have demonstrated considerable initiatives in pursuing and
securing opportunities under institutional constraints. Private ownership is one of
major signi ﬁcance in the development of free market economy, since it not only
represents a force for contributing to the country ’s economic growth but also as a
change agency for restructuring the economy. In a turbulent and changing mar-ketplace, it is not surprising that many such entrepreneurs have emerged as a newclass of plutocrats and business magnates. Their entrepreneurial ingenuity has led totheir pursuing and securing business opportunities. They are innovative,opportunity-driven and risk-taking. Regardless of a lack of resources, they develop92 P. Zheng and M.-A. Akhmad
tactics and networks to gain access to required skills and capital. There is a growing
number of technology driven entrepreneurs and series entrepreneurs who are takingthe lead role in technology innovations. They are extremely adaptive and ﬂexible;
their pursuit for new opportunities and collective actions for achieving their own
interests have created unintended consequences for the technological and socioe-
conomic change. As such, they are key agents of technological change and societaltransformation.
Take Chinese SME, Alibaba for example; it was regarded as the most valuable
IPO in human history, valued at 45 billion USD in the New York Stock market. AsJack Ma, founder of Alibaba addressed in one of his public speeches in the US, hecreated Alibaba with the intention to help Chinese SMEs to access overseas marketswhich he not only in ﬂuenced the way SMEs internationalised and developed but
also had led key inventions in internet communication technologies to help secure
online payment e.g. Zhifubao (Chinese version of Paypal system) and facerecognition software for online payment. Another example of an in ﬂuential entre-
preneur is Elon Musk, a co-founder of and CEO of Tesla Motors. He has pushedforward the technological development of capable electric vehicle technology. Hisvisions for sustainable technologies are in ﬂuencing investment in alternative and
disruptive technology research, which include; the SpaceX (advancing the state ofrocket technology), the Hyperloop transportation system, SolarCity ’s photovoltaic
technology and now his aspiration of helping humanity to colonise Mars (Cop-
pinger 2016 ). In the new era of rapid technological change, ethical and responsible
entrepreneurship is important. After the 2009 ﬁnancial crisis triggered in America
and extended to the UK and EU countries, there had been many calls frompoliticians for ‘responsible capitalism ’in the United Kingdom (Rae 2015 ). Clearly,
the pursuit of sole pro ﬁt-driven goal has led the banking crisis and market failure.
Entrepreneurship under malicious pro ﬁt maximisation has become destructive in
value creation. Entrepreneurs as the important agents of change are expected to
work in socially responsible and environmentally sustainable ways. This need
becomes more crucial in the new tech era where there are increasing involvementsof technology-based entrepreneurial ventures as the driving force for innovations.
4.2.3.4 Nation States as Agents of Change
Often state-owned or state-sponsored organisations are highly policy-led and
characterised as reactive to restructuring demands as they are inclined to conform
with institutional arrangements. Their resource-focused competitive advantage isbuilt upon on a grant-aided policy (Hassard et al. 2010 ). They can enjoy privileged
access to state resources and have favourable policy lobby in foreign markets. Forexample, in China, mergers and acquisitions are taking place between medium tolarge state organisations across sectors, which enable diverse state-owned busi-nesses to agglomerate to form larger industrial clusters. Such a strategy is creatingquasi-monopoly industries that dominate vital sectors of the Chinese economy,4 How Change Agencies Can Affect Our Path … 93
such as, for example, telecom and real estate sectors (Zheng and Scase 2012 ).
Nation states can play a critical role in shaping the technological development instate-owned organisations and institutes. According to OECD ( 2012 ), the country ’
spending on R&D has showed steady increase, US at 2.7 percent, Japan at 3.2
percent and China is about 1.6 percent. It is estimated that China ’s double-digit
growth in R&D spending is expected to match and surpass that of the US by about2023, as R&D growth over the past 15 years has consistently exceeded 10 percent(R&D Magazine 2011 :2). In June 2013, China ’s supercomputer —‘Tianhe-2 ’(de-
veloped by the National University of Defence Technology) —became the world ’s
most powerful computing system, superseding the current US system. This is partof the government ’s effort to make the country ’s high-tech industries more com-
petitive and less dependent on overseas rivals. It is evident that Chinese government
has made signi ﬁcant improvement in intellectual property laws and indigenous
innovation policy. Its continuous restructuring of free market and socioeconomicenvironments yield rapid growth of both private economy and state sectors. Thesepurposive actions and interplay of change agents have also created those super-power companies (e.g. China Petro, China Telecom, Huawei and Haier) who haveleading technological innovations in their ﬁelds and have in ﬂuential power over
domestic international marketplace. In China, owing to direct state control over theactions of state sectors that dominate large segments of the economy, the Chinese
governments can quickly intervene or order them to ramp up investments and
leading to leverage economic growth and market change (Zheng and Scase 2013 ).
This type of interdependence will continue to shape the evolution of technologicalchange process in China.
4.3 Key Drivers of Technology Research and Their
Impact
Although there are a plethora of potential technologies that could contribute to a
singularity event, there are key future technologies which are predicted to make adirect impact to a path towards the singularity. These include the exact science ofhuman brain functions, quantum computing, advanced global communicationnetworks, arti ﬁcial intelligence programming, nanotechnology, new methods for
energy generation and speci ﬁc defence technologies. The quest to fully understand
and replicate human brain functions is on going. Incremental approaches to
understand and simulate brain functionality have been based around analysing
individual synapses and neurons and manufacturing equivalent arti ﬁcial mecha-
nisms to simulate a very small portion of the potential parallel computing capa-bility. This method of development will allow this technology to scale up further,which will lead to larger simulated neural networks to be formed, paving the way toa complete simulation of human brain processing. One such current project looking94 P. Zheng and M.-A. Akhmad
at exactly this is the Human Brain Project, and international collaborative pro-
gramme funded by the European Commission (The European Commission 2015 ).
From a general research and development point of view the complete under-
standing of the human brain is an important component underpinning a singularity
event. Not only does the goal of understanding the human brain stem from tech-
nology development to explore the human condition from a biological perspective,it is an important milestone to reverse engineer for computer science. As the humanbrain is currently considered as the most advanced computer in existence, it is onlylogical to reverse engineer, modify and improve on what has naturally evolved todevelop more advanced computing technology. Although this particular develop-ment path is not the only way toward computers with human capability equivalence(such as traditional electronics design and quantum computing), when it is suc-
cessful it will: (1) be a signi ﬁcant marker event to a singularity when human brain
function is simulated on an arti ﬁcial computing platform and (2) de ﬁnitively prove
that compatibilities exist between arti ﬁcial computing technologies and human
brain functions meaning that augmented human brain technology can be realised(an aspect of transhumanism).
Quantum technology research has been gaining momentum over the last ten
years and is now dubbed as “quantum 2.0 ”, as quantum physics is now being
applied directly to technological innovation. This includes, quantum computing,
quantum based sensors, imaging and cryptography. As future quantum technology
may be viewed as ‘disruptive ’, i.e. it may create new product markets that may
displace similar technology with lesser capability. It has been an area of investmentthat has been increasing. For example, in 2014 the UK Government has announced£270 million funding to assist the transition of UK quantum research into thecommercial sector (Physics World, p.7 Feb 2014 ). Quantum 2.0 technology is
important, as it is likely to be the foundation of technologies that will cause anevolution of a singularity scenario. Quantum computing will provide processing
platforms that are orders of magnitude more ef ﬁciency that current systems,
allowing for computers with higher capacities than the human brain (Kurzweil2009 ). Quantum research and development will also perpetuate the general
miniaturisation of technology and the need to develop systems to cope withextremely larges amounts of information.
Research into arti ﬁcial intelligence, is also a fundamental technology component
for a singularity to occur, especially in the case of a future AI/AGI establishingitself. At present, AI developments are characterised into speci ﬁc areas including,
machine vision, perception, logical decision-making and problem solving, motion
and manipulation. Slowly but surely, the automated systems that we directly andindirectly come across are becoming more powerful, with more control overoutputs.
The rise of ‘Cyber ’technologies is also an important factor to the future singu-
larity. Speci ﬁcally, the rise of cyber defence research and development and the range
of technologies it impacts. Over the last decade there has been a surge of investmentin this area (both in the public and private sector) primarily due to the exponential4 How Change Agencies Can Affect Our Path … 95
growth and usage of the Internet. Of course, cyber defence/security is quite broad
and ranges from an individual protecting their information from being used andmanipulated to overt and covert military based cyber attacks between countries.However, as technology is intertwined with this issue, a computer or mobile device
connected to the Internet for the individual to advanced computing systems con-
trolling critical infrastructure (such as power generation, telecommunications andtransportation) the potential cyber threats to these systems spur research anddevelopment into protecting against such advanced threats. There is increaseddemand for the use of more powerful and intelligent computing platforms to thwartof threats faster. Larger and more ef ﬁcient information management systems are
required to understand cyber threats and assist in decision-making. Advanced,semi-autonomous computer programs designed to simulate, detect and defend
against attacks on critical systems. It is possible that the very motivation to develop
these technologies (in this case to keep the technological advantage, among others)will signi ﬁcantly contribute to a singularity occurring.
One perspective of Cyber defence/security technology providing a good foun-
dation for a singularity type event is with the perpetuation of an AGI. All majorgovernments have some sort of Cyber defence organisation. All of which invest inthe latest computing technology and have extensive budgets to maintain andupgrade their capability to maintain an advantage. This is the same with large
corporations such as IBM, and Apple, although the emphasis would be to protect
their corporate information and the manipulation thereof, rather than to preventnational cyber crimes. This technology comprises of the latest processing power,advanced computer programs, vast data management and storage capabilities, all ofwhich are key for an AGI to perpetuate itself, if one is born in the future. At thevery least the Cyber defence industry is signi ﬁcantly pushing forward the devel-
opment of important technologies for a singularity event to occur.
4.4 The Anti-singularity Postulate
As previously mentioned ‘The Observer Effect ’in quantum physics also applies to
the possibility of an anti-singularity arising. That is a series of events which eitherleads to (a) the expected or random selection of future technologies not convergingto produce a singularity; (b) the technologies to create an singularity exist but are
controlled to ensure a singularity event does not occur; or (c) stagnant development
of relevant technologies due to externalities (i.e. economic crisis, war or naturaldisasters).
The fact that the above paragraph has been written means that it reinforces other
contemplations about the potential threats from a singularity event, speci ﬁcally the
rise of an arti ﬁcial intelligence that is uncontrolled and could negatively impact the
human race on a whim. Therefore, the stated and recorded thoughts regarding asingularity event from the introduction of Asimov ’s laws of robotics in 1942 to the96 P. Zheng and M.-A. Akhmad
present will inherently change the course of current and future relevant technology
development to ensure that it does not happen at all. Or if the environment of thesingularity does arise, it is controlled and monitored to ensure a ‘big bang ’event
does not occur. This also implies that there is a certain amount of human ‘common
sense’embedded into in ﬁnite variables that make up part of the ‘existence proba-
bility of a singularity ’equation.
The potential negative impacts of a singularity has generated (at least for now)
soft guidelines and policies regarding monitoring and development of key tech-nologies to an extent. Soft arti ﬁcial intelligence is being continually developed, to
improve the autonomy of smart devices and systems and to generally improve thereliability of intelligent hardware and software that the human race is dependentupon to preserve our way of life. One example of this is Apple ’s Siri and Micro-
soft’s Cortana voice activated personal assistants on smart mobile devices. They are
certainly intelligent enough to understand what the user is saying and to respondwith the appropriate action accurately as well as having the ability to continuallylearn from past action to improve its own service to the end user. However, thereare safeguards in place to switch off this intelligent system for people who do notwant their privacy to affected. i.e. the collation of the end user ’s personal behaviour
and actions to improve the service of these quasi AI interfaces. Privacy guidelinesand policies relating to soft AI interfaces may ensure that there is always a ‘hard
on/off ’switch in-built into these systems, which ultimately mean that they can be
reset, controlled, monitored and less susceptible to contribute to a singularity event(see Chap. 2Risks, 4.1.3).
Another technology area which may directly contribute to an anti-singularity are
Cyber defence technologies. For most large corporations and governments, cyberdefence mechanisms will be in place to prevent things such as hacking and com-puter virus infection. Due to the fundamental nature these protection measures, itmay limit the effects of AI trying to establish itself in a potential singularity event
from trying to utilise networked computing resources. One example of this is the
extensive use of software and hardware encryption. If we assume that an AIbecomes conscious and decides to replicate and control signi ﬁcant computer
resources, it will still have to learn how to overcome and bypass many differenttypes of encryption, providing enough time for automated defence systems and tobe made aware of its breach attempts and to affect an appropriate response. That initself poses and interesting point; how can a potential hard AI singularity threat (inits initial stages) attempting to take control possibly win against a plethora of quasi
AI cyber defence programs? All of these programs are speci ﬁcally designed and
honed to stop unwanted breaches of its systems. The likelihood is that it would failin the initial stages of trying to establish itself and quickly highlight itself to being athreat to be dealt with.
There is also the possibility of stagnant development of key technologies causing
a singularity event to occur. This could be because of a natural disaster (earthquake, tsunami etc.) meaning the technology development shifts to remedy andprevent damage from further disasters or man-made disasters such as wars. Of4 How Change Agencies Can Affect Our Path … 97
course there is signi ﬁcant evidence showing that leaps in technology were apparent
during wartime, such as the atomic bomb in World War Two. However, since warcauses enemies to constantly counter each other ’s developments (technological or
otherwise) whilst battling for control of resources. This also enforces the argument
that an optimal environment for a destructive singularity may not exist in this
context.
4.5 Conclusions
Institutions re ﬂect the order and conventions of a society that is often de ﬁned by
law and regulations (Dobbin 2004 ). Thus, governments as agencies represent
institutions that prescribes certain practices while circumscribing others. Industrialorganisations play collective actions to set up norms and standards of practice tostabilise their position in the marketplace. Entrepreneurs and powerful individualsoften play active role within the existing institutions with deliberate or improvisedattempts to promote new institutional arrangements that meet their needs andinterests. The institutional change process is dynamic and interactive which drivesthe evolutional advancement of society. Technological and cultural development
will inevitably trigger institutional changes, as incompatibilities between agencies
and institutional structures arise. Incompatibilities and threats arising from currentcontrol and regulatory systems will encourage the creation of adapted and newinstitutions that can co-evolve with new technological norms and demands.
We argue that change agency theory can incorporate a variety of technological
developments, where new innovation and their related risks provide not a barrierbut an opportunity for new institutional environments to occur. Further explorationof the interactive effect between change agents and institutional and social contexts
is required to understand how AI and human agency relationships could be insti-
tutionalised and legitimated to provide a positive human/AGI coexistence. To doso, it would be necessary for governments to develop ‘AI Institutions ’where a
systematic arrangement of institutions can be established to regularise and structurethe co-existence of AGI and human race. AI Institutions may include the followingaspects:
(1) Policing mechanisms to use AI to regulate AGI.
(2) Economic policies focusing on the role of AGI in general economic
development.
(3) Financial policies targeted on the role and function of AGI in the capital
market.
(4) Social networks mechanisms to monitor the future role, societal impact and
moral hazards of AGI applications.
(5)‘Think Tank ’organisations such as the Oxford University Future of Humanity
Institute continually identifying and understanding the evolving cultural, moraland cognitive norms of AGI in society ( http://www.fhi.ox.ac.uk/ ).98 P. Zheng and M.-A. Akhmad
(6) Global collaboration between governments to build transparency in AGI
developments and an effective worldwide AI monitoring scheme. This may beaided by the general increase in globalisation.
It is important to recognise that, in the revolutionary process leading to a
potential singularity, the change agencies such as individuals, organisations andgovernments play key roles in determining not only how innovations are carried outand applied but also what institutional change may occur to regularise the tech-nological development. The co-evolution of the human race and technology isgenerally an iterative process, providing the opportunity to predict implementstructural and institutional control mechanisms to minimise the risk of an unwantedsingularity event. Possible future prevention measures may also incorporate counter
productive change agencies such as non-cooperative governments and organisa-
tions that may try to circumvent AGI development regulation.
We need to develop our awareness and measures against potential dangers and
be proactive to make prevention mechanisms. We need to have con ﬁdence and faith
in mankind and believe that AI will continue to be assistance to humans.
References
Baker, T.; Miner, A. S. and Eesley, D. T. (2003), ‘Improvising Firms: Briolage, Account Giving
and Improvisational Competencies in the Founding Process ’,Research Policy , 32, 255 –76.
Battilana, J. (2006), ‘Agency and Institutions: The Enabling Role of Individuals ’Social Position ’,
Organisation , 13 (5), 653 –76.
Battilana, J. and Casciaro, T. (2012), ‘Change agents, networks, and institutions: a contingency
theory of organisational change ’,Academy of Management Journal , 55 (2), 381 –398.
Bostrom, N. (2002), ‘Existential risks: analysing human extinction scenarios and related harzards ’,
Journal of Evolution and Technology ,9 .
Boudreau, M. C. and Robey, D. (2005), ‘Enacting integrated information technology: a human
agency perspective ’,Organization Science , 16 (1), 3 –18.
Chesbrough H. (2006), Open Business Models: How to Thrive in the New Innovation Landscape .
Boston, MA: Harvard Business School Press.
Coleman, J. S. (1986), ‘Social theory, social research, and a theory of action ’,American Journal of
Sociology , 6, 1309 –35.
Coppinger, R. (2016), ‘Elon Musk outlines Mars colony vision ’, Science & Environment, 27th
September 2016, BBC News .(http://www.bbc.co.uk/news/science-environment-37486372
accessed on 29th September 2016)
Compagni, A., Mele, V. and Ravasi, D. (2015), ‘How early implementations in ﬂuence later
adoptions of innovation: social positioning and skill reproduction in the diffusion of robotic
surgery ’,Academy of Management Journal , 58 (1), 242 –278.
Dobbin, F. (2004), The New Economic Sociology. Princeton, New York: Princeton University
Press.
Eden, A. H., Moor, J. H., Soraker, J. H., Steinhart, E. (2013), ‘Singularity Hypotheses: A Scienti ﬁc
and Philosophical Assessment ’, Berlin: Springer.
Emirbayer, M. and Mische, A. (1998), ‘What is Agency? ’,American Journal of Sociology , 103
(4), 962 –1023.4 How Change Agencies Can Affect Our Path … 99
Garud, R.; Hardy, C. and Maguire, S. (2007), ‘Institutional entrepreneurship as embedded agency:
an introduction to the special issue ’,Organisation Studies , 28, 957 –969.
Garud, R.; Jain, S.; and Kumaraswamy, A. (2002), ‘Institutional entrepreneurship in the
sponsorship of common technological standards: the Case of sun Microsystems and Java ’,
Academy of Management Journal, 45 (1), 196 –214.
Hassard, J., Morris, J., Sheehan, J. and Xiao, J. (2010), ‘China ’s state-owned enterprises: economic
reform and organisational restructuring ’,Journal of Organisational Change Management, 23
(5), 500 –516.
Heisenberg, W. (1958), Physics and Philosophy: the revolution in modern science , London:
Unwin University Books.
Hodgson, G. M. (2007), ‘Institutions and individuals: interaction and evolution ’,Organisation
Studies, 28 (1), 95 –116.
Hoffman, A. J. (1999), ‘Institutional evolution and change: environmentalism and the US chemical
industry ’,Academy of Management Journal , 42, 351 –371.
Kurzweil, R. (2009), The singularity is near ,2nded. London: Gerald Duckworth.
Lamberg, J. A. and Pajunen, K. (2010), ‘Agency, institutional change, and continuity: The case of
the Finish civil war ’,Journal of Management Studies , 47 (5), 815 –36.
Leonardi, P. M. (2009), ‘Organising technology: toward a theory of sociomaterial imbrication ’,
Academy of Management Annual Meeting Proceedings .
Lukes, S. (1973), Individualism , Oxford: Basil Blackwell.
Miller, J. D. (2012), ‘Some economic incentives facing a business that might bring about a
technological singularity ’, chapter 8 in “Singularity Hypotheses, The Frontiers Collection ”
(eds) by Eden et al., Berlin: Springer.
North, D. C. (1990), Institutions, institutional change and economic performance , Cambridge:
Cambridge University Press.
OECD (Organisation for Economic Co-operation and Development) (2012), ‘Active with the
People ’s Republic of China ’, pp. 1 –61.
Oliver, C. (1996), ‘The institutional embeddedness of economic activity ’,Advances in Strategic
Management , 13, 163 –186.
Orlikowski, W. J. (2000), ‘Using technology and constituting structures: a practice lens for
studying technology in organisations ’,Organisation Science , 11 (4), 404 –428.
Physic World (2014), February Issue, p 2.R&D Magazine (2011), ‘China ’s R&D Momentum: 2012 Global R&D Funding Forecasting ’,
R&D Magazine , 53 (7), 60 –61.
Rae, D. (2015), Opportunity-centred Entrepreneurship ,2
nded. London: Palgrave.
Redding, G. and Witt, M. A. (2009), ‘China ’s business system and its future trajectory ’,Asia
Paciﬁc Journal of Management , September, 26 (3), 381 –399.
Robertson, T. S. (1986), ‘Competitive effects on technology diffusion ’,Journal of Marketing ,5 0
(1), 1 –12.
Scott, W. R. (1987), ‘The adolesence of institutional theory ’,Administrative Science Quarterly ,3 2
(4), 493.
Seo, M-G. and Creed, W. E. D. (2002), ‘Institutional Contradictions, Praxis, and Institutional
Change: A Dialectical Perspective ’,Academy of Management Review , 27 (2), 222 –247.
Suarez-Villa, L. (2009), Technocapitalism , Philadephia: Temple University Press.
The European Commission http://www.humanbrainproject.eu (accessed on 1stMarch 2015).
Waarts, E., Yvonne, M. v. E. and Hillegersberg, J. v. (2002), ‘The dynamics of factors affecting the
adoption of innovations ’,The Journal of Product Innovation Management , 19, 412 –423.
Webster, F. E. (1969), ‘New Product adoption in industrial markets: a framework for analysis ’,
Journal of Marketing , 33 (1), 35 –39.
Yoo, Y., Boland Jr, R. J., Lyytinen, K. and Majchrzak, A. (2012), ‘Organising for innovation in
the digitalized world ’,Organisation Science , 23 (5), 1398 –1408.100 P. Zheng and M.-A. Akhmad
Zheng, P. and Scase, R. (2012), ‘The restructuring of market socialism: the contribution of
‘Agency ’theoretical perspective ’,Thunderbird International Business Review , 55 (1),
103–114.
Zheng, P. and Scase, R. (2013), Emerging business ventures under market socialism:
entrepreneurship in China , Routledge Studies in International Business and The World
Economy, London: Routledge.4 How Change Agencies Can Affect Our Path … 101


Diminishing Returns and Recursive Self Improving Artificial Intelligence

Chapter 7
Diminishing Returns and Recursive Self
Improving Arti ﬁcial Intelligence
Andrew Majot and Roman Yampolskiy
7.1 Introduction
The concept of a recursively self-improving intelligence has been around for quite a
while (Good 1965 ). At its core is the belief that the creation of a human or
superhuman intelligence level is far over the horizon of current science, but a moremodest and simple intelligence could be possible within a reasonable amount oftime. This reasonable timeframe is unfortunately one of the best examples of amoving target that computer science has ever produced. Initial estimates in the1960s were that this kind of intelligence, or for that matter any kind of intelligence,
might be possible by 1980. When this proved not to be the case the estimate was
moved again and again. Some of the current estimates have taken lesson fromprevious attempts at forecasting AI developments and put the estimate at any timebetween 20 and 100 years (Sotala and Yampolskiy 2013 ). The main reasons for this
constantly shifting estimate is our continued ﬂoundering in the creation of any
remotely workable models for cognition. We can describe overall cognitive func-tioning in some detail, but have no real concept behind things as basic as whatconstitutes an arti ﬁcial thought or how to code a machine with common sense.
These basic roadblocks to intelligence are open problems whose solutions we don ’t
even have an inkling of yet.
The idea of a “Seed AI ”capable of recursive self-improvement was created to
get around these bottlenecks. It seems inherently more dif ﬁcult to create a
super-intelligent AI from the start than it is to let a less intelligent one ﬁgure out
A. Majot ( ✉)⋅R. Yampolskiy
Department of Computer Engineering and Computer Science,
University of Louisville, Louisville, USA
e-mail: amajot@gmail.com
R. Yampolskiy
e-mail: roman.yampolskiy@louisville.edu
© Springer-Verlag GmbH Germany 2017
V. Callaghan et al. (eds.), The Technological Singularity ,
The Frontiers Collection, DOI 10.1007/978-3-662-54033-6\_7141
how to improve itself. The human intellect is currently barely able to begin com-
prehending itself, and the thought of it being able to generate something far moreintelligent than itself is a bit far fetched. So instead of solving every single problemrelated to creating super-intelligence why don ’t we have the AI itself do it? That is
the main proposal behind Seed AI. Humans would only need to produce one AI
capable of a modicum of cognition, programmed with the drive to improve itself,and capable of some degree of original thought and imagination. Over time this AIwould examine its own source code, picking up various algorithms it has andanalyzing them to look for weaknesses. Once identi ﬁed, these weaknesses would be
improved upon and the new algorithm tested, with the AI modeling itself and itsnew potential. If the algorithm works and produces improvement, the AI would editits source code to include the upgraded component. This would be done ad
inﬁnitum until a limit was reached, or the AI chooses to stop improving itself and
instead embark on a new quest. We will go over the details of how this recursiveimprovement would work later in the chapter, and show that some of the methodsused for this process have some possible limitations.
Admittedly, this whole concept is consigned to the semi-distant future, so why
do we need to worry ourselves over it now? There are many people who wouldargue that our time and resources are much better spent on other problems(McDermott 2012 ). Others think that once this type of intelligence is created, it
would have the ability to solve all of the most pressing and critical to survival
problems that we currently face (Chalmers 2010 ; Yudkowsky 2007 ). Such an
intelligence would have a wholly unique view on any problems we face. It wouldprobably not think like a human, have the biases inherent in any human culture, orhave the same motivations as a human. This alone makes an arti ﬁcial intelligence an
amazing asset capable of independent analysis and thought. Even if it doesn ’t solve
all of our problems overnight, such a being should be valued as an ultimate way togain inhuman perspectives on human problems. Who knows what kind of insights a
circuit-based life form can provide to us carbon-based life forms who struggle daily
just to play nicely with one another.
The ideal and most helpful example of an arti ﬁcial intelligence would be as
general as possible in order to mimic the capacities of the human brain. Thereforethis AI falls under the Arti ﬁcial General Intelligence (AGI) category (Yampolskiy
and Fox 2012 ; Goertzel and Pitt 2012 ). One could make any type of intelligence
you wished, chess playing, weather forecasting, disease curing, bomb designing,genocide planning, or election rigging. But the most useful kind of intelligence
would likely be of the AGI category since it would be able to draw upon any variant
of knowledge or deduction to solve problems. This general-purpose thinking AIprobably wouldn ’t have many of the shortcomings that human thinkers have. They
could remain focused on their goal, have no need of sleep, and depending on howthey are set up, they would have access to the bulk of human knowledge if allowedaccess to the internet. An internet hookup would certainly be a risky move however.It may be best to ensure a somewhat guided approach to the AI ’s knowledge
acquisition, as there have been recent examples where unintended consequences
result from bulk information ingestion. One of the more recent, and humorous142 A. Majot and R. Yampolskiy
examples can be found from IBM ’s Watson super computer. In order to get Watson
more comfortable with the nuances in human language, its programmers crawledthe site urbandictionary.com. This had the unintended side effect of making Watsoncurse like a sailor because it could not tell the difference between polite conver-
sation and crude jokes (Lewis 2013 ).
The main fear with a recursively improving AGI is that its strengths could
exploit our own weaknesses. That is to say that they would quickly outpace us inbrainpower while also not having a moral code to inhibit their actions and moti-vations. These actions and motivations might not necessarily have humanity ’s best
interests in mind. We fear that the AGI ’s recursive growth could be exponential and
within months of being turned on it could have control of the planet (Yampolskiy2015b ; Sotala and Yampolskiy 2013 ; Omohundro 2008 ; Turney 1991 ; Goertzel and
Pitt2012 ; Armstrong et al. 2012 ). Omohundro speci ﬁcally details what could drive
an AI towards self-improvement and why it would want to do so. But what if we arelooking for solutions to a problem that wouldn ’t actually exist? Is it really possible
for an AI to improve upon itself so drastically that our intelligence would be to itlike ants are to us?
To summarize: we show that this scenario is not likely, at least under current
mathematical assumptions coupled with a dash of common sense.
Yes, the AI could improve upon itself, creating novel algorithms and raising its
Machine Intelligence Quotient (MIQ) quite a bit, but it would still be bound byinherent limits of algorithm complexity.
The law of diminishing marginal returns would probably take effect, causing
greater and greater computational resource usage for smaller and smaller per-
formance gains.
Any AI with a goal will need to work towards this goal, limiting the time
available for self-improvement since after a certain point spending time onimprovement would become prohibitive to completing its goal.
The logistical chain for creating computers from initial mineral mining and
reﬁning to design and construction naturally allows for opportunities to limit an
AI bent on runaway hardware improvement.
7.2 Self-improvement
The concept of recursive self-improvement was ﬁrst mentioned in 1965 by John
Irving Good. He postulated that as soon as a reasonably intelligent machine wascreated, it would go about improving upon itself. It would then inhabit this newimprovement, and use the increase in ef ﬁciency and ability to create an even better
version of itself. This would continually take place until the machine ’s intelligence
would be completely inconceivable to us lowly humans. The machine would beable to solve nearly any current problem bothering mankind, and so would be the
“last invention that man need ever make ”(Good 1965 ). What might be the path for7 Diminishing Returns and Recursive Self Improving … 143
this AI to improve upon itself? In order to understand how an AI like this may be
developed it might be a good idea to take a look at some potential methods for itsinitial creation, and how it could go about improving itself.
7.2.1 Evolutionary Algorithms
One example of self-improving algorithms is found in evolutionary algorithms. Inthese algorithms the root source code stays the same, but any parameters the code isgiven can be changed, thus improving performance. These algorithms mimic thenatural process of evolution within code by generating a population of algorithms or
problem solutions, mutating and “breeding ”them, and only allowing the ﬁttest
solutions to pass on to the next generation. This idea has been around for a while,and has successfully solved many dif ﬁcult problems that may have taken much
longer to solve otherwise (Yampolskiy 2015a ; Eiben and Smith 2003 ).
The components of most evolutionary algorithms are as follows:
Creation of Initial Population of algorithms or solutions
Combinatorial/breeding function
Mutation function
Selection (Fitness Test)
While certainly not the only means of producing new, better iterations of
intelligence, an evolutionary implementation would seem to make the most sense
for an AI to better itself. The AI would initially create several candidateimprovement algorithms and combine/mutate them according to need. Piecingtogether what we have learned so far, the determination of the MIQ score for agiven self-improving AI would probably be the ﬁtness test in this evolutionary
setup. But in order to fully test the MIQ score for an AI, it would need to be able tosimulate the various algorithms in its candidate population. This would requireextensive resources to be able to run a full or even a limited simulation and, unless
the AI has a plethora of resources, would need to be carefully scheduled with
existing goals and duties so that they are not adversely affected. Once the simulationis loaded a MIQ test can take place, and the rendered score could be used for theevolutionary selection and ﬁtness test. Only the top results would be allowed to
move to the next generation, where they would be cross-bred and further mutatedfor evolutionary simulation. Perhaps if the simulations provided guaranteedimprovement the new algorithm(s) would be loaded as the default cognitivealgorithms for the AI. This would allow the next generation of candidates to be
created faster, as the AI would run smarter and/or more ef ﬁciently. Another key part
of this process would be the mutation algorithm. If the AI could guide mutationaway from obviously dead ends and prune the possible evolutionary tree it wouldspeed up generational improvement. Theoretically, over time this whole processwould create intelligences much more capable than the original.144 A. Majot and R. Yampolskiy
7.2.2 Learning Algorithms
Other self-improving algorithms could be powered by a learning based algorithm.
There has already been some research for self-improving and self-optimizing
algorithms using this approach (Ailon et al. 2011 ; Lin 1992 ). In basic machine
learning algorithms the agent is given a base learning algorithm and a number oftraining data sets. Along with the data sets, expected processed results are alsoprovided so that the agent can learn to associate certain types of input with certainoutputs. These training data sets allow the agent to ﬁne-tune itself to a point that it
can handle new data sets and reliably provide accurate results. The underlyingalgorithms behind this learning process can be arti ﬁcial neural nets, association rule
learning, inductive logic programming, support vector machines, bayesian net-
works, representation learning, and several other methods (Mitchell 1997 ). Each of
these methods has their pros and cons, and would need to be weighed accordingly.Ideally the best methods would be unsupervised learning since handholding an AImay be slow going for initial improvement.
It would seem reasonable for an AI to have the ability to improve by learning for
each of its algorithms responsible for cognition. If each function has the ability tolearn and improve its own performance, then all of their combined improvementswould theoretically increase the MIQ of the AI. But that may just increase per-
formance, it might not provide the imaginative leaps needed for sustained
improvement over time. To compensate for this a function separate from the baseAI intellect may be called for. This function could analyze cognitive algorithms,their performance, and their effectiveness and attempt to generate new versions.These new versions could then be simulated and the results benchmarked, with thefunction learning from its mistakes and successes to create better and better versionsof cognitive algorithms. Eventually this approach may lead to the leaps andbreakthroughs needed to avoid premature, stagnant dead ends.
7.3 Limits of Recursively Improving Intelligent
Algorithms
Most of the fears behind constantly improving AIs revolve around the assumption
that they will get super intelligent rather quickly. This also assumes that thealgorithms employed will be so complex and daunting that humans would not be
able to understand their structure or purpose, creating a sort of black box AI. This
takes for granted a couple of important things in order for this scenario to beplausible. First, humanity is not the ultimate form of intelligence, and there existspossible intelligences that are better than ours. Here the authors do not disagree withthis notion, every time you turn on the television and watch the news there seemsample evidence that human intelligence can be improved upon. The secondassumption is that intelligence can be in ﬁnitely improved upon, or improved upon7 Diminishing Returns and Recursive Self Improving … 145
in such a way as to have a runaway improvement curve that humanity would be
hard pressed to follow. Here we do have an issue with the assumption, which wewill detail along with our reasoning. First though we need to think about what kindsof improvements an AI can look for to boost its MIQ. Broadly speaking these
improvements can fall under two categories, software and hardware.
7.3.1 Software Improvements
According to a number of previously referenced de ﬁnitions, intelligence can be
described and measured as several discrete components acting together to form an
intelligent being. Examples include components that handle sensory input like
visual or auditory cues, ones that can perform counting or mathematical functions,and others that can make decisions based on input, previous experiences andavailable data. In nature these separate functions can be described in the variousregions of the human and animal brain. Taken individually each of these regionscan perform speci ﬁc functions that contribute to overall intellect. Taken together
they create something that is much larger and more functional than just the simplesum of its parts. The regions and discrete functions within the brain interact in
complex and currently not well understood ways. This is evidenced by many of the
most successful intelligence tests that query the individual test subject in a varietyof separate categories. Spatial and verbal reasoning, logical deduction, and math-ematical prowess are all measured in standard IQ tests. Each of these has a region ofthe brain associated with it (Jung and Haier 2007 ). It would make sense for an AI to
behave in the same manner, with discrete algorithms for individual functions that,when combined, form intelligence much greater than the simple sum of its parts.
Since the only current examples of a working intelligence to be found are
ourselves, and to a lesser extent our animal relatives, it makes sense to start the
quest for AI by mimicking the intellectual structures found in intelligent creatures.Some have proposed doing just that (Yudkowsky 2007 ). Our seed AI would be
loaded with a full complement of individual algorithms whose full source code isavailable to itself and it would then be tasked with improving itself. Our initialattempts at creating this base level AI would tie these basic algorithms together intoa rough semblance of cooperation. This would be just enough intelligence toprovide a functional cohesion capable of limited intellect. Then the AI would churn
through computational cycles modeling improvements for each of its algorithms,
along with improvements on how each of these algorithms would interact. After anindeterminate amount of time it would come up with a better version of itself andthen install that new software and run using that upgrade, thus improving a sector ofits intelligence by a ﬁxed amount. This would be repeated over and over, until the
AI reaches and surpasses our level of intelligence.
But here we ﬁnd ourselves with a problem. Using this collection of algorithms
approach we would eventually run out of improvements to ﬁnd for each individual
algorithm. Algorithms would be improved to such a point that they would become146 A. Majot and R. Yampolskiy
provably the best at what they do (Legg 2008 ; McDermott 2012 ). From a software
perspective there would only be a couple of options. Hunt for bettercombinations/interactions of algorithms, or create novel, additional algorithms thatwould expand functionality. The combination of both does have some interesting
possibilities with potentially near-in ﬁnite permutations. But what would this actu-
ally look like? The AI would gain talents and abilities, but would it really be anymore intelligent? For example, let us say that it were to formulate and then improveto maximum ef ﬁciency an algorithm capable of modeling net energy gain nuclear
fusion in a Tokomak reactor. This would allow the AI to probe new insights intonuclear fusion, potentially solving a very large energy crisis in the bargain. Thiscould be an incredible accomplishment, and one which has been occupying thelives of many a graduate student for years (Braams and Stott 2002 ). Now we have
to ask ourselves, does this additional algorithm —one that may be executed indi-
vidually on another system to produce the same sort of result —constitute an
increase in “smartness ”or intelligence of the AI? Or would it just constitute an
increase in ability or adding new tricks up its virtual sleeves from a topologicalperspective? Sure the AI can now simulate complex nuclear fusion reactions andTokomak housing stresses, but so could a large, dumb supercomputer. It doesn ’t
necessarily make it “smarter ”than a human. Talented humans who are able to use
abstracted, complex tools are not always smart humans. This is a perfect example of
how dif ﬁcult it will be to measure MIQ within an AI.
Utilizing a large number of discrete algorithms and various structures might not
be the only route to intelligence. Eventually we may actually ﬁnd an algorithm that
composes many aspects of intelligence by wrapping all of the functionality into oneneat package. Perhaps a mutating, learning, general purpose algorithm can be foundthat unfolds into intelligence over time. We are a little skeptical that this is the likelypath to intelligence, but if it does happen then perhaps it would be in a form friendlyto recursive self-improvement. Intuitively this would seem to be a bit more dif ﬁcult
than the collection of algorithms approach since it would be a more all-or-nothing
sort of improvement. For instance, with many different algorithms to choose fromfor improvement in the previously mentioned architecture some results would bemore immediate. If the AI were to improve upon speech synthesis alone it wouldsee bene ﬁts in at least that area. In this single algorithm scenario it might be a tri ﬂe
more dif ﬁcult as many aspects would need to be concurrently modeled, tested, and
improved simultaneously. But we are talking about an AI here, one with a lot oftime on its virtual hands, and a built-in drive to succeed so it could still be possible.
Perhaps the quantum leaps needed to improve upon a single intellectual algo-
rithm could come from the experiences the AI has while it interacts with humans.While the AI tries to ﬁgure out its own path of improvement it can interact with
humans to gain more experiences in the real world. These experiences couldtranslate into lessons applicable to its own improvement, kindling the AI ’s imag-
ination to produce novel leaps in its intellect. This learning process would give usthe chance to mold our creation and better guide it to its full potential. Conversely,this process would allow us to handicap our AI and tightly control its growth. If we
want to be careful about what we show the AI then we simply limit what external7 Diminishing Returns and Recursive Self Improving … 147
stimuli it has access to. This would be similar to limiting the curriculum of a child
growing up in school. If you are worried about a particular child growing up todesign the next masterpiece of military weaponry capable of leveling cities, you canprevent this by never teaching that child the concepts of math or the written word.
The moral legitimacy of that action may lie in the grayest of areas, but it certainly
would prevent any super-weapons from being built. It would also prevent the childfrom developing many other, possibly more bene ﬁcial technologies.
In any case, AIs powered by multiple algorithms or a single algorithm would
come up against an ef ﬁciency limit. Even approaching the maximum ef ﬁciency of
an algorithm for granting intelligence would be dif ﬁcult. Yes, the AI would recurse
itself into better and better forms, but to get any more intelligence the AI wouldneed to invest greater and greater resources into ﬁnding smaller and smaller
improvements. Not only that, but the tests and simulations involved in proving that
these improvements are valid become more and more complex over time. A similarsetup can be observed with the economic law of diminishing marginal returns. Foran economic example one can see how adding new employees to increase pro-duction can be a good idea at ﬁrst, but eventually the employees will get in each
other’s way. This is likened to the classic “too many cooks in the kitchen ”problem.
It is also true that the monetary bene ﬁt of the increased production brought about by
adding workers will eventually be negated by the cost of employing the workers
themselves. Therefore, it does not make logical sense for an economic entity to
invest too many resources towards production of a good or service because thoseextraneous, additional resources would hinder its business.
Moving back to our AI problem, we could see how the further enhancements an
AI could create would fall heavily under this law of diminishing returns (SeeFig.7.1). If we are looking for an AI whose only purpose is to improve upon itself
Fig. 7.1 Diminishing returns
on intellectual investments148 A. Majot and R. Yampolskiy
then this might be ﬁne. Such an AI would be useless for other things as it would
have no other resources to perform useful work, with 100% of its computationalpower going towards experimentation and algorithmic modeling all the time. Thiswould be the only feasible way to probabilistically get any improvement past a
certain point on its improvement curve. We would then need to ask ourselves what
purpose the AI would have in existing. At best it is a platform to examine complexalgorithmic interactions whose lessons could be utilized in other AIs who havelegitimate purposes. At worst it simply becomes a self-perpetuating, CPUlog-jamming, MIQ high-score achiever with no interest in anything outside of itself,let alone causing humanity intentional harm.
Because of this, any reasonable AI would ﬁnd it in its own natural interests to
limit self-improvement. It would need to gauge at what point does continued
investment in improvements —improvements that may not even realize any useful
results —hinder its primary function. A logical AI would cut back investments and
put the bulk of its resources into performing other work. Its primary function couldbe to produce widgets or solve math problems, and it would derive some satis-faction from the completion of these goals. The drive to improve upon itself wouldlargely be a result of its desire to more intelligently produce widgets or solve hardermath problems. But there will come a time when it is just easier and more math-ematically sound to continue as-is and solve widget or math related problems with
its current MIQ level instead of attempting to invest valuable resources to gain
smaller and smaller increases in ef ﬁciency or productivity. It could attempt to get
around this lack of resources by stealing more resources to run additional com-putations on. However this would just introduce more problems, chief among themﬁnding itself in a larger version of the diminishing returns problem it was trying to
get out of. It would need to coordinate more resources, dispatch more improvementjobs, and spend more intellectual capital on improvement all while trying to combatan angered human populace who doesn ’t appreciate the theft of its computing
power.
7.3.2 Hardware Improvements
While algorithmic improvements are a cheap and easy means of realizing recursive
self-improvement, they are only one half of the equation. The other involves the
iterative improvement of hardware over time. It is here that the power of an AI
could come to see real improvements. As evidenced by Moore ’s law, the computer
industry has already seen a steady performance improvement for the past severaldecades. A self-improving AI capable of chip design and manufacturing couldprovide an amazing overall speed up for chip performance. The lessons learned inits own improvement would provide amazing bene ﬁts in commercial chip manu-
facturing. It would be interesting to see how an AI would evolve the ability to7 Diminishing Returns and Recursive Self Improving … 149
improve its own hardware architecture. VLSI and chip architecture design is a main
driving factor, but there are a lot of other technologies that need to be advanced atthe same time. One of the biggest of these is in materials sciences. As chips havebeen growing increasingly powerful we have done more than simply shrink the
transistor dies and crammed more switches per chip. Advances in materials sciences
have helped to drive this miniaturization and make new chips possible. In fact, thisis an example of a sort of macro recursive self-improvement in the real worldalready. Existing technology is used to design and prototype better, faster tech-nology, which is then manufactured at the upper limits of current technologicalability. This new hardware is used to design better hardware, which is thenimplemented, etc. …etc. For all of history up until this new AI is created, humans
will have been the ones advancing this technology. Humans have opposable
thumbs, intuition, imagination, motivation, and dumb luck which have allowed us
to advance as far as we have. The ﬁrst few recursively improving AI ’s may lack
some or all of these aspects. Speci ﬁcally the ability to physically manipulate objects
in an accurate enough manner to even begin doing materials science research.Having an Oracle style AI that does not have a robotic body would seem, at ﬁrst, to
be the solution to a runaway speed increase in a self-improving AI (Armstrong et al.2012 ). But there have been many proposed issues with the safety behind this
seemingly innocuous type of AI (Sotala and Yampolskiy 2013 ).
The jump from virtual simulations of materials to actual physical modeling and
testing would be quite a leap from a self-improvement latency standpoint. Even ifthe AI were to come up with the most ef ﬁcient assembly line and blazingly fast
prototyping machines, the amount of time it would take to physically construct thenew chip would allow human caretakers, subjects, or rebels to interrupt the processand prevent threatening improvement. Not only would this provide an opportunity,but there is an entire raw materials logistical chain that would be a critical path inthe hardware self-improvement for an AI. An AI without morals or imposed limits
could try to insinuate itself into this supply chain, which is why it would be
important to keep an eye on the chain and AI activities. Just deny the AI raw chipmaking materials and it will be forced to make do with its existing hardware. Theeffort involved and the inevitable consequences would hopefully discourage the AIfrom putting resources into that course of action. This is the same logic that couldbe imposed in the previously mentioned law of diminishing returns. There reaches apoint where it no longer becomes viable to put ever increasing resources intoimprovement, so the AI would make do with what it has and continue to innovate at
a steady, linear rate. This is a logical assumption that any truly intelligent AI would
hopefully be able to deduce on its own.
These assumptions about AI ’s coming to logical conclusions and limiting their
own self-improvement do take a few things into consideration. The number oneassumption is that the humans involved in its creation and maturation are alsological beings who have the best interests of themselves, humanity, and the AI inmind. As history has shown us time and time again, we are not the most infallible ofspecies. If a dedicated group of scientists wants to create a destructive150 A. Majot and R. Yampolskiy
self-replicating and self-improving AI, then there would certainly be ways to go
about it. They could give the AI higher thresholds for time investment so that itmoves farther along the improvement curve before stopping. It could be pro-grammed to spend more time developing improvements for its algorithms or pro-
cesses that would power whatever form of destruction or evil that its creators wish.
And its creators could give it access to raw materials, research facilities, and pre-cision robots necessary for it to recursively improve its hardware to incredibleheights. With enough time and resources the AI would eventually form into anunconventional super weapon with unknown capabilities. Because of this it may notbe the AIs that we need to focus our attention on for prevention of an apocalypticsingularity, but rather ourselves.
7.4 The Takeaway
Because we are so far away from a workable intelligence, we are not sure exactlywhat form it will take. Will there be a collection of algorithms which combinedgenerates intelligence? Or will there be some single intelligence algorithm that canhandle most of the general intelligence an AI needs? We have presented some of the
current thoughts on these matters, but they will be conclusively answered on the
way to the technological singularity. When it comes time to start seriously devel-oping this intelligence there will need to be many concerns addressed so that an AIcan be created in a manner safe for itself and humans. Many fear that if leftunchecked a self-improving AI will rapidly become super-intelligent and leavehumanity in the dust. Its motives and actions would no longer be comprehensible tous, and we would be to it as ants are to us. But self-improvement will likely belimited by the law of diminishing returns enough so that even in the event of an AI
becoming hostile, we would have enough time to act in order to prevent any
catastrophes. Cognitive algorithms will rapidly approach their best possible per-formance and ef ﬁciency, and anything beyond that would require ever increasing
resources for ever diminishing returns.
A logical AI would cease to pour all of its resources into this improvement and
instead go about its normal business performing regular duties. While its intelli-gence would most likely top out at greater than man ’s, there is no reason to believe
that it would be completely incomprehensible. This is especially true when con-
sidering what would be necessary for it to attain better and better hardware. The AI
could take over some machines and design new hardware, but it still needs rawmaterials to create the entire supply chain and industrial base to even begin creatinghardware. This signi ﬁcantly dampens the rapid expansion potential of arti ﬁcial
intelligence and allows humans more time to understand the current iteration of theAI. With understanding comes a decreased probability of mishaps and mistakes,letting humans become more comfortable with the AI ’s capabilities.7 Diminishing Returns and Recursive Self Improving … 151
References
Ailon N, Chazelle B, Clarkson KL, Liu D, Mulzer W, Seshadhri C (2011) Self-Improving
Algorithms. SIAM Journal on Computing 40 (2):350 –375.
Armstrong S, Sandberg A, Bostrom N (2012) Thinking Inside the Box: Controlling and Using an
Oracle AI. Minds & Machines 22 (4):299 –324.
Braams CM, Stott PE (2002) Nuclear Fusion: Half a Century of Magnetic Con ﬁnement Fusion
research. IOP, Bristol; Philadelphia.
Chalmers DJ (2010) The Singularity: A Philosophical Analysis. Journal of Consciousness Studies
17 (9/10).
Eiben AE, Smith JE (2003) Introduction to Evolutionary Computing. Springer, New York.
Goertzel B, Pitt J (2012) Nine Ways to Bias Open-Source AGI Toward Friendliness. Journal of
Evolution & Technology 22 (1):116 –131.
Good IJ (1965) Speculations Concerning the First Ultraintelligent Machine. Advances in
computers 6 (31):88.
Jung RE, Haier RJ (2007) The Parieto-Frontal Integration Theory (P-FIT) of Intelligence:
Converging Neuroimaging Evidence. Behavioral and Brain Sciences 30 (02):135 –154.
Legg S (2008) Machine Super Intelligence.Lewis D (2013) Now I Know: The Revealing Stories Behind the World ’s Most Interesting Facts.
“F + W Media, Inc. ”,
Lin L-J (1992) Self-Improving Reactive Agents Based on Reinforcement Learning, Planning and
Teaching. Machine learning 8 (3 –4):293-321.
McDermott D (2012) Response to ‘The Singularity ’by David Chalmers. Journal of Consciousness
Studies 19 (1/2).
Mitchell TM (1997) Machine Learning. McGraw-Hill, New York.
Omohundro SM (2008) The Basic AI Drives. Frontiers in Arti ﬁcial Intelligence and applications
171:483.
Sotala K, Yampolskiy RV (2013) Responses to Catastrophic AGI Risk: A Survey. Machine
Intelligence Research Institute.
Turney P (1991) Controlling Super-Intelligent Machines. Canadian Arti ﬁcial Intelligence (27).
Yampolskiy R (2015a) Analysis of Types of Self-Improving Software. Paper presented at the
Eighth Conference on Arti ﬁcial General Intelligence (AGI2015), Berlin, Germany, July 22 –25,
2015.
Yampolskiy RV (2015b) On the Limits of Recursively Self-Improving AGI. Paper presented at the
Eighth Conference on Arti ﬁcial General Intelligence (AGI2015), Berlin, Germany, July 22 –25,
2015.
Yampolskiy RV, Fox J (2012) Arti ﬁcial General Intelligence and the Human Mental Model. In:
Singularity Hypotheses. Springer, pp 129 –145.
Yudkowsky E (2007) Levels of Organization in General Intelligence. In: Arti ﬁcial General
Intelligence. Springer, pp 389 –501.152 A. Majot and R. Yampolskiy