Risks and Mitigation Strategies for Oracle AI

Risks and mitigation strategies for
Oracle AI

Abstract : There is no strong reason to believe human level intelligence represents an
upper limit of the capacity of artificial intelligence, should it be realized. This poses
serious safety issues, since a superintelligent system would have great power to direct
the future according to its possibly flawed goals or motivation systems.  Oracle AIs
(OAI) , confined AIs that can only answer questions, are one particular approa ch to
this problem. However even  Oracl es are not particularly safe:  humans are still
vulnerable to traps, social engineering, or simply becoming dependent on the OAI.
But OAIs are still strictly safer than general AIs, and there are many extra layers of
precautions we can add on top of these. This paper looks at some of them and
analyses their strengths and weaknesses.

Keywords : Artificial Intelligence, Superintelligence, Security, Risks, Motivational
control, Capability control

1 Introduction

While most considerations about the mechanisation of labour has focused on AI with
intelligence up to the human level there is no strong reason to believe humans
represent an upper limit of possible intelligence. The human brain has evolved under
various biological constraints (e.g. food availability, bir th canal size, trade -offs with
other organs, the requirement of using biological materials) which do not exist for an
artificial system. Beside different hardware an AI might employ more effective
algorithms that cannot be implemented well in the human cog nitive architecture (e.g.
making use of very large and exact working memory, stacks, mathematical modules
or numerical modelling), or use abilities not feasible to humans, such  as running
multiple instances whose memories and conclusions are eventually mer ged. In
addition, if an AI system possesses sufficient abilities, it would be able to assist in
developing better AI. Since AI development is an expression of human intelligence, at
least some AI might achieve this form of intelligence, and beyond a certai n point
would accelerate the development far beyond the current rate (Chalmers, 2010)
(Kurzweil, 2005)  (Bostrom N. , The Future of Human Evolution, 2004) .

The likeliho od of both superintelligent and human level AI are hotly debated – it isn‟t
even clear if the term „human level intelligence‟ is meaningful for an AI, as its mind
may be completely alien to us. This paper will not take any position on the likelihood
of the se intelligences, but merely assume that they have not been shown to be
impossible, and hence that the worrying policy questions surrounding them are
worthy of study.  Similarly, the paper will not look in detail at the various theoretical
and methodological approaches  to building the AI. These are certainly relevant to
how the AI will develop, and to what methods of control will be used. But it is ver y
hard to predic t, even in the broad est sense, which current or future approaches  would
succeed in constructing a general AI.  Hence the paper will be looking  at broad
problems and methods that apply to many different AI designs, similarly to the
approach  in (Omohundro, 2008) .

Now, s ince intelligence implies the ability to achieve goals, we should expect
superinte lligent systems to be significantly better at achieving their goals than
humans.  This produces a risky power differential. The appearance of superintelligence
appears to pose an existential risk: a possibility that humanity is annihilated or has its
potent ial drastically curtailed indefinitely (Bostrom N. , Existential Risks: Analyzing
Human Extinction Scenarios and Related Hazards, 2001) .

There are several approaches to AI risk. The most common at present is to hope that it
is no problem: either sufficiently advanced intelligences will converge towards
human -compatible behaviour, a solution will be found closer to the time when they
are actually built, or they cannot be built in the first place.  These are not strategies that
can be heavily relied on, obviously. Other approaches, such as balancing superagents
or institutions (Sandberg, 2001)  or “friendly utility functions” (Yudkowsky E. ,
Creating Friendly AI, 2001)  (Yudkowsky E. , Friendly AI 0.9., 2001) , are
underdeveloped.

Another solution that is often proposed is the so -called  Oracle  AI (OAI)1. The idea is
to construct an AI that does not act, but only answers questions. While
superintelligent “genies” that try to achieve the wishes of their owners and sovereign
AI that acts according to their own goals are obviously dangerous, oracles appear
more benign2. While owners could potentially use them in selfish or destructive ways
– and their answers might in themselves be dangerous  (Bostrom N. , 2009)  – they do
not themselves pose a risk. Or do they?

This paper attempts to  analyse the problem of “boxing” a potentially unfriendly
superintelligence. The key question is: how dangerous is an Oracle AI, does boxing
help, and what can we do to reduce the risk?

2 Power of AI

A human -comparable mind instantiated in a computer has great advantages over our
biological brains. For a start, it benefits from every iteration of Moore‟s Law,
becoming faster at an exponential rate  as hardware improves . I.J. Good suggested that
the AI would become able to improve their own design, thus beco ming more
intelligent, leading to further improvements in their design and a recursive
intelligence explosion  (Good, 1965) . Without going that far – it is by no means
obvious that the researcher‟s speed of thought is the dominatin g factor in Moore‟s
Law – being able to reason, say, a thousand times faster than a human would provide
great advantages. Research of all types would become much faster, and social skills
would be boosted considerably by having the time to carefully reason  out the correct

1 Another common term is “AI -in-a-box”.
2 Some proponents of embodiments might argue that non -embodied AIs are i mpossible – but it is
perfectly possible to have an embodied AI limited to a particular “box” and have it only able to
interact with the world outside the box through questions.
response. Similarly, the AI could have access to vast amounts of data, with huge
amounts of information stored on expanding hard drives (which follow their own
Moore‟s Law  (Walter, 2005) ). So an AI would be able  to think through every
response thoroughly, carefully researching all relevant data, without any humans -
noticeable slow -down.

Software can not only be run faster with more data, it can also be copied and
networked. An AI need be only trained in a particu lar skill once; from that point on, it
can be copied as much as required. Similarly, if AIs are subject to human -like
vicissitudes, such as fatigue or drop in motivation, this can be overcome by taking the
entity at the peak of its energy or motivation, an d reloading this every time the AI
starts to weaken. One could use this, for instance, to select AIs at moments when they
are particularly successful at group -interactions. Thus a group of AIs, trained in
different skills and with compatible motivations, c ould be networked together into a
super -committee. Such super -committees are a likely stable organism  (see (Shulman
C. , 2010)  for the full version of this argument)  and could become what people refer
to as “an AI”. Their abilities would likely be a superset of any human committee: a
major bottleneck in human organisations is the ability to share information rapidly
and work well together. Thus no human group could out -think them, and with the
ability to be trained once and copied at will, they could be used in any and all  roles
that humans fill today.

Another factor to consider is that evolution has made humans very skilled at social
abilities and spatial recognition, but less so at scientific and technical abilities. So a
“human comparable AI” that matches our social sk ills (a la Turing test  (Turing,
1950) ) is likely to be much more skilled than us at these scientific and technical tasks;
even if it isn‟t, it‟s likely to have a superior ability to interact with other softwares that
do have the se skills.

Above -human intelligence AIs would be even more impressive, of course, if ever they
existed. If we just constrained the AI into a box, interacting with it only through text
messages, then we would be safe, surely?

3 Introducing the Oracle

That is precisely what an Oracle AI is: a confined AI with no physical manipulators,
with which we can only interact through text messages. Though it feels superficially
safer, this still leaves us vulnerable to the OAIs most dangerous tool: social
engineering,  the OAIs fastest way of getting power for itself.

Even humans can get themselves „unboxed‟ purely through arguing. Eliezer
Yudkowsky  has performed experiments where he took on the role of a boxed AI, and
others took the position of a putative gate -keeper (with a $10 reward if they did not let
him „out of the box‟, meaning granted it full access to the outside world). Despite
being of human intelligence himself, in three out five attempts, he was able to
convince the gatekeepers, purely through argumentatio n, to let him out  (Yudkowsky
E. , 2002) .

A real OAI should be much more skilled at this: we should think of the OAI as
composed of a committee of extremely smart and convincing people, working
together perfectly at high speeds . Or, to shift the intuition pump, if we were the OAI,
with these sorts of intellectual resources, would we not be able to persuade, trick or
trap ourselves out of the box? Most people can think of three or four plausible
methods that could work; we can re st assured the OAI would think of these, and better
ones, and be capable of implementing it to the best of its ability, if it wanted to.

We are all vulnerable to the OAI‟s social engineering. An advanced OAI can literally
offer us the world. Genuine altru ism is no defence against this; it can offer a world
free of poverty, hunger, war, disease or death, as well as all personal trinkets we could
want. It could guess the weaknesses of each individual, and find the right arguments
to convince us that granting  it power or liberty is the moral and profitable thing to do.
It will be able to play on rivalries, institutional weaknesses and ethical dilemmas with
more virtuosity than any human could.

On top of that , the OAI can always trick us with brilliant technol ogy with
undetectable „poison pills‟, or simply become so important to human economy or
human society that we dare not turn it off, whatever it does. In no way could a
superintelligent AI be considered „safe‟ merely because it is boxed.

4 Why box the Oracle  at all?

In view of the preceding, one could wonder why one should study  Oracle s at all. If the
Oracle will inevitably convince or trick its way out of the box  if it wants to , why
bother investing in this type of precaution – shouldn‟t all our efforts be devoted to
preventing general AI from being developed in the first place? It is unlikely that we
can convince the world to back off from AI research, or permit the draconian checks
that would reliably prevent narrow AI research from leading to general arti ficial
intelligence . But trying along these lines is likely to have a tiny but real effect,
whereas worrying more about Oracles seem to have no impact.

This is too extreme a position to take. Precautions are useful, even if they are
imperfect. A well desi gned physical boxing system would prevent the OAI from
escaping other than through social engineering. And there are myriads of clever ways
an OAI could physically escape, if it had access to a manufacturing base; taking these
off the table is a definite a chievement.

Secondly, a safe Oracle is strictly easier that a safe autonomous AI, even one that also
lacked a manufacturing capability. To see this, consider all the different groups in the
world with different agendas and preferences, from libertarians to communists,
democrats, nationalists, Kantians, utilitarians, religious groups, neo -Nazis,
environmental groups, advocates of free love... The list goes on. If any of these
groups were to build an autonomous AI that they felt was “safe”, it would have to  be
imprinted with most of their values, which would differ from the values of the other
groups.  But if they wanted to build a safe OAI, they would all agree on what this
constituted: an AI that stayed confined in a box and answered questions accurately.
To phrase this colloquially, the difference between Hitler and Gand hi is not relevant
for safe OAI (but is relevant for safe AI). It is difficult to estimate how much easier an
OAI is, but if we use approximate Kolmogorov complexity (Solomonoff, 1960)  as a
heuristic, then we can see that the values and desires of these different groups require
a lot of description, making general safe AI many orders of magnitude harder tha n
safe OAI.  We don‟t need to take this argument too litera lly in order to  agree with the
general point: pushing for OAI over general AI can make a huge difference.

But lastly, we don‟t need to just box the OAI, pick the least corruptible gatekeeper we
can find, and hope for the best. Once boxed, there are many supplementary  methods
of control we can add on. Methods of capability and motivational control can all be
layered over the basic AI framework, and  we can add checks and traps to ensure that
the OAI is actually behaving well. Looking at these “methods of control” will b e the
subject of the next section, though a more thorough analysis can be found in our paper
(Armstrong, Sandberg, & Bostom, Thinking Inside the Box: Using and controlling an
Oracle AI, 2011) , which elaborates on all these met hods.

5 Methods of control

The different methods can be loosely grouped into three broad types. The first
category, capability control , deals with those methods that try to contain the OAI
through physically securing it or controlling the information it has access to or can
output. The second category, motivational control , introduces methods  of control the
OAI through its motivation, ensur ing that it has not propensity to cause any harm to
humans. And lastly, there are some methods designed not to prevent the OAI from
causing harm but to catch it when it does; these are in the checks and balances
category.  It should be noted that, depending on the design of the OAI, some methods
may be impossible  (such as rule or utility based motivational control). Since the
design of the OAI is not known, we won ‟t rule out any methods on these grounds.

5.1. Capability  control

Capability co ntrol is the most basic and the most obvious of way of controlling the
OAI. To control a  human, we put them in solitary confinement, denying them
interactions with the outside world. Similarly for an OAI, the first, minimum steps
should be to restrict its means of influencing the world: that is essentially what boxing
means .

The most basic methods are physical capability control. The OAI should be confined
to a box. The „box‟ might be a Faraday cage, filled with inert gases (or with nothing at
all). The OA I should be denied manipulators, and ideally would have no /little  control
over any appa ratus with moving parts – anything to prevent it from building a
manufacturing capability. The cage could be surrounded with high explosives,
triggered by some willing g atekeeper, and the whole set -up could be buried in a
concrete bunker.  There are no real limits to the number of physical security measures
that can be added by wary or paranoid developers, and th eir effects are well
understood – though we are far more likely to succeed with „dumb‟ precautions: an
OAI could conceivably hack a n extremely advanced  security lock  with a blinking
light, but couldn‟t kick down a wall without limbs.

Another possibility is to use epistemic capability control, to reduce the OAI ‟s
epistemic rather than physical abilities. It would be very useful to  throttle the OAI‟s
output, reducing the bandwidth it can use.  If the OAI is reduced to outputting only
three answers – „yes‟, „no‟ or „undetermined‟ – then it will take considerably longer
for it to mount a successful social engineering attack.  The risk, of course, is that the
more opportunities the OAI has in answering, the more opportunity there is of it
passing on a secret message. In the same way giving the OAI more possibilities for
each answer – such as allowing it to output the first three significant figures of a
probability estimate – increases the risk (though this may be compensated by the fact
that we will need to ask less follow -up questions).  And it does not prevent the OAI
from  manipulating us through the contents of its answers, pushing human
development is certain directions.

Other epistemically plausible  measure s, in order of decreasing usefulness (Armstrong,
Sandberg, & Bostom, Thinking Inside the Box: Using and controlling an Oracle AI,
2011)  would be to rese t the OAI after each run, add „fake levels of reality‟ by running
on virtual machines inside other virtual machines, and throttling the OAI‟s input.
Resetting the OAI prevents certain types  of long term plans from emerging while fake
levels of reality are useful if the OAI is a highly technical entity asked highly
technical questions.

Throttling the OAIs input is likely to be less useful: the OAI can deduce huge
amounts of information from any question we ask and any information we give it .
This can be best seen from the Kolmogorov complexity perspective (Solomonoff,
1960) ). Since we cannot compute the Kolmogorov  prior, but any superintelligence
would approximate  it better than we can, we cannot know how much information we
are inadvertently giving it through our inputs (outside of certain closed situations).
The potential for considerable undetected giveaways is immense, and these are errors
we couldn‟t even know  we had committed.

5.2. Checks and  balances

This sub -category of methods of control involve not restricting the physical capability
of the OAI, nor directly interfering with its motivation, but attempting to catch the
early stages of an OAI behaving dangerously (and then, presumably, terminate it).
This also works on the OAI‟s motivation, as the risk of getting caught would push the
OAI along more human -friendly paths. It should always be assumed that a useful OAI
is considerably more intelligent than  its programmers, so any naive „tricks‟ are
unlikely to work.

Nevertheless, we can get some  mileage by testing a random selection of the OAI‟s
answers. If we only gave the OAI questions that we ourselves could solve, given
enough time, and if we exhaustiv ely tested, say, a thousand of the answers it gave,
then we would have a pretty good estimate of its trustworthiness, no matter how many
questions we asked. The most important aspect of this is that the tested questions be
selected at random, after the OAI  has given its answers (a sort of „double blindness‟
for questions). Asking specific test questions is a no -no; if the OAI can deduce that a
certain question will be tested, it may answer differently on it.

Less convincing is the idea of leaving  tempting „honey -pots‟ of great value to the OAI
– extra resources, extra information, the answer it is seeking – in physical or mental
locations that it is forbidden from accessing. And then waiting to see whether it will
violate its instructions to do so.  This app roach is of mixed use: important when testing
out defection and betrayal during the OAI‟s initial development, and nearly
completely useless once the OAI has risen to super -intelligence. It is a „garden of
Eden‟ type scenario: the apple in a tree is only a  worthwhile test while Adam and Eve
are dumb; if they were smarter than God, they wouldn‟t have gone anywhere near
such an obvious trap.

Finally, we might play multiple (but slightly different) OAIs against each other, by
asking them the same question whi le preventing them from communicating with each
other. The OAIs are then confronted with the choice between answering truthfully and
telling a lie; but for the lie to work, all OAIs would have to stumble upon the same
one. If the OAIs are quite different, and especially if their motivations are different
and unknown to each other (a rare case where throttling the OAIs input will be of
use), then they are unlikely to stumble upon the same lie, and may be compelled to
tell the truth.

5.3. Motivational control

The main area for potential improvement in OAI control rest s in motivational control.
Capability control is useful, but we probably already have a good idea of how to do it.
Checks and balances are probably too uncertain to rely on. So the gains from further
study in these areas are probably minor.

If we could solve the  motivational control problem completely, then we would solve
the OAI problem completely: we would have a friendly AI (Yudkowsky E. , 2001a)
(Yudkowsky E. , 2001b). But even partial improv ements can increase our chances of
a safe OAI, especially given the other methods of control we have.

It is essentially a problem of reverse ethics: we know approximatly what we want the
OAI to do (sta y inside the box, answer questions truthfully, don‟t plot against us).
And we seek to find the OAI motivations that will result in this happening. Then, of
course, once we have found the correct motivations, we want to ensure that the OAI
actually follows them.

There is a certain tension between these two goals:  motivations which will most likely
be the safest are not those which are easy to check that the OAI is implementing. The
grading runs from black box motivations (such as reinforcement learning), through
rule-based motivations (such as direct programming) to utility function based
motivations. It is much easier to “teach” an AI to be safe through reinforcement
learning than to spell out explicitly the utility function that would accomplish the
same thing. But conversely, it is much easier to check that an A I is implementing a
utility function than to be sure that the messy incomprehensible  result of
reinforcement learning is actually doing what we hope it is.

5.3.1.  Rule -based motivational control

Rule -based methods are the ones that spring most naturally to mind,  and – provided
the terms are clearly defined – potentially the most secure at keeping the OAI under
control. Defining the terms and the rules so that the OAI does what we want is the
challenge; programmers know many examples of tightly written code that m esses
everything up by doing exactly what it was written to do but not what it was intended
to do. The story stretches back as far as the golem of Prague (Idel, 1990) , and is
familiar to any parent: rules must be refined to deal with every imaginable speci al
case.

The biggest challenge is to define space and time.  If the OAI can have a correct
understanding of space and time, many other restrictions can be easily added to it. But
to achieve this through direct programming, an ontology must be specified, th at the
OAI can understand in the same way that we do. It must be able to understand its own
existence, its physical location, the proper workings of causality, and its own position
within the causal network of the universe – i.e. the consequences of its ac tions.

For instance, to specify what constitutes the OAI, one would presumably define the
physical OAI itself, along with any copy of its code, any copy of its code with slight
modifications, any code with functionally similar behaviour, any assemblage of
subprograms that combine to behave as the OAI would, and so on.

That brief list gives an idea of the problems involved; all the human -understandable
terms of the previous paragraph need to be made rigorous for the OAI for this
approach to work.  There is a similar challenge with time. An OAI that can freely
redefine its own chronology is a security risk. The risk becomes more acute when we
realise how much of human definitions and meanings revolve around a concept of
time, and thus that redefining duration  can redefine meaning. To illustrate with an
admittedly trivial example, if the OAI had the reasonable -sounding moral premise that
“painlessly killing a human being, who is going to die in a micro -second anyway, in
order to gain some other good, is not a c rime”, we would not want it to be able to
redefine millennia as seconds. Nor would we want it to do this when considering the
length of happiness or pain that it grants or imposes on us. More subtle errors will
certainly creep up in more complicated scenar ios; we have had little experience
dealing with beings that can redefine their own clocks, so we have not had to spell out
how many of our assumptions rest on this not happening.

Once time and space are defined, other problems become easier – though not e asy.
Confining the OAI to stay in the box involves defining “the OAI” (presumably
including all exact copies, all similar copies, all instantiation of algorithmic systems
that would be similar to the OAI, and maybe those that would have similar goals) and
“the box” (an easier problem) and “staying in” (a harder problem – waste heat
reaching the outside world from the OAI is ok, but building a laser to blast the outside
is not). We also must preclude the possibility of the OAI discovering a new theory of
physics, realising that „electrons‟ or „wave -functions‟ don‟t actually exist, and thus
that all its restrictions are meaningless.

With space and time well in hand, we can define other rules, requiring the OAI to
answer within a certain time frame, to use onl y specific resources (if we can define
those), and to  respect other constraints we may want to define.

5.3.2.  Black -box motivational control

Human feedback , and such methods as reinforcement  learning (Sutton & Barto,
1998) , can allo w programs to internalise complex concepts without humans having to
fully specify them. Indeed, an OAI trained in this way may spot a pattern we didn‟t
realise was there, and „know some things without us needing to tell it‟. The risks are
that we can‟t be sure how the OAI internalises these concepts: knowing its code
doesn‟t help us if we don‟t understand it.  The risk is especially great when the OAI
transitions out of its learning stage: we cannot be sure that the same concepts mean
the same thing for an O AI that has taken on a new “role”.

But, if the OAI does understand the concepts as we intend it to, we can be sure that
the OAI will obey the spirit and not the letter of these restrictions.  Moreover, it is
possible that we may understand the resulting co de‟s behaviour, even if we could not
code it in ourselves: this would be the ideal outcome from these methods.

Even without that unlikely ideal, it is important that these methods be well
understood , since i t is very possible that a seed AI (Yudkowsky E. , General
Intelligence and Seed AI 2.3)  will be trained mainly by human feedback .

Some of the problems touched upon for rule -based motivations are easier to solve
using black -box methods. The AI can be trained to recognise time  through the tick of
a clock; to internalise spatial restrictions through experience in embodiment (Steels &
Brooks, 1995)  or to come to think of themselves as “one copy” through similar
methods. These methods are somewhat more robust to changes in ontology, not being
so dependent on getting the definitions exactly right.

There still remain  grave risks for complex questions  whose definitions are unclear to
us, and  when the OAI is placed in situations far removed from its training
environment. And this approach also trains the OAI in the skills and usefulness of
social engineering: it will learn that certain specific response s increase the likelihood
of reward, and will extensively model the complex system – us – that is most
responsible for these rewards.

5.4. Utility -based motivational control

The ultimate goal of the friendly AI project  (Yudkowsky E. , 2001a) (Yudkowsky E. ,
2001b)  is to design a utility function for an AI that will provably prevent the AI from
acting in a way detrimental to humanity. Most work in this field has not actually been
directed towards building such a utility function, but towards showing how hard it
would be to do so, and how dangerous the naive ideas on the subject are.

The friendliness problem for an OAI is not much simpler, making this a very hard
approach to use. And if a friendly OAI could be designed, t hen it is most likely that a
friendly AI could also be designed, obviating the need to restrict to an Oracle design
in the first place.  In practice, however, the best that can be realistically hoped for is
that different component pieces of friendly AI the ory could be added onto an OAI as
extra precautions.

Even without a full friendly utility function, there are some tools specifically available
for utility based OAI. For instance, we could  impose a high discount rate on the
utility, to prevent the OAI fr om planning to far ahead. Other more exotic
manipulations of the OAIs utility function should also be possible; one is detailed in
(Armstrong, Utility Indifference, 2010) . This updates the OAI‟s utility function so
that it is in different to the consequences of a single probabilistic event, and could be
used to make it indifferent to the explosives packed around it, or, more interestingly
for an Oracle, indifferent to the consequences upon humans of its answers.

5.5. Extra measures

An important aspect of the whole OAI design  is that there are many extra methods
that can be implemented and added on top of the basic measures. Exotic methods such
as proxy measures of human survival and utility indifference are detailed in our pape r
(Armstrong, Sandberg, & Bostom, Thinking Inside the Box: Using and controlling an
Oracle AI, 2011) .

6 Conclusions

Analysing the different putative solutions to the OAI -control problem has been a
generally discouraging exerci se. The physical methods of control, which should be
implemented in all cases, are not enough to ensure safe OAI. The other methods of
control have been variously insufficient, problematic, or even downright dangerous.

It is not a question of little hope,  however, but of little current progress. Control
methods used in the real world have been the subject of extensive theoretical analysis
or long practical refinement. The lack of intensive study in AI safety leaves methods
in this field very underdeveloped . But this is an opportunity: much progress can be
expected at relatively little effort. There is no reason that a few good ideas would not
be enough to put the concepts of space and time on a sufficiently firm basis for
rigorous coding, for instance.

And even the seeming failures are of use, it they have inoculated us against dismissive
optimism: the problem of AI control is genuinely hard, and nothing can be gained by
not realising this essential truth. A list of approaches to avoid is invaluable, and ma y
act as a brake on AI research if it wanders into dangerous directions.

On the other hand, there are strong reasons to believe the oracle AI approach is safer
than the general AI approach. The accuracy and containment problems are strictly
simpler than t he general AI safety problem, and many more tools are available to us:
physical and epistemic capability control mainly rely on having the AI boxed, while
many motivational control methods are enhanced by this fact. Hence there are strong
grounds to direct  high-intelligence AI research towards the oracle AI model.

The creation of super -human artificial intelligence may turn out to be potentially
survivable.

7 Acknowledgements

I would like to thank and acknowledge the help of Anders Sandberg, Nick Bostrom,
Vincent  Müller, Owen Cotton -Barratt,  Will Crouch,  Katja Grace,  Robin Hanson, Lisa
Makros, Moshe Looks,  Eric Mandelbaum,  Toby Ord, Carl Shulman , Anna Salomon,
and Eliezer  Yudkowsky.

8 Bibliography
Armstrong, S. (2010). Utility Indifference. FHI Technical Report  .
Armstrong, S., Sandberg, A., & Bostom, N. (2011). Thinking Inside the Box: Using
and controlling an Oracle AI. accepted by Minds and Machines  .
Asimov, I. (1942). Runaround. In Astounding Science Fiction.
Bostrom, N. (2003). Ethical issues in advanced artificial intelligence. Cognitive,
Emotive and Ethical Aspects of Decision Making in Humans  , 2.
Bostrom, N. (2001). Existential Risks: Analyzing Human Extinction Scenarios and
Related Hazards. Journal of Evolution and Technology  , 9.
Bostrom, N. (2009). Information Hazards: A Typology of Potential Harms from
Knowledge.  Retrieved from http://www.nickbostrom.com/information -hazards.pdf.
Bostrom, N. (2000). Predictions from Philosoph y? Coloquia Manilana (PDCIS)  , 7.
Bostrom, N. (2004). The Future of Human Evolution. In C. Tandy (Ed.), Death and
Anti-Death: Two Hundred Years After Kant, Fifty Years After Turing,  (pp. 339 -371).
Palo Alto, California : Ria University Press.
Bostrom, N., & Salamon, A. (2011). The Intelligence Explosion . Retrieved from The
Singularity Hypothesis:
http://singularityhypothesis.blogspot.com/2011/01/intelligence -explosion -
extended.html
Caplan, B. (2008). The totalitarian threat. In N. Bostrom, & M. Cirkovic (Ed s.),
Global Catastrophic Risks  (pp. 504 -519). Oxford University Press.
Chalmers, D. J. (2010). The Singularity: A Philosophical Analysis.  Retrieved from
http://consc.net/papers/singularity.pdf.
Cook, S. (1971). The complexity of theorem proving procedures.  Proceedings of the
Third Annual ACM Symposium on Theory of Computing , (pp. 151 –158).
Evolutionary Algorithm . (n.d.). Retrieved from
http://en.wikipedia.org/wiki/Evolutionary_algorithm
Good, I. (1965). Speculations Concerning the First Ultraintelligent Mac hine.
Advances in Computers  , 6.
Hanson, R. (2000). Long -Term Growth As A Sequence of Exponential Modes.
Retrieved from http://hanson.gmu.edu/longgrow.pdf.
Idel, M. (1990). Golem: Jewish magical and mystical traditions on the artificial
anthropoid.  Albany,  New York: State University of New York Press.
Kahnemand, D., Slovicand, P., & Tversky, A. (1982). Judgement under Uncertainty:
Heuristics and Biases.  Cambridge University Press.
Kurzweil, R. (2005). The Singularity is Near.  Penguin Group.
Mallery, J. C. ( 1988). Thinking about foreign policy: Finding an appropriate role for
artificial intelligence computers. Cambridge, MA: MIT Political Science Department.
McCarthy, J., Minsky, M., Rochester, N., & Shannon, C. (1956). Dartmouth
Conference. Dartmouth Summer Research Conference on Artificial Intelligence.
Omohundro, S. (2008). The basic AI drives. In B. G. P. Wang (Ed.), Proceedings of
the First AGI Conference.  171. Frontiers in Artificial Intelligence and Applications,
IOS Press.
Ord, T., Hillerbrand, R., & Sandberg, A. (2010). Probing the improbable:
Methodological challenges for risks with low probabilities and high stakes. Journal of
Risk Research  (13), 191 -205.
Paperclip Maximiser . (n.d.). Retrieved from
http://wiki.lesswrong.com/wiki/Paperclip_maximizer
Russell, S., & Norvig, P. (1995). Artificial Intelligence: A Modern Approac.  Prentice -
Hall.
Salomon, A. (2009). When Software Goes Mental: Why Artificial Minds Mean Fast
Endogenous Growth.  Retrieved from http://singinst.org/files/htdraft.pdf
Sandberg, A. ( 2001). Friendly Superintelligence . Retrieved from Extropian 5:
http://www.nada.kth.se/~asa/Extro5/Friendly%20Superintelligence.htm
Shulman, C. p. (2010). Omohundro's "Basic AI Drives" and Catastrophic Risks.
Retrieved from http://singinst.org/upload/ai -resource -drives.pdf.
Shulman, C. (2010). Whole Brain Emulation and the Evolution of . Retrieved from
Singularity Institute for Artificial Intelligence: http://singinst.org/upload/WBE -
superorganisms.pdf
Simon, H. A. (1965). The shape of automation for men and m anagement.  Harper &
Row.
Solomonoff, R. (1960). A Preliminary Report on a General Theory of Inductive
Inference.  Cambridge, Ma.
Steels, L., & Brooks, R. (1995). The Artificial Life Route to Artificial Intelligence:
Building Embodied, Situated Agents.
Sutton, R., & Barto, A. (1998). Reinforcement Learning: an Introduction.  Cambridge,
MA: MIT Press.
Turing, A. (1950). Computing Machinery and Intelligence. Mind LIX  , 236 , 433 –460.
von Neumann, J., & Morgenstern, O. (1944). Theory of Games and Economic
Behavior.  Princeton, NJ: Princeton University Press.
Walter, C. (2005). Kryder's Law. Scientific American  .
Yudkowsky, E. (2001). Creating Friendly AI.  Retrieved from
http://singinst.org/CFAI/.
Yudkowsky, E. (2001). Friendly AI 0.9.  Retrieved from
http://sing inst.org/CaTAI/friendly/contents.html.
Yudkowsky, E. (n.d.). General Intelligence and Seed AI 2.3.  Retrieved from
http://singinst.org/ourresearch/publications/GISAI/.
Yudkowsky, E. (2002). The AI -Box Experiment . Retrieved from Singularity Institute:
http:/ /yudkowsky.net/singularity/aibox


The Evolved Radio and its Implications for Modelling the Evolution of Novel Sensors

Jon Bird1 and Paul Layzell2
1Centre for Computational Neuroscience and Robotics
School of Biological Sciences
University of Sussex
Brighton BN1 9QG, UK
jonba@cogs.susx.ac.uk

2Hewlett-Packard Laboratories
Bristol BS34 8QZ, UK
paul_layzell@hp.com

Abstract – Sensor evolution research typically uses
evolutionary algorithms (EAs) to generate sensors that near-
optimally satisfy large numbers of constraints. This is
qualitatively different from the phylogenetic process found in
nature that has resulted, for example, in the mammalian
auditory ossicles evolving from the jaw bones of amphibians and
reptiles, that in turn had previously acted as gill arches in fish.
This paper describes an evolvable hardware experiment that
resulted in a network of transistors sensing and utilising the
radio waves emanating from nearby PCs. We argue that this
evolved ‘radio’ is only the second device ever whose sensors
were constructed in a way that in key aspects is analogous to
that found in nature. We highlight the advantages and
disadvantages of this approach and show why it is practically
impossible to implement a similar process in simulation.

I. INTRODUCTION
The molecular mechanisms underlying energy production
and protein synthesis are virtually identical in all organisms.
The diversity of species is evident in the multitude of
different ways that organisms sense and protect themselves
from changing conditions. About 5% of the molecular
machinery in E. Coli is for sensing and motion, whereas in
humans these processes constitute the majority of our bulk
[1]. One of the key theoretical issues in sensor evolution
research is to explain this increase in complexity: what
processes lead to the development of novel sensors and
effectors [2]? The hope is that theoretical insights might be
applied to the engineering of robot sensors.
Sensor evolution research often uses evolutionary
algorithms (EAs) to investigate the relationship between
sensors and environmental conditions [2]. Typically,
experiments simulate agents acting in environments and
involve searching through large parameter sets to determine
the near-optimal values that satisfy multiple constraints.  For
example, different researchers have coevolved robot sensor
morphology and controllers [3,4] which can lead to insights
into how particular environmental conditions affect sensory
processing by agents [5]. However, the use of EAs has so far not lead to insights into how novel sensors evolve: sensors
that transduce environmental stimuli not previously utilised
by an organism.
Darwin was one of the first to observe that, “throughout
nature almost every part of each living being has probably
served, in a slightly modified condition, for diverse
purposes” [6], a process exemplified by the changing role of
the hyomandibular bone from a brachial structure in fish to
part of the ear in mammals. This paper shows why it is
practically impossible to implement an analogous process in
a simulated environment. This is an important issue for two
reasons. Firstly, it is necessary to be clear about the
differences between EAs and natural evolution if the aim is a
theoretical understanding of the evolution of novel sensors in
organisms [7,8]. Secondly, from an engineering perspective,
when designing robot sensors it is essential to be aware of the
limitations, as well as the strengths, of particular
methodologies.
 In order to highlight some of the differences between EAs,
as typically used in sensor evolution research, and
phylogenetic processes, this paper describes a number of
unconstrained hardware evolution (HE) experiments where
circuits were evolved intrinsically: that is, their fitness was
determined by instantiating and evaluating them in hardware,
rather than in simulation. A key advantage of testing in real-
world, physical environments is that the circuits are free to
take advantage of a wide range of environmental invariants,
none of which have to be specified by the experimenter at the
outset. This range is further extended when the constraints
adopted in conventional electronic engineering to ensure
robust and predictable operation are relaxed [9]. Evolution is
then free to explore very unusual designs: circuits with
strange structures and intricate dynamical behaviours beyond
the scope of conventional design. In unconstrained HE, the
circuit primitives do not have their behaviour constrained
within specific input and output ranges or by temporal
coordination, nor are they restricted to playing specific
functional roles. Consequently, the process of unconstrained
intrinsic HE is more like tinkering than conventional
engineering [10,11] and in some key aspects is analogous to
natural evolution.
In particular, this paper details an unconstrained, intrinsic
HE experiment where a network of transistors sensed and
utilised the radio waves emanating from a nearby PC.
Essentially, the EA led to the construction of a radio.  This is,
as far as the authors know, only the second example of a
physical device whose sensors were constructed by a process
analogous to that of phylogenetic change. We compare the
circuit to the first device constructed in this way: Gordon
Pask’s electrochemical ear [12]. We argue that both of these
devices display three key characteristics: they were
constructed and tested in real environments; their basic
primitives were not constrained to experimenter specified
functional roles; and the primitives were sensitive to a wide
range of environmental stimuli. We highlight the difficulties
in implementing comparable processes in simulation and
argue that only unconstrained physical systems situated in
real-world environments can ever construct novel sensors in a
way analogous to the phylogenetic process found in nature.

II. EVOLUTIONARY TINKERING
“Evolution proceeds like a tinkerer who, during millions of
years, has slowly modified his products, retouching, cutting,
lengthening, using all opportunities to transform and create”
[11]. Through this process evolution has generated novel
sensors “often utilizing organs not originally ‘intended’ for
the purpose they serve at present” [2]. Formalizing this
generative process is one of the key challenges in modelling
the evolution of novel sensors.

A. Evolution of mammalian middle ear ossicles
It is instructive to trace the evolution of the auditory
ossicles in the middle ear of vertebrates as this exemplifies
how homologous structures play different functional roles
over the course of evolutionary time.
In mammals, the function of the middle ear is to act as an
impedance transformer between the low impedance tympanic
membrane and the high impedance oval window of the
cochlea. These two membranes are linked by the three middle
ear ossicles: the malleus, incus and stapes. Without this
impedance matching much of the sound energy arriving at the
ear would be reflected back into the environment. The
primary mechanism of impedance matching is that the area of
the tympanic membrane is larger than that of the oval
window: for example, in the cat it is about 35 times larger
[13]. The pressure acting on the oval window is increased by
the ratio of the two areas. The second mechanism is the lever
action of the middle ear ossicles: the arm of the incus is
shorter than that of the malleus and this causes an increased
force on the stapes.
In fish, the homologue of the auditory ossicles is the
hyomandibular, which was once part of the gill apparatus and
then later functioned as a jaw prop [14]. In tetrapods, this
bone functioned as a structural support and as a transmitter of
vibrations (stapes). Gradually, the bone became finer and less
attached and more and more suited to the task of vibration transmission. Mammals evolved a new joint system for the
jaw and the older skeletal elements became the malleus and
incus. The radical change in the function of the
hyomandibular bone is a good illustration of the tinkering
process of evolution; as Romer and Sturges memorably put it:
“Breathing aids have become feeding aids and finally
hearing aids” [14].

B. Contrast of Engineering and Tinkering
It is useful to compare the engineering and tinkering
approaches to constructing objects. When EAs or other
optimisation methods are applied to an engineering problem,
such as component placement and routing, a clear goal is
defined which is not necessarily constrained by previous
solutions to design problems [11]. The design problem is
represented by a set of alternatives (command variables in the
terminology of Simon [15]) that have to adapt to a set of
environmental parameters whose values are known with
certainty or in terms of a probability distribution. The goal is
then to find the values of the command variables that
maximise the fitness (or other utility) function, given the
values of the environmental parameters and any other
constraints. The choice of command variables is usually
determined by a ‘divide and conquer’ methodology: a system
is functionally decomposed into semi-independent
subsystems, each with separate functional roles, that interact
through their functions, rather than the details of their
implementation. For example, field-programmable gate
arrays (FPGAs) implement Boolean logic using high-gain
groups of analogue transistors that result in the output of each
cell rapidly saturating high or low. In conventional electronic
design the interaction of the cells and the overall behaviour of
FPGAs is viewed at a functional, logic gate level, rather than
in terms of transistor dynamics [16].

Engineering Tinkering
Clear goal/plan Often no goal/plan
Not necessarily dependent on
previous designs Uses whatever is to hand
Aims for best solution given
constraints Makes some kind of workable object
Insulates subsystems and minimises
unforeseen side effects Combines systems or transforms
them for new uses

TABLE ONE – A COMPARISON OF ENGINEERING AND TINKERING

The design and implementation of computational models
follows the engineering methodology. We outline some of the
difficulties that this causes for simulating the evolution of
novel sensors in the next section.

III. MODELS OF SENSOR EVOLUTION
Sensor evolution research investigates the relationship
between sensors and environmental conditions [2]. One of the
outstanding questions in this field is how to use EAs to
construct sensors that are, “able to tap new information
channels in simulated and real-world (hardware)
environmentsº [17].
Sensor evolution research using EAs tends to use static
fitness functions; this is very much the engineering
methodology where the goal is to find a near-optimal solution
to a well-defined problem. This is clearly not analogous to
natural evolution, where the fitness landscape is dynamic and
there is no clearly defined goal  [18]. Some sensor evolution
research tries to make the search less constrained by allowing
limited changes in the dimensionality of the search space. For
example, the number of sensors and/or the size of the
controller, that maps sensor states to behaviour, are varied
[3,5,19]. Menczer and Belew [20] argue that fitness functions
should be implicit, for example, based on energy levels, in
order to allow “creative, ‘open-ended’ evolution”. However,
these approaches cannot overcome a fundamental constraint
in simulating sensor evolution: the experimenter sets a bound
on the possible interactions between the agent and the
environment. This is a direct consequence of the simulation
process: firstly, the experimenter has to model explicitly how
different environmental stimuli change the state of the
sensors; secondly, experimenters only simulate those aspects
of the environment that they think are relevant to their
experiment, otherwise the simulation would become
computationally intractable. These constraints make it very
difficult to see how there can be a simulation of the evolution
of novel sensors, as the possible sensor/environment
interactions are prespecified and cannot vary: an external
observer can model the system deterministically [21]. It
might be argued that a simulation can model the evolution of
a novel sensor from an agent’s perspective. However,
constructing a novel sensor does not involve selecting which
environmental stimulus to utilise from a prespecified finite
list. Lewontin [22] points out that the world can be
partitioned a priori into an infinite number of ecological
niches but that we can only know which of these partitions
are niches by the presence of an organism. The same
argument holds for environmental stimuli, which can only be
defined by reference to an organism. Novel sensors are
constructed when a device, rather than an experimenter,
determines which of the infinite number of environmental
perturbations act as useful stimuli.
The next section details some hardware evolution
experiments which demonstrate the conditions under which
novel sensors can be constructed by an EA.

IV. UNCONSTRAINED INTRINSIC HARDWARE
EVOLUTION
Unconstrained intrinsic HE design usually comprises a
computer running an EA and a reconfigurable device, such as
an FPGA, on which individual genotypes are instantiated as
physical electronic circuits. The fitness of a given circuit is
determined solely by its real time behaviour and other
factors, such as topology, are not considered. For example,
Thompson [9] evolved a circuit on a small corner of a Xilinx
XC6216 FPGA that was able to discriminate between two
square wave inputs of 1 kHz and 10 kHz without using any of
the counters/timers or RC networks that conventional design
would require for this task. The evolved circuit contained several continuous-time recurrent loops and the timing
mechanism relied on a subtle analogue property - possibly
parasitic capacitance - which affected delays in the internal
signal paths according to the input frequency [23]. Both the
loops and the timing mechanism would have been forbidden
under conventional design procedure, but the evolved circuit
made more parsimonious use of the silicon.
Unconstrained, intrinsic HE therefore shows potential for
the design of analogue dynamical systems that may prove
more successful for certain tasks than conventional design.
This approach may also lead to the discovery of novel
electronic ‘tricks’ not yet exploited by conventional design.
Layzell [24] developed the Evolvable Motherboard (EM) to
investigate some of the key issues in intrinsic HE, in
particular to evaluate the relative merits of different basic
components, methods of analysis and interconnection
architectures. The next section gives an overview of this
testbed and describes an experiment where he intrinsically
evolved the first oscillators to reach their target frequency.

A. The Evolvable Motherboard (EM)
The evolvable motherboard is essentially a triangular
matrix of analogue switches, into which daughterboards
containing the desired circuit primitives for evolution can be
inserted.  Any component from transistors and operational
amplifiers to function-level integrated circuits may be used.
Each daughterboard takes up to 8 lines on the switch matrix,
plus a further 8 connections to allow for various power lines
and I/O which may be required by certain components. The
matrix is designed to provide the minimum number of
switches necessary so that every combination of
interconnection between primitives can be configured. By the

Figure 1:  the evolvable motherboard (EM)

appropriate choice of genotype to phenotype mapping, more
or less restrictive interconnection architectures can be
investigated. The analogue switches are configured via an
interface card plugged into a host PC's internal I/O ports,
enabling genotypes to be instantiated in less than 1ms.
The analogue switches are themselves semiconductor
devices, contained within integrated circuits. They behave
like low value resistors, but also exhibit a small degree of
capacitance and inductance, and may therefore play an active
part in any evolving circuit.

B. Oscillator Experiments
There are established techniques for designing oscillators.
In conventional circuits the necessary timing is supplied by a
capacitor whose charge release is controlled by a resistor; this
combination of components is known as an RC time constant.
As the desired frequency decreases, the value of the RC
product increases. Large value capacitors are difficult to
implement in VLSI and are generally provided externally, at
some expense. The motivation was to evolve an oscillator of
a precise frequency without using capacitors. The tone
discriminator experiment discussed above had demonstrated
that evolution can make use of parasitic properties to form
suitable time constants. However, oscillator evolution is a
difficult task when the basic components are transistors.
Whereas oscillation is the likely outcome of recurrent loops
of digital gates or operational amplifiers, precise operating
points must be established before it can be produced by a
network of transistors. These conditions are extremely
unlikely to occur by chance, a fact that was confirmed by
Layzell when he performed some preliminary experiments
where only frequency and amplitude of oscillation were
rewarded. Therefore, he found it necessary to reward output
amplitude, even if the signal was just noise, in order to kick-
start the evolutionary process.
The experiment used 10 bipolar transistors as the circuit
primitives. A generational GA was used, with single point
crossover, rank-based selection and elitism.

Figure 2:  the oscillator experimental setup

The output of the candidate circuits was sampled directly
using a hardware frequency to voltage (f/v) converter placed
between the EM and an a/d converter on the host PC. This
arrangement prevented aliasing errors. If a signal of
amplitude greater than 10 mV is present at its input, the f/v
converter outputs a d.c. voltage in the range [0, 6V] which is
equal to the input frequency multiplied by a constant, k,
whose value is determined by the midpoint of  the f/v
converter’s range. The fitness function was as follows:

( )
(1)
Hz 60   if ,Hz 60  if ,
fitnesstarget target
maxmin

<=> −− +
=
faffffffka
 fa  and    represent respectively output amplitude and
frequency, averaged over 20 samples, each taken at 100 µs
intervals. max   min andff  are the minimum and maximum of
the 20 frequencies sampled. targetf is the target frequency.
The ratio of the minimum and maximum frequencies rewards
constant output frequency.  The f/v converter’s time constant
was configured so that the target frequency corresponded to
3V output, ensuring that the function was smooth for
frequencies above 60 Hz. This lower bound was chosen to
ensure that the f/v converter was detecting oscillation and not
mains hum, which is 50 Hz in the UK. The target frequency
was 25 kHz.
After the genotypes had been instantiated as circuits, there
was a 5 ms delay to allow the f/v converter and rectifier to
stabilise. From 20 runs, 10 resulted in successful oscillation,
attaining the target frequency within 1% and with minimum
amplitude of 100 mV. These represent the first intrinsically
evolved oscillators to reach their target frequencies.
It has proved difficult to clarify exactly how these circuits
work. Probing a typical one with an oscilloscope has shown
that it does not use beat frequencies to achieve the target
frequency. If the transistors are swapped for nominally
identical ones, then the output frequency changes by as much
as 30%. A simulation was created that incorporated all the
parasitic capacitance expected to exist within the physical
circuit, but the simulated circuits failed to oscillate. The
programmable switches almost certainly play an important
role in the behaviour of the circuit and it is only possible to
probe their input and output connections and not the circuitry
in which they are embedded.

C. The Evolved Radio
Some of the circuits achieved high fitness, but when they
were examined with an oscilloscope they did not oscillate
stably: the signals were of the order of 10 – 50 mV amplitude
with rapidly fluctuating frequency. The evolutionary process
had taken advantage of the fact that the fitness function
rewarded amplifiers, even if the output signal was noise. It
seems that some circuits had amplified radio signals present
in the air that were stable enough over the 2 ms sampling
period to give good fitness scores. These signals were
generated by nearby PCs in the laboratory where the
experiments took place.
In order to pick up radio signals the circuits need an aerial
and an extremely high input impedance. This was achieved
by using as an input the printed circuit board tracks on the
EM connected to an open programmable switch whose
impedance is at least 100 MΩ. The high impedance was
confirmed by an electrometer behaviour observed in many of
the non-oscillating circuits: if a person’s hand was brought
close to the circuit, then the d.c. output voltage rose; if the
person remained there, the output voltage remained high,
falling if the person was earthed. The evolutionary process
had utilised not only the EM’s transistors, but also the
analogue switches and the printed circuit to which they were
connected.

D. Other Environmental Effects
In earlier experiments Layzell [25] found that circuits
utilised the oscilloscope used to measure their behaviour as a
path to 0V, via the 10 MΩ impedance of the oscilloscope. If
the oscilloscope was unplugged, the circuit did not work. In a
SPICE simulation where the oscilloscope was represented by
a resistance, the circuit worked, confirming its functional
role.
Some of the evolved oscillators worked successfully until a
soldering iron on a nearby workbench was disconnected from
the mains, at which point oscillation ceased. This occurred
despite high quality laboratory power supplies and extensive
mains filtering. The circuit was apparently sensitive to tiny
transients in its voltage supply. The circuit worked if it was
reinstantiated on the EM, regardless of whether the soldering
iron was on or off. However, tests showed that it failed to
oscillate if during instantiation the programmable switches
were set in a different order to that used originally. It seems
that the circuit was dependent on some initial condition, such
as charge, that only occurred if the switches were set in a
particular sequence.
These results demonstrate that unconstrained, intrinsic HE
will potentially exploit any physical characteristic that can
influence circuit behaviour, and that these characteristics are
present in the entire evolutionary environment. The fact that
the circuits sometimes utilise very particular environmental
conditions and component properties does mean that they do
not always generalise well. This is also the case with many
organisms that live in environments of low variability, as
these niches can be effectively exploited by efficient
specialisations; general solutions are only found in organisms
that inhabit high variability environments [26]. If we
constrain the evolutionary process then we can make the
circuits more transparent, but we also lose any possible
advantages of unconventional design, one of which is the
construction of novel sensors.
We now describe the first device to construct its sensors in
a way analogous to the tinkering process of natural evolution:
Gordon Pask’s electrochemical ear. We then highlight the
key properties that it shares with the evolved radio and which
enable the construction of novel sensors.

V. PASK’S ELECTROCHEMICAL EAR
In 1958 Gordon Pask demonstrated a number of
remarkable mechanisms that were able to construct novel
sensors and thereby determine the relations between their
own states and the environment. In other words, these devices
were able to generate and explore their own state space. Any
observer trying to model the behaviour of these devices
would be forced to change the dimensionality of their model
over time as the devices can transform the underlying
generative system.

A. Description of the Mechanism
The devices are electrochemical assemblages consisting of
a number of small platinum electrodes that are inserted in a
dish of ferrous sulphate solution and connected to a current limited electrical source. Depending on the activity of the
system, these electrodes can act as sinks or sources of
current. Metallic iron threads tend to form between electrodes
where maximum lines of current are flowing. These metallic
threads have a low resistance relative to the solution and so
current will tend to flow down them if the electrical
activation is repeated. Consequently, the potentials at the
electrodes are modified by the formation of threads. If no
current passes through a thread, then it tends to dissolve back
into the acidic solution. The system therefore fundamentally
consists of two opposing processes: one which builds
metallic threads out of ions on relatively negative electrodes
(sinks); and one that dissolves metallic threads back into ions.
The trial and error process of thread development is also
constrained by the concurrent development of neighbouring
threads and also by previously developed structures. Slender
branches extend from a thread in many directions and most of
these dissolve except for the one following the path of
maximum current. If there is an ambiguous path then a thread
can bifurcate. As the total current entering the system is
restricted, threads compete for resources. However, when
there are a number of neighbouring unstable structures, the
threads can amalgamate and form one cooperative structure.
Over time a network of threads can form that is dynamically
stable: the electrochemical mechanism literally grows.
It is possible to associate some of the electrodes with
output devices that enable the behaviour of the system to be
assessed by a user. A reward consists of an increase in the
limited current supply to the assemblage and is therefore a
form of positive reinforcement. Regardless of how the
electrodes are configured, the assemblage will develop a
thread structure that leads to current flowing in such a way
that the user rewards the system. Importantly, the reward is
simply an increased capacity for growth and there is not any
specification of what form it should take.
Critically, the system is not just electrically connected to
the external world: due to the physical nature of the
components, thread formation is also sensitive to
temperature, chemical environment, vibrations and magnetic
fields. Any of these arbitrary disturbances can be viewed as
an input to the system, especially if they affect the
performance of the mechanism so that its current supply is
changed. The system can grow structures that are sensitive to
different environmental stimuli. Pask was able to train an
assemblage to act as an ‘ear’ that could discriminate between
a 50 Hz and 100 Hz tone in about half a day. He was also
able to grow a system that could detect magnetism and one
that was sensitive to pH differences. The development of
sensors constitutes a change in the state space of the
assemblage that was not specified by a designer explicitly.

VI. DISCUSSION
We have described an unconstrained, intrinsic HE
experiment that resulted in the construction of a novel radio
wave sensor. The EM is the second ever experimental system
to construct novel sensors, unconstrained by prespecified
sensor/environment channels. Like Pask’s ear, the evolved
radio determined the nature of its relation to, and knowledge
of, the world. Both of these devices are epistemically
autonomous: they are not restricted to experimenter specified
information channels [27]. By using a process analogous to
the tinkering of natural evolution, epistemically autonomous
devices alter their relationship with the environment
depending on whether a particular configuration generates
rewarded behaviour.
We have argued that there are three key properties that
devices must embody in order for selection pressure to form
them into novel sensors:

• they are situated in the physical world;
• they consist of primitives with no fixed functional
roles;
• and the primitives are sensitive to a wide range of
environmental stimuli.

In Pask’s ear, the second property stems from the fact that
electrochemical devices initially consist of raw material,
which has no specified structure or function; in the evolved
radio this property follows from releasing electronic
components from the constraints of their conventional
operating ranges.
We argue that devices such as this are useful for
highlighting the practical impossibility of simulating the
evolution of novel sensors: programming a simulation
necessarily involves prespecifying the possible
sensor/environment interactions. Novel sensors are
constructed when a device, rather than an experimenter,
determines which of the infinite number of environmental
perturbations act as useful stimuli. Unconstrained, intrinsic
HE has provided a concrete example of such a device and is
potentially a powerful approach to designing robot sensors as
it enables circuits to exploit the rich dynamics of
semiconductor physics and thereby explore regions of design
space that are inaccessible to the conventional engineering
approach.

References
[1] D.S. Goodsell. The Machinery of Life. Copernicus, Springer-
Verlag, 1998.
[2] K. Dautenhahn, D. Polani and T. Uthmann. Guest Editors’
Introduction: Special Issue on Sensor Evolution. Artificial Life 7:
95-97, 2001.
[3] D. Cliff, I. Harvey and P. Husbands. Explorations in Evolutionary
Robotics. Adaptive Behavior, Vol. 2, 1, 73 – 110.
[4] P. Husbands, T.M.C. Smith, N. Jakobi and M. O’Shea. Better
Living through Chemistry: Evolving Gasnets for Robot Control.
Connection Science, 10 (3-4), 185 – 210.
[5] K. Balakrishnan and V. Honovar. On Sensor Evolution in
Robotics. In J. Koza, D. Goldberg, D. Fogel and R.Riolo (Eds.),
Proceedings of the First International Conference on Genetic
Programming (GP-96), MIT Press, 455 – 460, 1996.
[6] C. Darwin. The Various Contrivances by which Orchids are
Fertilised by Insects. D. Appleton, 1886.
[7] E.A. Di Paolo, J. Noble and S. Bullock. Simulation Models as
Opaque Thought Experiments. In M.A. Bedau, J.S. McCaskill,
N.H. Packard and S. Rasmussen (Eds.), Artificial Life VII: The
Seventh International Conference on the Simulation and
Synthesis of Living Systems, Reed College, Portland, Oregon,
USA, 1 - 6 August, 497 – 506, 2000.  [8] J. Bird. Can Robots Illuminate Adaptive Behaviour? In J.
Kelemen and P. Sosík, (Eds.), Advances in Artificial Life:
Proceedings of the Sixth European Conference on Artificial Life,
Springer, 495 - 498, 2001.
[9] A. Thompson. An evolved circuit, intrinsic in silicon, entwined
with physics. International Conference on Evolvable Systems
(ICES96), Vol. 1259, Springer LNCS, 640 – 656, 1997.
[10] D. C. Dennett. Cognitive Science as Reverse Engineering:
Several Meanings of “Top-Down” and “Bottom-Up”. In D.
Prawitz, B. Skyrms and D. Westerstahl (Eds.), Logic,
Methodology, and Philosophy of Science IX, Elsevier Science,
679 – 689, 1994.
[11] F. Jacob. The Possible and the Actual. Penguin Books, 1989.
[12] G. Pask. Physical Analogues to the Growth of a Concept. In
Mechanisation of Thought Processes: Proceedings of a
Symposium held at the National Physical Laboratory on 24-27
November, 1958, Vol. II H.M.S.O., 879 – 928, 1958.
[13] J. O. Pickles. An Introduction to the Physiology of Hearing.
Second Edition. Academic Press, 1988.
[14] A. S. Romer and T.S. Sturges. The Vertebrate Body. Sixth
Edition. Saunders College, 1986.
[15] H.A. Simon. The Sciences of the Artificial. Third Edition. MIT
Press, 1996.
[16] A. Thompson. Artificial Evolution in the Physical World. In T.
Gomi (Ed.), Evolutionary Robotics: From Intelligent Robots to
Artificial Life (ER'97), AAI Books, 101 – 125, 1997.
[17] A. Liese, D. Polani and T. Uthmann. A Study of the Simulated
Evolution of the Spectral Sensitivity of Visual Agent Receptors.
Artificial Life 7: 99 – 124, 2001.
[18] I. Harvey. Artificial Evolution:  A Continuing SAGA. In T. Gomi
(Ed), Evolutionary Robotics: From Intelligent Robots to Artificial
Life, Proc. of 8th Intl. Symposium on Evolutionary Robotics
(ER2001), 2217 LNCS, Springer, 2001.
[19] M. D. Bugajska and A.C. Schultz. Co-Evolution of Form and
Function in the Design of Autonomous Agents: Micro Air
Vehicle Project. GECCO-2000 Workshop on Evolution of Sensors
in Nature, Hardware and Simulation, 240 – 244, 2000.
[20] F. Menczer and R.K. Belew. Evolving sensors in environments of
controlled complexity. In P. Maes and R.A. Brooks (Eds.),
Artificial Life IV: Proceedings of the Fourth International
Workshop on the Synthesis and Simulation of Living Systems.
MIT press, 210 – 221, 1994.
[21] P. Cariani. Emergence and Artificial Life. In C.G. Langton, C.
Taylor, J.D. Farmer and S. Rasmussen (Eds.), Artificial Life II,
Santa Fe Studies in the Sciences of Complexity, Proceedings, 10,
Addison-Wesley, 775-797,1991.
[22] R. Lewontin. Organism and Environment. In H.C. Plotkin (Ed.),
Learning, Development and Culture: Essays in Evolutionary
Epistemology, Wiley, 1982.
[23] A. Thompson and P. Layzell. Analysis of Unconventional
Evolved Electronics. Communications of the ACM, 42 (4), 71 –
79, 1999.
[24] P. Layzell. A New Research Tool for Intrinsic Hardware
Evolution. In M. Sipper, D. Mange and A. Perez-Uribe (Eds.)
Proceedings of the Second International Conference On
Evolvable Systems (ICES98), 1478 LNCS, Springer, 47-56, 1998.
[25] P. Layzell. Hardware Evolution: On the Nature of Artificially
Evolved Electronic Circuits. DPhil Thesis, School of Cognitive
and Computing Sciences, University of Sussex, 2001.
[26] M. Conrad. Adaptability: The significance of variability from
molecule to ecosystem. Plenum, 1983.
[27] P. Cariani. To evolve an ear: Epistemological implications of
Gordon Pask’s electrochemical devices. Systems Research, 10
(3), 19 – 33, 1993.