A Framework for the Safety of Agent-Environment Systems

A F ramew ork for the F ormal Analysis ofMulti/-Agen t SystemsIn Pro c/. F ormal Appr o aches to Multi/-A gent Systems /#28F AMAS/#29 /, a/#0Eliated withET APS /2/0/0/3/, April /1/2/, /2/0/0/3/. W arsa w/, P oland/.Ramesh Bharadw ajCen ter for High Assurance Computer SystemsNa v al Researc h Lab oratoryW ashington DC /2/0/3/7/5 USAAbstract/. In this pap er w e presen ta n i n tegrated formal framew ork forthe sp eci/#0Ccation and analysis of Multi/-Agen t Systems /#28MAS/#29/. Agen ts aresp eci/#0Ced in a sync hronous programming language called Secure Op era/-tions Language /#28SOL/#29 whic h supp orts the mo dular dev elopmen to f se cur eagen ts/. Multi/-agen t systems are constructed from individual agen tm o d /-ules b y using the comp osition op erator of SOL/, the seman tics of whic hare guaran teed to preserv e certain individual agen t prop erties/. The for/-mal seman tics and the underlying framew ork of SOL also serv e as thebasis for analysis and transformation tec hniques suc h as abstraction/,consistency c hec king/, v eri/#0Ccation b y mo del c hec king or theorem pro ving/,and automatic syn thesis of agen t co de/. Based on this framew ork/, w e arecurren tly dev eloping a suite of analysis and transformation to ols for theformal sp eci/#0Ccation/, analysis/, and syn thesis of m ulti/-agen t systems/./1 In tro ductionBuilding trusted applications is hard/, esp ecially in a distributed or mobile set/-ting/. Existing metho ds and to ols are inadequate to deal with the m ultitude ofc hallenges p osed b y distributed application dev elopmen t/. The problem is exac/-erbated in a hostile en vironmen t suc h as the In ternet where/, in addition/, ap/-plications are vulnerable to malicious attac ks/. It is widely ac kno wledged thatin telligen t soft w are agen ts pro vide the righ t paradigm for dev eloping agile/, re/-con/#0Cgurable/, and e/#0Ecien t distributed applications/. Distributed pro cessing ingeneral carries with it risks suc h as denial of service/, T ro jan horses/, informa/-tion leaks/, and malicious co de/. Agen t tec hnology /,b yi n tro ducing autonom y andco de mobilit y /,m a y exacerbate some of these problems/. In particular/, a maliciousagen t could do serious damage to an unprotected host/, and malicious hosts coulddamage agen ts or corrupt agen t data/.Secure Infrastructure for Net w ork ed Systems /#28SINS/#29 b eing dev elop ed at theNa v al Researc h Lab oratory is a middlew are for secure agen ts in tended to pro videthe required degree of trust for mobile agen ts/, in addition to ensuring theircompliance with a set of enforceable securit y p olicies/. An infrastructure suc ha sSINS is cen tral to the successful deplo ymen t and transfer of distributed agen ttec hnology to Industry b ecause securit y is a necessary prerequisite for distributedcomputing/./1
Report Documentation PageForm Approved
OMB No. 0704-0188
Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for r eviewing instructions, searching existing data sources, gathering and
maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden es timate or any other aspect of this collection of information,
including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington
VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a pen alty for failing to comply with a collection of information if it
does not display a currently valid OMB control number.
1. REPORT DATE
2003 2. REPORT TYPE 3. DATES COVERED
 00-00-2003 to 00-00-2003
4. TITLE AND SUBTITLE
A Framework for the Formal Analysis of Multi-Agent Systems 5a. CONTRACT NUMBER
5b. GRANT NUMBER
5c. PROGRAM ELEMENT NUMBER
6. AUTHOR(S) 5d. PROJECT NUMBER
5e. TASK NUMBER
5f. WORK UNIT NUMBER
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)
Naval Research Laboratory,Center for High Assurance ComputerSystems,4555 Overlook Avenue, SW,Washington,DC,20375 8. PERFORMING ORGANIZATION
REPORT NUMBER
9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S)
11. SPONSOR/MONITOR’S REPORT NUMBER(S)
12. DISTRIBUTION/AVAILABILITY STATEMENT
Approved for public release; distribution unlimited
13. SUPPLEMENTARY NOTES
14. ABSTRACT
15. SUBJECT TERMS
16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF
ABSTRACT 18. NUMBER
OF PAGES
13 19a. NAME OF
RESPONSIBLE PERSON
a. REPORT
unclassified b. ABSTRACT
unclassified c. THIS PAGE
unclassified
Standard Form 298 (Rev. 8-98)
Prescribed by ANSI Std Z39-18
/2 SINS Arc hitectureFigure /1 sho ws the arc hitecture of SINS/. Agen ts are created in a sp ecial purp osesync hronous programming language called Secure Op erations Language /#28SOL/#29/#5B/5/#7B/7/#5D/. A SOL application comprises a set of agen t mo dules/, eac h of whic h runson an Agen tI n terpreter /#28AI/#29/. The AI executes the mo dule on a giv en host incompliance with a set of lo cally enforced securit y p olicies/. A SOL m ulti/-agen tsystem ma y run on one or more AIs/, spanning m ultiple hosts across m ultiple ad/-ministrativ e domains/. Agen tI n terpreters comm unicate among themselv es usingan in ter/-agen t proto col /#5B/1/8/#5D/, similar to SO AP/#2FXML /#5B/1/9/#5D/./3 A Brief In tro duction to SOLA mo dule is the unit of sp eci/#0Ccation in SOL and comprises v ariable declarations/,assumptions and guaran tees/, and de/#0Cnitions/. The assumptions section includesassumptions ab out the en vironmen t of the agen t/. Execution ab orts when an yo fthese assumptions are violated b y the en vironmen t/. The required safet y prop/-erties of an agen t are sp eci/#0Ced in the guarantees section/. The definitionssection sp eci/#0Ces up dates to in ternal and con trolled v ariables as functions /#28ormore generally as relations/#29/. A SOL mo dule describ es the required relation b e/-t w een monitor e d variables /,v ariables in the en vironmen t that the agen t monitors/,and c ontr ol le d variables /, v ariables in the en vironmen t that the agen t con trols/.Additional in ternal v ariables are often in tro duced to mak e the description of theagen t concise/. In this pap er/, w e only distinguish be t w een monitored v ariables/,i/.e/./, v ariables whose v alues are sp eci/#0Ced b y the en vironmen t/, and dep endent vari/-ables /, i/.e/./, v ariables whose v alues dep end on the v alues of monitored v ariables/.Dep enden tv ariables include all the con trolled v ariables and in ternal v ariables ofan agen t mo dule/./3/./1 Ev en tsSOL b orro ws from SCR the notion of events /#5B/1/3/#5D/. Informally /, an SCR ev en tdenotes a c hange of state/, i/.e/./, an ev en t is said to o ccur when a state v ariable
Host
Host
Host
AgentInterpreterAgentAgent
Interpreter
InterpreterAgents
Agents
AgentsEncryptEncryptE
ncryp
tE
n
crypt
E
ncry
ptFig/. /1/. Arc hitecture of SINS/./2
c hanges v alue/. SCR systems are ev en t/-driv en and the SCR mo del includes asp ecial notation for denoting them/. The notation /@T/#28c/#29 denotes the ev en t /#5Ccon/-dition c b ecame true/"/, /@F/#28c/#29 denotes /#5Ccondition c b ecame false/" and /@C/#28x/#29the ev en t /#5Cthe v alue of expression x has c hanged/"/. These constructs are de/#0Cnedformally b elo w/. In the sequel/, PREV/#28x/#29 denotes the v alue of expression x in thepr evious state /./@T/#28c/#29
def/= /: PREV /#28 c /#29 /^ c/@F/#28c/#29
def/= PREV /#28 c /#29 /^/: c/@C/#28c/#29
def/= PREV /#28 c /#29 /6/= cEv en ts ma y be triggered predicated up on a condition b y including a /#5C when /"clause/. Informally /, the expression follo wing the k eyw ord when is /#5Caged/" /#28i/.e/./,ev aluated in the pr evious state/#29 and the ev en t o ccurs only when this expressionhas ev aluated to true /.F ormally /,a c onditione d event /, de/#0Cned as/@T/#28c/#29 when d
def/= /: PREV /#28 c /#29 /^ c /^ PREV /#28 d /#29 /;denotes the ev en t /#5Ccondition c b ecame true when condition d w as true in theprevious state/"/. Conditioned ev en ts in v olving the t w o other ev en t constructs arede/#0Cned along similar lines/.In SOL w e extend the SCR ev en t construct to include ev en ts that are trig/-gered b y the in v o cation of a metho d /#28i/.e/./, a pro cedure or function call/#29 of theem b edding language/. F or example/, the ev en t asso ciated with the in v o cation ofmetho d push/#28x/#29 of a stac k is denoted as /@push /. This pro vides users the abilit yto implemen t se curity automata /, a sp ecial class of B /#7F uc hi automata that acceptsafet y prop erties /#5B/1/, /1/7/#5D/./3/./2 De/#0CnitionsAv ariable de/#0Cnition is either a one/-state or a two/-state de/#0Cnition/. A one/-statede/#0Cnition/, of the form x /= expr /#28where expr is an expression/#29/, de/#0Cnes the v alueof v ariable x in terms of the v alues of other v ariables in the same state /.A t w o/-state v ariable de/#0Cnition/, of the form x /= initially init then expr /#28whereexpr is at w o/-state expression/#29/, requires the initial v alue of x to equal expres/-sion init /; the v alue of x in eac h subsequen t state is determined in terms of thev alues of v ariables in that state as wel l as the pr evious state /#28sp eci/#0Ced usingop erator PREV or b y a when clause/#29/. A c onditional expr ession /, consisting of asequence of branc hes /#5C /#5B/#5D guard /! expression/"/, is in tro duced b y the k eyw ord/#5C if /" and enclosed in braces /#28 /"/#7B/" and /"/#7D/" /#29/. A guard is a b o olean expression/. Theseman tics of the conditional expression if f /#5B/#5D g/1
/! expr/1
/#5B/#5D g/2
/! expr/2
/:/:/: g isde/#0Cned along the lines of Dijkstra/'s guar de d c ommands /#5B/1/0/#5D /#7B in a giv en state/,its v alue is equiv alen t to expression expri
whose asso ciated guard gi
is true/. Ifmore than one guard is true/, the expression is nondeterministic/. It is an error if
none of the guards ev aluates to true /, and execution ab orts/. The c ase expr essioncase expr f /#5B/#5D v/1
/! expr/1
/#5B/#5D v/2
/! expr/2
/:/:/: g is equiv alen t to the conditional/3
deterministic reactive enforcement modulesafestack/#28integer max/\_depth/#29 /#7B/#2F/#2F assumption/: max/\_depth /#3E /0interfacesvoid push/#28integer x/#29/;void pop/#28/#29/;integer top/#28/#29/;internal variables/#7Bempty/, nonempty/#7D status/;integer in /#5B/0/:max/\_depth/#5D depth/;guaranteesINV/1 /=/#28status /=/= empty/#29 /#3C/=/#3E /#28depth /=/= /0/#29/;definitions/#5Bstatus/, depth/#5D /= initially /#5Bempty/, /0/#5D thencase PREV/#28status/#29 /#7B/#5B/#5D empty /-/#3Eif /#7B/#5B/#5D /@push /-/#3E /#5Bnonempty/, PREV/#28depth/#29 /+ /1/#5D/#2F/#2F other operations illegal/!/#7D/#5B/#5D nonempty /-/#3Eif /#7B/#5B/#5D /@top /-/#3E/#5BPREV/#28status/#29/, PREV/#28depth/#29/#5D/#5B/#5D /@pop when /#28depth /#3E /1/#29 /-/#3E/#5Bnonempty/, PREV/#28depth/#29 /- /1/#5D/#5B/#5D /@pop when /#28depth /=/= /1/#29 /-/#3E/#5Bempty/, /0/#5D/#5B/#5D /@push when /#28depth/#3Cmax/\_depth/#29 /-/#3E/#5Bnonempty/, PREV/#28depth/#29 /+ /1/#5D/#2F/#2F /@push when /#28depth /=/= max/\_depth/#29 illegal/!/#7D/#7D/; /#2F/#2F end case/#7D /#2F/#2F end module safestackFig/. /2/. Agen t mo dule for safestac k/.expression if f /#5B/#5D/#28 expr /=/= v/1
/#29 /! expr/1
/#5B/#5D/#28 expr /=/= v/2
/#29 /! expr/2
/:/:/: g /. The con/-ditional expression and the case expression ma y optionally ha v ea n otherwiseclause with the ob vious meaning/./3/./3 An Example/: Safet y P olicy Enforcemen tW e examine ho w SOL agen ts are used to enforce safet y p olicies on a giv enagen ti n terpreter/. The example w e shall use is a stac k/, whic h has the asso ciatedmetho ds push /, pop /, and top /. Informally /, push/#28x/#29 pushes the v alue of in tegerv ariable x on the stac k and pop/#28/#29 p ops the topmost v alue o/#0B the stac k/. Themetho d top/#28/#29 returns the curren tv alue at the top of the stac k and lea v es thestac k unc hanged/. The stac k can accommo date at most max depth items/. W e/4
deterministic reactive enforcement module SecureRead /#7Binterfacesstring file/\_read/#28string filename/, integer position/, integer size/#29/;void send/#28string address/, string data/#29/;internal variables/#7Bno/\_reads/, read/\_performed/#7D status/;definitionsstatus /= initially no/\_reads thencase PREV/#28status/#29 /#7B/#5B/#5D no/\_reads /-/#3Eif /#7B/#5B/#5D /@send /-/#3E PREV/#28status/#29/#5B/#5D /@file/\_read /-/#3E read/\_performed/#7D/#5B/#5D read/\_performed /-/#3Eif /#7B/#5B/#5D /@file/\_read /-/#3E read/\_performed/#2F/#2F /@send illegal/!/#7D/#7D/; /#2F/#2F end case/#7D /#2F/#2F end module SecureReadFig/. /3/. A SOL mo dule that enforces safe access to lo cal /#0Cles/.assume that other agen ts /#28not sho wn/#29 access the stac kb yi n v oking its metho ds/.The safet y p olicies w e wish to enforce are/: /#28i/#29 No more than max depth itemsare pushed on the stac k/. /#28ii/#29 In v o cations of metho ds top and pop are disallo w edon an empt y stac k/. Figure /2 sho ws a SOL enforcemen t agen t safestack whic henforces these safet y p olicies on al l other SOL agen ts whic h use the stac k ob ject/#28implemen ted in the em b edding language/#29/. Note that b y delib erately omittingthe otherwise clauses in the if statemen ts/, w e ab ort the execution of a SOLagen t when none of the guards is true during execution/. If this is to o drastic/,correctiv e action ma y b e sp eci/#0Ced in an otherwise clause/; for example/, to ignoreall push actions when the stac k is full/./3/./4 Securit y AutomataW e use the example from /#5B/1/7/#5D to illustrate ho ww em a y enforce a securit y p olicythat allo ws a soft w are agen t to send data to remote hosts /#28using metho d send /#29as w ell as read lo cal /#0Cles /#28using metho d file read /#29/. Ho w ev er/, in v o cations ofsend subsequen t to file read are disallo w ed/. It is di/#0Ecult/, if not imp ossible/,to con/#0Cgure curren t systems to enforce suc h a p olicy /.F or example/, it cannot b eenforced in the /#5Csandb o x/" mo del of Ja v a /#5B/1/1/#5D in whic h one ma y either alw a ysor nev er allo w access to a system resource/. As sho wn in Figure /3/, this p olicy iseasily implemen ted in SOL/./5
/4 F ormal Seman ticsState Machines A SOL mo dule describ es a state mac hine /#5B/6/#5D/. A state machine/#06 is a quadruple /#28 V /; S/; /#02 /; /#1A /#29/, where V /= f v/1
/;v/2
/;/:/:/: /;vn
g is a /#0Cnite set of statevariables /; S is a nonempt y set of states where eac h state s /2 S maps eac h v /2 Vto its range of legal v alues/; /#02 /: S /! b o ole an is a predicate c haracterizing the setof initial states /; /#1A /: S /#02 S /! b o ole an is a predicate c haracterizing the tr ansitionr elation /.W e write /#02 as a logical form ula in v olving the names of v ariables in V /.Predicate /#1A relates the v alues of the state v ariables in a previous state s /2 S totheir v alues in the curren t state s
/0/2 S /.W e write /#1A as a logical form ula in v olvingthe v alues of state v ariables in the previous state /#28sp eci/#0Ced using op erator PREVor b ya when clause/#29 and in the curren t state/.SOL Pr e dic ates Giv en a state mac hine /#06 /=/#28 V /; S/; /#02 /; /#1A /#29w e classify a predicatep /: S /! b o ole an as a one/-state predicate of /#06 and a predicate q /: S /#02 S /! b o ole anas a two/-state predicate of /#06 /.More generally /, SOL pr e dic ate refers to either a one/-state or t w o/-state predi/-cate/, and SOL expr ession refers to logical form ulae or terms con taining referencesto curren t or previous v alues of state v ariables in V /.R e achability Giv en a state mac hine /#06 /=/#28 V /; S/; /#02 /; /#1A /#29/, a state s /2 S is r e achable/#28denoted R e achable/#06
/#28 s /#29/#29 if/#28i/#29 /#02 /#28 s /#29o r/#28ii/#29 /9 s
/0/2 S /: R eachabl e/#06
/#28 s
/0/#29 and /#1A /#28 s
/0/;s /#29Invariants A one/-state predicate p is a state invariant of /#06 if and only if/8 s /: R e achable/#06
/#28 s /#29 /#29 p /#28 s /#29At w o/-state predicate q is a tr ansition invariant of /#06 if and only if/8 s/; s
/0/:/#28 R e achable/#06
/#28 s /#29 /^ /#1A /#28 s/; s
/0/#29/#29 /#29 q /#28 s/; s
/0/#29More generally /, a SOL predicate x is an invariant of /#06 if x is a state in v arian tor transition in v arian to f /#06 /.V eri/#0Cc ation F or a SOL mo dule describing a state mac hine /#06 /, and a set of SOLpredicates X /= f x/1
/;x/2
/;/:/:/: xm
g /,v eri/#0Ccation is the pro cess of establishing thateac h SOL predicate xi
/2 X is an in v arian to f /#06 /./5 SOL Mo duleA SOL mo dule describ es b oth an agen t/'s en vironmen t/, whic h is usually non/-deterministic/, and the required agen t b eha vior/, whic h is usually determinis/-tic /#5B/8/, /1/2/#5D/. Recall that for eac h agen t w e distinguish be t w een its monitored/6
v ariables/, i/.e/./, v ariables in its en vironmen t/, and dep endent variables /, i/.e/./, v ari/-ables whose v alues are determined b y the agen t/. Dep enden t v ariables includeall the con trolled v ariables and in ternal v ariables of an agen t mo dule/. In the se/-quel/, w e assume that v ariables v/1
/;v/2
/;/:/:/: /;vI
are an agen t/'s monitored v ariables/,and that v ariables vI /+/1
/;vI /+/2
/;/:/:/: /;vn
are the agen t/'s dep enden t v ariables/. Thenotation NC /#28 v/1
/;v/2
/;/:/:/: /;vk
/#29 is used as an abbreviation for the SOL predicate/#28 v/1
/= PR E V /#28 v/1
/#29/#29 /^ /#28 v/2
/= PR E V /#28 v/2
/#29/#29 /^ /:/:/: /^ /#28 vk
/= PR E V /#28 vk
/#29/#29/.Comp onen ts of the state mac hine /#06 /=/#28 V /; S/; /#02 /; /#1A /#29 are sp eci/#0Ced in the sectiondefinitions of a SOL mo dule/. The initial predicate /#02 is sp eci/#0Ced in terms ofthe initial v alues for eac h v ariable in V /, i/.e/./, as predicates /#12v/1
/;/#12v/2
/;/:/:/: /;/#12vn
/, sothat /#02 /= /#12v/1
/^ /#12v/2
/^ /:/:/: /^ /#12vn
/. The transition relation /#1A is sp eci/#0Ced as a setof assignmen ts/, one for eac h dep enden t v ariable of /#06 /, i/.e/./, as SOL predicates/#1AvI /+/1
/;/#1AvI /+/2
/;/:/:/:/;/#1Avn
/, eac h of whic h is of the form/:vi
/=
/8/#3E/#3E
/#3E
/#3C/#3E/#3E
/#3E
/:
e/1
if g/1e/2
if g/2/././.ek
if gkwhere I /+/1 /#14 i /#14 n /, and e/1
/;e/2
/;/:/:/:/;ek
are SOL expressions/, and g/1
/;g/2
/;/:/:/:/;gk
areSOL predicates/. T oa v oid circular de/#0Cnitions/, w e imp ose an additional restrictionon the o ccurrences of state v ariables in these expressions as b elo w/:De/#0Cne dep endency r elations Dnew
/, Dold
/, and D on V /#02 V as follo ws/:F or v ariables vi
and vj
/, the pair /#28 vi
/;vj
/#29 /2 Dnew
i/#0B vj
o ccurs outside aPR E V /#28/#29 clause in the SOL expression de/#0Cning vi
/; the pair /#28 vi
/;vj
/#29 /2 Doldi/#0B PR E V /#28 vj
/#29 o ccurs in the SOL expression de/#0Cning vi
/; and D /= Dnew
/#5BDold
/. W e require D
/+new
/, the transitiv e closure of the Dnew
relation/, tode/#0Cne a partial order/./5/./1 Comp osing SOL Mo dulesConsider t w o SOL mo dules describing the state mac hines /#06/1
/=/#28 V/1
/;S/1
/;/#02/1
/;/#1A/1
/#29and /#06/2
/= /#28 V/2
/;S/2
/;/#02/2
/;/#1A/2
/#29/. W e de/#0Cne the c omp osition of the t w o SOL agen ts/#06 /=/#28 V /; S/; /#02 /; /#1A /#29a s /#06 /= /#06/1
k /#06/2
whereV /= V/1
/#5B V/2/#02 /= /#02/1
/^ /#02/2/#1A /= /#1A/1
/^ /#1A/2Eac h s /2 S maps eac h v /2 V to its range of legal v aluespro vided that there is no circularit y in the o ccurrences of v ariables in /#1A /. Also b yassumption/, it is the case that /#1A/1
and /#1A/2
de/#0Cne disjoin t sets of state v ariables/./6 V eri/#0CcationIn this section/, w e discuss ho wt w ow ell/-kno wn v eri/#0Ccation approac hes ma yb eused for establishing the in v ariance of predicates for a state mac hine /#06 /./7
/6/./1 Theorem Pro vingThe /#0Crst approac h/, whic h uses induction /, is p opularly kno wn as theorem pro v/-ing /. Due to its use of lo gic al we akening /, this approac ha v oids the explicit con/-struction of the state space and the calculation of predicate R e achable /.Pro of RulesR ule SINV Let p be a one/-state predicate of /#06 /. The follo wing are su/#0Ecien tconditions to sho w that p is an in v arian to f /#06 /, i/.e/./, /8 s /: R e achable/#1B
/#28 s /#29 /#29 p /#28 s /#29/:S/1/: /8 s /: /#02 /#28 s /#29 /#29 p /#28 s /#29 andS/2/: /8 s/; s
/0/:/#28 p /#28 s /#29 /^ /#1A /#28 s/; s
/0/#29/#29 /#29 p /#28 s
/0/#29/.R ule TINV Let q be a t w o/-state predicate of /#06 /. The follo wing are su/#0Ecien tconditions to sho w that q is a transition in v arian to f /#06 /:T/1/: /8 s/; s
/0/:/#28 /#02 /#28 s /#29 /^ /#1A /#28 s/; s
/0/#29/#29 /#29 q /#28 s/; s
/0/#29T/2/: /8 s/; s
/0/;s
/0/0/:/#28 q /#28 s/; s
/0/#29 /^ /#1A /#28 s
/0/;s
/0/0/#29/#29 /#29 q /#28 s
/0/;s
/0/0/#29Pr o of/: The soundness of the ab o v e rules follo ws b y induction from the de/#0Cnitionof R e achable /.Pro of Rules of SOLv er W e are constructing an automatic v eri/#0Ccation to olSOLv er/, based on theorem pro ving b y induction/, for the v eri/#0Ccation of agen tprop erties/. The pro of rules w e use for v eri/#0Ccation are w eak er forms of the pro ofrules SINV and TINV /. The to ol SOLv er is based up on our paten ted tec hnologydev elop ed in connection with the formal v eri/#0Ccation to ol Salsa /#5B/9/#5D/.R ule SINV/-W Let p b e a one/-state predicate of /#06 /. The follo wing are su/#0Ecien tconditions to sho w that p is an in v arian to f /#06 /:S/1/-W/: /8 s /: /#02 /#28 s /#29 /#29 p /#28 s /#29 andS/2/-W/: /8 s/; s
/0/: /#1A /#28 s/; s
/0/#29 /#29 p /#28 s
/0/#29/.Pr o of/: This is a w eak er form of SINV /./8
R ule TINV/-W Let q be a t w o/-state predicate of /#06 /. The follo wing is a su/#0Ecien tcondition to sho w that q is a transition in v arian to f /#06 /:T/1/-W/: /8 s/; s
/0/: /#1A /#28 s/; s
/0/#29 /#29 q /#28 s/; s
/0/#29Pr o of/: The result follo ws directly from the de/#0Cnition of transition in v arian t/./6/./2 Mo del Chec kingIn general/, the approac h p opularly kno wn as mo del che cking computes the setc haracterized b y the predicate R e achable either explicitly or implicitly /. Explicitstate mo del c hec king computes the set b ye n umerating eac h state in the statespace/. Symb olic mo del che cking computes predicate R e achable b y sym b olic execu/-tion/, using a canonical represen tation /#28suc h as BDDs/#29 for logical form ulae c harac/-terizing sets of states/. One imp ortan t adv an tage of mo del c hec king o v er theorempro ving is its abilit yt op r o vide a c ounter example b y whic hw e mean a sequenceof states s/0
/;s/1
/;/:/:/:/;sn /, /1
/;sn
suc h that /#02 /#28 s/0
/#29 and /8 /#28/1 /#14 i /#14 n /#29/: /#1A /#28 si /, /1
/;si
/#29/, andeither /: x /#28 sn
/#29 /#28for a one/-state predicate x /#29o r /: x /#28 sn /, /1
/;sn
/#29 /#28for a t w o/-state pred/-icate x /#29 where x is a presumed in v arian t of state mac hine /#06
/1/. In addition toits limited applicabilit y to /#0Cnite state systems/, a ma jor disadv an tage of mo delc hec king is state explosion whic h refers to the in tractable complexit y of the al/-gorithms for computing predicate R e achable /./7 AbstractionThe general idea b ehind abstraction is that in order to v erify the in v arianceof a SOL predicate x for a state mac hine /#06 /=/#28 V /; S/; /#02 /; /#1A /#29/, one v eri/#0Ces insteadthat x is an in v arian t of a di/#0Beren t state mac hine /#06A
/=/#28 VA
/;SA
/;/#02A
/;/#1AA
/#29/. W esa y that /#06A
is a sound abstr action of /#06 relativ e to the in v ariance of x if x is anin v arian to f /#06A
implies that x is an in v arian to f /#06 /.W es a y that /#06A
is a c ompleteabstr action of /#06 relativ e to the in v ariance of x if x is an in v arian to f /#06 impliesthat x is an in v arian to f /#06A
/. If /#06A
is a sound and complete abstraction of /#06relativ e to the in v ariance of x /, it is also called an exact abstr action /. In general/,the quotien t system /#06A
/, generated b y an equiv alence relation on S that is abisimulation on /#06 /, will b e exact for al l the in v arian ts of /#06 /.F or some in terestingmetho ds that are b oth sound and complete/, w e refer the reader to /#5B/8/#5D/.The follo wing theorems ab out abstraction are a generalization of man yo ft h epractical /#5Cabstraction metho ds/" used in the v eri/#0Ccation of in v arian ts for SCRsp eci/#0Ccations /#5B/8/, /1/2/#5D/./1Theorem pro ving do es pro vide some information/, whic h ma y be though t of as a/#5Cpartial coun terexample/"/, up on pro of failure/. F or example/, when application of thepro of rule SINV /#28see previous section/#29 fails/, one can usually extract informationab out the state or pair of states for whic h one of the premises of the rule is false/./9
/7/./1 A Theorem of Sound AbstractionsLet /#06 /=/#28 V /; S/; /#02 /; /#1A /#29 b e a state mac hine and let /#06R
/=/#28 V /; S/; /#02A
/;/#1AA
/#29 b e anotherstate mac hine constructed suc h that/:/#28a/#29 /8 s /2 S /: /#02 /#28 s /#29 /#29 /#02A
/#28 s /#29/, and/#28b/#29 /8 s/; s
/0/2 S /: /#1A /#28 s/; s
/0/#29 /#29 /#1AA
/#28 s/; s
/0/#29then /#06R
is a sound abstraction of /#06 with resp ect to the in v ariance of an y SOLpredicate x /, i/.e/./, if x is an in v arian to f /#06R
/, then x is also an in v arian to f /#06 /.Pr o of/: W e initially pro v e the follo wing lemma/:L emma /1/: F or state mac hines /#06 and /#06R
de/#0Cned as ab o v e/,/8 s /: R e achable/#06
/#28 s /#29 /#29 R e achable/#06R
/#28 s /#29/.Pro of/: Let s b e a state of /#06 /. Supp ose s is reac hable in /#06 /. Then/, b y thede/#0Cnition of R e achable/#06
/, /#02 /#28 s /#29 or /9 s/0
/;s/1
/;/:/:/:/;sn /, /1
/;s /: /#02 /#28 s/0
/#29 /^ /#1A /#28 s/0
/;s/1
/#29 /^/:/:/: /^ /#1A /#28 sn /, /1
/;s /#29/. F rom conditions /#28a/#29 and /#28b/#29 ab o v e/, and b y the de/#0Cnitionof R e achable/#06R
/, the conclusion follo ws/.Case /1/: Supp ose x is a one/-state SOL predicate/. Since x is a state in v arian to f/#06R
/#28premise/#29/, i/.e/./, /8 s /: R eachabl e/#06R
/#28 s /#29 /#29 x /#28 s /#29/, the conclusion follo ws from theapplication of Lemma /1/.Case /2/: Supp ose x is a t w o/-state SOL predicate/. Since x is a transition in v arian tof /#06R
/#28premise/#29/, i/.e/./, /8 s/; s
/0/:/#28 R e achable/#06R
/#28 s /#29 /^ R e achable/#06R
/#28 s
/0/#29 /^ /#1AA
/#28 s/; s
/0/#29/#29 /#29q /#28 s/; s
/0/#29/, the conclusion follo ws from condition /#28b/#29 ab o v e and b y the applicationof Lemma /1/./7/./2 Quotien t AutomataConsider the state mac hine /#06R
/=/#28 V /; S/; /#02A
/;/#1AA
/#29/. Let VA
/= v/1
/;v/2
/;/:/:/:vk
b e theset of state v ariables whose names app ear in the logical form ulae
/2c haracterizingpredicates /#02A
and /#1AA
/. Eac h state s /2 S is an in terpretation of the state v ariablesof V /. Let s /#28 v/1
/#29 /;s /#28 v/2
/#29 /;/:/:/:s /#28 vk
/#29 /;/:/:/:s /#28 vn
/#29 denote the in terpretation of v ariablesv/1
/;v/2
/;/:/:/:vk
/;/:/:/: vn
in state s /. The follo wing equiv alence relation /#19 on S /:f /#28 s/1
/;s/2
/#29 j /#28 s/1
/#28 v/1
/#29/= s/2
/#28 v/1
/#29/#29 /^ /#28 s/1
/#28 v/2
/#29/= s/2
/#28 v/2
/#29/#29 /^ /:/:/: /#28 s/1
/#28 vk
/#29/= s/2
/#28 vk
/#29/#29 gcan b e sho wn to b e a bisim ulation on /#06R
/. This follo ws from the de/#0Cnition of abisim ulation relation /#7B b y de/#0Cnition/, R is a bisim ulation relation if R progressesto R itself/; therefore/, since relation /#19 collapses all states that are equal in toan equiv alence class/, transitions emanating from an y state in that set m ust b eiden tical/. Therefore /#06A
/=/#28 VA
/;SA
/;/#02A
/;/#1AA
/#29/, the quotient automaton with resp ectto the bisim ulation/, whose state space SA
is the set of equiv alence classes inducedb y /#19 /, will b e a sound and complete abstraction of /#06R
for al l in v arian ts/. Since/#06R
is a sound abstraction of /#06 relativ e to the in v ariance of all SOL predicates/,/#06A
is a sound abstraction of /#06 for all SOL in v arian ts/. W e call this the GeneralTheorem of Sound Abstractions /./2W e assume that these form ulae ha v e b een simpli/#0Ced to their normal form/./1/0
/7/./3 The SOL Theorem of Sound AbstractionsA corollary of the General Theorem of Sound Abstractions is that b y establishingthe in v ariance of an SOL predicate x for any subset of a collection of SOLmo dules/, one establishes the in v ariance of x for the c omplete SOL m ulti/-agen tsystem/. The pro of of this follo ws from the observ ation that the comp onen ts/#28corresp onding to the dep enden t v ariables/#29 of the initial predicate /#02 and thetransition relation /#1A of the state mac hine describ ed b y a SOL mo dule are sp eci/#0Cedas a conjunction of predicates/, eac h one corresp onding to a dep enden tv ariable inV /. Selection of a subset of SOL de/#0Cnitions /#28whic h de/#0Cne the v alues of dep enden tv ariables/#29 amoun ts to /#5Cthro wing a w a y/" the conjuncts that corresp ond to thev ariables whose de/#0Cnitions are b eing eliminated/. Therefore/, since/#28 /:/:/: /^ /#12rI /+/1
/^ /#12rI /+/2
/:/:/: /^ /#12rk
/:/:/: /^ /#12rn
/#29 /#29 /#28 /:/:/: /#12rI /+/1
/^ /#12rI /+/2
/:/:/: /^ /#12rk
/#29/#28 /:/:/: /^ /#1ArI /+/1
/^ /#1ArI /+/2
/:/:/: /^ /#1Ark
/:/:/: /^ /#1Arn
/#29 /#29 /#28 /:/:/: /^ /#1ArI /+/1
/^ /#1ArI /+/2
/:/:/: /^ /#1Ark
/#29it follo ws from the General Theorem of Sound Abstractions that an in v arian to fa reduced SOL mo dule with dep enden tv ariables vI /+/1
/;vI /+/2
/;/:/:/: /;vk
will also b e anin v arian t of a SOL mo dule with dep enden tv ariables vI /+/1
/;vI /+/2
/;/:/:/:vk
/;vk /+/1
/;/:/:/: vn
/./8 Discussion and Related W orkSOL is based on ideas in tro duced in the Soft w are Cost Reduction /#28SCR/#29 pro ject/#5B/1/4/, /1/5/#5D of the Na v al Researc h Lab oratory whic h dates bac k to the late sev en ties/.The design of SOL w as directly in/#0Duenced b y the design of SAL /#28the SCR Ab/-stract Language/#29/, a sp eci/#0Ccation language based on the SCR F ormal Mo del /#5B/1/3/#5D/.SOL includes certain k ey features of SAL including the notion of events and mo d/-ularit y /. The notation of SCR has directly or indirectly in/#0Duenced more recen tnotations suc h as Reactiv e Mo dules /#5B/4/#5D/. Whereas the application areas of Reac/-tiv e Mo dules are primarily hardw are and comm unication proto cols/, the fo cus ofSCR and related languages has primarily b een on sp ecifying soft w are systems/.Because of in/#0Cnite or v ery large state spaces of systems describ ed in SOL/,/#0Cnite/-state v eri/#0Ccation metho ds suc h as that of Alur and Henzinger /#5B/3/#5D and to olssuc h as Mo c ha /#5B/2/#5D are limited in scop e/. F or v eri/#0Ccation to succeed/, abstrac/-tion /#5B/1/2/, /8/#5D therefore pla y sav ery imp ortan t role when applying mo del c hec king/.Although man y researc hers routinely apply abstractions during mo del c hec k/-ing/, the abstractions are often applied in an ad/-ho c w a y and are therefore notalw a ys sound/. In our approac h/, a systematic application of abstraction guar/-an tees soundness /#5B/8/#5D/. An imp ortan t adv an tage of mo del c hec k ers is their usefor r efutation since the coun terexamples they pro vide when a c hec k fails serv erspractitioners as a v aluable debugging aid /#5B/8/#5D/.Theorem pro ving/, the only other viable alternativ e/, has the asso ciated prob/-lems of incompleteness /#28i/.e/./, a prop ert y that is not pro v able ma y indeed b e v alid/#29and lac k of user guidance/, along the lines of coun terexamples pro vided b y mo delc hec k ers/, in case of pro of failure/. Also/, /#5Ctraditional/" theorem pro ving systemssuc h as PVS /#5B/1/6/#5D require man ual e/#0Bort and mathematical sophistication to use/./1/1
The to ol SOLv er/, curren tly under dev elopmen t/, is based on the to ol Salsa /#5B/9/#5Dwhic h uses SAL as the input language/. Although a theorem pro v er/, Salsa a/#0Bords/#5Cpush/-button/" automation/, ease of use/, and coun terexample generation that t yp/-ify mo del c hec k ers/. The results of our exp erimen ts with Salsa and a comparisonof its p erformance with those of p opular mo del c hec k ers and the theorem pro v erPVS are presen ted in /#5B/9/#5D/. Since the arc hitecture of SOLv er is based on Salsa/#28see /#5B/9/#5D for details/#29/, and the algorithms of SOLv er are extensions of those inSalsa/, w e exp ect SOLv er to ha v e comparable p erformance and to p ossess similarattributes as Salsa/./9 ConclusionIn this pap er w e presen t an in tegrated formal framew ork for the sp eci/#0Ccationand analysis of Multi/-Agen t Systems /#28MAS/#29/. Our framew ork uses a comp ositionop erator the seman tics of whic h are guaran teed to preserv e certain individualagen t prop erties/. W e demonstrate that the formal framew ork ma y serv e as thebasis for v arious analysis and transformation tec hniques suc h abstraction/, andv eri/#0Ccation b y mo del c hec king or theorem pro ving/. W e are curren tly dev elopinga suite of analysis and transformation to ols for SOL based on this framew ork/./1/0 Ac kno wledgemen tsI thank Connie Heitmey er and the anon ymous referees for their v ery usefulcommen ts on previous drafts of the pap er/.References/1/. B/. Alp ern and F/. B/. Sc hneider/. Recognizing safet y and liv eness/. Distribute d Com/-puting /, /2/:/1/1/7/#7B/1/2/6/, /1/9/8/7/./2/. Ra jeev Alur et al/. Mo c ha/: Mo dularit y in mo del c hec king/. In Pr o c/. Computer/-A ide dV eri/#0Cc ation/, /1/0th A nnual Conf/. /#28CA V/'/9/8/#29 /,V ancouv er/, Canada/, /1/9/9/8/. Springer/./3/. Ra jeev Alur and Thomas A/. Henzinger/. Computer A ide dV eri/#0Cc ation/: A n Intr o/-duction to Mo del Building and Mo del Che cking for Concurr ent Systems /. Draft/,www/-cad/.eecs/.berkeley/.edu/#2F/~tah/#2F CavB ook /, /1/9/9/8/./4/. Ra jeev Alur and Thomas A/. Henzinger/. Reactiv e mo dules/. F ormal Metho ds inSystem Design /, /1/5/:/7/#7B/4/8/, /1/9/9/9/./5/. R/. Bharadw a j/. SINS/: a middlew are for autonomous agen ts and secure co de mobil/-it y /.I n Pr o c/. Se c ond International Workshop on Se curity of/, Moble Multi/-A gent Sys/-tems /#28SEMAS/-/0/2/#29/, First International Joint Confer enc e on A utonomous A gentsand Multi/-A gent Systems /#28AAMAS/'/0/2/#29 /, Bologna/, Italy /, July /2/0/0/2/./6/. R/. Bharadw aj /. SOL/: A v eri/#0Cable sync hronous language for reactiv e systems/.In Pr o c/. Synchr/. L anguages/, Apps/./, and Pr o gr amming/, ET APS /2/0/0/2 /, Grenoble/,F rance/, April /2/0/0/2/./7/. R/. Bharadw a j et al/. An infrastructure for secure in terop erabilit y of agen ts/. In Pr o c/.Sixth World Multic onfer enc e on Systemics/, Cyb ernetics/, and Informatics /, Orlando/,Florida/, July /2/0/0/2/./1/2
/8/. R/. Bharadw a j and C/. Heitmey er/. Mo del c hec king complete requiremen ts sp eci/#0C/-cations using abstraction/. A utomate d Softwar e Engine ering /, /6/#28/1/#29/, Jan uary /1/9/9/9/./9/. R/. Bharadw aj and S/. Sims/. Salsa/: Com bining constrain t solv ers with BDDs forautomatic in v arian tc hec king/. In Pr o c/. /6
thInternational Confer enc eo n T o ols andA lgorithms for the Construction and A nalysis of Systems /#28T A CAS/'/2/0/0/0/#29/, ET APS/2/0/0/0 /, Berlin/, Marc h /2/0/0/0/./1/0/. E/. W/. Dijkstra/. A Discipline of Pr o gr amming /. Pren tice/-Hall/, /1/9/7/6/./1/1/. L/. Gong/. Ja v a Securit y/: Presen t and near future/. IEEE Micr o /, /1/5/#28/3/#29/:/1/4/#7B/1/9/, /1/9/9/7/./1/2/. C/. Heitmey er/, J/. Kirb y /, B/. Laba w/, M/. Arc her/, and R/. Bharadw a j/. Using abstractionand mo del c hec king to detect safet y violations in requiremen ts sp eci/#0Ccations/. IEEET r ans/. on Softw/. Eng/. /, /2/4/#28/1/1/#29/, No v em b er /1/9/9/8/./1/3/. C/. L/. Heitmey er/, R/. D/. Je/#0Bords/, and B/. G/. Laba w/. Automated consistency c hec kingof requiremen ts sp eci/#0Ccations/. A CM T r ansactions on Softwar e Engine ering andMetho dolo gy /, /5/#28/3/#29/:/2/3/1/#7B/2/6/1/, April/#7BJune /1/9/9/6/./1/4/. K/. Heninger/, D/. L/. P arnas/, J/. E/. Shore/, and J/. W/. Kallander/. Soft w are requiremen tsfor the A/-/7E aircraft/. T ec hnical Rep ort /3/8/7/6/, Na v al Researc h Lab/./, W ash/./, DC/,/1/9/7/8/./1/5/. K/. L/. Heninger/. Sp ecifying soft w are requiremen ts for complex systems/: New tec h/-niques and their application/. IEEE TSE /, SE/-/6/#28/1/#29/:/2/#7B/1/3/, Jan uary /1/9/8/0/./1/6/. Sam Owre/, John Rush b y /, Natara jan Shank ar/, and F riedric hv on Henk e/. F ormalv eri/#0Ccation for fault/-toleran t arc hitectures/: Prolegomena to the design of PVS/.IEEE T r ansactions on Softwar e Engine ering /, /2/1/#28/2/#29/:/1/0/7/#7B/1/2/5/, F ebruary /1/9/9/5/./1/7/. F/. B/. Sc hneider/. Enforceable securit y p olicies/. A CM T r ans/. Infor/. and SystemSe curity /, /3/#28/1/#29/:/3/0/#7B/5/0/, F ebruary /2/0/0/0/./1/8/. E/. T ressler/. In ter/-agen t proto col for distributed SOL pro cessing/. T ec hnical Rep ortT o App ear/, Na v al Researc h Lab oratory /,W ashington/, DC/, /2/0/0/2/./1/9/. W/3C/. Simple Ob ject Access Proto col /#28SO AP/#29 /1/./1/. T ec hnical Rep ort W/3C Note/0/8/, The W orld Wide W eb Consortium/, Ma y /2/0/0/0/./1/3


. GAMES AND ECONOMIC BEHAVIOR 20,3]24 1997
ARTICLE NO . GA970536
On the Interpretation of Decision Problems with
Imperfect Recall\*
Michele Piccione
Department of Economics, Uni ¨ersity of British Columbia, Vancou ¨er, Canada
and
Ariel Rubinstein
School of Economics, Tel A ¨i¨Uni¨ersity, Tel A ¨i¨, Israel
Received September 18, 1995
 We argue that in extensive decision problems extensive games with a single
. player with imperfect recall care must be taken in interpreting information sets
and strategies. Alternative interpretations allow for different kinds of analysis. Weaddress the following issues: 1. randomization at information sets; 2. consistentbeliefs; 3. time consistency of optimal plans; 4. the multiselves approach to decisionmaking. We illustrate our discussion through an example that we call the ``paradoxof the absentminded driver.'' Journal of Economic Literature Classi®cation Num-
bers: C7, D0.
Q1997 Academic Press
1. INTRODUCTION
This paper is an examination of some modelling problems regarding
imperfect recall within the model of extensive games. It is argued that, ifthe assumption of perfect recall is violated, care must be taken in inter-preting the main elements of the model. Interpretations that are inconse-quential under perfect recall have important implications in the analysis ofgames with imperfect recall.
\* The ®rst author acknowledges ®nancial support from SSHRC-Canada. The second author
acknowledges partial ®nancial support from the United States ]Israel Binational Science
Foundation, Grant Number 1011-341. We are grateful to the University of Oslo and NewYork University for their hospitality, and to Cafe Tamar in Tel Aviv for providing an inspiringatmosphere. We thank Bob Aumann, Paolo Battigalli, Sergiu Hart, Dana Heller, BartLipman , Avishai Margalit, Roger Myerson, Hugh Neary, Motty Perry, Tim Van Zandt, andRuth Weintraub for their comments on earlier versions of this paper.
3
0899-8256 r97 $25.00
Copyright Q1997 by Academic Press
All rights of reproduction in any form reserved.
PICCIONE AND RUBINSTEIN 4
The distinction between perfect and imperfect recall for extensive games
. was introduced in Kuhn 1953 . Since then, traditional game theory has
excluded games with imperfect recall from its scope. In this paper, we wishto readdress this topic. Since the interpretative issues on the agendaappear within the framework of extensive games with a single player, wecon®ne our discussion to decision problems with imperfect recall.
Extensive decision problems are a special case of extensive games in
that the set of players is a singleton. Our basic understanding of anextensive decision problem includes the following assertions:
1. Informational assumptions are modeled by partitioning the situa-
. tions histories where the decision maker takes an action into ``informa-
tion sets.'' The interpretation of the ``informational structure'' is that thedecision maker knows the information set he is at but, if an informationset is not a singleton, he cannot distinguish among the points in the set hehas reached. The decision maker, however, can make inferences or form abelief.
2. A strategy for the decision maker assigns to each information set
one distinct action which is executed whenever an information set is
reached. The decision maker cannot plan to assign different actions to twohistories which lie in the same information set.
3. If the decision maker assesses his strategy at an information set
including more than one history, he forms beliefs about the histories whichled to it. These beliefs are the basis for his considerations.
A decision problem exhibits imperfect recall if, at a point of time, the
decision maker holds information which is forgotten later on. Speci®cally,an information set includes some histories which are incompatible withpreviously held information. Figures 1 ]3 are examples that illustrate three
main aspects of imperfect recall in decision problems.
The standard motive of imperfect recall appears in Example 2. The
decision maker is initially informed about the move of chance and loses
FIG. 1. Example 1.
DECISIONS AND IMPERFECT RECALL 5
FIG. 2. Example 2.
this information upon reaching the information set d. In Example 3, the3
decision maker forgets his initial move at the information set d. Example2
.1 which is central in this paper is rather unconventional; the decision
maker does not distinguish between the ®rst and the second nodes.Reaching the second node, he loses the information that he had previouslymade a choice. The inability of a decision maker to distinguish betweentwo histories on the same path will be referred to as absentmindedness .
In the above examples, the information sets determine the ability of the
decision maker to recall. In practice, a decision maker can affect what heremembers. In this paper, however, we assume that the decision maker isnot allowed to employ an external device to assist him in keeping track ofthe information which he would otherwise lose. Thus, we sidestep thedecision maker's considerations regarding the trade-off between ``morememory'' and ``memory costs.''
FIG. 3. Example 3.
PICCIONE AND RUBINSTEIN 6
The common interpretation of a decision problem with imperfect recall
refers to a situation in which an individual takes several successive actions
and faces memory constraints. Imperfect recall, however, is not necessarily
related to the mental phenomenon of memory loss and may also re¯ect
the imperfect ability to make the inferences necessary for distinguishing
among different points in the same information set. A decision maker may
not realize that he is at the 17th exit along a highway, either because he
does not recall whether he has passed the 16th intersection, or because he
cannot infer that he is at the 17th intersection, despite the perfect picturesof each intersection in his mind. The latter interpretation of imperfect
recall brings our discussion closer to the topic of ``bounded rationality.''
. An alternative interpretation is found in Isbell 1957 . The decision
maker is an organization consisting of many agents who have the same
interests and act at different possible instances. In this case, the decision
process of the organization may exhibit imperfect recall either because it
must keep the instructions given to agents acting in successive situationssimple or because of communication problems between agents. Agents
receive instructions on how to behave and the collection of these instruc-
tions is equivalent to the notion of a strategy.
From a psychological point of view, imperfect recall is a very important
phenomenon as it puts severe constraints on a decision maker's behavior.
We believe, however, that the ultimate proof for its relevance in economic
analysis can only come from an interesting model which clearly explains aneconomic phenomenon. Constructing such a model is beyond the con®nes
of this paper.
The structure of the paper is as follows. We begin by presenting an
example which we call the `` paradox of the absentminded dri
¨er.'' This
example is used to illustrate many of the points of this paper and will be
referred to repeatedly.
After providing the formal de®nition of the model, we discuss two issues
of importance for decision problems with absent-mindedness. First, in
Section 4, we review the circumstances in which whether the decisionmaker is allowed to randomize affect the analysis. Second, in Section 5, we
show that the extension of Bayesian updating to decision problems with
absentmindedness is not trivial.
The two main topics which will be addressed are the timing of decision
and the multiself approaches. The signi®cance of these issues is marginal
in decision problems with perfect recall since possible answers are inconse-
quential for the analysis.
.iTiming of decision . This issue deals with the interpretation of an
information set as either a point of decision or a point in which a strategy
is executed. In Section 6, we will show that for decision problems with
DECISIONS AND IMPERFECT RECALL 7
imperfect recall optimal strategies may be time inconsistent and that
strategies which are time consistent may not be optimal. The timing ofdecisions can be an important consideration for a decision maker whereas,with perfect recall, there is no reason to make some decisions at aparticular point of time.
.iiThe multisel
¨es approach to decision making . Standard dynamic
inconsistencies are generally addressed by assuming that the decisionmaker acts as a collection of distinct ``selves'' who behave independently.The behavior of the decision maker is analyzed as an equilibrium. InSection 7, we extend the multiself approach to decision problems withimperfect recall and show that in this extension an optimal strategy isdynamically consistent.
2. THE PARADOX OF THE ABSENTMINDED DRIVER
An individual is sitting late at night in a bar planning his midnight trip
home. In order to get home he has to take the highway and get off at the
. second exit. Turning at the ®rst exit leads into a disastrous area payoff 0 .
. Turning at the second exit yields the highest reward payoff 4 . If he
continues beyond the second exit, he cannot go back and at the end of the
. highway he will ®nd a motel where he can spend the night payoff 1 . The
driver is absentminded and is aware of this fact. At an intersection, hecannot tell whether it is the ®rst or the second intersection and he cannot
 remember how many he has passed one can make the situation more
. realistic by referring to the 17th intersection . While sitting at the bar, all
he can do is to decide whether or not to exit at an intersection. We excludeat this stage the possibility that the decision maker can include randomelements in his strategy. Example 1 describes this situation.
Planning his trip at the bar, the decision maker must conclude that it is
impossible for him to get home and that he should not exit when reachingan intersection. Thus, his optimal plan will lead him to spend the night atthe motel and yield a payoff of 1. Now, suppose that he reaches anintersection. If he had decided to exit, he would have concluded that he isat the ®rst intersection. Having chosen the strategy to continue, heconcludes that he is at the ®rst intersection with probability 1 r2. Then,
reviewing his plan, he ®nds that it is optimal for him to leave the highwaysince it yields an expected payoff of 2. Despite no new information and nochange in his preferences, the decision maker would like to change hisinitial plan once he reaches an intersection! Note that if the decisionmaker now infers that he would have exited the highway had he passed the®rst intersection, his reasoning becomes circular; he must conclude that heis at the ®rst intersection and that it is optimal to continue.
PICCIONE AND RUBINSTEIN 8
We wish to emphasize that this is not a standard example of time
inconsistency. Usually, time inconsistency is obtained as a consequence of
. changes in either preferences tastes or information regarding the moves
of nature during the execution of the optimal plan. Here, preferences over®nal outcome are constant and the only factor intervening between plan-ning and execution of the optimal strategy is the occurrence of thesituation which calls for execution, that is, reaching the intersection.
We ®nd this example paradoxical as it exhibits a con¯ict between two
ways of reasoning at the intersection. The ®rst is based on a quite minimalprinciple of rationality; having chosen an optimal strategy, one does nothave to verify its optimality at the time of execution unless there is achange in information or in preferences. In our example, the decisionmaker knew he would reach the intersection with certainty and hispreferences are constant. This principle leads to the conclusion that thedecision maker should stick to his plan to continue. The second way isbased on the principle which calls at each instance to maximize expectedpayoffs given the relevant beliefs. In our example, this principle leads tothe conclusion of exiting. The con¯ict between these two potential lines ofreasoning is at the root of the apparent ambiguity of our example.
3. EXTENSIVE DECISION MODEL
In this section, a formal de®nition of the extensive decision model is
. given. The presentation follows that of Osborne and Rubinstein 1994 .
The reader can easily identify the model with the standard de®nition in the``tree'' language.
.  :A ®nite decision problem is a ®ve-tuple GsH,u,C,
r,I, where:
.aHis a ®nite set of sequences . We assume that the empty se-
. quence, f, is an element of Hand that if a,..., agHand1 K
.  .a,..., a/fthen a,..., a gH.1 K 1 Ky1
. We interpret a history a,..., agHas a feasible sequence of1 K
. actions taken by the decision maker or by chance. The history a,..., a1 K
. gHisterminal if there is no a,..., a,agH. The set of terminal1 K
histories is denoted by Z. The set of actions available to the decision
. maker or chance after a nonterminal history his de®ned by Ah sa:
. 4 . h,agH. To avoid degenerate cases we assume that Ah contains at
least two elements. When presenting a decision problem diagramatically,we draw Has a tree whose nodes are the set of histories with root
fand
.  . whose edges combine a node a,..., awith a node a,..., a .1 K 1 Kq1
.  .bu:ZªRis a utility function which assigns a number payoff to
each of the terminal histories. Preferences are de®ned on the set of alllotteries over terminal histories and satisfy the VNM assumptions.
DECISIONS AND IMPERFECT RECALL 9
.cCis a subset of H. We assume that the chance player moves after
histories in C.
.dris the decision maker's belief about the chance player's behav-
. ior. rassigns to each history hgCa probability measure on Ah .T o
. avoid degeneracy, we assume that rh,ais strictly positive for all hgC
. and agAh .
Thus, the set of histories His partitioned into three subsets:
Z, the set of terminal histories;
C, the set of histories after which chance moves;
DsHyZyC, the set of histories after which the decision maker
moves.
.e The set of information sets, which is denoted by I, is a partition of
. D. We assume that for all h,h9in the same cell of the partition Ah s
.Ah9; i.e., the sets of actions available to the decision maker at histories in
the same information set are identical. For convenience, with a slight
abuse of notation, we will sometimes denote the set of actions which are
. available at a history in XbyAX .
Note that, in contrast to some authors, we do not exclude from the class
 of decision problems those which exhibit absentmindedness see de®nition
. below .
If all information sets in Iare singletons we say that Gis a decision
problem with perfect information .
.Apure strategy ,f, is a function which assigns to every history hgD
. an element of Ah with the restriction that if hand h9are in the same
.  . information set fh sfh9. Notice that this de®nition requires that the
decision maker plans an action at histories which he will not reach if hefollows the strategy.
We are now ready for the main de®nitions of this paper. The experience
. of the decision maker at a history hinD, denoted by exp h, is the
sequence of information sets and actions of the decision maker along thehistory h. We adopt the convention that the last element in the sequence
.exp his the information set which contains h. Thus, in Example 1,
. . .  .exp
fsdand exp Bsd,B,d.11 1
A decision problem has perfect recall if for any two histories, h,h9gD,
.  . which lie in the same information set, exp hsexp h9. Thus, in a
decision problem with perfect recall, the decision maker ``remembers'' thesuccession of the information sets he has faced and the actions he hastaken. A decision problem for which the above condition is violated isreferred to as a decision problem with imperfect recall .
. Given a history hsa,..., a and L-K, the history h9s
1 K
.a,..., ais a subhistory ofh. We say that a decision problem Gexhibits1 L
PICCIONE AND RUBINSTEIN 10
absentmindedness if there are two histories hand h9such that h9is a
subhistory of hand both belong to the same information set.
The decision problem illustrated in Example 1 exhibits absentminded-
. ness since history Band its subhistory fare in the same information set.
4. THE VALUE OF RANDOMIZATION
In this section, we discuss the implications of enlarging the strategy set
of a decision maker to include random strategies. Given that the decisionmaker behaves as an expected utility maximizer, randomization over purestrategies is redundant for problems of either perfect or imperfect recall.De®ne a mixed strategy to be a probability distribution over the set of pure
strategies. It describes a behavior in which randomization occurs only atthe outset, before the decision problem unfolds. Each pure strategyinduces a lottery over Z. A mixed strategy induces a lottery over Zwhich
is the compound lottery of the lotteries induced by each of the purestrategies in its support. Therefore, no mixed strategy can be strictlypreferred to all the pure strategies.
Behavior strategies perform a different method of randomization. A
beha
¨ioral strategy ,b, is a function which assigns to every history hgD,a
. . .  . distribution b h over Ah such that bh sbh9for any two histories h
and h9which lie in the same information set. In decision problems without
. absentmindedness bh is a lottery which is realized when the information
set which contains his reached. For decision problems with absentminded-
. ness we take bh to be a random device which is activated independently
every time the information set which includes his reached.
Consider again Example 1. In this problem there are two pure strategies,
``B'' and `` E'' which yield payoffs of 1 and 0, respectively. Although the
absentminded driver cannot use a pure strategy to reach home withcertainty, he can toss a coin and obtain an expected payoff of 1.25. Note
1that his optimal behavioral strategy is to exit with probability and yields34the expected payoff of .3
It turns out that absentmindedness is necessary for behavioral strategy
. to be strictly optimal. This was shown in Isbell 1957 and we provide the
proof for completeness.
PROPOSITION 1.Suppose Gdoes not exhibit absentmindedness .Then for
any beha ¨ioral strategy there is a pure strategy which yields a payoff at least as
high.
: Con¨ersely ,suppose GsH,u,C,r,I exhibits absentmindedness .Then ,
: there exist a decision problem G9sH,u9,C,r,I and a beha ¨ioral strategy
which yields a payoff strictly higher than any payoff achie ¨ed by a pure strategy .
DECISIONS AND IMPERFECT RECALL 11
Proof. See the Appendix.
The Paradox of the Absentminded Dri ¨er Re ¨isited . The inconsistency
discussed in Section 2 is not a consequence of the restriction that thestrategy set includes only pure strategies and persists when the decisionmaker is allowed to choose random actions. The optimal behavioral
2strategy is to choose Bwith probability . Reaching the intersection, the3
driver will form beliefs about where he is. Denote by athe probability he
assigns to being at the ®rst intersection. Then, his expected payoff is
w2. x. w. x apq41ypp q1yapq41yp, where pis the probability of
. 4 not exiting. The optimal pis now max 0, 7 ay3r6a. This is inconsis-
tent with his original plan unless as1. In other words, his original plan is
time consistent if and only if he believes that there is no chance he haspassed the ®rst intersection. We ®nd such a belief unreasonable. Given hisstrategy it seems natural to assign to the second intersection a probability
2which is times the probability assigned to the ®rst intersection, which3
implies as0.6. The issue of consistent beliefs will be discussed in the
next section.
This type of time inconsistency can appear also in decision problems in
which the optimal strategy is pure. Consider the following example shownin Fig. 4.
The optimal behavioral strategy is the pure strategy which selects Lat
dand Latd. To verify it, denote by
aand bthe probabilities of12
2.  choosing Latdand d, respectively, and note that abq3a1ya112
. ybis strictly less than 1 unless asbs1. Upon reaching the informa-
tion set d, if the decision maker concludes that he is at the two histories1
with equal probabilities, the strategy Ratdand Ratdyields the higher12
expected payoff of 1.5.
FIG. 4. Example 4.
PICCIONE AND RUBINSTEIN 12
5. CONSISTENT BELIEFS
If information sets are to be interpreted as points of decision, Examples
1 and 4 suggest that a decision maker who acts on the basis of expected
utility maximization may be unable to execute the optimal strategy. The
®rst step in addressing this issue is to specify the decision maker's beliefs
at an information set which is not a singleton. As we shall see, ®nding an
appropriate speci®cation for decision problems with absentmindedness is
not conceptually trivial.
We de®ne a belief system as a function mwhich assigns to any
<. information set Xand any history hgXa nonnegative number mhX
<. such that  mhX s1. The interpretation is that the decision maker,hgX
<. upon reaching X, assigns probability mhX to the possibility that he is at
<. h. Let ph h 9,bbe the probability that, conditional on reaching h9, the
history hwill be realized when executing the strategy b. We denote
<. <. phf,bbyph b .
Several alternatives are conceivable for the speci®cation of the decision
maker's beliefs. Since our objective is to examine the optimality of a
strategy during its execution, we ®nd it natural to assume that the beliefs
of the decision maker are related in a systematic way with the strategy to
be assessed. The condition that we require a belief system mto satisfy to
be consistent with a behavioral strategy bmirrors the frequency approach
to belief formation. Namely, if an information set Xis reached with
<. positive probability, mhX is assumed to be equal to the long run
proportion of times in which ``visiting'' the information set Xinvolves
being in hfor a decision maker who plays the decision problem again and
again and follows b.
DEFINITION . A belief system misconsistent with the behavioral strat-
egy bif for every information set Xwhich is reached with positive
<. <. <. probability and for every hgX,mhX sph b r ph9b.h9gX
Our de®nition of consistency imposes restrictions only on beliefs at
information sets which are reached with positive probability. Notice that,
for decision problems without absentmindedness, consistency is equivalent
to Bayes' formula. For decision problem with absentmindedness, however,
the denominator can be greater than one. The similarity with Bayes'
formula is only notational.
To clarify this de®nition, consider ®rst the absentminded driver example
1and the strategy that selects Bwith probability equal to . A consistent22belief, conditional upon the information set d, assigns probability to1 3
. being at the ®rst intersection. Consider next Example 5 see Fig. 5 and the
strategy which selects Bwith probability 1.
DECISIONS AND IMPERFECT RECALL 13
FIG. 5. Example 5.
1Consistent beliefs at the information set dassign probability to each1 3
history in the information set.
Our de®nition of consistent beliefs can also be motivated as being
derived from a probability space which includes the time at which the
decision maker can be. As an illustration consider Example 5 and assume
that each action takes one unit of time. The relevant space of instances
. . . .  . consists of L,1 , L,2 , R, 1 , and R, 2 , where x,tis the instance in
which the chance player chooses xand time is t. Assuming equal probabil-
ities for each instance is consistent with the description of the chanceplayer in the decision problem. Then, a decision maker who is told that heis at dupdates his belief by the Bayesian formula. For example, the
1
unconditional probability of the second node after Rispr4, where pis
the probability of choosing Batd, and the unconditional probability of11the node after Ris . If ps1, the conditional probability of each of the41three nodes in dis .1 3
In this paper, we adopt the above de®nition of consistency. However, we
®nd the following de®nition of consistency reasonable as well.
DEFINITION . A belief system misZ-consistent with the behavioral
strategy bif for every information set Xwhich is reached with positive
PICCIONE AND RUBINSTEIN 14
probability and for every hgX
< mhX.
<4  pz b rah9¬h9gXand is a subhistory of z .z<his a subhistory of z4s .<  pz b.z<zhas a subhistory in X4
The rationale behind this de®nition is that, given the behavioral strategy
b, the probability of the event that the random elements which determine
<. the terminal history zare realized is pz b ; if the history zincludes K
histories in which the decision maker is asked to act then he assigns to
<. each of them the ex ante probability pz b rK.
To illustrate the Z-consistency, consider again the absentminded driver
1example and the strategy that selects Bwith probability . A Z-consistent23belief, conditional upon the information set d, assigns probability to1 4
being at the ®rst intersection. In Example 5, if the behavioral strategy
selects Bwith probability 1, Z-consistent beliefs at the information set d111. assign probability to the history Land probability to each of the24
.  . histories Rand R,B.
Note that the paradoxical ¯avor of the absentminded driver example is
unaffected by the type of consistency we adopt. Also, if the decisionproblem does not exhibit absentmindedness then consistency and Z-con-
sistency are identical.
We do not have a ®rm view about the ``right'' de®nition of consistent
beliefs. The issue deserves further investigation. We wish to point out thatif the decision maker adopts Z-consistency then he is exposed to a sort of
``money pump.'' Consider Example 5 and the behavioral strategy thatselects Bwith probability 1. Given Z-consistent beliefs a risk neutral
decision maker, upon reaching the information set, will always accept an
. agreement in which he gains $1.1 if he is at Land loses $1 otherwise. If
such an agreement is offered whenever the information set is reached, theresulting undesirable lottery yields the decision maker $1.1 with probability0.5 and y$2 with probability 0.5.
6. TIME CONSISTENCY
When does the decision maker make his decision? Can he decide about
the point of time at which a decision is made? To what extent can thedecision maker commit to decisions he makes? These questions aresuper¯uous for decision problems with perfect recall since the ex ante
optimal strategy remains optimal during its execution. In the presence ofimperfect recall, the optimal strategy may cease to be optimal along itsexecution and answers to these questions are signi®cant for the analysis.
DECISIONS AND IMPERFECT RECALL 15
We say that a strategy is time consistent if there is not, at any informa-
tion set which is reached as the decision problem unfolds, a differentstrategy for the remainder of the decision problem which yields a higherexpected payoff. Notice that we require the optimality to be assessed onlyat the information sets and not before the initial history .
The signi®cance of the statement that a strategy is optimal at an
information set depends on the beliefs that the decision maker is assumedto hold. Requiring that beliefs be consistent leads to the following de®ni-tion.
D
EFINITION . A behavioral strategy bistime consistent if there is a
belief mconsistent with bsuch that for every information set Xwhich is
reached with positive probability under b,
mhp z ¬h,buz G mhp z ¬h,b9uz .  .. .  ..
hgXz gZh gXz gZ
for any behavioral strategy b9.
The notion of time consistency that we use is a very strong criterion. We
must emphasize that it does not rest on a model of how decisions are madeat each information set. The de®nition presumes that the decision maker,in principle, can commit to a future course of action. Of course, if thedecision maker is aware that a revision of the plan is possible at futureinformation sets, expectations of commitments are naive.
Time consistency is an assessment as to whether, conditional upon the
realization of information set, there exists an alternative rule of behaviorwhich yields the decision maker a higher payoff. If an optimal strategysatis®es this criterion, the distinction with respect to information sets aspoints of decision or points of execution is inconsequential for a decisionmaker who is implementing it and whose beliefs are consistent. If, how-ever, an optimal strategy fails this criterion, this distinction becomessigni®cant and alternative speci®cations of the domain of choice and thetiming of decision of the decision maker can give rise to different analyses.
A well-known result states that for decision problems with perfect recall
a strategy is optimal if and only if it is time consistent. This is not the casefor problems with imperfect recall. As we have seen, the optimal behav-ioral strategy for the absentminded driver is not time consistent. It is easy
5to see that the only time consistent strategy is to exit with probability .9
The problem of time consistency can also arise without absentminded-
ness as Example 2 shows. The optimal strategy is to choose Satd,Bat1
d, and Ratd. However, upon reaching dthe decision maker prefers23 1
using Batdand Latd. A decision maker who can postpone his13
decisions until he has reached dordcan make a better decision12
regarding the choice at dthan a decision maker who decides ex ante .A3
PICCIONE AND RUBINSTEIN 16
complete model should indicate if the domain of choice of the decision
maker includes timing of the decision and the extent of his ability to usehis knowledge at dand dto in¯uence his choice at d.
12 3
In the case of absentmindedness, the divergence between optimality and
time consistency is two-sided. The only optimal strategy is not timeconsistent and the only time consistent strategy is not optimal. The next
. example Fig. 6 shows that, even without absentmindedness, a strategy
may be time consistent and not optimal.
Consider the strategy where the decision maker plays Bat both infor-
mation sets. The belief system consistent with this strategy assigns equalprobabilities to each of the histories in each of the information sets.Therefore, this strategy is time consistent although the optimal strategy isto choose Sat both information sets.
The next proposition provides conditions under which optimal and time
consistent strategies are equivalent. Two histories are said to split athifh
is their longest common subhistory.
P
ROPOSITION 2.LetGbe a decision problem without absentmindedness
for which
for any information set X and two histories h 9,h0gX which split at
hgC,the information sets which appear in exp h9are the same as in.
exp h0.. .)
A beha ¨ioral strategy for Gis optimal if and only if it is time consistent .
FIG. 6. Example 6.
DECISIONS AND IMPERFECT RECALL 17
Proof. See the Appendix.
. To clarify the meaning of condition )in Proposition 2, consider ®rst
. . . Example 2. Both histories L,Band R,Bare in d, but exp L,Bs3
.  . .  .d,B,dand exp R,Bsd,B,d.)is violated, since at dthe13 23 3
decision maker loses information about the move of chance. In Example 6,
even though at no information set the decision maker knows nature's
. .  .  . move, ) is violated since exp LsL,d and exp R,Bs2
.R,d,B,d. Notice that, however, if the decision maker had beliefs12
consistent with the strategy assigning Btodand Stod, he would assign121probability one to nature's move Rimmediately after Rand probability2
to both of nature's moves when Ris followed by his action B.
7. THE MULTISELF APPROACHES
. For standard dynamic inconsistencies, Strotz 1956 provides a frame-
work of analysis in which every information set is assumed to be a point ofdecision and the decision maker is unable to control his behavior at futureinformation sets. A decision maker acts as a collection of hypothetical
. agents selves whose plans form an equilibrium. More precisely, for any
. decision problem Gde®ne GGto be the extensive game in which each
information set of Gis assigned a distinct player, and all players have the
same payoff as the decision maker. The behavior of the decision maker in
. Gis then analyzed as an equilibrium of GG. As is well known, any
optimal play for Gis the play induced by some subgame perfect equilib-
. . rium of GG, and any subgame perfect equilibrium of GGcorresponds
to an optimal strategy. This property has the consequence that the back-ward induction algorithm is a procedure for solving a decision problemwith perfect information. An analogous result is valid for decision prob-lems with perfect recall and imperfect information. In this case we use thesolution concept of sequential equilibrium which combines sequentialrationality with the requirement of consistent beliefs. The set of distribu-tions over the terminal nodes generated by the sequential equilibria of
.GGis identical to the set of distributions generated by the optimal
. strategies of Gsee Hendon, Jacobsen, and Sloth, 1993 .
These results are called the ``no single improvement'' property; a strat-
egy is optimal if, conditional upon reaching an information set, thedecision maker cannot improve his expected payoff by changing only theaction prescribed by the strategy for that information set. It implies that itmakes no difference whether we model a decision problem piecemeal or inits entirety. The equivalence of the single-self and the multiselves ap-proaches for decision problems with perfect recall breaks down when we
PICCIONE AND RUBINSTEIN 18
analyze decision problems with imperfect recall. Consider Example 3. The
. optimal strategy R,rcorresponds to a sequential equilibrium in the
corresponding multiself game. However, the multiself approach does not
. rule out the inferior strategy L,l; the decision maker cannot improve his
payoff by simply changing the action prescribed at a single information set.
Conditional upon being at d, the choice of Lis optimal if the decision1
maker treats his behavior at das given and unchangeable and believes he2
will play l.
The Paradox of the Absentminded Dri ¨er Re -re¨isited . Consider again the
paradox of the absentminded driver. The formal notion of sequential
equilibrium is not applicable to decision problems with absentmindedness.However, the requirement that the behavioral strategy at each information
set is optimal, given the consistent beliefs, is meaningful. Despite the fact
that there is only one information set, this requirement generates a strong
dependence between behavioral strategies and beliefs. As we have noted
before, the only strategy which satis®es this requirement is to exit with
5probability , whereas the strategy that maximizes expected payoffs ex ante91is to exit with probability . Nevertheless, one can modify the multiself3
approach to avoid the possibility that an optimal strategy does not satisfy
sequential rationality with respect to consistent beliefs.
Beliefs consistent with the optimal behavioral strategy in the paradox of
3the absentminded driver assign probability to the ®rst intersection. If the5
decision maker anticipates that his ``twin-self,'' if asked to move again, will
1 1exit with probability , then it is optimal for him to exit with probability3 332 wx wx as he is indifferent between Eyielding the expected payoff 0 q 45583 1 2 2 8. w. . xw x . s and Byielding an expected payoff of 4 q 1q 1s .I n55 3 3 5 5
this modi®cation, when considering a deviation at an information set X
from the behavioral strategy assigned to X, the ``self'' assumes that his
``twin-self'' will use the equilibrium strategy if asked again to act, regardless
of the choice he now makes. The requirement that the decision maker uses
identical randomization at all histories in the same information set is
retained only in equilibrium. In an actual play of the decision problem, if
he decides to deviate, he may use different randomizations at different
histories in the same information set.
Also notice that, for the optimal strategy to satisfy sequential rationality
in this modi®cation of the multiself approach, the decision maker must
3believe that he is at the ®rst intersection with probability . In particular, if5
one replaces our de®nition of consistency of beliefs with Z-consistency,
sequential rationality fails in this modi®cation as well. We now de®ne the
``modi®ed multiself'' approach formally and prove that every optimal
 strategy is modi®ed multiself consistent discussions with Bob Aumann and
. Roger Myerson were very helpful in shaping this part of the paper .
DECISIONS AND IMPERFECT RECALL 19
DEFINITION . A behavioral strategy bismodi®ed multiself consistent if
there exists a belief mconsistent with bsuch that for every information set
. Xwhich is reached with positive probability and for every action agAX
. .  . for which bh a )0 for hgX, there is no a9gAX such that
mhP z ¬h,a9,buz .  . . .
hgXz gZ
) mhP z ¬h,a,buz . .  . . .
hgXz gZ
Note that in contrast with the approach discussed in the ®rst part of this
section, the above de®nition imposes requirements only on the selveswhich are reached with positive probability.
P
ROPOSITION 3.If a beha ¨ioral strategy is optimal then it is modi®ed
multiself consistent .
Proof. See the Appendix.
Proposition 3 demonstrates that, within the framework of the ``modi®ed''
multiself approach, the inconsistency of optimal plans for decision prob-lems with absentmindedness evaporates. The interpretative ambiguitiesoriginating in the paradox of the absentminded driver also vanish as the
2strategy to exit with probability is singled out as the only optimal and3
consistent strategy. The value of this resolution, however, depends on theappropriateness of the ``modi®ed'' multiself approach as a theory ofdecision making under imperfect recall. The behavioral assumption of themultiself approach is that each self maximizes his conditional expectedpayoff given his own beliefs. In decision problems with perfect recall,beliefs at each information set are independent of the choices of previousselves and each self can determine the optimal decisions of future selvesby recursion. In decision problems with imperfect recall, however, theconditional payoff of each self depends on his beliefs about the behavior ofearlier selves and the independence of beliefs and actions from previouschoices is an additional assumption. For decision problems with absent-mindedness, the ``modi®ed'' multiself approach assumes that a decision
maker, upon reaching an information set, takes his actions to be im-mutable at future occurrences of that information set, no matter whichcourse of action he is contemplating now. At the other extreme one ®ndsthe opposite axiom for which only one self resides in the information setand expects that, were the information set to occur again, he would adoptwhichever behavioral rule he adopts now. In this case, the strategy to exit
5with probability would be the consistent rule of behavior for Example 1.9
These two assumptions re¯ect two alternative ways of reasoning at the
PICCIONE AND RUBINSTEIN 20
information set. We ®nd both of them to have some intuitive appeal and
neither to be universally valid.
8. FINAL COMMENTS
Our observations are only a limited exercise intended to suggest some of
the issues which a comprehensive theory of imperfect recall must confront.We conclude our discussion with the following three comments:
a.Other types of imperfect recall . We model imperfect recall by includ-
ing in the same information set histories which contain different experi-ences. Such histories are assumed to be indistinguishable for the decisionmaker. A different type of imperfect recall is represented by a situation inwhich ``I know that at the ®rst intersection I would be aware of this but atthe second intersection I do not know whether I am at the ®rst or thesecond intersection.'' One way to model such a scenario is by nonparti-
.  . tional informational structures see Geanakopolos, 1990 . Rubinstein 1991
discussed the dif®culty of modelling considerations such as ``At the ®rstintersection I know that I am there but if I reach the second intersection I
 think that there is a 10% chance that I am at the ®rst intersection'' see
. also Fluck, 1994 . It seems to us that new analytical frameworks are
needed to address these issues as well as issues such as partial recall ofstrategy and imperfect ability to make inferences.
b.Games with imperfect recall . This paper focuses on issues related to
modeling decision problems with imperfect recall. Obviously, such issuescarry over to extensive games. Isbell and Kuhn show that the nonequiva-lence between mixed and behavioral strategies can cause nonexistence ofNash equilibria in behavioral strategies. Time inconsistency of optimalstrategies for some decision problems with imperfect recall makes itpossible to construct games with imperfect recall that have no equilibriawhich satisfy sequential rationality with respect to consistent beliefs.
Ambiguities in the interpretation of games with imperfect recall is
probably the reason why only a handful of papers have dealt with the topic.
. . The few exceptions include Isbell 1957 and Alpern 1988 , who prove the
existence of mixed behavioral strategy equilibrium in a class of games with
. imperfect recall, and Binmore 1992 who relates imperfect recall and
nonpartitional knowledge. The literature on machines playing repeated
. games see, for example, Rubinstein, 1986 may also be interpreted as an
analysis of a class of repeated games with imperfect recall, where it iscostly for the players to distinguish between different histories of thegame. Further investigation is needed to adapt solution concepts designedfor games with perfect recall to games with imperfect recall.
DECISIONS AND IMPERFECT RECALL 21
c.The paradox of the absentminded dri ¨er. In all its forms the absent-
minded driver example exhibits a con¯ict between two types of reasoning.Commitment to the ex ante optimal behavioral strategy is obviously the
normative rule of behavior.
We do not have a ®rm view about the resolution of the paradox. We
have investigated one resolution which requires dividing a decision makerinto multiple independent selves. Another resolution would entail therejection of expected utility maximization given consistent beliefs when theinformation set includes histories whose probabilities depend on thedecision maker's actions at that information set. Savage's theory views astate as a description of a scenario which is independent of the act. Incontrast, ``being at the second intersection'' is a state which is not indepen-dent from the action taken at the ®rst, and, consequently, at the secondintersection.
APPENDIX
Proof of Proposition 1. Consider an information set Xand hgX. Let
. . bbe a behavioral strategy and bh a denote the probability of choosing
. agAh ath. Since Gdoes not exhibit absentmindedness, the expected
. . payoff from bcan be written as  bh ag qg, where each
agAh. a
. . coef®cient gand gare independent of bh. Consider a)gAh whicha
.  . achieves the highest g. Setting bh a )s1 yields a payoff at least asa
high. The claim follows by repeating the argument for every information
set.
Suppose that Gexhibits absentmindedness. Then, there exist a ®nal
.  . history zand two distinct subhistories of z,h,aand h9,a9, such that h
and h9belong to the same information set and a/a9. Consider a payoff
. . function u9such that u9zs1 and u9zs0 for z/z. The claim
follows, since zcannot be reached by any pure strategy, and is reached
with positive probability by any behavioral strategy which assigns a positiveprobability to each action. Q.E.D.
. Proof of Proposition 2. Let
pbdenote the payoff that the decision
. maker obtains by playing b. Given an information set X, let ZX denote
. the set of ®nal histories having subhistories in X. Then, we can write pb
as
< pbs pz¬buz q pz buz . .  ..  ..
. . zgZrZX z gZX
. Since Gdoes not exhibit absentmindedness, a terminal history zinZX
<. <. <. has a unique subhistory hinX. Thus, pz b sph bpzh ,band
PICCIONE AND RUBINSTEIN 22
<. pz h 9,bs0 for h9/h,h9gX. Therefore,
<< pbs pz buz q ph b pz ¬h,buz . .  ..  .  ..
. hgX . zgZrZX z gZX
Suppose bis optimal and not time consistent. Then, there is an informa-
tion set Xand a behavioral strategy b9such that Xis reached with
positive probability under band
mhp z ¬h,buz - mhp z ¬h,b9uz , .  .. .  ..
hgXz gZh gXz gZ
where mis consistent with b. Then, by the de®nition of consistent beliefs,
pb- pz¬buz .  ..
. zgZrZX
q ph¬bp z ¬h,b9uz . .  .  .
hgX . zgZX
. Denote the right-hand side of the above inequality by p9. De®ne HX to
be the union of Xand set of histories having subhistories in X. Consider a
decision problem G9obtained from Gby ``splitting'' any information set Y
.  . . . containing histories in both HX and HyHX into YlHX and
 . . YlHyHX . De®ne bto be a strategy for G9which is identical to b9
atXand at any information sets after Xand identical to botherwise. By
construction, byields p9. Then, by Proposition 1, there exists a pure
strategy f9which yields in G9a payoff of at least p9. We now show that, by
suitably modifying f9, one can obtain a strategy fforGwhich yields the
same payoff as f9forG9. The construction of feasily follows if we show
that, for any information set YforGwhich has been split into Y9and Y0
inG9, it is not possible that both Y9and Y0are reached with positive
probability under f9. Suppose that h9gY9and h0gY0are both reached
. with positive probability. By construction, one and only one of exp h9and
.exp h0lists X. Denoting the longest history common subhistory of h9and
ÄÄ. h0byh,)implies that hfC. A contradiction is then obtained since f9
Ä is a pure strategy and assigns positive probability to at most one edge at h.
Thus, we can construct a strategy for Gwhich yields at least p9. This
contradicts the optimality of b.
Now suppose that bis time consistent. First notice that if fgD,n o
absentmindedness implies that bis optimal. Suppose now that fgCand
letDbe the set of information sets which contain some histories with no
subhistories in D. We ®rst show that XgDimplies that no history in X
has a subhistory in D. Suppose not and let hbe a history in XgDwith a
. subhistory h9in the information set X9. By construction, exp hlists X9
. and, by no absentmindedness, X/X9.)then yields a contradiction
DECISIONS AND IMPERFECT RECALL 23
since, by de®nition, Xcontains a history with no subhistory in D. Hence,
< pbs ph b pz ¬h,buz qconstant .  .  ..
XgDhgX . zgZX
<. and ph b is independent of bsince no action of the decision maker
precedes information sets in D. Since bis time consistent, bmaximizes
<ph b pz ¬h,buz .  .  .
hgX . zgZX
 <. for any XgDwhich is reached with positive probability i.e.,  ph bhgX
.)0 . The claim follows. Q.E.D.
Proof of Proposition 3. Let b\* be a strategy and suppose that there exist
a belief m\* consistent with b\*, an information set Xwhich is reached with
.  .  . positive probability, agAX for which b\*ha )0 for hgXand
. a9gAX such that
m\*hp z ¬h,a9,b\*uz .  . . .
hgXz gZ
) m\*hp z ¬h,a,b\*uz . .  . . .
hgXz gZ
We will show that b\* is not optimal. Let bbe a behavioral strategy whiche
. . . . .  . is identical to b\* except for bha sb\*ha yeand bha 9se e
.  . .b\*ha9qe. Let pbdenote the expected payoff of playing b. We claim
. . that for suf®ciently small positive e,pb)pb\*.e
. Note that pbis de®ned by a polynomial in the probabilities assigned
by the strategy bto all possible actions. Fix all the probabilities at the
levels assigned by b\*, except for the actions aand a9at the information
setX, and denote those probabilities by pand p9, respectively. We use
.  . the notation b\*Xto denote b\*h, for any hgX. We shall assume that
. .b\*Xa 9)0. The reader will easily see that this assumption keeps the
notation simple and does not affect the argument of the proof.
We obtain a polynomial
. .dz d9z dz.d9z.Vp ,p9s uzp p 9 pz¬b\*rb\*Xa b \*Xa 9 , .  . . .  . .  .
zgZ
. where dzis the number of times that the terminal history zincludes a
 . . play of aatXand similarly for d9z.
 .   . .  . .. It is enough to show that at the point p,p9sb\*Xa ,b\*Xa 9
. . we have dV p ,p9rdp-dV p ,p9rdp9.
.  . Since m\* is consistent with b\* we have that m\*hsph¬b\*r
PICCIONE AND RUBINSTEIN 24
. ph9¬b\* . Thus, it is suf®cient to verify that at the pointh9gX
  . .  . ..b\*Xa ,b\*Xa 9
dV p ,p9rdp9s d9zp z ¬b\*rb\*Xa 9uz .  . .  .  .  . .
zgZ
s ph¬b\* pz¬h,a9,b\*uz .  .  . .
hgXz gZ
and similarly for the partial derivative with respect to p. Q.E.D.
REFERENCES
. Alpern, S. 1988 . ``Games with Repeated Decisions,'' SIAM J .Control Optim .26, No. 2,
468]477.
. Binmore, K. 1992 . Fun and Games . Boston: Heath.
. Fluck, Z. 1994 . ``Games with Forgetfulness,'' mimeo.
. Geanakopolos, J. 1990 . ``Game Theory without Partitions, and Applications to Speculation
and Consensus,'' Santa-Fe Institute, DP. 90-018.
. Hendon, E., Jacobsen, J., and Sloth, B. 1993 . ``The One-Shot-Deviation Principle for
Sequential Rationality,'' Games Econ .Beh., to appear.
. Isbell, J. 1957 . ``Finitary Games,'' in Contributions to the Theory of Games III , pp. 79 ]96.
Princeton, NJ: Princeton Univ. Press.
. Kuhn, H. W. 1953 . ``Extensive Games and the Problem of Information,'' in Contributions to
the Theory of Games II . Princeton, NJ: Princeton Univ. Press.
. Osborne, M., and Rubinstein, A. 1994 . A Course in Game Theory , Cambridge, MA: MIT
Press.
. Rubinstein, A. 1986 . ``Finite Automata Play a Repeated Prisioner's Dilemma,'' J.Econ .
Theory 39,83]96.
. Rubinstein, A. 1991 . ``Comments on the Interpretation of Game Theory,'' Econometrica 59,
909]924.
. Strotz, R. H. 1956 . ``Myopia and Inconsistency in Dynamic Utility Maximization,'' Re¨.
Econ .Stud .23,165]180.