Provably Bounded-Optimal Agents

Journal of Arti/#0Ccial In telligence Researc h /2 /#28/1/9/9/5/#29 /5/7/5/-/6/0/9 Submitted /1/2/#2F/9/4/; published /5/#2F/9/5Pro v ably Bounded/-Optimal Agen tsStuart J/. Russell r ussell/@cs/.berkeley/.eduComputer Scienc e Division/, University of CaliforniaBerkeley/, CA /9/4/7/2/0/, USADevik a Subramanian devika/@cs/.cornell/.eduComputer Scienc e Dep artment/, Cornel l UniversityIthac a/, NY /1/4/8/5/3/, USAAbstractSince its inception/, arti/#0Ccial in telligence has relied up on a theoretical foundation cen/-tred around p erfe ct r ationality as the desired prop ert yo f i n telligen t systems/. W e argue/,as others ha v e done/, that this foundation is inadequate b ecause it imp oses fundamen tallyunsatis/#0Cable requiremen ts/. As a result/, there has arisen a wide gap b et w een theory andpractice in AI/, hindering progress in the /#0Celd/. W e prop ose instead a prop ert y called b ounde doptimality /. Roughly sp eaking/, an agen t is b ounded/-optimal if its program is a solution tothe constrained optimization problem presen ted b y its arc hitecture and the task en viron/-men t/. W e sho wh o w to construct agen ts with this prop ert y for a simple class of mac hinearc hitectures in a broad class of real/-time en vironmen ts/. W e illustrate these results usinga simple mo del of an automated mail sorting facilit y /. W e also de/#0Cne a w eak er prop ert y /,asymptotic b ounde d optimality /#28ABO/#29/, that generalizes the notion of optimalit y in classicalcomplexit y theory /. W e then construct universal ABO programs/, i/.e/./, programs that areABO no matter what real/-time constrain ts are applied/. Univ ersal ABO programs can b eused as building blo c ks for more complex systems/. W e conclude with a discussion of theprosp ects for b ounded optimalit y as a theoretical basis for AI/, and relate it to similar trendsin philosoph y /, economics/, and game theory /./1/. In tro ductionSince b efore the b eginning of arti/#0Ccial in telligence/, philosophers/, con trol theorists andeconomists ha v el o o k ed for a satisfactory de/#0Cnition of rational b eha viour/. This is needed tounderpin theories of ethics/, inductiv e learning/, reasoning/, optimal con trol/, decision/-making/,and economic mo delling/. Do yle /#28/1/9/8/3/#29 has prop osed that AI itself b e de/#0Cned as the com/-putational study of rational b eha viour/|e/#0Bectiv ely equating rational b eha viour with in telli/-gence/. The role of suc h de/#0Cnitions in AI is to ensure that theory and practice are correctlyaligned/. If w e de/#0Cne some prop ert y P /, then w e hop e to b e able to design a system thatpro v ably p ossesses prop ert y P /. Theory meets practice when our systems exhibit P in real/-it y /.F urthermore/, that they exhibit P in realit y should b e something that w e actually careab out/. In a sense/, the c hoice of what P to study determines the nature of the /#0Celd/.There are a n um b er of p ossible c hoices for P /:/#0F Perfe ct r ationality /: the classical notion of rationalit y in economics and philosoph y /.A p erfectly rational agen t acts at ev ery instan ti ns u c ha w a y as to maximize itsexp ected utilit y /, giv en the information it has acquired from the en vironmen t/. Sinceaction selection requires computation/, and computation tak es time/, p erfectly rationalagen ts do not exist for non/-trivial en vironmen ts/.c/#0D /1/9/9/5 AI Access F oundation and Morgan Kaufmann Publishers/. All righ ts reserv ed/.
R ussell /& Subramanian/#0F Calculative r ationality /: the notion of rationalit y studied in AI/. A calculativ ely rationalagen te v en tually returns what w ould ha v e b een the rational c hoice at the b eginning ofits delib eration/. There exist systems suc h as in/#0Duence diagram ev aluators that exhibitthis prop ert y for a decision/-theoretic de/#0Cnition of rational c hoice/, and systems suc has nonlinear planners that exhibit it for a logical de/#0Cnition of rational c hoice/. Thisis assumed to b e an in teresting prop ert y for a system to exhibit since it constitutesan /#5Cin/-principle/" capacit y to do the righ t thing/. Calculativ e rationalit y is of limitedv alue in practice/, b ecause the actual b eha viour exhibited b y suc h systems is absurdlyfar from b eing rational/; for example/, a calculativ ely rational c hess program will c ho osethe righ tm o v e/, but ma y tak e/1 /0
/5/0times to o long to do so/. As a result/, AI system/-builders often ignore theoretical dev elopmen ts/, b eing forced to rely on trial/-and/-errorengineering to ac hiev e their goals/. Ev en in simple domains suc ha sc hess/, there is littletheory for designing and analysing high/-p erformance programs/./#0F Metalevel r ationality /: a natural resp onse to the problems of calculativ e rationalit y /.A metalev el rational system optimizes o v er the ob ject/-lev el computations to b e p er/-formed in the service of selecting actions/. In other w ords/, for eac h decision it /#0Cndsthe optimal com bination of computation/-sequence/-plus/-action/, under the constrain tthat the action m ust b e selected b y the computation/. F ull metalev el rationalit yi sseldom useful b ecause the metalev el computations themselv es tak e time/, and the met/-alev el decision problem is often more di/#0Ecult than the ob ject/-lev el problem/. Simpleappro ximations to metalev el rationalit yh a v e pro v ed useful in practice/|for exam/-ple/, metalev el p olicies that limit lo ok ahead in c hess programs/|but these engineeringexp edien ts merely serv e to illustrate the lac k of a theoretical basis for agen t design/./#0F Bounde d optimality /: a b ounded optimal agen t b eha v es as w ell as p ossible giv en itscomputational resources/. Bounded optimalit y sp eci/#0Ces optimal pr o gr ams rather thanoptimal actions or optimal c omputation se quenc es /. Only b y the former approac hcan w ea v oid placing constrain ts on in telligen t agen ts that cannot b e met b ya n yprogram/. Actions and computations are/, after all/, generated b y programs/, and it iso v er programs that designers ha v e con trol/.W e mak e three claims/:/1/. A system that exhibits b ounded optimalit y is desirable in realit y /./2/. It is p ossible to construct pro v ably b ounded optimal programs/./3/. Arti/#0Ccial in telligence can b e usefully c haracterized as the study of b ounded optimalit y /,particularly in the con text of complex task en vironmen ts and reasonably p o w erfulcomputing devices/.The /#0Crst claim is unlik ely to b e con tro v ersial/. This pap er supp orts the second claim indetail/. The third claim ma y /,o rm a y not/, stand the test of time/.W e b egin in section /2 with a necessarily brief discussion of the relationship b et w eenb ounded optimalit y and earlier notions of rationalit y /.W e note in particular that some im/-p ortan t distinctions can b e missed without precise de/#0Cnitions of terms/. Th us in section /3 w epro vide formal de/#0Cnitions of agen ts/, their programs/, their b eha viour and their rationalit y /./5/7/6
Pr o v abl y bounded/-optimal a gentsT ogether with formal descriptions of task en vironmen ts/, these elemen ts allo wu st o p r o v ethat a giv en agen t exhibits b ounded optimalit y /. Section /4 examines a class of agen t arc hi/-tectures for whic h the problem of generating b ounded optimal con/#0Cgurations is e/#0Ecien tlysoluble/. The solution in v olv es a class of in teresting and practically relev an t optimizationproblems that do not app ear to ha v e b een addressed in the sc heduling literature/. W e il/-lustrate the results b y sho wing ho w the throughput of an automated mail/-sorting facilit ymigh t b e impro v ed/. Section /5 initiates a discussion of ho w b ounded optimal con/#0Cgurationsmigh t b e learned from exp erience in an en vironmen t/. In section /6/, w e de/#0Cne a w eak er prop/-ert y /, asymptotic b ounde d optimality /#28ABO/#29/, that ma y b e more robust and tractable thanthe strict v ersion of b ounded optimalit y /. In particular/, w e can construct universal ABOprograms/. A program is univ ersally ABO if it is ABO regardless of the sp eci/#0Cc form oftime dep endence of the utilit y function/.
/1Univ ersal ABO programs can therefore b e used asbuilding blo c ks for more complex systems/. W e conclude with an assessmen t of the prosp ectsfor further dev elopmen t of this approac h to arti/#0Ccial in telligence/./2/. Historical P ersp ectiv eThe classical idea of p erfect rationalit y /, whic h dev elop ed from Aristotle/'s theories of ethics/,w ork b y Arnauld and others on c hoice under uncertain t y /, and Mill/'s utilitarianism/, w as puton a formal fo oting in de cision the ory b y Ramsey /#28/1/9/3/1/#29 and v onNeumann and Morgernstern/#28/1/9/4/7/#29/. It stipulates that a rational agen t alw a ys act so as to maximize its exp e cte d utilit y /.The exp ectation is tak en according to the agen t/'s o wn b eliefs/; th us/, p erfect rationalit yd o e snot require omniscience/.In arti/#0Ccial in telligence/, the logical de/#0Cnition of rationalit y /, kno wn in philosoph y as the/#5Cpractical syllogism/"/, w as put forw ard b y McCarth y /#28/1/9/5/8/#29/, and reiterated strongly b yNew ell /#28/1/9/8/1/#29/. Under this de/#0Cnition/, an agen t should tak ea n y action that it b eliev es isguaran teed to ac hiev ea n y of its goals/. If AI can b e said to ha v e had a theoretical foun/-dation/, then this de/#0Cnition of rationalit y has pro vided it/. McCarth y b eliev ed/, probablycorrectly /, that in the early stages of the /#0Celd it w as imp ortan t to concen trate on /#5Cepistemo/-logical adequacy/" b efore /#5Cheuristic adequacy/" /| that is/, capabilit y in principle rather thanin practice/. The metho dology that has resulted in v olv es designing programs that exhibitcalculativ e rationalit y /, and then using v arious sp eedup tec hniques and appro ximations inthe hop e of getting as close as p ossible to p erfect rationalit y /. Our b elief/, alb eit unpro v en/, isthat the simple agen t designs that ful/#0Cll the sp eci/#0Ccation of calculativ e rationalit ym a y notpro vide go o d starting p oin ts from whic h to approac h b ounded optimalit y /. Moreo v er/, a the/-oretical foundation based on calculativ e rationalit y cannot pro vide the necessary guidancein the searc h/.It is not clear that AI w ould ha v ee m bark ed on the quest for calculativ e rationalit y had itnot b een op erating in the halcy on da ys b efore formal in tractabilit y results w ere disco v ered/.One resp onse to the sp ectre of complexit y has b een to rule it out of b ounds/. Lev esque andBrac hman /#28/1/9/8/7/#29 suggest limiting the complexit y of the en vironmen t so that calculativ e andp erfect rationalit y coincide/. Do yle and P atil /#28/1/9/9/1/#29 argue strongly against this p osition/./1/. This usage of the term /#5Cuniv ersal/" deriv es from its use in the sc heduling of randomized algorithms b yLub y /, Sinclair and Zuc k erman /#28/1/9/9/3/#29/./5/7/7
R ussell /& SubramanianEconomists ha v e used p erfect rationalit y as an abstract mo del of economic en tities/, forthe purp oses of economic forecasting and designing mark et mec hanisms/. This mak es itp ossible to pro v e theorems ab out the prop erties of mark ets in equilibrium/. Unfortunately /,as Simon /#28/1/9/8/2/#29 p oin ted out/, real economic en tities ha v e limited time and limited p o w ersof delib eration/. He prop osed the study of b ounde dr ationality /,i n v estigating /#5C /:/:/: the shap eof a system in whic h e/#0Bectiv eness in computation is one of the most imp ortan tw eap onsof surviv al/./" Simon/'s w ork fo cussed mainly on satis/#0Ccing designs/, whic h delib erate un tilreac hing some solution satisfying a preset /#5Caspiration lev el/./" The results ha v e descrip/-tiv ev alue for mo delling v arious actual en tities and p olicies/, but no general prescriptiv eframew ork for b ounded rationalit yw as dev elop ed/. Although it pro v ed p ossible to calculateoptimal aspiration lev els for certain problems/, no structural v ariation w as allo w ed in theagen t design/.In the theory of games/, b ounds on the complexit y of pla y ers ha v e b ecome a topic ofin tense in terest/. F or example/, it is a troubling fact that defection is the only equilibriumstrategy for un b ounded agen ts pla ying a /#0Cxed n um b er of rounds of the Prisoners/' Dilemmagame/. Neyman/'s theorem /#28Neyman/, /1/9/8/5/#29/, recen tly pro v ed b yP apadimitriou and Y an/-nak akis /#28/1/9/9/4/#29/, sho ws that an essen tially co op erativ e equilibrium exists if eac h agen ti s a/#0Cnite automaton with a n um b er of states that is less than exp onen tial in the n um be r o frounds/. This is essen tially a b ounded optimalit y result/, where the b ound is on space ratherthan on sp eed of computation/. This t yp e of result is made p ossible b y a shift from theproblem of selecting actions to the problem of selecting pr o gr ams /.I/. J/. Go o d /#28/1/9/7/1/#29 distinguished b et w een p erfect or /#5Ct yp e I/" rationalit y /, and metalev elor /#5Ct yp e I I/" rationalit y /. He de/#0Cnes this as /#5Cthe maximization of exp ected utilit y taking intoac c ount delib er ation c osts /./" Simon /#28/1/9/7/6/#29 also sa ys/: /#5CThe global optimization problem isto /#0Cnd the least/-cost or b est/-return decision/, net of computational costs/./" Although t yp e I Irationalit y seems to b e a step in the righ t direction/, it is not en tirely clear whether it can b emade precise in a w a y that resp ects the desirable in tuition that computation is imp ortan t/.W e will try one in terpretation/, although there ma y b e others/.
/2The k ey issue is the spaceo v er whic h the /#5Cmaximization/" or /#5Coptimization/" o ccurs/. Both Go o d and Simon seem tob e referring to the space of p ossible delib erations asso ciated with a p articular de cision /.Conceptually /, there is an /#5Cob ject/-lev el mac hine/" that executes a sequence of computationsunder the con trol of a /#5Cmeta/-lev el mac hine/./" The outcome of the sequence is the selection ofan external action/. An agen t exhibits t yp e I I rationalit y if at the end of its delib eration andsubsequen t action/, its utilit y is maximized compared to all p ossible delib erate/#2Fact pairs inwhic h it could ha v e engaged/. F or example/, Go o d discusses one p ossible application of t yp eI I rationalit yi n c hess programs/. In this case/, the ob ject/-lev el steps are no de expansions inthe game tree/, follo w ed b y bac king up of leaf no de ev aluations to sho w the b est mo v e/. F orsimplicit yw e will assume a p er/-mo v e time limit/. Then a t yp e I I rational agen t will executewhic hev er sequence of no de expansions c ho oses the b est mo v e/, of all those that /#0Cnish b efore/2/. F or example/, it is conceiv able that Go o d and Simon really in tended to refer to /#0Cnding an agen t designthat minimizes delib eration costs in general/. All their discussions/, ho w ev er/, seem to b e couc hed in termsof /#0Cnding the righ t delib eration for eac h decision/. Th us/, t yp e I I or metalev el rationalit y coincides withb ounded optimalit y if the b ounded optimal agen t is b eing designed for a single decision in a singlesituation/./5/7/8
Pr o v abl y bounded/-optimal a gentsthe time limit/.
/3Unfortunately /, the computations required in the /#5Cmetalev el mac hine/" toselect the ob ject/-lev el delib eration ma y b e extremely exp ensiv e/. Go o d actually prop oses afairly simple /#28and nearly practical/#29 metalev el decision pro cedure for c hess/, but it is far fromoptimal/. It is hard to see ho wa t yp e I I rational agen t could justify executing a sub optimalob ject/-lev el computation sequence if w e limit the scop e of the optimization problem to asingle decision/. The di/#0Ecult y can only b e resolv ed b y thinking ab out the design of theagen t program/, whic h generates an un b ounded set of p ossible delib erations in resp onse toan un b ounded set of circumstances that ma y arise during the life of the agen t/.Philosoph y has also seen a gradual ev olution in the de/#0Cnition of rationalit y /. There hasb een a shift from consideration of act utilitarianism /| the rationalit y of individual acts /| torule utilitarianism /, or the rationalit y of general p olicies for acting/. This shift has b een causedb y di/#0Eculties with individual v ersus so cietal rationalit y /, rather than an y consideration ofthe di/#0Ecult y of computing rational acts/. Some consideration has b een giv en more recen tlyto the tractabilit y of general moral p olicies/, with a view to making them understandableand usable b y p ersons of a v erage in telligence /#28Brandt/, /1/9/5/3/#29/. Cherniak /#28/1/9/8/6/#29 has suggesteda de/#0Cnition of /#5Cminimal rationalit y/"/, sp ecifying lo w er b ounds on the reasoning p o w ers ofan y rational agen t/, instead of upp er b ounds/. A philosophical prop osal generally consisten twith the notion of b ounded optimalit y can b e found in Dennett/'s /#5CMoral First Aid Man ual/"/#28/1/9/8/6/#29/. Dennett explicitly discusses the idea of reac hing equilibrium within the space ofdecision pro cedures/. He uses as an example the PhD admissions pro cedure of a philosoph ydepartmen t/. He concludes/, as do w e/, that the b est pro cedure ma y b e neither elegan t norilluminating/. The existence of suc h a pro cedure/, and the pro cess of reac hing it/, are themain p oin ts of in terest/.Man y researc hers in AI/, some of whose w ork is discussed b elo w/, ha v ew ork ed on theproblem of designing agen ts with limited computational resources/. The /1/9/8/9 AAAI Sym/-p osium on AI and Limited Rationalit y/#28 F ehling /& Russell/, /1/9/8/9/#29 con tains an in terestingv ariet yo fw ork on the topic/. Muc h of this w ork is concerned with metalev el rationalit y /.Metareasoning /| reasoning ab out reasoning /| is an imp ortan t tec hnique in this area/,since it enables an agen t to con trol its delib erations according to their costs and b ene/#0Cts/.Com bined with the idea of anytime /#28Dean /& Bo ddy /, /1/9/8/8/#29 or /#0Dexible algorithms /#28Horvitz/,/1/9/8/7/#29/, that return b etter results as time go es b y /, a simple form of metareasoning allo wsan agen t to b eha v ew ell in a real/-time en vironmen t/. A simple example is pro vided b yiterativ e/-deep ening algorithms used in game/-pla ying/. Breese and F ehling /#28/1/9/9/0/#29 apply sim/-ilar ideas to con trolling m ultiple decision pro cedures/. Russell and W efald /#28/1/9/8/9/#29 giv eageneral metho d for precompiling certain asp ects of metareasoning so that a system can e/#0E/-cien tly estimate the e/#0Bects of individual computations on its in ten tions/, giving /#0Cne/-grainedcon trol of reasoning/. These tec hniques can all b e seen as appro ximating metalev el rational/-it y/; they pro vide useful insigh ts in to the general problem of con trol of reasoning/, but thereis no reason to supp ose that the appro ximations used are optimal in an y sense/.The in tuitiv e notion of b ounded optimalit y seems to ha v e b ecome curren t in the AIcomm unit y in the mid/-/1/9/8/0/'s/. Horvitz /#28/1/9/8/7/#29 uses the term b ounde d optimality to referto /#5Cthe optimization of computational utilit y giv en a set of assumptions ab out exp ected/3/. One w ould imagine that in most cases the mo v e selected will b e the same mo v e selected b ya T yp eI agen t/, but this is in a sense /#5Cacciden tal/" b ecause further delib eration migh t cause the program toabandon it/./5/7/9
R ussell /& Subramanianproblems and constrain ts in reasoning resources/./" Russell and W efald /#28/1/9/9/1/#29 sa y that anagen t exhibits b ounded optimalit y for a giv en task en vironmen t /#5Cif its program is a solutionto the constrained optimization problem presen ted b y its arc hitecture/./" Recen tw ork b yEtzioni /#28/1/9/8/9/#29 and Russell and Zilb erstein /#28/1/9/9/1/#29 can b e seen as optimizing o v er a w ell/-de/#0Cned set of agen t designs/, thereb y making the notion of b ounded optimalit y more precise/.In the next section/, w e build a suitable set of general de/#0Cnitions from the ground up/, sothat w e can b egin to demonstrate examples of pro v ably b ounded optimal agen ts/./3/. Agen ts/, Arc hitectures and ProgramsIn tuitiv ely /,a n agent is just a ph ysical en tit y that w e wish to view in terms of its p er c eptionsand actions /. What coun ts in the /#0Crst instance is what it do es/, not necessarily what it thinks/,or ev en whether it thinks at all/. This initial refusal to consider further constrain ts on thein ternal w orkings of the agen t /#28suc h as that it should reason logically /, for example/#29 helps inthree w a ys/: /#0Crst/, it allo ws us to view suc h /#5Ccognitiv e faculties/" as planning and reasoningas o ccurring in the servic eo f /#0Cnding the righ t thing to do/; second/, it mak es ro om for thoseamong us /#28Agre /& Chapman/, /1/9/8/7/; Bro oks/, /1/9/8/6/#29 who tak e the p osition that systems cando the righ t thing without suc h cognitiv e faculties/; third/, it allo ws more freedom to considerv arious sp eci/#0Ccations/, b oundaries and in terconnections of subsystems/.W e b egin b y de/#0Cning agen ts and en vironmen ts in terms of the actions and p erceptsthat they exc hange/, and the sequence of states they go through/. The agen t is describ edb ya n agent function from p ercept sequences to actions/. This treatmen t is fairly standard/#28see/, e/.g/./, Genesereth /& Nilsson/, /1/9/8/7/#29/. W e then go /#5Cinside/" the agen t t ol o o ka t t h e agentpr o gr am that generates its actions/, and de/#0Cne the /#5Cimplemen tation/" relationship b et w eena program and the corresp onding agen t function/. W e consider p erformance measures onagen ts/, and the problem of designing agen ts to optimize the p erformance measure/./3/./1 Sp ecifying agen ts and en vironmen tsAn agen t can b e describ ed abstractly as a mapping /#28the agent function /#29 from p erceptsequences to actions/. Let O b e the set of p ercepts that the agen t can receiv ea ta n y instan t/,and A b e the set of p ossible actions the agen t can carry out in the external w orld/. Sincew e are in terested in the b eha viour of the agen to v er time/, w ei n tro duce a set of time p oin tsor instan ts/, T /. The set T is totally ordered b y the /#3C relation with a unique least elemen t/.Without loss of generalit y /,w e let T b e the set of non/-negativ ei n tegers/.The p ercept history of an agen t is a sequence of p ercepts indexed b y time/. W e de/#0Cnethe set of p ercept histories to b e O
T/= f O
T/: T /! O g /. The pre/#0Cx of a history O
T/2 O
Ttill time t is denoted O
tand is the pro jection of O
Ton /#5B/0 /:/:t /#5D/. W e can de/#0Cne the set ofp ercept history pre/#0Cxes as O
t/= f O
tj t /2 T /;O
T/2 O
Tg /. Similarly /,w e de/#0Cne the set ofaction histories A
T/= f A
T/: T /! A g /. The set of action history pre/#0Cxes is A
t/, de/#0Cned asthe set of pro jections A
tof histories A
T/2 A
T/.De/#0Cnition /1 A gent function/: a mappingf /: O
t/! A/5/8/0
Pr o v abl y bounded/-optimal a gentswher eA
T/#28 t /#29/= f /#28 O
t/#29Note that the agen t function is an en tirely abstract en tit y /, unlik e the agen t program thatimplemen ts it/. Note also that the /#5Coutput/" of the agen t function for a giv en p ercept sequencema yb e an ull action/, for example if the agen t is still thinking ab out what to do/. The agen tfunction sp eci/#0Ces what the agen t d o e sa te a c h time step/. This is crucial to the distinctionbe t w een p erfect rationalit y and calculativ e rationalit y /.Agen ts liv ei ne n vironmen ts/. The states of an en vironmen t E are dra wn from a set X /.The set of p ossible state tra jectories is de/#0Cned as X
T/= f X
T/: T /! X g /. The agen t do es notnecessarily ha v e full access to the curren t state X
T/#28 t /#29/, but the p ercept receiv ed b y the agen tdo es dep end on the curren t state through the p er c eptual /#0Cltering function fp
/. The e/#0Bectsof the agen t/'s actions are represen ted b y the en vironmen t/'s tr ansition function fe
/, whic hsp eci/#0Ces the next state giv en the curren t state and the agen t/'s action/. An en vironmen ti stherefore de/#0Cned as follo ws/:De/#0Cnition /2 Envir onment E /: a set of states X with a distinguishe d initial state X/0
/,atr ansition function fe
and a p er c eptual /#0Clter function fp
such thatX
T/#28/0/#29 /= X/0X
T/#28 t /+/1 /#29 /= fe
/#28 A
T/#28 t /#29 /;X
T/#28 t /#29/#29O
T/#28 t /#29 /= fp
/#28 X
T/#28 t /#29/#29The state history X
Tis th us determined b y the en vironmen t and the agen t function/. W euse the notation e/#0Be cts /#28 f/; E /#29 to denote the state history generated b y an agen t functionf op erating in an en vironmen t E /. W e will also use the notation /#5B E/; A
t/#5D to denote thestate history generated b y applying the action sequence A
tstarting in the initial state ofen vironmen t E /.Notice that the en vironmen t is discrete and deterministic in this form ulation/. W e canextend the de/#0Cnitions to co v er non/-deterministic and con tin uous en vironmen ts/, but at thecost of additional complexit y in the exp osition/. None of our results dep end in a signi/#0Ccan tw a y on discreteness or determinism/./3/./2 Sp ecifying agen t implemen tationsW e will consider a ph ysical agen t as consisting of an arc hitecture and a program/. Thearc hitecture is resp onsible for in terfacing b et w een the program and the en vironmen t/, andfor running the program itself/. With eac h arc hitecture M /,w e asso ciate a /#0Cnite programminglanguage LM
/, whic h is just the set of all programs runnable b y the arc hitecture/. An agentpr o gr am is a program l /2LM
that tak es a p ercept as input and has an in ternal state dra wnfrom a set I with initial state i/0
/. /#28The initial in ternal state dep ends on the program l /,but w e will usually suppress this argumen t/./#29 The set of p ossible in ternal state histories isI
T/= f I
T/: T /! I g /. The pre/#0Cx of an in ternal state history I
T/2 I
Ttill time t is denoted I
tand is the pro jection of I
Ton /#5B/0 /:/:t /#5D/./5/8/1
R ussell /& SubramanianDe/#0Cnition /3 A na r chite ctur e M is a /#0Cxe d interpr eter for an agent pr o gr am that runs thepr o gr am for a single time step/, up dating its internal state and gener ating an action/:M /: LM
/#02 I /#02 O /! I /#02 Awher eh I
T/#28 t /+/1 /#29 /;A
T/#28 t /#29 i /= M /#28 l/; I
T/#28 t /#29 /;O
T/#28 t /#29/#29Th us/, the arc hitecture generates a stream of actions according to the dictates of the program/.Because of the ph ysical prop erties of the arc hitecture/, running the program for a singletime step results in the execution of only a /#0Cnite n um b er of instructions/. The program ma yoften fail to reac h a /#5Cdecision/" in that time step/, and as a result the action pro duced b y thearc hitecture ma yb e n ull /#28or the same as the previous action/, dep ending on the programdesign/#29/./3/./3 Relating agen t sp eci/#0Ccations and implemen tationsW e can no w relate agen t programs to the corresp onding agen t functions/. W e will sa y that anagen t program l running on a mac hine M implements the agen t function Ag ent /#28 l/; M /#29/. Theagen t function is constructed in the follo wing de/#0Cnition b y sp ecifying the action sequencespro duced b y l running on M for all p ossible p ercept sequences/. Note the imp ortance of the/#5CMark o vian/" construction using the in ternal state of the agen t to ensure that actions canonly b e based on the past/, not the future/.De/#0Cnition /4 Ap r o gr am l running on M implements the agent function f /= Ag ent /#28 l/; M /#29 /,de/#0Cne d as fol lows/. F or any envir onment E /=/#28 X /;fe
/;fp
/#29 /, f /#28 O
t/#29/= A
T/#28 t /#29 wher eh I
T/#28 t /+/1 /#29 /;A
T/#28 t /#29 i /= M /#28 l/; I
T/#28 t /#29 /;O
T/#28 t /#29/#29O
T/#28 t /#29 /= fp
/#28 X
T/#28 t /#29/#29X
T/#28 t /+/1 /#29 /= fe
/#28 A
T/#28 t /#29 /;X
T/#28 t /#29/#29X
T/#28/0/#29 /= X/0I
T/#28/0/#29 /= i/0Although ev ery program l induces a corresp onding agen t function Ag ent /#28 l/; M /#29/, theaction that follo ws a giv en p ercept is not necessarily the agen t/'s /#5Cresp onse/" to that p ercept/;b ecause of the dela y incurred b y delib eration/, it ma y only re/#0Dect p ercepts o ccurring m uc hearlier in the sequence/. F urthermore/, it is not p ossible to map ev ery agen t function to animplemen tation l /2LM
/. W e can de/#0Cne a subset of the set of agen t functions f that areimplemen table on a giv en arc hitecture M and language LM
/:F easibl e /#28 M /#29/= f f j/9 l /2LM
/;f /= Ag ent /#28 l/; M /#29 gF easibilit y is related to/, but clearly distinct from/, the notion of computabilit y /. Computabil/-it y refers to the existence of a program that eventual ly returns the output sp eci/#0Ced b ya function/, whereas feasibilit y refers to the pro duction of the output at the appropriatep oin t in time/. The set of feasible agen t functions is therefore m uc h smaller than the set ofcomputable agen t functions/./5/8/2
Pr o v abl y bounded/-optimal a gents/3/./4 P erformance measures for agen tsT oe v aluate an agen t/'s p erformance in the w orld/, w e de/#0Cne a real/-v alued utilit y function Uon state histories/:U /: X
T/!/#3CThe utilit y function should b e seen as external to the agen t and its en vironmen t/. It de/#0Cnesthe problem to b e solv ed b y the designer of the agen t/. Some agen t designs ma y incorp oratean explicit represen tation of the utilit y function/, but this is b y no means required/. W e willuse the term task envir onment to denote the com bination of an en vironmen t and a utilit yfunction/.Recall that the agen t/'s actions driv e the en vironmen t E through a particular sequenceof states in accordance with the function e/#0Be cts /#28 f/; E /#29/. W e can de/#0Cne the v alue of an agen tfunction f in the en vironmen t E as the utilit y of the state history it generates/:V /#28 f/; E /#29/= U /#28 e/#0Be cts /#28 f/; E /#29/#29If the designer has a set E of en vironmen ts with a probabilit y distribution p o v er them/,instead of a single en vironmen t E /, then the v alue of the agen ti n E is de/#0Cned as theexp ected v alue o v er the elemen ts of E /. By a sligh t abuse of notation/,V /#28 f/; E /#29/=
XE /2 E
p /#28 E /#29 V /#28 f/; E /#29W e can assign a v alue V /#28 l/; M /; E /#29 to a program l executed b y the arc hitecture M in theen vironmen t E simply b y lo oking at the e/#0Bect of the agen t function implemen ted b y theprogram/:V /#28 l/; M /; E /#29/= V /#28 Ag ent /#28 l/; M /#29 /;E /#29/= U /#28 e/#0Be cts /#28 Ag ent /#28 l/; M /#29 /;E /#29/#29As ab o v e/, w e can extend this to a set of p ossible en vironmen ts as follo ws/:V /#28 l/; M /; E /#29/=
XE /2 E
p /#28 E /#29 V /#28 l/; M /; E /#29/3/./5 P erfect rationalit y and b ounded optimalit yAs discussed in Section /2/, a p erfectly rational agen t selects the action that maximizes itsexp ected utilit y /, giv en the p ercepts so far/. In our framew ork/, this amoun ts to an agen tfunction that maximizes V /#28 f/; E /#29o v er all p ossible agen t functions/.De/#0Cnition /5 Ap erfe ctly r ational agent for a set E of envir onments has an agent functionfopt
such thatfopt
/= argmaxf
/#28 V /#28 f/; E /#29/#29This de/#0Cnition is a p ersuasiv e sp eci/#0Ccation of an optimal agen t function for a giv enset of en vironmen ts/, and underlies sev eral recen t pro jects in in telligen t agen t design /#28Dean/&W ellman/,/1/9/9/1/; Do yle/, /1/9/8/8/; Hansson /& Ma y er/, /1/9/8/9/#29/. A direct implemen tation of thissp eci/#0Ccation/, whic h ignores the dela y incurred b y delib eration/, do es not yield a reasonable/5/8/3
R ussell /& Subramaniansolution to our problem /#7B the calculation of exp ected utilities tak es time for an y real agen t/.In terms of our simple formal description of agen ts in tro duced ab o v e/, it is easy to see wherethe di/#0Ecult y has arisen/. In designing the agen t program/, logicists and decision theoristsha v e concen trated on sp ecifying an optimal agen t function fopt
in order to guaran tee theselection of the b est action history /. The function fopt
is indep enden t of the arc hitecture M /.Unfortunately /, no real program in LM
implemen ts this function in a non/-trivial en vironmen t/,b ecause optimal actions cannot usually b e computed b efore the next p ercept arriv es/. Thatis/, quite frequen tly /, fopt
/6/2 F easibl e /#28 M /#29/.Supp ose the en vironmen t consists of games of c hess under tournamen t rules against somep opulation of h uman grandmasters/, and supp ose M is some standard p ersonal computer/.Then fopt
describ es an agen t that alw a ys pla ys in suc haw a y as to maximize its totalexp ected p oin ts against the opp osition/, where the maximization is o v er the moves it mak es/.W e claim that no p ossible program can pla y this w a y /. It is quite p ossible/, using depth/-/#0Crstalpha/-b eta searc h to termination/, to execute the program that c ho oses /#28sa y/#29 the optimalminimax mo v e in eac h situation/, but the agen t function induced b y this program is not thesame as fopt
/. In particular/, it ignores suc h p ercepts as the dropping of its /#0Dag indicating aloss on time/.The trouble with the p erfect rationalit y de/#0Cnition arose b ecause of unconstrained op/-timization o v er the space of f /'s in the determination of fopt
/, without regard to feasibilit y /./#28Similarly /, metalev el rationalit y assumes unconstrained optimization o v er the space of de/-lib erations/./#29 T o escap e this quandary /,w e prop ose a mac hine/-dep enden t standard of ratio/-nalit y /, in whic hw e maximize V o v er the implemen table set of agen t functions F easibl e /#28 M /#29/.That is/, w e imp ose optimalit y constrain ts on pr o gr ams rather than on agent functions ordelib er ations /.De/#0Cnition /6 Ab ounde d/-optimal agent with ar chite ctur e M for a set E of envir onmentshas an agent pr o gr am lopt
such thatlopt
/= argmaxl /2LM
V /#28 l/; M /; E /#29W e can see immediately that this sp eci/#0Ccation a v oids the most ob vious problems withT yp e I and T yp e I I rationalit y /. Consider our c hess example/, and supp ose the computer hasa total program memory of /8 megab ytes/. Then there are /2
/2
/2/6p ossible programs that canb e represen ted in the mac hine/, of whic ham uc h smaller n um b er pla y legal c hess/. Undertournamen t conditions/, one or more of these programs will ha v e the b est exp ected p erfor/-mance/. Eac h is a suitable candidate for lopt
/.T h us b ounded optimalit y is/, b y de/#0Cnition/, afeasible sp eci/#0Ccation/; moreo v er/, a program that ac hiev es it is highly desirable/. W e are noty et ready to announce the iden tit yo f lopt
for c hess on an eigh t/-megab yte PC/, so w e willb egin with a more restricted problem/./4/. Pro v ably Bounded/-Optimal Agen tsIn order to construct a pro v ably b ounded optimal agen t/, w em ust carry out the follo wingsteps/:/#0F Sp ecify the prop erties of the en vironmen t in whic h actions will b e tak en/, and theutilit y function on the b eha viours/./5/8/4
Pr o v abl y bounded/-optimal a gents/#0F Sp ecify a class of mac hines on whic h programs are to b e run/./#0F Prop ose a construction metho d/./#0F Pro v e that the construction metho d succeeds in building b ounded optimal agen ts/.The metho dology is similar to the formal analysis used in the /#0Celd of optimal con trol/, whic hstudies the design of c ontr ol lers /#28agen ts/#29 for plants /#28en vironmen ts/#29/. In optimal con troltheory /, a con troller is view ed as an essen tially instan taneous implemen tation of an optimalagen t function/. In con trast/, w e fo cus on the computation time required b y the agen t/, andthe relation b et w een computation time and the dynamics of the en vironmen t/./4/./1 Episo dic/, real/-time task en vironmen tsIn this section/, w e will consider a restricted class of task en vironmen ts whic hw e call episo dicen vironmen ts/. In an episo dic task en vironmen t/, the state history generated b y the actions ofthe agen t can b e considered as divided in to a series of episo des /, eac h of whic h is terminatedb y an action/. Let A/?
/#12 A b e a distinguished set of actions that terminate an episo de/.The utilit y of the complete history is giv en b y the sum of the utilities of eac h episo de/,whic h is determined in turn b y the state sequence/. After eac h A /2 A/?
/, the en vironmen t/#5Cresets/" to a state c hosen at random from a stationary probabilit y distribution Pinit
/. Inorder to include the e/#0Bects of the c hoice of A in the utilit y of the episo de/, w e notionallydivide the en vironmen t state in to a /#5Ccon/#0Cguration/" part and a /#5Cv alue/" part/, suc h thatthe con/#0Cguration part determines the state transitions while the v alue part determines theutilit y of a state sequence/. Actions in A/?
reset the con/#0Cguration part/, while their /#5Cv alue/"is recorded in the v alue part/. These restrictions mean that eac h episo de can b e treated asa separate decision problem/, and translate in to the follo wing prop ert y/: if agen t program l/1has higher exp ected utilit y on individual episo des than agen t l/2
/, it will ha v e higher exp ectedutilit y in the corresp onding episo dic task en vironmen t/.A real/-time task en vironmen t is one in whic h the utilit y of an action dep ends on thetime at whic h it is executed/. Usually /, this dep endence will b e su/#0Ecien tly strong to mak ecalculativ e rationalit y an unacceptably bad appro ximation to p erfect rationalit y /.An automated mail sorter
/4pro vides an illustrativ e example of an episo dic task en viron/-men t /#28see Figure /1/#29/. Suc h a mac hine scans handwritten or prin ted addresses /#28zip co des/#29 onmail pieces and dispatc hes them to appropriate bins/. Eac h episo de starts with the arriv al ofa new mail piece and terminates with the execution of the ph ysical action recommended b ythe sorter/: routing of the piece to a sp eci/#0Cc bin/. The /#5Ccon/#0Cguration part/" of the en viron/-men t corresp onds to the letter feeder side/, whic h pro vides a new/, randomly selected letterafter the previous letter is sorted/. The /#5Cv alue part/" of the state corresp onds to the state ofthe receiving bins/, whic h determines the utilit y of the pro cess/. The aim is to maximize theaccuracy of sorting while minimizing the reject p ercen tage and a v oiding jams/. A jam o ccursif the curren t piece is not routed to the appropriate bin/, or rejected/, b efore the arriv al ofthe next piece/.W en o w pro vide formal de/#0Cnitions for three v arieties of real/-time task en vironmen ts/:/#0Cxed deadlines/, /#0Cxed time cost and sto c hastic deadlines/./4/. See /#28Sac kinger et al/. /1/9/9/2/; Boser et al/. /1/9/9/2/#29 for details of an actual system/. The application w assuggested to us b y Bernhard Boser after an early presen tation of our w ork at the /1/9/9/2 NEC Symp osium/./5/8/5
R ussell /& Subramanian
sacks of mailcamera
rejectzipcode
bucketsFigure /1/: An automated mail/-sorting facilit y pro vides a simple example of an episo dic/,real/-time task en vironmen t/./4/./1/./1 Fixed deadlinesThe simplest and most commonly studied kind of real/-time task en vironmen t con tains adeadline at a kno wn time/. In most w ork on real/-time systems/, suc h deadlines are describ edinformally and systems are built to meet the deadline/. Here/, w e need a formal sp eci/#0Ccationin order to connect the description of the deadline to the prop erties of agen ts running indeadline task en vironmen ts/. One migh t think that deadlines are part of the en vironmen tdescription/, but in fact they are mainly realized as constrain ts on the utilit y function/. Onecan see this b y considering the opp osite of a deadline /| the /#5Cstarter/'s pistol/./" The t w oare distinguished b y di/#0Bering constrain ts on the utilities of acting b efore or after a sp eci/#0Cctime/.De/#0Cnition /7 Fixe dd e ad line/: The task envir onment h E/; U i has a /#0Cxe dd e ad line at time tdif the fol lowing c onditions hold/./#0F T aking an action in A/?
at any time b efor e the de ad line r esults in the same utility/:U /#28/#5B E/; A
t/1
/#5D/#29 /= U /#28/#5B E/; A
/#28 td
/, /1/#29/2
/#01 A
T/1
/#28 t /#29/#5D/#29wher e/#5C /#01 /" denotes se quenc ec onc atenation/, t /#14 td
/, A
T/1
/#28 t /#29 /2 A/?
/, and A
/#28 t /, /1/#29/1
and A
/#28 td
/, /1/#29/2c ontain no action in A/?
/./#0F A ctions taken after td
have no e/#0Be ct on utility/:U /#28/#5B E/; A
t/1
/#5D/#29 /#14 U /#28/#5B E/; A
t/2
/#5D/#29 if U /#28/#5B E/; A
td/1
/#5D/#29 /#14 U /#28/#5B E/; A
td/2
/#5D/#29 and t /#15 td/4/./1/./2 Fixed time costT ask en vironmen ts with appro ximately /#0Cxed time cost are also v ery common/. Examplesinclude consultations with la wy ers/, k eeping a taxi w aiting/, or dithering o v er where to in v estone/'s money /. W e can de/#0Cne a task en vironmen t with /#0Cxed time cost c b y comparing theutilities of actions tak en at di/#0Beren t times/./5/8/6
Pr o v abl y bounded/-optimal a gentsDe/#0Cnition /8 Fixe d time c ost/: The task envir onment h E/; U i has a /#0Cxe d time c ost if/, forany action history pr e/#0Cxes A
t/1/1
and A
t/2/2
satisfying/#28/1/#29 A
T/1
/#28 t/1
/#29 /2 A/?
and A
T/2
/#28 t/2
/#29/= A
T/1
/#28 t/1
/#29/#28/2/#29 A
/#28 t/1
/, /1/#29/1
and A
/#28 t/2
/, /1/#29/2
c ontain no action in A/?the utilities di/#0Ber by the di/#0Ber enc e in time c ost/:U /#28/#5B E/; A
t/2/2
/#5D/#29 /= U /#28/#5B E/; A
t/1/1
/#5D/#29 /, c /#28 t/2
/, t/1
/#29Strictly sp eaking/, there are no task en vironmen ts with /#0Cxed time cost/. Utilit yv alues ha v ea/#0Cnite range/, so one cannot con tin ue incurring time costs inde/#0Cnitely /.F or reasonably shorttimes and reasonably small costs/, a linear utilit y p enalt y is a useful appro ximation/./4/./1/./3 Stochastic deadlinesWhile /#0Cxed/-deadline and /#0Cxed/-cost task en vironmen ts o ccur frequen tly in the design ofreal/-time systems/, uncertain t y ab out the time/-dep endence of the utilit y function is morecommon/. It also turns out to b e more in teresting/, as w e see b elo w/.A sto c hastic deadline is represen ted b y uncertain t y concerning the time of o ccurrence ofa /#0Cxed deadline/. In other w ords/, the agen t has a probabilit y distribution pd
for the deadlinetime td
/. W e assume that the deadline m ust come ev en tually /, so that
Pt /2 T
pd
/#28 t /#29/= /1 /. W ealso de/#0Cne the cum ulativ e deadline distribution Pd
/.If the deadline do es not o ccur at a kno wn time/, then w e need to distinguish b et w eent w o cases/:/#0F The agen t receiv es a p ercept/, called a her ald /#28Dean /& Bo ddy /, /1/9/8/8/#29/, whic h announcesan imp ending deadline/. W e mo del this using a distinguished p ercept Od
/:O
T/#28 td
/#29/= OdIf the agen t resp onds immediately /, then it /#5Cmeets the deadline/./"/#0F No suc h p ercept is a v ailable/, in whic h case the agen ti s w alking blindfolded to w ardsthe utilit y cli/#0B/. By delib erating further/, the agen t risks missing the deadline but ma yimpro v e its decision qualit y /. An example familiar to most readers is that of decidingwhether to publish a pap er in its curren t form/, or to em b ellish it further and riskb eing /#5Csco op ed/./" W e do not treat this case in the curren t pap er/.F ormally /, the sto c hastic deadline case is similar to the /#0Cxed deadline case/, except that td
isdra wn from the distribution pd
/. The utilit y of executing an action history pre/#0Cx A
tin E isthe exp ectation of the utilities of that state history pre/#0Cx o v er the p ossible deadline times/.De/#0Cnition /9 Sto chastic de ad line/: A task envir onment class h E /;U i of /#0Cxe d/-de ad line taskenvir onments has a sto chastic de ad line distribute da c c or ding to pd
if/, for any action historypr e/#0Cx A
t/,U /#28/#5B E /;A
t/#5D/#29 /=
Xt
/0/2 T
pd
/#28 t
/0/#29 U /#28/#5B Et
/0/;A
t/#5D/#29wher e h Et
/0/;U i is a task envir onment in h E /;U i with a /#0Cxe dd e ad line at t
/0/./5/8/7
R ussell /& SubramanianThe mail sorter example is w ell describ ed b y a sto c hastic deadline/. The time b et w eenthe arriv al of mail pieces at the image pro cessing station is distributed according to a densit yfunction pd
/, whic h will usually b e P oisson/./4/./2 Agen t programs and agen t arc hitectureW e consider simple agen t programs for episo dic task en vironmen ts/, constructed from ele/-men ts of a set R /= f r/1
/;/:/:/: /;rn
g of decision pro cedures or rules/. Eac h decision pro cedurerecommends /#28but do es not execute/#29 an action Ai
/2 A/?
/, and an agen t program is a /#0Cxedsequence of decision pro cedures/. F or our purp oses/, a decision pro cedure is a blac kb o x witht w o parameters/:/#0F a run time ti
/#15 /0/, whic hi sa n i n teger that represen ts the time tak en b y the pro cedureto compute an action/./#0F a qualit y qi
/#15 /0/, whic h is a real n um b er/. This giv es the exp ected rew ard resultingfrom executing its action Ai
at the start of an episo de/:qi
/= U /#28/#5B E/; Ai
/#5D/#29 /#28/1/#29Let MJ
denote an agen t arc hitecture that executes decision pro cedures in the language J /.Let tM
denote the maxim um run time of the decision pro cedures that can b e accommo datedin M /. F or example/, if the run time of a feedforw ard neural net w ork is prop ortional to itssize/, then tM
will b e the run time of the largest neural net w ork that /#0Cts in M /.The arc hitecture M executes an agen t program s /= s/1
/:/:/: sm
b y running eac h decisionpro cedure in turn/, pro viding the same input to eac h as obtained from the initial p ercept/.When a deadline arriv es /#28at a /#0Cxed time td
/, or heralded b y the p ercept Od
/#29/, or whenthe en tire sequence has b een completed/, the agen t selects the action recommended b y thehighest/-qualit y pro cedure it has executed/:M /#28 s/; I
T/#28 td
/#29 /;O
T/#28 td
/#29/#29 /= h i/0
/; action/#28 I
T/#28 td
/#29/#29 iM /#28 s/; I
T/#28 ts
/#29 /;O
T/#28 ts
/#29/#29 /= h i/0
/; action/#28 I
T/#28 ts
/#29/#29 i where ts
/=
Psi
/2 s
ti
/#28/2/#29M /#28 s/; I
T/#28 t /#29 /;Od
/#29 /= h i/0
/; action/#28 I
T/#28 t /#29/#29 iwhere M up dates the agen t/'s in ternal state history I
T/#28 t /#29 suc h that action/#28 I
T/#28 t /#29/#29 is theaction recommended b y a completed decision pro cedure with the highest qualit y /. Whenthis action is executed/, the in ternal state of the agen t is re/-initialized to i/0
/. This agen tdesign w orks in all three of the task en vironmen t categories describ ed ab o v e/.Next w e deriv e the v alue V /#28 s/; M /; E /#29 of an agen t program s in en vironmen t E runningon M for the three real/-time regimes and sho wh o w to construct b ounded optimal agen tsfor these task en vironmen ts/./4/./3 Bounded optimalit y with /#0Cxed deadlinesF rom Equation /2/, w e kno w that the agen t pic ks the action in A/?
recommended b y thedecision pro cedure r with the highest qualit y that is executed b efore the deadline td
arriv es/./5/8/8
Pr o v abl y bounded/-optimal a gentsLet s/1
/:/:/:sj
b e the longest pre/#0Cx of the program s suc h that
Pji /=/1
ti
/#14 td
/.F rom De/#0Cnition /7and Equation /1/, it follo ws thatV /#28 s/; M /; E /#29/= Qj
/#28/3/#29where Qi
/= max f q/1
/;/:/:/:/;qi
g /. Giv en this expression for the v alue of the agen t program/, w ecan easily sho w the follo wing/:Theorem /1 L et r /#03 /= arg maxri
/2 R /;ti
/#14 td
qi
/. The singleton se quenc e r /#03 is a b ounde d optimalpr o gr am for M in an episo dic task envir onment with a known de ad line td
/.That is/, the b est program is the single decision pro cedure of maxim um qualit y whose run timeis less than the deadline/./4/./4 Bounded optimalit y with /#0Cxed time costF rom Equation /2/, w e kno w that the agen t pic ks the action in A/?
recommended b y the b estdecision pro cedure in the sequence/, since M runs the en tire sequence s /= s/1
/:/:/:sm
whenthere is no deadline/. F rom De/#0Cnition /8 and Equation /1/, w eh a v eV /#28 s/; M /; E /#29/= Qm
/, c
mXi /=/1
ti
/#28/4/#29Giv en this expression for the v alue of the agen t program/, w e can easily sho w the follo wing/:Theorem /2 L et r /#03 /= arg maxri
/2 R
qi
/, cti
/. The singleton se quenc e r /#03 is a b ounde d optimalpr o gr am for M in an episo dic task envir onment with a /#0Cxe d time c ost c /.That is/, the optimal program is the single decision pro cedure whose qualit y /, net of timecost/, is highest/./4/./5 Bounded optimalit y with sto c hastic deadlinesWith a sto c hastic deadline distributed according to pd
/, the v alue of an agen t pro/-gram s /= s/1
/:/:/:sm
is an exp ectation/. F rom De/#0Cnition /9/, w e can calculate this asPt /2 T
pd
/#28 t /#29 V /#28 s/; M /; Et
/#29/, where h Et
/;U i is a task en vironmen t with a /#0Cxed deadline at t /. Af/-ter substituting for V /#28 s/; M /; Et
/#29 from Equation /3/, this expression simpli/#0Ces to a summation/,o v er the pro cedures in the sequence/, of the probabilit yo f i n terruption after the i
thpro cedurein the sequence m ultiplied b y the qualit y of the b est completed decision pro cedure/:V /#28 s /#29 /#11 V /#28 s/; M /; E /#29/=
mXi /=/1
/#5B Pd
/#28
Pi /+/1j /=/1
tj
/#29 /, Pd
/#28
Pij /=/1
tj
/#29/#5D Qi
/#28/5/#29where Pd
/#28 t /#29/=
Rt/,/1
pd
/#28 t
/0/#29 dt
/0and Pd
/#28 t /#29 /= /1 for t /#15
Pmi /=/1
ti
/.A simple example serv es to illustrate the v alue function/. Consider R /= f r/1
/;r/2
/;r/3
g /. Therule r/1
has a qualit y of /0/./2 and needs /2 seconds to run/: w e will represen t this b y r/1
/=/#28 /0 /: /2 /; /2/#29/.The other rules are r/2
/=/#28 /0 /: /5 /; /5/#29 /; r/3
/=/#28 /0 /: /7 /; /7/#29/. The deadline distribution function pd
is auniform distribution o v er /0 to /1/0 seconds/. The v alue of the sequence r/1
r/2
r/3
isV /#28 r/1
r/2
r/3
/#29/= /#5B /: /7 /, /: /2/#5D /: /2/+ /#5B /1 /, /: /7/#5D /: /5/+ /#5B /1 /, /1/#5D /: /7/= /: /2/5A geometric in tuition is giv en b y the notion of a p erformanc ep r o/#0Cle /,a ss h o wn in Fig/-ure /2/./5/8/9
R ussell /& Subramanian
0.2q
t0.50.7
25 7p(t)Figure /2/: P erformance pro/#0Cle for r/1
r/2
r/3
/, with pd
sup erimp osed/.De/#0Cnition /1/0 Performanc ep r o/#0Cle/: F o ras e quenc e s /, the p erformanc ep r o/#0Cle Qs
/#28 t /#29 givesthe quality of the action r eturne d if the agent is interrupte da t t /:Qs
/#28 t /#29/= max f qi
/:
iXj /=/1
tj
/#14 t gF or a uniform deadline densit y function/, the v alue of a sequence is prop ortional to thearea under the p erformance pro/#0Cle up to the last p ossible in terrupt time/. Note that theheigh t of the pro/#0Cle during the in terv al of length ti
while rule i is running is the qualit yo fthe b est of the pr evious rules/.F rom De/#0Cnition /1/0/, w eh a v e the follo wing ob vious prop ert y/:Lemma /1 The p erformanc ep r o/#0Cle of any se quenc e is monotonic al ly nonde cr e asing/.It is also the case that a sequence with higher qualit y decisions at all times is a b ettersequence/:Lemma /2 If /8 tQs/1
/#28 t /#29 /#15 Qs/2
/#28 t /#29 /, then V /#28 s/1
/#29 /#15 V /#28 s/2
/#29 /.In this case w es a y that Qs/1
dominates Qs/2
/.W e can use the idea of p erformance pro/#0Cles to establish some useful prop erties of optimalsequences/.Lemma /3 Ther e exists an optimal se quenc e that is sorte d in incr e asing or der of q /'s/.Without Lemma /3/, there are
Pni /=/1
i /! p ossible sequences to consider/. The ordering con/-strain t eliminates all but /2
nsequences/. It also means that in pro ofs of prop erties of se/-quences/, w en o w need consider only ordered sequences/. In addition/, w e can replace Qi
inEquation /5 b y qi
/.The follo wing lemma establishes that a sequence can alw a ys b e impro v ed b y the additionof a b etter rule at the end/:Lemma /4 F or every se quenc e s /= s/1
/:/:/:sm
sorte d in incr e asing or der of quality/, and singlestep z with qz
/#15 qsm
/, V /#28 sz /#29 /#15 V /#28 s /#29 /./5/9/0
Pr o v abl y bounded/-optimal a gentsCorollary /1 Ther e exists an optimal se quenc e ending with the highest/-quality rule in R /.The follo wing lemma re/#0Dects the ob vious in tuition that if one can get a b etter result inless time/, there/'s no p oin t sp ending more time to get a w orse result/:Lemma /5 Ther e exists an optimal se quenc e whose rules ar e in nonde cr e asing or der of ti
/.W en o w apply these preparatory results to deriv e algorithms that construct b oundedoptimal programs for v arious deadline distributions/./4/./5/./1 General distributionsF or a general deadline distribution/, the dynamic pr o gr amming metho d can b e used to obtainan optimal sequence of decision rules in pseudo/-p olynomial time/. W e construct an optimalsequence b y using the de/#0Cnition of V /#28 s/; M /; E /#29 in Equation /5/. Optimal sequences generatedb y the metho ds are ordered b y qi
/, in accordance with Lemma /3/.W e construct the table S /#28 i/; t /#29/, where eac he n try in the table is the highest v alue ofan y sequence that ends with rule ri
at time t /.W e assume the rule indices are arranged inincreasing order of qualit y /, and t ranges from the start time /0 to the end time L /=
Pri
/2 R
ti
/.The up date rule is/:S /#28 i/; t /#29/= maxk /2 /#5B/0 /:/:/: i /, /1/#5D
/#5B S /#28 k/; t /, ti
/#29/+/#28 qi
/, qk
/#29/#5B/1 /, Pd
/#28 t /#29/#5D/#5Dwith b oundary conditionS /#28 i/; /0/#29 /= /0 for eac h rule i and S /#28/0 /;t /#29 /= /0 for eac h time tF rom Corollary /1/, w e can read o/#0B the b est sequence from the highest v alue in ro w n of thematrix S /.Theorem /3 The DP algorithm c omputes an optimal se quenc e in time O /#28 n
/2L /#29 wher e n isthe numb er of de cision pr o c e dur es in R /.The dep endence on L in the time complexit y of the DP algorithm means that the algo/-rithm is not p olynomial in the input size/. Using standard rounding and scaling metho ds/,ho w ev er/, a fully p olynomial appro ximation sc heme can b e constructed/. Although w ed on o tha v e a hardness pro of for the problem/, John Binder /#28/1/9/9/4/#29 has sho wn that if the deadlinedistribution is used as a constan t/-time oracle for /#0Cnding v alues of P /#28 t /#29/, an y algorithm willrequire an exp onen tial n um b er of calls to the oracle in the w orst case/./4/./5/./2 Long unif orm distributionsIf the deadline is uniformly distributed o v er a time in terv al greater than the sum of therunning times of the rules/, w e will call the distribution a long uniform distribution/. Considerthe rule sequence s /= s/1
/:/:/:sm
dra wn from the rule set R /. With a long uniform distribution/,the probabilit y that the deadline arriv es during rule si
of the sequence s is indep enden to fthe time at whic h si
starts/. This p ermits a simpler form of Equation /5/:V /#28 s/; M /; E /#29/=
Pm /, /1i /=/1
Pd
/#28 ti /+/1
/#29 qi
/+ qm
/#28/1 /,
Pmi /=/1
Pd
/#28 ti
/#29/#29 /#28/6/#29/5/9/1
R ussell /& SubramanianT o deriv e an optimal sequence under a long uniform distribution/, w e obtain a recursiv esp eci/#0Ccation of the v alue of a sequence as with a /2 R and s /= s/1
/:/:/:sm
b eing some sequencein R /.V /#28 as/; M /; E /#29/= V /#28 s/; M /; E /#29/+ qa
Pd
/#28 t/1
/#29 /, qm
Pd
/#28 ta
/#29 /#28/7/#29This allo ws us to de/#0Cne a dynamic programming sc heme for calculating an optimal sequenceusing a state function S /#28 i/; j /#29 denoting the highest v alue of a rule sequence that starts withrule i and ends in rule j /.F rom Lemma /3 and Equation /7/, the up date rule is/:S /#28 i/; j /#29 /= maxi/#3Ck /#14 j
/#5B S /#28 k/; j /#29/+ Pd
/#28 tk
/#29 qi
/, Pd
/#28 ti
/#29 qj
/#5D /#28/8/#29with b oundary conditionS /#28 i/; i /#29/= /#28 /1 /, Pd
/#28 ti
/#29/#29 qi
/#28/9/#29F rom Corollary /1/, w e kno w that an optimal sequence for the long uniform distribution endsin rn
/, the rule with the highest qualit yi n R /. Th us/, w e only need to examine S /#28 i/; n /#29 /; /1 /#14i /#14 n /. Eac he n try requires O /#28 n /#29 computation/, and there are n en tries to compute/. Th us/,the optimal sequence for the long uniform case can b e calculated in O /#28 n
/2/#29/.Theorem /4 A n optimal se quenc eo f d e cision pr o c e dur es for a long uniform de ad line dis/-tribution c an b e determine di n O /#28 n
/2/#29 time wher e n is the numb er of de cision pr o c e dur es inR /./4/./5/./3 Shor t unif orm distributionsWhen
Pni /=/1
Pd
/#28 ti
/#29 /#3E /1/, for a uniform deadline distribution Pd
/,w e call it short /. This meansthat some sequences are longer than the last p ossible deadline time/, and therefore some rulesin those sequences ha v e no p ossibilit y of executing b efore the deadline/. F or suc h sequences/,w e cannot use Equation /7 to calculate V /#28 s /#29/. Ho w ev er/, an y suc h sequence can b e truncatedb y remo ving all rules that w ould complete execution after the last p ossible deadline/. Thev alue of the sequence is una/#0Bected b y truncation/, and for truncated sequences the use ofEquation /7 is justi/#0Ced/. F urthermore/, there is an optimal sequence that is a truncatedsequence/.Since the up date rule /8 correctly computes S /#28 i/; j /#29 for truncated sequences/, w e can useit with short uniform distributions pro vided w e add a c hec k to ensure that the sequencesconsidered are truncated/. Unlik e the long uniform case/, ho w ev er/, the iden tit y of the last rulein an optimal sequence is unkno wn/, so w e need to compute all n
/2en tries in the S /#28 i/; j /#29 table/.Eac he n try computation tak es O /#28 n /#29 time/, th us the time to compute an optimal sequence isO /#28 n
/3/#29/.Theorem /5 A n optimal se quenc eo f d e cision pr o c e dur es for a short uniform de ad line dis/-tribution c an b e determine di n O /#28 n
/3/#29 time wher e n is the numb er of de cision pr o c e dur es inR /./5/9/2
Pr o v abl y bounded/-optimal a gents/4/./5/./4 Exponential distributionsF or an exp onen tial distribution/, Pd
/#28 t /#29/= /1 /, e
/, /#0Ct/. Exp onen tial distributions allo w an optimalsequence to b e computed in p olynomial time/. Let pi
stand for the probabilit y that rule iis in terrupted/, assuming it starts at /0/. Then pi
/= Pd
/#28 ti
/#29/= /1 /, e
/, /#0Cti/: F or the exp onen tialdistribution/, V /#28 s/; M /; E /#29 simpli/#0Ces out as/:V /#28 s/; M /; E /#29/=
m /, /1Xi /=/1
h/#05
ij /=/1
/#28/1 /, pj
/#29
ipi /+/1
qi
/+
h/#05
mj /=/1
/#28/1 /, pj
/#29
iqmThis yields a simple recursiv e sp eci/#0Ccation of the v alue V /#28 as/; M /; E /#29 of a sequence thatb egins with the rule a /:V /#28 as/; M /; E /#29/= /#28 /1 /, pa
/#29 p/1
qa
/+/#28 /1 /, pa
/#29 V /#28 s/; M /; E /#29W e will use the state function S /#28 i/; j /#29 whic h represen ts the highest v alue of an y rule sequencestarting with i and ending in j /.S /#28 i/; j /#29/= maxi/#3Ck /#14 j
/#5B /#28/1 /, pi
/#29 pk
qi
/+/#28 /1 /, pi
/#29 S /#28 k/; j /#29/#5Dwith b oundary condition S /#28 i/; i /#29/= qi
/#28/1 /, pi
/#29/. F or an y giv en j /, S /#28 i/; j /#29 can b e calculated inO /#28 n
/2/#29/. F rom Corollary /1/, w e kno w that there is an optimal sequence whose last elemen ti sthe highest/-v alued rule in R /.Theorem /6 A n optimal se quenc eo fd e cision pr o c e dur es for an exp onential ly distribute dsto chastic de ad line c an b e determine di n O /#28 n
/2/#29 time wher e n is the numb er of de cisionpr o c e dur es in R /.The pro of is similar to the long uniform distribution case/./4/./6 Sim ulation results for a mail/-sorterThe preceding results pro vide a set of algorithms for optimizing the construction of an agen tprogram for a v ariet y of general task en vironmen t classes/. In this section/, w e illustrate theseresults and the p ossible gains that can b e realized in a sp eci/#0Cc task en vironmen t/, namely /,a sim ulated mail/-sorter/.First/, let us b e more precise ab out the utilit y function U on episo des/. There are fourp ossible outcomes/; the utilit y of outcome i is ui
/./1/. The zip co de is successfully read and the letter is sen t to the correct bin for deliv ery /./2/. The zip co de is misread and the letter go es to the wrong bin/./3/. The letter is sen t to the reject bin/./4/. The next letter arriv es b efore the recognizer has /#0Cnished/, and there is a jam/. Sinceletter arriv al is heralded/, jams cannot o ccur with the mac hine arc hitecture giv en inEquation /2/./5/9/3
R ussell /& Subramanian
00.20.40.60.81
0 2 4 6 8 10Accuracy
Computation Time (sec)lambda=0.9
00.20.40.60.81
0 2 4 6 8 10P(t)
Time (sec)mu=9Figure /3/: /#28a/#29 Accuracy pro/#0Cle /#28/1 /, e
/, /#15x/#29/, for /#15 /=/0 /: /9/. /#28b/#29 P oisson arriv al distribution/, formean /#16 /= /9 secWithout loss of generalit y /,w e set u/1
/=/1 /: /0 and u/2
/=/0 /: /0/. If the probabilit y of a rulerecommending a correct destination bin is pi
/, then qi
/= pi
u/1
/+/#28 /1 /, pi
/#29 u/2
/= pi
/.W e assumethat u/2
/#14 u/3
/, hence there is a threshold probabilit y b elo w whic h the letter should b e sen tto the reject bin instead/. W e will therefore include in the rule set R a rule rr ej ect
thathas zero run time and recommends rejection/. The sequence construction algorithm will thenautomatically exclude rules with qualit yl o w er than qr ej ect
/= u/3
/. The o v erall utilit y for anepiso de is c hosen to b e a linear com bination of the qualit y of sorting /#28 qi
/#29/, the probabilit yo frejection or the rejection rate /#28giv en b y P /#28 t/1
/#29/, where t/1
is the run time of the /#0Crst non/-rejectrule executed/#29/, and the sp eed of sorting /#28measured b y the arriv al time mean/#29/.The agen t program in /#28Boser et al/. /1/9/9/2/#29 uses a single neural net w ork on a c hip/.W e sho w that under a v ariet y of conditions an optimized sequence of net w orks can dosigni/#0Ccan tly b etter than an y single net w ork in terms of throughput or accuracy /.W e examinethe follo wing exp erimen tal conditions/:/#0F W e assume that a net w ork that executes in time t has a recognition accuracy p thatdep ends on t /.W e consider p /=/1 /, e
/, /#15t/. The particular c hoice of /#15 is irrelev an t b ecausethe scale c hosen for t is arbitrary /.W ec ho ose /#15 /=/0 /: /9/, for con v enience /#28Figure /3/#28a/#29/#29/.W e include rr ej ect
with qr ej ect
/= u/3
and tr ej ect
/=/0 /./#0F W e consider arriv al time distributions that are P oisson with v arying means/. Fig/-ure /3/#28b/#29 sho ws three example distributions/, for means /1/, /5/, and /9 seconds/./#0F W e create optimized sequences from sets of /4/0 net w orks with execution times tak enat equal in terv als from t /= /1 to /4/0/./#0F W e compare/#28a/#29 BO sequence/: a b ounded optimal sequence/;/#28b/#29 Best singleton/: the b est single rule/;/#28c/#29 /5/0/#25 rule/: the rule whose execution time is the mean of the distribution /#28i/.e/./, itwill complete in /5/0/#25 of cases/#29/;/5/9/4
Pr o v abl y bounded/-optimal a gents
00.20.40.60.81
5 10 15 20 25 30 35 40Average utility per second
Mean arrival timeBO Sequence
Best Singleton
50% Rule
90% RuleFigure /4/: Graph sho wing the ac hiev able utilit y p er second as a function of the a v erage timep er letter/, for the four program t yp es/. /#15 /=/0 /: /9/./#28d/#29 /9/0/#25 rule/: the rule whose execution time guaran tees that it will complete in /9/0/#25of cases/.In the last three cases/, w e add rr ej ect
as an initial step/; the BO sequence will includeit automatically /./#0F W e measure the utilit y p er second as a function of the mean arriv al rate /#28Figure /4/#29/.This sho ws that there is an optimal setting of the sorting mac hinery at /6 letters p ermin ute /#28in ter/-arriv al time /= /1/0 seconds/#29 for the b ounded optimal program/, giv en thatw eh a v e /#0Cxed /#15 at /0/./9/./#0F Finally /,w ei n v estigate the e/#0Bect of the v ariance of the arriv al time on the relativ ep erformance of the four program t yp es/. F or this purp ose/, w e use a uniform distributioncen tered around /2/0 seconds but with di/#0Beren t widths to v ary the v ariance withouta/#0Becting the mean /#28Figure /5/#29/.W e notice sev eral in teresting things ab out these results/:/#0F The p olicy of c ho osing a rule with a /9/0/#25 probabilit y of completion p erforms p o orlyfor rapid arriv al rates /#28 /#16 /#14 /3/#29/, but catc hes up with the p erformance of the b est singlerule for slo w er arriv al rates /#28 /#16/#3E /4/#29/. This is an artifact of the exp onen tial accuracypro/#0Cle for an y /#15/#3E /0 /: /5/, where the di/#0Berence in qualit y of the rules with run timesgreater than /6 seconds is quite small/./#0F The p olicy of c ho osing a rule with a /5/0/#25 probabilit y of completion fares as w ell asthe b est single rule for v ery high arriv al rates /#28 /#16 /#14 /2/#29/, but rapidly div erges from itthereafter/, p erforming far w orse for arriv al time means greater than /5 seconds/./5/9/5
R ussell /& Subramanian
00.20.40.60.81
0 20 40 60 80 100 120Average utility per second
Variance in arrival timeBO Sequence
Best Singleton
50% Rule
90% RuleFigure /5/: Graphs sho wing the utilit y gain p er second as a function of the arriv al timev ariance/, for the four program t yp es for the uniform distribution with a mean of/2/0 seconds/./#0F Both the b est sequence and the b est single rule giv e their b est o v erall p erformanceat an arriv al rate of around /6 letters p er min ute/. The p erformance adv an tage of theoptimal sequence o v er the b est single rule is ab out /7/#25 at this arriv al rate/. It shouldb e noted that this is a signi/#0Ccan t p erformance adv an tage that is obtainable with noextra computational resources/. F or slo w er arriv al rates /#28 /#16 /#15 /7/#29/, the di/#0Berence b et w eenthe p erformance of the b est rule and the b est sequence arises from the decreasedrejection rate of the b est sequence/. With the exp onen tial accuracy pro/#0Cle /#28 /#15 /#15 /0 /: /5/#29the adv an tage of running a rule with a shorter completion time ahead of a longer ruleis the abilit y to reduce the probabilit y of rejecting a letter/. F or high arriv al rates/#28in ter/-arriv al times of /1 to /4 seconds/#29/, it is useful to ha v e a few short rules instead ofa longer single rule/./#0F Figure /5 sho ws that the b est sequence p erforms b etter than the b est single rule as thev ariance of the arriv al time increases/.
/5The p erformance of the optimal sequence alsoapp ears to b e largely una/#0Bected b yv ariance/. This is exactly the b eha viour w e exp ectto observ e /| the abilit y to run a sequence of rules instead of committing to a singleone giv es it robustness in the face of increasing v ariance/. Since realistic en vironmen tscan in v olv e unexp ected demands of man y kinds/, the p ossession of a v ariet y of defaultb eha viours of graded sophistication w ould seem to b e an optimal design c hoice for ab ounded agen t/./5/. The p erformance of the /5/0/#25 rule is /#0Dat b ecause the uniform distributions used in this exp erimen th a v e/#0Cxed mean and are symmetric/, so that the /5/0/#25 rule is alw a ys the rule that runs for /2/0 seconds/. The/9/0/#25 rule c hanges with the v ariance/, and the curv e exhibits some discretization e/#0Bects/. These could b eeliminated using a /#0Cner/-grained set of rules/./5/9/6
Pr o v abl y bounded/-optimal a gents/5/. Learning Appro ximately Bounded/-Optimal ProgramsThe ab o v e deriv ations assume that a suitable rule set R is a v ailable ab initio /, with correctqualities qi
and run times ti
/, and that the deadline distribution is kno wn/. In this section/, w estudy w a ys in whic h some of this information can b e learned/, and the implications of thisfor the b ounded optimalit y of the resulting system/. W e will concen trate on learning rulesand their qualities/, lea ving run times and deadline distributions for future w ork/.The basic idea is that the learning algorithms will con v erge/, o v er time/, to a set ofoptimal comp onen ts /| the most accurate rules and the most accurate qualit y estimates forthem/. As this happ ens/, the v alue of the agen t constructed from the rules/, using the qualit yestimates/, con v erges to the v alue of lopt
/.T h us there are t w o sources of sub optimalit y in thelearned agen t/:/#0F The rules in R ma y not b e the b est p ossible rules /| they ma y recommend actionsthat are of lo w er utilit y than those that w ould b e recommended b y some other rules/./#0F There ma y b e errors in estimating the exp ected utilit y of the rule/. This can cause thealgorithms giv en ab o v e to construct sub optimal sequences/, ev en if the b est rules area v ailable/.Our notional metho d for constructing b ounded optimal agen ts /#28/1/#29 learns sets of indi/-vidual decision pro cedures from episo dic in teractions/, and /#28/2/#29 arranges them in a sequenceusing one of the algorithms describ ed earlier so that the p erformance of an agen t usingthe sequence is at least as go o d as that of an y other suc h agen t/. W e assume a parameter/-ized learning algorithm LJ /;k
that will b e used to learn one rule for eac h p ossible run timek /2f /1 /;/:/:/:/;tM
g /. Since there is nev er a need to include t w o rules with the same run timein the R /, this ob viates the need to consider the en tire rule language J in the optimizationpro cess/.Our setting places somewhat un usual requiremen ts on the learning algorithm/. Lik emost learning algorithms/, LJ /;k
w orks b y observing a collection T of training episo des in E /,including the utilit y obtained for eac h episo de/. W e do not/, ho w ev er/, mak ea n y assumptionsab out the form of the c orr e ct decision rule/. Instead/, w e mak e assumptions ab out theh yp otheses/, namely that they come from some /#0Cnite language Jk
/, the set of programsin J of complexit y at most k /. This setting has b een called the agnostic le arning setting b yKearns/, Sc hapire and Sellie /#28/1/9/9/2/#29/, b ecause no assumptions are made ab out the en vironmen tat all/. It has b een sho wn /#28Theorems /4 and /5 in Kearns/, Sc hapire and Sellie/, /1/9/9/2/#29 that/, forsome languages J /, the error in the learned appro ximation can b e b ounded to within /#0F ofthe b est rule in Jk
that /#0Cts the examples/, with probabilit y/1 /, /#0E /. The sample size neededto guaran tee these b ounds is p olynomial in the complexit y parameter k /,a sw ell as
/1/#0F
and
/1/#0E
/.In addition to constructing the decision pro cedures/, LJ /;k
outputs estimates of theirqualit y qi
/. Standard Cherno/#0B/-Ho e/#0Bding b ounds can b e used to limit the error in the qualit yestimate to b e within /#0Fq
with probabilit y/1 /, /#0Eq
/. The sample size for the estimation of qualit yis also p olynomial in
/1/#0Fq
and
/1/#0Eq
/.Th us the error in eac h agnostically learned rule is b ounded to within /#0F of the b est rulein its complexit y class with probabilit y/1 /, /#0E /. The error in the qualit y estimation of theserules is b ounded b y /#0Fq
with probabilit y/1 /, /#0Eq
/.F rom these b ounds/, w e can calculate a b oundon the utilit y de/#0Ccit in the agen t program that w e construct/, in comparison to lopt
/:/5/9/7
R ussell /& SubramanianTheorem /7 Assume an ar chite ctur e MJ
that exe cutes se quenc es of de cision pr o c e dur es inan agnostic al ly le arnable language J whose runtimes r ange over /#5B/1 /:/:tM
/#5D /.F or r e al time taskenvir onments with /#0Cxe d time c ost/, /#0Cxe dd e ad line/, and sto chastic de ad line/, we c an c onstructap r o gr am l such thatV /#28 lopt
/;M /; E /#29 /, V /#28 l/; M /; E /#29 /#14 /#0F /+/2 /#0Fqwith pr ob ability gr e ater than /1 /, m /#28 /#0E /+ /#0Eq
/#29 /, wher e m is the numb er of de cision pr o c e dur es inlopt
/.Pro of/: W e pro v e this theorem for the sto c hastic deadline regime/, where the b oundedoptimal program is a sequence of decision pro cedures/. The pro ofs for the /#0Cxed cost and/#0Cxed deadline regimes/, where the b ounded optimal program is a singleton/, follo wa s asp ecial case/. Let the b est decision pro cedures for E b e the set R
/#03/= f r/1
/#03/;/:/:/:/;rn
/#03g /, andlet lopt
/= s/1
/#03/:/:/:sm
/#03b e an optimal sequence constructed from R
/#03/. Let R /= f r/1
/;/:/:/:rn
g bethe set of decision pro cedures returned b y the learning algorithm/. With probabilit y greaterthan /1 /, m/#0E /, q
/#03i
/, qi
/#14 /#0F for all i /, where qi
refers to the true qualit yo f ri
/. The error in theestimate d qualit y/^ qi
of decision pro cedure ri
is also b ounded/: with probabilit y greater than/1 /, m/#0Eq
/, j /^ qi
/, qi
j/#14 /#0Fq
for all i /.Let s /= s/1
/:/:/:sm
b e those rules in R that come from the same run time classes as therules s/1
/#03/:/:/:sm
/#03in R
/#03/. Then/, b y Equation /5/, w eh a v eV /#28 lopt
/;M /; E /#29 /, V /#28 s/; M /; E /#29 /#14 /#0Fb ecause the error in V is a w eigh ted a v erage of the errors in the individual qi
/. Similarly /,w eha v ej
/^V /#28 s/; M /; E /#29 /, V /#28 s/; M /; E /#29 j/#14 /#0FqNo w supp ose that the sequence construction algorithm applied to R pro duces a sequencel /= s/1
/0/:/:/: sl
/0/. By de/#0Cnition/, this sequence app e ars to b e optimal according to the estimatedv alue function
/^V /. Hence/^V /#28 l/; M /; E /#29 /#15
/^V /#28 s/; M /; E /#29As b efore/, w e can b ound the error on the estimated v alue/:j
/^V /#28 l/; M /; E /#29 /, V /#28 l/; M /; E /#29 j/#14 /#0FqCom bining the ab o v e inequalities/, w eh a v eV /#28 lopt
/;M /; E /#29 /, V /#28 l/; M /; E /#29 /#14 /#0F /+/2 /#0Fq/2Although the theorem has practical applications/, it is mainly in tended as an illustrationof ho w a learning pro cedure can con v erge on a b ounded optimal con/#0Cguration/. With someadditional w ork/, more general error b ounds can b e deriv ed for the case in whic h the ruleexecution times ti
and the real/-time utilit yv ariation /#28time cost/, /#0Cxed deadline/, or deadlinedistribution/#29 are all estimated from the training episo des/. W e can also obtain error b oundsfor the case in whic h the rule language J is divided up in to a smaller n um b er of coarserrun time classes/, rather than the p oten tially h uge n um b er that w e curren tly use/./5/9/8
Pr o v abl y bounded/-optimal a gents/6/. Asymptotic Bounded Optimalit yThe strict notion of b ounded optimalit ym a y b e a useful philosophical landmark from whic hto explore arti/#0Ccial in telligence/, but it ma y b e to o strong to allo w man yi n teresting/, generalresults to b e obtained/. The same observ ation can b e made in ordinary complexit y theory/:although absolute e/#0Eciency is the aim/, asymptotic e/#0Eciency is the game/. That a sortingalgorithm is O /#28 n log n /#29 rather than O /#28 n
/2/#29 is considered signi/#0Ccan t/, but replacing a /#5Cm ultiplyb y/2 /"b y a /#5Cshift/-left /1 bit/" is not considered a real adv ance/. The slac k allo w ed b y thede/#0Cnitions of complexit y classes is essen tial in building on earlier results/, in obtaining robustresults that are not restricted to sp eci/#0Cc implemen tations/, and in analysing the complexit yo falgorithms that use other algorithms as subroutines/. In this section/, w e b egin b y reviewingclassical complexit y /. W e then prop ose de/#0Cnitions of asymptotic b ounded optimalit y thatha v e some of the same adv an tages/, and sho w that classical optimalit y is a sp ecial case ofasymptotic b ounded optimalit y /. Lastly /,w e rep ort on some preliminary in v estigations in tothe use of asymptotic b ounded optimalit y as a theoretical to ol in constructing univ ersalreal/-time systems/./6/./1 Classical complexit yA pr oblem /, in the classical sense/, is de/#0Cned b y a pair of predicates /#1E and /#20 suc h that outputz is a solution for input x if and only if /#1E /#28 x /#29 and /#20 /#28 x/; z /#29 hold/. A pr oblem instanc e is aninput satisfying /#1E /, and an algorithm for the problem class alw a ys terminates with an outputz satisfying /#20 /#28 x/; z /#29 giv en an input x satisfying /#1E /#28 x /#29/. Asymptotic complexit y describ es thegro wth rate of the w orst/-case run time of an algorithm as a function of the input size/. W ecan de/#0Cne this formally as follo ws/. Let Ta
/#28 x /#29 b e the run time of algorithm a on input x /,and let T
/#03a
/#28 n /#29 b e the maxim um run time of a on an y input of size n /. Then algorithm a hascomplexit y O /#28 f /#28 n /#29/#29 if/9 k/; n/0
/8 n n/#3En/0
/#29 T
/#03a
/#28 n /#29 /#14 kf /#28 n /#29In tuitiv ely /, a classically optimal algorithm is one that has the lo w est p ossible complexit y /.F or the purp oses of constructing an asymptotic notion of b ounded optimalit y /, it will b euseful to ha v e a de/#0Cnition of classical optimalit y that do es not men tion the complexit ydirectly /. This can b e done as follo ws/:De/#0Cnition /1/1 Classic al ly optimal algorithm/: A n algorithm a is classic al ly optimal if andonly if/9 k/; n/0
/8 a
/0/; n n/#3En/0
/#29 T
/#03a
/#28 n /#29 /#14 kT
/#03a
/0
/#28 n /#29T o relate classical complexit y to our framew ork/, w e will need to de/#0Cne the sp ecial case of tasken vironmen ts in whic h traditional programs are appropriate/. In suc h task en vironmen ts/,an input is pro vided to the program as the initial p ercept/, and the utilit y function onen vironmen t histories ob eys the follo wing constrain t/:De/#0Cnition /1/2 Classic al task envir onment/: h EP
/;U i is a classic al task envir onment forpr oblem P ifV /#28 l/; M /; EP
/#29 /=
/#28u /#28 T /#28 l/; M /; EP
/#29/#29 if l outputs a c orr e ct solution for P/0 otherwise/5/9/9
R ussell /& Subramanianwher e T /#28 l/; M /; EP
/#29 is the running time for l in EP
on M /, M is a universal T uring machine/,and u is some p ositive de cr e asing function/.The notion of a pr oblem class in classical complexit y theory th us corresp onds to a class ofclassical task en vironmen ts of un b ounded complexit y /.F or example/, the T ra v eling Salesp er/-son Problem con tains instances with arbitrarily large n um b ers of cities/./6/./2 V arieties of asymptotic b ounded optimalit yThe /#0Crst thing w e will need is a complexit y measure on en vironmen ts/. Let n /#28 E /#29 b e a suitablemeasure of the complexit yo fa n e n vironmen t/. W e will assume the existence of en vironmen tclasses that are of un b ounded complexit y /. Then/, b y analogy with the de/#0Cnition of classicaloptimalit y /,w e can de/#0Cne a w orst/-case notion of asymptotic b ounded optimalit y /#28ABO/#29/.Letting V
/#03/#28 l/; M /; n /; E /#29 b e the minim um v alue of V /#28 l/; M /; E /#29 for all E in E of complexit y n /,w eh a v eDe/#0Cnition /1/3 Worst/-c ase asymptotic b ounde d optimality/: an agent pr o gr am l is timewise/#28or sp ac ewise/#29 worst/-c ase asymptotic al ly b ounde d optimal in E on M i/#0B/9 k/; n/0
/8 l
/0/;n n/#3En/0
/#29 V
/#03/#28 l/; k M /; n /; E /#29 /#15 V
/#03/#28 l
/0/;M /;n /; E /#29wher e kM denotes a version of the machine M sp e e de d up by a factor k /#28or with k timesmor e memory/#29/.In English/, this means that the program is basically along the righ t lines if it just needs afaster /#28larger/#29 mac hine to ha v ew orst/-case b eha viour as go o d as that of an y other programin all en vironmen ts/.If a probabilit y distribution is asso ciated with the en vironmen t class E /, then w e can usethe exp ected v alue V /#28 l/; M /; E /#29 to de/#0Cne an a v erage/-case notion of ABO/:De/#0Cnition /1/4 A ver age/-c ase asymptotic b ounde d optimality/: an agent pr o gr am l is timewise/#28or sp ac ewise/#29 aver age/-c ase asymptotic al ly b ounde d optimal in E on M i/#0B/9 k /8 l
/0V /#28 l/; k M /; E /#29 /#15 V /#28 l
/0/;M /; E /#29F or b oth the w orst/-case and a v erage/-case de/#0Cnitions of ABO/, w ew ould b e happ y with aprogram that w as ABO for a non trivial en vironmen t on a non trivial arc hitecture M /, unlessk w ere enormous/.
/6In the rest of the pap er/, w e will use the w orst/-case de/#0Cnition of ABO/.Almost iden tical results can b e obtained using the a v erage/-case de/#0Cnition/.The /#0Crst observ ation that can b e made ab out ABO programs is that classically optimalprograms are a sp ecial case of ABO programs/:
/7/6/. The classical de/#0Cnitions allo w for optimalit y up to a constan t factor k in the run time of the algorithms/.One migh tw onder wh yw ec hose to use the constan t factor to expand the mac hine capabilities/, ratherthan to increase the time a v ailable to the program/. In the con text of ordinary complexit y theory /, thet w o alternativ es are exactly equiv alen t/, but in the con text of general time/-dep enden t utilities/, only theformer is appropriate/. It w ould not b e p ossible to simply /#5Clet l run k times longer/,/" b ecause the programsw e wish to consider con trol their o wn execution time/, trading it o/#0B against solution qualit y /. One couldimagine slo wing do wn the en tire en vironmen tb y a factor of k /, but this is merely a less realistic v ersionof what w e prop ose/./7/. This connection w as suggested b y Bart Selman/./6/0/0
Pr o v abl y bounded/-optimal a gentsTheorem /8 Ap r o gr am is classic al ly optimal for a given pr oblem P if and only if it istimewise worst/-c ase ABO for the c orr esp onding classic al task envir onment class h EP
/;U i /.This observ ation follo ws directly from De/#0Cnitions /1/1/, /1/2/, and /1/3/.In summary /, the notion of ABO will pro vide the same degree of theoretical robustnessand mac hine/-indep endence for the study of b ounded systems as asymptotic complexit yd o e sfor classical programs/. Ha ving set up a basic framew ork/, w e can no w b egin to exercise thede/#0Cnitions/./6/./3 Univ ersal asymptotic b ounded optimalit yAsymptotic b ounded optimalit y is de/#0Cned with resp ect to a sp eci/#0Cc v alue function V /. Inconstructing real/-time systems/, w ew ould prefer a certain degree of indep endence from thetemp oral v ariation in the v alue function/. W e can ac hiev e this b y de/#0Cning a family V of v aluefunctions/, di/#0Bering only in their temp oral v ariation/. By this w e mean that the v alue functionpreserv es the preference ordering of external actions o v er time/, with all v alue functions inthe family ha ving the same preference ordering/.
/8F or example/, in the /#0Cxed/-cost regime w e can v ary the time cost c to generate a family ofv alue functions/; in the sto c hastic deadline case/, w e can v ary the deadline distribution Pd
togenerate another family /. Also/, since eac h of the three regimes uses the same qualit y measurefor actions/, then the union of the three corresp onding families is also a family /. What w e willsho w is that a single program/, whic hw e call a univ ersal program/, can b e asymptoticallyb ounded/-optimal regardless of whic hv alue function is c hosen within an y particular family /.De/#0Cnition /1/5 Universal asymptotic b ounde d optimality /#28UABO/#29/: A n agent pr o gr am l isUABO in envir onment class E on M for the family of value functions V i/#0B l is ABO in Eon M for every Vi
/2V /.A UABO program m ust comp ete with the ABO programs for ev ery individual v alue functionin the family /. A UABO program is therefore a univ ersal real/-time solution for a giv en task/.Do UABO programs exist/? If so/, ho w can w e construct them/?It turns out that w e can use the sc heduling construction from /#28Russell /& Zilb erstein/,/1/9/9/1/#29 to design UABO programs/. This construction w as designed to reduce task en viron/-men ts with unkno wn in terrupt times to the case of kno wn deadlines/, and the same insigh tapplies here/. The construction requires the arc hitecture M to pro vide program concatena/-tion /#28e/.g/./, the LISP prog construct/#29/, a conditional/-return construct/, and the n ull program/#03/. The univ ersal program lU
has the form of a concatenation of individual programs ofincreasing run time/, with an appropriate termination test after eac h/. It can b e written aslU
/=/#5B l/0
/#01 l/1
/#01/#01/#01 lj
/#01/#01/#01 /#5Dwhere eac h lj
consists of a program and a termination test/. The program part in lj
is an yprogram in LM
that is ABO in E for a v alue function Vj
that corresp onds to a /#0Cxed deadlineat td
/=/2
j/#0F /, where /#0F is a time incremen t smaller than the execution time of an y non/-n ullprogram in LM
/./8/. The v alue function m ust therefore b e sep ar able /#28Russell /& W efald/, /1/9/8/9/#29/, since this preserv ation of rankorder allo ws a separate time cost to b e de/#0Cned/. See c hapter /9 of /#28Keeney /& Rai/#0Ba/, /1/9/7/6/#29 for a thoroughdiscussion of time/-dep enden t utilit y /./6/0/1
R ussell /& Subramanian
0.2q
t0.50.7
p(t)
0 10lopton  MlUon  4MFigure /6/: P erformance pro/#0Cles for lU
running on /4 M /, and for lopt
running on MBefore pro ceeding to a statemen t that lU
is indeed UABO/, let us lo ok at an example/.Consider the simple/, sequen tial mac hine arc hitecture describ ed earlier/. Supp ose w e canselect rules from a three/-rule set with r/1
/=/#28 /0 /: /2 /; /2/#29/, r/2
/=/#28 /0 /: /5 /; /5/#29 and r/3
/=/#28 /0 /: /7 /; /7/#29/. Sincethe shortest run time of these rules is /2 seconds/, w e let /#0F /= /1/. Then w e lo ok at the optimalprograms l/0
/;l/1
/;l/2
/;l/3
/;/:/:/: for the /#0Cxed/-deadline task en vironmen ts with td
/=/1 /; /2 /; /4 /; /8 /;/:/:/: /.These are/:l/0
/=/#03 /; l/1
/= r/1
/; l/2
/= r/1
/; l/3
/= r/3
/; /:/:/:Hence the sequence of programs in lU
is /#5B/#03 /;r/1
/;r/1
/;r/3
/;/:/:/: /#5D/.No w consider a task en vironmen t class with a v alue function Vi
that sp eci/#0Ces a sto c hasticdeadline uniformly distributed o v er the range /#5B/0 /:/:/: /1/0/#5D/. F or this class/, lopt
/= r/1
r/2
is ab ounded optimal sequence/.
/9It turns out that lU
has higher utilit y than lopt
pro vided it isrun on a mac hine that is four times faster/. W e can see this b y plotting the t w o p erformancepro/#0Cles/: QU
for lU
on /4 M and Qopt
for lopt
on M /. QU
dominates Qopt
/,a ss h o wn in Figure /6/.T o establish that the lU
construction yields UABO programs in general/, w e need tode/#0Cne a notion of w orst/-case p erformance pro/#0Cle/. Let Q
/#03/#28 t/; l /; M /; n/; E /#29 b e the minim umv alue obtained b yi n terrupting l at t /,o v er all E in E of complexit y n /.W e kno w that eac hlj
in lU
satis/#0Ces the follo wing/:/8 l
/0/; n n/#3Enj
/#29 V
/#03j
/#28 lj
/;kj
M/; n /; E /#29 /#15 V
/#03j
/#28 l
/0/;M /;n /; E /#29for constan ts kj
/, nj
/. The aim is to pro v e that/8 Vi
/2V /9 k/; n/0
/8 l
/0/; n n/#3En/0
/#29 V
/#03i
/#28 lU
/;k M /;n /; E /#29 /#15 V
/#03i
/#28 l
/0/;M /;n /; E /#29Giv en the de/#0Cnition of w orst/-case p erformance pro/#0Cle/, it is fairly easy to sho w the follo winglemma /#28the pro of is essen tially iden tical to the pro of of Theorem /1 in Russell and Zilb erstein/,/1/9/9/1/#29/:/9/. Notice that/, in our simple mo del/, the output qualit y of a rule dep ends only on its execution time andnot on the input complexit y /. This also means that w orst/-case and a v erage/-case b eha viour are the same/./6/0/2
Pr o v abl y bounded/-optimal a gents
00.20.40.60.81
5 10 15 20 25 30 35 40Average utility per second
Mean arrival timeBO Sequence
ABO sequenceFigure /7/: Throughput and accuracy impro v emen to f lU
o v er l
iopt
/, as a function of meanarriv al time/, /#15 /= /0/./2/, P oisson arriv als/.Lemma /6 If lU
is a universal pr o gr am in E for V /, and li
is ABO on M in E for Vi
/2V /,then Q
/#03/#28 t/; lU
/;k M /;n /; E /#29 dominates Q
/#03/#28 t/; li
/;M /;n /; E /#29 for k /#15 /4 maxj
kj
/, n/#3E maxj
nj
/.This lemma establishes that/, for a small constan t p enalt y /,w e can ignore the sp eci/#0Cc real/-time nature of the task en vironmen t in constructing b ounded optimal programs/. Ho w ev er/,w e still need to deal with the issue of termination/. It is not p ossible in general for lUto terminate at an appropriate time without access to information concerning the time/-dep endence of the utilit y function/. F or example/, in a /#0Cxed/-time/-cost task en vironmen t/, theappropriate termination time dep ends on the v alue of the time cost c /.F or the general case with deterministic time/-dep endence/, w e can help out lU
b y sup/-plying/, for eac h Vi
/, an /#5Caspiration lev el/" Q
/#03i
/#28 ti
/;li
/;M /;n /; E /#29/, where ti
is the time at whic hli
acts/. lU
terminates when it has completed an lj
suc h that qj
/#15 Q
/#03i
/#28 ti
/;li
/;M /;n /; E /#29/. Byconstruction/, this will happ en no later than ti
b ecause of Lemma /6/.Theorem /9 In task envir onments with deterministic time/-dep endenc e/, an lU
with a suitableaspir ation level is UABO in E on M /.With deadline heralds/, the termination test is somewhat simpler and do es not require an yadditional input to lU
/.Theorem /1/0 In a task envir onment with sto chastic de ad lines/, lU
is UABO in E on M ifit terminates when the her ald arrives/.Returning to the mail/-sorting example/, it is fairly easy to see that lU
/#28whic h consists ofa sequence of net w orks/, lik e the optimal programs for the sto c hastic deadline case/#29 will b eABO in the /#0Cxed/-deadline regime/. It is not so ob vious that it is also ABO in an y particular/6/0/3
R ussell /& Subramaniansto c hastic deadline case /| recall that b oth regimes can b e considered as a single family /.W eh a v e programmed a constructor function for univ ersal programs/, and applied it to themail/-sorter en vironmen t class/. V arying the letter arriv al distribution giv es us di/#0Beren tv aluefunctions Vi
/2V /. Figure /7 sho ws that lU
/#28on /4 M /#29 has higher throughput and accuracythan l
iopt
across the en tire range of arriv al distributions/.Giv en the existence of UABO programs/, it is p ossible to consider the b eha viour of com/-p ositions thereof/. The simplest form of comp osition is functional comp osition/, in whic hthe output of one program is used as input b y another/. More complex/, nested comp osi/-tional structures can b e en tertained/, including lo ops and conditionals /#28Zilb erstein/, /1/9/9/3/#29/.The main issue in constructing UABO comp ositions is ho w to allo cate time among thecomp onen ts/. Pro vided that w e can solv e the time allo cation problem when w e kno w thetotal run time allo w ed/, w e can use the same construction tec hnique as used ab o v e to gen/-erate comp osite UABO programs/, where optimalit y is among all p ossible comp ositions ofthe comp onen ts/. Zilb erstein and Russell /#28/1/9/9/3/#29/, sho w that the allo cation problem can b esolv ed in linear time in the size of the comp osite system/, pro vided the comp osition is a treeof b ounded degree/./7/. Conclusions And F urther W orkW e examined three p ossible formal bases for arti/#0Ccial in telligence/, and concluded thatb ounded optimalit y pro vides the most appropriate goal in constructing in telligen t systems/.W e also noted that similar notions ha v e arisen in philosoph y and game theory for more orless the same reason/: the mismatc hb e t w een classically optimal actions and what w eh a v ecalled fe asible b eha viours/|those that can b e generated b y an agen t program running on acomputing device of /#0Cnite sp eed and size/.W e sho w ed that with careful sp eci/#0Ccation of the task en vironmen t and the computingdevice one can design pro v ably b ounded/-optimal agen ts/. W e exhibited only v ery simpleagen ts/, and it is lik ely that b ounded optimalit y in the strict sense is a di/#0Ecult goal toac hiev e when a larger space of agen t programs is considered/. More relaxed notions suc has asymptotic b ounded optimalit y /#28ABO/#29 ma y pro vide more theoretically robust to ols forfurther progress/. In particular/, ABO promises to yield useful results on comp osite agen tdesigns/, allo wing us to separate the problem of designing complex ABO agen ts in to a discretestructural problem and a con tin uous temp oral optimization problem that is tractable inman y cases/. Hence/, w eh a v e reason to b e optimistic that arti/#0Ccial in telligence can b eusefully c haracterized as the study of b ounded optimalit y /.W em a y sp eculate that pro videdthe computing device is neither to o small /#28so that small c hanges in sp eed or size causesigni/#0Ccan tc hanges in the optimal program design/#29 nor to o p o w erful /#28so that classicallyoptimal decisions can b e computed feasibly/#29/, ABO designs should b e stable o v er reasonablywide v ariations in mac hine sp eed and size and in en vironmen tal complexit y /. The details ofthe optimal designs ma y b e rather arcane/, and learning pro cesses will pla y a large part intheir disco v ery/; w e exp ect that the fo cus of this t yp e of researc h will b e more on questionsof con v ergence to optimalit y for v arious structural classes than on the end result itself/.P erhaps the most imp ortan t implication/, b ey ond the conceptual foundations of the /#0Celditself/, is that researc h on b ounded optimalit y applies/, b y design/, to the pr actic e of arti/#0Ccialin telligence in a w a y that idealized/, in/#0Cnite/-resource mo dels ma y not/. W eh a v e giv en/, b y/6/0/4
Pr o v abl y bounded/-optimal a gentsw a y of illustrating this de/#0Cnition/, a b ounded optimal agen t/: the design of a simple systemconsisting of sequences of decision pro cedures that is pro v ably b etter than an y other programin its class/. A theorem that exhibits a b ounded optimal design translates/, b y de/#0Cnition/,in to an agen t whose actual b eha viour is desirable/.There app ear to b e plen t yo f w orth while directions in whic h to con tin ue the explorationof b ounded optimalit y /. F rom a foundational p oin t of view/, one of the most in terestingquestions is ho w the concept applies to agen ts that can incorp orate a learning comp onen t/./#28Note that in section /5/, the learning algorithm w as external to the agen t/./#29 In suc hacase/, there will not necessarily b e a largely stable b ounded optimal con/#0Cguration if the
agen t program is not large enough/; instead/, the agen t will ha v e to adapt to a shorter/-termhorizon and rewrite itself as it b ecomes obsolete/.With results on the preserv ation of ABO under comp osition/, w e can start to examinem uc h more in teresting arc hitectures than the simple pro duction system studied ab o v e/. F orexample/, w e can lo ok at optimal searc h algorithms/, where the algorithm is constrained toapply a metalev el decision pro cedure at eac h step to decide whic h no de to expand/, if an y/#28Russell /& W efald/, /1/9/8/9/#29/. W e can also extend the w ork on asymptotic b ounded optimalit yto pro vide a utilit y/-based analogue to /#5Cbig/-O/" notation for describing the p erformance ofagen t designs/, including those that are sub optimal/.In the con text of computational learning theory /,i t i s o b vious that the stationarit yrequiremen t on the en vironmen t/, whic h is necessary to satisfy the preconditions of P A Cresults/, is to o restrictiv e/. The fact that the agen t learns ma yh a v e some e/#0Bect on thedistribution of future episo des/, and little is kno wn ab out learning in suc h cases /#28Aldous /&V azirani/, /1/9/9/0/#29/. W e could also relax the deterministic and episo dic requiremen t to allo wnon/-immediate rew ards/, thereb y making connections to curren t researc h on reinforcemen tlearning/.The computation sc heduling problem w e examined is in teresting in itself/, and do es notapp ear to ha v e b een studied in the op erations researc ho rc o m binatorial optimization liter/-ature/. Sc heduling algorithms usually deal with ph ysical rather than computational tasks/,hence the ob jectiv e function usually in v olv es summation of outputs rather than pic king theb est/. W ew ould lik e to resolv e the formal question of its tractabilit y in the general case/, andalso to lo ok at cases in whic h the solution qualities of individual pro cesses are in terdep enden t/#28suc h as when one can use the results of another/#29/. Practical extensions include computationsc heduling for parallel mac hines or m ultiple agen ts/, and sc heduling com binations of compu/-tational and ph ysical /#28e/.g/./, job/-shop and /#0Do w/-shop/#29 pro cesses/, where ob jectiv e functions area com bination of summation and maximization/. The latter extension broadens the scop eof applications considerably /. An industrial pro cess/, suc h as designing and man ufacturing acar/, consists of b oth computational steps /#28design/, logistics/, factory sc heduling/, insp ectionetc/./#29 and ph ysical pro cesses /#28stamping/, assem bling/, pain ting etc/./#29/. One can easily imagineman y other applications in real/-time /#0Cnancial/, industrial/, and military con texts/.It ma y turn out that b ounded optimalit y is found w an ting as a theoretical framew ork/. Ifthis is the case/, w e hop e that it is refuted in an in teresting w a y /, so that a b etter framew orkcan b e created in the pro cess/./6/0/5
R ussell /& SubramanianApp endix/: Additional Pro ofsThis app endix con tains formal pro ofs for three subsidiary lemmata in the main b o dy of thepap er/.Lemma /3 Ther e exists an optimal se quenc e that is sorte d in incr e asing or der of q /'s/.Pro of/: Supp ose this is not the case/, and s is an optimal sequence/. Then there m ust b et w o adjacen t rules i /, i /+ /1 where qi
/#3Eqi /+/1
/#28see Figure /8/#29/. Remo v al of rule i /+ /1 yields asequence s
/0suc h that Qs
/0/#28 t /#29 /#15 Qs
/#28 t /#29/, from Lemma /1 and the fact that ti /+/2
/#14 ti /+/1
/+ ti /+/2
/.B yLemma /2/, s
/0m ust also b e optimal/. W e can rep eat this remo v al pro cess un til s
/0is orderedb y qi
/, pro ving the theorem b y reductio ad absurdum/. /2Lemma /4 F or every se quenc e s /= s/1
/:/:/:sm
sorte d in incr e asing or der of quality/, and singlestep z with qz
/#15 qsm
/, V /#28 sz /#29 /#15 V /#28 s /#29 /.Pro of/: W e calculate V /#28 sz /#29 /, V /#28 s /#29 using Equation /5 and sho w that it is non/-negativ e/:V /#28 sz /#29 /, V /#28 s /#29 /= qz
/#5B/1 /, Pd
/#28/#28
Pmj /=/1
tj
/#29/+ tz
/#29/#5D /, qm
/#5B/1 /, Pd
/#28/#28
Pmj /=/1
tj
/#29/+ tz
/#29/#5D/= /#28 qz
/, qm
/#29/#5B/1 /, Pd
/#28/#28
Pmj /=/1
tj
/#29/+ tz
/#29/#5Dwhic h is non/-negativ e since qz
/#15 qm
/. /2
ti+1qi+2
qi
qi-1
tiq
tti+2
qi+1Figure /8/: Pro of for ordering b y qi
/;l o w er dotted line indicates original pro/#0Cle/; upp er dottedline indicates pro/#0Cle after remo v al of rule i /+/1 /.Lemma /5 Ther e exists an optimal se quenc e whose rules ar e in nonde cr e asing or der of ti
/.Pro of/: Supp ose this is not the case/, and s is an optimal sequence/. Then there m ust b et w o adjacen t rules i /, i /+ /1 where qi
/#14 qi /+/1
and ti
/#3Eti /+/1
/#28see Figure /9/#29/. Remo v al of rule iyields a sequence s
/0suc h that Qs
/0/#28 t /#29 /#15 Qs
/#28 t /#29/, from Lemma /1/. By Lemma /2/, s
/0m ust also b eoptimal/. W e can rep eat this remo v al pro cess un til s
/0is ordered b y ti
/, pro ving the theoremb y reductio ad absurdum/. /2/6/0/6
Pr o v abl y bounded/-optimal a gents
ti+1qi+1
qi
qi-1
tiq
tti+1Figure /9/: Pro of for ordering b y ti
/; dotted line indicates pro/#0Cle after remo v al of rule i /.Ac kno wledgemen tsW ew ould lik et oa c kno wledge stim ulating discussions with Mic hael F ehling/, Mic hael Gene/-sereth/, Russ Greiner/, Eric Horvitz/, Henry Kautz/, Daphne Koller/, and Bart Selman on thesub ject of b ounded optimalit y/; with Dorit Ho c h baum/, Nimro d Megiddo/, and Kevin Glaze/-bro ok on the sub ject of dynamic programming for sc heduling problems/; and with Nic kLittlestone and Mic hael Kearns on the sub ject of agnostic learning/. W ew ould also lik et othank the review ers for their man y constructiv e suggestions/. Man y of the early ideas onwhic h this w ork is based arose in discussions with the late Eric W efald/. Thanks also toRon P arr for his w ork on the uniform/-distribution case/, Rhonda Righ ter for extending theresults to the exp onen tial distribution/, and P atric k Ziesk e for help in implemen ting the dy/-namic programming algorithm/. The /#0Crst author w as supp orted b y NSF gran ts IRI/-/8/9/0/3/1/4/6/,IRI/-/9/2/1/1/5/1/2 and IRI/-/9/0/5/8/4/2/7/, b y a visiting fello wship from the SER C while on sabbaticalin the UK/, and b y the NEC Researc h Institute/. The second author w as supp orted b y NSFgran t IRI/-/8/9/0/2/7/2/1/.ReferencesAgre/, P /./, /& Chapman/, D/. /#28/1/9/8/7/#29/. P engi/: An implemen tation of a theory of activit y /. InPr o c/. /6th National Confer enc eo n A rti/#0Ccial Intel ligenc e/, Se attle/, W A /. Morghan Kauf/-mann/.Aldous/, D/./, /& V azirani/, U/. /#28/1/9/9/0/#29/. A mark o vian extension of v alian t/'s learning mo del/. InPr o c/. /3/1st A nnual Symp osium on F oundations of Computer Scienc e/, St/. L ouis/, MO /.IEEE Comput/. So c/. Press/.Binder/, J/. /#28/1/9/9/4/#29/. On the complexit y of delib eration sc heduling with sto c hastic deadlines/./.Boser/, B/. E/./, Sac kinger/, E/./, Bromley /, J/./, /& LeCun/, Y/. /#28/1/9/9/2/#29/. Hardw are requiremen ts forneural net w ork pattern classi/#0Cers /| a case study and implemen tation/. IEEE Micr o /,/1/2 /, /3/2/#7B/4/0/.Brandt/, R/. /#28/1/9/5/3/#29/. In searc h of a credible form of rule utilitarianism/. In Nakhnikian/, G/./, /&Castaneda/, H/. /#28Eds/./#29/, Mor ality and the L anguage of Conduct /./6/0/7
R ussell /& SubramanianBreese/, J/. S/./, /& F ehling/, M/. R/. /#28/1/9/9/0/#29/. Con trol of problem/-solving/: Principles and arc hitec/-ture/. In Shac h ter/, R/. D/./, Levitt/, T/./, Kanal/, L/./, /& Lemmer/, J/. /#28Eds/./#29/, Unc ertainty inA rti/#0Ccial Intel ligenc e/4 /. North Holland/: Amsterdam/.Bro oks/, R/. A/. /#28/1/9/8/6/#29/. A robust/, la y ered con trol system for a mobile rob ot/. IEEE Journalof R ob otics and A utomation /, /2 /, /1/4/#7B/2/3/.Cherniak/, C/. /#28/1/9/8/6/#29/. Minimal r ationality /. MIT Press/: Cam bridge/.Dean/, T/./, /& Bo ddy /, M/. /#28/1/9/8/8/#29/. An analysis of time/-dep enden t planning/. In Pr o c/. of AAAI/-/8/8 /, pp/. /4/9/#7B/5/4/.Dean/, T/. L/./, /& W ellman/, M/. P /. /#28/1/9/9/1/#29/. Planning and c ontr ol /. Morgan Kaufmann/: SanMateo/, CA/.Dennett/, D/. /#28/1/9/8/6/#29/. The moral /#0Crst aid man ual/. T anner lectures on h uman v alues/, Univ ersit yof Mic higan/.Do yle/, J/. /#28/1/9/8/3/#29/. What is rational psyc hology/? to w ard a mo dern men tal philosoph y /. AIMagazine /, /4 /, /5/0/#7B/5/3/.Do yle/, J/. /#28/1/9/8/8/#29/. Arti/#0Ccial in telligence and rational self/-go v ernmen t/. T ec h/. rep/./. T ec hnicalrep ort CMU/-CS/-/8/8/-/1/2/4/.Do yle/, J/./, /& P atil/, R/. /#28/1/9/9/1/#29/. Tw o theses of kno wledge represen tation/: language restric/-tions/, taxonomic classi/#0Ccation/, and the utilit y of represen tation services/. A rti/#0Ccialintel ligenc e /, /4/8 /, /2/6/1/#7B/2/9/7/.Etzioni/, O/. /#28/1/9/8/9/#29/. T ractable decision/-analytic con trol/. In Pr o c/. of /1st International Con/-fer enc e on Know le dge R epr esentation and R e asoning /, pp/. /1/1/4/#7B/1/2/5/.F ehling/, M/./, /& Russell/, S/. J/. /#28/1/9/8/9/#29/. Pr o c e e dings of the AAAI Spring Symp osium on Limite dR ationality /. AAAI/.Genesereth/, M/. R/./, /& Nilsson/, N/. J/. /#28/1/9/8/7/#29/. L o gic al F oundations of A rti/#0Ccial Intel ligenc e /.Morgan Kaufmann/: Mateo/, CA/.Go o d/, I/. J/. /#28/1/9/7/1/#29/. Tw en t y/-sev en principles of rationalit y /. I nI nG o d a m be /, V /. P /./, /& Sprott/,D/. A/. /#28Eds/./#29/, F oundations of Statistic al Infer enc e /, pp/. /1/0/8/#7B/1/4/1/. Holt/, Rinehart/, Win/-ston/./: T oron to/.Hansson/, O/./, /& Ma y er/, A/. /#28/1/9/8/9/#29/. Heuristic searc h as eviden tial reasoning/. In Pr o c e e dingsof the Fifth Workshop on Unc ertainty in A rti/#0Ccial Intel ligenc e/, Windsor/, Ontario /.Horvitz/, E/. J/. /#28/1/9/8/8/#29/. Reasoning ab out b eliefs and actions under computational resourceconstrain ts/. In Levitt/, T/./, Lemmer/, J/./, /& Kanal/, L/. /#28Eds/./#29/, Unc ertainty in A rti/#0CcialIntel ligenc e/3 /. North Holland/: Amsterdam/.Kearns/, M/./, Sc hapire/, R/./, /& Sellie/, L/. /#28/1/9/9/2/#29/. T o w ard e/#0Ecien t agnostic learning/. In Pr o c/. /5thA nn/. Workshop on Computational L e arning The ory/, Pittsbur gh/, P A /. Morgan Kauf/-mann/./6/0/8
Pr o v abl y bounded/-optimal a gentsKeeney /, R/./, /& Rai/#0Ba/, H/. /#28/1/9/7/6/#29/. De cisions with multiple obje ctives/: Pr efer enc es and valuetr ade o/#0Bs /. Wiley/: New Y ork/.Lev esque/, H/./, /& Brac hman/, R/. /#28/1/9/8/7/#29/. Expressiv eness and tractabilit y in kno wledge repre/-sen tation and reasoning/. Computational Intel ligenc e /, /3 /, /7/8/#7B/9/3/.Lub y /, M/./, Sinclair/, A/./, /& Zuc k erman/, D/. /#28/1/9/9/3/#29/. Optimal sp eedup of las v egas algorithms/.Information Pr o c essing L etters /, /4/7 /, /1/7/3/#7B/8/0/.McCarth y /, J/. /#28/1/9/5/8/#29/. Programs with common sense/. In Pr o c e e dings of the Symp osium onthe Me chanization of Thought Pr o c esses/, T e ddington/, England/: HMSO /.New ell/, A/. /#28/1/9/8/1/#29/. The kno wledge lev el/. AI Magazine /, /2 /, /1/#7B/2/0/.Neyman/, A/. /#28/1/9/8/5/#29/. Bounded complexit y justi/#0Ces co op eration in the /#0Cnitely rep eated pris/-oners/' dilemma/. Ec onomics L etters /, /1/9 /, /2/2/7/#7B/2/2/9/.P apadimitriou/, C/./, /& Y annak akis/, M/. /#28/1/9/9/4/#29/. On complexit y as b ounded rationalit y /. InPr o c/. A CM Symp osium on the The ory of Computation /.Ramsey /,F /.P /. /#28/1/9/3/1/#29/. T ruth and probabilit y /. In Braith w aite/, R/. /#28Ed/./#29/, The foundations ofmathematics and other lo gic al essays /. Harcourt Brace Jo v ano vic h/: New Y ork/.Russell/, S/. J/./, /& W efald/, E/. H/. /#28/1/9/8/9a/#29/. On optimal game tree searc h using rational metar/-easoning/. In Pr o c/. IJCAI/-/8/9 /.Russell/, S/. J/./, /& W efald/, E/. H/. /#28/1/9/8/9b/#29/. Principles of metareasoning/. In Pr o c/. KR/-/8/9 /.Russell/, S/. J/./, /& W efald/, E/. H/. /#28/1/9/9/1/#29/. Do the right thing/: Studies in limite dr ationality /.MIT Press/: Cam bridge/, MA/.Russell/, S/. J/./, /& Zilb erstein/, S/. /#28/1/9/9/1/#29/. Comp osing real/-time systems/. In Pr o c/. IJCAI/-/9/1/,Sydney /.Sac kinger/, E/./, Boser/, B/. E/./, Bromley /, J/./, /& LeCun/, Y/. /#28/1/9/9/2/#29/. Application of the annaneural net w ork c hip to high/-sp eed c haracter recognition/. IEEE T r ansactions on Neur alNetworks /, /3 /, /4/9/8/#7B/5/0/5/.Simon/, H/. A/. /#28/1/9/7/6/#29/. On ho w to decide what to do/. In Mo dels of b ounde dr ationality/,V olume /2 /.Simon/, H/. A/. /#28/1/9/8/2/#29/. Mo dels of b ounde dr ationality/, V olume /2 /. MIT Press/: Cam bridge/.v on Neumann/, J/./, /& Morgenstern/, O/. /#28/1/9/4/7/#29/. The ory of games and e c onomic b ehavior /.Princeton Univ ersit y Press/: Princeton/.Zilb erstein/, S/. /#28/1/9/9/3/#29/. Op er ational R ationality Thr ough Compilation of A nytime A lgorithms /.Ph/.D/. thesis/, Computer Science Division/, Univ ersit y of California/, Berk eley /.Zilb erstein/, S/./, /& Russell/, S/. /#28/1/9/9/3/#29/. Optimal comp osition of real/-time systems/. Submittedto A rti/#0Ccial Intel ligenc e /./6/0/9