Artificial Intelligence A Modern Approach

Second Edition

FORSYTH & PONCE GRAHAM

JURAFSKY & MARTIN NEAPOLITAN RUSSELL & NORVIG

Computer Vision: A Modern Approach ANSI Common Lisp

Speech and Language Processing Learning Bayesian Networks

Artificial Intelligence: A Modern Approach

Artificial Intelligence A Modern Approach

Second Edition

Stuart J. Russell and Peter Norvig

Contributing writers: John FE. Canny Douglas D. Edwards Jitendra M. Malik Sebastian Thrun

|SESI(ES Hall

Pearson Education International

If you purchased this book within the United States or Canada you should be aware that it has been wrongfully imported without the approval of the Publisher or the Author.

Vice President and Editorial Director, ECS: Marcia J. Horton

Publisher: Alan R. Apt

Associate Editor: Toni Dianne Holm

Editorial Assistant: Patrick Lindner

Vice President and Director of Production and Manufacturing, ESM: David W. Riccardi Executive Managing Editor: Vince O’Brien

Assistant Managing Editor: Camille Trentacoste

Production Editor: Irwin Zucker

Manufacturing Manager: Trudy Pisciotti

Manufacturing Buyer: Lisa McDowell

Director, Creative Services: Paul Belfanti

Creative Director: Carole Anson

Art Editor: Greg Dulles

Art Director: Heather Scott

Assistant to Art Director: Geoffrey Cassar

Cover Designers: Stuart Russell and Peter Norvig

Cover Image Creation: Stuart Russell and Peter Norvig; Tamara Newnam and Patrice Van Acker Interior Designer: Stuart Russell and Peter Norvig

Marketing Manager: Pamela Shaffer

Marketing Assistant: Barrie Reinhold

[OIE © 2003, 1995 by Pearson Education, Inc. ERI Pearson Education, Inc., B® {Jpper Saddle River, New Jersey 07458

All rights reserved. No part of this book may be reproduced, in any form or by any means, without permission in writing from the publisher.

The author and publisher of this book have used their best efforts in preparing this book. These efforts include the development, research, and testing of the theories and programs to determine their effectiveness. The author and publisher make no warranty of any kind, express or implied, with regard to these programs or the documentation contained in this book. The author and publisher shall not be liable in any event for incidental or consequential damages in connection with, or arising out of, the furnishing, performance,

or use of these programs.

Printed in the United States of America

109876543

ISBN 0-13-080302-¢

Pearson Education Ltd., London

Pearson Education Australia Pty. Ltd., Sydney

Pearson Education Singapore, Pte. Ltd.

Pearson Education North Asia Ltd., Hong Kong Pearson Education Canada, Inc., Toronto

Pearson Educacién de Mexico, S.A. de C.V.

Pearson Education—Japan, Tokyo

Pearson Education Malaysia, Pte. Ltd.

Pearson Education, Inc., Upper Saddle River, New Jersey

For Loy, Gordon, and Lucy — SJ.R.

For Kris, Isabella, and Juliet — P.N.

Preface

Artificial Intelligence (AI) is a big field, and this is a big book. We have tried to explore the full breadth of the field, which encompasses logic, probability, and continuous mathematics; perception, reasoning, learning, and action; and everything from microelectronic devices to robotic planetary explorers. The book is also big because we go into some depth in presenting results, although we strive to cover only the most central ideas in the main part of each chapter. Pointers are given to further results in the bibliographical notes at the end of each chapter.

The subtitle of this book is “A Modern Approach.” The intended meaning of this rather empty phrase is that we have tried to synthesize what is now known into a common framework, rather than trying to explain each subfield of Al in its own historical context. We apologize to those whose subfields are, as a result, less recognizable than they might otherwise have been.

The main unifying theme is the idea of an intelligent agent. We define Al as the study of agents that receive percepts from the environment and perform actions. Each such agent implements a function that maps percept sequences to actions, and we cover different ways to represent these functions, such as production systems, reactive agents, real-time conditional planners, neural networks, and decision-theoretic systems. We explain the role of learning as extending the reach of the designer into unknown environments, and we show how that role constrains agent design, favoring explicit knowledge representation and reasoning. We treat robotics and vision not as independently defined problems, but as occurring in the service of achieving goals. We stress the importance of the task environment in determining the appropriate agent design.

Our primary aim is to convey the ideas that have emerged over the past fifty years of Al research and the past two millenia of related work. We have tried to avoid excessive formality in the presentation of these ideas while retaining precision. Wherever appropriate, we have included pseudocode algorithms to make the ideas concrete; our pseudocode is described briefly in Appendix B. Implementations in several programming languages are available on the book’s Web site, aima.cs.berkeley.edu.

This book is primarily intended for use in an undergraduate course or course sequence. It can also be used in a graduate-level course (perhaps with the addition of some of the primary sources suggested in the bibliographical notes). Because of its comprehensive coverage and large number of detailed algorithms, it is useful as a primary reference volume for Al graduate students and professionals wishing to branch out beyond their own subfield. The only prerequisite is familiarity with basic concepts of computer science (algorithms, data structures, complexity) at a sophomore level. Freshman calculus is useful for understanding neural networks and statistical learning in detail. Some of the required mathematical background is supplied in Appendix A.

Overview of the book

The book is divided into eight parts. Part 1, Artificial Intelligence, offers a view of the Al enterprise based around the idea of intelligent agents—systems that can decide what to do and then do it. Part IL, Problem Solving, concentrates on methods for deciding what to do when one needs to think ahead several steps—for example in navigating across a country or playing chess. Part III, Knowledge and Reasoning, discusses ways to represent knowledge about the world—how it works, what it is currently like, and what one’s actions might do—and how to reason logically with that knowledge. Part IV, Planning, then discusses how to use these reasoning methods to decide what to do, particularly by constructing plans. Part V, Uncertain Knowledge and Reasoning, is analogous to Parts III and IV, but it concentrates on reasoning and decision making in the presence of uncertainty about the world, as might be faced, for example, by a system for medical diagnosis and treatment.

Together, Parts [I-V describe that part of the intelligent agent responsible for reaching decisions. Part VI, Learning, describes methods for generating the knowledge required by these decision-making

vil

viii

Preface

components. Part VII, Communicating, Perceiving, and Acting, describes ways in which an intelligent agent can perceive its environment so as to know what is going on, whether by vision, touch, hearing, or understanding language, and ways in which it can turn its plans into real actions, either as robot motion or as natural language utterances. Finally, Part VIII, Conclusions, analyzes the past and future of Al and the philosophical and ethical implications of artificial intelligence.

Changes from the first edition

Much has changed in Al since the publication of the first edition in 1995, and much has changed in this book. Every chapter has been significantly rewritten to reflect the latest work in the field, to reinterpret old work in a way that is more cohesive with new findings, and to improve the pedagogical flow of ideas. Followers of Al should be encouraged that current techniques are much more practical than those of 1995; for example the planning algorithms in the first edition could generate plans of only dozens of steps, while the algorithms in this edition scale up to tens of thousands of steps. Similar orders-of-magnitude improvements are seen in probabilistic inference, language processing, and other subfields. The following are the most notable changes in the book:

o In Part I, we acknowledge the historical contributions of control theory, game theory, economics, and neuroscience. This helps set the tone for a more integrated coverage of these ideas in subsequent chapters.

e In Part II, online search algorithms are covered and a new chapter on constraint satisfaction has been added. The latter provides a natural connection to the material on logic.

o In Part III, propositional logic, which was presented as a stepping-stone to first-order logic in the first edition, is now presented as a useful representation language in its own right, with fast inference algorithms and circuit-based agent designs. The chapters on first-order logic have been reorganized to present the material more clearly and we have added the Internet shopping domain as an example.

e In Part IV, we include newer planning methods such as GRAPHPLAN and satisfiability-based planning, and we increase coverage of scheduling, conditional planning, hierarchical planning, and multiagent planning.

o In Part V, we have augmented the material on Bayesian networks with new algorithms, such as variable elimination and Markov Chain Monte Carlo, and we have created a new chapter on uncertain temporal reasoning, covering hidden Markov models, Kalman filters, and dynamic Bayesian networks. The coverage of Markov decision processes is deepened, and we add sections on game theory and mechanism design.

o In Part VI, we tie together work in statistical, symbolic, and neural learning and add sections on boosting algorithms, the EM algorithm, instance-based learning, and kernel methods (support vector machines).

o In Part VII, coverage of language processing adds sections on discourse processing and grammar induction, as well as a chapter on probabilistic language models, with applications to information retrieval and machine translation. The coverage of robotics stresses the integration of uncertain sensor data, and the chapter on vision has updated material on object recognition.

o In Part VIII, we introduce a section on the ethical implications of AL

Using this book

The book has 27 chapters, each requiring about a week’s worth of lectures, so working through the whole book requires a two-semester sequence. Alternatively, a course can be tailored to suit the interests of the instructor and student. Through its broad coverage, the book can be used to support such

Preface

ix

courses, whether they are short, introductory undergraduate courses or specialized graduate courses on advanced topics. Sample syllabi from the more than 600 universities and colleges that have adopted the first edition are shown on the Web at aima.cs.berkeley.edu, along with suggestions to help you find a sequence appropriate to your needs.

The book includes 385 exercises. Exercises requiring significant programming are marked with a keyboard icon. These exercises can best be solved by taking advantage of the code repository at aima.cs.berkeley.edu. Some of them are large enough to be considered term projects. A number of exercises require some investigation of the literature; these are marked with a book icon.

Throughout the book, important points are marked with a pointing icon. We have included an extensive index of around 10,000 items to make it easy to find things in the book. Wherever a new term is first defined, it is also marked in the margin.

Using the Web site

At the aima.cs.berkeley.edu Web site you will find:

implementations of the algorithms in the book in several programming languages,

a list of over 600 schools that have used the book, many with links to online course materials, an annotated list of over 800 links to sites around the web with useful Al content,

a chapter by chapter list of supplementary material and links,

instructions on how to join a discussion group for the book,

instructions on how to contact the authors with questions or comments,

instructions on how to report errors in the book, in the likely event that some exist, and

copies of the figures in the book, along with slides and other material for instructors.

Acknowledgments

Jitendra Malik wrote most of Chapter 24 (on vision). Most of Chapter 25 (on robotics) was written by Sebastian Thrun in this edition and by John Canny in the first edition. Doug Edwards researched the historical notes for the first edition. Tim Huang, Mark Paskin, and Cynthia Bruyns helped with formatting of the diagrams and algorithms. Alan Apt, Sondra Chavez, Toni Holm, Jake Warde, Irwin Zucker, and Camille Trentacoste at Prentice Hall tried their best to keep us on schedule and made many helpful suggestions on the book’s design and content.

Stuart would like to thank his parents for their continued support and encouragement and his wife, Loy Sheflott, for her endless patience and boundless wisdom. He hopes that Gordon and Lucy will soon be reading this. RUGS (Russell’s Unusual Group of Students) have been unusually helpful.

Peter would like to thank his parents (Torsten and Gerda) for getting him started, and his wife (Kris), children, and friends for encouraging and tolerating him through the long hours of writing and longer hours of rewriting.

We are indebted to the librarians at Berkeley, Stanford, MIT, and NASA, and to the developers of CiteSeer and Google, who have revolutionized the way we do research.

We can’t thank all the people who have used the book and made suggestions, but we would like to acknowledge the especially helpful comments of Eyal Amir, Krzysztof Apt, Ellery Aziel, Jeff Van Baalen, Brian Baker, Don Barker, Tony Barrett, James Newton Bass, Don Beal, Howard Beck, Wolfgang Bibel, John Binder, Larry Bookman, David R. Boxall, Gerhard Brewka, Selmer Bringsjord, Carla Brodley, Chris Brown, Wilhelm Burger, Lauren Burka, Joao Cachopo, Murray Campbell, Norman Carver, Emmanuel Castro, Anil Chakravarthy, Dan Chisarick, Roberto Cipolla, David Cohen, James Coleman, Julie Ann Comparini, Gary Cottrell, Ernest Davis, Rina Dechter, Tom Dietterich, Chuck Dyer, Barbara Engelhardt, Doug Edwards, Kutluhan Erol, Oren Etzioni, Hana Filip, Douglas

Preface

Fisher, Jeffrey Forbes, Ken Ford, John Fosler, Alex Franz, Bob Futrelle, Marek Galecki, Stefan Gerberding, Stuart Gill, Sabine Glesner, Seth Golub, Gosta Grahne, Russ Greiner, Eric Grimson, Barbara Grosz, Larry Hall, Steve Hanks, Othar Hansson, Ernst Heinz, Jim Hendler, Christoph Herrmann, Vasant Honavar, Tim Huang, Seth Hutchinson, Joost Jacob, Magnus Johansson, Dan Jurafsky, Leslie Kaelbling, Keiji Kanazawa, Surekha Kasibhatla, Simon Kasif, Henry Kautz, Gernot Kerschbaumer, Richard Kirby, Kevin Knight, Sven Koenig, Daphne Koller, Rich Korf, James Kurien, John Lafferty, Gus Larsson, John Lazzaro, Jon LeBlanc, Jason Leatherman, Frank Lee, Edward Lim, Pierre Louveaux, Don Loveland, Sridhar Mahadevan, Jim Martin, Andy Mayer, David McGrane, Jay Mendelsohn, Brian Milch, Steve Minton, Vibhu Mittal, Leora Morgenstern, Stephen Muggleton, Kevin Murphy, Ron Musick, Sung Myaeng, Lee Naish, Pandu Nayak, Bernhard Nebel, Stuart Nelson, XuanLong Nguyen, Illah Nourbakhsh, Steve Omohundro, David Page, David Palmer, David Parkes, Ron Parr, Mark Paskin, Tony Passera, Michael Pazzani, Wim Pijls, Ira Pohl, Martha Pollack, David Poole, Bruce Porter, Malcolm Pradhan, Bill Pringle, Lorraine Prior, Greg Provan, William Rapaport, Philip Resnik, Francesca Rossi, Jonathan Schaeffer, Richard Scher], Lars Schuster, Soheil Shams, Stuart Shapiro, Jude Shavlik, Satinder Singh, Daniel Sleator, David Smith, Bryan So, Robert Sproull, Lynn Stein, Larry Stephens, Andreas Stolcke, Paul Stradling, Devika Subramanian, Rich Sutton, Jonathan Tash, Austin Tate, Michael Thielscher, William Thompson, Sebastian Thrun, Eric Tiedemann, Mark Torrance, Randall Upham, Paul Utgoff, Peter van Beek, Hal Varian, Sunil Vemuri, Jim Waldo, Bonnie Webber, Dan Weld, Michael Wellman, Michael Dean White, Kamin Whitehouse, Brian Williams, David Wolfe, Bill Woods, Alden Wright, Richard Yen, Weixiong Zhang, Shlomo Zilberstein, and the anonymous reviewers provided by Prentice Hall.

About the Cover

The cover image was designed by the authors and executed by Lisa Marie Sardegna and Maryann Simmons using SGI Inventor ™ and Adobe Photoshop™. The cover depicts the following items from the history of Al:

p—

Aristotle’s planning algorithm from De Motu Animalium (c. 400 B.C.).

Ramon Lull’s concept generator from Ars Magna (c. 1300 A.D.).

Charles Babbage’s Difference Engine, a prototype for the first universal computer (1848). Gottlob Frege's notation for first-order logic (1789).

Lewis Carroll's diagrams for logical reasoning (1886).

Sewall Wright's probabilistic network notation (1921).

Alan Turing (1912-1954).

Shakey the Robot (1969-1973).

A modern diagnostic expert system (1993).

OX NN Wo

About the Authors

Stuart Russell was born in 1962 in Portsmouth, England. He received his B.A. with first-class honours in physics from Oxford University in 1982, and his Ph.D. in computer science from Stanford in 1986. He then joined the faculty of the University of California at Berkeley, where he is a professor of computer science, director of the Center for Intelligent Systems, and holder of the Smith-Zadeh Chair in Engineering. In 1990, he received the Presidential Young Investigator Award of the National Science Foundation, and in 1995 he was cowinner of the Computers and Thought Award. He was a 1996 Miller Professor of the University of California and was appointed to a Chancellor’s Professorship in 2000. In 1998, he gave the Forsythe Memorial Lectures at Stanford University. He is a Fellow and former Executive Council member of the American Association for Artificial Intelligence. He has published over 100 papers on a wide range of topics in artificial intelligence. His other books include The Use of Knowledge in Analogy and Induction and (with Eric Wefald) Do the Right Thing: Studies in Limited Rationality.

Peter Norvig is director of Search Quality at Google, Inc. He is a Fellow and Executive Council member of the American Association for Artificial Intelligence. Previously, he was head of the Computational Sciences Division at NASA Ames Research Center, where he oversaw NASA’s research and development in artificial intelligence and robotics. Before that he served as chief scientist at Junglee, where he helped develop one of the first Internet information extraction services, and as a senior scientist at Sun Microsystems Laboratories working on intelligent information retrieval. He received a B.S. in applied mathematics from Brown University and a Ph.D. in computer science from the University of California at Berkeley. He has been a professor at the University of Southern California and a research faculty member at Berkeley. He has over 50 publications in computer science including the books Paradigms of AI Programming: Case Studies in Common Lisp, Verbmobil: A Translation System for Face-to-Face Dialog, and Intelligent Help Systems for UNIX.

x1

Summary of Contents

I Artificial Intelligence

1 Introduction. ..........iiieiuieiiiiinniinneniiiiiiiasssssessssssncncsnsnsnsans 1 2 Intelligent AGentS......ouoeninin iene inirineiscassorssasiencsncesnsssasaenns 32 IT Problem-solving 3 Solving Problems by Searching...........c.coiiiiiiiiniiiiiiniiiiiiiiniininnen. 59 4 Informed Search and Exploration ............cciiuiiiiiiiiiiiiiiiennceneenanes 94 5 Constraint Satisfaction Problems. .............ccoiiiiiiiiiiiiiiiiiiiiiiinina... 137 6 Adversarial Search ..........ooiiuiiiiiiiiiii iii iii irre cie ieee 161 IIT Knowledge and reasoning 7 Logical ABents......c.ouveiiiniiiiiiusastninisiiassstassrsstnrrscansasssnnns 194 8 First-Order Logic..........coiiiiiiiiiiiiiiiiiiiii iii iiataiiii ii icasearnasnns 240 9 Inference in First-Order Logic .........ouitiiiiiiiniiinnniennscsntecersnnsncces 272 10 Knowledge Representation ..........c.oeivieienineinnieciiacnsiecinsnennconnens 320 IV Planning 8 0 1111 375 12 Planning and Acting inthe Real World ............cooiiiiiiiiiiiiiiiianiienenn, 417 V Uncertain knowledge and reasoning 13 Uncertainty «.ovvuereennenn inne inn eneseeeneanconsansesseceacessnscsscsnrenssens 462 14 Probabilistic Reasoning .........cocvieiiininiiiieniiiiiininieiiessisinsnsnsncens 492 15 Probabilistic Reasoning over Time......cccoviiiiiiiiiiiiiiiniiiieiiinnnnnieee, 537 16 Making Simple Decisions. .......ocvieieininiiiiiiiiiininiisneiiiiiensnsseannss 584 17 Making Complex Decisions. .... cou. iiiiiiaiaiiiniiiieiiiiiniiaiateincncansanas 613 VI Learning 18 Learning from Observations ...........cocoiiiiienniiiieininitinenieisasnscnss 649 19 Knowledge in Learning .....coouvuieieiinieeitnetneninisnressesnnssccenrssncnns 678 20 Statistical Learning Methods. ..........cciiuiiuiiiieiiiiiiiniincanrecencinran.. 712 21 Reinforcement Learning .........ooiiiutennrenceeecennsenncsntecesacsssnnsencs 763 VII Communicating, perceiving, and acting 22 Communication .......ccovvueinieniiniiienaeenneiinsnsscssssscenssessacnsanses 790 23 Probabilistic Language Processing. ......couooviuiiiineiiienteiineinsesanssnnens 834 7 2 {171 1) 1 863 25 RODOICS. ceiver iiianse ii intetiuarasssnsesnseescsssssnsscassesnseasssssnnens 901 VIII Conclusions 26 Philosophical Foundations ........c.ccoiuiiiiintiinniinseensecsesncasssnssscces 947 27 AL: Present and Future . ......coovitiiiiiiineiinnecaaineinncanensennsnrenranses 968 A Mathematical background ..........c.cieiiuiuiiiiiiiiiscsntereciciciernrnnaenss 977 B Notes on Languages and Algorithms...........c.coiiiiiiiiiiirniiiiiiirnnenenne. 984 Bibliography 987 Index 1045

xiii

LT Whatis AI? . ©... ee Acting humanly: The Turing Test approach . . . . . . ................ Thinking humanly: The cognitive modeling approach . . . . . ........... Thinking rationally: The “laws of thought” approach. . . . . ............ Acting rationally: The rational agent approach . . . . ................

1.2 The Foundations of Artificial Intelligence . . . . . . . ................ Philosophy (428 B.C.—present) . . . . . . . oc Mathematics (c. 800—present) . . . . . . . oe Economics (1776—present) . . . . . 00 ee Neuroscience (1861—present) . . . . . . . eee Psychology (1879—present) . . . . . . . Computer engineering (1940-present) . . . . . . .. oo Control theory and Cybernetics (1948—present) . . . . . . . . ............ Linguistics (1957—present) . . . . . . oe

1.3 The History of Artificial Intelligence . . . . . .................... The gestation of artificial intelligence (1943-1955). . . . . . . . .......... The birth of artificial intelligence (1956) . . . . . . . ................ Early enthusiasm, great expectations (1952-1969) . . . .. .. ........... Adose of reality (1966-1973) . . . . . . Lo Knowledge-based systems: The key to power? (1969-1979) . . . . . ........ Al becomes an industry (1980—present) . . . . . . . . The return of neural networks (1986—present) . . . . . . .............. Al becomes a science (1987—present) . . . . . . . The emergence of intelligent agents (1995—present) . . . . . ............

14 The Stateofthe Art . . . . LL

LS Summary. . o.oo

Bibliographical and Historical Notes . . . . . . . .. 0...

EXercises . . . eee

2 Intelligent Agents

2.1 Agents and Environments . . . . o.oo ce

2.2 Good Behavior: The Concept of Rationality . . . .................. Performance measures . . . . . . Lo Lo Rationality . . . . . 0. ee Omniscience, learning, and autonomy . . . . . . . . . 00

23 TheNature of Environments. . . . . . . Specifying the task environment . . . . . . .. LL 0 00 Properties of task environments . . . . . .. Loo L

24 The Structure of Agents . . . . . . Le

AZENE Programs . . . . . «o.oo e Simple reflex agents . . . . . . LL ee Model-based reflex agents . . . . . . . LL

XV

~N ha RRL

LR BBB BBD BIBI Dd ms hm hm pm rm pm pm em ee SOO INN hh ~=0ITOAONWVEADNOWO

xvi Contents

Goal-based agents . . . . . .. Le 49 Utility-based agents . . . . . . . LL ee 51 Learning agents . . . . . . . . oe 51 25 SUMMAELY ©... coo ee ee 54 Bibliographical and Historical Notes . . . . .. ............... . ....... 55 EXEICISEs . . . . . © co ee 56

IT Problem-solving

3 Solving Problems by Searching 59 3.1 Problem-Solving Agents. . . . . . Le 59 Well-defined problems and solutions. . . . .. ............. ...... 62 Formulating problems . . . . . . Lo oo 62

32 Example Problems . . . . . . .. 64 Toy problems . . . . Le 64 Real-world problems . . . . . . oo 67

33 Searching for Solutions . . . .. ..... ee 69 Measuring problem-solving performance . . . . .. . . LL 71

3.4 Uninformed Search Strategies . . . . . . . . . . 0 73 Breadth-firstsearch . . . . . . o.oo oo 73 Depth-first search . . . o.oo a 75 Depth-limited search . . . . . oo... ooo a 77 Iterative deepening depth-firstsearch . . . . . . ......... LL 78 Bidirectional search . . . . . . . LLL oa 79 Comparing uninformed search strategies . . . . . . . ...... 81

3.5 Avoiding Repeated States . . . . . . . . oe 81 3.6 Searching with Partial Information . . . ............... .. 0... 83 Sensorless problems . . . LLL oo 84 Contingency problems . . . . . . . LL. oe 86

37 SUMMATY . . . oo cco ee 87 Bibliographical and Historical Notes . . . . . .... ........... ... ...... 88 EXEICiSeS . . «oo eee 89 4 Informed Search and Exploration 94 4.1 Informed (Heuristic) Search Strategies . . . . . ................... 94 Greedy best-firstsearch . . . . . . LL... Lee 95

A\* search: Minimizing the total estimated solutioncost . . . . ........... 97 Memory-bounded heuristic search. . . . o.oo oo oo 101 Learning to search better . . . . . . .. . 104

42 Heuristic Functions . . . . .. .. oe 105 The effect of heuristic accuracy on performance . . . . .. ............. 106 Inventing admissible heuristic functions . . . . ...... 0 LLL. 107 Learning heuristics from experience . . . . . . . Lo. 109

43 Local Search Algorithms and Optimization Problems . . . . . ........... 110 Hill-climbing search . . . . . . . . ee 111 Simulated annealing search . . . . LL... oo oo Lo 115 Localbeamsearch . . . . . .. . 115 Genetic algorithms . . . . . . . «LL ee 116

44 Local Search in Continuous Spaces . . . . . . ..... 119

Contents XVvii

4.5 Online Search Agents and Unknown Environments . . . . . . ........... 122

Online search problems . . . . . . oo 123

Online search agents . . . . . . viii 125

Online local search... . . . oa 126

Learning inonlinesearch . . . . . . . 0 127

4.6 SUMMATY . «© © © cco ee ee ee 129

Bibliographical and Historical Notes . . . . . . . ............ ... ...... 130

EXErcises . . . . . . 134

5 Constraint Satisfaction Problems 137

5.1 Constraint Satisfaction Problems . . . . . . . 000. 137

5.2 Backtracking Search for CSPs . . . . . o.oo oo ooo 141

Variable and value ordering . . . . . . . ooo 143

Propagating information through constraints . . . . . ... oo 0 144

Intelligent backtracking: looking backward . . . . . .. ............... 148

5.3 Local Search for Constraint Satisfaction Problems . . . . . . ............ 150

54 The Structure of Problems . . . . . . oo ooo ooo 151

55 Summary . oo... LL ee 155

Bibliographical and Historical Notes . . . . . . ... 0... 156

EXErciSes . . oo ov oi 158

6 Adversarial Search 161

6.1 Games . . . Lo... ee 161

6.2 Optimal Decisions in Games . . . . . . . «co vce 162

Optimal strategies . . . . . . LL 163

The minimax algorithm . . . . . . 165

Optimal decisions in multiplayer games. . . . . . . . 165

6.3 Alpha-BetaPruning . . . . . . oe 167

6.4 Imperfect, Real-Time Decisions . . . . ......... 171

Evaluation functions . . . . . . . LLL 171

Cutting off search . . . . . .. oo 173

6.5 Games That Include an Elementof Chance . . . . .................. 175

Position evaluation in games with chancenodes . . . ..... ........... 177

Complexity of expectiminimax . . . . . . . . oo... 177

Card games . . . «oe eee 179

6.6 State-of-the-Art Game Programs . . . . . o.oo oc 180

6.7 Discussion . . . . oe ee 183

6.8 Summary. . . . Le 185

Bibliographical and Historical Notes . . . . . . . .. 186

EXErcises . . . o.oo ee ee eee 189 IIT Knowledge and reasoning

7 Logical Agents 194

7.1 Knowledge-Based Agents . . . . . .. .. Lo ee 195

7.2 TheWumpusWorld . . . . Le 197

73 LOZIC . + vv ee ee ee eee 200

7.4 Propositional Logic: A Very Simple Logic . . . . .................. 204

SYmtax . oo... ee 204

Xviil Contents

Semantics . . . LL eee 206

A simple knowledgebase . . . . .... LL. Lo 208 Inference . . . . . LL eee 208 Equivalence, validity, and satisfiability . . . ................. .... 210

7.5 Reasoning Patterns in Propositional Logic . . . .. ................. 211 Resolution . . . . . . eee 213 Forward and backward chaining . . . . . ....... 217

7.6 Effective propositional inference . . . . . .... 0 LL 220 A complete backtracking algorithm . . . . . Lo... LL LoL 221 Local-search algorithms . . . . .. ....... ... .......... ...... 222

Hard satisfiability problems . . . . . ......... 0... 224

7.7 Agents Based on Propositional Logic . . . . ......... 225 Finding pits and wumpuses using logical inference . . . . . . . ........... 225 Keeping track of location and orientation . . . . . .................. 227 Circuit-based agents . . . . . . «tie ee eee 227 Acomparison . . . ....ee e eee e 231

7.8 SUMINATY . . oo. ee ee ee 232 Bibliographical and Historical Notes . . . . . . . .. ............. ....... 233 Exercises . . . . oo eee eee 236 8 First-Order Logic 240 8.1 Representation Revisited . . . ............ oo... 240 8.2 Syntax and Semantics of First-Order Logic . . . . . ................. 245 Models for first-order logic . . . . . .. Le 245 Symbols and interpretations . . . . . . LL... eee 246

Terms . . oo eee 248 AtOMICSENEENCES . . . «vo vo i see 248 Complex sentences . . . . . «coo ee 249 Quantifiers . . . oo LLL 249 Equality . .. .. ee 253

83 Using First-Order Logic . . . . . . 253 Assertions and queries in first-order logic . . . . . . Lo. Loo oo. 253

The kinshipdomain . . . . . . . 254 Numbers, sets, and lists . . . . . . ee 256

The wumpus world... . . 258

8.4 Knowledge Engineering in First-Order Logic . . . . . . ............... 260 The knowledge engineering process . . . . . . . . . . ce. 261

The electronic circuitsdomain. . . . . . .. Loe 262

85 Summary... ... eee 266 Bibliographical and Historical Notes . . . . . .. ...................... 267 Exercises . . . oe eee 268 9 Inference in First-Order Logic 272 9.1 Propositional vs. First-Order Inference . . . . . . ......... ..... .... 272 Inference rules for quantifiers . . . . . .... LoL. 273 Reduction to propositional inference . . . . . ......... 0... 274

9.2 Unificationand Lifting . . .......... . ....... ..... ....... 275 A first-order inferencerule . . . . LL... LL LL oo 275

Unification . . . . . . . oe ee ee 276

Contents Xix Storage and retrieval . . . . LLL LLL LoL 278

9.3 Forward Chaining . . . . . . . . «i i i tee eee 280 First-order definite clauses . . . . . . ........ 280

A simple forward-chaining algorithm . . . . . ....... oo LL. 281 Efficient forward chaining . . . . . ............... 283

94 Backward Chaining . . . ............. 287 A backward chaining algorithm . . . . . . . o.oo ooo 287

Logic programming . . . . . . . «oo... eee ee 289 Efficient implementation of logic programs . . . . . . . . . . . 290 Redundant inference and infinite loops . . . . . ....... Lo LL. 292 Constraint logic programming . . . . . . . . . . ... 0... 294

95 Resolution . . ...... 295 Conjunctive normal form for first-order logic . . . . ................. 295

The resolution inference rule . . . . . . . ... LL LLL. 297 Example proofs . . . . . . ee 297 Completeness of resolution . . . . .. .. Lo oo 300 Dealing with equality . . . ..................... ......... 303 Resolution Strategies . . . . . . . . . . . . Lio 304 Theorem provers . . . . . oo . cL ee ee eee 306

9.6 SUMMATY . . . . . «oo tee ee eee eee 310 Bibliographical and Historical Notes . . . . . .. ...................... 310 EXEICISES . . «ot tt tt Ce eC 315 10 Knowledge Representation 320 10.1 Ontological Engineering . . . . . . . . ........ .. . ..... 320 10.2 Categoriesand Objects . . . . . . . . oe 322 Physical composition . . . . ..... LL 324 MEASUreImMENtS . . . . . . cc vt tte ee ee eee 325 Substances and objects . . . LL... LLL LL cee 327

10.3 Actions, Situations, and Events . . . . . . . eee 328 The ontology of situation calculus. . . . . .... .................. 329 Describing actions in situation calculus . . . . . ..... LoL 330 Solving the representational frame problem . . . . . . . ......... LL. 332 Solving the inferential frame problem . . . . . . ................... 333

Time and eventcalculus . . . . . . .. LL 334 Generalized events . . . . . . . . LL... eee 335 Processes. . . . Lee 337 Intervals . . . . Le 338 Fluents and objects . . . . . . .. 339

10.4 Mental Events and Mental Objects . . . . . ..................... 341 A formal theory of beliefs . . . . . ... 0000. 341 Knowledge and belief . . . . .................... 343 Knowledge, time, and action . . . . . ............. ........... 344

10.5 The Internet Shopping World . . . . . .... ooo. 344 Comparing offers . . . . .. LL 348

10.6 Reasoning Systems for Categories . . . . . . .. ..... ot... 349 Semanticnetworks . . . . . LL LLL Lo 350 Description logics . . . . . . . o.oo 353

10.7 Reasoning with Default Information . . . ...................... 354

XX

Contents

Openandclosedworlds . . . . . ..................... Negation as failure and stable model semantics . . . . . . ........ Circumscription and defaultlogic . . . . . ................ 10.8 Truth Maintenance Systems . . . . . . .. ................ 109 Summary. . . . . LL. Lee eee eee Bibliographical and Historical Notes . . . . . .. ................ Exercises . . . . . . . eee

IV Planning

11 Planning 11.1 The Planning Problem . . . . . .. .................... The language of planning problems . . . . . . .............. Expressiveness and extensions . . . .. . ..... LL. Example: Aircargo transport . . . .. ... ............... Example: The spare tire problem . . . . . . ............... Example: Theblocksworld . . . . .................... 11.2 Planning with State-Space Search . . . . .... ............. Forward state-space search. . . . ... ................. Backward state-spacesearch . . . ...... Loo. LLL. Heuristics for state-space search. . . . . . . .. ............. 11.3 Partial-Order Planning . . . . . . . .................... A partial-order planning example . . . . . ....... LL. Partial-order planning with unbound variables . . . . .......... Heuristics for partial-order planning . . . . . . ............. 11.4 Planning Graphs . . . . . . . Le. Planning graphs for heuristic estimation . . . .............. The GRAPHPLAN algorithm . . . . .. ..................

11.5 Planning with Propositional Logic . . . . ................ Describing planning problems in propositional logic . . . . .. ..... Complexity of propositional encodings . . . . ..............

11.6 Analysis of Planning Approaches . . . .. ................

11.7 Summary . . . . . Lo ee eee

Bibliographical and Historical Notes . . . . . ..................

12 Planning and Acting in the Real World 12.1 Time, Schedules, and Resources. . . . . . . . . . . . ov...

12.2 Hierarchical Task Network Planning . . . .. .............. Representing action decompositions. . . . . . .. ........... Modifying the planner for decompositions . . . . ............ Discussion . . . . . . Le

12.3 Planning and Acting in Nondeterministic Domains . . . . . .......

12.4 Conditional Planning . . . . . .. ....................

Conditional planning in partially observable environments . . . . . . . .

Contents

XXi

13

14

126 Continuous Planning . . . . . . . . . LL. 12.7 MultiAgent Planning . . . . . . . . Cooperation: Joint goalsand plans . . . . . . .................... Multibody planning . . . . . . . LLL Coordination mechanisms . . . . . . . . . .... Competition . . . . .. . . Lee 12.8 Summary . . . . oo. Bibliographical and Historical Notes . . . . . . ....................... Exercises . . . . oo.

Uncertain knowledge and reasoning

Uncertainty 13.1 Actingunder Uncertainty . . . .. .......................... Handling uncertainknowledge . . .. ........................ Uncertainty and rational decisions . . . . . .. .................... Design for a decision-theoreticagent . . . . . ... ................. 13.2 Basic Probability Notation. . . . . .......................... Propositions . . . . . LL... a Atomic events . . . . . Lo... Le eee Prior probability . . . . . .. .. LL Conditional probability . . .............................. 13.3 The Axioms of Probability . ............... .............. Using the axioms of probability . . . . . ....................... Why the axioms of probability are reasonable . . . . ................ 13.4 Inference Using Full Joint Distributions . . . . . . .................. 13.5 Independence . . . .......... ee 13.6 Bayes’RuleandItsUse . . ................. ............. Applying Bayes’ rule: The simplecase . . . . .................... Using Bayes’ rule: Combining evidence . . . .................... 13.7 The Wumpus World Revisited . . . . . . ... .................... 13.8 Summary . . . . . LL. ee ee Bibliographical and Historical Notes . . . . . . . ...................... Exercises . . . . oo...

Probabilistic Reasoning

14.1 Representing Knowledge in an Uncertain Domain . . . .. .............

14.2 The Semantics of Bayesian Networks . . . . ..................... Representing the full joint distribution . . . . . ................... Conditional independence relations in Bayesian networks . . . . ..........

14.3 Efficient Representation of Conditional Distributions . . . . . . ...........

14.4 Exact Inference in Bayesian Networks . . . . . ................... Inference by enumeration . . . . . .... LL LLL LL LLL. The variable elimination algorithm . . . . . ..................... The complexity of exactinference . . . . . . . .................... Clustering algorithms . . . . . . ................

14.5 Approximate Inference in Bayesian Networks . . . . . . .............. Direct sampling methods . . . . . .......................... Inference by Markov chain simulation . . . .....................

445 449 450 451 452 454 454 455 459

XXii Contents

14.6 Extending Probability to First-Order Representations . . . . . ........... 519 14.7 Other Approaches to Uncertain Reasoning . . . . . ................. 523 Rule-based methods for uncertain reasoning . . . . .. ............... 524 Representing ignorance: Dempster-Shafer theory . . . ............... 525 Representing vagueness: Fuzzy sets and fuzzy logic . . . .............. 526

148 Summary. . . . . LLL. eee 528 Bibliographical and Historical Notes . . . . . . . . . .................... 528 Exercises . . . . . oe 533 15 Probabilistic Reasoning over Time 537 15.1 Time and Uncertainty . . . . . . . .. .... «oii 537 States and observations . . . . . .. LL... LLL ee 538 Stationary processes and the Markov assumption . . . . . .............. 538

15.2 Inferencein Temporal Models . . . . . . ....................... 541 Filtering and prediction . . . . .. ........ 542 Smoothing . . . . . . . ee 544 Finding the most likely sequence . . . . . ...................... 547

15.3 Hidden MarkovModels . . . . ..................... ....... 549 Simplified matrix algorithms . . . . . .... LLL LoL 549

154 KalmanFilters . . . . . . . oe 551 Updating Gaussian distributions . . . . . . ...................... 553

A simple one-dimensional example . . . . . .... LLL... 554

The generalcase . . . . .. ... a 556 Applicability of Kalman filtering . . . . . ...................... 557

15.5 Dynamic Bayesian Networks . . . . ........... .............. 559 Constructing DBNs . . . . . . .. oa 560 Exactinferencein DBNs . . . ............................ 563 Approximate inference in DBNs . . . . ....... 565

15.6 SpeechRecognition . . . ............... .......... ...... 568 Speechsounds . . . .. ......... ee 570 Words... Le ee eee 572 Sentences . . . . Lo... ee ee ee 574 Building a speechrecognizer . . . .................. 0... 576

15.7 Summary . . . . LL 578 Bibliographical and Historical Notes . . . . . ........................ 578 Exercises . . . . . . . . e ee 581 16 Making Simple Decisions 584 16.1 Combining Beliefs and Desires under Uncertainty . . . . .............. 584 16.2 The Basis of Utility Theory . . . . ............ . .... ......... 586 Constraints on rational preferences . . . . . . . ................... 586 Andthenthere was Utility . . . . . . ......................... 588

16.3 Utility Functions . . . . . ..... 589 Theutilityof money . . . . . ....... 589 Utility scales and utility assessment . . . . . . .................... 591

16.4 Multiattribute Utility Functions . . . . . ....................... 593 Dominance . . . . . . ... ee 594 Preference structure and multiattribute utility . . . . ................. 596

16.5 Decision Networks. . . . . . . . ee 597

Contents

xxiii

Representing a decision problem with a decision network . . . ........... Evaluating decision networks . . . . . . .... LL. LL 16.6 The Value of Information . . . ................. ........... Asimpleexample . . . ... LLL Ageneralformula . . ........... LL Properties of the value of information . . . . . .................... Implementing an information-gathering agent . . . . . ............... 16.7 Decision-Theoretic Expert Systems . . . . . . .. .................. 16.8 Summary . . . . . LL. Bibliographical and Historical Notes . . . . . . . ...................... Exercises . . . . . o.oo eee

17 Making Complex Decisions

17.1 Sequential Decision Problems . . . . . . .............. Anexample . . .. Lee Optimality in sequential decision problems . . . . . . . .. .............

17.2 Valuelteration . . . . . . . . . Utilities of states . . . . . . . . LL... The value iteration algorithm . . . . . . . ...................... Convergence of value iteration . . . . . . ... . ..

17.3 Policy Iteration. . . . . . . ee

17.4 Partially observable MDPs . . . . . .... Lo

17.5 Decision-Theoretic Agents . . . . . . . . iii

17.6 Decisions with Multiple Agents: Game Theory . . . . . . ..............

17.7 Mechanism Design . . . . . .

17.8 Summary . . . . oo... ee eee eee

Bibliographical and Historical Notes . . . . . . . ......................

Exercises . . . . . . Le ee eee

VI Learning

18 Learning from Observations 18.1 FormsofLearning . . . . . . . . . . . a 182 Inductive Learning . . . . . . . . . LL... 18.3 Learning Decision Trees . . . . . . . ......................... Decision trees as performance elements . . . . . . . ...... LL. Expressiveness of decisiontrees . . . . .. ...................... Inducing decision trees from examples . . . . . .. LLL LLL Choosing attribute tests. . . . . . . LL... Assessing the performance of the learning algorithm . . . . . . ........... Noise and overfitting . . . . . . . . eee Broadening the applicability of decisiontrees. . . . . . . .............. 18.4 Ensemble Learning . . . . . . . . .. Lee 18.5 Why Learning Works: Computational Learning Theory . . . . . .......... How many examples are needed? . . . . ....................... Learning decision lists . . . . . . . . . LL. Discussion . . . . . . LLL eee 18.6 Summary . . . . . . . . . ee Bibliographical and Historical Notes . . . . . . . . . ....................

598 599 600 600 601 602 603 604 607 607 609

613 613 613 616 618 619 620 620 624 625 629 631 640 643 644 646

Xxiv Contents

EXercises . . . . . Le 676 19 Knowledge in Learning 678 19.1 A Logical Formulation of Learning . . . . . . ..... ............... 678 Examples and hypotheses . . . . . .. . oo. 678 Current-best-hypothesis search . . . . ........................ 680 Least-commitment search . . . . . . . ........ LL... 683

19.2 Knowledgein Learning . . . . ........... 636 Some simple examples . . . . .... LLL 687

Some general schemes . . . . . . . .. LL... LL 688

19.3 Explanation-Based Learning . . . .. ................ ........ 690 Extracting general rules fromexamples . . . . . ... LoL LLL 691 Improving efficiency . . . . . . . LLL 693

19.4 Learning Using Relevance Information . . . . .................... 694 Determining the hypothesis space . . . . . ...................... 695 Learning and using relevance information . . . . . ................. 695

19.5 Inductive Logic Programming . . . . . . . . . ....... 697 Anexample . . .. . ee 699 Top-down inductive learning methods . . . . . . .. ................. 701 Inductive learning with inverse deduction . . . . . .................. 703 Making discoveries with inductive logic programming . . . . . ........... 705

19.6 Summary . . . . LL. LL 707 Bibliographical and Historical Notes . . . . . ........................ 708 EXEICiSes . . . oo oi ee ee ee ee 710 20 Statistical Learning Methods 712 20.1 Statistical Learning . . . . . . Lo. Lee 712 20.2 Learning with Complete Data . . . . . .. ...................... 716 Maximum-likelihood parameter learning: Discrete models . . . . . . ........ 716

Naive Bayesmodels . . . . . .... 718 Maximum-likelihood parameter learning: Continuous models . . . . . ....... 719 Bayesian parameter learning . . . . . . LL... LLL 720 Learning Bayes net structures . . . . . . . .. . ... 722

20.3 Learning with Hidden Variables: The EM Algorithm . . . . . . ........... 724 Unsupervised clustering: Learning mixtures of Gaussians . . . . .......... 725 Learning Bayesian networks with hidden variables . . . . . ............. 727 Learning hidden Markovmodels . . . ........................ 731

The general form of the EM algorithm . . . ..................... 731 Learning Bayes net structures with hidden variables . . . . .. ........... 732

20.4 Instance-Based Learming . . ............... 733 Nearest-neighbor models . . . . . . . ....... 733 Kemelmodels . . . . ... 735

20.5 Neural Networks . . . . . . . . Le 736 Unitsinneuralnetworks . . . . . . . ... Lo 737 Network structures . . . . . . . . . 738

Single layer feed-forward neural networks (perceptrons) . . . . ........... 740 Multilayer feed-forward neural networks . . . . . . ...... LL. 744 Learning neural network structures . . . . . . . . .... LL. 748

20.6 Kernel Machines . . . . . . . . . . . ee 749

Contents XXV 20.7 Case Study: Handwritten Digit Recognition . . . . ................. 752 20.8 Summary . . . . Lo... ee eee 754 Bibliographical and Historical Notes . . . . . .. .. .................... 755 Exercises . . . . Lee 759

21 Reinforcement Learning 763 21.1 Introduction . . . . . . . . eee ee ee 763 21.2 Passive Reinforcement Learning . . . . . ...................... 765

Direct utility estimation . . . . . . . . . LLL ee 766 Adaptive dynamic programming . . . . ... LL... eee 767 Temporal difference learning . . . . . .. .... oo LoL 767 21.3 Active Reinforcement Learning . ... . . ....................... 77 Exploration . . . . . . .. eee 771 Learning an Action-Value Function . . . . ... ................... 775 21.4 Generalization in Reinforcement Learning . . . . . . ................ 771 Applications to game-playing . . . . . . ... Lo 780 Applicationtorobotcontrol . . . . ... LL. LoL 780 21.5 PolicySearch . . .. . . . eee 781 21.6 SUMMArY . . . . oe ee ee eee eee 784 Bibliographical and Historical Notes . . . . . . ....................... 785 Exercises . . . . . ee 788

VII Communicating, perceiving, and acting

22 Communication 790 22.1 Communication as ACORN . . . . . . . . . . tite 790

Fundamentals of language . . . . . . . ............. 791 The component steps of communication . . . .................... 792 22.2 A Formal Grammar for a Fragment of English . . . . ................ 795 The Lexiconof &; . . . . . . . ea 795 The Grammarof & . . . . . . . eee 796 22.3 Syntactic Analysis (Parsing) . . . . .. .. oo 798 Efficientparsing . . . . . . . . . eee 800 22.4 Augmented Grammars . . . . . Lo... LL. 806 Verb subcategorization. . . . . . . . LLL. LLL 808 Generative capacity of augmented grammars . . . . . . .. ............. 809 22.5 Semantic Interpretation . . . . .. .. LLL LL 810 The semantics of an English fragment. . . . . . ................... 811 Timeandtense . . . . . . . . Le 812 Quantification . . . . . . LL. Lee eee 813 Pragmatic Interpretation . . . . . . .. .. LL LL 815 Language generation withDCGs . . . . ....................... 817 22.6 Ambiguity and Disambiguation . . . . .. ... LLL. 818 Disambiguation . . . . . . .. ee 820 227 Discourse Understanding . . . . . ......... 0... 821 Reference resolution . . . . . . . ... 821 The structure of coherent discourse . . . . . ............ ......... 823 228 GrammarInduction .......... LL... 824 229 Summary . . . . o.oo ee ee ee ee eee 826

XXVi Contents

Bibliographical and Historical Notes . . . . . . . ...................... 827 Exercises . . . . . eee ee 831 23 Probabilistic Language Processing 834 23.1 Probabilistic Language Models . . . . ........................ 834 Probabilistic context-free grammars . . . . . .... LL LL LLL. 836 Learning probabilities for PCFGs . . . . . ...................... 839 Learning rule structure for PCFGs . . . . . .. ................... 840

23.2 Information Retrieval . . . . ....... . ........ 840 Evaluating IR systems . . . . . . . ... ..... 842 IRrefinements . . . . . .. .. eee 844 Presentation of resultsets . . . . .. ......... .......L. 845 Implementing IR systems . . . . . .................. 846

23.3 Information Extraction. . . . . . . ....... 848 234 Machine Translation . . . . .. .. ........ ee 850 Machine translation systems . . . . . . . LL... LL. ee 852 Statistical machine translation . . . . . . . .. LL LL. 853 Learning probabilities for machine translation . . . ................. 856

235 Summary . . Lo... ee ee 857 Bibliographical and Historical Notes . . . . . ........................ 858 EXerciSes . . . . oie ee eee 861 24 Perception 863 24.1 Introduction . . . . . . . ee eee 863 24.2 TImageFormation. . . . . . . . . 865 Images without lenses: the pinhole camera . . . . . ................. 865 Lenssystems . . . . . . Lee eee 866

Light: the photometry of image formation . . . ................... 867

Color: the spectrophotometry of image formation . . ................ 868

24.3 Early Image Processing Operations . . . . . ..................... 869 Edgedetection . . . . . . . Lee eee 870

Image segmentation . . . . .. ... LL... 872

24.4 Extracting Three-Dimensional Information . . . . . ................. 873 Motion . . . . «eee 875 Binocular stereopsis . . . . . LL... LLL ee 876 Texture gradients. . . . . . . ... LLL 879 Shading . . . . . . eee 880 Contour . . . . LL. ee eee 881

24.5 ObjectRecognition . . . . . . .. eee 885 Brightness-based recognition . . . .. .... LL... 887 Feature-based recognition . . . . . . . . ... LLL 888

Pose Estimation . . . . . ..... Le 890

24.6 Using Vision for Manipulation and Navigation . . . . . ............... 892 247 SUMMATY . . «Lt tee ee ee ee ee ee 894 Bibliographical and Historical Notes . . . . . .. ...................... 895 Exercises . . . . ee 898 25 Robotics 901

25.1 Introduction . . . . . . . 901

Contents

XX Vil

Bibliographical and Historical Notes

VIII Conclusions

26 Philosophical Foundations

The argument from disability

The Chineseroom . . . . . . . .. 26.3 The Ethics and Risks of Developing Artificial Intelligence . . . . . ......... 26.4 SUMMATY . . . . . «oot te ee ee ee

Bibliographical and Historical Notes

27 AI Present and Future

903 903 904 907 908 913 915 916 916 919 922 923 924 926 927 929 930 932 932 933 934 935 938 939 942

947 947 948 949 950 952 954 955 956 958 960 964 964 967

Xxviii

Contents

A Mathematical background

B Notes on Languages and Algorithms B.1 Defining Languages with Backus-Naur Form (BNF)

Bibliography

Index

INTRODUCTION

ARTIFICIAL INTELLIGENCE

In which we try to explain why we consider artificial intelligence to be a subject most worthy of study, and in which we try to decide what exactly it is, this being a good thing to decide before embarking.

We call ourselves Homo sapiens—man the wise—because our mental capacities are so important to us. For thousands of years, we have tried to understand how we think; that is, how a mere handful of stuff can perceive, understand, predict, and manipulate a world far larger and more complicated than itself. The field of artificial intelligence, or Al, goes further still: it attempts not just to understand but also to build intelligent entities.

Al is one of the newest sciences. Work started in earnest soon after World War II, and the name itself was coined in 1956. Along with molecular biology, Al is regularly cited as the “field I would most like to be in” by scientists in other disciplines. A student in physics might reasonably feel that all the good ideas have already been taken by Galileo, Newton, Einstein, and the rest. Al, on the other hand, still has openings for several full-time Einsteins.

Al currently encompasses a huge variety of subfields, ranging from general-purpose areas, such as learning and perception to such specific tasks as playing chess, proving mathematical theorems, writing poetry, and diagnosing diseases. AI systematizes and automates intellectual tasks and is therefore potentially relevant to any sphere of human intellectual activity. In this sense, it is truly a universal field.

RATIONALITY

We have claimed that Al is exciting, but we have not said what it is. Definitions of artificial intelligence according to eight textbooks are shown in Figure 1.1. These definitions vary along two main dimensions. Roughly, the ones on top are concerned with thought processes and reasoning, whereas the ones on the bottom address behavior. The definitions on the left measure success in terms of fidelity to Auman performance, whereas the ones on the right measure against an ideal concept of intelligence, which we will call rationality. A system is rational if it does the “right thing,” given what it knows.

1

Chapter 1.

TURING TEST

NATURAL LANGUAGE PROCESSING

Systems that think like humans

Systems that think rationally

“The exciting new effort to make computers think ... machines with minds, in the full and literal sense.” (Haugeland, 1985)

“[The automation of] activities that we associate with human thinking, activities such as decision-making, problem solving, learning . . .” (Bellman, 1978)

“The study of mental faculties through the use of computational models.” (Charniak and McDermott, 1985)

“The study of the computations that make it possible to perceive, reason, and act.” (Winston, 1992)

Systems that act like humans

Systems that act rationally

“The art of creating machines that perform functions that require intelligence

“Computational Intelligence is the study of the design of intelligent agents.” (Poole

when performed by people.” (Kurzweil, et al., 1998)

1990)

“The study of how to make computers do things at which, at the moment, people are better.” (Rich and Knight, 1991)

“Al ...is concerned with intelligent behavior in artifacts.” (Nilsson, 1998)

Figure 1.1 Some definitions of artificial intelligence, organized into four categories.

Historically, all four approaches to Al have been followed. As one might expect, a tension exists between approaches centered around humans and approaches centered around rationality.! A human-centered approach must be an empirical science, involving hypothesis and experimental confirmation. A rationalist approach involves a combination of mathematics and engineering. Each group has both disparaged and helped the other. Let us look at the four approaches in more detail.

Acting humanly: The Turing Test approach

The Turing Test, proposed by Alan Turing (1950), was designed to provide a satisfactory operational definition of intelligence. Rather than proposing a long and perhaps controversial list of qualifications required for intelligence, he suggested a test based on indistinguishability from undeniably intelligent entities—human beings. The computer passes the test if a human interrogator, after posing some written questions, cannot tell whether the written responses come from a person or not. Chapter 26 discusses the details of the test and whether a computer is really intelligent if it passes. For now, we note that programming a computer to pass the test provides plenty to work on. The computer would need to possess the following capabilities:

{> natural language processing to enable it to communicate successfully in English.

1 We should point out that, by distinguishing between human and rational behavior, we are not suggesting that humans are necessarily “irrational” in the sense of “emotionally unstable” or “insane.” One merely need note that we are not perfect: we are not all chess grandmasters, even those of us who know all the rules of chess; and, unfortunately, not everyone gets an A on the exam. Some systematic errors in human reasoning are cataloged by Kahneman et al. (1982).

Section 1.1.

What is AI? 3

KNOWLEDGE REPRESENTATION

AUTOMATED REASONING

MACHINE LEARNING

TOTAL TURING TEST

COMPUTER VISION

ROBOTICS

COGNITIVE SCIENCE

¢& knowledge representation to store what it knows or hears;

{> automated reasoning to use the stored information to answer questions and to draw new conclusions;

<> machine learning to adapt to new circumstances and to detect and extrapolate patterns.

Turing’s test deliberately avoided direct physical interaction between the interrogator and the computer, because physical simulation of a person is unnecessary for intelligence. However, the so-called total Turing Test includes a video signal so that the interrogator can test the subject’s perceptual abilities, as well as the opportunity for the interrogator to pass physical objects “through the hatch.” To pass the total Turing Test, the computer will need

{> computer vision to perceive objects, and ¢ robotics to manipulate objects and move about.

These six disciplines compose most of Al, and Turing deserves credit for designing a test that remains relevant 50 years later. Yet Al researchers have devoted little effort to passing the Turing test, believing that it is more important to study the underlying principles of intelligence than to duplicate an exemplar. The quest for “artificial flight” succeeded when the Wright brothers and others stopped imitating birds and learned about aerodynamics. Aeronautical engineering texts do not define the goal of their field as making “machines that fly so exactly like pigeons that they can fool even other pigeons.”

Thinking humanly: The cognitive modeling approach

If we are going to say that a given program thinks like a human, we must have some way of determining how humans think. We need to get inside the actual workings of human minds. There are two ways to do this: through introspection—trying to catch our own thoughts as they go by—and through psychological experiments. Once we have a sufficiently precise theory of the mind, it becomes possible to express the theory as a computer program. If the program’s input/output and timing behaviors match corresponding human behaviors, that is evidence that some of the program’s mechanisms could also be operating in humans. For example, Allen Newell and Herbert Simon, who developed GPS, the “General Problem Solver” (Newell and Simon, 1961), were not content to have their program solve problems correctly. They were more concerned with comparing the trace of its reasoning steps to traces of human subjects solving the same problems. The interdisciplinary field of cognitive science brings together computer models from Al and experimental techniques from psychology to try to construct precise and testable theories of the workings of the human mind.

Cognitive science is a fascinating field, worthy of an encyclopedia in itself (Wilson and Keil, 1999). We will not attempt to describe what is known of human cognition in this book. We will occasionally comment on similarities or differences between Al techniques and human cognition. Real cognitive science, however, is necessarily based on experimental investigation of actual humans or animals, and we assume that the reader has access only to a computer for experimentation.

In the early days of Al there was often confusion between the approaches: an author would argue that an algorithm performs well on a task and that it is therefore a good model

Chapter 1. Introduction

SYLLOGISMS

LOGIC

LOGICIST

AGENT

RATIONAL AGENT

of human performance, or vice versa. Modern authors separate the two kinds of claims; this distinction has allowed both AI and cognitive science to develop more rapidly. The two fields continue to fertilize each other, especially in the areas of vision and natural language. Vision in particular has recently made advances via an integrated approach that considers neurophysiological evidence and computational models.

Thinking rationally: The “laws of thought” approach

The Greek philosopher Aristotle was one of the first to attempt to codify “right thinking,” that is, irrefutable reasoning processes. His syllogisms provided patterns for argument structures that always yielded correct conclusions when given correct premises—for example, “Socrates is a man; all men are mortal; therefore, Socrates is mortal.” These laws of thought were supposed to govern the operation of the mind; their study initiated the field called logic.

Logicians in the 19th century developed a precise notation for statements about all kinds of things in the world and about the relations among them. (Contrast this with ordinary arithmetic notation, which provides mainly for equality and inequality statements about numbers.) By 1965, programs existed that could, in principle, solve any solvable problem described in logical notation.” The so-called logicist tradition within artificial intelligence hopes to build on such programs to create intelligent systems.

There are two main obstacles to this approach. First, it is not easy to take informal knowledge and state it in the formal terms required by logical notation, particularly when the knowledge is less than 100% certain. Second, there is a big difference between being able to solve a problem “in principle” and doing so in practice. Even problems with just a few dozen facts can exhaust the computational resources of any computer unless it has some guidance as to which reasoning steps to try first. Although both of these obstacles apply to any attempt to build computational reasoning systems, they appeared first in the logicist tradition.

Acting rationally: The rational agent approach

An agent is just something that acts (agent comes from the Latin agere, to do). But computer agents are expected to have other attributes that distinguish them from mere “programs,” such as operating under autonomous control, perceiving their environment, persisting over a prolonged time period, adapting to change, and being capable of taking on another’s goals. A rational agent is one that acts so as to achieve the best outcome or, when there is uncertainty, the best expected outcome.

In the “laws of thought” approach to Al, the emphasis was on correct inferences. Making correct inferences is sometimes part of being a rational agent, because one way to act rationally is to reason logically to the conclusion that a given action will achieve one’s goals and then to act on that conclusion. On the other hand, correct inference is not all of rationality, because there are often situations where there is no provably correct thing to do, yet something must still be done. There are also ways of acting rationally that cannot be said to involve inference. For example, recoiling from a hot stove is a reflex action that is usually more successful than a slower action taken after careful deliberation.

2 If there is no solution, the program might never stop looking for one.

Section 1.2. The Foundations of Artificial Intelligence 5

All the skills needed for the Turing Test are there to allow rational actions. Thus, we need the ability to represent knowledge and reason with it because this enables us to reach good decisions in a wide variety of situations. We need to be able to generate comprehensible sentences in natural language because saying those sentences helps us get by in a complex society. We need learning not just for erudition, but because having a better idea of how the world works enables us to generate more effective strategies for dealing with it. We need visual perception not just because seeing is fun, but to get a better idea of what an action might achieve—for example, being able to see a tasty morsel helps one to move toward it.

For these reasons, the study of Al as rational-agent design has at least two advantages. First, it is more general than the ‘laws of thought” approach, because correct inference is just one of several possible mechanisms for achieving rationality. Second, it is more amenable to scientific development than are approaches based on human behavior or human thought because the standard of rationality is clearly defined and completely general. Human behavior, on the other hand, is well-adapted for one specific environment and is the product, in part, of a complicated and largely unknown evolutionary process that still is far from producing perfection. This book will therefore concentrate on general principles of rational agents and on components for constructing them. We will see that despite the apparent simplicity with which the problem can be stated, an enormous variety of issues come up when we try to solve it. Chapter 2 outlines some of these issues in more detail.

One important point to keep in mind: We will see before too long that achieving perfect rationality—always doing the right thing—is not feasible in complicated environments. The computational demands are just too high. For most of the book, however, we will adopt the working hypothesis that perfect rationality is a good starting point for analysis. It simplifies the problem and provides the appropriate setting for most of the foundational material in Ty the field. Chapters 6 and 17 deal explicitly with the issue of limited rationality—acting

appropriately when there is not enough time to do all the computations one might like.

1.2 THE FOUNDATIONS OF ARTIFICIAL INTELLIGENCE

In this section, we provide a brief history of the disciplines that contributed ideas, viewpoints, and techniques to Al. Like any history, this one is forced to concentrate on a small number of people, events, and ideas and to ignore others that also were important. We organize the history around a series of questions. We certainly would not wish to give the impression that these questions are the only ones the disciplines address or that the disciplines have all been working toward Al as their ultimate fruition.

Philosophy (428 B.C.—present)

e Can formal rules be used to draw valid conclusions?

e How does the mental mind arise from a physical brain? e Where does knowledge come from?

e How does knowledge lead to action?

Chapter 1. Introduction

DUALISM

MATERIALISM

EMPIRICISM

INDUCTION

LOGICAL POSITIVISM OBSERVATION SENTENCES CONFIRMATION THEORY

Aristotle (384-322 B.C.) was the first to formulate a precise set of laws governing the rational part of the mind. He developed an informal system of syllogisms for proper reasoning, which in principle allowed one to generate conclusions mechanically, given initial premises. Much later, Ramon Lull (d. 1315) had the idea that useful reasoning could actually be carried out by a mechanical artifact. His “concept wheels” are on the cover of this book. Thomas Hobbes (1588-1679) proposed that reasoning was like numerical computation, that “we add and subtract in our silent thoughts.” The automation of computation itself was already well under way; around 1500, Leonardo da Vinci (1452-1519) designed but did not build a mechanical calculator; recent reconstructions have shown the design to be functional. The first known calculating machine was constructed around 1623 by the German scientist Wilhelm Schickard (1592-1635), although the Pascaline, built in 1642 by Blaise Pascal (1623-1662), is more famous. Pascal wrote that “the arithmetical machine produces effects which appear nearer to thought than all the actions of animals.” Gottfried Wilhelm Leibniz (1646-1716) built a mechanical device intended to carry out operations on concepts rather than numbers, but its scope was rather limited.

Now that we have the idea of a set of rules that can describe the formal, rational part of the mind, the next step is to consider the mind as a physical system. René Descartes (1596-1650) gave the first clear discussion of the distinction between mind and matter and of the problems that arise. One problem with a purely physical conception of the mind is that it seems to leave little room for free will: if the mind is governed entirely by physical laws, then it has no more free will than a rock “deciding” to fall toward the center of the earth. Although a strong advocate of the power of reasoning, Descartes was also a proponent of dualism. He held that there is a part of the human mind (or soul or spirit) that is outside of nature, exempt from physical laws. Animals, on the other hand, did not possess this dual quality; they could be treated as machines. An alternative to dualism is materialism, which holds that the brain’s operation according to the laws of physics constitutes the mind. Free will is simply the way that the perception of available choices appears to the choice process.

Given a physical mind that manipulates knowledge, the next problem is to establish the source of knowledge. The empiricism movement, starting with Francis Bacon’s (1561-1626) Novum Organum,’ is characterized by a dictum of John Locke (1632-1704): “Nothing is in the understanding, which was not first in the senses.” David Hume's (1711-1776) A Treatise of Human Nature (Hume, 1739) proposed what is now known as the principle of induction: that general rules are acquired by exposure to repeated associations between their elements. Building on the work of Ludwig Wittgenstein (1889-1951) and Bertrand Russell (1872— 1970), the famous Vienna Circle, led by Rudolf Carnap (1891-1970), developed the doctrine of logical positivism. This doctrine holds that all knowledge can be characterized by logical theories connected, ultimately, to observation sentences that correspond to sensory inputs.\* The confirmation theory of Carnap and Carl Hempel (1905-1997) attempted to understand how knowledge can be acquired from experience. Carnap’s book The Logical Structure of

3 An update of Aristotle’s Organon, or instrument of thought.

4 In this picture, all meaningful statements can be verified or falsified either by analyzing the meaning of the words or by carrying out experiments. Because this rules out most of metaphysics, as was the intention, logical positivism was unpopular in some circles.

Section 1.2.

The Foundations of Artificial Intelligence 7

the World (1928) defined an explicit computational procedure for extracting knowledge from elementary experiences. It was probably the first theory of mind as a computational process.

The final element in the philosophical picture of the mind is the connection between knowledge and action. This question is vital to Al, because intelligence requires action as well as reasoning. Moreover, only by understanding how actions are justified can we understand how to build an agent whose actions are justifiable (or rational). Aristotle argued that actions are justified by a logical connection between goals and knowledge of the action’s outcome (the last part of this extract also appears on the front cover of this book):

But how does it happen that thinking is sometimes accompanied by action and sometimes

not, sometimes by motion, and sometimes not? It looks as if almost the same thing

happens as in the case of reasoning and making inferences about unchanging objects. But

in that case the end is a speculative proposition ... whereas here the conclusion which

results from the two premises is an action. ... I need covering; a cloak is a covering. 1

need a cloak. What I need, I have to make; I need a cloak. I have to make a cloak. And

the conclusion, the “I have to make a cloak,” is an action. (Nussbaum, 1978, p. 40)

In the Nicomachean Ethics (Book III. 3, 1112b), Aristotle further elaborates on this topic, suggesting an algorithm: We deliberate not about ends, but about means. For a doctor does not deliberate whether he shall heal, nor an orator whether he shall persuade, ... They assume the end and consider how and by what means it is attained, and if it seems easily and best produced thereby; while if it is achieved by one means only they consider how it will be achieved by this and by what means this will be achieved, till they come to the first cause, ... and what is last in the order of analysis seems to be first in the order of becoming. And if we come on an impossibility, we give up the search, e.g. if we need money and this cannot be got; but if a thing appears possible we try to do it.

Aristotle’s algorithm was implemented 2300 years later by Newell and Simon in their GPS program. We would now call it a regression planning system. (See Chapter 11.)

Goal-based analysis is useful, but does not say what to do when several actions will achieve the goal, or when no action will achieve it completely. Antoine Arnauld (1612-1694) correctly described a quantitative formula for deciding what action to take in cases like this (see Chapter 16). John Stuart Mill’s (1806-1873) book Utilitarianism (Mill, 1863) promoted the idea of rational decision criteria in all spheres of human activity. The more formal theory of decisions is discussed in the following section.

Mathematics (c. 800—present)

eo What are the formal rules to draw valid conclusions? e What can be computed? eo How do we reason with uncertain information?

Philosophers staked out most of the important ideas of Al, but the leap to a formal science required a level of mathematical formalization in three fundamental areas: logic, computation, and probability.

The idea of formal logic can be traced back to the philosophers of ancient Greece (see Chapter 7), but its mathematical development really began with the work of George Boole

Chapter 1. Introduction

ALGORITHM

INCOMPLETENESS THEOREM

INTRACTABILITY

(1815-1864), who worked out the details of propositional, or Boolean, logic (Boole, 1847). In 1879, Gottlob Frege (1848-1925) extended Boole’s logic to include objects and relations, creating the first-order logic that is used today as the most basic knowledge representation system. Alfred Tarski (1902-1983) introduced a theory of reference that shows how to relate the objects in a logic to objects in the real world. The next step was to determine the limits of what could be done with logic and computation.

The first nontrivial algorithm is thought to be Euclid’s algorithm for computing greatest common denominators. The study of algorithms as objects in themselves goes back to al-Khowarazmi, a Persian mathematician of the 9th century, whose writings also introduced Arabic numerals and algebra to Europe. Boole and others discussed algorithms for logical deduction, and, by the late 19th century, efforts were under way to formalize general mathematical reasoning as logical deduction. In 1900, David Hilbert (1862-1943) presented a list of 23 problems that he correctly predicted would occupy mathematicians for the bulk of the century. The final problem asks whether there is an algorithm for deciding the truth of any logical proposition involving the natural numbers—the famous Entscheidungsproblem, or decision problem. Essentially, Hilbert was asking whether there were fundamental limits to the power of effective proof procedures. In 1930, Kurt Godel (1906-1978) showed that there exists an effective procedure to prove any true statement in the first-order logic of Frege and Russell, but that first-order logic could not capture the principle of mathematical induction needed to characterize the natural numbers. In 1931, he showed that real limits do exist. His incompleteness theorem showed that in any language expressive enough to describe the properties of the natural numbers, there are true statements that are undecidable in the sense that their truth cannot be established by any algorithm.

This fundamental result can also be interpreted as showing that there are some functions on the integers that cannot be represented by an algorithm—that is, they cannot be computed. This motivated Alan Turing (1912-1954) to try to characterize exactly which functions are capable of being computed. This notion is actually slightly problematic, because the notion of a computation or effective procedure really cannot be given a formal definition. However, the Church-Turing thesis, which states that the Turing machine (Turing, 1936) is capable of computing any computable function, is generally accepted as providing a sufficient definition. Turing also showed that there were some functions that no Turing machine can compute. For example, no machine can tell in general whether a given program will return an answer on a given input or run forever.

Although undecidability and noncomputability are important to an understanding of computation, the notion of intractability has had a much greater impact. Roughly speaking, a problem is called intractable if the time required to solve instances of the problem grows exponentially with the size of the instances. The distinction between polynomial and exponential growth in complexity was first emphasized in the mid-1960s (Cobham, 1964; Edmonds, 1965). It is important because exponential growth means that even moderately large instances cannot be solved in any reasonable time. Therefore, one should strive to divide

5 Frege's proposed notation for first-order logic never became popular, for reasons that are apparent immediately from the example on the front cover.

Section 1.2.

The Foundations of Artificial Intelligence 9

NP-COMPLETENESS

PROBABILITY

DECISION THEORY

the overall problem of generating intelligent behavior into tractable subproblems rather than intractable ones.

How can one recognize an intractable problem? The theory of NP-completeness, pioneered by Steven Cook (1971) and Richard Karp (1972), provides a method. Cook and Karp showed the existence of large classes of canonical combinatorial search and reasoning problems that are NP-complete. Any problem class to which the class of NP-complete problems can be reduced is likely to be intractable. (Although it has not been proved that NP-complete problems are necessarily intractable, most theoreticians believe it.) These results contrast with the optimism with which the popular press greeted the first computers— Electronic Super-Brains” that were “Faster than Einstein!” Despite the increasing speed of computers, careful use of resources will characterize intelligent systems. Put crudely, the world is an extremely large problem instance! In recent years, Al has helped explain why some instances of NP-complete problems are hard, yet others are easy (Cheeseman et al., 1991).

Besides logic and computation, the third great contribution of mathematics to Al is the theory of probability. The Italian Gerolamo Cardano (1501-1576) first framed the idea of probability, describing it in terms of the possible outcomes of gambling events. Probability quickly became an invaluable part of all the quantitative sciences, helping to deal with uncertain measurements and incomplete theories. Pierre Fermat (1601-1665), Blaise Pascal (1623-1662), James Bernoulli (1654-1705), Pierre Laplace (1749-1827), and others advanced the theory and introduced new statistical methods. Thomas Bayes (1702-1761) proposed a rule for updating probabilities in the light of new evidence. Bayes’ rule and the resulting field called Bayesian analysis form the basis of most modern approaches to uncertain reasoning in Al systems.

Economics (1776—present)

e How should we make decisions so as to maximize payoff? e How should we do this when others may not go along? e How should we do this when the payoff may be far in the future?

The science of economics got its start in 1776, when Scottish philosopher Adam Smith (1723-1790) published An Inquiry into the Nature and Causes of the Wealth of Nations. While the ancient Greeks and others had made contributions to economic thought, Smith was the first to treat it as a science, using the idea that economies can be thought of as consisting of individual agents maximizing their own economic well-being. Most people think of economics as being about money, but economists will say that they are really studying how people make choices that lead to preferred outcomes. The mathematical treatment of “preferred outcomes” or utility was first formalized by Léon Walras (pronounced “Valrasse™) (1834-1910) and was improved by Frank Ramsey (1931) and later by John von Neumann and Oskar Morgenstern in their book The Theory of Games and Economic Behavior (1944). Decision theory, which combines probability theory with utility theory, provides a formal and complete framework for decisions (economic or otherwise) made under uncertainty— that is, in cases where probabilistic descriptions appropriately capture the decision-maker’s environment. This is suitable for “large” economies where each agent need pay no attention

10

Chapter 1. Introduction

GAME THEORY

OPERATIONS RESEARCH

SATISFICING

NEUROSCIENCE

NEURONS

to the actions of other agents as individuals. For “small” economies, the situation is much more like a game: the actions of one player can significantly affect the utility of another (either positively or negatively). Von Neumann and Morgenstern’s development of game theory (see also Luce and Raiffa, 1957) included the surprising result that, for some games, a rational agent should act in a random fashion, or at least in a way that appears random to the adversaries.

For the most part, economists did not address the third question listed above, namely, how to make rational decisions when payoffs from actions are not immediate but instead result from several actions taken in sequence. This topic was pursued in the field of operations research, which emerged in World War II from efforts in Britain to optimize radar installations, and later found civilian applications in complex management decisions. The work of Richard Bellman (1957) formalized a class of sequential decision problems called Markov decision processes, which we study in Chapters 17 and 21.

Work in economics and operations research has contributed much to our notion of rational agents, yet for many years Al research developed along entirely separate paths. One reason was the apparent complexity of making rational decisions. Herbert Simon (19162001), the pioneering Al researcher, won the Nobel prize in economics in 1978 for his early work showing that models based on satisficing—making decisions that are “good enough,” rather than laboriously calculating an optimal decision—gave a better description of actual human behavior (Simon, 1947). In the 1990s, there has been a resurgence of interest in decision-theoretic techniques for agent systems (Wellman, 1995).

Neuroscience (1861-present)

e How do brains process information?

Neuroscience is the study of the nervous system, particularly the brain. The exact way in which the brain enables thought is one of the great mysteries of science. It has been appreciated for thousands of years that the brain is somehow involved in thought, because of the evidence that strong blows to the head can lead to mental incapacitation. It has also long been known that human brains are somehow different; in about 335 B.C. Aristotle wrote, “Of all the animals, man has the largest brain in proportion to his size.” ® Still, it was not until the middle of the 18th century that the brain was widely recognized as the seat of consciousness. Before then, candidate locations included the heart, the spleen, and the pineal gland.

Paul Broca’s (1824-1880) study of aphasia (speech deficit) in brain-damaged patients in 1861 reinvigorated the field and persuaded the medical establishment of the existence of localized areas of the brain responsible for specific cognitive functions. In particular, he showed that speech production was localized to a portion of the left hemisphere now called Broca’s area.” By that time, it was known that the brain consisted of nerve cells or neurons, but it was not until 1873 that Camillo Golgi (1843-1926) developed a staining technique allowing the observation of individual neurons in the brain (see Figure 1.2). This technique

6 Since then, it has been discovered that some species of dolphins and whales have relatively larger brains. The large size of human brains is now thought to be enabled in part by recent improvements in its cooling system. 7 Many cite Alexander Hood (1824) as a possible prior source.

Section 1.2. The Foundations of Artificial Intelligence 11

Axonal arborization

Axon from another cell

Synapse

Dendrite

Nucleus

Synapses

Figure 1.2 The parts of a nerve cell or neuron. Each neuron consists of a cell body, or soma, that contains a cell nucleus. Branching out from the cell body are a number of fibers called dendrites and a single long fiber called the axon. The axon stretches out for a long distance, much longer than the scale in this diagram indicates. Typically they are 1 cm long (100 times the diameter of the cell body), but can reach up to 1 meter. A neuron makes connections with 10 to 100,000 other neurons at junctions called synapses. Signals are propagated from neuron to neuron by a complicated electrochemical reaction. The signals control brain activity in the short term, and also enable long-term changes in the position and connectivity of neurons. These mechanisms are thought to form the basis for learning in the brain. Most information processing goes on in the cerebral cortex, the outer layer of the brain. The basic organizational unit appears to be a column of tissue about 0.5 mm in diameter, extending the full depth of the cortex, which is about 4 mm in humans. A column contains about 20,000 neurons.

was used by Santiago Ramon y Cajal (1852-1934) in his pioneering studies of the brain’s neuronal structures?

We now have some data on the mapping between areas of the brain and the parts of the body that they control or from which they receive sensory input. Such mappings are able to change radically over the course of a few weeks, and some animals seem to have multiple maps. Moreover, we do not fully understand how other areas can take over functions when one area is damaged. There is almost no theory on how an individual memory is stored.

The measurement of intact brain activity began in 1929 with the invention by Hans Berger of the electroencephalograph (EEG). The recent development of functional magnetic resonance imaging (fMRI) (Ogawa et al., 1990) is giving neuroscientists unprecedentedly detailed images of brain activity, enabling measurements that correspond in interesting ways to ongoing cognitive processes. These are augmented by advances in single-cell recording of

8 Golgi persisted in his belief that the brain’s functions were carried out primarily in a continuous medium in which neurons were embedded, whereas Cajal propounded the “neuronal doctrine.” The two shared the Nobel prize in 1906 but gave rather antagonistic acceptance speeches.

12 Chapter 1. Introduction Computer Human Brain Computational units 1 CPU, 10? gates 10! neurons Storage units 100 bits RAM 10! neurons 10" bits disk 10'4 synapses Cycle time 1079 sec 1073 sec Bandwidth 1010 bits/sec 10% bits/sec Memory updates/sec 10° 1014 Figure 1.3 A crude comparison of the raw computational resources available to computers (circa 2003) and brains. The computer’s numbers have all increased by at least a factor of 10 since the first edition of this book, and are expected to do so again this decade. The brain’s numbers have not changed in the last 10,000 years.

neuron activity. Despite these advances, we are still a long way from understanding how any of these cognitive processes actually work.

The truly amazing conclusion is that a collection of simple cells can lead to thought, action, and consciousness or, in other words, that brains cause minds (Searle, 1992). The only real alternative theory is mysticism: that there is some mystical realm in which minds operate that is beyond physical science.

Brains and digital computers perform quite different tasks and have different properties. Figure 1.3 shows that there are 1000 times more neurons in the typical human brain than there are gates in the CPU of a typical high-end computer. Moore’s Law” predicts that the CPU’s gate count will equal the brain’s neuron count around 2020. Of course, little can be inferred from such predictions; moreover, the difference in storage capacity is minor compared to the difference in switching speed and in parallelism. Computer chips can execute an instruction in a nanosecond, whereas neurons are millions of times slower. Brains more than make up for this, however, because all the neurons and synapses are active simultaneously, whereas most current computers have only one or at most a few CPUs. Thus, even though a computer is a million times faster in raw switching speed, the brain ends up being 100,000 times faster at what it does.

Psychology (1879—-present)

eo How do humans and animals think and act?

The origins of scientific psychology are usually traced to the work of the German physicist Hermann von Helmholtz (1821-1894) and his student Wilhelm Wundt (1832-1920). Helmholtz applied the scientific method to the study of human vision, and his Handbook of Physiological Optics is even now described as “the single most important treatise on the physics and physiology of human vision” (Nalwa, 1993, p.15). In 1879, Wundt opened the first laboratory of experimental psychology at the University of Leipzig. Wundt insisted on carefully controlled experiments in which his workers would perform a perceptual or associa

® Moore’s Law says that the number of transistors per square inch doubles every 1 to 1.5 years. Human brain capacity doubles roughly every 2 to 4 million years.

Section 1.2.

The Foundations of Artificial Intelligence 13

BEHAVIORISM

COGNITIVE PSYCHOLOGY

COGNITIVE SCIENCE

tive task while introspecting on their thought processes. The careful controls went a long way toward making psychology a science, but the subjective nature of the data made it unlikely that an experimenter would ever disconfirm his or her own theories. Biologists studying animal behavior, on the other hand, lacked introspective data and developed an objective methodology, as described by H. S. Jennings (1906) in his influential work Behavior of the Lower Organisms. Applying this viewpoint to humans, the behaviorism movement, led by John Watson (1878-1958), rejected any theory involving mental processes on the grounds that introspection could not provide reliable evidence. Behaviorists insisted on studying only objective measures of the percepts (or stimulus) given to an animal and its resulting actions (or response). Mental constructs such as knowledge, beliefs, goals, and reasoning steps were dismissed as unscientific “folk psychology.” Behaviorism discovered a lot about rats and pigeons, but had less success at understanding humans. Nevertheless, it exerted a strong hold on psychology (especially in the United States) from about 1920 to 1960.

The view of the brain as an information-processing device, which is a principal characteristic of cognitive psychology, can be traced back at least to the works of William James'® (1842-1910). Helmholtz also insisted that perception involved a form of unconscious logical inference. The cognitive viewpoint was largely eclipsed by behaviorism in the United States, but at Cambridge’s Applied Psychology Unit, directed by Frederic Bartlett (1886— 1969), cognitive modeling was able to flourish. The Nature of Explanation, by Bartlett’s student and successor Kenneth Craik (1943), forcefully reestablished the legitimacy of such “mental” terms as beliefs and goals, arguing that they are just as scientific as, say, using pressure and temperature to talk about gases, despite their being made of molecules that have neither. Craik specified the three key steps of a knowledge-based agent: (1) the stimulus must be translated into an internal representation, (2) the representation is manipulated by cognitive processes to derive new internal representations, and (3) these are in turn retranslated back into action. He clearly explained why this was a good design for an agent:

If the organism carries a “small-scale model” of external reality and of its own possible actions within its head, it is able to try out various alternatives, conclude which is the best of them, react to future situations before they arise, utilize the knowledge of past events in dealing with the present and future, and in every way to react in a much fuller, safer, and more competent manner to the emergencies which face it. (Craik, 1943)

After Craik’s death in a bicycle accident in 1945, his work was continued by Donald Broadbent, whose book Perception and Communication (1958) included some of the first information-processing models of psychological phenomena. Meanwhile, in the United States, the development of computer modeling led to the creation of the field of cognitive science. The field can be said to have started at a workshop in September 1956 at MIT. (We shall see that this is just two months after the conference at which Al itself was “born.”) At the workshop, George Miller presented The Magic Number Seven, Noam Chomsky presented Three Models of Language, and Allen Newell and Herbert Simon presented The Logic Theory Machine. These three influential papers showed how computer models could be used to

10 ‘William James was the brother of novelist Henry James. It is said that Henry wrote fiction as if it were psychology and William wrote psychology as if it were fiction.

14

Chapter 1. Introduction

address the psychology of memory, language, and logical thinking, respectively. It is now a common view among psychologists that “a cognitive theory should be like a computer program” (Anderson, 1980), that is, it should describe a detailed information-processing mechanism whereby some cognitive function might be implemented.

Computer engineering (1940-present)

e How can we build an efficient computer?

For artificial intelligence to succeed, we need two things: intelligence and an artifact. The computer has been the artifact of choice. The modern digital electronic computer was invented independently and almost simultaneously by scientists in three countries embattled in World War II. The first operational computer was the electromechanical Heath Robinson,'! built in 1940 by Alan Turing’s team for a single purpose: deciphering German messages. In 1943, the same group developed the Colossus, a powerful general-purpose machine based on vacuum tubes.'? The first operational programmable computer was the Z-3, the invention of Konrad Zuse in Germany in 1941. Zuse also invented floating-point numbers and the first high-level programming language, Plankalkiil. The first electronic computer, the ABC, was assembled by John Atanasoff and his student Clifford Berry between 1940 and 1942 at Iowa State University. Atanasoff’s research received little support or recognition; it was the ENIAC, developed as part of a secret military project at the University of Pennsylvania by a team including John Mauchly and John Eckert, that proved to be the most influential forerunner of modern computers.

In the half-century since then, each generation of computer hardware has brought an increase in speed and capacity and a decrease in price. Performance doubles every 18 months or so, with a decade or two to go at this rate of increase. After that, we will need molecular engineering or some other new technology.

Of course, there were calculating devices before the electronic computer. The earliest automated machines, dating from the 17th century, were discussed on page 6. The first programmable machine was a loom devised in 1805 by Joseph Marie Jacquard (1752-1834) that used punched cards to store instructions for the pattern to be woven. In the mid-19th century, Charles Babbage (1792-1871) designed two machines, neither of which he completed. The “Difference Engine,” which appears on the cover of this book, was intended to compute mathematical tables for engineering and scientific projects. It was finally built and shown to work in 1991 at the Science Museum in London (Swade, 1993). Babbage’s “Analytical Engine” was far more ambitious: it included addressable memory, stored programs, and conditional jumps and was the first artifact capable of universal computation. Babbage’s colleague Ada Lovelace, daughter of the poet Lord Byron, was perhaps the world’s first programmer. (The programming language Ada is named after her.) She wrote programs for the unfinished Analytical Engine and even speculated that the machine could play chess or compose music.

11 Heath Robinson was a cartoonist famous for his depictions of whimsical and absurdly complicated contraptions for everyday tasks such as buttering toast.

12 In the postwar period, Turing wanted to use these computers for Al research—for example, one of the first chess programs (Turing et al., 1953). His efforts were blocked by the British government.

Section 1.2.

The Foundations of Artificial Intelligence 15

CONTROL THEORY

CYBERNETICS

OBJECTIVE FUNCTION

Al also owes a debt to the software side of computer science, which has supplied the operating systems, programming languages, and tools needed to write modern programs (and papers about them). But this is one area where the debt has been repaid: work in Al has pioneered many ideas that have made their way back to mainstream computer science, including time sharing, interactive interpreters, personal computers with windows and mice, rapid development environments, the linked list data type, automatic storage management, and key concepts of symbolic, functional, dynamic, and object-oriented programming.

Control theory and Cybernetics (1948—present) e How can artifacts operate under their own control?

Ktesibios of Alexandria (c. 250 B.C.) built the first self-controlling machine: a water clock with a regulator that kept the flow of water running through it at a constant, predictable pace. This invention changed the definition of what an artifact could do. Previously, only living things could modify their behavior in response to changes in the environment. Other examples of self-regulating feedback control systems include the steam engine governor, created by James Watt (1736-1819), and the thermostat, invented by Cornelis Drebbel (1572-1633), who also invented the submarine. The mathematical theory of stable feedback systems was developed in the 19th century.

The central figure in the creation of what is now called control theory was Norbert Wiener (1894-1964). Wiener was a brilliant mathematician who worked with Bertrand Russell, among others, before developing an interest in biological and mechanical control systems and their connection to cognition. Like Craik (who also used control systems as psychological models), Wiener and his colleagues Arturo Rosenblueth and Julian Bigelow challenged the behaviorist orthodoxy (Rosenblueth er al., 1943). They viewed purposive behavior as arising from a regulatory mechanism trying to minimize “error”—the difference between current state and goal state. In the late 1940s, Wiener, along with Warren McCulloch, Walter Pitts, and John von Neumann, organized a series of conferences that explored the new mathematical and computational models of cognition and influenced many other researchers in the behavioral sciences. Wiener’s book Cybernetics (1948) became a bestseller and awoke the public to the possibility of artificially intelligent machines.

Modern control theory, especially the branch known as stochastic optimal control, has as its goal the design of systems that maximize an objective function over time. This roughly matches our view of Al: designing systems that behave optimally. Why, then, are Al and control theory two different fields, especially given the close connections among their founders? The answer lies in the close coupling between the mathematical techniques that were familiar to the participants and the corresponding sets of problems that were encompassed in each world view. Calculus and matrix algebra, the tools of control theory, lend themselves to systems that are describable by fixed sets of continuous variables; furthermore, exact analysis is typically feasible only for linear systems. AI was founded in part as a way to escape from the limitations of the mathematics of control theory in the 1950s. The tools of logical inference and computation allowed Al researchers to consider some problems such as language, vision, and planning, that fell completely outside the control theorist’s purview.

16 Chapter 1. Introduction

Linguistics (1957—-present) ¢ How does language relate to thought?

In 1957, B. FE. Skinner published Verbal Behavior. This was a comprehensive, detailed account of the behaviorist approach to language learning, written by the foremost expert in the field. But curiously, a review of the book became as well known as the book itself, and served to almost kill off interest in behaviorism. The author of the review was Noam Chomsky, who had just published a book on his own theory, Syntactic Structures. Chomsky showed how the behaviorist theory did not address the notion of creativity in language—it did not explain how a child could understand and make up sentences that he or she had never heard before. Chomsky’s theory—based on syntactic models going back to the Indian linguist Panini (c. 350 B.C.)—could explain this, and unlike previous theories, it was formal enough that it could in principle be programmed. Modern linguistics and Al, then, were “born” at about the same time, and grew up COMPLTATIONAL together, intersecting in a hybrid field called computational linguistics or natural language processing. The problem of understanding language soon turned out to be considerably more complex than it seemed in 1957. Understanding language requires an understanding of the subject matter and context, not just an understanding of the structure of sentences. This might seem obvious, but it was not widely appreciated until the 1960s. Much of the early work in knowledge representation (the study of how to put knowledge into a form that a computer can reason with) was tied to language and informed by research in linguistics, which was connected in turn to decades of work on the philosophical analysis of language.

1.3 THE HISTORY OF ARTIFICIAL INTELLIGENCE

With the background material behind us, we are ready to cover the development of Al itself.

The gestation of artificial intelligence (1943-1955)

The first work that is now generally recognized as Al was done by Warren McCulloch and Walter Pitts (1943). They drew on three sources: knowledge of the basic physiology and function of neurons in the brain; a formal analysis of propositional logic due to Russell and Whitehead; and Turing’s theory of computation. They proposed a model of artificial neurons in which each neuron is characterized as being “on” or “off,” with a switch to “on” occurring in response to stimulation by a sufficient number of neighboring neurons. The state of a neuron was conceived of as “factually equivalent to a proposition which proposed its adequate stimulus.” They showed, for example, that any computable function could be computed by some network of connected neurons, and that all the logical connectives (and, or, not, etc.) could be implemented by simple net structures. McCulloch and Pitts also suggested that suitably defined networks could learn. Donald Hebb (1949) demonstrated a simple updating rule for modifying the connection strengths between neurons. His rule, now called Hebbian learning, remains an influential model to this day.

Section 1.3.

The History of Artificial Intelligence 17

Two graduate students in the Princeton mathematics department, Marvin Minsky and Dean Edmonds, built the first neural network computer in 1951. The SNARC, as it was called, used 3000 vacuum tubes and a surplus automatic pilot mechanism from a B-24 bomber to simulate a network of 40 neurons. Minsky’s Ph.D. committee was skeptical about whether this kind of work should be considered mathematics, but von Neumann reportedly said, “If it isn’t now, it will be someday.” Minsky was later to prove influential theorems showing the limitations of neural network research.

There were a number of early examples of work that can be characterized as Al but it was Alan Turing who first articulated a complete vision of Al in his 1950 article “Computing Machinery and Intelligence.” Therein, he introduced the Turing test, machine learning, genetic algorithms, and reinforcement learning.

The birth of artificial intelligence (1956)

Princeton was home to another influential figure in Al, John McCarthy. After graduation, McCarthy moved to Dartmouth College, which was to become the official birthplace of the field. McCarthy convinced Minsky, Claude Shannon, and Nathaniel Rochester to help him bring together U.S. researchers interested in automata theory, neural nets, and the study of intelligence. They organized a two-month workshop at Dartmouth in the summer of 1956. There were 10 attendees in all, including Trenchard More from Princeton, Arthur Samuel from IBM, and Ray Solomonoff and Oliver Selfridge from MIT.

Two researchers from Carnegie Tech,!® Allen Newell and Herbert Simon, rather stole the show. Although the others had ideas and in some cases programs for particular applications such as checkers, Newell and Simon already had a reasoning program, the Logic Theorist (LT), about which Simon claimed, “We have invented a computer program capable of thinking non-numerically, and thereby solved the venerable mind-body problem.”'\* Soon after the workshop, the program was able to prove most of the theorems in Chapter 2 of Russell and Whitehead’s Principia Mathematica. Russell was reportedly delighted when Simon showed him that the program had come up with a proof for one theorem that was shorter than the one in Principia. The editors of the Journal of Symbolic Logic were less impressed; they rejected a paper coauthored by Newell, Simon, and Logic Theorist.

The Dartmouth workshop did not lead to any new breakthroughs, but it did introduce all the major figures to each other. For the next 20 years, the field would be dominated by these people and their students and colleagues at MIT, CMU, Stanford, and IBM. Perhaps the longest-lasting thing to come out of the workshop was an agreement to adopt McCarthy’s new name for the field: artificial intelligence. Perhaps “computational rationality” would have been better, but “AI” has stuck.

Looking at the proposal for the Dartmouth workshop (McCarthy et al., 1955), we can see why it was necessary for Al to become a separate field. Why couldn’t all the work done

13 Now Carnegie Mellon University (CMU).

14 Newell and Simon also invented a list-processing language, IPL, to write LT. They had no compiler, and translated it into machine code by hand. To avoid errors, they worked in parallel, calling out binary numbers to each other as they wrote each instruction to make sure they agreed.

18

Chapter 1. Introduction

PHYSICAL SYMBOL SYSTEM

LISP

in AI have taken place under the name of control theory, or operations research, or decision theory, which, after all, have objectives similar to those of AI? Or why isn’t Al a branch of mathematics? The first answer is that Al from the start embraced the idea of duplicating human faculties like creativity, self-improvement, and language use. None of the other fields were addressing these issues. The second answer is methodology. Al is the only one of these fields that is clearly a branch of computer science (although operations research does share an emphasis on computer simulations), and Al is the only field to attempt to build machines that will function autonomously in complex, changing environments.

Early enthusiasm, great expectations (1952-1969)

The early years of Al were full of successes—in a limited way. Given the primitive computers and programming tools of the time, and the fact that only a few years earlier computers were seen as things that could do arithmetic and no more, it was astonishing whenever a computer did anything remotely clever. The intellectual establishment, by and large, preferred to believe that “a machine can never do X.” (See Chapter 26 for a long list of X’s gathered by Turing.) Al researchers naturally responded by demonstrating one X after another. John McCarthy referred to this period as the “Look, Ma, no hands!” era.

Newell and Simon’s early success was followed up with the General Problem Solver, or GPS. Unlike Logic Theorist, this program was designed from the start to imitate human problem-solving protocols. Within the limited class of puzzles it could handle, it turned out that the order in which the program considered subgoals and possible actions was similar to that in which humans approached the same problems. Thus, GPS was probably the first program to embody the “thinking humanly” approach. The success of GPS and subsequent programs as models of cognition led Newell and Simon (1976) to formulate the famous physical symbol system hypothesis, which states that “a physical symbol system has the necessary and sufficient means for general intelligent action.” What they meant is that any system (human or machine) exhibiting intelligence must operate by manipulating data structures composed of symbols. We will see later that this hypothesis has been challenged from many directions.

At IBM, Nathaniel Rochester and his colleagues produced some of the first Al programs. Herbert Gelernter (1959) constructed the Geometry Theorem Prover, which was able to prove theorems that many students of mathematics would find quite tricky. Starting in 1952, Arthur Samuel wrote a series of programs for checkers (draughts) that eventually learned to play at a strong amateur level. Along the way, he disproved the idea that computers can do only what they are told to: his program quickly learned to play a better game than its creator. The program was demonstrated on television in February 1956, creating a very strong impression. Like Turing, Samuel had trouble finding computer time. Working at night, he used machines that were still on the testing floor at IBM’s manufacturing plant. Chapter 6 covers game playing, and Chapter 21 describes and expands on the learning techniques used by Samuel.

John McCarthy moved from Dartmouth to MIT and there made three crucial contributions in one historic year: 1958. In MIT AI Lab Memo No. 1, McCarthy defined the high-level language Lisp, which was to become the dominant Al programming language. Lisp is the

Section 1.3.

The History of Artificial Intelligence 19

MICROWORLDS

second-oldest major high-level language in current use, one year younger than FORTRAN. With Lisp, McCarthy had the tool he needed, but access to scarce and expensive computing resources was also a serious problem. In response, he and others at MIT invented time sharing. Also in 1958, McCarthy published a paper entitled Programs with Common Sense, in which he described the Advice Taker, a hypothetical program that can be seen as the first complete Al system. Like the Logic Theorist and Geometry Theorem Prover, McCarthy’s program was designed to use knowledge to search for solutions to problems. But unlike the others, it was to embody general knowledge of the world. For example, he showed how some simple axioms would enable the program to generate a plan to drive to the airport to catch a plane. The program was also designed so that it could accept new axioms in the normal course of operation, thereby allowing it to achieve competence in new areas without being reprogrammed. The Advice Taker thus embodied the central principles of knowledge representation and reasoning: that it is useful to have a formal, explicit representation of the world and of the way an agent’s actions affect the world and to be able to manipulate these representations with deductive processes. It is remarkable how much of the 1958 paper remains relevant even today.

1958 also marked the year that Marvin Minsky moved to MIT. His initial collaboration with McCarthy did not last, however. McCarthy stressed representation and reasoning in formal logic, whereas Minsky was more interested in getting programs to work and eventually developed an anti-logical outlook. In 1963, McCarthy started the Al lab at Stanford. His plan to use logic to build the ultimate Advice Taker was advanced by J. A. Robinson’s discovery of the resolution method (a complete theorem-proving algorithm for first-order logic; see Chapter 9). Work at Stanford emphasized general-purpose methods for logical reasoning. Applications of logic included Cordell Green’s question-answering and planning systems (Green, 1969b) and the Shakey robotics project at the new Stanford Research Institute (SRI). The latter project, discussed further in Chapter 25, was the first to demonstrate the complete integration of logical reasoning and physical activity.

Minsky supervised a series of students who chose limited problems that appeared to require intelligence to solve. These limited domains became known as microworlds. James Slagle’s SAINT program (1963a) was able to solve closed-form calculus integration problems typical of first-year college courses. Tom Evans’s ANALOGY program (1968) solved geometric analogy problems that appear in IQ tests, such as the one in Figure 1.4. Daniel Bobrow’s STUDENT program (1967) solved algebra story problems, such as the following:

If the number of customers Tom gets is twice the square of 20 percent of the number of advertisements he runs, and the number of advertisements he runs is 45, what is the number of customers Tom gets?

The most famous microworld was the blocks world, which consists of a set of solid blocks placed on a tabletop (or more often, a simulation of a tabletop), as shown in Figure 1.5. A typical task in this world is to rearrange the blocks in a certain way, using a robot hand that can pick up one block at a time. The blocks world was home to the vision project of David Huffman (1971), the vision and constraint-propagation work of David Waltz (1975), the learning theory of Patrick Winston (1970), the natural language understanding program

[]

20

Section 1.3.

The History of Artificial Intelligence 21

1960; Widrow, 1962), who called his networks adalines, and by Frank Rosenblatt (1962) with his perceptrons. Rosenblatt proved the perceptron convergence theorem, showing that his learning algorithm could adjust the connection strengths of a perceptron to match any input data, provided such a match existed. These topics are covered in Chapter 20.

A dose of reality (1966-1973)

From the beginning, Al researchers were not shy about making predictions of their coming successes. The following statement by Herbert Simon in 1957 is often quoted:

It is not my aim to surprise or shock you—but the simplest way I can summarize is to say that there are now in the world machines that think, that learn and that create. Moreover, their ability to do these things is going to increase rapidly until—in a visible future—the range of problems they can handle will be coextensive with the range to which the human mind has been applied.

Terms such as “visible future” can be interpreted in various ways, but Simon also made a more concrete prediction: that within 10 years a computer would be chess champion, and a significant mathematical theorem would be proved by machine. These predictions came true (or approximately true) within 40 years rather than 10. Simon’s over-confidence was due to the promising performance of early Al systems on simple examples. In almost all cases, however, these early systems turned out to fail miserably when tried out on wider selections of problems and on more difficult problems.

The first kind of difficulty arose because most early programs contained little or no knowledge of their subject matter; they succeeded by means of simple syntactic manipulations. A typical story occurred in early machine translation efforts, which were generously funded by the U.S. National Research Council in an attempt to speed up the translation of Russian scientific papers in the wake of the Sputnik launch in 1957. It was thought initially that simple syntactic transformations based on the grammars of Russian and English, and word replacement using an electronic dictionary, would suffice to preserve the exact meanings of sentences. The fact is that translation requires general knowledge of the subject matter in order to resolve ambiguity and establish the content of the sentence. The famous re-translation of “the spirit is willing but the flesh is weak” as “the vodka is good but the meat is rotten” illustrates the difficulties encountered. In 1966, a report by an advisory committee found that “there has been no machine translation of general scientific text, and none is in immediate prospect.” All U.S. government funding for academic translation projects was canceled. Today, machine translation is an imperfect but widely used tool for technical, commercial, government, and Internet documents.

The second kind of difficulty was the intractability of many of the problems that Al was attempting to solve. Most of the early Al programs solved problems by trying out different combinations of steps until the solution was found. This strategy worked initially because microworlds contained very few objects and hence very few possible actions and very short solution sequences. Before the theory of computational complexity was developed, it was widely thought that “scaling up” to larger problems was simply a matter of taster hardware and larger memories. The optimism that accompanied the development of resolution theorem

22

Chapter 1. Introduction

= lL £2

MACHINE EVOLUTION

WEAK METHODS

proving, for example, was soon dampened when researchers failed to prove theorems involving more than a few dozen facts. The fact that a program can find a solution in principle does not mean that the program contains any of the mechanisms needed to find it in practice.

The illusion of unlimited computational power was not confined to problem-solving programs. Early experiments in machine evolution (now called genetic algorithms) (Friedberg, 1958; Friedberg ef al, 1959) were based on the undoubtedly correct belief that by making an appropriate series of small mutations to a machine code program, one can generate a program with good performance for any particular simple task. The idea, then, was to try random mutations with a selection process to preserve mutations that seemed useful. Despite thousands of hours of CPU time, almost no progress was demonstrated. Modern genetic algorithms use better representations and have shown more success.

Failure to come to grips with the “combinatorial explosion” was one of the main criticisms of Al contained in the Lighthill report (Lighthill, 1973), which formed the basis for the decision by the British government to end support for Al research in all but two universities. (Oral tradition paints a somewhat different and more colorful picture, with political ambitions and personal animosities whose description is beside the point.)

A third difficulty arose because of some fundamental limitations on the basic structures being used to generate intelligent behavior. For example, Minsky and Papert’s book Perceptrons (1969) proved that, although perceptrons (a simple form of neural network) could be shown to learn anything they were capable of representing, they could represent very little. In particular, a two-input perceptron could not be trained to recognize when its two inputs were different. Although their results did not apply to more complex, multilayer networks, research funding for neural-net research soon dwindled to almost nothing. Ironically, the new back-propagation learning algorithms for multilayer networks that were to cause an enormous resurgence in neural-net research in the late 1980s were actually discovered first in 1969 (Bryson and Ho, 1969).

Knowledge-based systems: The key to power? (1969-1979)

The picture of problem solving that had arisen during the first decade of Al research was of a general-purpose search mechanism trying to string together elementary reasoning steps to find complete solutions. Such approaches have been called weak methods, because, although general, they do not scale up to large or difficult problem instances. The alternative to weak methods is to use more powerful, domain-specific knowledge that allows larger reasoning steps and can more easily handle typically occurring cases in narrow areas of expertise. One might say that to solve a hard problem, you have to almost know the answer already.

The DENDRAL program (Buchanan et al., 1969) was an early example of this approach. It was developed at Stanford, where Ed Feigenbaum (a former student of Herbert Simon), Bruce Buchanan (a philosopher turned computer scientist), and Joshua Lederberg (a Nobel laureate geneticist) teamed up to solve the problem of inferring molecular structure from the information provided by a mass spectrometer. The input to the program consists of the elementary formula of the molecule (e.g., C¢H13NO3) and the mass spectrum giving the masses of the various fragments of the molecule generated when itis bombarded by an electron beam.

Section 1.3.

The History of Artificial Intelligence 23

EXPERT SYSTEMS

For example, the mass spectrum might contain a peak at m = 15, corresponding to the mass of a methyl (CH3) fragment.

The naive version of the program generated all possible structures consistent with the formula, and then predicted what mass spectrum would be observed for each, comparing this with the actual spectrum. As one might expect, this is intractable for decent-sized molecules. The DENDRAL researchers consulted analytical chemists and found that they worked by looking for well-known patterns of peaks in the spectrum that suggested common substructures in the molecule. For example, the following rule is used to recognize a ketone (C=0) subgroup (which weighs 28):

if there are two peaks at x, and x5 such that

(@y x1 + x2 = M + 28 (M is the mass of the whole molecule), (b) x; — 28 is a high peak;

(c) zz — 28 is a high peak;

(d) At least one of x; and zs is high.

then there is a ketone subgroup

Recognizing that the molecule contains a particular substructure reduces the number of possible candidates enormously. DENDRAL was powerful because

All the relevant theoretical knowledge to solve these problems has been mapped over from its general form in the [spectrum prediction component] (“first principles”) to efficient special forms (“cookbook recipes”). (Feigenbaum et al., 1971)

The significance of DENDRAL was that it was the first successful knowledge-intensive system: its expertise derived from large numbers of special-purpose rules. Later systems also incorporated the main theme of McCarthy’s Advice Taker approach—the clean separation of the knowledge (in the form of rules) from the reasoning component.

With this lesson in mind, Feigenbaum and others at Stanford began the Heuristic Programming Project (HPP), to investigate the extent to which the new methodology of expert systems could be applied to other areas of human expertise. The next major effort was in the area of medical diagnosis. Feigenbaum, Buchanan, and Dr. Edward Shortliffe developed MYCIN to diagnose blood infections. With about 450 rules, MYCIN was able to perform as well as some experts, and considerably better than junior doctors. It also contained two major differences from DENDRAL. First, unlike the DENDRAL rules, no general theoretical model existed from which the MYCIN rules could be deduced. They had to be acquired from extensive interviewing of experts, who in turn acquired them from textbooks, other experts, and direct experience of cases. Second, the rules had to reflect the uncertainty associated with medical knowledge. MYCIN incorporated a calculus of uncertainty called certainty factors (see Chapter 13), which seemed (at the time) to fit well with how doctors assessed the impact of evidence on the diagnosis.

The importance of domain knowledge was also apparent in the area of understanding natural language. Although Winograd’s SHRDLU system for understanding natural language had engendered a good deal of excitement, its dependence on syntactic analysis caused some of the same problems as occurred in the early machine translation work. It was able to overcome ambiguity and understand pronoun references, but this was mainly because it was

24

Chapter 1. Introduction

FRAMES

designed specifically for one area—the blocks world. Several researchers, including Eugene Charniak, a fellow graduate student of Winograd’s at MIT, suggested that robust language understanding would require general knowledge about the world and a general method for using that knowledge.

At Yale, the linguist-turnedAl-researcher Roger Schank emphasized this point, claiming, “There is no such thing as syntax,” which upset a lot of linguists, but did serve to start a useful discussion. Schank and his students built a series of programs (Schank and Abelson, 1977; Wilensky, 1978; Schank and Riesbeck, 1981; Dyer, 1983) that all had the task of understanding natural language. The emphasis, however, was less on language per se and more on the problems of representing and reasoning with the knowledge required for language understanding. The problems included representing stereotypical situations (Cullingford, 1981), describing human memory organization (Rieger, 1976; Kolodner, 1983), and understanding plans and goals (Wilensky, 1983).

The widespread growth of applications to real-world problems caused a concurrent increase in the demands for workable knowledge representation schemes. A large number of different representation and reasoning languages were developed. Some were based on logic—for example, the Prolog language became popular in Europe, and the PLANNER family in the United States. Others, following Minsky’s idea of frames (1975), adopted a more structured approach, assembling facts about particular object and event types and arranging the types into a large taxonomic hierarchy analogous to a biological taxonomy.

Al becomes an industry (1980—present)

The first successful commercial expert system, R1, began operation at the Digital Equipment Corporation (McDermott, 1982). The program helped configure orders for new computer systems; by 1986, it was saving the company an estimated $40 million a year. By 1988, DEC’s AI group had 40 expert systems deployed, with more on the way. Du Pont had 100 in use and 500 in development, saving an estimated $10 million a year. Nearly every major U.S. corporation had its own AI group and was either using or investigating expert systems.

In 1981, the Japanese announced the “Fifth Generation” project, a 10-year plan to build intelligent computers running Prolog. In response the United States formed the Microelectronics and Computer Technology Corporation (MCC) as a research consortium designed to assure national competitiveness. In both cases, Al was part of a broad effort, including chip design and human-interface research. However, the AI components of MCC and the Fifth Generation projects never met their ambitious goals. In Britain, the Alvey report reinstated the funding that was cut by the Lighthill report."

Overall, the Al industry boomed from a few million dollars in 1980 to billions of dollars in 1988. Soon after that came a period called the “Al Winter,” in which many companies suffered as they failed to deliver on extravagant promises.

15 To save embarrassment, a new field called IKBS (Intelligent Knowledge-Based Systems) was invented because Artificial Intelligence had been officially canceled.

Section 1.3.

The History of Artificial Intelligence 25

CONNECTIONIST

The return of neural networks (1986—present)

Although computer science had largely abandoned the field of neural networks in the late 1970s, work continued in other fields. Physicists such as John Hopfield (1982) used techniques from statistical mechanics to analyze the storage and optimization properties of networks, treating collections of nodes like collections of atoms. Psychologists including David Rumelhart and Geoff Hinton continued the study of neural-net models of memory. As we discuss in Chapter 20, the real impetus came in the mid-1980s when at least four different groups reinvented the back-propagation learning algorithm first found in 1969 by Bryson and Ho. The algorithm was applied to many learning problems in computer science and psychology, and the widespread dissemination of the results in the collection Parallel Distributed Processing (Rumelhart and McClelland, 1986) caused great excitement.

These so-called connectionist models of intelligent systems were seen by some as direct competitors both to the symbolic models promoted by Newell and Simon and to the logicist approach of McCarthy and others (Smolensky, 1988). It might seem obvious that at some level humans manipulate symbols—in fact, Terrence Deacon’s book The Symbolic Species (1997) suggests that this is the defining characteristic of humans, but the most ardent connectionists questioned whether symbol manipulation had any real explanatory role in detailed models of cognition. This question remains unanswered, but the current view is that connectionist and symbolic approaches are complementary, not competing.

Al becomes a science (1987—-present)

Recent years have seen a revolution in both the content and the methodology of work in artificial intelligence.'6 It is now more common to build on existing theories than to propose brand new ones, to base claims on rigorous theorems or hard experimental evidence rather than on intuition, and to show relevance to real-world applications rather than toy examples.

Al was founded in part as a rebellion against the limitations of existing fields like control theory and statistics, but now it is embracing those fields. As David McAllester (1998) put it,

In the early period of Al it seemed plausible that new forms of symbolic computation, e.g., frames and semantic networks, made much of classical theory obsolete. This led to a form of isolationism in which AI became largely separated from the rest of computer science. This isolationism is currently being abandoned. There is a recognition that machine learning should not be isolated from information theory, that uncertain reasoning should not be isolated from stochastic modeling, that search should not be isolated from classical optimization and control, and that automated reasoning should not be isolated from formal methods and static analysis.

In terms of methodology, Al has finally come firmly under the scientific method. To be accepted, hypotheses must be subjected to rigorous empirical experiments, and the results must

16 Some have characterized this change as a victory of the neats—those who think that AI theories should be grounded in mathematical rigor—over the scruffies—those who would rather try out lots of ideas, write some programs, and then assess what seems to be working. Both approaches are important, A shift toward neatness implies that the field has reached a level of stability and maturity. Whether that stability will be disrupted by a new scruffy idea is another question,

26

Chapter 1. Introduction

DATA MINING

be analyzed statistically for their importance (Cohen, 1995). Through the use of the Internet and shared repositories of test data and code, it is now possible to replicate experiments.

The field of speech recognition illustrates the pattern. In the 1970s, a wide variety of different architectures and approaches were tried. Many of these were rather ad hoc and fragile, and were demonstrated on only a few specially selected examples. In recent years, approaches based on hidden Markov models (HMMs) have come to dominate the area. Two aspects of HMMs are relevant. First, they are based on a rigorous mathematical theory. This has allowed speech researchers to build on several decades of mathematical results developed in other fields. Second, they are generated by a process of training on a large corpus of real speech data. This ensures that the performance is robust, and in rigorous blind tests the HMMs have been improving their scores steadily. Speech technology and the related field of handwritten character recognition are already making the transition to widespread industrial and consumer applications.

Neural networks also fit this trend. Much of the work on neural nets in the 1980s was done in an attempt to scope out what could be done and to learn how neural nets differ from “traditional” techniques. Using improved methodology and theoretical frameworks, the field arrived at an understanding in which neural nets can now be compared with corresponding techniques from statistics, pattern recognition, and machine learning, and the most promising technique can be applied to each application. As a result of these developments, so-called data mining technology has spawned a vigorous new industry.

Judea Pearl’s (1988) Probabilistic Reasoning in Intelligent Systems led to a new acceptance of probability and decision theory in Al, following a resurgence of interest epitomized by Peter Cheeseman’s (1985) article “In Defense of Probability.” The Bayesian network formalism was invented to allow efficient representation of, and rigorous reasoning with, uncertain knowledge. This approach largely overcomes many problems of the probabilistic reasoning systems of the 1960s and 1970s; it now dominates Al research on uncertain reasoning and expert systems. The approach allows for learning from experience, and it combines the best of classical AI and neural nets. Work by Judea Pearl (1982a) and by Eric Horvitz and David Heckerman (Horvitz and Heckerman, 1986; Horvitz er al., 1986) promoted the idea of normative expert systems: ones that act rationally according to the laws of decision theory and do not try to imitate the thought steps of human experts. The Windows™ operating system includes several normative diagnostic expert systems for correcting problems. Chapters 13 to 16 cover this area.

Similar gentle revolutions have occurred in robotics, computer vision, and knowledge representation. A better understanding of the problems and their complexity properties, combined with increased mathematical sophistication, has led to workable research agendas and robust methods. In many cases, formalization and specialization have also led to fragmentation: topics such as vision and robotics are increasingly isolated from “mainstream” Al work. The unifying view of Al as rational agent design is one that can bring unity back to these disparate fields.

Section 1.4.

The State of the Art 27

The emergence of intelligent agents (1995—present)

Perhaps encouraged by the progress in solving the subproblems of Al, researchers have also started to look at the “whole agent” problem again. The work of Allen Newell, John Laird, and Paul Rosenbloom on SOAR (Newell, 1990; Laird et al., 1987) is the best-known example of a complete agent architecture. The so-called situated movement aims to understand the workings of agents embedded in real environments with continuous sensory inputs. One of the most important environments for intelligent agents is the Internet. Al systems have become so common in web-based applications that the “-bot” suffix has entered everyday language. Moreover, Al technologies underlie many Internet tools, such as search engines, recommender systems, and Web site construction systems.

Besides the first edition of this text (Russell and Norvig, 1995), other recent texts have also adopted the agent perspective (Poole et al., 1998; Nilsson, 1998). One consequence of trying to build complete agents is the realization that the previously isolated subfields of Al might need to be reorganized somewhat when their results are to be tied together. In particular, it is now widely appreciated that sensory systems (vision, sonar, speech recognition, etc.) cannot deliver perfectly reliable information about the environment. Hence, reasoning and planning systems must be able to handle uncertainty. A second major consequence of the agent perspective is that Al has been drawn into much closer contact with other fields, such as control theory and economics, that also deal with agents.

What can Al do today? A concise answer is difficult, because there are so many activities in so many subfields. Here we sample a few applications; others appear throughout the book.

Autonomous planning and scheduling: A hundred million miles from Earth, NASA's Remote Agent program became the first on-board autonomous planning program to control the scheduling of operations for a spacecraft (Jonsson er al., 2000). Remote Agent generated plans from high-level goals specified from the ground, and it monitored the operation of the spacecraft as the plans were executed—detecting, diagnosing, and recovering from problems as they occurred.

Game playing: IBM’s Deep Blue became the first computer program to defeat the world champion in a chess match when it bested Garry Kasparov by a score of 3.5 to 2.5 in an exhibition match (Goodman and Keene, 1997). Kasparov said that he felt a “new kind of intelligence” across the board from him. Newsweek magazine described the match as “The brain’s last stand.” The value of IBM’s stock increased by $18 billion.

Autonomous control: The ALVINN computer vision system was trained to steer a car to keep it following a lane. It was placed in CMU’s NAVLAB computer-controlled minivan and used to navigate across the United States—for 2850 miles it was in control of steering the vehicle 98% of the time. A human took over the other 2%, mostly at exit ramps. NAVL AB has video cameras that transmit road images to ALVINN, which then computes the best direction to steer, based on experience from previous training runs.

28

Chapter 1. Introduction

Diagnosis: Medical diagnosis programs based on probabilistic analysis have been able to perform at the level of an expert physician in several areas of medicine. Heckerman (1991) describes a case where a leading expert on lymph-node pathology scoffs at a program’s diagnosis of an especially difficult case. The creators of the program suggest he ask the computer for an explanation of the diagnosis. The machine points out the major factors influencing its decision and explains the subtle interaction of several of the symptoms in this case. Eventually, the expert agrees with the program.

Logistics Planning: During the Persian Gulf crisis of 1991, U.S. forces deployed a Dynamic Analysis and Replanning Tool, DART (Cross and Walker, 1994), to do automated logistics planning and scheduling for transportation. This involved up to 50,000 vehicles, cargo, and people at a time, and had to account for starting points, destinations, routes, and conflict resolution among all parameters. The Al planning techniques allowed a plan to be generated in hours that would have taken weeks with older methods. The Defense Advanced Research Project Agency (DARPA) stated that this single application more than paid back DARPA’s 30-year investment in AL

Robotics: Many surgeons now use robot assistants in microsurgery. HipNav (DiGioia et al., 1996) is a system that uses computer vision techniques to create a three-dimensional model of a patient’s internal anatomy and then uses robotic control to guide the insertion of a hip replacement prosthesis.

Language understanding and problem solving: PROVERB (Littman et al., 1999) is a computer program that solves crossword puzzles better than most humans, using constraints on possible word fillers, a large database of past puzzles, and a variety of information sources including dictionaries and online databases such as a list of movies and the actors that appear in them. For example, it determines that the clue “Nice Story” can be solved by “ETAGE” because its database includes the clue/solution pair “Story in France/ETAGE” and because it recognizes that the patterns “Nice X” and “X in France” often have the same solution. The program does not know that Nice is a city in France, but it can solve the puzzle.

These are just a few examples of artificial intelligence systems that exist today. Not magic or science fiction—but rather science, engineering, and mathematics, to which this book provides an introduction.

This chapter defines AI and establishes the cultural background against which it has developed. Some of the important points are as follows:

o Different people think of Al differently. Two important questions to ask are: Are you concerned with thinking or behavior? Do you want to model humans or work from an ideal standard?

¢ In this book, we adopt the view that intelligence is concerned mainly with rational action. Ideally, an intelligent agent takes the best possible action in a situation. We will study the problem of building agents that are intelligent in this sense.

Section 1.5. Summary 29

¢ Philosophers (going back to 400 B.C.) made Al conceivable by considering the ideas that the mind is in some ways like a machine, that it operates on knowledge encoded in some internal language, and that thought can be used to choose what actions to take.

¢ Mathematicians provided the tools to manipulate statements of logical certainty as well as uncertain, probabilistic statements. They also set the groundwork for understanding computation and reasoning about algorithms.

o Economists formalized the problem of making decisions that maximize the expected outcome to the decision-maker.

s Psychologists adopted the idea that humans and animals can be considered informationprocessing machines. Linguists showed that language use fits into this model.

s Computer engineers provided the artifacts that make Al applications possible. Al programs tend to be large, and they could not work without the great advances in speed and memory that the computer industry has provided.

¢ Control theory deals with designing devices that act optimally on the basis of feedback from the environment. Initially, the mathematical tools of control theory were quite different from Al, but the fields are coming closer together.

s The history of Al has had cycles of success, misplaced optimism, and resulting cutbacks in enthusiasm and funding. There have also been cycles of introducing new creative approaches and systematically refining the best ones.

¢ Al has advanced more rapidly in the past decade because of greater use of the scientific method in experimenting with and comparing approaches.

o Recent progress in understanding the theoretical basis for intelligence has gone hand in hand with improvements in the capabilities of real systems. The subfields of AI have become more integrated, and Al has found common ground with other disciplines.

BIBLIOGRAPHICAL AND HISTORICAL NOTES

The methodological status of artificial intelligence is investigated in The Sciences of the Artificial, by Herb Simon (1981), which discusses research areas concerned with complex artifacts. It explains how Al can be viewed as both science and mathematics. Cohen (1995) gives an overview of experimental methodology within Al. Ford and Hayes (1995) give an opinionated view of the usefulness of the Turing Test.

Artificial Intelligence: The Very Idea, by John Haugeland (1985) gives a readable account of the philosophical and practical problems of Al. Cognitive science is well described by several recent texts (Johnson-Laird, 1988; Stillings et al., 1995; Thagard, 1996) and by the Encyclopedia of the Cognitive Sciences (Wilson and Keil, 1999). Baker (1989) covers the syntactic part of modern linguistics, and Chierchia and McConnell-Ginet (1990) cover semantics. Jurafsky and Martin (2000) cover computational linguistics.

Early Al is described in Feigenbaum and Feldman’s Computers and Thought (1963), Minsky’s Semantic Information Processing (1968), and the Machine Intelligence series edited by Donald Michie. A large number of influential papers have been anthologized by Webber

30

Chapter 1. Introduction

and Nilsson (1981) and by Luger (1995). Early papers on neural networks are collected in Neurocomputing (Anderson and Rosenfeld, 1988). The Encyclopedia of AI (Shapiro, 1992) contains survey articles on almost every topic in AL These articles usually provide a good entry point into the research literature on each topic.

The most recent work appears in the proceedings of the major AI conferences: the biennial International Joint Conference on AI (ICAI), the annual European Conference on Al (ECALI), and the National Conference on Al, more often known as AAAI, after its sponsoring organization. The major journals for general Al are Artificial Intelligence, Computational Intelligence, the IEEE Transactions on Pattern Analysis and Machine Intelligence, IEEE Intelligent Systems, and the electronic Journal of Artificial Intelligence Research. There are also many conferences and journals devoted to specific areas, which we cover in the appropriate chapters. The main professional societies for Al are the American Association for Artificial Intelligence (AAAI), the ACM Special Interest Group in Artificial Intelligence (SIGART), and the Society for Artificial Intelligence and Simulation of Behaviour (AISB). AAAI’s Al Magazine contains many topical and tutorial articles, and its website, aaai.org, contains news and background information.

EXERCISES

These exercises are intended to stimulate discussion, and some might be set as term projects. Alternatively, preliminary attempts can be made now, and these attempts can be reviewed after the completion of the book.

1.1 Define in your own words: (a) intelligence, (b) artificial intelligence, (c) agent.

1.2 Read Turing’s original paper on Al (Turing, 1950). In the paper, he discusses several potential objections to his proposed enterprise and his test for intelligence. Which objections still carry some weight? Are his refutations valid? Can you think of new objections arising from developments since he wrote the paper? In the paper, he predicts that, by the year 2000, a computer will have a 30% chance of passing a five-minute Turing Test with an unskilled interrogator. What chance do you think a computer would have today? In another 50 years?

1.3 Every year the Loebner prize is awarded to the program that comes closest to passing a version of the Turing test. Research and report on the latest winner of the Loebner prize. What techniques does it use? How does it advance the state of the art in AI?

1.4 There are well-known classes of problems that are intractably difficult for computers, and other classes that are provably undecidable. Does this mean that Al is impossible?

1.5 Suppose we extend Evans’s ANALOGY program so that it can score 200 on a standard IQ test. Would we then have a program more intelligent than a human? Explain.

1.6 How could introspection—reporting on one’s inner thoughts—be inaccurate? Could I be wrong about what I’m thinking? Discuss.

Section 1.5. Summary 31

1.7 Examine the AI literature to discover whether the following tasks can currently be solved by computers:

a. Playing a decent game of table tennis (ping-pong).

. Driving in the center of Cairo.

Buying a week’s worth of groceries at the market. Buying a week’s worth of groceries on the web.

. Playing a decent game of bridge at a competitive level. Discovering and proving new mathematical theorems. . Writing an intentionally funny story.

FowRome Ba

Giving competent legal advice in a specialized area of law. i. Translating spoken English into spoken Swedish in real time. Jj. Performing a complex surgical operation.

For the currently infeasible tasks, try to find out what the difficulties are and predict when, if ever, they will be overcome.

1.8 Some authors have claimed that perception and motor skills are the most important part of intelligence, and that “higher level” capacities are necessarily parasitic—simple add-ons to these underlying facilities. Certainly, most of evolution and a large part of the brain have been devoted to perception and motor skills, whereas Al has found tasks such as game playing and logical inference to be easier, in many ways, than perceiving and acting in the real world. Do you think that Al’s traditional focus on higher-level cognitive abilities is misplaced?

1.9 Why would evolution tend to result in systems that act rationally? What goals are such systems designed to achieve?

1.10 Are reflex actions (such as moving your hand away from a hot stove) rational? Are they intelligent?

1.11 “Surely computers cannot be intelligent—they can do only what their programmers tell them.” Is the latter statement true, and does it imply the former?

1.12 “Surely animals cannot be intelligent—they can do only what their genes tell them.” Is the latter statement true, and does it imply the former?

1.13 “Surely animals, humans, and computers cannot be intelligent—they can do only what their constituent atoms are told to do by the laws of physics.” Is the latter statement true, and does it imply the former?

INTELLIGENT AGENTS

In which we discuss the nature of agents, perfect or otherwise, the diversity of environments, and the resulting menagerie of agent types.

Chapter 1 identified the concept of rational agents as central to our approach to artificial intelligence. In this chapter, we make this notion more concrete. We will see that the concept of rationality can be applied to a wide variety of agents operating in any imaginable environment. Our plan in this book is to use this concept to develop a small set of design principles for building successful agents—systems that can reasonably be called intelligent.

We will begin by examining agents, environments, and the coupling between them. The observation that some agents behave better than others leads naturally to the idea of a rational agent—one that behaves as well as possible. How well an agent can behave depends on the nature of the environment; some environments are more difficult than others. We give a crude categorization of environments and show how properties of an environment influence the design of suitable agents for that environment. We describe a number of basic “skeleton” agent designs, which will be fleshed out in the rest of the book.

ENVIRONMENT SENSOR ACTUATOR

PERCEPT PERCEPT SEQUENCE

An agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators. This simple idea is illustrated in Figure 2.1. A human agent has eyes, ears, and other organs for sensors and hands, legs, mouth, and other body parts for actuators. A robotic agent might have cameras and infrared range finders for sensors and various motors for actuators. A software agent receives keystrokes, file contents, and network packets as sensory inputs and acts on the environment by displaying on the screen, writing files, and sending network packets. We will make the general assumption that every agent can perceive its own actions (but not always the effects).

We use the term percept to refer to the agent’s perceptual inputs at any given instant. An agent's percept sequence is the complete history of everything the agent has ever perceived. In general, an agent’s choice of action at any given instant can depend on the entire percept sequence observed to date. If we can specify the agent’s choice of action for every possible

32

Section 2.1.

Agents and Environments 33

[]

AGENT FUNCTION

AGENT PROGRAM

Figure 2.1 Agents interact with environments through sensors and actuators.

percept sequence, then we have said more or less everything there is to say about the agent. Mathematically speaking, we say that an agent’s behavior is described by the agent function that maps any given percept sequence to an action.

We can imagine tabulating the agent function that describes any given agent; for most agents, this would be a very large table—infinite, in fact, unless we place a bound on the length of percept sequences we want to consider. Given an agent to experiment with, we can, in principle, construct this table by trying out all possible percept sequences and recording which actions the agent does in response.! The table is, of course, an external characterization of the agent. Internally, the agent function for an artificial agent will be implemented by an agent program. It is important to keep these two ideas distinct. The agent function is an abstract mathematical description; the agent program is a concrete implementation, running on the agent architecture.

To illustrate these ideas, we will use a very simple example—the vacuum-cleaner world shown in Figure 2.2. This world is so simple that we can describe everything that happens; it’s also a made-up world, so we can invent many variations. This particular world has just two locations: squares A and B. The vacuum agent perceives which square it is in and whether there is dirt in the square. It can choose to move left, move right, suck up the dirt, or do nothing. One very simple agent function is the following: if the current square is dirty, then suck, otherwise move to the other square. A partial tabulation of this agent function is shown in Figure 2.3. A simple agent program for this agent function is given later in the chapter, in Figure 2.8.

Looking at Figure 2.3, we see that various vacuum-world agents can be defined simply by filling in the right-hand column in various ways. The obvious question, then, is this: What

1 If the agent uses some randomization to choose its actions, then we would have to try each sequence many times to identify the probability of each action. One might imagine that acting randomly is rather silly, but we’ll see later in this chapter that it can be very intelligent.

34

Chapter 2. Intelligent Agents

Figure 2.2 A vacuum-cleaner world with just two locations.

Percept sequence Action [A, Clean] Right [A, Dirty) Suck [B, Clean] Left [B, Dirty] Suck [4, Clean], [A, Clean] Right [4, Clean], [A, Dirty] Suck [4, Clean], [A, Clean], [A, Clean) Right [A, Clean], [A, Clean], [A, Dirty] Suck

Figure 2.3 Partial tabulation of a simple agent function for the vacuum-cleaner world shown in Figure 2.2.

\_

is the right way to fill out the table? In other words, what makes an agent good or bad, intelligent or stupid? We answer these questions in the next section.

Before closing this section, we will remark that the notion of an agent is meant to be a tool for analyzing systems, not an absolute characterization that divides the world into agents and non-agents. One could view a hand-held calculator as an agent that chooses the action of displaying “4” when given the percept sequence “2 + 2 =,” but such an analysis would hardly aid our understanding of the calculator,

RATIONAL AGENT

A rational agent is one that does the right thing—conceptually speaking, every entry in the table for the agent function is filled out correctly. Obviously, doing the right thing is better than doing the wrong thing, but what does it mean to do the right thing? As a first approximation, we will say that the right action is the one that will cause the agent to be

Section 2.2.

Good Behavior: The Concept of Rationality 35

PERFORMANCE MEASURE

most successful. Therefore, we will need some way to measure success. Together with the description of the environment and the sensors and actuators of the agent, this will provide a complete specification of the task facing the agent. Given this, we can define more precisely what it means to be rational.

Performance measures

A performance measure embodies the criterion for success of an agent’s behavior. When an agent is plunked down in an environment, it generates a sequence of actions according to the percepts it receives. This sequence of actions causes the environment to go through a sequence of states. If the sequence is desirable, then the agent has performed well. Obviously, there is not one fixed measure suitable for all agents. We could ask the agent for a subjective opinion of how happy it is with its own performance, but some agents would be unable to answer, and others would delude themselves.” Therefore, we will insist on an objective performance measure, typically one imposed by the designer who is constructing the agent.

Consider the vacuum-cleaner agent from the preceding section. We might propose to measure performance by the amount of dirt cleaned up in a single eight-hour shift. With a rational agent, of course, what you ask for is what you get. A rational agent can maximize this performance measure by cleaning up the dirt, then dumping it all on the floor, then cleaning it up again, and so on. A more suitable performance measure would reward the agent for having a clean floor. For example, one point could be awarded for each clean square at each time step (perhaps with a penalty for electricity consumed and noise generated). As a general rule, it is better to design performance measures according to what one actually wants in the environment, rather than according to how one thinks the agent should behave.

The selection of a performance measure is not always easy. For example, the notion of “clean floor” in the preceding paragraph is based on average cleanliness over time. Yet the same average cleanliness can be achieved by two different agents, one of which does a mediocre job all the time while the other cleans energetically but takes long breaks. Which is preferable might seem to be a fine point of janitorial science, but in fact it is a deep philosophical question with far-reaching implications. Which is better—a reckless life of highs and lows, or a safe but humdrum existence? Which is better—an economy where everyone lives in moderate poverty, or one in which some live in plenty while others are very poor? We will leave these questions as an exercise for the diligent reader.

Rationality

What is rational at any given time depends on four things:

o The performance measure that defines the criterion of success. e The agent’s prior knowledge of the environment.

e The actions that the agent can perform.

o The agent’s percept sequence to date.

2 Human agents in particular are notorious for “sour grapes”—believing they did not really want something after not getting it, as in, “Oh well, never mind, I didn’t want that stupid Nobel prize anyway.”

36

Chapter 2. Intelligent Agents

DEFINITION OF A RATIONAL AGENT

[e3 fy

OMNISCIENCE

This leads to a definition of a rational agent:

For each possible percept sequence, a rational agent should select an action that is expected to maximize its performance measure, given the evidence provided by the percept sequence and whatever built-in knowledge the agent has.

Consider the simple vacuum-cleaner agent that cleans a square if it is dirty and moves to the other square if not; this is the agent function tabulated in Figure 2.3. Is this a rational agent? That depends! First, we need to say what the performance measure is, what is known about the environment, and what sensors and actuators the agent has. Let us assume the following:

o The performance measure awards one point for each clean square at each time step, over a “lifetime” of 1000 time steps.

o The “geography” of the environment is known a priori (Figure 2.2) but the dirt distribution and the initial location of the agent are not. Clean squares stay clean and sucking cleans the current square. The Left and Right actions move the agent left and right except when this would take the agent outside the environment, in which case the agent remains where it is.

e The only available actions are Left, Right, Suck, and NoOp (do nothing). o The agent correctly perceives its location and whether that location contains dirt.

We claim that under these circumstances the agent is indeed rational; its expected performance is at least as high as any other agent’s. Exercise 2.4 asks you to prove this.

One can see easily that the same agent would be irrational under different circumstances. For example, once all the dirt is cleaned up it will oscillate needlessly back and forth; if the performance measure includes a penalty of one point for each movement left or right, the agent will fare poorly. A better agent for this case would do nothing once it is sure that all the squares are clean. If clean squares can become dirty again, the agent should occasionally check and re-clean them if needed. If the geography of the environment is unknown, the agent will need to explore it rather than stick to squares A and B. Exercise 2.4 asks you to design agents for these cases.

Omniscience, learning, and autonomy

We need to be careful to distinguish between rationality and omniscience. An omniscient agent knows the actual outcome of its actions and can act accordingly; but omniscience is impossible in reality. Consider the following example: I am walking along the Champs Elysées one day and I see an old friend across the street. There is no traffic nearby and I'm not otherwise engaged, so, being rational, [ start to cross the street. Meanwhile, at 33,000 feet, a cargo door falls off a passing airliner,® and before I make it to the other side of the street I am flattened. Was [irrational to cross the street? It is unlikely that my obituary would read “Idiot attempts to cross street.”

This example shows that rationality is not the same as perfection. Rationality maximizes expected performance, while perfection maximizes actual performance. Retreating from a requirement of perfection is not just a question of being fair to agents. The point is

3 See N. Henderson, “New door latches urged for Boeing 747 jumbo jets,” Washington Post, August 24, 1989.

Section 2.2.

Good Behavior: The Concept of Rationality 37

INFORMATION GATHERING

EXPLORATION

LEARNING

AUTONOMY

that if we expect an agent to do what turns out to be the best action after the fact, it will be impossible to design an agent to fulfill this specification—unless we improve the performance of crystal balls or time machines.

Our definition of rationality does not require omniscience, then, because the rational choice depends only on the percept sequence fo date. We must also ensure that we haven't inadvertently allowed the agent to engage in decidedly underintelligent activities. For example, if an agent does not look both ways before crossing a busy road, then its percept sequence will not tell it that there is a large truck approaching at high speed. Does our definition of rationality say that it’s now OK to cross the road? Far from it! First, it would not be rational to cross the road given this uninformative percept sequence: the risk of accident from crossing without looking is too great. Second, a rational agent should choose the “looking” action before stepping into the street, because looking helps maximize the expected performance. Doing actions in order to modify future percepts—sometimes called information gathering—is an important part of rationality and is covered in depth in Chapter 16. A second example of information gathering is provided by the exploration that must be undertaken by a vacuum-cleaning agent in an initially unknown environment.

Our definition requires a rational agent not only to gather information, but also to learn as much as possible from what it perceives. The agent’s initial configuration could reflect some prior knowledge of the environment, but as the agent gains experience this may be modified and augmented. There are extreme cases in which the environment is completely known a priori. In such cases, the agent need not perceive or learn; it simply acts correctly. Of course, such agents are very fragile. Consider the lowly dung beetle. After digging its nest and laying its eggs, it fetches a ball of dung from a nearby heap to plug the entrance. If the ball of dung is removed from its grasp en route, the beetle continues on and pantomimes plugging the nest with the nonexistent dung ball, never noticing that it is missing. Evolution has built an assumption into the beetle’s behavior, and when it is violated, unsuccessful behavior results. Slightly more intelligent is the sphex wasp. The female sphex will dig a burrow, go out and sting a caterpillar and drag it to the burrow, enter the burrow again to check all is well, drag the caterpillar inside, and lay its eggs. The caterpillar serves as a food source when the eggs hatch. So far so good, but if an entomologist moves the caterpillar a few inches away while the sphex is doing the check, it will revert back to the “drag” step of its plan, and will continue the plan without modification, even after dozens of caterpillar-moving interventions. The sphex is unable to learn that its innate plan is failing, and thus will not change it.

Successful agents split the task of computing the agent function into three different periods: when the agent is being designed, some of the computation is done by its designers; when it is deliberating on its next action, the agent does more computation; and as it learns from experience, it does even more computation to decide how to modify its behavior.

To the extent that an agent relies on the prior knowledge of its designer rather than on its own percepts, we say that the agent lacks autonomy. A rational agent should be autonomous—it should learn what it can to compensate for partial or incorrect prior knowledge. For example, a vacuum-cleaning agent that learns to foresee where and when additional dirt will appear will do better than one that does not. As a practical matter, one seldom requires complete autonomy from the start: when the agent has had little or no experience, it

38

Chapter 2. Intelligent Agents

would have to act randomly unless the designer gave some assistance. So, just as evolution provides animals with enough built-in reflexes so that they can survive long enough to learn for themselves, it would be reasonable to provide an artificial intelligent agent with some initial knowledge as well as an ability to learn. After sufficient experience of its environment, the behavior of a rational agent can become effectively independent of its prior knowledge. Hence, the incorporation of learning allows one to design a single rational agent that will succeed in a vast variety of environments.

TASK ENVIRONMENTS

PEAS

Now that we have a definition of rationality, we are almost ready to think about building rational agents. First, however, we must think about task environments, which are essentially the “problems” to which rational agents are the “solutions.” We begin by showing how to specify a task environment, illustrating the process with a number of examples. We then show that task environments come in a variety of flavors. The flavor of the task environment directly affects the appropriate design for the agent program.

Specifying the task environment

In our discussion of the rationality of the simple vacuum-cleaner agent, we had to specify the performance measure, the environment, and the agent’s actuators and sensors. We will group all these together under the heading of the task environment. For the acronymically minded, we call this the PEAS (Performance, Environment, Actuators, Sensors) description. In designing an agent, the first step must always be to specify the task environment as fully as possible.

The vacuum world was a simple example; let us consider a more complex problem: an automated taxi driver. We will use this example throughout the rest of the chapter. We should point out, before the reader becomes alarmed, that a fully automated taxi is currently somewhat beyond the capabilities of existing technology. (See page 27 for a description of an existing driving robot, or look at recent proceedings of the conferences on Intelligent Transportation Systems.) The full driving task is extremely open-ended. There is no limit to the novel combinations of circumstances that can arise—another reason we chose it as a focus for discussion. Figure 2.4 summarizes the PEAS description for the taxi’s task environment. We discuss each element in more detail in the following paragraphs.

First, what is the performance measure to which we would like our automated driver to aspire? Desirable qualities include getting to the correct destination; minimizing fuel consumption and wear and tear; minimizing the trip time and/or cost; minimizing violations of traffic laws and disturbances to other drivers; maximizing safety and passenger comfort; maximizing profits. Obviously, some of these goals conflict, so there will be tradeoffs involved.

Next, what is the driving environment that the taxi will face? Any taxi driver must deal with a variety of roads, ranging from rural lanes and urban alleys to 12-lane freeways. The roads contain other traffic, pedestrians, stray animals, road works, police cars, puddles,

Section 2.3.

39

[]

and potholes. The taxi must also interact with potential and actual passengers. There are also some optional choices. The taxi might need to operate in Southern California, where snow is seldom a problem, or in Alaska, where it seldom is not. It could always be driving on the right, or we might want it to be flexible enough to drive on the left when in Britain or Japan. Obviously, the more restricted the environment, the easier the design problem.

The actuators available to an automated taxi will be more or less the same as those available to a human driver: control over the engine through the accelerator and control over steering and braking. In addition, it will need output to a display screen or voice synthesizer to talk back to the passengers, and perhaps some way to communicate with other vehicles, politely or otherwise.

To achieve its goals in the driving environment, the taxi will need to know where it is, what else is on the road, and how fast it is going. Its basic sensors should therefore include one or more controllable TV cameras, the speedometer, and the odometer. To control the vehicle properly, especially on curves, it should have an accelerometer; it will also need to know the mechanical state of the vehicle, so it will need the usual array of engine and electrical system sensors. It might have instruments that are not available to the average human driver: a satellite global positioning system (GPS) to give it accurate position information with respect to an electronic map, and infrared or sonar sensors to detect distances to other cars and obstacles. Finally, it will need a keyboard or microphone for the passenger to request a destination.

In Figure 2.5, we have sketched the basic PEAS elements for a number of additional agent types. Further examples appear in Exercise 2.5. It may come as a surprise to some readers that we include in our list of agent types some programs that operate in the entirely artificial environment defined by keyboard input and character output on a screen. “Surely,” one might say, “this is not a real environment, is it?” In fact, what matters is not the distinction between “real” and “artificial” environments, but the complexity of the relationship among the behavior of the agent, the percept sequence generated by the environment, and the performance measure. Some “real” environments are actually quite simple. For example, a robot designed to inspect parts as they come by on a conveyor belt can make use of a number of simplifying assumptions: that the lighting is always just so, that the only thing on the conveyer belt will be parts of a kind that it knows about, and that there are only two actions (accept or reject).

[]

40 Chapter 2. Intelligent Agents Agent Type Performance Environment Actuators Sensors Measure Medical Healthy patient, Patient, hospital, Display Keyboard entry diagnosis system minimize costs, staff questions, tests, of symptoms, lawsuits diagnoses, findings, patient’s treatments, answers referrals Satellite image Correct image Downlink from Display Color pixel analysis system categorization orbiting satellite categorization of arrays scene Part-picking Percentage of Conveyor belt Jointed arm and Camera, joint robot parts in correct with parts; bins hand angle sensors bins Refinery Maximize purity, Refinery, Valves, pumps, Temperature, controller yield, safety operators heaters, displays pressure, chemical sensors Interactive Maximize Set of students, Display Keyboard entry English tutor student’s score testing agency exercises, on test suggestions, corrections Figure 2.5 Examples of agent types and their PEAS descriptions. SOFTWARE AGENTS In contrast, some software agents (or software robots or softbots) exist in rich, unSOFTBOTS limited domains. Imagine a softbot designed to fly a flight simulator for a large commercial

airplane. The simulator is a very detailed, complex environment including other aircraft and ground operations, and the software agent must choose from a wide variety of actions in real time. Or imagine a softbot designed to scan Internet news sources and show the interesting items to its customers. To do well, it will need some natural language processing abilities, it will need to learn what each customer is interested in, and it will need to change its plans dynamically—for example, when the connection for one news source goes down or when a new one comes online. The Internet is an environment whose complexity rivals that of the physical world and whose inhabitants include many artificial agents.

Properties of task environments

The range of task environments that might arise in Al is obviously vast. We can, however, identify a fairly small number of dimensions along which task environments can be categorized. These dimensions determine, to a large extent, the appropriate agent design and the

The Nature of Environments 41

FULLY OBSERVABLE

DETERMINISTIC STOCHASTIC

STRATEGIC

EPISODIC SEQUENTIAL

applicability of each of the principal families of techniques for agent implementation. First, we list the dimensions, then we analyze several task environments to illustrate the ideas. The definitions here are informal; later chapters provide more precise statements and examples of each kind of environment.

{> Fully observable vs. partially observable.

If an agent’s sensors give it access to the complete state of the environment at each point in time, then we say that the task environment is fully observable.\* A task environment is effectively fully observable if the sensors detect all aspects that are relevant to the choice of action; relevance, in turn, depends on the performance measure. Fully observable environments are convenient because the agent need not maintain any internal state to keep track of the world. An environment might be partially observable because of noisy and inaccurate sensors or because parts of the state are simply missing from the sensor data—for example, a vacuum agent with only a local dirt sensor cannot tell whether there is dirt in other squares, and an automated taxi cannot see what other drivers are thinking.

{> Deterministic vs. stochastic.

If the next state of the environment is completely determined by the current state and the action executed by the agent, then we say the environment is deterministic; otherwise, it is stochastic. In principle, an agent need not worry about uncertainty in a fully observable, deterministic environment. If the environment is partially observable, however, then it could appear to be stochastic. This is particularly true if the environment is complex, making it hard to keep track of all the unobserved aspects. Thus, it is often better to think of an environment as deterministic or stochastic from the point of view of the agent. Taxi driving is clearly stochastic in this sense, because one can never predict the behavior of traffic exactly; moreover, one’s tires blow out and one’s engine seizes up without warning. The vacuum world as we described it is deterministic, but variations can include stochastic elements such as randomly appearing dirt and an unreliable suction mechanism (Exercise 2.12). If the environment is deterministic except for the actions of other agents, we say that the environment is strategic.

¢ Episodic vs. sequential > In an episodic task environment, the agent’s experience is divided into atomic episodes. Each episode consists of the agent perceiving and then performing a single action. Crucially, the next episode does not depend on the actions taken in previous episodes. In episodic environments, the choice of action in each episode depends only on the episode itself. Many classification tasks are episodic. For example, an agent that has to spot defective parts on an assembly line bases each decision on the current part, regardless of previous decisions; moreover, the current decision doesn’t affect whether the next

4 The first edition of this book used the terms accessible and inaccessible instead of fully and partially observable; nondeterministic instead of stochastic; and nonepisodic instead of sequential. The new terminology is more consistent with established usage.

5 The word “sequential” is also used in computer science as the antonym of “parallel.” The two meanings are largely unrelated.

42

Chapter 2. Intelligent Agents

STATIC DYNAMIC

SEMIDYNAMIC

DISCRETE CONTINUOUS

SINGLE AGENT MULTIAGENT

COMPETITIVE

COOPERATIVE

part is defective. In sequential environments, on the other hand, the current decision could affect all future decisions. Chess and taxi driving are sequential: in both cases, short-term actions can have long-term consequences. Episodic environments are much simpler than sequential environments because the agent does not need to think ahead.

Static vs. dynamic.

If the environment can change while an agent is deliberating, then we say the environment is dynamic for that agent; otherwise, it is static. Static environments are easy to deal with because the agent need not keep looking at the world while it is deciding on an action, nor need it worry about the passage of time. Dynamic environments, on the other hand, are continuously asking the agent what it wants to do; if it hasn’t decided yet, that counts as deciding to do nothing. If the environment itself does not change with the passage of time but the agent’s performance score does, then we say the environment is semidynamic. Taxi driving is clearly dynamic: the other cars and the taxi itself keep moving while the driving algorithm dithers about what to do next. Chess, when played with a clock, is semidynamic. Crossword puzzles are static.

The discrete/continuous distinction can be applied to the stare of the environment, to the way time is handled, and to the percepts and actions of the agent. For example, a discrete-state environment such as a chess game has a finite number of distinct states. Chess also has a discrete set of percepts and actions. Taxi driving is a continuousstate and continuous-time problem: the speed and location of the taxi and of the other vehicles sweep through a range of continuous values and do so smoothly over time. Taxi-driving actions are also continuous (steering angles, etc.). Input from digital cameras is discrete, strictly speaking, but is typically treated as representing continuously varying intensities and locations.

Single agent vs. multiagent.

The distinction between single-agent and multiagent environments may seem simple enough. For example, an agent solving a crossword puzzle by itself is clearly in a single-agent environment, whereas an agent playing chess is in a two-agent environment. There are, however, some subtle issues. First, we have described how an entity may be viewed as an agent, but we have not explained which entities must be viewed as agents. Does an agent A (the taxi driver for example) have to treat an object B (another vehicle) as an agent, or can it be treated merely as a stochastically behaving object, analogous to waves at the beach or leaves blowing in the wind? The key distinction is whether B’s behavior is best described as maximizing a performance measure whose value depends on agent A’s behavior. For example, in chess, the opponent entity B is trying to maximize its performance measure, which, by the rules of chess, minimizes agent A’s performance measure. Thus, chess is a competitive multiagent environment. In the taxi-driving environment, on the other hand, avoiding collisions maximizes the performance measure of all agents, so it is a partially cooperative multiagent environment. It is also partially competitive because, for example, only one car can occupy a parking space. The agent-design problems arising in multiagent environments are often

Section 2.3.

The Nature of Environments 43

[]

quite different from those in single-agent environments; for example, communication often emerges as a rational behavior in multiagent environments; in some partially observable competitive environments, stochastic behavior is rational because it avoids the pitfalls of predictability.

As one might expect, the hardest case is partially observable, stochastic, sequential, dynamic, continuous, and multiagent. It also turns out that most real situations are so complex that whether they are really deterministic is a moot point. For practical purposes, they must be treated as stochastic. Taxi driving is hard in all these senses.

Figure 2.6 lists the properties of a number of familiar environments. Note that the answers are not always cut and dried. For example, we have listed chess as fully observable; strictly speaking, this is false because certain rules about castling, en passant capture, and draws by repetition require remembering some facts about the game history that are not observable as part of the board state. These exceptions to observability are of course minor compared to those faced by the taxi driver, the English tutor, or the medical diagnosis system.

Some other answers in the table depend on how the task environment is defined. We have listed the medical-diagnosis task as single-agent because the disease process in a patient is not profitably modeled as an agent; but a medical-diagnosis system might also have to deal with recalcitrant patients and skeptical staff, so the environment could have a multiagent aspect. Furthermore, medical diagnosis is episodic if one conceives of the task as selecting a diagnosis given a list of symptoms; the problem is sequential if the task can include proposing a series of tests, evaluating progress over the course of treatment, and so on. Also, many environments are episodic at higher levels than the agent’s individual actions. For example, a chess tournament consists of a sequence of games; each game is an episode, because (by and large) the contribution of the moves in one game to the agent’s overall performance is not affected by the moves in its previous game. On the other hand, decision making within a single game is certainly sequential.

44

Chapter 2. Intelligent Agents

ENVIRONMENT CLASS

ENVIRONMENT GENERATOR

The code repository associated with this book (aima.cs.berkeley.edu) includes implementations of a number of environments, together with a general-purpose environment simulator that places one or more agents in a simulated environment, observes their behavior over time, and evaluates them according to a given performance measure. Such experiments are often carried out not for a single environment, but for many environments drawn from an environment class. For example, to evaluate a taxi driver in simulated traffic, we would want to run many simulations with different traffic, lighting, and weather conditions. If we designed the agent for a single scenario, we might be able to take advantage of specific properties of the particular case but might not identify a good design for driving in general. For this reason, the code repository also includes an environment generator for each environment class that selects particular environments (with certain likelihoods) in which to run the agent. For example, the vacuum environment generator initializes the dirt pattern and agent location randomly. We are then interested in the agent’s average performance over the environment class. A rational agent for a given environment class maximizes this average performance. Exercises 2.7 to 2.12 take you through the process of developing an environment class and evaluating various agents therein.

AGENT PROGRAM

ARCHITECTURE

So far we have talked about agents by describing behavior—the action that is performed after any given sequence of percepts. Now, we will have to bite the bullet and talk about how the insides work. The job of Al is to design the agent program that implements the agent function mapping percepts to actions. We assume this program will run on some sort of computing device with physical sensors and actuators—we call this the architecture:

agent = architecture + program .

Obviously, the program we choose has to be one that is appropriate for the architecture. If the program is going to recommend actions like Walk, the architecture had better have legs. The architecture might be just an ordinary PC, or it might be a robotic car with several onboard computers, cameras, and other sensors. In general, the architecture makes the percepts from the sensors available to the program, runs the program, and feeds the program’s action choices to the actuators as they are generated. Most of this book is about designing agent programs, although Chapters 24 and 25 deal directly with the sensors and actuators.

Agent programs

The agent programs that we will design in this book all have the same skeleton: they take the current percept as input from the sensors and return an action to the actuators.’ Notice the difference between the agent program, which takes the current percept as input, and the agent function, which takes the entire percept history. The agent program takes just the current 6 There are other choices for the agent program skeleton; for example, we could have the agent programs be

coroutines that run asynchronously with the environment. Each such coroutine has an input and output port and consists of a loop that reads the input port for percepts and writes actions to the output port.

Section 2.4.

The Structure of Agents 45

function TABLE-DRIVEN-AGENT( percept) returns an action static: percepts, a sequence, initially empty table, a table of actions, indexed by percept sequences, initially fully specified

append percept to the end of percepts action «— LOOKUP( percepts, table) return action

Figure 2.7 The TABLE-DRIVEN-AGENT program is invoked for each new percept and returns an action each time. It keeps track of the percept sequence using its own private data structure.

percept as input because nothing more is available from the environment; if the agent’s actions depend on the entire percept sequence, the agent will have to remember the percepts.

We will describe the agent programs via the simple pseudocode language that is defined in Appendix B. (The online code repository contains implementations in real programming languages.) For example, Figure 2.7 shows a rather trivial agent program that keeps track of the percept sequence and then uses it to index into a table of actions to decide what to do. The table represents explicitly the agent function that the agent program embodies. To build a rational agent in this way, we as designers must construct a table that contains the appropriate action for every possible percept sequence.

It is instructive to consider why the table-driven approach to agent construction is doomed to failure. Let P be the set of possible percepts and let 1" be the lifetime of the agent (the total number of percepts it will receive). The lookup table will contain Y7\_, | P|! entries. Consider the automated taxi: the visual input from a single camera comes in at the rate of roughly 27 megabytes per second (30 frames per second, 640 x 480 pixels with 24 bits of color information). This gives a lookup table with over 1025%:000,000,000 entries for an hour’s driving. Even the lookup table for chess—a tiny, well-behaved fragment of the real world—would have at least 10! entries. The daunting size of these tables (the number of atoms in the observable universe is less than 108%) means that (a) no physical agent in this universe will have the space to store the table, (b) the designer would not have time to create the table, (c) no agent could ever learn all the right table entries from its experience, and (d) even if the environment is simple enough to yield a feasible table size, the designer still has no guidance about how to fill in the table entries.

Despite all this, TABLE-DRIVENAGENT does do what we want: it implements the desired agent function. The key challenge for Al is to find out how to write programs that, to the extent possible, produce rational behavior from a small amount of code rather than from a large number of table entries. We have many examples showing that this can be done successfully in other areas: for example, the huge tables of square roots used by engineers and schoolchildren prior to the 1970s have now been replaced by a five-line program for Newton’s method running on electronic calculators. The question is, can Al do for general intelligent behavior what Newton did for square roots? We believe the answer is yes.

46

Chapter 2. Intelligent Agents

SIMPLE REFLEX AGENT

CONDITION-ACTION RULE

function REFLEX-VACUUM-AGENT([location,status]) returns an action

if status = Dirty then return Suck else if location = A then return Right else if location = B then return Left

Figure 2.8 The agent program for a simple reflex agent in the two-state vacuum environment. This program implements the agent function tabulated in Figure 2.3.

In the remainder of this section, we outline four basic kinds of agent program that embody the principles underlying almost all intelligent systems:

e Simple reflex agents;

eo Model-based reflex agents; ® Goal-based agents; and

o Utility-based agents.

We then explain in general terms how to convert all these into learning agents.

Simple reflex agents

The simplest kind of agent is the simple reflex agent. These agents select actions on the basis of the current percept, ignoring the rest of the percept history. For example, the vacuum agent whose agent function is tabulated in Figure 2.3 is a simple reflex agent, because its decision is based only on the current location and on whether that contains dirt. An agent program for this agent is shown in Figure 2.8.

Notice that the vacuum agent program is very small indeed compared to the corresponding table. The most obvious reduction comes from ignoring the percept history, which cuts down the number of possibilities from 47 to just 4. A further, small reduction comes from the fact that, when the current square is dirty, the action does not depend on the location.

Imagine yourself as the driver of the automated taxi. If the car in front brakes, and its brake lights come on, then you should notice this and initiate braking. In other words, some processing is done on the visual input to establish the condition we call “The car in front is braking.” Then, this triggers some established connection in the agent program to the action “initiate braking.” We call such a connection a condition-action rule,” written as

if car-in-front-is-braking then initiate-braking.

Humans also have many such connections, some of which are learned responses (as for driving) and some of which are innate reflexes (such as blinking when something approaches the eye). In the course of the book, we will see several different ways in which such connections can be learned and implemented.

The program in Figure 2.8 is specific to one particular vacuum environment. A more general and flexible approach is first to build a general-purpose interpreter for condition

Section 2.4.

The Structure of Agents 47

[]

Figure 2.9 Schematic diagram of a simple reflex agent.

function SIMPLE-REFLEX-AGENT( percept) returns an action static: rules, a set of condition—action rules

state «— INTERPRET-INPUT( percept) rule «— RULE-MATCH(state, rules) action «— RULE-ACTION][ rule] return action

Figure 2.10 A simple reflex agent. It acts according to a rule whose condition matches the current state, as defined by the percept.

action rules and then to create rule sets for specific task environments. Figure 2.9 gives the structure of this general program in schematic form, showing how the condition—action rules allow the agent to make the connection from percept to action. (Do not worry if this seems trivial; it gets more interesting shortly.) We use rectangles to denote the current internal state of the agent’s decision process and ovals to represent the background information used in the process. The agent program, which is also very simple, is shown in Figure 2.10. The INTERPRET-INPUT function generates an abstracted description of the current state from the percept, and the RULE-MATCH function returns the first rule in the set of rules that matches the given state description. Note that the description in terms of “rules” and “matching” is purely conceptual; actual implementations can be as simple as a collection of logic gates implementing a Boolean circuit.

Simple reflex agents have the admirable property of being simple, but they turn out to be of very limited intelligence. The agent in Figure 2.10 will work only if the correct decision can be made on the basis of only the current percept—that is, only if the environment is Sully observable. Even a little bit of unobservability can cause serious trouble. For example,

48

Chapter 2. Intelligent Agents

RANDOMIZATION

INTERNAL STATE

MODEL-BASED AGENT

the braking rule given earlier assumes that the condition car-in-front-is-braking can be determined from the current percept—the current video image—if the car in front has a centrally mounted brake light. Unfortunately, older models have different configurations of taillights, brake lights, and turn-signal lights, and it is not always possible to tell from a single image whether the car is braking. A simple reflex agent driving behind such a car would either brake continuously and unnecessarily, or, worse, never brake at all.

We can see a similar problem arising in the vacuum world. Suppose that a simple reflex vacuum agent is deprived of its location sensor, and has only a dirt sensor. Such an agent has just two possible percepts: [Dirty] and [Clean]. It can Suck in response to [Dirty]; what should it do in response to [Clean]? Moving Left fails (for ever) if it happens to start in square A, and moving Right fails (for ever) if it happens to start in square B. Infinite loops are often unavoidable for simple reflex agents operating in partially observable environments.

Escape from infinite loops is possible if the agent can randomize its actions. For example, if the vacuum agent perceives [Clean], it might flip a coin to choose between Left and Right. It is easy to show that the agent will reach the other square in an average of two steps. Then, if that square is dirty, it will clean it and the cleaning task will be complete. Hence, a randomized simple reflex agent might outperform a deterministic simple reflex agent.

We mentioned in Section 2.3 that randomized behavior of the right kind can be rational in some multiagent environments. In single-agent environments, randomization is usually not rational. It is a useful trick that helps a simple reflex agent in some situations, but in most cases we can do much better with more sophisticated deterministic agents.

Model-based reflex agents

The most effective way to handle partial observability is for the agent to keep rrack of the part of the world it can’t see now. That is, the agent should maintain some sort of internal state that depends on the percept history and thereby reflects at least some of the unobserved aspects of the current state. For the braking problem, the internal state is not too extensive— just the previous frame from the camera, allowing the agent to detect when two red lights at the edge of the vehicle go on or off simultaneously. For other driving tasks such as changing lanes, the agent needs to keep track of where the other cars are if it can’t see them all at once.

Updating this internal state information as time goes by requires two kinds of knowledge to be encoded in the agent program. First, we need some information about how the world evolves independently of the agent—for example, that an overtaking car generally will be closer behind than it was a moment ago. Second, we need some information about how the agent’s own actions affect the world—for example, that when the agent turns the steering wheel clockwise, the car turns to the right or that after driving for five minutes northbound on the freeway one is usually about five miles north of where one was five minutes ago. This knowledge about “how the world works”—whether implemented in simple Boolean circuits or in complete scientific theories—is called a model of the world. An agent that uses such a model is called a model-based agent.

Figure 2.11 gives the structure of the reflex agent with internal state, showing how the current percept is combined with the old internal state to generate the updated description

Section 2.4.

The Structure of Agents 49

GOAL

| \ i What the world y = . is like now Ie

3

Figure 2.11 A model-based reflex agent.

function REFLEX-AGENT-WITH-STATE( percept) returns an action static: state, a description of the current world state rules, a set of condition—action rules action, the most recent action, initially none

state «— UPDATE-STATE(stale, action, percept) rule «— RULE-MATCH(state, rules)

action «+— RULE-ACTION[rule]

return action

Figure 2.12 A model-based reflex agent. It keeps track of the current state of the world using an internal model. It then chooses an action in the same way as the reflex agent.

of the current state. The agent program is shown in Figure 2.12. The interesting part is the function UPDATESTATE, which is responsible for creating the new internal state description. As well as interpreting the new percept in the light of existing knowledge about the state, it uses information about how the world evolves to keep track of the unseen parts of the world, and also must know about what the agent’s actions do to the state of the world. Detailed examples appear in Chapters 10 and 17.

Goal-based agents

Knowing about the current state of the environment is not always enough to decide what to do. For example, at a road junction, the taxi can turn left, turn right, or go straight on. The correct decision depends on where the taxi is trying to get to. In other words, as well as a current state description, the agent needs some sort of goal information that describes situations that are desirable—for example, being at the passenger’s destination. The agent

50

Chapter 2. Intelligent Agents

[]

Figure 2.13 A model-based, goal-based agent. It keeps track of the world state as well as a set of goals it is trying to achieve, and chooses an action that will (eventually) lead to the achievement of its goals.

program can combine this with information about the results of possible actions (the same information as was used to update internal state in the reflex agent) in order to choose actions that achieve the goal. Figure 2.13 shows the goal-based agent’s structure.

Sometimes goal-based action selection is straightforward, when goal satisfaction results immediately from a single action. Sometimes it will be more tricky, when the agent has to consider long sequences of twists and turns to find a way to achieve the goal. Search (Chapters 3 to 6) and planning (Chapters 11 and 12) are the subfields of AI devoted to finding action sequences that achieve the agent’s goals.

Notice that decision making of this kind is fundamentally different from the condition— action rules described earlier, in that it involves consideration of the future—both “What will happen if I do such-and-such?” and “Will that make me happy?” In the reflex agent designs, this information is not explicitly represented, because the built-in rules map directly from percepts to actions. The reflex agent brakes when it sees brake lights. A goal-based agent, in principle, could reason that if the car in front has its brake lights on, it will slow down. Given the way the world usually evolves, the only action that will achieve the goal of not hitting other cars is to brake.

Although the goal-based agent appears less efficient, it is more flexible because the knowledge that supports its decisions is represented explicitly and can be modified. If it starts to rain, the agent can update its knowledge of how effectively its brakes will operate; this will automatically cause all of the relevant behaviors to be altered to suit the new conditions. For the reflex agent, on the other hand, we would have to rewrite many condition—action rules. The goal-based agent’s behavior can easily be changed to go to a different location. The reflex agent’s rules for when to turn and when to go straight will work only for a single destination; they must all be replaced to go somewhere new.

Section 2.4.

The Structure of Agents 51

UTILITY UTILITY FUNCTION

Utility-based agents

Goals alone are not really enough to generate high-quality behavior in most environments. For example, there are many action sequences that will get the taxi to its destination (thereby achieving the goal) but some are quicker, safer, more reliable, or cheaper than others. Goals just provide a crude binary distinction between “happy” and “unhappy” states, whereas a more general performance measure should allow a comparison of different world states according to exactly how happy they would make the agent if they could be achieved. Because “happy” does not sound very scientific, the customary terminology is to say that if one world state is preferred to another, then it has higher utility for the agent.®

A utility function maps a state (or a sequence of states) onto a real number, which describes the associated degree of happiness. A complete specification of the utility function allows rational decisions in two kinds of cases where goals are inadequate. First, when there are conflicting goals, only some of which can be achieved (for example, speed and safety), the utility function specifies the appropriate tradeoff. Second, when there are several goals that the agent can aim for, none of which can be achieved with certainty, utility provides a way in which the likelihood of success can be weighed up against the importance of the goals.

In Chapter 16, we will show that any rational agent must behave as if it possesses a utility function whose expected value it tries to maximize. An agent that possesses an explicit utility function therefore can make rational decisions, and it can do so via a general-purpose algorithm that does not depend on the specific utility function being maximized. In this way, the “global” definition of rationality—designating as rational those agent functions that have the highest performance—is turned into a “local” constraint on rational-agent designs that can be expressed in a simple program.

The utility-based agent structure appears in Figure 2.14. Utility-based agent programs appear in Part V, where we design decision making agents that must handle the uncertainty inherent in partially observable environments.

Learning agents

We have described agent programs with various methods for selecting actions. We have not, so far, explained how the agent programs come into being. In his famous early paper, Turing (1950) considers the idea of actually programming his intelligent machines by hand. He estimates how much work this might take and concludes “Some more expeditious method seems desirable.” The method he proposes is to build learning machines and then to teach them. In many areas of Al, this is now the preferred method for creating state-of-the-art systems. Learning has another advantage, as we noted earlier: it allows the agent to operate in initially unknown environments and to become more competent than its initial knowledge alone might allow. In this section, we briefly introduce the main ideas of learning agents. In almost every chapter of the book, we will comment on opportunities and methods for learning in particular kinds of agents. Part VI goes into much more depth on the various learning algorithms themselves.

8 The word “utility” here refers to “the quality of being useful,” not to the electric company or water works.

[]

52 Chapter 2. Intelligent Agents ( TT TT 2 ~~ Sensors = State KS { What the world How the world evolves is like now f 7 ‘What my actions do | pas i he =. = ¥ © (utility y— How h: I will be 5 os ow happy I wi in such a state o x What action I should do now Actuators

LEARNING ELEMENT

PERFORMANCE ELEMENT

CRITIC

PROBLEM GENERATOR

Figure 2.14 A model-based, utility-based agent. It uses a model of the world, along with a utility function that measures its preferences among states of the world. Then it chooses the action that leads to the best expected utility, where expected utility is computed by averaging over all possible outcome states, weighted by the probability of the outcome.

A learning agent can be divided into four conceptual components, as shown in Figure 2.15. The most important distinction is between the learning element, which is responsible for making improvements, and the performance element, which is responsible for selecting external actions. The performance element is what we have previously considered to be the entire agent: it takes in percepts and decides on actions. The learning element uses feedback from the critic on how the agent is doing and determines how the performance element should be modified to do better in the future.

The design of the learning element depends very much on the design of the performance element. When trying to design an agent that learns a certain capability, the first question is not “How am 1 going to get it to learn this?” but “What kind of performance element will my agent need to do this once it has learned how?” Given an agent design, learning mechanisms can be constructed to improve every part of the agent.

The critic tells the learning element how well the agent is doing with respect to a fixed performance standard. The critic is necessary because the percepts themselves provide no indication of the agent's success. For example, a chess program could receive a percept indicating that it has checkmated its opponent, but it needs a performance standard to know that this is a good thing; the percept itself does not say so. It is important that the performance standard be fixed. Conceptually, one should think of it as being outside the agent altogether, because the agent must not modify it to fit its own behavior.

The last component of the learning agent is the problem generator. It is responsible for suggesting actions that will lead to new and informative experiences. The point is that if the performance element had its way, it would keep doing the actions that are best, given what it knows. But if the agent is willing to explore a little, and do some perhaps suboptimal actions in the short run, it might discover much better actions for the long run. The problem

Section 2.4.

The Structure of Agents 53

[]

Figure 2.15 A general model of learning agents.

generators job is to suggest these exploratory actions. This is what scientists do when they carry out experiments. Galileo did not think that dropping rocks from the top of a tower in Pisa was valuable in itself. He was not trying to break the rocks, nor to modify the brains of unfortunate passers-by. His aim was to modify his own brain, by identifying a better theory of the motion of objects.

To make the overall design more concrete, let us return to the automated taxi example. The performance element consists of whatever collection of knowledge and procedures the taxi has for selecting its driving actions. The taxi goes out on the road and drives, using this performance element. The critic observes the world and passes information along to the learning element. For example, after the taxi makes a quick left turn across three lanes of traffic, the critic observes the shocking language used by other drivers. From this experience, the learning element is able to formulate a rule saying this was a bad action, and the performance element is modified by installing the new rule. The problem generator might identify certain areas of behavior in need of improvement and suggest experiments, such as trying out the brakes on different road surfaces under different conditions.

The learning element can make changes to any of the “knowledge” components shown in the agent diagrams (Figures 2.9, 2.11, 2.13, and 2.14). The simplest cases involve learning directly from the percept sequence. Observation of pairs of successive states of the environment can allow the agent to learn “How the world evolves,” and observation of the results of its actions can allow the agent to learn “What my actions do.” For example, if the taxi exerts a certain braking pressure when driving on a wet road, then it will soon find out how much deceleration is actually achieved. Clearly, these two learning tasks are more difficult if the environment is only partially observable.

The forms of learning in the preceding paragraph do not need to access the external performance standard—in a sense, the standard is the universal one of making predictions

54

Chapter 2. Intelligent Agents

that agree with experiment. The situation is slightly more complex for a utility-based agent that wishes to learn utility information. For example, suppose the taxi-driving agent receives no tips from passengers who have been thoroughly shaken up during the trip. The external performance standard must inform the agent that the loss of tips is a negative contribution to its overall performance; then the agent might be able to learn that violent maneuvers do not contribute to its own utility. In a sense, the performance standard distinguishes part of the incoming percept as a reward (or penalty) that provides direct feedback on the quality of the agent’s behavior. Hard-wired performance standards such as pain and hunger in animals can be understood in this way. This issue is discussed further in Chapter 21.

In summary, agents have a variety of components, and those components can be represented in many ways within the agent program, so there appears to be great variety among learning methods. There is, however, a single unifying theme. Learning in intelligent agents can be summarized as a process of modification of each component of the agent to bring the components into closer agreement with the available feedback information, thereby improving the overall performance of the agent.

This chapter has been something of a whirlwind tour of Al, which we have conceived of as the science of agent design. The major points to recall are as follows:

e An agent is something that perceives and acts in an environment. The agent function for an agent specifies the action taken by the agent in response to any percept sequence.

¢ The performance measure evaluates the behavior of the agent in an environment. A rational agent acts so as to maximize the expected value of the performance measure, given the percept sequence it has seen so far.

¢ A task environment specification includes the performance measure, the external environment, the actuators, and the sensors. In designing an agent, the first step must always be to specify the task environment as fully as possible.

o Task environments vary along several significant dimensions. They can be fully or partially observable, deterministic or stochastic, episodic or sequential, static or dynamic, discrete or continuous, and single-agent or multiagent.

e The agent program implements the agent function. There exists a variety of basic agent-program designs, reflecting the kind of information made explicit and used in the decision process. The designs vary in efficiency, compactness, and flexibility. The appropriate design of the agent program depends on the nature of the environment.

+ Simple reflex agents respond directly to percepts, whereas model-based reflex agents maintain internal state to track aspects of the world that are not evident in the current percept. Goal-based agents act to achieve their goals, and utility-based agents try to maximize their own expected “happiness.”

e All agents can improve their performance through learning.

Section 2.5. Summary 55

BIBLIOGRAPHICAL AND HISTORICAL NOTES

The central role of action in intelligence—the notion of practical reasoning—goes back at least as far as Aristotle’s Nicomachean Ethics. Practical reasoning was also the subject of McCarthy’s (1958) influential paper “Programs with Common Sense.” The fields of robotics and control theory are, by their very nature, concerned principally with the construction of

CONTROLLER physical agents. The concept of a controller in control theory is identical to that of an agent in Al Perhaps surprisingly, Al has concentrated for most of its history on isolated components of agents—question-answering systems, theorem-provers, vision systems, and so on—rather than on whole agents. The discussion of agents in the text by Genesereth and Nilsson (1987) was an influential exception. The whole-agent view is now widely accepted in the field and is a central theme in recent texts (Poole ef al., 1998; Nilsson, 1998).

Chapter 1 traced the roots of the concept of rationality in philosophy and economics. In Al, the concept was of peripheral interest until the mid-1980s, when it began to suffuse many discussions about the proper technical foundations of the field. A paper by Jon Doyle (1983) predicted that rational agent design would come to be seen as the core mission of Al, while other popular topics would spin off to form new disciplines.

Careful attention to the properties of the environment and their consequences for rational agent design is most apparent in the control theory tradition—for example, classical control systems (Dorf and Bishop, 1999) handle fully observable, deterministic environments; stochastic optimal control (Kumar and Varaiya, 1986) handles partially observable, stochastic environments; and hybrid control (Henzinger and Sastry, 1998) deals with environments containing both discrete and continuous elements. The distinction between fully and partially observable environments is also central in the dynamic programming literature developed in the field of operations research (Puterman, 1994), which we will discuss in Chapter 17.

Reflex agents were the primary model for psychological behaviorists such as Skinner (1953), who attempted to reduce the psychology of organisms strictly to input/output or stimulus/response mappings. The advance from behaviorism to functionalism in psychology, which was at least partly driven by the application of the computer metaphor to agents (Putnam, 1960; Lewis, 1966), introduced the internal state of the agent into the picture. Most work in Al views the idea of pure reflex agents with state as too simple to provide much leverage, but work by Rosenschein (1985) and Brooks (1986) questioned this assumption (see Chapter 25). In recent years, a great deal of work has gone into finding efficient algorithms for keeping track of complex environments (Hamscher et al., 1992). The Remote Agent program that controlled the Deep Space One spacecraft (described on page 27) is a particularly impressive example (Muscettola et al., 1998; Jonsson et al., 2000).

Goal-based agents are presupposed in everything from Aristotle’s view of practical reasoning to McCarthy’s early papers on logical Al. Shakey the Robot (Fikes and Nilsson, 1971; Nilsson, 1984) was the first robotic embodiment of a logical, goal-based agent. A full logical analysis of goal-based agents appeared in Genesereth and Nilsson (1987), and a goal-based programming methodology called agent-oriented programming was developed by Shoham (1993).

56

Chapter 2. Intelligent Agents

The goal-based view also dominates the cognitive psychology tradition in the area of problem solving, beginning with the enormously influential Human Problem Solving (Newell and Simon, 1972) and running through all of Newell's later work (Newell, 1990). Goals, further analyzed as desires (general) and intentions (currently pursued), are central to the theory of agents developed by Bratman (1987). This theory has been influential both in natural language understanding and multiagent systems.

Horvitz et al. (1988) specifically suggest the use of rationality conceived as the maximization of expected utility as a basis for AL. The text by Pearl (1988) was the first in Al to cover probability and utility theory in depth; its exposition of practical methods for reasoning and decision making under uncertainty was probably the single biggest factor in the rapid shift towards utility-based agents in the 1990s (see Part V).

The general design for learning agents portrayed in Figure 2.15 is classic in the machine learning literature (Buchanan et al., 1978; Mitchell, 1997). Examples of the design, as embodied in programs, go back at least as far as Arthur Samuel’s (1959, 1967) learning program for playing checkers. Learning agents are discussed in depth in Part VI.

Interest in agents and in agent design has risen rapidly in recent years, partly because of the growth of the Internet and the perceived need for automated and mobile softbots (Etzioni and Weld, 1994). Relevant papers are collected in Readings in Agents (Huhns and Singh, 1998) and Foundations of Rational Agency (Wooldridge and Rao, 1999). Multiagent Systems (Weiss, 1999) provides a solid foundation for many aspects of agent design. Conferences devoted to agents include the International Conference on Autonomous Agents, the International Workshop on Agent Theories, Architectures, and Languages, and the International Conference on Multiagent Systems. Finally, Dung Beetle Ecology (Hanski and Cambefort, 1991) provides a wealth of interesting information on the behavior of dung beetles.

EXERCISES

2.1 Define in your own words the following terms: agent, agent function, agent program, rationality, autonomy, reflex agent, model-based agent, goal-based agent, utility-based agent, learning agent.

2.2 Both the performance measure and the utility function measure how well an agent is doing. Explain the difference between the two.

2.3 This exercise explores the differences between agent functions and agent programs.

a. Can there be more than one agent program that implements a given agent function? Give an example, or show why one is not possible.

b. Are there agent functions that cannot be implemented by any agent program?

¢. Given a fixed machine architecture, does each agent program implement exactly one agent function?

d. Given an architecture with n bits of storage, how many different possible agent programs are there?

Section 2.5. Summary 57

2.4 Let us examine the rationality of various vacuum-cleaner agent functions.

a. Show that the simple vacuum-cleaner agent function described in Figure 2.3 is indeed rational under the assumptions listed on page 36.

b. Describe a rational agent function for the modified performance measure that deducts one point for each movement. Does the corresponding agent program require internal state?

c¢. Discuss possible agent designs for the cases in which clean squares can become dirty and the geography of the environment is unknown. Does it make sense for the agent to learn from its experience in these cases? If so, what should it learn?

2.5 For each of the following agents, develop a PEAS description of the task environment:

Robot soccer player; Internet book-shopping agent; Autonomous Mars rover;

Bo Fe

Mathematician’s theorem-proving assistant. 2.6 For each of the agent types listed in Exercise 2.5, characterize the environment according to the properties given in Section 2.3, and select a suitable agent design.

The following exercises all concern the implementation of environments and agents for the vacuum-cleaner world.

2.7 Implement a performance-measuring environment simulator for the vacuum-cleaner world depicted in Figure 2.2 and specified on page 36. Your implementation should be modular, so that the sensors, actuators, and environment characteristics (size, shape, dirt placement, etc.) can be changed easily. (Note: for some choices of programming language and operating system there are already implementations in the online code repository.)

2.8 Implement a simple reflex agent for the vacuum environment in Exercise 2.7. Run the environment simulator with this agent for all possible initial dirt configurations and agent locations. Record the agent’s performance score for each configuration and its overall average score.

2.9 Consider a modified version of the vacuum environment in Exercise 2.7, in which the agent is penalized one point for each movement.

a. Can a simple reflex agent be perfectly rational for this environment? Explain.

b. What about a reflex agent with state? Design such an agent.

¢. How do your answers to a and b change if the agent’s percepts give it the clean/dirty status of every square in the environment?

2.10 Consider a modified version of the vacuum environment in Exercise 2.7, in which the geography of the environment—its extent, boundaries, and obstacles—is unknown, as is the initial dirt configuration. (The agent can go Up and Down as well as Left and Right.)

a. Can a simple reflex agent be perfectly rational for this environment? Explain.

58

Chapter 2. Intelligent Agents

b. Can a simple reflex agent with a randomized agent function outperform a simple reflex agent? Design such an agent and measure its performance on several environments.

c. Can you design an environment in which your randomized agent will perform very poorly? Show your results.

d. Can a reflex agent with state outperform a simple reflex agent? Design such an agent and measure its performance on several environments. Can you design a rational agent of this type?

2.11 Repeat Exercise 2.10 for the case in which the location sensor is replaced with a “bump” sensor that detects the agent’s attempts to move into an obstacle or to cross the boundaries of the environment. Suppose the bump sensor stops working; how should the agent behave?

2.12 The vacuum environments in the preceding exercises have all been deterministic. Discuss possible agent programs for each of the following stochastic versions:

a. Murphy’s law: twenty-five percent of the time, the Suck action fails to clean the floor if it is dirty and deposits dirt onto the floor if the floor is clean. How is your agent program affected if the dirt sensor gives the wrong answer 10% of the time?

b. Small children: At each time step, each clean square has a 10% chance of becoming dirty. Can you come up with a rational agent design for this case?

SOLVING PROBLEMS BY SEARCHING

PROBLEM-SOLVING AGENT

In which we see how an agent can find a sequence of actions that achieves its goals, when no single action will do.

The simplest agents discussed in Chapter 2 were the reflex agents, which base their actions on a direct mapping from states to actions. Such agents cannot operate well in environments for which this mapping would be too large to store and would take too long to learn. Goal-based agents, on the other hand, can succeed by considering future actions and the desirability of their outcomes.

This chapter describes one kind of goal-based agent called a problem-solving agent. Problem-solving agents decide what to do by finding sequences of actions that lead to desirable states. We start by defining precisely the elements that constitute a “problem” and its “solution,” and give several examples to illustrate these definitions. We then describe several general-purpose search algorithms that can be used to solve these problems and compare the advantages of each algorithm. The algorithms are uninformed, in the sense that they are given no information about the problem other than its definition. Chapter 4 deals with informed search algorithms, ones that have some idea of where to look for solutions.

This chapter uses concepts from the analysis of algorithms. Readers unfamiliar with the concepts of asymptotic complexity (that is, O() notation) and NP-completeness should consult Appendix A.

Intelligent agents are supposed to maximize their performance measure. As we mentioned in Chapter 2, achieving this is sometimes simplified if the agent can adopt a goal and aim at satisfying it. Let us first look at why and how an agent might do this.

Imagine an agent in the city of Arad, Romania, enjoying a touring holiday. The agent's performance measure contains many factors; it wants to improve its suntan, improve its Romanian, take in the sights, enjoy the nightlife (such as it is), avoid hangovers, and so on. The decision problem is a complex one involving many tradeoffs and careful reading of guidebooks. Now, suppose the agent has a nonrefundable ticket to fly out of Bucharest the follow

59

60

Chapter 3. Solving Problems by Searching

GOAL FORMULATION

PROBLEM FORMULATION

SOLUTION EXECUTION

ing day. In that case, it makes sense for the agent to adopt the goal of getting to Bucharest. Courses of action that don’t reach Bucharest on time can be rejected without further consideration and the agent’s decision problem is greatly simplified. Goals help organize behavior by limiting the objectives that the agent is trying to achieve. Goal formulation, based on the current situation and the agent’s performance measure, is the first step in problem solving.

We will consider a goal to be a set of world states—exactly those states in which the goal is satisfied. The agent’s task is to find out which sequence of actions will get it to a goal state. Before it can do this, it needs to decide what sorts of actions and states to consider. If it were to try to consider actions at the level of “move the left foot forward an inch” or “turn the steering wheel one degree left,” the agent would probably never find its way out of the parking lot, let alone to Bucharest, because at that level of detail there is too much uncertainty in the world and there would be too many steps in a solution. Problem formulation is the process of deciding what actions and states to consider, given a goal. We will discuss this process in more detail later. For now, let us assume that the agent will consider actions at the level of driving from one major town to another. The states it will consider therefore correspond to being in a particular town. !

Our agent has now adopted the goal of driving to Bucharest, and is considering where to go from Arad. There are three roads out of Arad, one toward Sibiu, one to Timisoara, and one to Zerind. None of these achieves the goal, so unless the agent is very familiar with the geography of Romania, it will not know which road to follow. In other words, the agent will not know which of its possible actions is best, because it does not know enough about the state that results from taking each action. If the agent has no additional knowledge, then it is stuck. The best it can do is choose one of the actions at random.

But suppose the agent has a map of Romania, either on paper or in its memory. The point of a map is to provide the agent with information about the states it might get itself into, and the actions it can take. The agent can use this information to consider subsequent stages of a hypothetical journey via each of the three towns, trying to find a journey that eventually gets to Bucharest. Once it has found a path on the map from Arad to Bucharest, it can achieve its goal by carrying out the driving actions that correspond to the legs of the journey. In general, an agent with several immediate options of unknown value can decide what to do by first examining different possible sequences of actions that lead to states of known value, and then choosing the best sequence.

This process of looking for such a sequence is called search. A search algorithm takes a problem as input and returns a solution in the form of an action sequence. Once a solution is found, the actions it recommends can be carried out. This is called the execution phase. Thus, we have a simple “formulate, search, execute” design for the agent, as shown in Figure 3.1. After formulating a goal and a problem to solve, the agent calls a search procedure to solve it. It then uses the solution to guide its actions, doing whatever the solution recommends as

1 Notice that each of these “states” actually corresponds to a large set of world states, because a real world state specifies every aspect of reality. It is important to keep in mind the distinction between states in problem solving and world states.

2 We are assuming that most readers are in the same position and can easily imagine themselves to be as clueless as our agent. We apologize to Romanian readers who are unable to take advantage of this pedagogical device.

Section 3.1.

Problem-Solving Agents 61

OPEN-LOOP

function SIMPLE-PROBLEM-SOLVING-AGENT( percept) returns an action inputs: percept, a percept static: seq, an action sequence, initially empty state, some description of the current world state goal, a goal, initially null problem, a problem formulation

state «— UPDATE-STATE(state, percept)

if seq is empty then do goal — FORMULATE-GOAL(state) problem «— FORMULATE-PROBLEM(state, goal) seq «+ SEARCH( problem)

action «— FIRST(seq)

seq — REST(seq)

return action

Figure 3.1 A simple problem-solving agent. It first formulates a goal and a problem, searches for a sequence of actions that would solve the problem, and then executes the actions one at a time. When this is complete, it formulates another goal and starts over. Note that when it is executing the sequence it ignores its percepts: it assumes that the solution it has found will always work.

the next thing to do—typically, the first action of the sequence—and then removing that step from the sequence. Once the solution has been executed, the agent will formulate a new goal.

We first describe the process of problem formulation, and then devote the bulk of the chapter to various algorithms for the SEARCH function. We will not discuss the workings of the UPDATE-STATE and FORMULATE-GOAL functions further in this chapter.

Before plunging into the details, let us pause briefly to see where problem-solving agents fit into the discussion of agents and environments in Chapter 2. The agent design in Figure 3.1 assumes that the environment is static, because formulating and solving the problem is done without paying attention to any changes that might be occurring in the environment. The agent design also assumes that the initial state is known; knowing it is easiest if the environment is observable. The idea of enumerating “alternative courses of action” assumes that the environment can be viewed as discrete. Finally, and most importantly, the agent design assumes that the environment is deterministic. Solutions to problems are single sequences of actions, so they cannot handle any unexpected events; moreover, solutions are executed without paying attention to the percepts! An agent that carries out its plans with its eyes closed, so to speak, must be quite certain of what is going on. (Control theorists call this an open-loop system, because ignoring the percepts breaks the loop between agent and environment.) All these assumptions mean that we are dealing with the easiest kinds of environments, which is one reason this chapter comes early on in the book. Section 3.6 takes a brief look at what happens when we relax the assumptions of observability and determinism. Chapters 12 and 17 go into much greater depth.

62

Chapter 3. Solving Problems by Searching

PROBLEM

INITIAL STATE

SUCCESSOR FUNCTION

STATE SPACE

PATH

GOAL TEST

PATH COST

STEP COST

OPTIMAL SOLUTION

Well-defined problems and solutions

A problem can be defined formally by four components:

o The initial state that the agent starts in. For example, the initial state for our agent in Romania might be described as In( Arad).

e A description of the possible actions available to the agent. The most common formulation® uses a successor function. Given a particular state x, SUCCESSOR-FN(z) returns a set of (action, successor) ordered pairs, where each action is one of the legal actions in state x and each successor is a state that can be reached from x by applying the action. For example, from the state In( Arad), the successor function for the Romania problem would return

{{Go(Sibiu), In(Sibiv)), {Go( Timisoara), In{ Timisoara)), {Go(Zerind), In(Zerind))}

Together, the initial state and successor function implicitly define the state space of the problem—the set of all states reachable from the initial state. The state space forms a graph in which the nodes are states and the arcs between nodes are actions. (The map of Romania shown in Figure 3.2 can be interpreted as a state space graph if we view each road as standing for two driving actions, one in each direction.) A path in the state space is a sequence of states connected by a sequence of actions.

e The goal test, which determines whether a given state is a goal state. Sometimes there is an explicit set of possible goal states, and the test simply checks whether the given state is one of them. The agent’s goal in Romania is the singleton set { In( Bucharest)}. Sometimes the goal is specified by an abstract property rather than an explicitly enumerated set of states. For example, in chess, the goal is to reach a state called “checkmate,” where the opponent’s king is under attack and can’t escape.

e A path cost function that assigns a numeric cost to each path. The problem-solving agent chooses a cost function that reflects its own performance measure. For the agent trying to get to Bucharest, time is of the essence, so the cost of a path might be its length in kilometers. In this chapter, we assume that the cost of a path can be described as the sum of the costs of the individual actions along the path. The step cost of taking action a to go from state z to state y is denoted by c¢(z, a, y). The step costs for Romania are shown in Figure 3.2 as route distances. We will assume that step costs are nonnegative.\*

The preceding elements define a problem and can be gathered together into a single data structure that is given as input to a problem-solving algorithm. A solution to a problem is a path from the initial state to a goal state. Solution quality is measured by the path cost function, and an optimal solution has the lowest path cost among all solutions.

Formulating problems

In the preceding section we proposed a formulation of the problem of getting to Bucharest in terms of the initial state, successor function, goal test, and path cost. This formulation seems

3 An alternative formulation uses a set of operators that can be applied to a state to generate successors. 4 The implications of negative costs are explored in Exercise 3,17.

Section 3.1.

Problem-Solving Agents 63

ABSTRACTION

Arad Ek

Fagaras

id Vaslui

imi Rimnicu Vilcea pn Timisoara bm C

142

111 Pitesti

id Lugoj

70 p Ld Hirsova

1 Mehadia

Urziceni

75 86

Dobreta

Bucharest

Craiova Mi Giurgiu Eforie

Figure 3.2 A simplified road map of part of Romania.

reasonable, yet it omits a great many aspects of the real world. Compare the simple state description we have chosen, In(Arad), to an actual cross-country trip, where the state of the world includes so many things: the traveling companions, what is on the radio, the scenery out of the window, whether there are any law enforcement officers nearby, how far it is to the next rest stop, the condition of the road, the weather, and so on. All these considerations are left out of our state descriptions because they are irrelevant to the problem of finding a route to Bucharest. The process of removing detail from a representation is called abstraction.

In addition to abstracting the state description, we must abstract the actions themselves. A driving action has many effects. Besides changing the location of the vehicle and its occupants, it takes up time, consumes fuel, generates pollution, and changes the agent (as they say, travel is broadening). In our formulation, we take into account only the change in location. Also, there are many actions that we will omit altogether: turning on the radio, looking out of the window, slowing down for law enforcement officers, and so on. And of course, we don’t specify actions at the level of “turn steering wheel to the left by three degrees.”

Can we be more precise about defining the appropriate level of abstraction? Think of the abstract states and actions we have chosen as corresponding to large sets of detailed world states and detailed action sequences. Now consider a solution to the abstract problem: for example, the path from Arad to Sibiu to Rimnicu Vilcea to Pitesti to Bucharest. This abstract solution corresponds to a large number of more detailed paths. For example, we could drive with the radio on between Sibiu and Rimnicu Vilcea, and then switch it off for the rest of the trip. The abstraction is valid if we can expand any abstract solution into a solution in the more detailed world; a sufficient condition is that for every detailed state that is “in Arad,”

64

Chapter 3. Solving Problems by Searching

there is a detailed path to some state that is “in Sibiu,” and so on. The abstraction is useful if carrying out each of the actions in the solution is easier than the original problem; in this case they are easy enough that they can be carried out without further search or planning by an average driving agent. The choice of a good abstraction thus involves removing as much detail as possible while retaining validity and ensuring that the abstract actions are easy to carry out. Were it not for the ability to construct useful abstractions, intelligent agents would be completely swamped by the real world.

TOY PROBLEM

REAL-WORLD PROBLEM

8-PUZZLE

The problem-solving approach has been applied to a vast array of task environments. We list some of the best known here, distinguishing between toy and real-world problems. A toy problem is intended to illustrate or exercise various problem-solving methods. It can be given a concise, exact description. This means that it can be used easily by different researchers to compare the performance of algorithms. A real-world problem is one whose solutions people actually care about. They tend not to have a single agreed-upon description, but we will attempt to give the general flavor of their formulations.

Toy problems

The first example we will examine is the vacuum world first introduced in Chapter 2. (See Figure 2.2.) This can be formulated as a problem as follows:

¢ States: The agent is in one of two locations, each of which might or might not contain dirt. Thus there are 2 x 2° = 8 possible world states.

¢ Initial state: Any state can be designated as the initial state.

¢ Successor function: This generates the legal states that result from trying the three actions (Left, Right, and Suck). The complete state space is shown in Figure 3.3.

¢& Goal test: This checks whether all the squares are clean.

<> Path cost: Each step costs 1, so the path cost is the number of steps in the path.

Compared with the real world, this toy problem has discrete locations, discrete dirt, reliable cleaning, and it never gets messed up once cleaned. (In Section 3.6, we will relax these assumptions.) One important thing to note is that the state is determined by both the agent location and the dirt locations. A larger environment with n locations has n 2" states.

The 8-puzzle, an instance of which is shown in Figure 3.4, consists of a 3x3 board with eight numbered tiles and a blank space. A tile adjacent to the blank space can slide into the space. The object is to reach a specified goal state, such as the one shown on the right of the figure. The standard formulation is as follows:

¢ States: A state description specifies the location of each of the eight tiles and the blank in one of the nine squares.

¢ Initial state: Any state can be designated as the initial state. Note that any given goal can be reached from exactly half of the possible initial states (Exercise 3.4).

Section 3.2.

Example Problems 65

[]

Figure 3.3 The state space for the vacuum world. Arcs denote actions: L = Left, R =

Right, S = Suck.

<> Successor function: This generates the legal states that result from trying the four actions (blank moves Left, Right, Up, or Down).

¢& Goal test: This checks whether the state matches the goal configuration shown in Figure 3.4. (Other goal configurations are possible.)

<> Path cost: Each step costs 1, so the path cost is the number of steps in the path.

What abstractions have we included here? The actions are abstracted to their beginning and final states, ignoring the intermediate locations where the block is sliding. We've abstracted away actions such as shaking the board when pieces get stuck, or extracting the pieces with a knife and putting them back again. We’re left with a description of the rules of the puzzle, avoiding all the details of physical manipulations.

1 7 | 2 4 1 2 5 6 3 4 5 ba) Tre 8 il 3 Ij 1 6 7 | 8 Start State Goal State Figure 3.4 A typical instance of the 8-puzzle.

66 Chapter 3. Solving Problems by Searching

SLIDING BLOCK The 8-puzzle belongs to the family of sliding-block puzzles, which are often used as test problems for new search algorithms in Al This general class is known to be NP-complete, so one does not expect to find methods significantly better in the worst case than the search algorithms described in this chapter and the next. The 8-puzzle has 9!/2 = 181, 440 reachable states and is easily solved. The 15-puzzle (on a 4 x 4 board) has around 1.3 trillion states, and random instances can be solved optimally in a few milliseconds by the best search algorithms. The 24-puzzle (on a 5 x 5 board) has around 10° states, and random instances are still quite difficult to solve optimally with current machines and algorithms.

8-QUEENS PROBLEM The goal of the 8-queens problem is to place eight queens on a chessboard such that no queen attacks any other. (A queen attacks any piece in the same row, column or diagonal.) Figure 3.5 shows an attempted solution that fails: the queen in the rightmost column is attacked by the queen at the top left.

Figure 3.5 Almost a solution to the 8-queens problem. (Solution is left as an exercise.)

Although efficient special-purpose algorithms exist for this problem and the whole nqueens family, it remains an interesting test problem for search algorithms. There are two INCrEMENT main kinds of formulation. An incremental formulation involves operators that augment the state description, starting with an empty state; for the 8-queens problem, this means that COMPLETESTATE each action adds a queen to the state. A complete-state formulation starts with all 8 queens on the board and moves them around. In either case, the path cost is of no interest because

only the final state counts. The first incremental formulation one might try is the following:

{> States: Any arrangement of 0 to 8 queens on the board is a state. {> Initial state: No queens on the board. {> Successor function: Add a queen to any empty square.

{> Goal test: 8 queens are on the board, none attacked.

Section 3.2.

Example Problems 67

ROUTE-FINDING PROBLEM

TOURING PROBLEMS

In this formulation, we have 64 63 --57 ~ 3 x 10'\* possible sequences to investigate. A better formulation would prohibit placing a queen in any square that is already attacked:

$ States: Arrangements of n queens (0 < n < 8), one per column in the leftmost n columns, with no queen attacking another are states.

¢& Successor function: Add a queen to any square in the leftmost empty column such that it is not attacked by any other queen.

This formulation reduces the 8-queens state space from 3 x 10'4 to just 2,057, and solutions are easy to find. On the other hand, for 100 queens the initial formulation has roughly 10400 states whereas the improved formulation has about 10°2 states (Exercise 3.5). This is a huge reduction, but the improved state space is still too big for the algorithms in this chapter to handle. Chapter 4 describes the complete-state formulation and Chapter 5 gives a simple algorithm that makes even the million-queens problem easy to solve.

Real-world problems

We have already seen how the route-finding problem is defined in terms of specified locations and transitions along links between them. Route-finding algorithms are used in a variety of applications, such as routing in computer networks, military operations planning, and airline travel planning systems. These problems are typically complex to specify. Consider a simplified example of an airline travel problem specified as follows:

{> States: Each is represented by a location (e.g., an airport) and the current time.

¢> Initial state: This is specified by the problem.

{> Successor function: This returns the states resulting from taking any scheduled flight (perhaps further specified by seat class and location), leaving later than the current time plus the within-airport transit time, from the current airport to another.

¢{ Goal test: Are we at the destination by some prespecified time?

{> Path cost: This depends on monetary cost, waiting time, flight time, customs and immigration procedures, seat quality, time of day, type of airplane, frequent-flyer mileage awards, and so on.

Commercial travel advice systems use a problem formulation of this kind, with many additional complications to handle the byzantine fare structures that airlines impose. Any seasoned traveller knows, however, that not all air travel goes according to plan. A really good system should include contingency plans—such as backup reservations on alternate flights— to the extent that these are justified by the cost and likelihood of failure of the original plan.

Touring problems are closely related to route-finding problems, but with an important difference. Consider, for example, the problem, “Visit every city in Figure 3.2 at least once, starting and ending in Bucharest.” As with route finding, the actions correspond to trips between adjacent cities. The state space, however, is quite different. Each state must include not just the current location but also the set of cities the agent has visited. So the initial state would be “In Bucharest; visited {Bucharest}, a typical intermediate state would be “In Vaslui; visited {Bucharest,Urziceni, Vaslui },” and the goal test would check whether the agent is in Bucharest and all 20 cities have been visited.

68

Chapter 3. Solving Problems by Searching

TRAVELING SALESPERSON PROBLEM

VLSI LAYOUT

ROBOT NAVIGATION

AUTOMATIC ASSEMBLY SEQUENCING

PROTEIN DESIGN

INTERNET SEARCHING

The traveling salesperson problem (TSP) is a touring problem in which each city must be visited exactly once. The aim is to find the shortest tour. The problem is known to be NP-hard, but an enormous amount of effort has been expended to improve the capabilities of TSP algorithms. In addition to planning trips for traveling salespersons, these algorithms have been used for tasks such as planning movements of automatic circuit-board drills and of stocking machines on shop floors.

A VLSI layout problem requires positioning millions of components and connections on a chip to minimize area, minimize circuit delays, minimize stray capacitances, and maximize manufacturing yield. The layout problem comes after the logical design phase, and is usually split into two parts: cell layout and channel routing. In cell layout, the primitive components of the circuit are grouped into cells, each of which performs some recognized function. Each cell has a fixed footprint (size and shape) and requires a certain number of connections to each of the other cells. The aim is to place the cells on the chip so that they do not overlap and so that there is room for the connecting wires to be placed between the cells. Channel routing finds a specific route for each wire through the gaps between the cells. These search problems are extremely complex, but definitely worth solving. In Chapter 4, we will see some algorithms capable of solving them.

Robot navigation is a generalization of the route-finding problem described earlier. Rather than a discrete set of routes, a robot can move in a continuous space with (in principle) an infinite set of possible actions and states. For a circular robot moving on a flat surface, the space is essentially two-dimensional. When the robot has arms and legs or wheels that must also be controlled, the search space becomes many-dimensional. Advanced techniques are required just to make the search space finite. We examine some of these methods in Chapter 25. In addition to the complexity of the problem, real robots must also deal with errors in their sensor readings and motor controls.

Automatic assembly sequencing of complex objects by a robot was first demonstrated by FREDDY (Michie, 1972). Progress since then has been slow but sure, to the point where the assembly of intricate objects such as electric motors is economically feasible. In assembly problems, the aim is to find an order in which to assemble the parts of some object. If the wrong order is chosen, there will be no way to add some part later in the sequence without undoing some of the work already done. Checking a step in the sequence for feasibility is a difficult geometrical search problem closely related to robot navigation. Thus, the generation of legal successors is the expensive part of assembly sequencing. Any practical algorithm must avoid exploring all but a tiny fraction of the state space. Another important assembly problem is protein design, in which the goal is to find a sequence of amino acids that will fold into a three-dimensional protein with the right properties to cure some disease.

In recent years there has been increased demand for software robots that perform Internet searching, looking for answers to questions, for related information, or for shopping deals. This is a good application for search techniques, because it is easy to conceptualize the Internet as a graph of nodes (pages) connected by links. A full description of Internet search is deferred until Chapter 10.

Section 3.3. Searching for Solutions 69

3.3 SEARCHING FOR SOLUTIONS

Having formulated some problems, we now need to solve them. This is done by a search through the state space. This chapter deals with search techniques that use an explicit search SEARCH TREE tree that is generated by the initial state and the successor function that together define the state space. In general, we may have a search graph rather than a search tree, when the same state can be reached from multiple paths. We defer consideration of this important complication until Section 3.5. Figure 3.6 shows some of the expansions in the search tree for finding a route from SEARCH NODE Arad to Bucharest. The root of the search tree is a search node corresponding to the initial state, In(Arad). The first step is to test whether this is a goal state. Clearly it is not, but 1t is important to check so that we can solve trick problems like “starting in Arad, get to Arad.” Because this is not a goal state, we need to consider some other states. This is done

EXPANDING by expanding the current state; that is, applying the successor function to the current state,

GENERATING thereby generating a new set of states. In this case, we get three new states: In(Sibiu), In(Timisoara), and In(Zerind). Now we must choose which of these three possibilities to consider further.

This is the essence of search—following up one option now and putting the others aside for later, in case the first choice does not lead to a solution. Suppose we choose Sibiu first. We check to see whether it is a goal state (it is not) and then expand it to get In(Arad), In(Fagaras), In(Oradea), and In(RimnicuVilcea). We can then choose any of these four, or go back and choose Timisoara or Zerind. We continue choosing, testing, and expanding until either a solution is found or there are no more states to be expanded. The choice of which

sEARCHSTRATEGY state to expand is determined by the search strategy. The general tree-search algorithm is described informally in Figure 3.7.

It is important to distinguish between the state space and the search tree. For the route finding problem, there are only 20 states in the state space, one for each city. But there are an infinite number of paths in this state space, so the search tree has an infinite number of nodes. For example, the three paths Arad-Sibiu, Arad-Sibiu—-Arad, Arad-Sibiu—Arad-Sibiu are the first three of an infinite sequence of paths. (Obviously, a good search algorithm avoids following such repeated paths; Section 3.5 shows how.)

There are many ways to represent nodes, but we will assume that a node is a data structure with five components:

e STATE: the state in the state space to which the node corresponds; e PARENT-NODE: the node in the search tree that generated this node; e ACTION: the action that was applied to the parent to generate the node;

e PATH-COST: the cost, traditionally denoted by g(n), of the path from the initial state to the node, as indicated by the parent pointers; and

e DEPTH: the number of steps along the path from the initial state.

It is important to remember the distinction between nodes and states. A node is a bookkeeping data structure used to represent the search tree. A state corresponds to a configuration of the

70

Chapter 3. Solving Problems by Searching

FRINGE LEAF NODE

world. Thus, nodes are on particular paths, as defined by PARENT-NODE pointers, whereas states are not. Furthermore, two different nodes can contain the same world state, if that state is generated via two different search paths. The node data structure is depicted in Figure 3.8.

We also need to represent the collection of nodes that have been generated but not yet expanded—this collection is called the fringe. Each element of the fringe is a leaf node, that

[]

Figure 3.6 Partial search trees for finding a route from Arad to Bucharest. Nodes that have been expanded are shaded; nodes that have been generated but not yet expanded are outlined in bold; nodes that have not yet been generated are shown in faint dashed lines.

function TREE-SEARCH( problem, strategy) returns a solution, or failure initialize the search tree using the initial state of problem loop do if there are no candidates for expansion then return failure choose a leaf node for expansion according to strategy if the node contains a goal state then return the corresponding solution else expand the node and add the resulting nodes to the search tree

Figure 3.7 An informal description of the general tree-search algorithm.

Section 3.3.

Searching for Solutions 71

QUEUE

COMPLETENESS OPTIMALITY TIME COMPLEXITY

SPACE COMPLEXITY

PARENT-NODE

ACTION = right DEPTH = 6 PATH-COST =6

Figure 3.8 Nodes are the data structures from which the search tree is constructed. Each has a parent, a state, and various bookkeeping fields. Arrows point from child to parent.

is, a node with no successors in the tree. In Figure 3.6, the fringe of each tree consists of those nodes with bold outlines. The simplest representation of the fringe would be a set of nodes. The search strategy then would be a function that selects the next node to be expanded from this set. Although this is conceptually straightforward, it could be computationally expensive, because the strategy function might have to look at every element of the set to choose the best one. Therefore, we will assume that the collection of nodes is implemented as a queue. The operations on a queue are as follows:

o MAKE-QUEUE(element, ...) creates a queue with the given element(s).

EMPTY ?(queue) returns true only if there are no more elements in the queue.

FIRST(gueue) returns the first element of the queue.

REMOVE-FIRST(queue) returns FIRST(queue) and removes it from the queue.

INSERT(element, queue) inserts an element into the queue and returns the resulting queue. (We will see that different types of queues insert elements in different orders.)

INSERT-ALL(elements, queue) inserts a set of elements into the queue and returns the resulting queue.

With these definitions, we can write the more formal version of the general tree-search algorithm shown in Figure 3.9.

Measuring problem-solving performance

The output of a problem-solving algorithm is either failure or a solution. (Some algorithms might get stuck in an infinite loop and never return an output.) We will evaluate an algorithm’s performance in four ways:

¢$ Completeness: Is the algorithm guaranteed to find a solution when there is one?

¢& Optimality: Does the strategy find the optimal solution, as defined on page 62?

¢ Time complexity: How long does it take to find a solution?

{> Space complexity: How much memory is needed to perform the search?

72

Chapter 3. Solving Problems by Searching

BRANCHING FACTOR

SEARCH COST

TOTAL COST

function TREE-SEARCH( problem, fringe) returns a solution, or failure

fringe — INSERT(MAKE-NODE(INITIAL-STATE[ problem]), fringe) loop do if EMPTY ?( fringe) then return failure node — REMOVE-FIRST( fringe) if GOAL-TEST( problem] applied to STATE[node] succeeds then return SOLUTION(node) fringe «— INSERT-ALL(EXPAND(node, problem), fringe)

function EXPAND( node, problem) returns a set of nodes

successors — the empty set for each (action, result) in SUCCESSOR-FN[ problem ](STATE[node]) do s «— anew NODE STATE[s] «— result PARENT-NODE[s] + node ACTION[s] «+ action PATH-COST[s] + PATH-COST[node] + STEP-COST(node, action, s) DEPTH[s] «+ DEPTH[node] + 1 add s to successors return successors

Figure 3.9 The general tree-search algorithm. (Note that the fringe argument must be an empty queue, and the type of the queue will affect the order of the search.) The SOLUTION function returns the sequence of actions obtained by following parent pointers back to the root.

Time and space complexity are always considered with respect to some measure of the problem difficulty. In theoretical computer science, the typical measure is the size of the state space graph, because the graph is viewed as an explicit data structure that is input to the search program. (The map of Romania is an example of this.) In AI, where the graph is represented implicitly by the initial state and successor function and is frequently infinite, complexity is expressed in terms of three quantities: b, the branching factor or maximum number of successors of any node; d, the depth of the shallowest goal node; and m, the maximum length of any path in the state space.

Time is often measured in terms of the number of nodes generated® during the search, and space in terms of the maximum number of nodes stored in memory,

To assess the effectiveness of a search algorithm, we can consider just the search cost— which typically depends on the time complexity but can also include a term for memory usage—or we can use the total cost, which combines the search cost and the path cost of the solution found. For the problem of finding a route from Arad to Bucharest, the search cost

5 Some texts measure time in terms of the number of node expansions instead. The two measures differ by at most a factor of b. It seems to us that the execution time of a node expansion increases with the number of nodes generated in that expansion.

Section 3.4.

Uninformed Search Strategies 73

is the amount of time taken by the search and the solution cost is the total length of the path in kilometers. Thus, to compute the total cost, we have to add kilometers and milliseconds. There is no “official exchange rate” between the two, but it might be reasonable in this case to convert kilometers into milliseconds by using an estimate of the car’s average speed (because time is what the agent cares about). This enables the agent to find an optimal tradeoff point at which further computation to find a shorter path becomes counterproductive. The more general problem of tradeoffs between different goods will be taken up in Chapter 16.

\

UNINFORMED SEARCH

INFORMED SEARCH HEURISTIC SEARCH

BREADTH-FIRST SEARCH

This section covers five search strategies that come under the heading of uninformed search (also called blind search). The term means that they have no additional information about states beyond that provided in the problem definition. All they can do is generate successors and distinguish a goal state from a nongoal state. Strategies that know whether one nongoal state is “more promising” than another are called informed search or heuristic search strategies; they will be covered in Chapter 4. All search strategies are distinguished by the order in which nodes are expanded.

Breadth-first search

Breadth-first search is a simple strategy in which the root node is expanded first, then all the successors of the root node are expanded next, then their successors, and so on. In general, all the nodes are expanded at a given depth in the search tree before any nodes at the next level are expanded.

Breadth-first search can be implemented by calling TREE-SEARCH with an empty fringe that is a first-in-first-out (FIFO) queue, assuring that the nodes that are visited first will be expanded first. In other words, calling TREE-SEARCH( problem, FIFO-QUEUE()) results in a breadth-first search. The FIFO queue puts all newly generated successors at the end of the queue, which means that shallow nodes are expanded before deeper nodes. Figure 3.10 shows the progress of the search on a simple binary tree.

We will evaluate breadth-first search using the four criteria from the previous section. We can easily see that it is complete—if the shallowest goal node is at some finite depth d, breadth-first search will eventually find it after expanding all shallower nodes (provided the branching factor b is finite). The shallowest goal node is not necessarily the optimal one; technically, breadth-first search is optimal if the path cost is a nondecreasing function of the depth of the node. (For example, when all actions have the same cost.)

So far, the news about breadth-first search has been good. To see why it is not always the strategy of choice, we have to consider the amount of time and memory it takes to complete a search. To do this, we consider a hypothetical state space where every state has b successors. The root of the search tree generates b nodes at the first level, each of which generates b more nodes, for a total of b2 at the second level. Each of these generates b more nodes, yielding b3 nodes at the third level, and so on. Now suppose that the solution is at depth d. In the worst

74

Chapter 3. Solving Problems by Searching

case, we would expand all but the last node at level d (since the goal itself is not expanded), generating bt! — b nodes at level d + 1. Then the total number of nodes generated is

b4+b? +03 +--+ b+ (pH —b) = O41).

Every node that is generated must remain in memory, because it is either part of the fringe or is an ancestor of a fringe node. The space complexity is, therefore, the same as the time complexity (plus one node for the root).

Those who do complexity analysis are worried (or excited, if they like a challenge) by exponential complexity bounds such as O(b%\*1). Figure 3.11 shows why. It lists the time and memory required for a breadth-first search with branching factor b = 10, for various values of the solution depth d. The table assumes that 10,000 nodes can be generated per second and that a node requires 1000 bytes of storage. Many search problems fit roughly within these assumptions (give or take a factor of 100) when run on a modern personal computer.

There are two lessons to be learned from Figure 3.11. First, the memory requirements are a bigger problem for breadth-first search than is the execution time. 31 hours would not be too long to wait for the solution to an important problem of depth 8, but few computers have the terabyte of main memory it would take. Fortunately, there are other search strategies that require less memory.

The second lesson is that the time requirements are still a major factor. If your problem has a solution at depth 12, then (given our assumptions) it will take 35 years for breadth-first search (or indeed any uninformed search) to find it. In general, exponential-complexity search problems cannot be solved by uninformed methods for any but the smallest instances.

P®

Figure 3.10 Breadth-first search on a simple binary tree. At each stage, the node to be expanded next is indicated by a marker.

Depth Nodes Time Memory 2 1100 .11 seconds 1 megabyte 4 111,100 11 seconds 106 megabytes 6 107 19 minutes 10 gigabytes 8 10° 31 hours 1 terabytes 10 101! 129 days 101 terabytes 12 1013 35 years 10 petabytes 14 1015 3,523 years 1 exabyte

Figure 3.11 Time and memory requirements for breadth-first search. The numbers shown assume branching factor b = 10; 10,000 nodes/second; 1000 bytes/node.

Section 3.4.

Uninformed Search Strategies 75

UNIFORM-COST SEARCH

DEPTH-FIRST SEARCH

Uniform-cost search

Breadth-first search is optimal when all step costs are equal, because it always expands the shallowest unexpanded node. By a simple extension, we can find an algorithm that is optimal with any step cost function. Instead of expanding the shallowest node, uniform-cost search expands the node n with the lowest path cost. Note that if all step costs are equal, this is identical to breadth-first search.

Uniform-cost search does not care about the number of steps a path has, but only about their total cost. Therefore, it will get stuck in an infinite loop if it ever expands a node that has a zero-cost action leading back to the same state (for example, a NoOp action). We can guarantee completeness provided the cost of every step is greater than or equal to some small positive constant €. This condition is also sufficient to ensure optimality. It means that the cost of a path always increases as we go along the path. From this property, it is easy to see that the algorithm expands nodes in order of increasing path cost. Therefore, the first goal node selected for expansion is the optimal solution. (Remember that TREE-SEARCH applies the goal test only to the nodes that are selected for expansion.) We recommend trying the algorithm out to find the shortest path to Bucharest.

Uniform-cost search is guided by path costs rather than depths, so its complexity cannot easily be characterized in terms of b and d. Instead, let C\* be the cost of the optimal solution, and assume that every action costs at least €. Then the algorithm’s worst-case time and space complexity is O(b+LC"/¢] ), which can be much greater than b. This is because uniform-cost search can, and often does, explore large trees of small steps before exploring paths involving large and perhaps useful steps. When all step costs are equal, of course, b+" is just be.

Depth-first search

Depth-first search always expands the deepest node in the current fringe of the search tree. The progress of the search is illustrated in Figure 3.12. The search proceeds immediately to the deepest level of the search tree, where the nodes have no successors. As those nodes are expanded, they are dropped from the fringe, so then the search “backs up” to the next shallowest node that still has unexplored successors.

This strategy can be implemented by TREE-SEARCH with a last-in-first-out (LIFO) queue, also known as a stack. As an alternative to the TREE-SEARCH implementation, it is common to implement depth-first search with a recursive function that calls itself on each of its children in turn. (A recursive depth-first algorithm incorporating a depth limit is shown in Figure 3.13.)

Depth-first search has very modest memory requirements. It needs to store only a single path from the root to a leaf node, along with the remaining unexpanded sibling nodes for each node on the path. Once a node has been expanded, it can be removed from memory as soon as all its descendants have been fully explored. (See Figure 3.12.) For a state space with branching factor b and maximum depth m, depth-first search requires storage of only bm + 1 nodes. Using the same assumptions as Figure 3.11, and assuming that nodes at the same depth as the goal node have no successors, we find that depth-first search would require 118 kilobytes instead of 10 petabytes at depth d = 12, a factor of 10 billion times less space.

76

Chapter 3. Solving Problems by Searching

[]

BACKTRACKING SEARCH

Figure 3.12 Depth-first search on a binary tree. Nodes that have been expanded and have no descendants in the fringe can be removed from memory; these are shown in black. Nodes at depth 3 are assumed to have no successors and M is the only goal node.

A variant of depth-first search called backtracking search uses still less memory. In backtracking, only one successor is generated at a time rather than all successors; each partially expanded node remembers which successor to generate next. In this way, only O(m) memory is needed rather than O(bm). Backtracking search facilitates yet another memorysaving (and time-saving) trick: the idea of generating a successor by modifying the current state description directly rather than copying it first. This reduces the memory requirements to just one state description and O(m) actions. For this to work, we must be able to undo each modification when we go back to generate the next successor. For problems with large state descriptions, such as robotic assembly, these techniques are critical to success.

The drawback of depth-first search is that it can make a wrong choice and get stuck going down a very long (or even infinite) path when a different choice would lead to a solution near the root of the search tree. For example, in Figure 3.12, depth-first search will explore the entire left subtree even if node C is a goal node. If node J were also a goal node, then depth-first search would return it as a solution; hence, depth-first search is not optimal. If

Section 3.4.

Uninformed Search Strategies 77

DEPTH-LIMITED SEARCH

DIAMETER

function DEPTH-LIMITED-SEARCH( problem, limit) returns a solution, or failure/cutoff return RECURSIVE-DLS(MAKE-NODE(INITIAL-STATE|[ problem]), problem, limit)

function RECURSIVE-DLS (node, problem, limit) returns a solution, or failure/cutoff cutoff-occurred? «— false if GOAL-TEST[ problem](STATE[node]) then return SOLUTION(node) else if DEPTH[ node] = limit then return cutoff else for each successor in EXPAND(node, problem) do result +— RECURSIVE-DLS (successor, problem, limit) if result = cutoff then cutoff\_occurred? + true else if result # failure then return result if cutoff\_occurred? then return cutoff else return failure

Figure 3.13 A recursive implementation of depth-limited search.

the left subtree were of unbounded depth but contained no solutions, depth-first search would never terminate; hence, it is not complete. In the worst case, depth-first search will generate all of the O(b™) nodes in the search tree, where m is the maximum depth of any node. Note that m can be much larger than d (the depth of the shallowest solution), and is infinite if the tree is unbounded.

Depth-limited search

The problem of unbounded trees can be alleviated by supplying depth-first search with a predetermined depth limit ¢. That is, nodes at depth ¢ are treated as if they have no successors. This approach is called depth-limited search. The depth limit solves the infinite-path problem. Unfortunately, it also introduces an additional source of incompleteness if we choose ¢ < d, that is, the shallowest goal is beyond the depth limit. (This is not unlikely when d is unknown.) Depth-limited search will also be nonoptimal if we choose £ > d. Its time complexity is O(b\*) and its space complexity is O(b?). Depth-first search can be viewed as a special case of depth-limited search with £= oc.

Sometimes, depth limits can be based on knowledge of the problem. For example, on the map of Romania there are 20 cities. Therefore, we know that if there is a solution, it must be of length 19 at the longest, so £ = 19 is a possible choice. But in fact if we studied the map carefully, we would discover that any city can be reached from any other city in at most 9 steps. This number, known as the diameter of the state space, gives us a better depth limit, which leads to a more efficient depth-limited search. For most problems, however, we will not know a good depth limit until we have solved the problem.

Depth-limited search can be implemented as a simple modification to the general treesearch algorithm or to the recursive depth-first search algorithm. We show the pseudocode for recursive depth-limited search in Figure 3.13. Notice that depth-limited search can terminate with two kinds of failure: the standard failure value indicates no solution; the cutoff value indicates no solution within the depth limit.

Chapter 3. Solving Problems by Searching

EEPENING SEARCH

Iterative deepening depth-first search

Iterative deepening search (or iterative deepening depth-first search) is a general strategy, often used in combination with depth-first search, that finds the best depth limit. It does this by gradually increasing the limit—first 0, then 1, then 2, and so on—until a goal is found. This will occur when the depth limit reaches d, the depth of the shallowest goal node. The algorithm is shown in Figure 3.14. Iterative deepening combines the benefits of depth-first and breadth-first search. Like depth-first search, its memory requirements are very modest: O(bd) to be precise. Like breadth-first search, it is complete when the branching factor is finite and optimal when the path cost is a nondecreasing function of the depth of the node. Figure 3.15 shows four iterations of ITERATIVE-DEEPENING-SEARCH on a binary search tree, where the solution is found on the fourth iteration.

Iterative deepening search may seem wasteful, because states are generated multiple times. It turns out this is not very costly. The reason is that in a search tree with the same (or nearly the same) branching factor at each level, most of the nodes are in the bottom level, so it does not matter much that the upper levels are generated multiple times. In an iterative deepening search, the nodes on the bottom level (depth d) are generated once, those on the next to bottom level are generated twice, and so on, up to the children of the root, which are generated d times. So the total number of nodes generated is

NDS) = (d)b + (d — 1)b\* +--+ (1)b?, which gives a time complexity of O(b%). We can compare this to the nodes generated by a breadth-first search:

N(BFS) =b+b\* ++ +b + (pT —b) Notice that breadth-first search generates some nodes at depth d+ 1, whereas iterative deepen

ing does not. The result is that iterative deepening is actually faster than breadth-first search, despite the repeated generation of states. For example, if b = 10 and d = 5, the numbers are

N(IDS) = 50 + 400 + 3,000 + 20, 000 + 100, 000 = 123, 450 N(BFS) = 10+ 100 + 1,000 + 10,000 + 100,000 + 999,990 = 1,111, 100.

In general, iterative deepening is the preferred uninformed search method when there is a large search space and the depth of the solution is not known.

function ITERATIVE-DEEPENING-SEARCH( problem) returns a solution, or failure inputs: problem, a problem

for depth «— 0 to co do result + DEPTH-LIMITED-SEARCH( problem, depth) if result # cutoff then return result

Figure 3.14 The iterative deepening search algorithm, which repeatedly applies depthlimited search with increasing limits. It terminates when a solution is found or if the depthlimited search returns failure, meaning that no solution exists.

Uninformed Search Strategies 79

[]

ITERATIVE LENGTHENING SEARCH

Figure 3.15 Four iterations of iterative deepening search on a binary tree.

Iterative deepening search is analogous to breadth-first search in that it explores a complete layer of new nodes at each iteration before going on to the next layer. It would seem worthwhile to develop an iterative analog to uniform-cost search, inheriting the latter algorithm’s optimality guarantees while avoiding its memory requirements. The idea is to use increasing path-cost limits instead of increasing depth limits. The resulting algorithm, called iterative lengthening search, is explored in Exercise 3.11. It turns out, unfortunately, that iterative lengthening incurs substantial overhead compared to uniform-cost search.

Bidirectional search

The idea behind bidirectional search is to run two simultaneous searches—one forward from the initial state and the other backward from the goal, stopping when the two searches meet

80

Chapter 3. Solving Problems by Searching

PREDECESSORS

2 eT

Figure 3.16 A schematic view of a bidirectional search that is about to succeed, when a branch from the start node meets a branch from the goal node.

in the middle (Figure 3.16). The motivation is that 5/2 + b%/? is much less than b?, or in the figure, the area of the two small circles is less than the area of one big circle centered on the start and reaching to the goal.

Bidirectional search is implemented by having one or both of the searches check each node before it is expanded to see if it is in the fringe of the other search tree; if so, a solution has been found. For example, if a problem has solution depth d = 6, and each direction runs breadth-first search one node at a time, then in the worst case the two searches meet when each has expanded all but one of the nodes at depth 3. For b= 10, this means a total of 22,200 node generations, compared with 11,111,100 for a standard breadth-first search. Checking a node for membership in the other search tree can be done in constant time with a hash table, so the time complexity of bidirectional search is O(b%/2). Atleast one of the search trees must be kept in memory so that the membership check can be done, hence the space complexity is also O(b%/2). This space requirement is the most significant weakness of bidirectional search. The algorithm is complete and optimal (for uniform step costs) if both searches are breadth-first; other combinations may sacrifice completeness, optimality, or both.

The reduction in time complexity makes bidirectional search attractive, but how do we search backwards? This is not as easy as it sounds. Let the predecessors of a node n, Pred(n), be all those nodes that have n as a successor. Bidirectional search requires that Pred(n) be efficiently computable. The easiest case is when all the actions in the state space are reversible, so that Pred(n) = Succ(n). Other cases may require substantial ingenuity.

Consider the question of what we mean by “the goal” in searching “backward from the goal.” For the 8-puzzle and for finding a route in Romania, there is just one goal state, so the backward search is very much like the forward search. If there are several explicitly listed goal states—for example, the two dirt-free goal states in Figure 3.3—then we can construct a new dummy goal state whose immediate predecessors are all the actual goal states. Alternatively, some redundant node generations can be avoided by viewing the set of goal states as a single state, each of whose predecessors is also a set of states—specifically, the set of states having a corresponding successor in the set of goal states. (See also Section 3.6.)

The most difficult case for bidirectional search is when the goal test gives only an implicit description of some possibly large set of goal states—for example, all the states satisfy

Section 3.5.

Avoiding Repeated States 81

ing the “checkmate” goal test in chess. A backward search would need to construct compact descriptions of “all states that lead to checkmate by move 7,” and so on; and those descriptions would have to be tested against the states generated by the forward search. There is no general way to do this efficiently.

Comparing uninformed search strategies

Figure 3.17 compares search strategies in terms of the four evaluation criteria set forth in Section 3.4.

[]

the shallowest solution; m is the maximum depth of the search tree; [ is the depth limit. Superscript caveats are as follows: ¢ complete if b is finite; ® complete if step costs > ¢ for positive €; © optimal if step costs are all identical; ¢ if both directions use breadth-first search.

RECTANGULAR GRID

Up to this point, we have all but ignored one of the most important complications to the search process: the possibility of wasting time by expanding states that have already been encountered and expanded before. For some problems, this possibility never comes up; the state space is a tree and there is only one path to each state. The efficient formulation of the §8queens problem (where each new queen is placed in the leftmost empty column) is efficient in large part because of this—each state can be reached only through one path. If we formulate the 8-queens problem so that a queen can be placed in any column, then each state with n queens can be reached by n! different paths.

For some problems, repeated states are unavoidable. This includes all problems where the actions are reversible, such as route-finding problems and sliding-blocks puzzles. The search trees for these problems are infinite, but if we prune some of the repeated states, we can cut the search tree down to finite size, generating only the portion of the tree that spans the state-space graph. Considering just the search tree up to a fixed depth, it is easy to find cases where eliminating repeated states yields an exponential reduction in search cost. In the extreme case, a state space of size d + 1 (Figure 3.18(a)) becomes a tree with 2¢ leaves (Figure 3.18(b)). A more realistic example is the rectangular grid as illustrated in Figure 3.18(c). On a grid, each state has four successors, so the search tree including repeated

82

Chapter 3. Solving Problems by Searching

CLOSED LIST OPEN LIST

states has 47 leaves; but there are only about 2d? distinct states within d steps of any given state. For d = 20, this means about a trillion nodes but only about 800 distinct states.

Repeated states, then, can cause a solvable problem to become unsolvable if the algorithm does not detect them. Detection usually means comparing the node about to be expanded to those that have been expanded already; if a match is found, then the algorithm has discovered two paths to the same state and can discard one of them.

For depth-first search, the only nodes in memory are those on the path from the root to the current node. Comparing those nodes to the current node allows the algorithm to detect looping paths that can be discarded immediately. This is fine for ensuring that finite state spaces do not become infinite search trees because of loops; unfortunately, it does not avoid the exponential proliferation of nonlooping paths in problems such as those in Figure 3.18. The only way to avoid these is to keep more nodes in memory. There is a fundamental tradeoff between space and time. Algorithms that forget their history are doomed to repeat it.

If an algorithm remembers every state that it has visited, then it can be viewed as exploring the state-space graph directly. We can modify the general TREE-SEARCH algorithm to include a data structure called the closed list, which stores every expanded node. (The fringe of unexpanded nodes is sometimes called the open list.) If the current node matches a node on the closed list, it is discarded instead of being expanded. The new algorithm is called GRAPH-SEARCH (Figure 3.19). On problems with many repeated states, GRAPH-SEARCH is much more efficient than TREE-SEARCH. Its worst-case time and space requirements are proportional to the size of the state space. This may be much smaller than O(b%).

Optimality for graph search is a tricky issue. We said earlier that when a repeated state is detected, the algorithm has found two paths to the same state. The GRAPH-SEARCH algorithm in Figure 3.19 always discards the newly discovered path; obviously, if the newly discovered path is shorter than the original one, GRAPH-SEARCH could miss an optimal solution. Fortunately, we can show (Exercise 3.12) that this cannot happen when using either

[]

(a) (b) (c)

Figure 3.18 State spaces that generate an exponentially larger search tree. (a) A state space in which there are two possible actions leading from A to B, two from B to C, and so on. The state space contains d + 1 states, where d is the maximum depth. (b) The corresponding search tree, which has 2¢ branches corresponding to the 2¢ paths through the space. (c) A rectangular grid space. States within 2 steps of the initial state (A) are shown in gray.

Section 3.6.

Searching with Partial Information 83

function GRAPH-SEARCH( problem, fringe) returns a solution, or failure

closed «+ an empty set fringe + INSERT(MAKE-NODE(INITIAL-STATE[ problem), fringe) loop do if EMPTY ?( fringe) then return failure node +— REMOVE-FIRST( fringe) if GOAL-TEST[ problem](STATE[node]) then return SOLUTION(node) if STATE[node] is not in closed then add STATE[node] to closed fringe « INSERT-ALL(EXPAND(node, problem), fringe)

Figure 3.19 The general graph-search algorithm. The set closed can be implemented with a hash table to allow efficient checking for repeated states. This algorithm assumes that the first path to a state s is the cheapest (see text).

uniform-cost search or breadth-first search with constant step costs; hence, these two optimal tree-search strategies are also optimal graph-search strategies. Iterative deepening search, on the other hand, uses depth-first expansion and can easily follow a suboptimal path to a node before finding the optimal one. Hence, iterative deepening graph search needs to check whether a newly discovered path to a node is better than the original one, and if so, it might need to revise the depths and path costs of that node’s descendants.

Note that the use of a closed list means that depth-first search and iterative deepening search no longer have linear space requirements. Because the GRAPH-SEARCH algorithm keeps every node in memory, some searches are infeasible because of memory limitations.

In Section 3.3 we assumed that the environment is fully observable and deterministic and that the agent knows what the effects of each action are. Therefore, the agent can calculate exactly which state results from any sequence of actions and always knows which state it is in. Its percepts provide no new information after each action. What happens when knowledge of the states or actions is incomplete? We find that different types of incompleteness lead to three distinct problem types:

1. Sensorless problems (also called conformant problems): If the agent has no sensors at all, then (as far as it knows) it could be in one of several possible initial states, and each action might therefore lead to one of several possible successor states.

2. Contingency problems: If the environment is partially observable or if actions are uncertain, then the agent’s percepts provide new information after each action. Each possible percept défines a contingency that must be planned for. A problem is called adversarial if the uncertainty is caused by the actions of another agent.

84

Chapter 3. Solving Problems by Searching

COERCION

BELIEF STATE

& 03s %

&

I) [FE] [B] FE

I [5] FE FE

Figure 3.20 The eight possible states of the vacuum world.

3. Exploration problems: When the states and actions of the environment are unknown, the agent must act to discover them. Exploration problems can be viewed as an extreme case of contingency problems.

As an example, we will use the vacuum world environment. Recall that the state space has eight states, as shown in Figure 3.20. There are three actions—Left, Right, and Suck—and the goal is to clean up all the dirt (states 7 and 8). If the environment is observable, deterministic, and completely known, then the problem is trivially solvable by any of the algorithms we have described. For example, if the initial state is 5, then the action sequence [Right Suck] will reach a goal state, 8. The remainder of this section deals with the sensorless and contingency versions of the problem. Exploration problems are covered in Section 4.5, adversarial problems in Chapter 6.

Sensorless problems

Suppose that the vacuum agent knows all the effects of its actions, but has no sensors. Then it knows only that its initial state is one of the set {1,2,3,4,5,6,7,8}. One might suppose that the agent’s predicament is hopeless, but in fact it can do quite well. Because it knows what its actions do, it can, for example, calculate that the action Right will cause it to be in one of the states {2, 4, 6,8}, and the action sequence [Right,Suck) will always end up in one of the states {4,8}. Finally, the sequence [Right,Suck,Left, Suck] is guaranteed to reach the goal state 7 no matter what the start state. We say that the agent can coerce the world into state 7, even when it doesn’t know where it started. To summarize: when the world is not fully observable, the agent must reason about sets of states that it might get to, rather than single states. We call each such set of states a belief state, representing the agent’s current belief about the possible physical states it might be in. (In a fully observable environment, each belief state contains one physical state.)

Section 3.6.

Searching with Partial Information 85

[]

Figure 3.21 The reachable portion of the belief state space for the deterministic, sensorless vacuum world. Each shaded box corresponds to a single belief state. At any given point, the agent is in a particular belief state but does not know which physical state it is in. The initial belief state (complete ignorance) is the top center box. Actions are represented by

labeled arcs. Self-loops are omitted for clarity.

To solve sensorless problems, we search in the space of belief states rather than physical states. The initial state is a belief state, and each action maps from a belief state to another belief state. An action is applied to a belief state by unioning the results of applying the action to each physical state in the belief state. A path now connects several belief states, and a solution is now a path that leads to a belief state, all of whose members are goal states. Figure 3.21 shows the reachable belief-state space for the deterministic, sensorless vacuum world. There are only 12 reachable belief states, but the entire belief state space contains every possible set of physical states, i.e., 28 = 256 belief states. In general, if the physical state space has S states, the belief state space has 25 belief states.

Our discussion of sensorless problems so far has assumed deterministic actions, but the analysis is essentially unchanged if the environment is nondeterministic—that is, if actions may have several possible outcomes. The reason is that, in the absence of sensors, the agent

86 Chapter 3. Solving Problems by Searching

has no way to tell which outcome actually occurred, so the various possible outcomes are just additional physical states in the successor belief state. For example, suppose the environment obeys Murphy’s Law: the so-called Suck action sometimes deposits dirt on the carpet but only if there is no dirt there already.® Then, if Suck is applied in physical state 4 (see Figure 3.20), there are two possible outcomes: states 2 and 4. Applied to the initial belief state, {1,2,3,4,5,6,7,8}, Suck now leads to the belief state that is the union of the outcome sets for the eight physical states. Calculating this, we find that the new belief state is {1,2,3,4,5,6,7,8}. So, for a sensorless agent in the Murphy’s Law world, the Suck action leaves the belief state unchanged! In fact, the problem is unsolvable. (See Exercise 3.18.) Intuitively, the reason is that the agent cannot tell whether the current square is dirty and hence cannot tell whether the Suck action will clean it up or create more dirt.

Contingency problems

When the environment is such that the agent can obtain new information from its sensors CONTINGENCY after acting, the agent faces a contingency problem. The solution to a contingency problem often takes the form of a tree, where each branch may be selected depending on the percepts received up to that point in the tree. For example, suppose that the agent is in the Murphy’s Law world and that it has a position sensor and a local dirt sensor, but no sensor capable of detecting dirt in other squares. Thus, the percept [L, Dirty] means that the agent is in one of the states {1, 3}. The agent might formulate the action sequence [Suck, Right, Suck]. Sucking would change the state to one of {5, 7}, and moving right would then change the state to one of {6,8}. Executing the final Suck action in state 6 takes us to state 8, a goal, but executing it in state 8 might take us back to state 6 (by Murphy’s Law), in which case the plan fails. By examining the belief-state space for this version of the problem, it can easily be determined that no fixed action sequence guarantees a solution to this problem. There is, however, a solution if we don’t insist on a fixed action sequence:

[Suck, Right if [R,Dirty] then Suck] .

This extends the space of solutions to include the possibility of selecting actions based on contingencies arising during execution. Many problems in the real, physical world are contingency problems, because exact prediction is impossible. For this reason, many people keep their eyes open while walking around or driving.

Contingency problems sometimes allow purely sequential solutions. For example, consider a fully observable Murphy's Law world. Contingencies arise if the agent performs a Suck action in a clean square, because dirt might or might not be deposited in the square. As long as the agent never does this, no contingencies arise and there is a sequential solution from every initial state (Exercise 3.18).

The algorithms for contingency problems are more complex than the standard search algorithms in this chapter; they are covered in Chapter 12. Contingency problems also lend themselves to a somewhat different agent design, in which the agent can act before it has found a guaranteed plan. This is useful because rather than considering in advance every

6 We assume that most readers face similar problems and can sympathize with our agent. We apologize to owners of modern, efficient home appliances who cannot take advantage of this pedagogical device.

Section 3.7.

INTERLEAVING

the goal to formulate a problem.

A problem consists of four parts: the initial state, a set of actions, a goal test function, and a path cost function. The environment of the problem is represented by a state space. A path through the state space from the initial state to a goal state is a solution. A single, general TREE-SEARCH algorithm can be used to solve any problem; specific variants of the algorithm embody different strategies.

Search algorithms are judged on the basis of completeness, optimality, time complexity, and space complexity. Complexity depends on b, the branching factor in the state space, and d, the depth of the shallowest solution.

Breadth-first search selects the shallowest unexpanded node in the search tree for expansion. It is complete, optimal for unit step costs, and has time and space complexity of O(b%). The space complexity makes it impractical in most cases. Uniform-cost search is similar to breadth-first search but expands the node with lowest path cost, g(n). It is complete and optimal if the cost of each step exceeds some positive bound e. Depth-first search selects the deepest unexpanded node in the search tree for expansion. It is neither complete nor optimal, and has time complexity of O(b™) and space complexity of O(bm), where m is the maximum depth of any path in the state space. Depth-limited search imposes a fixed depth limit on a depth-first search.

Iterative deepening search calls depth-limited search with increasing limits until a goal is found. It is complete, optimal for unit step costs, and has time complexity of O(b?%) and space complexity of O(bd).

Bidirectional search can enormously reduce time complexity, but it is not always applicable and may require too much space.

When the state space is a graph rather than a tree, it can pay off to check for repeated states in the search tree. The GRAPHSEARCH algorithm eliminates all duplicate states. When the environment is partially observable, the agent can apply search algorithms in the space of belief states, or sets of possible states that the agent might be in. In some cases, a single solution sequence can be constructed; in other cases, the agent needs a contingency plan to handle unknown circumstances that may arise.

88 Chapter 3. Solving Problems by Searching

BIBLIOGRAPHICAL AND HISTORICAL NOTES

Most of the state-space search problems analyzed in this chapter have a long history in the literature and are less trivial than they might seem. The missionaries and cannibals problem used in Exercise 3.9 was analyzed in detail by Amarel (1968). It had been considered earlier in AI by Simon and Newell (1961), and in operations research by Bellman and Dreyfus (1962). Studies such as these and Newell and Simon’s work on the Logic Theorist (1957) and GPS (1961) led to the establishment of search algorithms as the primary weapons in the armory of 1960s Al researchers and to the establishment of problem solving as the canonical Al task. Unfortunately, very little work was done on the automation of the problem formulation step. A more recent treatment of problem representation and abstraction, including Al programs that themselves perform these tasks (in part), is in Knoblock (1990).

The 8-puzzle is a smaller cousin of the 15-puzzle, which was invented by the famous American game designer Sam Loyd (1959) in the 1870s. The 15-puzzle quickly achieved immense popularity in the United States, comparable to the more recent sensation caused by Rubik’s Cube. It also quickly attracted the attention of mathematicians (Johnson and Story, 1879; Tait, 1880). The editors of the American Journal of Mathematics stated “The ‘15° puzzle for the last few weeks has been prominently before the American public, and may safely be said to have engaged the attention of nine out of ten persons of both sexes and all ages and conditions of the community. But this would not have weighed with the editors to induce them to insert articles upon such a subject in the American Journal of Mathematics, but for the fact that . . .” (there follows a summary of the mathematical interest of the 15-puzzle). An exhaustive analysis of the 8-puzzle was carried out with computer aid by Schofield (1967). Ratner and Warmuth (1986) showed that the general nn x n version of the 15-puzzle belongs to the class of NP-complete problems.

The 8-queens problem was first published anonymously in the German chess magazine Schach in 1848; it was later attributed to one Max Bezzel. It was republished in 1850 and at that time drew the attention of the eminent mathematician Carl Friedrich Gauss, who attempted to enumerate all possible solutions, but found only 72. Nauck published all 92 solutions later in 1850. Netto (1901) generalized the problem to n queens, and Abramson and Yung (1989) found an O(n) algorithm.

Each of the real-world search problems listed in the chapter has been the subject of a good deal of research effort. Methods for selecting optimal airline flights remain proprietary for the most part, but Carl de Marcken (personal communication) has shown that airline ticket pricing and restrictions have become so convoluted that the problem of selecting an optimal flight is formally undecidable. The traveling-salesperson problem is a standard combinatorial problem in theoretical computer science (Lawler, 1985; Lawler ef al., 1992). Karp (1972) proved the TSP to be NP-hard, but effective heuristic approximation methods were developed (Lin and Kernighan, 1973). Arora (1998) devised a fully polynomial approximation scheme for Euclidean TSPs. VLSI layout methods are surveyed by Shahookar and Mazumder (1991), and many layout optimization papers appear in VLSI journals. Robotic navigation and assembly problems are discussed in Chapter 25.

Section 3.7.

Summary 89

Uninformed search algorithms for problem solving are a central topic of classical computer science (Horowitz and Sahni, 1978) and operations research (Dreyfus, 1969); Deo and Pang (1984) and Gallo and Pallottino (1988) give more recent surveys. Breadth-first search was formulated for solving mazes by Moore (1959). The method of dynamic programming (Bellman and Dreyfus, 1962), which systematically records solutions for all subproblems of increasing lengths, can be seen as a form of breadth-first search on graphs. The two-point shortest-path algorithm of Dijkstra (1959) is the origin of uniform-cost search.

A version of iterative deepening designed to make efficient use of the chess clock was first used by Slate and Atkin (1977) in the CHESS 4.5 game-playing program, but the application to shortest path graph search is due to Korf (1985a). Bidirectional search, which was introduced by Pohl (1969, 1971), can also be very effective in some cases.

Partially observable and nondeterministic environments have not been studied in great depth within the problem-solving approach. Some efficiency issues in belief-state search have been investigated by Genesereth and Nourbakhsh (1993). Koenig and Simmons (1998) studied robot navigation from an unknown initial position, and Erdmann and Mason (1988) studied the problem of robotic manipulation without sensors, using a continuous form of belief-state search. Contingency search has been studied within the planning subfield. (See Chapter 12.) For the most part, planning and acting with uncertain information have been handled using the tools of probability and decision theory (see Chapter 17).

The textbooks by Nilsson (1971, 1980) are good general sources of information about classical search algorithms. A comprehensive and more up-to-date survey can be found in Korf (1988). Papers about new search algorithms—which, remarkably, continue to be discovered—appear in journals such as Artificial Intelligence.

EXERCISES

3.1 Define in your own words the following terms: state, state space, search tree, search node, goal, action, successor function, and branching factor.

3.2 Explain why problem formulation must follow goal formulation.

3.3 Suppose that LEGAL-ACTIONS(s) denotes the set of actions that are legal in state s, and RESULT(a, s) denotes the state that results from performing a legal action a in state s. Define SUCCESSOR-FN in terms of LEGAL-ACTIONS and RESULT, and vice versa.

3.4 Show that the 8-puzzle states are divided into two disjoint sets, such that no state in one set can be transformed into a state in the other set by any number of moves. (Hint: See Berlekamp et al. (1982).) Devise a procedure that will tell you which class a given state is in, and explain why this is a good thing to have for generating random states.

3.5 Consider the n-queens problem using the “efficient” incremental formulation given on page 67. Explain why the state space size is at least /n! and estimate the largest n for which exhaustive exploration is feasible. (Hint: Derive a lower bound on the branching factor by considering the maximum number of squares that a queen can attack in any column.)

90 Chapter 3. Solving Problems by Searching

3.6 Does a finite state space always lead to a finite search tree? How about a finite state space that is a tree? Can you be more precise about what types of state spaces always lead to finite search trees? (Adapted from Bender, 1996.)

3.7 Give the initial state, goal test, successor function, and cost function for each of the following. Choose a formulation that is precise enough to be implemented.

a. You have to color a planar map using only four colors, in such a way that no two adjacent regions have the same color.

b. A 3-foot-tall monkey is in a room where some bananas are suspended from the 8-foot ceiling. He would like to get the bananas. The room contains two stackable, movable, climbable 3-foot-high crates.

¢. You have a program that outputs the message “illegal input record” when fed a certain file of input records. You know that processing of each record is independent of the other records. You want to discover what record is illegal.

d. You have three jugs, measuring 12 gallons, 8 gallons, and 3 gallons, and a water faucet. You can fill the jugs up or empty them out from one to another or onto the ground. You need to measure out exactly one gallon.

3.8 Consider a state space where the start state is number 1 and the successor function for state n returns two states, numbers 2n and 2n + 1.

a. Draw the portion of the state space for states 1 to 15.

b. Suppose the goal state is 11. List the order in which nodes will be visited for breadthfirst search, depth-limited search with limit 3, and iterative deepening search.

¢. Would bidirectional search be appropriate for this problem? If so, describe in detail how it would work.

d. What is the branching factor in each direction of the bidirectional search?

e. Does the answer to (¢) suggest a reformulation of the problem that would allow you to solve the problem of getting from state 1 to a given goal state with almost no search?

3.9 The missionaries and cannibals problem is usually stated as follows. Three missionaries and three cannibals are on one side of a river, along with a boat that can hold one or two people. Find a way to get everyone to the other side, without ever leaving a group of missionaries in one place outnumbered by the cannibals in that place. This problem is famous in Al because it was the subject of the first paper that approached problem formulation from an analytical viewpoint (Amarel, 1968).

a. Formulate the problem precisely, making only those distinctions necessary to ensure a valid solution. Draw a diagram of the complete state space.

b. Implement and solve the problem optimally using an appropriate search algorithm. Is it a good idea to check for repeated states?

¢. Why do you think people have a hard time solving this puzzle, given that the state space is so simple?

Section 3.7.

Summary 91

3.10 Implement two versions of the successor function for the 8-puzzle: one that generates all the successors at once by copying and editing the 8-puzzle data structure, and one that generates one new successor each time it is called and works by modifying the parent state directly (and undoing the modifications as needed). Write versions of iterative deepening depth-first search that use these functions and compare their performance.

3.11 On page 79, we mentioned iterative lengthening search, an iterative analog of uniform cost search. The idea is to use increasing limits on path cost. If a node is generated whose path cost exceeds the current limit, it is immediately discarded. For each new iteration, the limit is set to the lowest path cost of any node discarded in the previous iteration.

a. Show that this algorithm is optimal for general path costs.

b. Consider a uniform tree with branching factor b, solution depth d, and unit step costs. How many iterations will iterative lengthening require?

¢. Now consider step costs drawn from the continuous range [0, 1] with a minimum positive cost e. How many iterations are required in the worst case?

d. Implement the algorithm and apply it to instances of the 8-puzzle and traveling salesperson problems. Compare the algorithm’s performance to that of uniform-cost search, and comment on your results.

3.12 Prove that uniform-cost search and breadth-first search with constant step costs are optimal when used with the GRAPH-SEARCH algorithm. Show a state space with constant step costs in which GRAPHSEARCH using iterative deepening finds a suboptimal solution.

3.13 Describe a state space in which iterative deepening search performs much worse than depth-first search (for example, O(n?) vs. O(n)).

3.14 Write a program that will take as input two Web page URLs and find a path of links from one to the other. What is an appropriate search strategy? Is bidirectional search a good idea? Could a search engine be used to implement a predecessor function?

3.15 Consider the problem of finding the shortest path between two points on a plane that has convex polygonal obstacles as shown in Figure 3.22. This is an idealization of the problem that a robot has to solve to navigate its way around a crowded environment.

a. Suppose the state space consists of all positions (x,y) in the plane. How many states are there? How many paths are there to the goal?

b. Explain briefly why the shortest path from one polygon vertex to any other in the scene must consist of straight-line segments joining some of the vertices of the polygons. Define a good state space now. How large is this state space?

¢. Define the necessary functions to implement the search problem, including a successor function that takes a vertex as input and returns the set of vertices that can be reached in a straight line from the given vertex. (Do not forget the neighbors on the same polygon.) Use the straight-line distance for the heuristic function.

d. Apply one or more of the algorithms in this chapter to solve a range of problems in the domain, and comment on their performance.

92

Chapter 3. Solving Problems by Searching

eG

Se.

Figure 3.22 A scene with polygonal obstacles.

3.16

We can turn the navigation problem in Exercise 3.15 into an environment as follows:

The percept will be a list of the positions, relative to the agent, of the visible vertices. The percept does not include the position of the robot! The robot must learn its own position from the map; for now, you can assume that each location has a different “view.” Each action will be a vector describing a straight-line path to follow. If the path is unobstructed, the action succeeds; otherwise, the robot stops at the point where its path first intersects an obstacle. If the agent returns a zero motion vector and is at the goal (which is fixed and known), then the environment should teleport the agent to a random location (not inside an obstacle).

The performance measure charges the agent 1 point for each unit of distance traversed and awards 1000 points each time the goal is reached.

Implement this environment and a problem-solving agent for it. The agent will need to formulate a new problem after each teleportation, which will involve discovering its current location.

as it moves around) and report its performance over 100 episodes.

Modify the environment so that 30% of the time the agent ends up at an unintended destination (chosen randomly from the other visible vertices if any, otherwise no move at all). This is a crude model of the motion errors of a real robot. Modify the agent so that when such an error is detected, it finds out where it is and then constructs a plan to get back to where it was and resume the old plan. Remember that sometimes getting back to where it was might also fail! Show an example of the agent successfully overcoming two successive motion errors and still reaching the goal.

Now try two different recovery schemes after an error: (1) Head for the closest vertex on the original route; and (2) replan a route to the goal from the new location. Compare the performance of the three recovery schemes. Would the inclusion of search costs affect the comparison?

Now suppose that there are locations from which the view is identical. (For example, suppose the world is a grid with square obstacles.) What kind of problem does the agent now face? What do solutions look like?

Suppose that actions can have arbitrarily large negative costs; explain why this possibility would force any optimal algorithm to explore the entire state space.

Does it help if we insist that step costs must be greater than or equal to some negative constant ¢? Consider both trees and graphs.

order results in no net change to the state. If all of these operators have negative cost, what does this imply about the optimal behavior for an agent in such an environment?

route finding. For example, some stretches of road might have such beautiful scenery as to far outweigh the normal costs in terms of time and fuel. Explain, in precise terms, within the context of state-space search, why humans do not drive round scenic loops indefinitely, and explain how to define the state space and operators for route finding so that artificial agents can also avoid looping.

Can you think of a real domain in which step costs are such as to cause looping?

Which of the algorithms defined in this chapter would be appropriate for this problem? Should the algorithm check for repeated states?

Apply your chosen algorithm to compute an optimal sequence of actions for a 3 x 3 world whose initial state has dirt in the three top squares and the agent in the center. Construct a search agent for the vacuum world, and evaluate its performance in a set of 3 x 3 worlds with probability 0.2 of dirt in each square. Include the search cost as well as path cost in the performance measure, using a reasonable exchange rate.

Compare your best search agent with a simple randomized reflex agent that sucks if there is dirt and otherwise moves randomly.

Consider what would happen if the world were enlarged to n x n. How does the performance of the search agent and of the reflex agent vary with n?

INFORMED SEARCH AND EXPLORATION

In which we see how information about the state space can prevent algorithms from blundering about in the dark.

Chapter 3 showed that uninformed search strategies can find solutions to problems by systematically generating new states and testing them against the goal. Unfortunately, these strategies are incredibly inefficient in most cases. This chapter shows how an informed search strategy—one that uses problem-specific knowledge—can find solutions more efficiently. Section 4.1 describes informed versions of the algorithms in Chapter 3, and Section 4.2 explains how the necessary problem-specific information can be obtained. Sections 4.3 and 4.4 cover algorithms that perform purely local search in the state space, evaluating and modifying one or more current states rather than systematically exploring paths from an initial state. These algorithms are suitable for problems in which the path cost is irrelevant and all that matters is the solution state itself. The family of local-search algorithms includes methods inspired by statistical physics (simulated annealing) and evolutionary biology (genetic algorithms). Finally, Section 4.5 investigates online search, in which the agent is faced with a state space that is completely unknown.

This section shows how an informed search strategy—one that uses problem-specific knowledge beyond the definition of the problem itself—can find solutions more efficiently than an uninformed strategy.

The general approach we will consider is called best-first search. Best-first search is an instance of the general TREE-SEARCH or GRAPH-SEARCH algorithm in which a node is selected for expansion based on an evaluation function, f(n). Traditionally, the node with the lowest evaluation is selected for expansion, because the evaluation measures distance to the goal. Best-first search can be implemented within our general search framework via a priority queue, a data structure that will maintain the fringe in ascending order of f-values.

The name “‘best-first search” is a venerable but inaccurate one. After all, if we could really expand the best node first, it would not be a search at all; it would be a straight march to

94

Section 4.1. Informed (Heuristic) Search Strategies 95

the goal. All we can do is choose the node that appears to be best according to the evaluation function. If the evaluation function is exactly accurate, then this will indeed be the best node; in reality, the evaluation function will sometimes be off, and can lead the search astray. Nevertheless, we will stick with the name “best-first search,” because “seemingly-best-first search” is a little awkward. There is a whole family of BEST-FIRST-SEARCH algorithms with different evaluation HEURISTIC functions.! A key component of these algorithms is a heuristic function, denoted h(n):

h(n) = estimated cost of the cheapest path from node 7 to a goal node.

For example, in Romania, one might estimate the cost of the cheapest path from Arad to Bucharest via the straight-line distance from Arad to Bucharest.

Heuristic functions are the most common form in which additional knowledge of the problem is imparted to the search algorithm. We will study heuristics in more depth in Section 4.2. For now, we will consider them to be arbitrary problem-specific functions, with one constraint: if n is a goal node, then h(n) = 0. The remainder of this section covers two ways to use heuristic information to guide search.

Greedy best-first search

GREEDVBESTFRST (Greedy best-first search? tries to expand the node that is closest to the goal, on the grounds that this is likely to lead to a solution quickly. Thus, it evaluates nodes by using just the heuristic function: f(n) = h(n). Let us see how this works for route-finding problems in Romania, using the straightSIRAIGHTLINE line distance heuristic, which we will call hgzp. If the goal is Bucharest, we will need to know the straight-line distances to Bucharest, which are shown in Figure 4.1. For example, hspp(In(Arad)) = 366. Notice that the values of hg p cannot be computed from the problem description itself. Moreover, it takes a certain amount of experience to know that hgzp is correlated with actual road distances and is, therefore, a useful heuristic.

[]

1 Exercise 4.3 asks you to show that this family includes several familiar uninformed algorithms.

2 A heuristic function h(n) takes a node as input, but it depends only on the state at that node.

3 Our first edition called this greedy search; other authors have called it best-first search. Our more general usage of the latter term follows Pearl (1984).

96

Chapter 4. Informed Search and Exploration

(a) The initial state

(b) After expanding Arad

253 329 374

(c) After expanding Sibiu

(d) After expanding Fagaras

0

Figure 4.2 Stages in a greedy best-first search for Bucharest using the straight-line distance heuristic hgz,p. Nodes are labeled with their A-values.

Figure 4.2 shows the progress of a greedy best-first search using hszp to find a path from Arad to Bucharest. The first node to be expanded from Arad will be Sibiu, because it is closer to Bucharest than either Zerind or Timisoara. The next node to be expanded will be Fagaras, because it is closest. Fagaras in turn generates Bucharest, which is the goal. For this particular problem, greedy best-first search using hg p finds a solution without ever expanding a node that is not on the solution path; hence, its search cost is minimal. It is not optimal, however: the path via Sibiu and Fagaras to Bucharest is 32 kilometers longer than the path through Rimnicu Vilcea and Pitesti. This shows why the algorithm is called “greedy”—at each step it tries to get as close to the goal as it can.

Minimizing h(n) is susceptible to false starts, Consider the problem of getting from lasi to Fagaras. The heuristic suggests that Neamt be expanded first, because it is closest

Section 4.1.

Informed (Heuristic) Search Strategies 97

A" SEARCH

ADMISSIBLE HEURISTIC

to Fagaras, but it is a dead end. The solution is to go first to Vaslui—a step that is actually farther from the goal according to the heuristic—and then to continue to Urziceni, Bucharest, and Fagaras. In this case, then, the heuristic causes unnecessary nodes to be expanded. Furthermore, if we are not careful to detect repeated states, the solution will never be found—the search will oscillate between Neamt and Iasi.

Greedy best-first search resembles depth-first search in the way it prefers to follow a single path all the way to the goal, but will back up when it hits a dead end. It suffers from the same defects as depth-first search—it is not optimal, and it is incomplete (because it can start down an infinite path and never return to try other possibilities). The worst-case time and space complexity is O(b™), where m is the maximum depth of the search space. With a good heuristic function, however, the complexity can be reduced substantially. The amount of the reduction depends on the particular problem and on the quality of the heuristic.

A\* search: Minimizing the total estimated solution cost

The most widely-known form of best-first search is called A\* search (pronounced “A-star search”). It evaluates nodes by combining g(n), the cost to reach the node, and h(n), the cost to get from the node to the goal:

fn) = g(n) + h(n) . Since g(n) gives the path cost from the start node to node n, and h(n) is the estimated cost of the cheapest path from 7 to the goal, we have

f(n) = estimated cost of the cheapest solution through 7 .

Thus, if we are trying to find the cheapest solution, a reasonable thing to try first is the node with the lowest value of g(n) + h(n). It turns out that this strategy is more than just reasonable: provided that the heuristic function h(n) satisfies certain conditions, A\* search is both complete and optimal.

The optimality of A" is straightforward to analyze if it is used with TREE-SEARCH. In this case, A\* is optimal if h(n) is an admissible heuristic—that is, provided that h(n) never overestimates the cost to reach the goal. Admissible heuristics are by nature optimistic, because they think the cost of solving the problem is less than it actually is. Since g(n) is the exact cost to reach n, we have as immediate consequence that f(n) never overestimates the true cost of a solution through n.

An obvious example of an admissible heuristic is the straight-line distance hgyp that we used in getting to Bucharest. Straight-line distance is admissible because the shortest path between any two points is a straight line, so the straight line cannot be an overestimate. In Figure 4.3, we show the progress of an A\* tree search for Bucharest. The values of ¢ are computed from the step costs in Figure 3.2, and the values of hg; p are given in Figure 4.1. Notice in particular that Bucharest first appears on the fringe at step (e), but it is not selected for expansion because its f-cost (450) is higher than that of Pitesti (417). Another way to say this is that there might be a solution through Pitesti whose cost is as low as 417, so the algorithm will not settle for a solution that costs 450. From this example, we can extract a general proof that A\* using TREE-SEARCH is optimal if h(n) is admissible. Suppose a

98

(a) The initial state 366=0+366 (b) After expanding Arad

393=140+253 447=118+329 449=75+374

(c) After expanding Sibiu

447=118+329

449=75+374

646=280+366 415=239+176 671=291+380 413=220+193

(d) After expanding Rimnicu Vilcea

449=75+374

526=366+160 417=317+100 553=300+253

(e) After expanding Fagaras

449=75+374

591=338+253 450=450+0 526=366+160 417=317+100 553=300+253

(f) After expanding Pitesti

449=75+374

C Craiova)

418=418+0 615=455+160 607=414+193

Figure 4.3 Stages in an A" search for Bucharest. Nodes are labeled with f = ¢ + h. The h values are the straight-line distances to Bucharest taken from Figure 4.1.

Section 4.1.

Informed (Heuristic) Search Strategies 99

CONSISTENCY MONOTONICITY

TRIANGLE INEQUALITY

suboptimal goal node G2 appears on the fringe, and let the cost of the optimal solution be C\*. Then, because G7 is suboptimal and because h(G2) = 0 (true for any goal node), we know

f(G2) = g(Ga) + h(Ga) = g(Ga) > C™. Now consider a fringe node 7 that is on an optimal solution path—for example, Pitesti in the

example of the preceding paragraph. (There must always be such a node if a solution exists.) If h(n) does not overestimate the cost of completing the solution path, then we know that

f(n) = g(n) + h(n)

Now we have shown that f(n) < C\* < f(G2), so G2 will not be expanded and A\* must return an optimal solution.

If we use the GRAPH-SEARCH algorithm of Figure 3.19 instead of TREE-SEARCH, then this proof breaks down. Suboptimal solutions can be returned because GRAPH-SEARCH can discard the optimal path to a repeated state if it is not the first one generated. (See Exercise 4.4.) There are two ways to fix this problem. The first solution is to extend GRAPHSEARCH so that it discards the more expensive of any two paths found to the same node. (See the discussion in Section 3.5.) The extra bookkeeping is messy, but it does guarantee optimality. The second solution is to ensure that the optimal path to any repeated state is always the first one followed—as is the case with uniform-cost search. This property holds if we impose an extra requirement on h(n), namely the requirement of consistency (also called monotonicity). A heuristic h(n) is consistent if, for every node n and every successor n’ of n generated by any action a, the estimated cost of reaching the goal from n is no greater than the step cost of getting to n’ plus the estimated cost of reaching the goal from n/':

h(n) < c(n,a,n’) + h(n) .

This is a form of the general triangle inequality, which stipulates that each side of a triangle cannot be longer than the sum of the other two sides. Here, the triangle is formed by n, n/, and the goal closest to n. It is fairly easy to show (Exercise 4.7) that every consistent heuristic is also admissible. The most important consequence of consistency is the following: A\* using GRAPH-SEARCH is optimal if h(n) is consistent.

Although consistency is a stricter requirement than admissibility, one has to work quite hard to concoct heuristics that are admissible but not consistent. All the admissible heuristics we discuss in this chapter are also consistent. Consider, for example, hsyp. We know that the general triangle inequality is satisfied when each side is measured by the straight-line distance, and that the straight-line distance between n and n’ is no greater than ¢(n, a, n’). Hence, hgrp is a consistent heuristic.

Another important consequence of consistency is the following: If h(n) is consistent, then the values of f(n) along any path are nondecreasing. The proof follows directly from the definition of consistency. Suppose n’ is a successor of n; then g(n’) = g(n) + c(n,a,n’) for some a, and we have

fn) = gn) + h(n) = g(n) + c(n,a,n') + h(n’) > g(n) + h(n) = f(n). It follows that the sequence of nodes expanded by A" using GRAPH-SEARCH is in nonde

creasing order of f(n). Hence, the first goal node selected for expansion must be an optimal solution, since all later nodes will be at least as expensive.

100

Chapter 4. Informed Search and Exploration

CONTOURS

PRUNING

Figure 4.4 Map of Romania showing contours at f = 380, f = 400 and f = 420, with Arad as the start state. Nodes inside a given contour have f-costs less than or equal to the contour value.

The fact that f-costs are nondecreasing along any path also means that we can draw contours in the state space, just like the contours in a topographic map. Figure 4.4 shows an example. Inside the contour labeled 400, all nodes have f(n) less than or equal to 400, and so on. Then, because A\* expands the fringe node of lowest f-cost, we can see that an A\* search fans out from the start node, adding nodes in concentric bands of increasing f-cost.

With uniform-cost search (A\* search using h(n) = 0), the bands will be “circular” around the start state. With more accurate heuristics, the bands will stretch toward the goal state and become more narrowly focused around the optimal path. If C\* is the cost of the optimal solution path, then we can say the following:

e A\* expands all nodes with f(n) < C\*.

e A\* might then expand some of the nodes right on the “goal contour” (where f(n) = Cx) before selecting a goal node.

Intuitively, it is obvious that the first solution found must be an optimal one, because goal nodes in all subsequent contours will have higher f-cost, and thus higher g-cost (because all goal nodes have h(n) = 0). Intuitively, it is also obvious that A\* search is complete. As we add bands of increasing f, we must eventually reach a band where f is equal to the cost of the path to a goal state.\*

Notice that A\* expands no nodes with f(n) > C\*—for example, Timisoara is not expanded in Figure 4.3 even though it is a child of the root. We say that the subtree below Timisoara is pruned; because hgzp is admissible, the algorithm can safely ignore this subtree

4 Completeness requires that there be only finitely many nodes with cost less than or equal to C\*, a condition that is true if all step costs exceed some finite ¢ and if b is finite.

Section 4.1.

Informed (Heuristic) Search Strategies 101

OPTIMALLY EFFICIENT

RECURSIVE BEST-FIRST SEARCH

while still guaranteeing optimality. The concept of pruning—eliminating possibilities from consideration without having to examine them—is important for many areas of Al.

One final observation is that among optimal algorithms of this type—algorithms that extend search paths from the root—A\* is optimally efficient for any given heuristic function. That is, no other optimal algorithm is guaranteed to expand fewer nodes than A\* (except possibly through tie-breaking among nodes with f(n) = C\*). This is because any algorithm that does not expand all nodes with f(n) < C\* runs the risk of missing the optimal solution.

That A\* search is complete, optimal, and optimally efficient among all such algorithms is rather satisfying. Unfortunately, it does not mean that A\* is the answer to all our searching needs. The catch is that, for most problems, the number of nodes within the goal contour search space is still exponential in the length of the solution. Although the proof of the result is beyond the scope of this book, it has been shown that exponential growth will occur unless the error in the heuristic function grows no faster than the logarithm of the actual path cost. In mathematical notation, the condition for subexponential growth is that

|i(n) = h\*(n)| < O(log h™(n)) ,

where h\*(n) is the true cost of getting from n to the goal. For almost all heuristics in practical use, the error is at least proportional to the path cost, and the resulting exponential growth eventually overtakes any computer. For this reason, it is often impractical to insist on finding an optimal solution. One can use variants of A\* that find suboptimal solutions quickly, or one can sometimes design heuristics that are more accurate, but not strictly admissible. In any case, the use of a good heuristic still provides enormous savings compared to the use of an uninformed search. In Section 4.2, we will look at the question of designing good heuristics. Computation time is not, however, A\*’s main drawback. Because it keeps all generated nodes in memory (as do all GRAPH-SEARCH algorithms), A\* usually runs out of space long before it runs out of time. For this reason, A\* is not practical for many large-scale problems. Recently developed algorithms have overcome the space problem without sacrificing optimality or completeness, at a small cost in execution time. These are discussed next.

Memory-bounded heuristic search

The simplest way to reduce memory requirements for A\* is to adapt the idea of iterative deepening to the heuristic search context, resulting in the iterative-deepening A\* (IDA\*) algorithm. The main difference between IDA" and standard iterative deepening is that the cutoff used is the f-cost (¢ + h) rather than the depth; at each iteration, the cutoff value is the smallest f-cost of any node that exceeded the cutoff on the previous iteration. IDA\* is practical for many problems with unit step costs and avoids the substantial overhead associated with keeping a sorted queue of nodes. Unfortunately, it suffers from the same difficulties with realvalued costs as does the iterative version of uniform-cost search described in Exercise 3.11. This section briefly examines two more recent memory-bounded algorithms, called RBFS and MA",

Recursive best-first search (RBFS) is a simple recursive algorithm that attempts to mimic the operation of standard best-first search, but using only linear space. The algorithm is shown in Figure 4.5. Its structure is similar to that of a recursive depth-first search, but rather

102

Chapter 4. Informed Search and Exploration

function RECURSIVE-BEST-FIRST-SEARCH( problem) returns a solution, or failure RBFS( problem, MAKE-NODE(INITIAL-STATE[ problem]), oo)

function RBFS( problem, node, f\_limit) returns a solution, or failure and a new f-cost limit if GOAL-TEST[ problem](STATE[node]) then return node successors — EXPAND(node, problem) if successors is empty then return failure, co for each s in successors do FIs] max(g(s) + h(s), f[node]) repeat best — the lowest f-value node in successors if f[best] > f-limit then return failure, f[best] alternative — the second-lowest f-value among successors result, flbest] «— RBES( problem, best, min( f\_limit, alternative)) if result # failure then return result

Figure 4.5 The algorithm for recursive best-first search.

than continuing indefinitely down the current path, it keeps track of the f-value of the best alternative path available from any ancestor of the current node. If the current node exceeds this limit, the recursion unwinds back to the alternative path. As the recursion unwinds, RBFS replaces the f-value of each node along the path with the best f-value of its children. In this way, RBFS remembers the f-value of the best leaf in the forgotten subtree and can therefore decide whether it’s worth reexpanding the subtree at some later time. Figure 4.6 shows how RBFS reaches Bucharest.

RBFS is somewhat more efficient than IDA\*, but still suffers from excessive node regeneration. In the example in Figure 4.6, RBFS first follows the path via Rimnicu Vilcea, then “changes its mind” and tries Fagaras, and then changes its mind back again. These mind changes occur because every time the current best path is extended, there is a good chance that its f-value will increase—\* is usually less optimistic for nodes closer to the goal. When this happens, particularly in large search spaces, the second-best path might become the best path, so the search has to backtrack to follow it. Each mind change corresponds to an iteration of IDA\*, and could require many reexpansions of forgotten nodes to recreate the best path and extend it one more node.

Like A\*, RBFS is an optimal algorithm if the heuristic function h(n) is admissible. Its space complexity is O(bd), but its time complexity is rather difficult to characterize: it depends both on the accuracy of the heuristic function and on how often the best path changes as nodes are expanded. Both IDA\* and RBFS are subject to the potentially exponential increase in complexity associated with searching on graphs (see Section 3.5), because they cannot check for repeated states other than those on the current path. Thus, they may explore the same state many times.

IDA\* and RBFS suffer from using too little memory. Between iterations, IDA" retains only a single number: the current f-cost limit. RBFS retains more information in memory,

Section 4.1.

Informed (Heuristic) Search Strategies 103

[]

MA

Figure 4.6 Stages in an RBFS search for the shortest route to Bucharest. The f-limit value for each recursive call is shown on top of each current node. (a) The path via Rimnicu Vilcea is followed until the current best leaf (Pitesti) has a value that is worse than the best alternative path (Fagaras). (b) The recursion unwinds and the best leaf value of the forgotten subtree (417) is backed up to Rimnicu Vilcea; then Fagaras is expanded, revealing a best leaf value of 450. (c) The recursion unwinds and the best leaf value of the forgotten subtree (450) is backed up to Fagaras; then Rimnicu Vilcea is expanded. This time, because the best

alternative path (through Timisoara) costs at least 447, the expansion continues to Bucharest.

but it uses only O(bd) memory: even if more memory were available, RBFS has no way to make use of it.

It seems sensible, therefore, to use all available memory. Two algorithms that do this are MA\* (memory-bounded A\*) and SMA\* (simplified MA\*). We will describe SMA\*, which

104

Chapter 4. Informed Search and Exploration

METALEVEL STATE SPACE

OBJECT-LEVEL STATE SPACE

is—well—simpler. SMA\* proceeds just like A", expanding the best leaf until memory is full. At this point, it cannot add a new node to the search tree without dropping an old one. SMA\* always drops the worst leaf node—the one with the highest f-value. Like RBFS, SMA\* then backs up the value of the forgotten node to its parent. In this way, the ancestor of a forgotten subtree knows the quality of the best path in that subtree. With this information, SMA" regenerates the subtree only when all other paths have been shown to look worse than the path it has forgotten. Another way of saying this is that, if all the descendants of a node n are forgotten, then we will not know which way to go from n, but we will still have an idea of how worthwhile it is to go anywhere from n.

The complete algorithm is too complicated to reproduce here, but there is one subtlety worth mentioning. We said that SMA\* expands the best leaf and deletes the worst leaf. What if all the leaf nodes have the same f-value? Then the algorithm might select the same node for deletion and expansion. SMA\* solves this problem by expanding the newest best leaf and deleting the oldest worst leaf. These can be the same node only if there is only one leaf; in that case, the current search tree must be a single path from root to leaf that fills all of memory. If the leaf is not a goal node, then even if it is on an optimal solution path, that solution is not reachable with the available memory. Therefore, the node can be discarded exactly as if it had no successors.

SMA\* is complete if there is any reachable solution—that is, if d, the depth of the shallowest goal node, is less than the memory size (expressed in nodes). It is optimal if any optimal solution is reachable; otherwise it returns the best reachable solution. In practical terms, SMA\* might well be the best general-purpose algorithm for finding optimal solutions, particularly when the state space is a graph, step costs are not uniform, and node generation is expensive compared to the additional overhead of maintaining the open and closed lists.

On very hard problems, however, it will often be the case that SMA\* is forced to switch back and forth continually between a set of candidate solution paths, only a small subset of which can fit in memory. (This resembles the problem of thrashing in disk paging systems.) Then the extra time required for repeated regeneration of the same nodes means that problems that would be practically solvable by A\*, given unlimited memory, become intractable for SMA\*. That is to say, memory limitations can make a problem intractable from the point of view of computation time. Although there is no theory to explain the tradeoff between time and memory, it seems that this is an inescapable problem. The only way out is to drop the optimality requirement.

Learning to search better

We have presented several fixed strategies—breadth-first, greedy best-first, and so on—that have been designed by computer scientists. Could an agent learn how to search better? The answer is yes, and the method rests on an important concept called the metalevel state space. Each state in a metalevel state space captures the internal (computational) state of a program that is searching in an object-level state space such as Romania. For example, the internal state of the A\* algorithm consists of the current search tree. Each action in the metalevel state

5 A rough sketch appeared in the first edition of this book.

Section 4.2.

Heuristic Functions 105

METALEVEL LEARNING

space is a computation step that alters the internal state; for example, each computation step in A\* expands a leaf node and adds its successors to the tree. Thus, Figure 4.3, which shows a sequence of larger and larger search trees, can be seen as depicting a path in the metalevel state space where each state on the path is an object-level search tree.

Now, the path in Figure 4.3 has five steps, including one step, the expansion of Fagaras, that is not especially helpful. For harder problems, there will be many such missteps, and a metalevel learning algorithm can learn from these experiences to avoid exploring unpromising subtrees. The techniques used for this kind of learning are described in Chapter 21. The goal of learning is to minimize the total cost of problem solving, trading off computational expense and path cost.

In this section, we will look at heuristics for the 8-puzzle, in order to shed light on the nature of heuristics in general.

The 8-puzzle was one of the earliest heuristic search problems. As mentioned in Section 3.2, the object of the puzzle is to slide the tiles horizontally or vertically into the empty space until the configuration matches the goal configuration (Figure 4.7).

[]

: 3 E 1 2 YR 3 il 4 5 6 || 7 8 Start State Goal State Figure 4.7 A typical instance of the 8-puzzle. The solution is 26 steps long.

The average solution cost for a randomly generated 8-puzzle instance is about 22 steps. The branching factor is about 3. (When the empty tile is in the middle, there are four possible moves; when it is in a corner there are two; and when it is along an edge there are three.) This means that an exhaustive search to depth 22 would look at about 322 ~ 3.1 x 10'° states. By keeping track of repeated states, we could cut this down by a factor of about 170,000, because there are only 9!/2 = 181, 440 distinct states that are reachable. (See Exercise 3.4.) This is a manageable number, but the corresponding number for the 15-puzzle is roughly 10'3, so the next order of business is to find a good heuristic function. If we want to find the shortest solutions by using A\*, we need a heuristic function that never overestimates the number of steps to the goal. There is a long history of such heuristics for the 15-puzzle; here are two commonly-used candidates:

106

Chapter 4. Informed Search and Exploration

MANHATTAN DISTANCE

EFFECTIVE BRANCHING FACTOR

DOMINATION

e h; = the number of misplaced tiles. For Figure 4.7, all of the eight tiles are out of position, so the start state would have hy = 8. hy is an admissible heuristic, because it is clear that any tile that is out of place must be moved at least once.

e ho = the sum of the distances of the tiles from their goal positions. Because tiles cannot move along diagonals, the distance we will count is the sum of the horizontal and vertical distances. This is sometimes called the city block distance or Manhattan distance. hs is also admissible, because all any move can do is move one tile one step closer to the goal. Tiles 1 to 8 in the start state give a Manhattan distance of

hg =3+1+24+2+2+34+3+2=18.

As we would hope, neither of these overestimates the true solution cost, which is 26.

The effect of heuristic accuracy on performance

One way to characterize the quality of a heuristic is the effective branching factor b\*. If the total number of nodes generated by A\* for a particular problem is /V, and the solution depth is d, then b\* is the branching factor that a uniform tree of depth d would have to have in order to contain N + 1 nodes. Thus,

NA+l=1+b"+ 02+ +09).

For example, if A\* finds a solution at depth 5 using 52 nodes, then the effective branching factor is 1.92. The effective branching factor can vary across problem instances, but usually it is fairly constant for sufficiently hard problems. Therefore, experimental measurements of b\* on a small set of problems can provide a good guide to the heuristic’s overall usefulness. A well-designed heuristic would have a value of b\* close to 1, allowing fairly large problems to be solved.

To test the heuristic functions h; and hy, we generated 1200 random problems with solution lengths from 2 to 24 (100 for each even number) and solved them with iterative deepening search and with A\* tree search using both h; and hs. Figure 4.8 gives the average number of nodes expanded by each strategy and the effective branching factor. The results suggest that ho is better than ka, and is far better than using iterative deepening search. On our solutions with length 14, A\* with hs is 30,000 times more efficient than uninformed iterative deepening search.

One might ask whether hy is always better than hi. The answer is yes. It is easy to see from the definitions of the two heuristics that, for any node n, ha(n) > hi(n). We thus say that ho dominates h;. Domination translates directly into efficiency: A\* using ho will never expand more nodes than A\* using h; (except possibly for some nodes with f(n)=C\*). The argument is simple. Recall the observation on page 100 that every node with f(n) < C\* will surely be expanded. This is the same as saying that every node with h(n) < C\* — g(n) will surely be expanded. But because hs» is at least as big as kh; for all nodes, every node that is surely expanded by A\* search with hg will also surely be expanded with hi, and h; might also cause other nodes to be expanded as well. Hence, it is always better to use a heuristic function with higher values, provided it does not overestimate and that the computation time for the heuristic is not too large.

[]

Section 4.2. Heuristic Functions 107 Search Cost Effective Branching Factor d IDS Ar(hy) A\*(hs) IDS A\*(hy) A\*(hs) 2 10 6 6 2.45 1.79 1.79 4 112 13 12 2.87 1.48 1.45 6 680 20 18 2.73 1.34 1.30 8 6384 39 25 2.80 1.33 1.24 10 47127 93 39 2.79 1.38 1.22 12 || 3644035 227 73 2.78 1.42 1.24 14 539 113 1.44 1.23 16 1301 211 1.45 1.25 18 3056 363 1.46 1.26 20 7276 676 1.47 1.27 22 18094 1219 1.48 1.28 24 39135 1641 1.48 1.26 Figure 4.8 Comparison of the search costs and effective branching factors for the ITERATIVE-DEEPENING-SEARCH and A\* algorithms with hj, ho. Data are averaged over 100 instances of the 8-puzzle, for various solution lengths.

RELAXED PROBLEM

Inventing admissible heuristic functions

We have seen that both i; (misplaced tiles) and ho (Manhattan distance) are fairly good heuristics for the 8-puzzle and that hs is better. How might one have come up with ho? Is it possible for a computer to invent such a heuristic mechanically?

hi and ho are estimates of the remaining path length for the 8-puzzle, but they are also perfectly accurate path lengths for simplified versions of the puzzle. If the rules of the puzzle were changed so that a tile could move anywhere, instead of just to the adjacent empty square, then h; would give the exact number of steps in the shortest solution. Similarly, if a tile could move one square in any direction, even onto an occupied square, then he would give the exact number of steps in the shortest solution. A problem with fewer restrictions on the actions is called a relaxed problem. The cost of an optimal solution to a relaxed problem is an admissible heuristic for the original problem. The heuristic is admissible because the optimal solution in the original problem is, by definition, also a solution in the relaxed problem and therefore must be at least as expensive as the optimal solution in the relaxed problem. Because the derived heuristic is an exact cost for the relaxed problem, it must obey the triangle inequality and is therefore consistent (see page 99).

If a problem definition is written down in a formal language, it is possible to construct relaxed problems automatically.® For example, if the 8-puzzle actions are described as

A tile can move from square A to square B if A is horizontally or vertically adjacent to B and B is blank,

6 In Chapters 8 and 11, we will describe formal languages suitable for this task; with formal descriptions that can be manipulated, the construction of relaxed problems can be automated. For now, we will use English.

108

Chapter 4. Informed Search and Exploration

SUBPROBLEM

PATTERN DATABASES

we can generate three relaxed problems by removing one or both of the conditions:

(a) A tile can move from square A to square B if A is adjacent to B. (b) A tile can move from square A to square B if B is blank. (c) A tile can move from square A to square B.

From (a), we can derive hy (Manhattan distance). The reasoning is that ho would be the proper score if we moved each tile in turn to its destination. The heuristic derived from (b) is discussed in Exercise 4.9. From (¢), we can derive h; (misplaced tiles), because it would be the proper score if tiles could move to their intended destination in one step. Notice that it is crucial that the relaxed problems generated by this technique can be solved essentially without search, because the relaxed rules allow the problem to be decomposed into eight independent subproblems. If the relaxed problem is hard to solve, then the values of the corresponding heuristic will be expensive to obtain.’

A program called ABSOLVER can generate heuristics automatically from problem definitions, using the “relaxed problem” method and various other techniques (Prieditis, 1993). ABSOLVER generated a new heuristic for the 8-puzzle better than any preexisting heuristic and found the first useful heuristic for the famous Rubik’s cube puzzle.

One problem with generating new heuristic functions is that one often fails to get one “clearly best” heuristic. If a collection of admissible heuristics hj ... hy, is available for a problem, and none of them dominates any of the others, which should we choose? As it turns out, we need not make a choice. We can have the best of all worlds, by defining

h(n) = max{hi(n),...,hm(n)} .

This composite heuristic uses whichever function is most accurate on the node in question. Because the component heuristics are admissible, h is admissible; it is also easy to prove that h is consistent. Furthermore, i dominates all of its component heuristics.

Admissible heuristics can also be derived from the solution cost of a subpreblem of a given problem. For example, Figure 4.9 shows a subproblem of the 8-puzzle instance in Figure 4.7. The subproblem involves getting tiles 1, 2, 3, 4 into their correct positions. Clearly, the cost of the optimal solution of this subproblem is a lower bound on the cost of the complete problem. It turns out to be substantially more accurate than Manhattan distance in some cases.

The idea behind pattern databases is to store these exact solution costs for every possible subproblem instance—in our example, every possible configuration of the four tiles and the blank. (Notice that the locations of the other four tiles are irrelevant for the purposes of solving the subproblem, but moves of those tiles do count towards the cost.) Then, we compute an admissible heuristic h pp for each complete state encountered during a search simply by looking up the corresponding subproblem configuration in the database. The database itself is constructed by searching backwards from the goal state and recording the cost of each new pattern encountered; the expense of this search is amortized over many subsequent problem instances.

7 Note that a perfect heuristic can be obtained simply by allowing A to run a full breadth-first search “on the sly.” Thus, there is a tradeoff between accuracy and computation time for heuristic functions.

Section 4.2. Heuristic Functions 109 «lf 2 Iff 4 2 1 \* \* 3 4 \* | \* 3 1 \* \* \* Start State Goal State Figure 4.9 A subproblem of the 8-puzzle instance given in Figure 4.7. The task is to get tiles 1, 2, 3, and 4 into their correct positions, without worrying about what happens to the other tiles.

DISJOINT PATTERN DATABASES

The choice of 1-2-3-4 is fairly arbitrary; we could also construct databases for 5-6-7-8, and for 2-4-6-8, and so on. Each database yields an admissible heuristic, and these heuristics can be combined, as explained earlier, by taking the maximum value. A combined heuristic of this kind is much more accurate than the Manhattan distance; the number of nodes generated when solving random 15-puzzles can be reduced by a factor of 1000.

One might wonder whether the heuristics obtained from the 1-2-3-4 database and the 5-6-7-8 could be added, since the two subproblems seem not to overlap. Would this still give an admissible heuristic? The answer is no, because the solutions of the 1-2-3-4 subproblem and the 5-6-7-8 subproblem for a given state will almost certainly share some moves—it is unlikely that 1-2-3-4 can be moved into place without touching 5-6-7-8, and vice versa. But what if we don’t count those moves? That is, we record not the total cost of solving the 1-2-3-4 subproblem, but just the number of moves involving 1-2-3-4. Then it is easy to see that the sum of the two costs is still a lower bound on the cost of solving the entire problem. This is the idea behind disjoint pattern databases. Using such databases, it is possible to solve random 15-puzzles in a few milliseconds—the number of nodes generated is reduced by a factor of 10,000 compared with using Manhattan distance. For 24-puzzles, a speedup of roughly a million can be obtained.

Disjoint pattern databases work for sliding-tile puzzles because the problem can be divided up in such a way that each move affects only one subproblem—because only one tile is moved at a time. For a problem such as Rubik’s cube, this kind of subdivision cannot be done because each move affects 8 or 9 of the 26 cubies. Currently, it is not clear how to define disjoint databases for such problems.

Learning heuristics from experience

A heuristic function h(n) is supposed to estimate the cost of a solution beginning from the state at node nn. How could an agent construct such a function? One solution was given in the preceding section—namely, to devise relaxed problems for which an optimal solution can be found easily. Another solution is to learn from experience. “Experience” here means solving lots of 8-puzzles, for instance. Each optimal solution to an 8-puzzle problem provides ex

110

Chapter 4. Informed Search and Exploration

FEATURES

amples from which h(n) can be learned. Each example consists of a state from the solution path and the actual cost of the solution from that point. From these examples, an inductive learning algorithm can be used to construct a function h(n) that can (with luck) predict solution costs for other states that arise during search. Techniques for doing just this using neural nets, decision trees, and other methods are demonstrated in Chapter 18. (The reinforcement learning methods described in Chapter 21 are also applicable.)

Inductive learning methods work best when supplied with features of a state that are relevant to its evaluation, rather than with just the raw state description. For example, the feature “number of misplaced tiles” might be helpful in predicting the actual distance of a state from the goal. Let’s call this feature x(n). We could take 100 randomly generated 8-puzzle configurations and gather statistics on their actual solution costs. We might find that when x1(n) is 5, the average solution cost is around 14, and so on. Given these data, the value of x1 can be used to predict h(n). Of course, we can use several features. A second feature z2(n) might be “number of pairs of adjacent tiles that are also adjacent in the goal state.” How should x(n) and z2(n) be combined to predict h(n)? A common approach is to use a linear combination:

h(n) = c1z1(n) 4 coxa(n) .

The constants ¢; and cp are adjusted to give the best fit to the actual data on solution costs. Presumably, c; should be positive and co should be negative.

LOCAL SEARCH CURRENT STATE

OPTIMIZATION PROBLEMS

OBJECTIVE FUNCTION

The search algorithms that we have seen so far are designed to explore search spaces systematically. This systematicity is achieved by keeping one or more paths in memory and by recording which alternatives have been explored at each point along the path and which have not. When a goal is found, the path to that goal also constitutes a solution to the problem.

In many problems, however, the path to the goal is irrelevant. For example, in the 8queens problem (see page 66), what matters is the final configuration of queens, not the order in which they are added. This class of problems includes many important applications such as integrated-circuit design, factory-floor layout, job-shop scheduling, automatic programming, telecommunications network optimization, vehicle routing, and portfolio management.

If the path to the goal does not matter, we might consider a different class of algorithms, ones that do not worry about paths at all. Local search algorithms operate using a single current state (rather than multiple paths) and generally move only to neighbors of that state. Typically, the paths followed by the search are not retained. Although local search algorithms are not systematic, they have two key advantages: (1) they use very little memory—usually a constant amount; and (2) they can often find reasonable solutions in large or infinite (continuous) state spaces for which systematic algorithms are unsuitable.

In addition to finding goals, local search algorithms are useful for solving pure optimization problems, in which the aim is to find the best state according to an objective function. Many optimization problems do not fit the “standard” search model introduced in

Local Search Algorithms and Optimization Problems 111

STATE SPACE LANDSCAPE

GLOBAL MINIMUM

GLOBAL MAXIMUM

HILL-CLIMBING

Chapter 3. For example, nature provides an objective function—reproductive fitness—that Darwinian evolution could be seen as attempting to optimize, but there is no “goal test” and no “path cost” for this problem.

To understand local search, we will find it very useful to consider the state space landscape (as in Figure 4.10). A landscape has both “location” (defined by the state) and “elevation” (defined by the value of the heuristic cost function or objective function). If elevation corresponds to cost, then the aim is to find the lowest valley—a global minimum; if elevation corresponds to an objective function, then the aim is to find the highest peak—a global maximum. (You can convert from one to the other just by inserting a minus sign.) Local search algorithms explore this landscape. A complete local search algorithm always finds a goal if one exists; an optimal algorithm always finds a global minimum/maximum.

objective function . | \_— global maximum

shoulder

local maximum

“flat” local maximum

— state space current state

Figure 4.10 A one-dimensional state space landscape in which elevation corresponds to the objective function. The aim is to find the global maximum. Hill-climbing search modifies the current state to try to improve it, as shown by the arrow. The various topographic features are defined in the text.

Hill-climbing search

The hill-climbing search algorithm is shown in Figure 4.11. It is simply a loop that continually moves in the direction of increasing value—that is, uphill. It terminates when it reaches a “peak” where no neighbor has a higher value. The algorithm does not maintain a search tree, so the current node data structure need only record the state and its objective function value. Hill-climbing does not look ahead beyond the immediate neighbors of the current state. This resembles trying to find the top of Mount Everest in a thick fog while suffering from amnesia.

To illustrate hill-climbing, we will use the 8-queens problem introduced on page 66. Local-search algorithms typically use a complete-state formulation, where each state has 8 queens on the board, one per column. The successor function returns all possible states generated by moving a single queen to another square in the same column (so each state has

112

loop do

neighbor, a node

Figure 4.11

[]

Figure 4.12

is more than one.

Local Search Algorithms and Optimization Problems 113

GREEDY LOCAL SEARCH

SHOULDER

SIDEWAYS MOVE

STOCHASTIC HILL CLIMBING

FIRST-CHOICE HILL CLIMBING

Hill climbing is sometimes called greedy local search because it grabs a good neighbor state without thinking ahead about where to go next. Although greed is considered one of the seven deadly sins, it turns out that greedy algorithms often perform quite well. Hill climbing often makes very rapid progress towards a solution, because it is usually quite easy to improve a bad state. For example, from the state in Figure 4.12(a), it takes just five steps to reach the state in Figure 4.12(b), which has h=1 and is very nearly a solution. Unfortunately, hill climbing often gets stuck for the following reasons:

<> Local maxima: a local maximum is a peak that is higher than each of its neighboring states, but lower than the global maximum. Hill-climbing algorithms that reach the vicinity of a local maximum will be drawn upwards towards the peak, but will then be stuck with nowhere else to go. Figure 4.10 illustrates the problem schematically. More concretely, the state in Figure 4.12(b) is in fact a local maximum (i.e., a local minimum for the cost h); every move of a single queen makes the situation worse.

¢{ Ridges: a ridge is shown in Figure 4.13. Ridges result in a sequence of local maxima that is very difficult for greedy algorithms to navigate.

{> Plateaux: a plateau is an area of the state space landscape where the evaluation function is flat. It can be a flat local maximum, from which no uphill exit exists, or a shoulder, from which it is possible to make progress. (See Figure 4.10.) A hill-climbing search might be unable to find its way off the plateau.

In each case, the algorithm reaches a point at which no progress is being made. Starting from arandomly generated 8-queens state, steepest-ascent hill climbing gets stuck 86% of the time, solving only 14% of problem instances. It works quickly, taking just 4 steps on average when it succeeds and 3 when it gets stuck—not bad for a state space with 8 a 17 million states.

The algorithm in Figure 4.11 halts if it reaches a plateau where the best successor has the same value as the current state. Might it not be a good idea to keep going—to allow a sideways move in the hope that the plateau is really a shoulder, as shown in Figure 4.10? The answer is usually yes, but we must take care. If we always allow sideways moves when there are no uphill moves, an infinite loop will occur whenever the algorithm reaches a flat local maximum that is not a shoulder. One common solution is to put a limit on the number of consecutive sideways moves allowed. For example, we could allow up to, say, 100 consecutive sideways moves in the 8-queens problem. This raises the percentage of problem instances solved by hill-climbing from 14% to 94%. Success comes at a cost: the algorithm averages roughly 21 steps for each successful instance and 64 for each failure.

Many variants of hill-climbing have been invented. Stochastic hill climbing chooses at random from among the uphill moves; the probability of selection can vary with the steepness of the uphill move. This usually converges more slowly than steepest ascent, but in some state landscapes it finds better solutions. First-cheice hill climbing implements stochastic hill climbing by generating successors randomly until one is generated that is better than the current state. This is a good strategy when a state has many (e.g., thousands) of successors. Exercise 4.16 asks you to investigate.

The hill-climbing algorithms described so far are incomplete—they often fail to find a goal when one exists because they can get stuck on local maxima. Random-restart hill

114

Chapter 4. Informed Search and Exploration

RANDOM-RESTART HILL CLIMBING

Figure 4.13 Illustration of why ridges cause difficulties for hill-climbing. The grid of states (dark circles) is superimposed on a ridge rising from left to right, creating a sequence of local maxima that are not directly connected to each other. From each local maximum, all

the available actions point downhill.

climbing adopts the well known adage, “If at first you don’t succeed, try, try again” It conducts a series of hill-climbing searches from randomly generated initial states,? stopping when a goal is found. It is complete with probability approaching 1, for the trivial reason that it will eventually generate a goal state as the initial state. If each hill-climbing search has a probability p of success, then the expected number of restarts required is 1/p. For 8-queens instances with no sideways moves allowed, p ~ 0.14, so we need roughly 7 iterations to find a goal (6 failures and 1 success). The expected number of steps is the cost of one successful iteration plus (1—p) /p times the cost of failure, or roughly 22 steps. When we allow sideways moves, 1/0.94 ~ 1.06 iterations are needed on average and (1 x 21)+(0.06/0.94) x 64 ~ 25 steps. For 8-queens, then, random-restart hill climbing is very effective indeed. Even for three million queens, the approach can find solutions in under a minute.’

The success of hill climbing depends very much on the shape of the state-space landscape: if there are few local maxima and plateaux, random-restart hill climbing will find a good solution very quickly. On the other hand, many real problems have a landscape that looks more like a family of porcupines on a flat floor, with miniature porcupines living on the tip of each porcupine needle, ad infinitum. NP-hard problems typically have an exponential number of local maxima to get stuck on. Despite this, a reasonably good local maximum can often be found after a small number of restarts.

& Generating a random state from an implicitly specified state space can be a hard problem in itself.

9 Luby et al. (1993) prove that it is best, in some cases, to restart a randomized search algorithm after a particular, fixed amount of time and that this can be much more efficient than letting each search continue indefinitely. Disallowing or limiting the number of sideways moves is an example of this.

Section 4.3.

Local Search Algorithms and Optimization Problems 115

SIMULATED ANNEALING

GRADIENT DESCENT

LOCAL BEAM SEARCH

Simulated annealing search

A hill-climbing algorithm that never makes “downhill” moves towards states with lower value (or higher cost) is guaranteed to be incomplete, because it can get stuck on a local maximum. In contrast, a purely random walk—that is, moving to a successor chosen uniformly at random from the set of successors—is complete, but extremely inefficient. Therefore, it seems reasonable to try to combine hill climbing with a random walk in some way that yields both efficiency and completeness. Simulated annealing is such an algorithm. In metallurgy, annealing is the process used to temper or harden metals and glass by heating them to a high temperature and then gradually cooling them, thus allowing the material to coalesce into a low-energy crystalline state. To understand simulated annealing, let’s switch our point of view from hill climbing to gradient descent (i.e., minimizing cost) and imagine the task of getting a ping-pong ball into the deepest crevice in a bumpy surface. If we just let the ball roll, it will come to rest at a local minimum. If we shake the surface, we can bounce the ball out of the local minimum. The trick is to shake just hard enough to bounce the ball out of local minima, but not hard enough to dislodge it from the global minimum. The simulatedannealing solution is to start by shaking hard (i.e., at a high temperature) and then gradually reduce the intensity of the shaking (i.e., lower the temperature).

The innermost loop of the simulated-annealing algorithm (Figure 4.14) is quite similar to hill climbing. Instead of picking the best move, however, it picks a random move. If the move improves the situation, it is always accepted. Otherwise, the algorithm accepts the move with some probability less than 1. The probability decreases exponentially with the “badness” of the move—the amount AE by which the evaluation is worsened. The probability also decreases as the “temperature” 7" goes down: “bad” moves are more likely to be allowed at the start when temperature is high, and they become more unlikely as 7" decreases. One can prove that if the schedule lowers T slowly enough, the algorithm will find a global optimum with probability approaching 1.

Simulated annealing was first used extensively to solve VLSI layout problems in the early 1980s. It has been applied widely to factory scheduling and other large-scale optimization tasks. In Exercise 4.16, you are asked to compare its performance to that of randomrestart hill climbing on the n-queens puzzle.

Local beam search

Keeping just one node in memory might seem to be an extreme reaction to the problem of memory limitations. The local beam search algorithm'? keeps track of k states rather than just one. It begins with £ randomly generated states. At each step, all the successors of all £ states are generated. If any one is a goal, the algorithm halts. Otherwise, it selects the k best successors from the complete list and repeats.

At first sight, a local beam search with k states might seem to be nothing more than running k random restarts in parallel instead of in sequence. In fact, the two algorithms are quite different. In a random-restart search, each search process runs independently of

10 Local beam search is an adaptation of beam search, which is a path-based algorithm.

116

Chapter 4. Informed Search and Exploration

STOCHASTIC BEAM SEARCH

GENETIC ALGORITHM

POPULATION

function SIMULATED-ANNEALING( problem, schedule) returns a solution state inputs: problem, a problem schedule, a mapping from time to “temperature” local variables: current, a node next, a node T, a “temperature” controlling the probability of downward steps

current «— MAKE-NODE(INITIAL-STATE[ problem]) for t — 1 to oo do T — schedule[t] if 7 =0 then return current next — a randomly selected successor of current AF «— VALUE[next] — VALUE[current] if AE > 0 then current — next

else current — next only with probability e2£/T

Figure 4.14 The simulated annealing search algorithm, a version of stochastic hill climbing where some downhill moves are allowed. Downhill moves are accepted readily early in the annealing schedule and then less often as time goes on. The schedule input determines the value of T as a function of time.

the others. In a local beam search, useful information is passed among the k parallel search threads. For example, if one state generates several good successors and the other k — 1 states all generate bad successors, then the effect is that the first state says to the others, “Come over here, the grass is greener!” The algorithm quickly abandons unfruitful searches and moves its resources to where the most progress is being made.

In its simplest form, local beam search can suffer from a lack of diversity among the k states—they can quickly become concentrated in a small region of the state space, making the search little more than an expensive version of hill climbing. A variant called stochastic beam search, analogous to stochastic hill climbing, helps to alleviate this problem. Instead of choosing the best &£ from the the pool of candidate successors, stochastic beam search chooses k successors at random, with the probability of choosing a given successor being an increasing function of its value. Stochastic beam search bears some resemblance to the process of natural selection, whereby the “successors” (offspring) of a “state” (organism) populate the next generation according to its “value” (fitness).

Genetic algorithms

A genetic algorithm (or GA) is a variant of stochastic beam search in which successor states are generated by combining two parent states, rather than by modifying a single state. The analogy to natural selection is the same as in stochastic beam search, except now we are dealing with sexual rather than asexual reproduction.

Like beam search, GAs begin with a set of k randomly generated states, called the population. Each state, or individual, is represented as a string over a finite alphabet—most

Section 4.3.

Local Search Algorithms and Optimization Problems 117

FITNESS FUNCTION

24748552

[32752411 | 32748552 || 3274¢Ik2 ] | 24748552 | 24752411 | 24752411 | 20 20 [32752411 ( 32752124 —={ 322124 | 1 1a] 24415124 [24415411 +] 24415417]

@ ®) © @ ©

Initial Population Fitness Function Selection Crossover Mutation

32752411 |\_23 29%

Figure 4.15 The genetic algorithm. The initial population in (a) is ranked by the fitness function in (b), resulting in pairs for mating in (c). They produce offspring in (d), which are subject to mutation in (e).

—— } mt ¥ PNW

HN BR | \_

Figure 4.16 The 8-queens states corresponding to the first two parents in Figure 4.15(c) and the first offspring in Figure 4.15(d). The shaded columns are lost in the crossover step and the unshaded columns are retained.

commonly, a string of Os and 1s. For example, an 8-queens state must specify the positions of 8 queens, each in a column of 8 squares, and so requires 8 x logy 8 = 24 bits. Alternatively, the state could be represented as 8 digits, each in the range from 1 to 8. (We will see later that the two encodings behave differently.) Figure 4.15(a) shows a population of four 8-digit strings representing 8-queens states.

The production of the next generation of states is shown in Figure 4.15(b)—(e). In (b), each state is rated by the evaluation function or (in GA terminology) the fitness function. A fitness function should return higher values for better states, so, for the 8-queens problem we use the number of nonattacking pairs of queens, which has a value of 28 for a solution. The values of the four states are 24, 23, 20, and 11. In this particular variant of the genetic algorithm, the probability of being chosen for reproducing is directly proportional to the fitness score, and the percentages are shown next to the raw scores.

In (¢), a random choice of two pairs is selected for reproduction, in accordance with the probabilities in (b). Notice that one individual is selected twice and one not at all.!! For each

11 There are many variants of this selection rule. The method of culling, in which all individuals below a given threshold are discarded, can be shown to converge faster than the random version (Baum ef al., 1995).

118

Chapter 4. Informed Search and Exploration

CROSSOVER

MUTATION

SCHEMA

pair to be mated, a crossover point is randomly chosen from the positions in the string. In Figure 4.15 the crossover points are after the third digit in the first pair and after the fifth digit in the second pair.'?

In (d), the offspring themselves are created by crossing over the parent strings at the crossover point. For example, the first child of the first pair gets the first three digits from the first parent and the remaining digits from the second parent, whereas the second child gets the first three digits from the second parent and the rest from the first parent. The 8-queens states involved in this reproduction step are shown in Figure 4.16. The example illustrates the fact that, when two parent states are quite different, the crossover operation can produce a state that is a long way from either parent state. It is often the case that the population is quite diverse early on in the process, so crossover (like simulated annealing) frequently takes large steps in the state space early in the search process and smaller steps later on when most individuals are quite similar.

Finally, in (e), each location is subject to random mutation with a small independent probability. One digit was mutated in the first, third, and fourth offspring. In the 8-queens problem, this corresponds to choosing a queen at random and moving it to a random square in its column. Figure 4.17 describes an algorithm that implements all these steps.

Like stochastic beam search, genetic algorithms combine an uphill tendency with random exploration and exchange of information among parallel search threads. The primary advantage, if any, of genetic algorithms comes from the crossover operation. Yet it can be shown mathematically that, if the positions of the genetic code is permuted initially in a random order, crossover conveys no advantage. Intuitively, the advantage comes from the ability of crossover to combine large blocks of letters that have evolved independently to perform useful functions, thus raising the level of granularity at which the search operates. For example, it could be that putting the first three queens in positions 2, 4, and 6 (where they do not attack each other) constitutes a useful block that can be combined with other blocks to construct a solution.

The theory of genetic algorithms explains how this works using the idea of a schema, which is a substring in which some of the positions can be left unspecified. For example, the schema 246\*\*\*\*\* describes all 8-queens states in which the first three queens are in positions 2, 4, and 6 respectively. Strings that match the schema (such as 24613578) are called instances of the schema. It can be shown that, if the average fitness of the instances of a schema is above the mean, then the number of instances of the schema within the population will grow over time. Clearly, this effect is unlikely to be significant if adjacent bits are totally unrelated to each other, because then there will be few contiguous blocks that provide a consistent benefit. Genetic algorithms work best when schemas correspond to meaningful components of a solution. For example, if the string is a representation of an antenna, then the schemas may represent components of the antenna, such as reflectors and deflectors. A good component is likely to be good in a variety of different designs. This suggests that successful use of genetic algorithms requires careful engineering of the representation.

12 Tt is here that the encoding matters. If a 24-bit encoding is used instead of 8 digits, then the crossover point has a 2/3 chance of being in the middle of a digit, which results in an essentially arbitrary mutation of that digit.

Section 4.4. Local Search in Continuous Spaces 119

function GENETIC-ALGORITHM( population, FITNESS-FN) returns an individual inputs: population, a set of individuals FITNESS-FN, a function that measures the fitness of an individual

repeat new-population — empty set loop for : from 1 to SIZE( population) do z «+ RANDOM-SELECTION( population, FITNESS-FN) yy — RANDOM-SELECTION( population, FITNESS-FN) child — REPRODUCE(z, ¥) if (small random probability) then child — MUTATE(child) add child to new\_population population «— new\_population until some individual is fit enough, or enough time has elapsed return the best individual in population, according to FITNESS-FN

function REPRODUCE(x, y) returns an individual inputs: x, y, parent individuals

n «— LENGTH(x) ¢ « random number from 1 to n return APPEND(SUBSTRING(z, 1, ¢), SUBSTRING(y, ¢ + 1, n))

Figure 4.17 A genetic algorithm. The algorithm is the same as the one diagrammed in Figure 4.15, with one variation: in this more popular version, each mating of two parents produces only one offspring, not two.

In practice, genetic algorithms have had a widespread impact on optimization problems, such as circuit layout and job-shop scheduling. At present, it is not clear whether the appeal of genetic algorithms arises from their performance or from their @sthetically pleasing origins in the theory of evolution. Much work remains to be done to identify the conditions under which genetic algorithms perform well.

4.4 LOCAL SEARCH IN CONTINUOUS SPACES

In Chapter 2, we explained the distinction between discrete and continuous environments, pointing out that most real-world environments are continuous. Yet none of the algorithms we have described can handle continuous state spaces—the successor function would in most cases return infinitely many states! This section provides a very brief introduction to some local search techniques for finding optimal solutions in continuous spaces. The literature on this topic is vast; many of the basic techniques originated in the 17th century, after the development of calculus by Newton and Leibniz.!\* We will find uses for these techniques at

13 A basic knowledge of multivariate calculus and vector arithmetic is useful when one is reading this section.

120 Chapter 4. Informed Search and Exploration

EVOLUTION AND SEARCH

The theory of evolution was developed in Charles Darwin’s On the Origin of Species by Means of Natural Selection (1859). The central idea is simple: variations (known as mutations) occur in reproduction and will be preserved in successive generations approximately in proportion to their effect on reproductive fitness.

Darwin’s theory was developed with no knowledge of how the traits of organisms can be inherited and modified. The probabilistic laws governing these processes were first identified by Gregor Mendel (1866), a monk who experimented with sweet peas using what he called artificial fertilization. Much later, Watson and Crick (1953) identified the structure of the DNA molecule and its alphabet, AGTC (adenine, guanine, thymine, cytosine). In the standard model, variation occurs both by point mutations in the letter sequence and by “crossover” (in which the DNA of an offspring is generated by combining long sections of DNA from each parent).

The analogy to local search algorithms has already been described; the principal difference between stochastic beam search and evolution is the use of sexual reproduction, wherein successors are generated from multiple organisms rather than just one. The actual mechanisms of evolution are, however, far richer than most genetic algorithms allow. For example, mutations can involve reversals, duplications, and movement of large chunks of DNA; some viruses borrow DNA from one organism and insert it in another; and there are transposable genes that do nothing but copy themselves many thousands of times within the genome. There are even genes that poison cells from potential mates that do not carry the gene, thereby increasing their chances of replication. Most important is the fact that the genes themselves encode the mechanisms whereby the genome is reproduced and translated into an organism. In genetic algorithms, those mechanisms are a separate program that is not represented within the strings being manipulated.

Darwinian evolution might well seem to be an inefficient mechanism, having generated blindly some 10% or so organisms without improving its search heuristics one iota. Fifty years before Darwin, however, the otherwise great French naturalist Jean Lamarck (1809) proposed a theory of evolution whereby traits acquired by adaptation during an organism’s lifetime would be passed on to its offspring. Such a process would be effective, but does not seem to occur in nature. Much later, James Baldwin (1896) proposed a superficially similar theory: that behavior learned during an organism’s lifetime could accelerate the rate of evolution. Unlike Lamarck’s, Baldwin’s theory is entirely consistent with Darwinian evolution, because it relies on selection pressures operating on individuals that have found local optima among the set of possible behaviors allowed by their genetic makeup. Modern computer simulations confirm that the “Baldwin effect” is real, provided that “ordinary” evolution can create organisms whose internal performance measure is somehow correlated with actual fitness.

Section 4.4.

Local Search in Continuous Spaces 121

GRADIENT

EMPIRICAL GRADIENT

LINE SEARCH

several places in the book, including the chapters on learning, vision, and robotics. In short, anything that deals with the real world.

Let us begin with an example. Suppose we want to place three new airports anywhere in Romania, such that the sum of squared distances from each city on the map (Figure 3.2) to its nearest airport is minimized. Then the state space is defined by the coordinates of the airports: (x1, 41), (z2,y2), and (x3,ys3). This is a six-dimensional space; we also say that states are defined by six variables. (In general, states are defined by an n-dimensional vector of variables, x.) Moving around in this space corresponds to moving one or more of the airports on the map. The objective function f(z1, yi, x2, y2, Z3,y3) is relatively easy to compute for any particular state once we compute the closest cities, but rather tricky to write down in general.

One way to avoid continuous problems is simply to discretize the neighborhood of each state. For example, we can move only one airport at a time in either the x or y direction by a fixed amount +4. With 6 variables, this gives 12 possible successors for each state. We can then apply any of the local search algorithms described previously. One can also apply stochastic hill climbing and simulated annealing directly, without discretizing the space. These algorithms choose successors randomly, which can be done by generating random vectors of length J.

There are many methods that attempt to use the gradient of the landscape to find a maximum. The gradient of the objective function is a vector V f that gives the magnitude and direction of the steepest slope. For our problem, we have

vio (OF of of of OF of

B (5; "Oy1’ Ox’ Oya’ Oxy’ 5s)

In some cases, we can find a maximum by solving the equation V f = 0. (This could be done, for example, if we were placing just one airport; the solution is the arithmetic mean of all the cities’ coordinates.) In many cases, however, this equation cannot be solved in closed form. For example, with three airports, the expression for the gradient depends on what cities are closest to each airport in the current state. This means we can compute the gradient locally but not globally. Even so, we can still perform steepest-ascent hill climbing by updating the current state via the formula

Xx —x+aVfx),

where « is a small constant. In other cases, the objective function might not be available in a differentiable form at all—for example, the value of a particular set of airport locations may be determined by running some large-scale economic simulation package. In those cases, a so-called empirical gradient can be determined by evaluating the response to small increments and decrements in each coordinate. Empirical gradient search is the same as steepest-ascent hill climbing in a discretized version of the state space.

Hidden beneath the phrase “« is a small constant” lies a huge variety of methods for adjusting a. The basic problem is that, if « is too small, too many steps are needed; if « is too large, the search could overshoot the maximum. The technique of line search tries to overcome this dilemma by extending the current gradient direction—usually by repeatedly doubling a—until f starts to decrease again. The point at which this occurs becomes the new

122 Chapter 4. Informed Search and Exploration

current state. There are several schools of thought about how the new direction should be chosen at this point.

NEWTON-RAPHSON For many problems, the most effective algorithm is the venerable Newton—Raphson method (Newton, 1671; Raphson, 1690). This is a general technique for finding roots of functions—that is, solving equations of the form g(x) =0. It works by computing a new estimate for the root x according to Newton’s formula

zz —glx)/g(z). To find a maximum or minimum of f, we need to find x such that the gradient is zero (i.e., V f(x) =0). Thus g(x) in Newton’s formula becomes V f(x), and the update equation can be written in matrix-vector form as

x —xHi (x) V(x),

HESSIAN where Hy (x) is the Hessian matrix of second derivatives, whose elements H;; are given by 0 f/0z;0z;. Since the Hessian has n? entries, Newton—Raphson becomes expensive in high-dimensional spaces, and many approximations have been developed.

Local search methods suffer from local maxima, ridges, and plateaux in continuous state spaces just as much as in discrete spaces. Random restarts and simulated annealing can be used and are often helpful. High-dimensional continuous spaces are, however, big places in which it is easy to get lost,

CONSTRAINED A final topic with which a passing acquaintance is useful is constrained optimization. An optimization problem is constrained if solutions must satisfy some hard constraints on the values of each variable. For example, in our airport-siting problem, we might constrain sites to be inside Romania and on dry land (rather than in the middle of lakes). The difficulty of constrained optimization problems depends on the nature of the constraints and the objec

HEA MMING tive function. The best-known category is that of linear programming problems, in which constraints must be linear inequalities forming a convex region and the objective function is also linear. Linear programming problems can be solved in time polynomial in the number of variables. Problems with different types of constraints and objective functions have also been studied—quadratic programming, second-order conic programming, and so on.

4.5 ONLINE SEARCH AGENTS AND UNKNOWN ENVIRONMENTS

OFFLINE SEARCH So far we have concentrated on agents that use offline search algorithms. They compute a complete solution before setting foot in the real world (see Figure 3.1), and then execute the ONLINE SEARCH solution without recourse to their percepts. In contrast, an online search'\* agent operates by interleaving computation and action: first it takes an action, then it observes the environment and computes the next action. Online search is a good idea in dynamic or semidynamic domains—domains where there is a penalty for sitting around and computing too long. Online search is an even better idea for stochastic domains. In general, an offline search would

14 The term “online” is commonly used in computer science to refer to algorithms that must process input data as they are received, rather than waiting for the entire input data set to become available.

Section 4.5.

Online Search Agents and Unknown Environments 123

EXPLORATION PROBLEM

COMPETITIVE RATIO

have to come up with an exponentially large contingency plan that considers all possible happenings, while an online search need only consider what actually does happen. For example, a chess playing agent is well-advised to make its first move long before it has figured out the complete course of the game.

Online search is a necessary idea for an exploration problem, where the states and actions are unknown to the agent. An agent in this state of ignorance must use its actions as experiments to determine what to do next, and hence must interleave computation and action.

The canonical example of online search is a robot that is placed in a new building and must explore it to build a map that it can use for getting from A to B. Methods for escaping from labyrinths—required knowledge for aspiring heroes of antiquity—are also examples of online search algorithms. Spatial exploration is not the only form of exploration, however. Consider a newborn baby: it has many possible actions, but knows the outcomes of none of them, and it has experienced only a few of the possible states that it can reach. The baby’s gradual discovery of how the world works is, in part, an online search process.

Online search problems

An online search problem can be solved only by an agent executing actions, rather than by a purely computational process. We will assume that the agent knows just the following:

e ACTIONS(s), which returns a list of actions allowed in state s;

e The step-cost function ¢(s, a, s')—note that this cannot be used until the agent knows that s’ is the outcome; and

eo GOAL-TEST(s).

Note in particular that the agent cannot access the successors of a state except by actually trying all the actions in that state. For example, in the maze problem shown in Figure 4.18, the agent does not know that going Up from (1,1) leads to (1,2); nor, having done that, does it know that going Down will take it back to (1,1). This degree of ignorance can be reduced in some applications—for example, a robot explorer might know how its movement actions work and be ignorant only of the locations of obstacles.

We will assume that the agent can always recognize a state that it has visited before, and we will assume that the actions are deterministic. (These last two assumptions are relaxed in Chapter 17.) Finally, the agent might have access to an admissible heuristic function h(s) that estimates the distance from the current state to a goal state. For example, in Figure 4.18, the agent might know the location of the goal and be able to use the Manhattan distance heuristic.

Typically, the agent’s objective is to reach a goal state while minimizing cost. (Another possible objective is simply to explore the entire environment.) The cost is the total path cost of the path that the agent actually travels. It is common to compare this cost with the path cost of the path the agent would follow if it knew the search space in advance—that is, the actual shortest path (or shortest complete exploration). In the language of online algorithms, this is called the competitive ratio; we would like it to be as small as possible.

Although this sounds like a reasonable request, it is easy to see that the best achievable competitive ratio is infinite in some cases. For example, if some actions are irreversible, the online search might accidentally reach a dead-end state from which no goal state is reachable.

124 Chapter 4. Informed Search and Exploration

3 G 2 11S

1 2 3

Figure 4.18 A simple maze problem. The agent starts at S and must reach GG, but knows nothing of the environment.

(b)

Figure 4.19 (a) Two state spaces that might lead an online search agent into a dead end. Any given agent will fail in at least one of these spaces. (b) A two-dimensional environment that can cause an online search agent to follow an arbitrarily inefficient route to the goal. Whichever choice the agent makes, the adversary blocks that route with another long, thin wall, so that the path followed is much longer than the best possible path.

Perhaps you find the term “accidentally” unconvincing—after all, there might be an algorithm that happens not to take the dead-end path as it explores. Our claim, to be more precise, is that

ie no algorithm can avoid dead ends in all state spaces. Consider the two dead-end state spaces in Figure 4.19(a). To an online search algorithm that has visited states .S and A, the two state spaces look identical, so it must make the same decision in both. Therefore, it will fail in

AERA one of them. This is an example of an adversary argument—we can imagine an adversary that constructs the state space while the agent explores it and can put the goals and dead ends wherever it likes.

Section 4.5.

Online Search Agents and Unknown Environments 125

SAFELY EXPLORABLE

Dead ends are a real difficulty for robot exploration—staircases, ramps, cliffs, and all kinds of natural terrain present opportunities for irreversible actions. To make progress, we will simply assume that the state space is safely explorable—that is, some goal state is reachable from every reachable state. State spaces with reversible actions, such as mazes and 8-puzzles, can be viewed as undirected graphs and are clearly safely explorable.

Even in safely explorable environments, no bounded competitive ratio can be guaranteed if there are paths of unbounded cost. This is easy to show in environments with irreversible actions, but in fact it remains true for the reversible case as well, as Figure 4.19(b) shows. For this reason, it is common to describe the performance of online search algorithms in terms of the size of the entire state space rather than just the depth of the shallowest goal.

Online search agents

After each action, an online agent receives a percept telling it what state it has reached; from this information, it can augment its map of the environment. The current map is used to decide where to go next. This interleaving of planning and action means that online search algorithms are quite different from the offline search algorithms we have seen previously. For example, offline algorithms such as A" have the ability to expand a node in one part of the space and then immediately expand a node in another part of the space, because node expansion involves simulated rather than real actions. An online algorithm, on the other hand, can expand only a node that it physically occupies. To avoid traveling all the way across the tree to expand the next node, it seems better to expand nodes in a local order. Depth-first search has exactly this property, because (except when backtracking) the next node expanded is a child of the previous node expanded.

An online depth-first search agent is shown in Figure 4.20. This agent stores its map in a table, result]a, s], that records the state resulting from executing action a in state s. Whenever an action from the current state has not been explored, the agent tries that action. The difficulty comes when the agent has tried all the actions in a state. In offline depth-first search, the state is simply dropped from the queue; in an online search, the agent has to backtrack physically. In depth-first search, this means going back to the state from which the agent entered the current state most recently. That is achieved by keeping a table that lists, for each state, the predecessor states to which the agent has not yet backtracked. If the agent has run out of states to which it can backtrack, then its search is complete.

We recommend that the reader trace through the progress of ONLINE-DFS-AGENT when applied to the maze given in Figure 4.18. Tt is fairly easy to see that the agent will, in the worst case, end up traversing every link in the state space exactly twice. For exploration, this is optimal; for finding a goal, on the other hand, the agent’s competitive ratio could be arbitrarily bad if it goes off on a long excursion when there is a goal right next to the initial state. An online variant of iterative deepening solves this problem; for an environment that is a uniform tree, the competitive ratio of such an agent is a small constant.

Because of its method of backtracking, ONLINE-DFS-AGENT works only in state spaces where the actions are reversible. There are slightly more complex algorithms that work in general state spaces, but no such algorithm has a bounded competitive ratio.

126 Chapter 4. Informed Search and Exploration function ONLINE-DFS-AGENT(s’) returns an action inputs: s’, a percept that identifies the current state static: result, a table, indexed by action and state, initially empty unexplored, a table that lists, for each visited state, the actions not yet tried unbacktracked, a table that lists, for each visited state, the backtracks not yet tried s, a, the previous state and action, initially null if GOAL-TEST(s’) then return stop if s’ is a new state then unexplored[s’] + ACTIONS(s’) if s is not null then do result[a, s] — s’ add s to the front of unbacktracked[s’] if unexplored[s’] is empty then if unbacktracked[s'] is empty then return stop else a <— an action b such that result[b, s'] = POP(unbacktracked[s']) else a — Por(unexplored[s’]) s« 8 return q Figure 4.20 An online search agent that uses depth-first exploration. The agent is applicable only in bidirected search spaces. Online local search Like depth-first search, hill-climbing search has the property of locality in its node expansions. In fact, because it keeps just one current state in memory, hill-climbing search is already an online search algorithm! Unfortunately, it is not very useful in its simplest form because it leaves the agent sitting at local maxima with nowhere to go. Moreover, random restarts cannot be used, because the agent cannot transport itself to a new state. RANDOM WALK Instead of random restarts, one might consider using a random walk to explore the

environment. A random walk simply selects at random one of the available actions from the current state; preference can be given to actions that have not yet been tried. It is easy to prove that a random walk will eventually find a goal or complete its exploration, provided that the space is finite.!> On the other hand, the process can be very slow. Figure 4.21 shows an environment in which a random walk will take exponentially many steps to find the goal, because, at each step, backward progress is twice as likely as forward progress. The example is contrived, of course, but there are many real-world state spaces whose topology causes these kinds of “traps” for random walks.

Augmenting hill climbing with memory rather than randomness turns out to be a more effective approach. The basic idea is to store a “current best estimate” H(s) of the cost to reach the goal from each state that has been visited. H(s) starts out being just the heuristic

15 The infinite case is much more tricky. Random walks are complete on infinite one-dimensional and two dimensional grids, but not on three-dimensional grids! In the latter case, the probability that the walk ever returns to the starting point is only about 0.3405. (See Hughes, 1995, for a general introduction.)

Online Search Agents and Unknown Environments 127

[]

OPTI

MISM UNDER RTAINTY

Figure 4.21 An environment in which a random walk will take exponentially many steps to find the goal.

estimate h(s) and is updated as the agent gains experience in the state space. Figure 4.22 shows a simple example in a one-dimensional state space. In (a), the agent seems to be stuck in a flat local minimum at the shaded state. Rather than staying where it is, the agent should follow what seems to be the best path to the goal based on the current cost estimates for its neighbors. The estimated cost to reach the goal through a neighbor s is the cost to get to s’ plus the estimated cost to get to a goal from there—that is, ¢(s,a,s’) + H(s'). In the example, there are two actions with estimated costs 1 +9 and 1 + 2, so it seems best to move right. Now, it is clear that the cost estimate of 2 for the shaded state was overly optimistic. Since the best move cost 1 and led to a state that is at least 2 steps from a goal, the shaded state must be at least 3 steps from a goal, so its H should be updated accordingly, as shown in Figure 4.22(b). Continuing this process, the agent will move back and forth twice more, updating H{ each time and “flattening out” the local minimum until it escapes to the right.

An agent implementing this scheme, which is called learning real-time A\* (LRTA"), is shown in Figure 4.23. Like ONLINE-DFSAGENT, it builds a map of the environment using the result table. It updates the cost estimate for the state it has just left and then chooses the “apparently best” move according to its current cost estimates. One important detail is that actions that have not yet been tried in a state s are always assumed to lead immediately to the goal with the least possible cost, namely A (ss). This optimism under uncertainty encourages the agent to explore new, possibly promising paths.

An LRTA" agent is guaranteed to find a goal in any finite, safely explorable environment. Unlike A\*, however, it is not complete for infinite state spaces—there are cases where it can be led infinitely astray. It can explore an environment of n states in O(n?) steps in the worst case, but often does much better. The LRTA\* agent is just one of a large family of online agents that can be defined by specifying the action selection rule and the update rule in different ways. We will discuss this family, which was developed originally for stochastic environments, in Chapter 21.

Learning in online search

The initial ignorance of online search agents provides several opportunities for learning. First, the agents learn a “map” of the environment—more precisely, the outcome of each action in each state—simply by recording each of their experiences. (Notice that the assumption of deterministic environments means that one experience is enough for each action.) Second, the local search agents acquire more accurate estimates of the value of each state by using local updating rules, as in LRTA\*. In Chapter 21 we will see that these updates eventually

128

Chapter 4. Informed Search and Exploration

[]

Figure 4.22 Five iterations of LRTA\* on a one-dimensional state space. Each state is labeled with H(s), the current cost estimate to reach a goal, and each arc is labeled with its step cost. The shaded state marks the location of the agent, and the updated values at each iteration are circled.

function LRTA \*AGENT(s’) returns an action inputs: s’, a percept that identifies the current state static: result, a table, indexed by action and state, initially empty H, a table of cost estimates indexed by state, initially empty 8, a, the previous state and action, initially null

if GOAL-TEST(s’) then return stop if s’ is a new state (not in H) then H[s'] — h(s’) unless s is null

result{a, s] — s

H[s]«— min LRTA\*-C0OST(s,b, result[b, s], H) be ACTIONS(s)

a «an action b in ACTIONS(s’) that minimizes LRTA\*-CoOST(s’, b, result[b, s'], H) ss’

return a

function LRTA\*-COST(s, a, s’, H) returns a cost estimate if s’ is undefined then return A(s) else return c(s, a, s’) + H[5]

Figure 423 LRTA\*-AGENT selects an action according to the values of neighboring states, which are updated as the agent moves about the state space.

Section 4.6.

Summary 129

converge to exact values for every state, provided that the agent explores the state space in the right way. Once exact values are known, optimal decisions can be taken simply by moving to the highest-valued successor—that is, pure hill climbing is then an optimal strategy.

If you followed our suggestion to trace the behavior of ONLINE-DFS-AGENT in the environment of Figure 4.18, you will have noticed that the agent is not very bright. For example, after it has seen that the Up action goes from (1,1) to (1,2), the agent still has no idea that the Down action goes back to (1,1), or that the Up action also goes from (2,1) to (2,2), from (2,2) to (2,3), and so on. In general, we would like the agent to learn that Up increases the y-coordinate unless there is a wall in the way, that Down reduces it, and so on. For this to happen, we need two things. First, we need a formal and explicitly manipulable representation for these kinds of general rules; so far, we have hidden the information inside the black box called the successor function. Part III is devoted to this issue. Second, we need algorithms that can construct suitable general rules from the specific observations made by the agent. These are covered in Chapter 18.

This chapter has examined the application of heuristics to reduce search costs. We have looked at a number of algorithms that use heuristics and found that optimality comes at a stiff price in terms of search cost, even with good heuristics.

o Best-first search is just GRAPH-SEARCH where the minimum-cost unexpanded nodes (according to some measure) are selected for expansion. Best-first algorithms typically use a heuristic function h(n) that estimates the cost of a solution from n.

o Greedy best-first search expands nodes with minimal h(n). It is not optimal, but is often efficient.

e A\* search expands nodes with minimal f(n) = g(n) + h(n). A" is complete and optimal, provided that we guarantee that h(n) is admissible (for TREE-SEARCH) or consistent (for GRAPH-SEARCH). The space complexity of A\* is still prohibitive.

¢ The performance of heuristic search algorithms depends on the quality of the heuristic function. Good heuristics can sometimes be constructed by relaxing the problem definition, by precomputing solution costs for subproblems in a pattern database, or by learning from experience with the problem class.

e RBFS and SMA" are robust, optimal search algorithms that use limited amounts of memory; given enough time, they can solve problems that A\* cannot solve because it runs out of memory.

e Local search methods such as hill climbing operate on complete-state formulations, keeping only a small number of nodes in memory. Several stochastic algorithms have been developed, including simulated annealing, which returns optimal solutions when given an appropriate cooling schedule. Many local search methods can also be used to solve problems in continuous spaces.

130 Chapter 4. Informed Search and Exploration

¢ A genetic algorithm is a stochastic hill-climbing search in which a large population of states is maintained. New states are generated by mutation and by crossover, which combines pairs of states from the population.

o Exploration problems arise when the agent has no idea about the states and actions of its environment. For safely explorable environments, online search agents can build a map and find a goal if one exists. Updating heuristic estimates from experience provides an effective method to escape from local minima.

BIBLIOGRAPHICAL AND HISTORICAL NOTES

The use of heuristic information in problem solving appears in an early paper by Simon and Newell (1958), but the phrase “heuristic search” and the use of heuristic functions that estimate the distance to the goal came somewhat later (Newell and Emst, 1965; Lin, 1965). Doran and Michie (1966) conducted extensive experimental studies of heuristic search as applied to a number of problems, especially the 8-puzzle and the 15-puzzle. Although Doran and Michie carried out theoretical analyses of path length and “penetrance” (the ratio of path length to the total number of nodes examined so far) in heuristic search, they appear to have ignored the information provided by current path length. The A\* algorithm, incorporating the current path length into heuristic search, was developed by Hart, Nilsson, and Raphael (1968), with some later corrections (Hart ef al., 1972). Dechter and Pearl (1985) demonstrated the optimal efficiency of A\*.

The original A\* paper introduced the consistency condition on heuristic functions. The monotone condition was introduced by Pohl (1977) as a simpler replacement, but Pearl (1984) showed that the two were equivalent. A number of algorithms predating A\* used the equivalent of open and closed lists; these include breadth-first, depth-first, and uniform-cost search (Bellman, 1957; Dijkstra, 1959). Bellman’s work in particular showed the importance of additive path costs in simplifying optimization algorithms.

Pohl (1970, 1977) pioneered the study of the relationship between the error in heuristic functions and the time complexity of A\*. The proof that A\* runs in linear time if the error in the heuristic function is bounded by a constant can be found in Pohl (1977) and in Gaschnig (1979). Pearl (1984) strengthened this result to allow a logarithmic growth in the error. The “effective branching factor” measure of the efficiency of heuristic search was proposed by Nilsson (1971).

There are many variations on the A\* algorithm. Pohl (1973) proposed the use of dynamic weighting, which uses a weighted sum fy, (n) = wqg(n) + wph(n) of the current path length and the heuristic function as an evaluation function, rather than the simple sum f(n)=g(n)+ h(n) used in A\*. The weights wg, and wy, are adjusted dynamically as the search progresses. Pohl’s algorithm can be shown to be e-admissible—that is, guaranteed to find solutions within a factor 1 + € of the optimal solution—where ¢ is a parameter supplied to the algorithm. The same property is exhibited by the A} algorithm (Pearl, 1984), which can select any node from the fringe provided its f-cost is within a factor 1 + € of the lowestf-cost fringe node. The selection can be done so as to minimize search cost.

Section 4.6.

Summary 131

ITERATIVE EXPANSION

A" and other state-space search algorithms are closely related to the branch-and-bound techniques that are widely used in operations research (Lawler and Wood, 1966). The relationships between state-space search and branch-and-bound have been investigated in depth (Kumar and Kanal, 1983; Nau ef al., 1984; Kumar ef al., 1988). Martelli and Montanari (1978) demonstrate a connection between dynamic programming (see Chapter 17) and certain types of state-space search. Kumar and Kanal (1988) attempt a “grand unification” of heuristic search, dynamic programming, and branch-and-bound techniques under the name of CDP—the “composite decision process.”

Because computers in the late 1950s and early 1960s had at most a few thousand words of main memory, memory-bounded heuristic search was an early research topic. The Graph Traverser (Doran and Michie, 1966), one of the earliest search programs, commits to an operator after searching best first up to the memory limit. IDA\* (Korf, 1985a, 1985b) was the first widely used optimal, memory-bounded, heuristic search algorithm, and a large number of variants have been developed. An analysis of the efficiency of IDA\* and of its difficulties with real-valued heuristics appears in Patrick et al. (1992).

RBFS (Korf, 1991, 1993) is actually somewhat more complicated than the algorithm shown in Figure 4.5, which is closer to an independently developed algorithm called iterative expansion, or IE (Russell, 1992). RBFS uses a lower bound as well as the upper bound; the two algorithms behave identically with admissible heuristics, but RBFS expands nodes in best-first order even with an inadmissible heuristic. The idea of keeping track of the best alternative path appeared earlier in Bratko’s (1986) elegant Prolog implementation of A\* and in the DTA\* algorithm (Russell and Wefald, 1991). The latter work also discusses metalevel state spaces and metalevel learning.

The MA\* algorithm appeared in Chakrabarti ef al. (1989). SMA\*, or Simplified MA\*, emerged from an attempt to implement MA\* as a comparison algorithm for IE (Russell, 1992). Kaindl and Khorsand (1994) have applied SMA\* to produce a bidirectional search algorithm that is substantially faster than previous algorithms. Korf and Zhang (2000) describe a divideand-conquer approach, and Zhou and Hansen (2002) introduce memory-bounded A\* graph search. Korf (1995) surveys memory-bounded search techniques.

The idea that admissible heuristics can be derived by problem relaxation appears in the seminal paper by Held and Karp (1970), who used the the minimum-spanning-tree heuristic to solve the TSP. (See Exercise 4.8.)

The automation of the relaxation process was implemented successfully by Prieditis (1993), building on earlier work with Mostow (Mostow and Prieditis, 1989). The use of pattern databases to derive admissible heuristics is due to Gasser (1995) and Culberson and Schaeffer (1998); disjoint pattern databases are described by Korf and Felner (2002). The probabilistic interpretation of heuristics was investigated in depth by Pearl (1984) and Hansson and Mayer (1989).

By far the most comprehensive source on heuristics and heuristic search algorithms is Pearl’s (1984) Heuristics text. This book provides especially good coverage of the wide variety of offshoots and variations of A\*, including rigorous proofs of their formal properties. Kanal and Kumar (1988) present an anthology of important articles on heuristic search. New results on search algorithms appear regularly in the journal Artificial Intelligence.

132

Chapter 4. Informed Search and Exploration

TABU SEARCH

HEAVY-TAILED DISTRIBUTION

EVOLUTION STRATEGIES

Local-search techniques have a long history in mathematics and computer science. Indeed, the Newton—Raphson method (Newton, 1671; Raphson, 1690) can be seen as a very efficient local-search method for continuous spaces in which gradient information is available. Brent (1973) is a classic reference for optimization algorithms that do not require such information. Beam search, which we have presented as a local-search algorithm, originated as a bounded-width variant of dynamic programming for speech recognition in the HARPY system (Lowerre, 1976). A related algorithm is analyzed in depth by Pearl (1984, Ch. 5).

The topic of local search has been reinvigorated in recent years by surprisingly good results for large constraint satisfaction problems such as n-queens (Minton ef al., 1992) and logical reasoning (Selman et al., 1992) and by the incorporation of randomness, multiple simultaneous searches, and other improvements. This renaissance of what Christos Papadimitriou has called “New Age” algorithms has also sparked increased interest among theoretical computer scientists (Koutsoupias and Papadimitriou, 1992; Aldous and Vazirani, 1994). In the field of operations research, a variant of hill climbing called tabu search has gained popularity (Glover, 1989; Glover and Laguna, 1997). Drawing on models of limited shortterm memory in humans, this algorithm maintains a tabu list of k previously visited states that cannot be revisited; as well as improving efficiency when searching graphs, this can allow the algorithm to escape from some local minima. Another useful improvement on hill climbing is the STAGE algorithm (Boyan and Moore, 1998). The idea is to use the local maxima found by random-restart hill climbing to get an idea of the overall shape of the landscape. The algorithm fits a smooth surface to the set of local maxima and then calculates the global maximum of that surface analytically. This becomes the new restart point. The algorithm has been shown to work in practice on hard problems. (Gomes et al., 1998) showed that the run time distributions of systematic backtracking algorithms often have a heavy-tailed distribution, which means that the probability of a very long run time is more than would be predicted if the run times were normally distributed. This provides a theoretical justification for random restarts.

Simulated annealing was first described by Kirkpatrick ef al. (1983), who borrowed directly from the Metropolis algorithm (which is used to simulate complex systems in physics (Metropolis et al., 1953) and was supposedly invented at a Los Alamos dinner party). Simulated annealing is now a field in itself, with hundreds of papers published every year.

Finding optimal solutions in continuous spaces is the subject matter of several fields, including optimization theory, optimal control theory, and the calculus of variations, Suitable (and practical) entry points are provided by Press er al. (2002) and Bishop (1995). Linear programming (LP) was one of the first applications of computers; the simplex algorithm (Wood and Dantzig, 1949; Dantzig, 1949) is still used despite worst-case exponential complexity. Karmarkar (1984) developed a practical polynomial-time algorithm for LP.

Work by Sewall Wright (1931) on the concept of a fitness landscape was an important precursor to the development of genetic algorithms. In the 1950s, several statisticians, including Box (1957) and Friedman (1959), used evolutionary techniques for optimization problems, but it wasn’t until Rechenberg (1965, 1973) introduced evolution strategies to solve optimization problems for airfoils that the approach gained popularity. In the 1960s and 1970s, John Holland (1975) championed genetic algorithms, both as a useful tool and

Section 4.6.

Summary 133

ARTIFICIAL LIFE

GENETIC PROGRAMMING

EULERIAN GRAPHS

as a method to expand our understanding of adaptation, biological or otherwise (Holland, 1995). The artificial life movement (Langton, 1995) takes this idea one step further, viewing the products of genetic algorithms as organisms rather than solutions to problems. Work in this field by Hinton and Nowlan (1987) and Ackley and Littman (1991) has done much to clarify the implications of the Baldwin effect. For general background on evolution, we strongly recommend Smith and Szathmary (1999).

Most comparisons of genetic algorithms to other approaches (especially stochastic hillclimbing) have found that the genetic algorithms are slower to converge (O'Reilly and Oppacher, 1994; Mitchell et al., 1996; Juels and Wattenberg, 1996; Baluja, 1997). Such findings are not universally popular within the GA community, but recent attempts within that community to understand population-based search as an approximate form of Bayesian learning (see Chapter 20) might help to close the gap between the field and its critics (Pelikan ef al., 1999). The theory of quadratic dynamical systems may also explain the performance of GAs (Rabani er al., 1998). See Lohn ef al. (2001) for an example of GAs applied to antenna design, and Larrafiaga er al. (1999) for an application to the traveling salesperson problem,

The field of genetic programming is closely related to genetic algorithms. The principal difference is that the representations that are mutated and combined are programs rather than bit strings. The programs are represented in the form of expression trees; the expressions can be in a standard language such as Lisp or can be specially designed to represent circuits, robot controllers, and so on. Crossover involves splicing together subtrees rather than substrings. This form of mutation guarantees that the offspring are well-formed expressions, which would not be the case if programs were manipulated as strings.

Recent interest in genetic programming was spurred by John Koza’s work (Koza, 1992, 1994), but it goes back at least to early experiments with machine code by Friedberg (1958) and with finite-state automata by Fogel ef al. (1966). As with genetic algorithms, there is debate about the effectiveness of the technique. Koza er al. (1999) describe a variety of experiments on the automated design of circuit devices using genetic programming.

The journals Evolutionary Computation and IEEE Transactions on Evolutionary Computation cover genetic algorithms and genetic programming; articles are also found in Complex Systems, Adaptive Behavior, and Artificial Life. The main conferences are the International Conference on Genetic Algorithms and the Conference on Genetic Programming, recently merged to form the Genetic and Evolutionary Computation Conference. The texts by Melanie Mitchell (1996) and David Fogel (2000) give good overviews of the field.

Algorithms for exploring unknown state spaces have been of interest for many centuries. Depth-first search in a maze can be implemented by keeping one’s left hand on the wall; loops can be avoided by marking each junction. Depth-first search fails with irreversible actions; the more general problem of exploring of Eulerian graphs (i.e., graphs in which each node has equal numbers of incoming and outgoing edges) was solved by an algorithm due to Hierholzer (1873). The first thorough algorithmic study of the exploration problem for arbitrary graphs was carried out by Deng and Papadimitriou (1990), who developed a completely general algorithm, but showed that no bounded competitive ratio is possible for exploring a general graph. Papadimitriou and Yannakakis (1991) examined the question of finding paths to a goal in geometric path-planning environments (where all actions are reversible). They showed that

134

Chapter 4. Informed Search and Exploration

REAL-TIME SEARCH

PARALLEL SEARCH

a small competitive ratio is achievable with square obstacles, but with general rectangular obstacles no bounded ratio can be achieved. (See Figure 4.19.)

The LRTA" algorithm was developed by Korf (1990) as part of an investigation into real-time search for environments in which the agent must act after searching for only a fixed amount of time (a much more common situation in two-player games). LRTA\* is in fact a special case of reinforcement learning algorithms for stochastic environments (Barto et al., 1995). Its policy of optimism under uncertainty—always head for the closest unvisited state—can result in an exploration pattern that is less efficient in the uninformed case than simple depth-first search (Koenig, 2000). Dasgupta er al. (1994) show that online iterative deepening search is optimally efficient for finding a goal in a uniform tree with no heuristic information. Several informed variants on the LRTA\* theme have been developed with different methods for searching and updating within the known portion of the graph (Pemberton and Korf, 1992). As yet, there is no good understanding of how to find goals with optimal efficiency when using heuristic information.

The topic of parallel search algorithms was not covered in the chapter, partly because it requires a lengthy discussion of parallel computer architectures. Parallel search is becoming an important topic in both Al and theoretical computer science. A brief introduction to the Al literature can be found in Mahanti and Daniels (1993).

EXERCISES

4.1 Trace the operation of A\* search applied to the problem of getting to Bucharest from Lugoj using the straight-line distance heuristic. That is, show the sequence of nodes that the algorithm will consider and the f, g, and h score for each node.

4.2 The heuristic path algorithm is a best-first search in which the objective function is f(n) = (2 — w)g(n) + wh(n). For what values of w is this algorithm guaranteed to be optimal? (You may assume that 4 is admissable.) What kind of search does this perform when w = 0? When w = 1? When w = 2?

4.3 Prove each of the following statements:

a. Breadth-first search is a special case of uniform-cost search.

b. Breadth-first search, depth-first search, and uniform-cost search are special cases of best-first search.

c. Uniform-cost search is a special case of A\* search.

4.4 Devise a state space in which A\* using GRAPH-SEARCH returns a suboptimal solution with an h(n) function that is admissible but inconsistent.

4.5 We saw on page 96 that the straight-line distance heuristic leads greedy best-first search astray on the problem of going from Iasi to Fagaras. However, the heuristic is perfect on the opposite problem: going from Fagaras to Iasi. Are there problems for which the heuristic is misleading in both directions?

Section 4.6.

Summary 135

4.6 Invent a heuristic function for the 8-puzzle that sometimes overestimates, and show how it can lead to a suboptimal solution on a particular problem. (You can use a computer to help if you want.) Prove that, if h never overestimates by more than ¢, A\* using A returns a solution whose cost exceeds that of the optimal solution by no more than c.

4.7 Prove that if a heuristic is consistent, it must be admissible. Construct an admissible heuristic that is not consistent.

4.8 The traveling salesperson problem (TSP) can be solved via the minimum spanning tree (MST) heuristic, which is used to estimate the cost of completing a tour, given that a partial tour has already been constructed. The MST cost of a set of cities is the smallest sum of the link costs of any tree that connects all the cities.

a. Show how this heuristic can be derived from a relaxed version of the TSP.

b. Show that the MST heuristic dominates straight-line distance.

c. Write a problem generator for instances of the TSP where cities are represented by random points in the unit square.

d. Find an efficient algorithm in the literature for constructing the MST, and use it with an admissible search algorithm to solve instances of the TSP.

4.9 On page 108, we defined the relaxation of the 8-puzzle in which a tile can move from square A to square B if B is blank. The exact solution of this problem defines Gaschnig’s heuristic (Gaschnig, 1979). Explain why Gaschnig’s heuristic is at least as accurate as hj (misplaced tiles), and show cases where it is more accurate than both kh; and hy (Manhattan distance). Can you suggest a way to calculate Gaschnig’s heuristic efficiently?

4.10 We gave two simple heuristics for the 8-puzzle: Manhattan distance and misplaced tiles. Several heuristics in the literature purport to improve on this—see, for example, Nilsson (1971), Mostow and Prieditis (1989), and Hansson et al. (1992). Test these claims by implementing the heuristics and comparing the performance of the resulting algorithms.

4.11 Give the name of the algorithm that results from each of the following special cases:

. Local beam search with k = 1.

a b. Local beam search with one initial state and no limit on the number of states retained. c. Simulated annealing with 7" = 0 at all times (and omitting the termination test).

d

. Genetic algorithm with population size N = 1.

4.12 Sometimes there is no good evaluation function for a problem, but there is a good comparison method: a way to tell whether one node is better than another, without assigning numerical values to either. Show that this is enough to do a best-first search. Is there an analog of A\*?

4.13 Relate the time complexity of LRTA\* to its space complexity.

4.14 Suppose that an agent is in a 3 x 3 maze environment like the one shown in Figure 4.18. The agent knows that its initial location is (1,1), that the goal is at (3,3), and that the

136

Chapter 4. Informed Search and Exploration

four actions Up, Down, Left, Right have their usual effects unless blocked by a wall. The agent does not know where the internal walls are. In any given state, the agent perceives the set of legal actions; it can also tell whether the state is one it has visited before or a new state.

a. Explain how this online search problem can be viewed as an offline search in belief state space, where the initial belief state includes all possible environment configurations. How large is the initial belief state? How large is the space of belief states?

b. How many distinct percepts are possible in the initial state?

¢. Describe the first few branches of a contingency plan for this problem. How large (roughly) is the complete plan?

Notice that this contingency plan is a solution for every possible environment fitting the given description. Therefore, interleaving of search and execution is not strictly necessary even in unknown environments.

4.15 In this exercise, we will explore the use of local search methods to solve TSPs of the type defined in Exercise 4.8.

a. Devise a hill-climbing approach to solve TSPs. Compare the results with optimal solutions obtained via the A" algorithm with the MST heuristic (Exercise 4.8).

b. Devise a genetic algorithm approach to the traveling salesperson problem. Compare results to the other approaches. You may want to consult Larrafiaga er al. (1999) for some suggestions for representations.

4.16 Generate a large number of 8-puzzle and 8-queens instances and solve them (where possible) by hill climbing (steepest-ascent and first-choice variants), hill climbing with random restart, and simulated annealing. Measure the search cost and percentage of solved problems and graph these against the optimal solution cost. Comment on your results.

4.17 In this exercise, we will examine hill climbing in the context of robot navigation, using the environment in Figure 3.22 as an example.

a. Repeat Exercise 3.16 using hill climbing. Does your agent ever get stuck in a local minimum? Is it possible for it to get stuck with convex obstacles?

b. Construct a nonconvex polygonal environment in which the agent gets stuck.

¢. Modify the hill-climbing algorithm so that, instead of doing a depth-1 search to decide where to go next, it does a depth-£ search. It should find the best k-step path and do one step along it, and then repeat the process.

d. Is there some k for which the new algorithm is guaranteed to escape from local minima?

e. Explain how LRTA" enables the agent to escape from local minima in this case.

4.18 Compare the performance of A\* and RBFS on a set of randomly generated problems in the 8-puzzle (with Manhattan distance) and TSP (with MST—see Exercise 4.8) domains. Discuss your results. What happens to the performance of RBFS when a small random number is added to the heuristic values in the 8-puzzle domain?

CONSTRAINT SATISFACTION PROBLEMS

BLACK BOX

REPRESENTATION

In which we see how treating states as more than just little black boxes leads to the invention of a range of powerful new search methods and a deeper understanding of problem structure and complexity.

Chapters 3 and 4 explored the idea that problems can be solved by searching in a space of states. These states can be evaluated by domain-specific heuristics and tested to see whether they are goal states. From the point of view of the search algorithm, however, each state is a black box with no discernible internal structure. It is represented by an arbitrary data structure that can be accessed only by the problem-specific routines—the successor function, heuristic function, and goal test.

This chapter examines constraint satisfaction problems, whose states and goal test conform to a standard, structured, and very simple representation (Section 5.1). Search algorithms can be defined that take advantage of the structure of states and use general-purpose rather than problem-specific heuristics to enable the solution of large problems (Sections 5.2— 5.3). Perhaps most importantly, the standard representation of the goal test reveals the structure of the problem itself (Section 5.4). This leads to methods for problem decomposition and to an understanding of the intimate connection between the structure of a problem and the difficulty of solving it.

CONSTRAINT SATISFACTION PROBLEM VARIABLES CONSTRAINTS DOMAIN VALUES ASSIGNMENT

CONSISTENT

OBJECTIVE FUNCTION

Formally speaking, a constraint satisfaction problem (or CSP) is defined by a set of variables, X, Xo, ...,X,, and a set of constraints, C1, (Co, ...,C,,. Each variable X; has a nonempty domain D; of possible values. Each constraint C; involves some subset of the variables and specifies the allowable combinations of values for that subset. A state of the problem is defined by an assignment of values to some or all of the variables, {X; = v;, X; = vj,...}. An assignment that does not violate any constraints is called a consistent or legal assignment. A complete assignment is one in which every variable is mentioned, and a solution to a CSP is a complete assignment that satisfies all the constraints. Some CSPs also require a solution that maximizes an objective function.

137

138

Chapter 5. Constraint Satisfaction Problems

CONSTRAINT GRAPH

So what does all this mean? Suppose that, having tired of Romania, we are looking at a map of Australia showing each of its states and territories, as in Figure 5.1(a), and that we are given the task of coloring each region either red, green, or blue in such a way that no neighboring regions have the same color. To formulate this as a CSP, we define the variables to be the regions: WA, NT, @, NSW, V, SA, and T. The domain of each variable is the set {red, green, blue}. The constraints require neighboring regions to have distinct colors; for example, the allowable combinations for WA and NT are the pairs

{(red, green), (red, blue), (green, red), (green, blue), (blue, red), (blue, green)} .

(The constraint can also be represented more succinctly as the inequality WA # NT, provided the constraint satisfaction algorithm has some way to evaluate such expressions.) There are many possible solutions, such as

{WA=red, NT = green,Q = red, NSW = green, V = red, SA = blue, T = red }.

It is helpful to visualize a CSP as a constraint graph, as shown in Figure 5.1(b). The nodes of the graph correspond to variables of the problem and the arcs correspond to constraints.

Treating a problem as a CSP confers several important benefits. Because the representation of states in a CSP conforms to a standard pattern—that is, a set of variables with assigned values—the successor function and goal test can written in a generic way that applies to all CSPs. Furthermore, we can develop effective, generic heuristics that require no additional, domain-specific expertise. Finally, the structure of the constraint graph can be used to simplify the solution process, in some cases giving an exponential reduction in complexity. The CSP representation is the first, and simplest, in a series of representation schemes that will be developed throughout the book.

[]

(a) (b)

Figure 5.1 (a) The principal states and territories of Australia. Coloring this map can be viewed as a constraint satisfaction problem. The goal is to assign colors to each region so that no neighboring regions have the same color. (b) The map-coloring problem represented

as a constraint graph.

Section 5.1.

Constraint Satisfaction Problems 139

FINITE DOMAINS

BOOLEAN CSPS

INFINITE DOMAINS

CONSTRAINT LANGUAGE

LINEAR CONSTRAINTS

NONLINEAR CONSTRAINTS

CONTINUOUS DOMAINS

It is fairly easy to see that a CSP can be given an incremental formulation as a standard search problem as follows:

< Initial state: the empty assignment { }, in which all variables are unassigned.

<> Successor function: a value can be assigned to any unassigned variable, provided that it does not conflict with previously assigned variables.

<> Goal test: the current assignment is complete. <> Path cost: a constant cost (e.g., 1) for every step.

Every solution must be a complete assignment and therefore appears at depth n if there are n variables. Furthermore, the search tree extends only to depth n. For these reasons, depthfirst search algorithms are popular for CSPs. (See Section 5.2.) It is also the case that the path by which a solution is reached is irrelevant. Hence, we can also use a complete-state formulation, in which every state is a complete assignment that might or might not satisfy the constraints. Local search methods work well for this formulation. (See Section 5.3.)

The simplest kind of CSP involves variables that are discrete and have finite domains. Map-coloring problems are of this kind. The 8-queens problem described in Chapter 3 can also be viewed as a finite-domain CSP, where the variables Q1,..., Js are the positions of each queen in columns 1, . .., 8 and each variable has the domain {1, 2, 3,4, 5,6, 7, 8}. If the maximum domain size of any variable in a CSP is d, then the number of possible complete assignments is O(d"™)—that is, exponential in the number of variables. Finite-domain CSPs include Boolean CSPs, whose variables can be either true or false. Boolean CSPs include as special cases some NP-complete problems, such as 3SAT. (See Chapter 7.) In the worst case, therefore, we cannot expect to solve finite-domain CSPs in less than exponential time. In most practical applications, however, general-purpose CSP algorithms can solve problems orders of magnitude larger than those solvable via the general-purpose search algorithms that we saw in Chapter 3.

Discrete variables can also have infinite domains—for example, the set of integers or the set of strings. For example, when scheduling construction jobs onto a calendar, each job's start date is a variable and the possible values are integer numbers of days from the current date. With infinite domains, it is no longer possible to describe constraints by enumerating all allowed combinations of values. Instead, a constraint language must be used. For example, if Joby, which takes five days, must precede Jobs, then we would need a constraint language of algebraic inequalities such as StartJob, + 5 < StartJobs. It is also no longer possible to solve such constraints by enumerating all possible assignments, because there are infinitely many of them. Special solution algorithms (which we will not discuss here) exist for linear constraints on integer variables—that is, constraints, such as the one just given, in which each variable appears only in linear form. It can be shown that no algorithm exists for solving general nonlinear constraints on integer variables. In some cases, we can reduce integer constraint problems to finite-domain problems simply by bounding the values of all the variables. For example, in a scheduling problem, we can set an upper bound equal to the total length of all the jobs to be scheduled.

Constraint satisfaction problems with continuous domains are very common in the real world and are widely studied in the field of operations research. For example, the scheduling

140

Chapter 5. Constraint Satisfaction Problems

LINEAR PROGRAMMING

UNARY CONSTRAINT

BINARY CONSTRAINT

CRYPTARITHMETIC

AUXILIARY VARIABLES

CONSTRAINT HYPERGRAPH

PREFERENCE

of experiments on the Hubble Space Telescope requires very precise timing of observations; the start and finish of each observation and maneuver are continuous-valued variables that must obey a variety of astronomical, precedence, and power constraints. The best-known category of continuous-domain CSPs is that of linear programming problems, where constraints must be linear inequalities forming a convex region. Linear programming problems can be solved in time polynomial in the number of variables. Problems with different types of constraints and objective functions have also been studied—quadratic programming, secondorder conic programming, and so on.

In addition to examining the types of variables that can appear in CSPs, it is useful to look at the types of constraints. The simplest type is the unary constraint, which restricts the value of a single variable. For example, it could be the case that South Australians actively dislike the color green. Every unary constraint can be eliminated simply by preprocessing the domain of the corresponding variable to remove any value that violates the constraint. A binary constraint relates two variables. For example, SA # NSW is a binary constraint. A binary CSP is one with only binary constraints; it can be represented as a constraint graph, as in Figure 5.1(b).

Higher-order constraints involve three or more variables. A familiar example is provided by cryptarithmetic puzzles. (See Figure 5.2(a).) It is usual to insist that each letter in a cryptarithmetic puzzle represent a different digit. For the case in Figure 5.2(a)), this would be represented as the six-variable constraint Alldiff (F,T,U, W, R, O). Alternatively, it can be represented by a collection of binary constraints such as F' # T'. The addition constraints on the four columns of the puzzle also involve several variables and can be written as

O+0=R+10-X; X1+W4+W=U+10-X, Xo0+T+T=0+10-X3 X3=F

where X, Xo, and X3 are auxiliary variables representing the digit (0 or 1) carried over into the next column. Higher-order constraints can be represented in a constraint hypergraph, such as the one shown in Figure 5.2(b). The sharp-eyed reader will have noticed that the Alldiff constraint can be broken down into binary constraints—F # T', F' # U, and so on. In fact, as Exercise 5.11 asks you to prove, every higher-order, finite-domain constraint can be reduced to a set of binary constraints if enough auxiliary variables are introduced. Because of this, we will deal only with binary constraints in this chapter.

The constraints we have described so far have all been absolute constraints, violation of which rules out a potential solution. Many real-world CSPs include preference constraints indicating which solutions are preferred. For example, in a university timetabling problem, Prof. X might prefer teaching in the morning whereas Prof. Y prefers teaching in the afternoon. A timetable that has Prof. X teaching at 2 p.m. would still be a solution (unless Prof. X happens to be the department chair), but would not be an optimal one. Preference constraints can often be encoded as costs on individual variable assignments—for example, assigning an afternoon slot for Prof. X costs 2 points against the overall objective function, whereas a morning slot costs 1. With this formulation, CSPs with preferences can be solved using opti

Section 5.2.

Backtracking Search for CSPs 141

TW O + T7T W O

FOUR

(a) ®)

Figure 5.2 (a) A cryptarithmetic problem. Each letter stands for a distinct digit; the aim is to find a substitution of digits for letters such that the resulting sum is arithmetically correct, with the added restriction that no leading zeroes are allowed. (b) The constraint hypergraph for the cryptarithmetic problem, showing the Alldiff constraint as well as the column addition constraints. Each constraint is a square box connected to the variables it constrains.

mization search methods, either path-based or local. We do not discuss such CSPs further in this chapter, but we provide some pointers in the bibliographical notes section.

COMMUTATIVITY

BACKTRACKING SEARCH

The preceding section gave a formulation of CSPs as search problems. Using this formulation, any of the search algorithms from Chapters 3 and 4 can solve CSPs. Suppose we apply breadth-first search to the generic CSP problem formulation given in the preceding section. We quickly notice something terrible: the branching factor at the top level is nd, because any of d values can be assigned to any of n variables. At the next level, the branching factor is (n— 1)d, and so on for n levels. We generate a tree with n!d" leaves, even though there are only d” possible complete assignments!

Our seemingly reasonable but naive problem formulation has ignored a crucial property common to all CSPs: commutativity. A problem is commutative if the order of application of any given set of actions has no effect on the outcome. This is the case for CSPs because, when assigning values to variables, we reach the same partial assignment, regardless of order. Therefore, all CSP search algorithms generate successors by considering possible assignments for only a single variable at each node in the search tree. For example, at the root node of a search tree for coloring the map of Australia, we might have a choice between SA =red, SA= green, and SA = blue, but we would never choose between SA = red and WA = blue. With this restriction, the number of leaves is d”, as we would hope.

The term backtracking search is used for a depth-first search that chooses values for one variable at a time and backtracks when a variable has no legal values left to assign. The algorithm is shown in Figure 5.3. Notice that it uses, in effect, the one-at-a-time method of

142

Chapter 5. Constraint Satisfaction Problems

function BACKTRACKING-SEARCH( csp) returns a solution, or failure return RECURSIVE-BACKTRACKING({ }, csp)

function RECURSIVE-BACKTRACKING(assignment, csp) returns a solution, or failure

if assignment is complete then return assignment

var «+ SELECT-UNASSIGNED-VARIABLE(VARIABLES( csp], assignment, csp)

for each value in ORDER-DOMAIN-VALUES(var, assignment, csp) do

if value is consistent with assignment according to CONSTRAINTS[ csp] then

add {var = value} to assignment result «— RECURSIVE-BACKTRACKING(assignment, csp) if result # failure then return result remove {var = value} from assignment

return failure

Figure 5.3 A simple backtracking algorithm for constraint satisfaction problems. The algorithm is modeled on the recursive depth-first search of Chapter 3. The functions SELECT-UNASSIGNEDVARIABLE and ORDER-DOMAIN-VALUES can be used to implement the general-purpose heuristics discussed in the text.

WA=blue

WA=red WA=red NT=green NT=blue — To WA=red WA=red NT=green NT=green Q=red Q=blue | — TT —

Figure 5.4 Part of the search tree generated by simple backtracking for the map-coloring problem in Figure 5.1.

incremental successor generation described on page 76. Also, it extends the current assignment to generate a successor, rather than copying it. Because the representation of CSPs is standardized, there is no need to supply BACKTRACKING-SEARCH with a domain-specific initial state, successor function, or goal test. Part of the search tree for the Australia problem is shown in Figure 5.4, where we have assigned variables in the order WA, NT',Q), .. ..

Plain backtracking is an uninformed algorithm in the terminology of Chapter 3, so we do not expect it to be very effective for large problems. The results for some sample problems are shown in the first column of Figure 5.5 and confirm our expectations.

In Chapter 4 we remedied the poor performance of uninformed search algorithms by supplying them with domain-specific heuristic functions derived from our knowledge of the problem. It turns out that we can solve CSPs efficiently without such domain-specific knowl

Section 5.2.

Backtracking Search for CSPs 143

[]

MINIMUM REMAINING VALUES

Figure 5.5 Comparison of various CSP algorithms on various problems. The algorithms from left to right, are simple backtracking, backtracking with the MRV heuristic, forward checking, forward checking with MRV, and minimum conflicts local search. Listed in each cell is the median number of consistency checks (over five runs) required to solve the problem; note that all entries except the two in the upper right are in thousands (K). Numbers in parentheses mean that no answer was found in the allotted number of checks. The first problem is finding a 4-coloring for the 50 states of the United States of America. The remaining problems are taken from Bacchus and van Run (1995), Table 1. The second problem counts the total number of checks required to solve all n-Queens problems for nn from 2 to 50. The third is the “Zebra Puzzle,” as described in Exercise 5.13. The last two are artificial random problems. (Min-conflicts was not run on these.) The results suggest that forward checking with the MRV heuristic is better on all these problems than the other backtracking algorithms,

but not always better than min-conflicts local search.

edge. Instead, we find general-purpose methods that address the following questions:

1. Which variable should be assigned next, and in what order should its values be tried?

2. What are the implications of the current variable assignments for the other unassigned variables?

3. When a path fails—that is, a state is reached in which a variable has no legal values— can the search avoid repeating this failure in subsequent paths?

The subsections that follow answer each of these questions in turn.

Variable and value ordering

The backtracking algorithm contains the line

var «+ SELECT-UNASSIGNEDVARIABLE(VARIABLES|[ csp], assignment, csp).

By default, SELECT-UNASSIGNEDVARIABLE simply selects the next unassigned variable in the order given by the list VARIABLES| csp]. This static variable ordering seldom results in the most efficient search. For example, after the assignments for WA = red and NT = green, there is only one possible value for SA, so it makes sense to assign SA = blue next rather than assigning (). In fact, after SA is assigned, the choices for QQ, NSW, and V are all forced. This intuitive idea—choosing the variable with the fewest “legal” values—is called the minimum remaining values (MRV) heuristic. It also has been called the “most constrained variable” or “fail-first” heuristic, the latter because it picks a variable that is most likely to cause a failure soon, thereby pruning the search tree. If there is a variable X with zero legal values remaining, the MRV heuristic will select X and failure will be detected immediately—avoiding pointless searches through other variables which always will fail when X is finally selected.

144

Chapter 5. Constraint Satisfaction Problems

DEGREE HEURISTIC

LEASTCONSTRAININGVALUE

FORWARD CHECKING

The second column of Figure 5.5, labeled BT+MRYV, shows the performance of this heuristic. The performance is 3 to 3,000 times better than simple backtracking, depending on the problem. Note that our performance measure ignores the extra cost of computing the heuristic values; the next subsection describes a method that makes this cost manageable.

The MRYV heuristic doesn’t help at all in choosing the first region to color in Australia, because initially every region has three legal colors. In this case, the degree heuristic comes in handy. It attempts to reduce the branching factor on future choices by selecting the variable that is involved in the largest number of constraints on other unassigned variables. In Figure 5.1, SA is the variable with highest degree, 5; the other variables have degree 2 or 3, except for T', which has 0. In fact, once SA is chosen, applying the degree heuristic solves the problem without any false steps—you can choose any consistent color at each choice point and still arrive at a solution with no backtracking. The minimum remaining values heuristic is usually a more powerful guide, but the degree heuristic can be useful as a tie-breaker.

Once a variable has been selected, the algorithm must decide on the order in which to examine its values. For this, the least-constraining-value heuristic can be effective in some cases. It prefers the value that rules out the fewest choices for the neighboring variables in the constraint graph. For example, suppose that in Figure 5.1 we have generated the partial assignment with WA = red and NT = green, and that our next choice is for . Blue would be a bad choice, because it eliminates the last legal value left for (’s neighbor, SA. The least-constraining-value heuristic therefore prefers red to blue. In general, the heuristic is trying to leave the maximum flexibility for subsequent variable assignments. Of course, if we are trying to find all the solutions to a problem, not just the first one, then the ordering does not matter because we have to consider every value anyway. The same holds if there are no solutions to the problem.

Propagating information through constraints

So far our search algorithm considers the constraints on a variable only at the time that the variable is chosen by SELECT-UNASSIGNED-VARIABLE. But by looking at some of the constraints earlier in the search, or even before the search has started, we can drastically reduce the search space.

Forward checking

One way to make better use of constraints during search is called forward checking. Whenever a variable X is assigned, the forward checking process looks at each unassigned variable Y that is connected to X by a constraint and deletes from Y’s domain any value that is inconsistent with the value chosen for X. Figure 5.6 shows the progress of a map-coloring search with forward checking. There are two important points to notice about this example. First, notice that after assigning WA = red and QQ = green, the domains of NT" and SA are reduced to a single value; we have eliminated branching on these variables altogether by propagating information from WA and J. The MRV heuristic, which is an obvious partner for forward checking, would automatically select SA and NT next. (Indeed, we can view forward checking as an efficient way to incrementally compute the information that the

Section 5.2.

Backtracking Search for CSPs 145

CONSTRAINT PROPAGATION

ARC CONSISTENCY

WA NT 0 NSW v SA T Initial domains | RG B|RG B|RG B|RG B|RGB|RG BIR GB After WA=red |® GB|RGB|RGB|RGB| GB|RGB After Q=green |® Bl © [rR B[RGB B|[RG B After V=blue |® Bl © |r RGB

Figure 5.6 The progress of a map-coloring search with forward checking. WA = red is assigned first; then forward checking deletes red from the domains of the neighboring variables NT" and SA. After (Q = green, green is deleted from the domains of NT, SA, and NSW. After V = blue, blue is deleted from the domains of NSW and SA, leaving SA with no legal values.

MRYV heuristic needs to do its job.) A second point to notice is that, after V = blue, the domain of SA is empty. Hence, forward checking has detected that the partial assignment {WA =red,Q = green, V = blue} is inconsistent with the constraints of the problem, and the algorithm will therefore backtrack immediately.

Constraint propagation

Although forward checking detects many inconsistencies, it does not detect all of them. For example, consider the third row of Figure 5.6. It shows that when WA is red and Q) is green, both NT and SA are forced to be blue. But they are adjacent and so cannot have the same value. Forward checking does not detect this as an inconsistency, because it does not look far enough ahead. Constraint propagation is the general term for propagating the implications of a constraint on one variable onto other variables; in this case we need to propagate from WA and Q onto NT and SA, (as was done by forward checking) and then onto the constraint between NT" and SA to detect the inconsistency. And we want to do this fast: it is no good reducing the amount of search if we spend more time propagating constraints than we would have spent doing a simple search.

The idea of arc consistency provides a fast method of constraint propagation that is substantially stronger than forward checking. Here, “arc” refers to a directed arc in the constraint graph, such as the arc from SA to NSW. Given the current domains of SA and NSW, the arc is consistent if, for every value x of SA, there is some value y of NSW that is consistent with z. In the third row of Figure 5.6, the current domains of SA and NSW are {blue} and {red, blue} respectively. For SA = blue, there is a consistent assignment for NSW, namely, NSW = red; therefore, the arc from SA to NSW is consistent. On the other hand, the reverse arc from NSW to SA is not consistent: for the assignment NSW = blue, there is no consistent assignment for SA. The arc can be made consistent by deleting the value blue from the domain of NST.

We can also apply arc consistency to the arc from SA to NT at the same stage in the search process. The third row of the table in Figure 5.6 shows that both variables have the domain {blue}. The result is that blue must be deleted from the domain of SA, leaving the domain empty. Thus, applying arc consistency has resulted in early detection of an inconsis

146

Chapter 5. Constraint Satisfaction Problems

function AC-3( csp) returns the CSP, possibly with reduced domains inputs: csp, a binary CSP with variables { X;, Xo, ..., X,} local variables: queue, a queue of arcs, initially all the arcs in csp

while gueue is not empty do (X;, X;) <— REMOVE-FIRST(queue) if REMOVE-INCONSISTENT-VALUES(X;, X;) then for each X;. in NEIGHBORS[X;] do add (Xj, X;) to queue

function REMOVE-INCONSISTENTVALUES (X;, X;) returns true iff we remove a value removed + false for each z in DOMAIN[X;] do if no value y in DOMAIN[X] allows (x,y) to satisfy the constraint between X, and X; then delete z from DOMAIN[X;]; removed « true return removed

Figure 5.7 The arc consistency algorithm AC-3. After applying AC-3, either every arc is arc-consistent, or some variable has an empty domain, indicating that the CSP cannot be made arc-consistent (and thus the CSP cannot be solved). The name “AC-3" was used by the algorithm’s inventor (Mackworth, 1977) because it’s the third version developed in the paper.

tency that is not detected by pure forward checking.

Arc consistency checking can be applied either as a preprocessing step before the beginning of the search process, or as a propagation step (like forward checking) after every assignment during search. (The latter algorithm is sometimes called MAC, for Maintaining Arc Consistency.) In either case, the process must be applied repeatedly until no more inconsistencies remain. This is because, whenever a value is deleted from some variable’s domain to remove an arc inconsistency, a new arc inconsistency could arise in arcs pointing to that variable. The full algorithm for arc consistency, AC-3, uses a queue to keep track of the arcs that need to be checked for inconsistency. (See Figure 5.7.) Each arc (Xj, X;) in turn is removed from the agenda and checked; if any values need to be deleted from the domain of Xi, then every arc (Xk, X;) pointing to X; must be reinserted on the queue for checking. The complexity of arc consistency checking can be analyzed as follows: a binary CSP has at most O(n?) arcs; each arc (Xj, X;) can be inserted on the agenda only d times, because X; has at most d values to delete; checking consistency of an arc can be done in O(d?) time; so the total worst-case time is O(n2d?). Although this is substantially more expensive than forward checking, the extra cost is usually worthwhile.

Because CSPs include 3SAT as a special case, we do not expect to find a polynomialtime algorithm that can decide whether a given CSP is consistent. Hence, we deduce that arc consistency does not reveal every possible inconsistency. For example, in Figure 5.1, the partial assignment { WA = red, NSW = red} is inconsistent, but AC-3 will not find the incon

1 The AC-4 algorithm, due to Mohr and Henderson (1986), runs in O(n2d?). See Exercise 5.10.

Section 5.2.

Backtracking Search for CSPs 147

K-CONSISTENCY

NODE CONSISTENCY

PATH CONSISTENCY

STRONGLY K-CONSISTENT

sistency. Stronger forms of propagation can be defined using the notion called k-consistency. A CSP is k-consistent if, for any set of & — 1 variables and for any consistent assignment to those variables, a consistent value can always be assigned to any kth variable. For example, 1-consistency means that each individual variable by itself is consistent; this is also called node consistency. 2-consistency is the same as arc consistency. 3-consistency means that any pair of adjacent variables can always be extended to a third neighboring variable; this is also called path consistency.

A graph is strongly k-consistent if it is k-consistent and is also (k — 1)-consistent, (k—2)-consistent, . . . all the way down to 1-consistent. Now suppose we have a CSP problem with n nodes and make it strongly n-consistent (i.e., strongly k-consistent for k =n). We can then solve the problem with no backtracking. First, we choose a consistent value for X;. We are then guaranteed to be able to choose a value for Xs because the graph is 2-consistent, for X3 because it is 3-consistent, and so on. For each variable X;, we need only search through the d values in the domain to find a value consistent with X1,..., X;\_;. We are guaranteed to find a solution in time O(nd). Of course, there is no free lunch: any algorithm for establishing n-consistency must take time exponential in n in the worst case.

There is a broad middle ground between n-consistency and arc consistency: running stronger consistency checks will take more time, but will have a greater effect in reducing the branching factor and detecting inconsistent partial assignments. It is possible to calculate the smallest value & such that running k-consistency ensures that the problem can be solved without backtracking (see Section 5.4), but this is often impractical. In practice, determining the appropriate level of consistency checking is mostly an empirical science.

Handling special constraints

Certain types of constraints occur frequently in real problems and can be handled using special-purpose algorithms that are more efficient than the general-purpose methods described so far. For example, the Alldiff constraint says that all the variables involved must have distinct values (as in the cryptarithmetic problem). One simple form of inconsistency detection for Alldiff constraints works as follows: if there are m variables involved in the constraint, and if they have n possible distinct values altogether, and m > n, then the constraint cannot be satisfied.

This leads to the following simple algorithm: First, remove any variable in the constraint that has a singleton domain, and delete that variable’s value from the domains of the remaining variables. Repeat as long as there are singleton variables. If at any point an empty domain is produced or there are more variables than domain values left, then an inconsistency has been detected.

We can use this method to detect the inconsistency in the partial assignment { WA = red, NSW = red} for Figure 5.1. Notice that the variables SA, NT, and @ are effectively connected by an Alldiff constraint because each pair must be a different color. After applying AC-3 with the partial assignment, the domain of each variable is reduced to {green, blue}. That is, we have three variables and only two colors, so the Alldiff constraint is violated. Thus, a simple consistency procedure for a higher-order constraint is sometimes more effec

148

Chapter 5. Constraint Satisfaction Problems

RESOURCE CONSTRAINT

BOUNDS PROPAGATION

CHRONOLOGICAL BACKTRACKING

CONFLICT SET

BACKJUMPING

tive than applying arc consistency to an equivalent set of binary constraints.

Perhaps the most important higher-order constraint is the resource constraint, sometimes called the atmost constraint. For example, let PA4,..., PA4 denote the numbers of personnel assigned to each of four tasks. The constraint that no more than 10 personnel are assigned in total is written as atmost(10, PA;, PAs, PAs, PA4). An inconsistency can be detected simply by checking the sum of the minimum values of the current domains; for example, if each variable has the domain {3,4, 5,6}, the atmost constraint cannot be satisfied. We can also enforce consistency by deleting the maximum value of any domain if it is not consistent with the minimum values of the other domains. Thus, if each variable in our example has the domain {2, 3,4, 5,6}, the values 5 and 6 can be deleted from each domain.

For large resource-limited problems with integer values—such as logistical problems involving moving thousands of people in hundreds of vehicles—it is usually not possible to represent the domain of each variable as a large set of integers and gradually reduce that set by consistency checking methods. Instead, domains are represented by upper and lower bounds and are managed by bounds propagation. For example, let’s suppose there are two flights, 271 and 272, for which the planes have capacities 165 and 385, respectively. The initial domains for the numbers of passengers on each flight are then

Flight271 € [0,165] and Flight272 € [0,385] .

Now suppose we have the additional constraint that the two flights together must carry 420 people: Flight271 + Flight272 € [420,420]. Propagating bounds constraints, we reduce the domains to

Flight271 € [35,165] and Flight272 € [255,385] .

We say that a CSP is bounds-consistent if for every variable X, and for both the lower bound and upper bound values of X, there exists some value of Y that satisfies the constraint between X and Y, for every variable Y. This kind of bounds propagation is widely used in practical constraint problems.

Intelligent backtracking: looking backward

The BACKTRACKING-SEARCH algorithm in Figure 5.3 has a very simple policy for what to do when a branch of the search fails: back up to the preceding variable and try a different value for it. This is called chronological backtracking, because the most recent decision point is revisited. In this subsection, we will see that there are much better ways.

Consider what happens when we apply simple backtracking in Figure 5.1 with a fixed variable ordering Q, NSW, V, T, SA, WA, NT. Suppose we have generated the partial assignment {Q = red, NSW = green, V = blue, T = red}. When we try the next variable, SA, we see that every value violates a constraint. We back up to 7" and try a new color for Tasmania! Obviously this is silly—recoloring Tasmania cannot resolve the problem with South Australia.

A more intelligent approach to backtracking is to go all the way back to one of the set of variables that caused the failure. This set is called the conflict set; here, the conflict set for SA is {Q, NSW ,V}. In general, the conflict set for variable X is the set of previously assigned variables that are connected to X by constraints. The backjumping method

Section 5.2.

Backtracking Search for CSPs 149

CONFLICT-DIRECTED BACKJUMPING

backtracks to the most recent variable in the conflict set; in this case, backjumping would jump over Tasmania and try a new value for VV. This is easily implemented by modifying BACKTRACKING-SEARCH so that it accumulates the conflict set while checking for a legal value to assign. If no legal value is found, it should return the most recent element of the conflict set along with the failure indicator.

The sharp-eyed reader will have noticed that forward checking can supply the conflict set with no extra work: whenever forward checking based on an assignment to X deletes a value from Y’s domain, it should add X to Ys conflict set. Also, every time the last value is deleted from Y's domain, the variables in the conflict set of Y are added to the conflict set of X. Then, when we get to Y, we know immediately where to backtrack if needed.

The eagle-eyed reader will have noticed something odd: backjumping occurs when every value in a domain is in conflict with the current assignment; but forward checking detects this event and prevents the search from ever reaching such a node! In fact, it can be shown that every branch pruned by backjumping is also pruned by forward checking. Hence, simple backjumping is redundant in a forward-checking search or, indeed, in a search that uses stronger consistency checking, such as MAC.

Despite the observations of the preceding paragraph, the idea behind backjumping remains a good one: to backtrack based on the reasons for failure. Backjumping notices failure when a variable’s domain becomes empty, but in many cases a branch is doomed long before this occurs. Consider again the partial assignment { WA = red, NSW = red } (which, from our earlier discussion, is inconsistent). Suppose we try 7" = red next and then assign NT, Q, V, SA. We know that no assignment can work for these last four variables, so eventually we run out of values to try at N71". Now, the question is, where to backtrack? Backjumping cannot work, because NT does have values consistent with the preceding assigned variables—NT' doesn’t have a complete conflict set of preceding variables that caused it to fail. We know, however, that the four variables NT’, Q, VV, and SA, taken together, failed because of a set of preceding variables, which must be those variables which directly conflict with the four. This leads to a deeper notion of the conflict set for a variable such as NT": it is that set of preceding variables that caused NT, together with any subsequent variables, to have no consistent solution. In this case, the set is WA and NSW, so the algorithm should backtrack to NSW and skip over Tasmania. A backjumping algorithm that uses conflict sets defined in this way is called conflict-directed backjumping.

We must now explain how these new conflict sets are computed. The method is in fact very simple. The “terminal” failure of a branch of the search always occurs because a variable’s domain becomes empty; that variable has a standard conflict set. In our example, SA fails, and its conflict set is (say) { WA, NT, Q}. We backjump to @, and @Q absorbs the conflict set from SA (minus @ itself, of course) into its own direct conflict set, which is {NT, NSW}, the new conflict set is { WA, NT, NSW}. That is, there is no solution from @Q onwards, given the preceding assignment to { WA, NT, NSW }. Therefore, we backtrack to NT, the most recent of these. NT absorbs { WA, NT, NSW} — {NT} into its own direct conflict set { WA}, giving { WA, NSW} (as stated in the previous paragraph). Now the algorithm backjumps to NSW, as we would hope. To summarize: let X; be the current variable, and let conf (X;) be its conflict set. If every possible value for X; fails, backjump

150 Chapter 5. Constraint Satisfaction Problems

to the most recent variable X; in conf (X;), and set conf (X;) — conf (X;) U conf(X;) — {Xi}. Conflict-directed backjumping takes us back to the right point in the search tree, but doesn’t

CONSTRAINT prevent us from making the same mistakes in another branch of the tree. Constraint learning actually modifies the CSP by adding a new constraint that is induced from these conflicts.

5.3 LoCAL SEARCH FOR CONSTRAINT SATISFACTION PROBLEMS

Local-search algorithms (see Section 4.3) turn out to be very effective in solving many CSPs. They use a complete-state formulation: the initial state assigns a value to every variable, and the successor function usually works by changing the value of one variable at a time. For example, in the 8-queens problem, the initial state might be a random configuration of 8 queens in 8 columns, and the successor function picks one queen and considers moving it elsewhere in its column. Another possibility would be start with the 8 queens, one per column in a permutation of the 8 rows, and to generate a successor by having two queens swap rows.’ We have actually already seen an example of local search for CSP solving: the application of hill climbing to the n-queens problem (page 112). The application of WALKS AT (page 223) to solve satisfiability problems, which are a special case of CSPs, is another.

In choosing a new value for a variable, the most obvious heuristic is to select the value

MIN-CONFLICTS that results in the minimum number of conflicts with other variables—the min-conflicts heuristic. The algorithm is shown in Figure 5.8 and its application to an 8-queens problem is diagrammed in Figure 5.9 and quantified in Figure 5.5.

Min-conflicts is surprisingly effective for many CSPs, particularly when given a reasonable initial state. Its performance is shown in the last column of Figure 5.5. Amazingly, on the n-queens problem, if you don’t count the initial placement of queens, the runtime of minconflicts is roughly independent of problem size. It solves even the million-queens problem in an average of 50 steps (after the initial assignment). This remarkable observation was the stimulus leading to a great deal of research in the 1990s on local search and the distinction between easy and hard problems, which we take up in Chapter 7. Roughly speaking, n-queens is easy for local search because solutions are densely distributed throughout the state space. Min-conflicts also works well for hard problems. For example, it has been used to schedule observations for the Hubble Space Telescope, reducing the time taken to schedule a week of observations from three weeks (!) to around 10 minutes.

Another advantage of local search is that it can be used in an online setting when the problem changes. This is particularly important in scheduling problems. A week’s airline schedule may involve thousands of flights and tens of thousands of personnel assignments, but bad weather at one airport can render the schedule infeasible. We would like to repair the schedule with a minimum number of changes. This can be easily done with a local search algorithm starting from the current schedule. A backtracking search with the new set of

2 Local search can easily be extended to CSPs with objective functions. In that case, all the techniques for hill climbing and simulated annealing can be applied to optimize the objective function.

Section 5.4.

The Structure of Problems 151

function MIN-CONFLICTS(csp, maz steps) returns a solution or failure inputs: csp, a constraint satisfaction problem maz \_steps, the number of steps allowed before giving up

current + an initial complete assignment for csp for i = 1 to max \_steps do if current is a solution for csp then return current var «a randomly chosen, conflicted variable from VARIABLES] csp] value « the value v for var that minimizes CONFLICTS(var, v, current, csp) set var = value in current return failure

Figure 5.8 The MIN-CONFLICTS algorithm for solving CSPs by local search. The initial state may be chosen randomly or by a greedy assignment process that chooses a minimalconflict value for each variable in turn. The CONFLICTS function counts the number of constraints violated by a particular value, given the rest of the current assignment.

[]

Figure 5.9 A two-step solution for an 8-queens problem using min-conflicts. At each stage, a queen is chosen for reassignment in its column. The number of conflicts (in this case, the number of attacking queens) is shown in each square. The algorithm moves the queen to the min-conflict square, breaking ties randomly.

constraints usually requires much more time and might find a solution with many changes from the current schedule.

In this section, we examine ways in which the structure of the problem, as represented by the constraint graph, can be used to find solutions quickly. Most of the approaches here are very general and are applicable to other problems besides CSPs, for example probabilistic reasoning. After all, the only way we can possibly hope to deal with the real world is to decompose it into many subproblems. Looking again at Figure 5.1(b) with a view to identifying problem

152

Chapter 5. Constraint Satisfaction Problems

INDEPENDENT SUBPROBLEMS

CONNECTED COMPONENTS

structure, one fact stands out: Tasmania is not connected to the mainland.? Intuitively, it is obvious that coloring Tasmania and coloring the mainland are independent subproblems—any solution for the mainland combined with any solution for Tasmania yields a solution for the whole map. Independence can be ascertained simply by looking for connected components of the constraint graph. Each component corresponds to a subproblem CSP;. If assignment S; is a solution of CSP;, then | J; S; is a solution of | J; CSP;. Why is this important? Consider the following: suppose each CSP; has c¢ variables from the total of n variables, where c is a constant. Then there are n/c subproblems, each of which takes at most d° work to solve. Hence, the total work is O(d“n/c), which is linear in n; without the decomposition, the total work is O(d™), which is exponential in 72. Let’s make this more concrete: dividing a Boolean CSP with n = 80 into four subproblems with ¢ = 20 reduces the worst-case solution time from the lifetime of the universe down to less than a second.

Completely independent subproblems are delicious, then, but rare. In most cases, the subproblems of a CSP are connected. The simplest case is when the constraint graph forms a tree: any two variables are connected by at most one path. Figure 5.10(a) shows a schematic example. We will show that any tree-structured CSP can be solved in time linear in the number of variables. The algorithm has the following steps:

1. Choose any variable as the root of the tree, and order the variables from the root to the leaves in such a way that every node’s parent in the tree precedes it in the ordering. (See Figure 5.10(b).) Label the variables X7,..., X,, in order. Now, every variable except the root has exactly one parent variable.

2. For j from n down to 2, apply arc consistency to the arc (X;, X;), where Xj; is the parent of X;, removing values from DOMAIN[.X;] as necessary.

3. For j from 1 to n, assign any value for X; consistent with the value assigned for Xj, where X; is the parent of X;.

There are two key points to note. First, after step 2 the CSP is directionally arc-consistent, so the assignment of values in step 3 requires no backtracking. (See the discussion of kconsistency on page 147.) Second, by applying the arc-consistency checks in reverse order in step 2, the algorithm ensures that any deleted values cannot endanger the consistency of arcs that have been processed already. The complete algorithm runs in time O(nd?).

Now that we have an efficient algorithm for trees, we can consider whether more general constraint graphs can be reduced to trees somehow. There are two primary ways to do this, one based on removing nodes and one based on collapsing nodes together.

The first approach involves assigning values to some variables so that the remaining variables form a tree. Consider the constraint graph for Australia, shown again in Figure 5.11(a). If we could delete South Australia, the graph would become a tree, as in (b). Fortunately, we can do this (in the graph, not the continent) by fixing a value for SA and deleting from the domains of the other variables any values that are inconsistent with the value chosen for SA.

3 A careful cartographer or patriotic Tasmanian might object that Tasmania should not be colored the same as

its nearest mainland neighbor, to avoid the impression that it might be part of that state. 4 Sadly, very few regions of the world, with the possible exception of Sulawesi, have tree-structured maps.

Section 5.4. The Structure of Problems 153

(a) (F) (b)

Figure 5.10 (a) The constraint graph of a tree-structured CSP. (b) A linear ordering of the variables consistent with the tree with A as the root.

(b)

Figure 5.11 (a) The original constraint graph from Figure 5.1. (b) The constraint graph after the removal of SA.

Now, any solution for the CSP after SA and its constraints are removed will be consistent with the value chosen for SA. (This works for binary CSPs; the situation is more complicated with higher-order constraints.) Therefore, we can solve the remaining tree with the algorithm given above and thus solve the whole problem. Of course, in the general case (as opposed to map coloring) the value chosen for SA could be the wrong one, so we would need to try each of them. The general algorithm is as follows:

1. Choose a subset S from VARIABLES[ csp] such that the constraint graph becomes a tree CYCLE CUTSET after removal of S. S' is called a cycle cutset. 2. For each possible assignment to the variables in S' that satisfies all constraints on 5, (a) remove from the domains of the remaining variables any values that are inconsistent with the assignment for 5, and (b) If the remaining CSP has a solution, return it together with the assignment for S.

If the cycle cutset has size c, then the total runtime is O(d° (n — ¢)d?). If the graph is “nearly a tree” then ¢ will be small and the savings over straight backtracking will be huge.

154

Chapter 5. Constraint Satisfaction Problems

CUTSET CONDITIONING

TREE DECOMPOSITION

In the worst case, however, ¢ can be as large as (n — 2). Finding the smallest cycle cutset is NP-hard, but several efficient approximation algorithms are known for this task. The overall algorithmic approach is called cutset conditioning; we will see it again in Chapter 14, where it is used for reasoning about probabilities.

The second approach is based on constructing a tree decomposition of the constraint graph into a set of connected subproblems. Each subproblem is solved independently, and the resulting solutions are then combined. Like most divide-and-conquer algorithms, this works well if no subproblem is too large. Figure 5.12 shows a tree decomposition of the mapcoloring problem into five subproblems. A tree decomposition must satisfy the following three requirements:

e Every variable in the original problem appears in at least one of the subproblems.

e If two variables are connected by a constraint in the original problem, they must appear together (along with the constraint) in at least one of the subproblems.

eo If a variable appears in two subproblems in the tree, it must appear in every subproblem along the path connecting those subproblems.

The first two conditions ensure that all the variables and constraints are represented in the decomposition. The third condition seems rather technical, but simply reflects the constraint that any given variable must have the same value in every subproblem in which it appears; the links joining subproblems in the tree enforce this constraint. For example, SA appears in all four of the connected subproblems in Figure 5.12. You can verify from Figure 5.11 that this decomposition makes sense.

Figure 5.12 A tree decomposition of the constraint graph in Figure 5.11(a).

We solve each subproblem independently; if any one has no solution, we know the entire problem has no solution. If we can solve all the subproblems, then we attempt to construct

Section 5.5.

Summary 155

TREE WIDTH

a global solution as follows. First, we view each subproblem as a “mega-variable” whose domain is the set of all solutions for the subproblem. For example, the leftmost subproblems in Figure 5.12 is a map-coloring problem with three variables and hence has six solutions—one is {WA = red, SA = blue, NT = green}. Then, we solve the constraints connecting the subproblems using the efficient algorithm for trees given earlier. The constraints between subproblems simply insist that the subproblem solutions agree on their shared variables. For example, given the solution { WA = red, SA = blue, NT = green} for the first subproblem, the only consistent solution for the next subproblem is { SA = blue, NT = green, Q = red}. A given constraint graph admits many tree decompositions; in choosing a decomposition, the aim is to make the subproblems as small as possible. The tree width of a tree decomposition of a graph is one less than the size of the largest subproblem; the tree width of the graph itself is defined to be the minimum tree width among all its tree decompositions. If a graph has tree width w, and we are given the corresponding tree decomposition, then the problem can be solved in O(nd“\*!) time. Hence, CSPs with constraint graphs of bounded tree width are solvable in polynomial time. Unfortunately, finding the decomposition with minimal tree width is NP-hard, but there are heuristic methods that work well in practice.

¢ Constraint satisfaction problems (or CSPs) consist of variables with constraints on them. Many important real-world problems can be described as CSPs. The structure of a CSP can be represented by its constraint graph.

e Backtracking search, a form of depth-first search, is commonly used for solving CSPs.

¢ The minimum remaining values and degree heuristics are domain-independent methods for deciding which variable to choose next in a backtracking search. The leastconstraining-value heuristic helps in ordering the variable values.

e By propagating the consequences of the partial assignments that it constructs, the backtracking algorithm can reduce greatly the branching factor of the problem. Forward checking is the simplest method for doing this. Arc consistency enforcement is a more powerful technique, but can be more expensive to run.

e Backtracking occurs when no legal assignment can be found for a variable. Conflictdirected backjumping backtracks directly to the source of the problem.

e Local search using the min-conflicts heuristic has been applied to constraint satisfaction problems with great success.

e The complexity of solving a CSP is strongly related to the structure of its constraint graph. Tree-structured problems can be solved in linear time. Cutset conditioning can reduce a general CSP to a tree-structured one and is very efficient if a small cutset can be found. Tree decomposition techniques transform the CSP into a tree of subproblems and are efficient if the tree width of the constraint graph is small.

156

Chapter 5. Constraint Satisfaction Problems

DIOPHANTINE EQUATIONS

GRAPH COLORING

The earliest work related to constraint satisfaction dealt largely with numerical constraints. Equational constraints with integer domains were studied by the Indian mathematician Brahmagupta in the seventh century; they are often called Diophantine equations, after the Greek mathematician Diophantus (c. 200-284), who actually considered the domain of positive rationals. Systematic methods for solving linear equations by variable elimination were studied by Gauss (1829); the solution of linear inequality constraints goes back to Fourier (1827).

Finite-domain constraint satisfaction problems also have a long history. For example, graph coloring (of which map coloring is a special case) is an old problem in mathematics. According to Biggs et al. (1986), the four-color conjecture (that every planar graph can be colored with four or fewer colors) was first made by Francis Guthrie, a student of De Morgan, in 1852. It resisted solution—despite several published claims to the contrary—until a proof was devised, with the aid of a computer, by Appel and Haken (1977).

Specific classes of constraint satisfaction problems occur throughout the history of computer science. One of the most influential early examples was the SKETCHPAD system (Sutherland, 1963), which solved geometric constraints in diagrams and was the forerunner of modern drawing programs and CAD tools. The identification of CSPs as a general class is due to Ugo Montanari (1974). The reduction of higher-order CSPs to purely binary CSPs with auxiliary variables (see Exercise 5.11) is due originally to the 19th-century logician Charles Sanders Peirce. It was introduced into the CSP literature by Dechter (1990b) and was elaborated by Bacchus and van Beek (1998). CSPs with preferences among solutions are studied widely in the optimization literature; see Bistarelli ef al. (1997) for a generalization of the CSP framework to allow for preferences. The bucket-elimination algorithm (Dechter, 1999) can also be applied to optimization problems.

Backtracking search for constraint satisfaction is due to Bitner and Reingold (1975), although they trace the basic algorithm back to the 19th century. Bitner and Reingold also introduced the MRV heuristic, which they called the most-constrained-variable heuristic. Brelaz (1979) used the degree heuristic as a tie-breaker after applying the MRV heuristic. The resulting algorithm, despite its simplicity, is still the best method for k-coloring arbitrary graphs. Haralick and Elliot (1980) proposed the least-constraining-value heuristic.

Constraint propagation methods were popularized by Waltz’s (1975) success on polyhedral line-labeling problems for computer vision. Waltz showed that, in many problems, propagation completely eliminates the need for backtracking. Montanari (1974) introduced the notion of constraint networks and propagation by path consistency. Alan Mackworth (1977) proposed the AC-3 algorithm for enforcing arc consistency as well as the general idea of combining backtracking with some degree of consistency enforcement. AC-4, a more efficient arc consistency algorithm, was developed by Mohr and Henderson (1986). Soon after Mackworth’s paper appeared, researchers began experimenting with the tradeoff between the cost of consistency enforcement and the benefits in terms of search reduction. Haralick and Elliot (1980) favored the minimal forward checking algorithm described by McGregor (1979), whereas Gaschnig (1979) suggested full arc consistency checking after each variable

Summary 157

DEPENDENCY DIRECTED BACKTRACKING

CONSTRAINT RECORDING

BACKMARKING

DYNAMIC BACKTRACKING

assignment—an algorithm later called MAC by Sabin and Freuder (1994). The latter paper provides somewhat convincing evidence that, on harder CSPs, full arc consistency checking pays off. Freuder (1978, 1982) investigated the notion of k-consistency and its relationship to the complexity of solving CSPs. Apt (1999) describes a generic algorithmic framework within which consistency propagation algorithms can be analyzed.

Special methods for handling higher-order constraints have been developed primarily within the context of constraint logic programming. Marriott and Stuckey (1998) provide excellent coverage of research in this area. The Alldiff constraint was studied by Regin (1994). Bounds constraints were incorporated into constraint logic programming by Van Hentenryck et al. (1998).

The basic backjumping method is due to John Gaschnig (1977, 1979). Kondrak and van Beek (1997) showed that this algorithm is essentially subsumed by forward checking. Conflict-directed backjumping was devised by Prosser (1993). The most general and powerful form of intelligent backtracking was actually developed very early on by Stallman and Sussman (1977). Their technique of dependency-directed backtracking led to the development of truth maintenance systems (Doyle, 1979), which we will discuss in Section 10.8. The connection between the two areas is analyzed by de Kleer (1989).

The work of Stallman and Sussman also introduced the idea of constraint recording, in which partial results obtained by search can be saved and reused later in the search. The idea was introduced formally into backtracking search by Dechter (1990a). Backmarking (Gaschnig, 1979) is a particularly simple method in which consistent and inconsistent pairwise assignments are saved and used to avoid rechecking constraints. Backmarking can be combined with conflict-directed backjumping; Kondrak and van Beek (1997) present a hybrid algorithm that provably subsumes either method taken separately. The method of dynamic backtracking (Ginsberg, 1993) retains successful partial assignments from later subsets of variables when backtracking over an earlier choice that does not invalidate the later success.

Local search in constraint satisfaction problems was popularized by the work of Kirkpatrick et al. (1983) on simulated annealing (see Chapter 4), which is widely used for scheduling problems. The min-conflicts heuristic was first proposed by Gu (1989) and was developed independently by Minton et al. (1992). Sosic and Gu (1994) showed how it could be applied to solve the 3,000,000 queens problem in less than a minute. The astounding success of local search using min-conflicts on the n-queens problem led to a reappraisal of the nature and prevalence of “easy” and “hard” problems. Peter Cheeseman ef al. (1991) explored the difficulty of randomly generated CSPs and discovered that almost all such problems either are trivially easy or have no solutions. Only if the parameters of the problem generator are set in a certain narrow range, within which roughly half of the problems are solvable, do we find “hard” problem instances. We discuss this phenomenon further in Chapter 7.

Work relating the structure and complexity of CSPs originates with Freuder (1985), who showed that search on arc-consistent trees works without any backtracking. A similar result, with extensions to acyclic hypergraphs, was developed in the database community (Beeri et al., 1983). Since those papers were published, there has been a great deal of progress in developing more general results relating the complexity of solving a CSP to the structure of

158

Chapter 5. Constraint Satisfaction Problems

its constraint graph. The notion of tree width was introduced by the graph theorists Robertson and Seymour (1986). Dechter and Pearl (1987, 1989), building on the work of Freuder, applied the same notion (which they called induced width) to constraint satisfaction problems and developed the tree decomposition approach sketched in Section 5.4. Drawing on this work and on results from database theory, Gottlob et al. (1999a, 1999b) developed a notion, hypertree width, that is based on the characterization of the CSP as a hypergraph. In addition to showing that any CSP with hypertree width w can be solved in time O(n“! logn), they also showed that hypertree width subsumes all previously defined measures of “width” in the sense that there are cases where the hypertree width is bounded and the other measures are unbounded.

There are several good surveys of CSP techniques, including those by Kumar (1992), Dechter and Frost (1999), and Bartak (2001); and the encyclopedia articles by Dechter (1992) and Mackworth (1992). Pearson and Jeavons (1997) survey tractable classes of CSPs, covering both structural decomposition methods and methods that rely on properties of the domains or constraints themselves. Kondrak and van Beek (1997) give an analytical survey of backtracking search algorithms, and Bacchus and van Run (1995) give a more empirical survey. The texts by Tsang (1993) and by Marriott and Stuckey (1998) go into much more depth than has been possible in this chapter. Several interesting applications are described in the collection edited by Freuder and Mackworth (1994). Papers on constraint satisfaction appear regularly in Artificial Intelligence and in the specialist journal, Constraints. The primary conference venue is the International Conference on Principles and Practice of Constraint Programming, often called CP.

EXERCISES

5.1 Define in your own words the terms constraint satisfaction problem, constraint, backtracking search, arc consistency, backjumping and min-conflicts.

5.2 How many solutions are there for the map-coloring problem in Figure 5.1?

5.3 Explain why it is a good heuristic to choose the variable that is most constrained, but the value that is least constraining in a CSP search.

5.4 Consider the problem of constructing (not solving) crossword puzzles: fitting words into a rectangular grid. The grid, which is given as part of the problem, specifies which squares are blank and which are shaded. Assume that a list of words (i.e., a dictionary) is provided and that the task is to fill in the blank squares using any subset of the list. Formulate this problem precisely in two ways:

a. As a general search problem. Choose an appropriate search algorithm, and specify a heuristic function, if you think one is needed. Is it better to fill in blanks one letter at a time or one word at a time?

5 Ginsberg et al. (1990) discuss several methods for constructing crossword puzzles. Littman et al. (1999) tackle the harder problem of solving them.

Section 5.5.

Summary 159

FLOOR-PLANNING

CLASS SCHEDULING

b. As a constraint satisfaction problem. Should the variables be words or letters?

Which formulation do you think will be better? Why?

5.5 Give precise formulations for each of the following as constraint satisfaction problems:

a. Rectilinear floor-planning: find nonoverlapping places in a large rectangle for a number of smaller rectangles.

b. Class scheduling: There is a fixed number of professors and classrooms, a list of classes to be offered, and a list of possible time slots for classes. Each professor has a set of classes that he or she can teach.

5.6 Solve the cryptarithmetic problem in Figure 5.2 by hand, using backtracking, forward checking, and the MRV and least-constraining-value heuristics.

5.7 Figure 5.5 tests out various algorithms on the n-queens problem. Try these same algorithms on map-coloring problems generated randomly as follows: scatter n points on the unit square; selecting a point X at random, connect X by a straight line to the nearest point Y such that X is not already connected to Y and the line crosses no other line; repeat the previous step until no more connections are possible. Construct the performance table for the largest n you can manage, using both d = 3 and d = 4 colors. Comment on your results.

5.8 Use the AC-3 algorithm to show that arc consistency is able to detect the inconsistency of the partial assignment { WA = red, V = blue} for the problem shown in Figure 5.1.

5.9 What is the worst-case complexity of running AC-3 on a tree-structured CSP?

5.10 AC-3 puts back on the queue every arc (Xj, X;) whenever any value is deleted from the domain of X;, even if each value of X, is consistent with several remaining values of X;. Suppose that, for every arc (Xj, X;), we keep track of the number of remaining values of X; that are consistent with each value of Xj. Explain how to update these numbers efficiently and hence show that arc consistency can be enforced in total time O(n2d?).

5.11 Show how a single ternary constraint such as “A + B = C” can be turned into three binary constraints by using an auxiliary variable. You may assume finite domains. (Hint: consider a new variable that takes on values which are pairs of other values, and consider constraints such as “X is the first element of the pair Y.”) Next, show how constraints with more than three variables can be treated similarly. Finally, show how unary constraints can be eliminated by altering the domains of variables. This completes the demonstration that any CSP can be transformed into a CSP with only binary constraints.

5.12 Suppose that a graph is known to have a cycle cutset of no more than k nodes. Describe a simple algorithm for finding a minimal cycle cutset whose runtime is not much more than O(n\*) for a CSP with 2 variables. Search the literature for methods for finding approximately minimal cycle cutsets in time that is polynomial in the size of the cutset. Does the existence of such algorithms make the cycle cutset method practical?

5.13 Consider the following logic puzzle: In five houses, each with a different color, live 5 persons of different nationalities, each of whom prefer a different brand of cigarette, a

160 Chapter 5. Constraint Satisfaction Problems

different drink, and a different pet. Given the following facts, the question to answer is “Where does the zebra live, and in which house do they drink water?”

The Englishman lives in the red house.

The Spaniard owns the dog.

The Norwegian lives in the first house on the left.

Kools are smoked in the yellow house.

The man who smokes Chesterfields lives in the house next to the man with the fox.

The Norwegian lives next to the blue house.

The Winston smoker owns snails.

The Lucky Strike smoker drinks orange juice.

The Ukrainian drinks tea.

The Japanese smokes Parliaments.

Kools are smoked in the house next to the house where the horse is kept.

Coffee is drunk in the green house.

The Green house is immediately to the right (your right) of the ivory house.

Milk is drunk in the middle house. Discuss different representations of this problem as a CSP. Why would one prefer one representation over another?

ADVERSARIAL SEARCH

In which we examine the problems that arise when we try to plan ahead in a world where other agents are planning against us.

GAMES

ZERO-SUM GAMES

PERFECT. INFORMATION

Chapter 2 introduced multiagent environments, in which any given agent will need to consider the actions of other agents and how they affect its own welfare. The unpredictability of these other agents can introduce many possible contingencies into the agent’s problemsolving process, as discussed in Chapter 3. The distinction between cooperative and competitive multiagent environments was also introduced in Chapter 2. Competitive environments, in which the agents’ goals are in conflict, give rise to adversarial search problems—often known as games.

Mathematical game theory, a branch of economics, views any multiagent environment as a game provided that the impact of each agent on the others is “significant,” regardless of whether the agents are cooperative or competitive.! In Al, “games” are usually of a rather specialized kind—what game theorists call deterministic, turn-taking, two-player, zero-sum games of perfect information. In our terminology, this means deterministic, fully observable environments in which there are two agents whose actions must alternate and in which the utility values at the end of the game are always equal and opposite. For example, if one player wins a game of chess (+1), the other player necessarily loses (1). It is this opposition between the agents’ utility functions that makes the situation adversarial. We will consider multiplayer games, non-zero-sum games, and stochastic games briefly in this chapter, but will delay discussion of game theory proper until Chapter 17.

Games have engaged the intellectual faculties of humans—sometimes to an alarming degree—for as long as civilization has existed. For Al researchers, the abstract nature of games makes them an appealing subject for study. The state of a game is easy to represent, and agents are usually restricted to a small number of actions whose outcomes are defined by

1 Environments with very many agents are best viewed as economies rather than games.

161

162

Chapter 6. Adversarial Search

IMPERFECT INFORMATION

precise rules. Physical games, such as croquet and ice hockey, have much more complicated descriptions, a much larger range of possible actions, and rather imprecise rules defining the legality of actions. With the exception of robot soccer, these physical games have not attracted much interest in the Al community.

Game playing was one of the first tasks undertaken in Al. By 1950, almost as soon as computers became programmable, chess had been tackled by Konrad Zuse (the inventor of the first programmable computer and the first programming language), by Claude Shannon (the inventor of information theory), by Norbert Wiener (the creator of modern control theory), and by Alan Turing. Since then, there has been steady progress in the standard of play, to the point that machines have surpassed humans in checkers and Othello, have defeated human champions (although not every time) in chess and backgammon, and are competitive in many other games. The main exception is Go, in which computers perform at the amateur level.

Games, unlike most of the toy problems studied in Chapter 3, are interesting because they are too hard to solve. For example, chess has an average branching factor of about 35, and games often go to 50 moves by each player, so the search tree has about 35 1% or 10 154 nodes (although the search graph has “only” about 10° distinct nodes). Games, like the real world, therefore require the ability to make some decision even when calculating the optimal decision is infeasible. Games also penalize inefficiency severely. Whereas an implementation of A\* search that is half as efficient will simply cost twice as much to run to completion, a chess program that is half as efficient in using its available time probably will be beaten into the ground, other things being equal. Game-playing research has therefore spawned a number of interesting ideas on how to make the best possible use of time.

We begin with a definition of the optimal move and an algorithm for finding it. We then look at techniques for choosing a good move when time is limited. Pruning allows us to ignore portions of the search tree that make no difference to the final choice, and heuristic evaluation functions allow us to approximate the true utility of a state without doing a complete search. Section 6.5 discusses games such as backgammon that include an element of chance; we also discuss bridge, which includes elements of imperfect information because not all cards are visible to each player. Finally, we look at how state-of-the-art game-playing programs fare against human opposition and at directions for future developments.

We will consider games with two players, whom we will call MAX and MIN for reasons that will soon become obvious. MAX moves first, and then they take turns moving until the game is over. At the end of the game, points are awarded to the winning player and penalties are given to the loser. A game can be formally defined as a kind of search problem with the following components:

e The initial state, which includes the board position and identifies the player to move.

o A successor function, which returns a list of (move, state) pairs, each indicating a legal move and the resulting state.

Section 6.2. Optimal Decisions in Games 163

TERMINAL TEST e A terminal test, which determines when the game is over. States where the game has ended are called terminal states. o A utility function (also called an objective function or payoff function), which gives a numeric value for the terminal states. In chess, the outcome is a win, loss, or draw, with values +1, —1, or 0. Some games have a wider variety of possible outcomes; the payoffs in backgammon range from +192 to —192. This chapter deals mainly with zero-sum games, although we will briefly mention non-zero-sum games.

GAME TREE The initial state and the legal moves for each side define the game tree for the game. Figure 6.1 shows part of the game tree for tic-tac-toe (noughts and crosses). From the initial state, MAX has nine possible moves. Play alternates between MAX’s placing an X and MIN’s placing an O until we reach leaf nodes corresponding to terminal states such that one player has three in a row or all the squares are filled. The number on each leaf node indicates the utility value of the terminal state from the point of view of MAX; high values are assumed to be good for MAX and bad for MIN (which is how the players get their names). It is MAX’s job to use the search tree (particularly the utility of terminal states) to determine the best move.

Optimal strategies

In a normal search problem, the optimal solution would be a sequence of moves leading to a goal state—a terminal state that is a win. In a game, on the other hand, MIN has something

STRATEGY to say about it. MAX therefore must find a contingent strategy, which specifies MAX’s move in the initial state, then MAX’s moves in the states resulting from every possible response by MIN, then MAX’s moves in the states resulting from every possible response by MIN to those moves, and so on. Roughly speaking, an optimal strategy leads to outcomes at least as good as any other strategy when one is playing an infallible opponent. We will begin by showing how to find this optimal strategy, even though it should be infeasible for MAX to compute it for games more complex than tic-tac-toe.

Even a simple game like tic-tac-toe is too complex for us to draw the entire game tree, so we will switch to the trivial game in Figure 6.2. The possible moves for MAX at the root node are labeled aq, as, and a3. The possible replies to a; for MIN are by, ba, b3, and so on. This particular game ends after one move each by MAX and MIN. (In game parlance, we say

PLY that this tree is one move deep, consisting of two half-moves, each of which is called a ply.) The utilities of the terminal states in this game range from 2 to 14. Given a game tree, the optimal strategy can be determined by examining the minimax MINIMAX VALUE value of each node, which we write as MINIMAX-VALUE(n). The minimax value of a node is the utility (for MAX) of being in the corresponding state, assuming that both players play optimally from there to the end of the game. Obviously, the minimax value of a terminal state is just its utility. Furthermore, given a choice, MAX will prefer to move to a state of maximum value, whereas MIN prefers a state of minimum value. So we have the following:

MINIMAX-VALUE(n) =

UTILITY(n) if n is a terminal state MAX Syccessors(n) MINIMAX-VALUE(s) if nis a MAX node Mine Gyecessors(n) MINIMAX-VALUE(s) if n is a MIN node.

164

Chapter 6. Adversarial Search

[]

XOX O(X| [X[O|X TERMINAL o[X X10] |X|0lo Utility —1 0 +l

Figure 6.1 A (partial) search tree for the game of tic-tac-toe. The top node is the initial state, and MAX moves first, placing an X in an empty square. We show part of the search tree, giving alternating moves by MIN (0) and MAX, until we eventually reach terminal states, which can be assigned utilities according to the rules of the game.

[]

Figure 6.2 A two-ply game tree. The A nodes are “MAX nodes,” in which it is MAX’s turn to move, and the V nodes are “MIN nodes.” The terminal nodes show the utility values for MAX; the other nodes are labeled with their minimax values. MAX’s best move at the root is a1, because it leads to the successor with the highest minimax value, and MIN’s best reply is by, because it leads to the successor with the lowest minimax value.

Let us apply these definitions to the game tree in Figure 6.2. The terminal nodes on the bottom level are already labeled with their utility values. The first MIN node, labeled B, has three successors with values 3, 12, and 8, so its minimax value is 3. Similarly, the other two MIN nodes have minimax value 2. The root node is a MAX node; its successors have minimax

Section 6.2.

Optimal Decisions in Games 165

MINIMAX DECISION

MINIMAX ALGORITHM

BACKED UP

values 3, 2, and 2; so it has a minimax value of 3. We can also identify the minimax decision at the root: action a4 is the optimal choice for MAX because it leads to the successor with the highest minimax value.

This definition of optimal play for MAX assumes that MIN also plays optimally—it maximizes the worst-case outcome for MAX. What if MIN does not play optimally? Then it is easy to show (Exercise 6.2) that MAX will do even better. There may be other strategies against suboptimal opponents that do better than the minimax strategy; but these strategies necessarily do worse against optimal opponents.

The minimax algorithm

The minimax algorithm (Figure 6.3) computes the minimax decision from the current state. It uses a simple recursive computation of the minimax values of each successor state, directly implementing the defining equations. The recursion proceeds all the way down to the leaves of the tree, and then the minimax values are backed up through the tree as the recursion unwinds. For example, in Figure 6.2, the algorithm first recurses down to the three bottomleft nodes, and uses the UTILITY function on them to discover that their values are 3, 12, and 8 respectively. Then it takes the minimum of these values, 3, and returns it as the backed-up value of node B. A similar process gives the backed up values of 2 for C' and 2 for D. Finally, we take the maximum of 3, 2, and 2 to get the backed-up value of 3 for the root node.

The minimax algorithm performs a complete depth-first exploration of the game tree. If the maximum depth of the tree is m, and there are b legal moves at each point, then the time complexity of the minimax algorithm is O(b™). The space complexity is O(bm) for an algorithm that generates all successors at once, or O{m) for an algorithm that generates successors one at a time (see page 76). For real games, of course, the time cost is totally impractical, but this algorithm serves as the basis for the mathematical analysis of games and for more practical algorithms.

Optimal decisions in multiplayer games

Many popular games allow more than two players. Let us examine how to extend the minimax idea to multiplayer games. This is straightforward from the technical viewpoint, but raises some interesting new conceptual issues.

First, we need to replace the single value for each node with a vector of values. For example, in a three-player game with players A, B, and C, a vector (v4, vp, vo) is associated with each node. For terminal states, this vector gives the utility of the state from each player’s viewpoint. (In two-player, zero-sum games, the two-element vector can be reduced to a single value because the values are always opposite.) The simplest way to implement this is to have the UTILITY function return a vector of utilities.

Now we have to consider nonterminal states. Consider the node marked X in the game tree shown in Figure 6.4. In that state, player C chooses what to do. The two choices lead to terminal states with utility vectors (v4 =1,vp =2,v¢c =6) and (vg =4,vp =2,vc =3). Since 6 is bigger than 3, C should choose the first move. This means that if state X is reached, subsequent play will lead to a terminal state with utilities (v4 =1,vg =2,vc =6). Hence,

166

Chapter 6. Adversarial Search

ALLIANCES

function MINIMAX-DECISION(state) returns an action inputs: state, current state in game

v + MAX-VALUE(state) return the action in SUCCESSORS(state) with value v

function MAX-VALUE(state) returns a utility value if TERMINAL-TEST(state) then return UTILITY (state) V——00 for a, sin SUCCESSORS(state) do © +— MAX(v, MIN-VALUE(s)) return v

function MIN-VALUE(state) returns a utility value if TERMINAL-TEST(state) then return UTILITY (state) V— 00 for a, s in SUCCESSORS(state) do © + MIN(v, MAX-VALUE(s)) return v

Figure 6.3 An algorithm for calculating minimax decisions. It returns the action corresponding to the best possible move, that is, the move that leads to the outcome with the best utility, under the assumption that the opponent plays to minimize utility. The functions MAX-VALUE and MIN-VALUE go through the whole game tree, all the way to the leaves, to determine the backed-up value of a state.

to move A

(1,2,6) (4,2,3) (6, 1,2) (7,41) G, LD (1,52) (@7,7,1) (5,49)

Figure 6.4 The first three ply of a game tree with three players (4, B, C). Each node is labeled with values from the viewpoint of each player. The best move is marked at the root.

the backed-up value of X is this vector. In general, the backed-up value of a node n is the utility vector of whichever successor has the highest value for the player choosing at 7. Anyone who plays multiplayer games, such as Diplomacy ™, quickly becomes aware that there is a lot more going on than in two-player games. Multiplayer games usually involve alliances, whether formal or informal, among the players. Alliances are made and broken

Section 6.3. Alpha—Beta Pruning 167

as the game proceeds. How are we to understand such behavior? Are alliances a natural consequence of optimal strategies for each player in a multiplayer game? It turns out that they can be. For example suppose A and B are in weak positions and C is in a stronger position. Then it is often optimal for both A and B to attack C rather than each other, lest C destroy each of them individually. In this way, collaboration emerges from purely selfish behavior. Of course, as soon as C' weakens under the joint onslaught, the alliance loses its value, and either A or B could violate the agreement. In some cases, explicit alliances merely make concrete what would have happened anyway. In other cases there is a social stigma to breaking an alliance, so players must balance the immediate advantage of breaking an alliance against the long-term disadvantage of being perceived as untrustworthy. See Section 17.6 for more on these complications.

If the game is not zero-sum, then collaboration can also occur with just two players. Suppose, for example, that there is a terminal state with utilities (v4 = 1000, vp = 1000), and that 1000 is the highest possible utility for each player. Then the optimal strategy is for both players to do everything possible to reach this state—that is, the players will automatically cooperate to achieve a mutually desirable goal.

6.3 ALPHA—BETA PRUNING

The problem with minimax search is that the number of game states it has to examine is exponential in the number of moves. Unfortunately we can’t eliminate the exponent, but we can effectively cut it in half. The trick is that it is possible to compute the correct minimax decision without looking at every node in the game tree. That is, we can borrow the idea of pruning from Chapter 4 in order to eliminate large parts of the tree from consideration.

ALPHIR BETA The particular technique we will examine is called alpha-beta pruning. When applied to a standard minimax tree, it returns the same move as minimax would, but prunes away branches that cannot possibly influence the final decision.

Consider again the two-ply game tree from Figure 6.2. Let’s go through the calculation of the optimal decision once more, this time paying careful attention to what we know at each point in the process. The steps are explained in Figure 6.5. The outcome is that we can identify the minimax decision without ever evaluating two of the leaf nodes.

Another way to look at this is as a simplification of the formula for MINIMA XVALUE. Let the two unevaluated successors of node C' in Figure 6.5 have values x and y and let z be the minimum of x and y. The value of the root node is given by

MINIMAX-VALUE(r00t) = max(min(3, 12,8), min(2, z,y), min(14, 5, 2)) = max(3, min(2,z,y), 2) = max(3, z,2) where z < 2 = 3.

In other words, the value of the root and hence the minimax decision are independent of the values of the pruned leaves x and y.

168

Chapter 6. Adversarial Search

[]

Figure 6.5 Stages in the calculation of the optimal decision for the game tree in Figure 6.2. At each point, we show the range of possible values for each node. (a) The first leaf below B has the value 3. Hence, B, which is a MIN node, has a value of at most 3. (b) The second leaf below B has a value of 12; MIN would avoid this move, so the value of B is still at most 3. (¢) The third leaf below B has a value of 8; we have seen all B’s successors, so the value of B is exactly 3. Now, we can infer that the value of the root is at least 3, because MAX has a choice worth 3 at the root. (d) The first leaf below C has the value 2. Hence, C, which is a MIN node, has a value of at most 2. But we know that B is worth 3, so MAX would never choose C. Therefore, there is no point in looking at the other successors of C'. This is an example of alpha—beta pruning. (e) The first leaf below D has the value 14, so D is worth ar most 14. This is still higher than MAX’s best alternative (i.e., 3), so we need to keep exploring D’s successors. Notice also that we now have bounds on all of the successors of the root, so the root’s value is also at most 14. (f) The second successor of D is worth 5, so again we need to keep exploring. The third successor is worth 2, so now D is worth exactly 2. MAX’s decision at the root is to move to B, giving a value of 3.

Alpha-beta pruning can be applied to trees of any depth, and it is often possible to prune entire subtrees rather than just leaves. The general principle is this: consider a node n somewhere in the tree (see Figure 6.6), such that Player has a choice of moving to that node. If Player has a better choice m either at the parent node of n or at any choice point further up, then n will never be reached in actual play. So once we have found out enough about n (by examining some of its descendants) to reach this conclusion, we can prune it.

Remember that minimax search is depth-first, so at any one time we just have to consider the nodes along a single path in the tree. Alpha—beta pruning gets its name from the

Section 6.3.

Alpha—Beta Pruning 169 Player Opponent Player Opponent Figure 6.6 Alpha-beta pruning: the general case. If m is better than n for Player, we will never get to n in play.

following two parameters that describe bounds on the backed-up values that appear anywhere along the path:

a = the value of the best (i.e., highest-value) choice we have found so far at any choice point along the path for MAX.

(8 = the value of the best (i.e., lowest-value) choice we have found so far at any choice point along the path for MIN.

Alpha-beta search updates the values of « and (3 as it goes along and prunes the remaining branches at a node (i.e., terminates the recursive call) as soon as the value of the current node is known to be worse than the current « or 3 value for MAX or MIN, respectively. The complete algorithm is given in Figure 6.7. We encourage the reader to trace its behavior when applied to the tree in Figure 6.5.

The effectiveness of alpha—beta pruning is highly dependent on the order in which the successors are examined. For example, in Figure 6.5(e) and (f), we could not prune any successors of ID at all because the worst successors (from the point of view of MIN) were generated first. If the third successor had been generated first, we would have been able to prune the other two. This suggests that it might be worthwhile to try to examine first the successors that are likely to be best.

If we assume that this can be done, then it turns out that alpha-beta needs to examine only O(b™/2) nodes to pick the best move, instead of O(b™) for minimax. This means that the effective branching factor becomes v/b instead of b—for chess, 6 instead of 35. Put another way, alpha—beta can look ahead roughly twice as far as minimax in the same amount of time. If successors are examined in random order rather than best-first, the total number of nodes examined will be roughly O(b\*™/\*) for moderate b. For chess, a fairly simple ordering function (such as trying captures first, then threats, then forward moves, and then backward moves) gets you to within about a factor of 2 of the best-case O(b™/2) result. Adding dynamic

2 Obviously, it cannot be done perfectly; otherwise the ordering function could be used to play a perfect game!

170

Chapter 6. Adversarial Search

TRANSPOSITIONS

function ALPHA-BETA-SEARCH(state) returns an action inputs: state, current state in game

v +— MAX-VALUE(state, —oo, +00) return the action in SUCCESSORS(state) with value v

function MAX-VALUE(state, o, 3) returns a utility value inputs: state, current state in game «, the value of the best alternative for MAX along the path to state 3, the value of the best alternative for MIN along the path to state

if TERMINAL-TEST(state) then return UTILITY (State) V—— for a, sin SUCCESSORS(state) do v «— MAX(v, MIN-VALUE(s, a, 8) if v > (3 then return v a +— MAX(a, v) return v

function MIN-VALUE(state, a, 5) returns a utility value inputs: state, current state in game a, the value of the best alternative for MAX along the path to state 3, the value of the best alternative for MIN along the path to state

if TERMINAL-TEST(state) then return UTILITY (state) V<+— +00 for a, sin SUCCESSORS(state) do v + MIN(v, MAX-VALUE(s, a, 8)) if v < « then return v 8 «— MIN(S, v) return v

Figure 6.7 The alpha-beta search algorithm. Notice that these routines are the same as the MINIMAX routines in Figure 6.3, except for the two lines in each of MINVALUE and MAX-VALUE that maintain « and 3 (and the bookkeeping to pass these parameters along).

move-ordering schemes, such as trying first the moves that were found to be best last time, brings us quite close to the theoretical limit.

In Chapter 3, we noted that repeated states in the search tree can cause an exponential increase in search cost. In games, repeated states occur frequently because of transpositions—different permutations of the move sequence that end up in the same position. For example, if White has one move a; that can be answered by Black with 4; and an unrelated move az on the other side of the board that can be answered by b2, then the sequences [a1,b1, a9, ba] and [aq, bz, az, by] both end up in the same position (as do the permutations beginning with as). It is worthwhile to store the evaluation of this position in a hash table the first time it is encountered, so that we don’t have to recompute it on subsequent occurrences.

Section 6.4.

Imperfect, Real-Time Decisions 171

TRANSPOSITION TABLE

The hash table of previously seen positions is traditionally called a transposition table; it is essentially identical to the closed list in GRAPH-SEARCH (page 83). Using a transposition table can have a dramatic effect, sometimes as much as doubling the reachable search depth in chess. On the other hand, if we are evaluating a million nodes per second, it is not practical to keep all of them in the transposition table. Various strategies have been used to choose the most valuable ones.

CUTOFF TEST

The minimax algorithm generates the entire game search space, whereas the alpha—beta algorithm allows us to prune large parts of it. However, alpha—beta still has to search all the way to terminal states for at least a portion of the search space. This depth is usually not practical, because moves must be made in a reasonable amount of time—typically a few minutes at most. Shannon’s 1950 paper, Programming a computer for playing chess, proposed instead that programs should cut off the search earlier and apply a heuristic evaluation function to states in the search, effectively turning nonterminal nodes into terminal leaves. In other words, the suggestion is to alter minimax or alpha—beta in two ways: the utility function is replaced by a heuristic evaluation function EVAL, which gives an estimate of the position’s utility, and the terminal test is replaced by a cutoff test that decides when to apply EVAL.

Evaluation functions

An evaluation function returns an estimate of the expected utility of the game from a given position, just as the heuristic functions of Chapter 4 return an estimate of the distance to the goal. The idea of an estimator was not new when Shannon proposed it. For centuries, chess players (and aficionados of other games) have developed ways of judging the value of a position, because humans are even more limited in the amount of search they can do than are computer programs. It should be clear that the performance of a game-playing program is dependent on the quality of its evaluation function. An inaccurate evaluation function will guide an agent toward positions that turn out to be lost. How exactly do we design good evaluation functions?

First, the evaluation function should order the terminal states in the same way as the true utility function; otherwise, an agent using it might select suboptimal moves even if it can see ahead all the way to the end of the game. Second, the computation must not take too long! (The evaluation function could call MINIMAXDECISION as a subroutine and calculate the exact value of the position, but that would defeat the whole purpose: to save time.) Third, for nonterminal states, the evaluation function should be strongly correlated with the actual chances of winning.

One might well wonder about the phrase “chances of winning.” After all, chess is not a game of chance: we know the current state with certainty, and there are no dice involved. But if the search must be cut off at nonterminal states, then the algorithm will necessarily be uncertain about the final outcomes of those states. This type of uncertainty is induced by

172 Chapter 6. Adversarial Search

computational, rather than informational, limitations. Given the limited amount of computation that the evaluation function is allowed to do for a given state, the best it can do is make a guess about the final outcome. Let us make this idea more concrete. Most evaluation functions work by calculating FEATURES various features of the state—for example, the number of pawns possessed by each side in a game of chess. The features, taken together, define various categories or equivalence classes of states: the states in each category have the same values for all the features. Any given category, generally speaking, will contain some states that lead to wins, some that lead to draws, and some that lead to losses. The evaluation function cannot know which states are which, but it can return a single value that reflects the proportion of states with each outcome. For example, suppose our experience suggests that 72% of the states encountered in the category lead to a win (utility +1); 20% to a loss (-1), and 8% to a draw (0). Then a ExPECTEDVALUE Teasonable evaluation for states in the category is the weighted average or expected value: (0.72 x +1) + (0.20 x —1) + (0.08 x 0) = 0.52. In principle, the expected value can be determined for each category, resulting in an evaluation function that works for any state. As with terminal states, the evaluation function need not return actual expected values, as long as the ordering of the states is the same.

In practice, this kind of analysis requires too many categories and hence too much experience to estimate all the probabilities of winning. Instead, most evaluation functions compute separate numerical contributions from each feature and then combine them to find

MATERIAL VALUE the total value. For example, introductory chess books give an approximate material value for each piece: each pawn is worth 1, a knight or bishop is worth 3, a rook 5, and the queen 9. Other features such as “good pawn structure” and “king safety” might be worth half a pawn, say. These feature values are then simply added up to obtain the evaluation of the position. A secure advantage equivalent to a pawn gives a substantial likelihood of winning, and a secure advantage equivalent to three pawns should give almost certain victory, as illustrated in Figure 6.8(a). Mathematically, this kind of evaluation function is called a weighted linear

NEGHIEDLINEAR function, because it can be expressed as

BVAL(s) = wi fi(s) + wafa(s) +--+ wafals) = > \_wifi(s), 1=1

where each w; is a weight and each f; is a feature of the position. For chess, the f; could be the numbers of each kind of piece on the board, and the w; could be the values of the pieces (1 for pawn, 3 for bishop, etc.).

Adding up the values of features seems like a reasonable thing to do, but in fact it involves a very strong assumption: that the contribution of each feature is independent of the values of the other features. For example, assigning the value 3 to a bishop ignores the fact that bishops are more powerful in the endgame, when they have a lot of space to maneuver. For this reason, current programs for chess and other games also use nonlinear combinations of features. For example, a pair of bishops might be worth slightly more than twice the value of a single bishop, and a bishop is worth more in the endgame than at the beginning.

The astute reader will have noticed that the features and weights are not part of the rules of chess! They come from centuries of human chess-playing experience. Given the

Section 6.4. Imperfect, Real-Time Decisions 173

[]

(a) White to move (b) White to move

Figure 6.8 Two slightly different chess positions. In (a), black has an advantage of a knight and two pawns and will win the game. In (b), black will lose after white captures the queen.

linear form of the evaluation, the features and weights result in the best approximation to the true ordering of states by value. In particular, experience suggests that a secure material advantage of more than one point will probably win the game, all other things being equal; a three-point advantage is sufficient for near-certain victory. In games where this kind of experience is not available, the weights of the evaluation function can be estimated by the machine learning techniques of Chapter 18. Reassuringly, applying these techniques to chess has confirmed that a bishop is indeed worth about three pawns.

Cutting off search

The next step is to modify ALPHA-BETA-SEARCH so that it will call the heuristic EVAL function when it is appropriate to cut off the search. In terms of implementation, we replace the two lines in Figure 6.7 that mention TERMINAL-TEST with the following line:

if CUTOFF-TEST(state, depth) then return EVAL(state)

We also must arrange for some bookkeeping so that the current depth is incremented on each recursive call. The most straightforward approach to controlling the amount of search is to set a fixed depth limit, so that CUTOFF-TEST(state, depth) returns true for all depth greater than some fixed depth d. (It must also return true for all terminal states, just as TERMINAL-TEST did.) The depth d is chosen so that the amount of time used will not exceed what the rules of the game allow.

A more robust approach is to apply iterative deepening, as defined in Chapter 3. When time runs out, the program returns the move selected by the deepest completed search. However, these approaches can lead to errors due to the approximate nature of the evaluation function. Consider again the simple evaluation function for chess based on material advantage. Suppose the program searches to the depth limit, reaching the position in Figure 6.8(b),

174

Chapter 6. Adversarial Search

QUIESCENCE

QUIESCENCE SEARCH

HORIZON EFFECT

SINGULAR EXTENSIONS

FORWARD PRUNING

where Black is ahead by a knight and two pawns. It would report this as the heuristic value of the state, thereby declaring that the state will likely lead to a win by Black. But White’s next move captures Black’s queen with no compensation. Hence, the position is really won for White, but this can be seen only by looking ahead one more ply.

Obviously, a more sophisticated cutoff test is needed. The evaluation function should be applied only to positions that are quiescent—that is, unlikely to exhibit wild swings in value in the near future. In chess, for example, positions in which favorable captures can be made are not quiescent for an evaluation function that just counts material. Nonquiescent positions can be expanded further until quiescent positions are reached. This extra search is called a quiescence search; sometimes it is restricted to consider only certain types of moves, such as capture moves, that will quickly resolve the uncertainties in the position.

The horizon effect is more difficult to eliminate. It arises when the program is facing a move by the opponent that causes serious damage and is ultimately unavoidable. Consider the chess game in Figure 6.9. Black is ahead in material, but if White can advance its pawn from the seventh row to the eighth, the pawn will become a queen and create an easy win for White. Black can forestall this outcome for 14 ply by checking White with the rook, but inevitably the pawn will become a queen. The problem with fixed-depth search is that it believes that these stalling moves have avoided the queening move—we say that the stalling moves push the inevitable queening move “over the search horizon” to a place where it cannot be detected.

As hardware improvements lead to deeper searches, one expects that the horizon effect will occur less frequently—very long delaying sequences are quite rare. The use of singular extensions has also been quite effective in avoiding the horizon effect without adding too much search cost. A singular extension is a move that is “clearly better” than all other moves in a given position. A singular-extension search can go beyond the normal depth limit without incurring much cost because its branching factor is 1. (Quiescence search can be thought of as a variant of singular extensions.) In Figure 6.9, a singular extension search will find the eventual queening move, provided that black’s checking moves and white’s king moves can be identified as “clearly better” than the alternatives.

So far we have talked about cutting off search at a certain level and about doing alpha— beta pruning that provably has no effect on the result. It is also possible to do forward pruning, meaning that some moves at a given node are pruned immediately without further consideration. Clearly, most humans playing chess only consider a few moves from each position (at least consciously). Unfortunately, the approach is rather dangerous because there is no guarantee that the best move will not be pruned away. This can be disastrous if applied near the root, because every so often the program will miss some “obvious” moves. Forward pruning can be used safely in special situations—for example, when two moves are symmetric or otherwise equivalent, only one of them need be considered—or for nodes that are deep in the search tree.

Combining all the techniques described here results in a program that can play creditable chess (or other games). Let us assume we have implemented an evaluation function for chess, a reasonable cutoff test with a quiescence search, and a large transposition table. Let us also assume that, after months of tedious bit-bashing, we can generate and evaluate

Section 6.5.

Games That Include an Element of Chance 175

x1 Io -a =e

Black to move

Figure 6.9 The horizon effect. A series of checks by the black rook forces the inevitable queening move by white “over the horizon” and makes this position look like a win for black,

when it is really a win for white.

around a million nodes per second on the latest PC, allowing us to search roughly 200 million nodes per move under standard time controls (three minutes per move). The branching factor for chess is about 35, on average, and 355 is about 50 million, so if we used minimax search we could look ahead only about five plies. Though not incompetent, such a program can be fooled easily by an average human chess player, who can occasionally plan six or eight plies ahead. With alpha—beta search we get to about 10 ply, which results in an expert level of play. Section 6.7 describes additional pruning techniques that can extend the effective search depth to roughly 14 plies. To reach grandmaster status we would need an extensively tuned evaluation function and a large database of optimal opening and endgame moves. It wouldn't hurt to have a supercomputer to run the program on.

In real life, there are many unpredictable external events that put us into unforeseen situations. Many games mirror this unpredictability by including a random element, such as the throwing of dice. In this way, they take us a step nearer reality, and it is worthwhile to see how this affects the decision-making process.

Backgammon is a typical game that combines luck and skill. Dice are rolled at the beginning of a player’s turn to determine the legal moves. In the backgammon position of Figure 6.10, for example, white has rolled a 6-5, and has four possible moves.

Although White knows what his or her own legal moves are, White does not know what Black is going to roll and thus does not know what Black’s legal moves will be. That means White cannot construct a standard game tree of the sort we saw in chess and tic-tac-toe. A

176 Chapter 6. Adversarial Search

[]

25 24 23 22 21 20 19 18 177 16 15 14 13

Figure 6.10 A typical backgammon position. The goal of the game is to move all one’s pieces off the board. White moves clockwise toward 25, and black moves counterclockwise toward 0. A piece can move to any position unless there are multiple opponent pieces there; if there is one opponent, it is captured and must start over. In the position shown, White has rolled 6-5 and must choose among four legal moves: (5-10,5-11), (5-11,19-24), (5-10,1016), and (5-11,11-16).

[]

Figure 6.11 Schematic game tree for a backgammon position.

Section 6.5.

Games That Include an Element of Chance 177

CHANCE NODES

EXPECTIMINIMAX VALUE

game tree in backgammon must include chance nodes in addition to MAX and MIN nodes. Chance nodes are shown as circles in Figure 6.11. The branches leading from each chance node denote the possible dice rolls, and each is labeled with the roll and the chance that it will occur. There are 36 ways to roll two dice, each equally likely; but because a 6-5 is the same as a 5-6, there are only 21 distinct rolls. The six doubles (1-1 through 6-6) have a 1/36 chance of coming up, the other 15 distinct rolls a 1/18 chance each.

The next step is to understand how to make correct decisions. Obviously, we still want to pick the move that leads to the best position. However, the resulting positions do not have definite minimax values. Instead, we can only calculate the expected value, where the expectation is taken over all the possible dice rolls that could occur. This leads us to generalize the minimax value for deterministic games to an expectiminimax value for games with chance nodes. Terminal nodes and MAX and MIN nodes (for which the dice roll is known) work exactly the same way as before; chance nodes are evaluated by taking the weighted average of the values resulting from all possible dice rolls, that is,

EXPECTIMINIMAX(n) = UTILITY (n) if n is a terminal state MAX se Syccessors(n) EXPECTIMINIMAX(S) if n is a MAX node Mine gyccessors(n) EXPECTIMINIMAX(S) if n is a MIN node

> se Successors(n) (8) EXPECTIMINIMAX(s) if n is a chance node

where the successor function for a chance node n simply augments the state of n with each possible dice roll to produce each successor s and P(s) is the probability that that dice roll occurs. These equations can be backed up recursively all the way to the root of the tree, just as in minimax. We leave the details of the algorithm as an exercise.

Position evaluation in games with chance nodes

As with minimax, the obvious approximation to make with expectiminimax is to cut the search off at some point and apply an evaluation function to each leaf. One might think that evaluation functions for games such as backgammon should be just like evaluation functions for chess—they just need to give higher scores to better positions. But in fact, the presence of chance nodes means that one has to be more careful about what the evaluation values mean. Figure 6.12 shows what happens: with an evaluation function that assigns values [1, 2, 3, 4] to the leaves, move Ap is best; with values [1, 20, 30, 400], move As is best. Hence, the program behaves totally differently if we make a change in the scale of some evaluation values! It turns out that, to avoid this sensitivity, the evaluation function must be a positive linear transformation of the probability of winning from a position (or, more generally, of the expected utility of the position). This is an important and general property of situations in which uncertainty is involved, and we discuss it further in Chapter 16.

Complexity of expectiminimax

If the program knew in advance all the dice rolls that would occur for the rest of the game, solving a game with dice would be just like solving a game without dice, which minimax

178

Chapter 6. Adversarial Search

MAX

CHANCE

MIN

200 20 30 30 1 1 400 400

Figure 6.12 An order-preserving transformation on leaf values changes the best move.

does in O(b™) time. Because expectiminimax is also considering all the possible dice-roll sequences, it will take O(b™n™), where n is the number of distinct rolls.

Even if the search depth is limited to some small depth d, the extra cost compared with that of minimax makes it unrealistic to consider looking ahead very far in most games of chance. In backgammon 7 is 21 and b is usually around 20, but in some situations can be as high as 4000 for dice rolls that are doubles. Three plies is probably all we could manage.

Another way to think about the problem is this: the advantage of alpha-beta is that it ignores future developments that just are not going to happen, given best play. Thus, it concentrates on likely occurrences. In games with dice, there are no likely sequences of moves, because for those moves to take place, the dice would first have to come out the right way to make them legal. This is a general problem whenever uncertainty enters the picture: the possibilities are multiplied enormously, and forming detailed plans of action becomes pointless, because the world probably will not play along.

No doubt it will have occurred to the reader that perhaps something like alpha-beta pruning could be applied to game trees with chance nodes. It turns out that it can. The analysis for MIN and MAX nodes is unchanged, but we can also prune chance nodes, using a bit of ingenuity. Consider the chance node C in Figure 6.11 and what happens to its value as we examine and evaluate its children. Is it possible to find an upper bound on the value of C before we have looked at all its children? (Recall that this is what alpha-beta needs to prune a node and its subtree.) At first sight, it might seem impossible, because the value of C' is the average of its children’s values. Until we have looked at all the dice rolls, this average could be anything, because the unexamined children might have any value at all. But if we put bounds on the possible values of the utility function, then we can arrive at bounds for the average. For example, if we say that all utility values are between +3 and —3, then the value of leaf nodes is bounded, and in turn we can place an upper bound on the value of a chance node without looking at all its children.

Section 6.5.

Games That Include an Element of Chance 179

Card games

Card games are interesting for many reasons besides their connection with gambling. Among the huge variety of games, we will focus on those in which cards are dealt randomly at the beginning of the game, with each player receiving a hand of cards that is not visible to the other players. Such games include bridge, whist, hearts, and some forms of poker.

At first sight, it might seem that card games are just like dice games: the cards are dealt randomly and determine the moves available to each player, but all the dice are rolled at the beginning! We will pursue this observation further. It will turn out to be quite useful in practice. It is also quite wrong, for interesting reasons.

Imagine two players, MAX and MIN, playing some practice hands of four-card two handed bridge with all the cards showing. The hands are as follows, with MAX to play first:

MAX: U6 $6 S98 MIN: O04 #2 &105.

Suppose that MAX leads the & 9. MIN must now follow suit, playing either the & 10 or the & 5. MIN plays the & 10 and wins the trick. MIN goes next and leads the & 2. MAX has no spades (and so cannot win the trick) and therefore must throw away some card. The obvious choice is the > 6 because the other two remaining cards are winners. Now, whichever card MIN leads for the next trick, MAX will win both remaining tricks and the game will be tied at two tricks each. It is easy to show, using a suitable variant of minimax (Exercise 6.12), that MAX’s lead of the & 9 is in fact an optimal choice. Now let’s modify MIN’s hand, replacing the ©’ 4 with the $ 4:

MAX: U6 $6 98 MIN: $4 #2 &105.

The two cases are entirely symmetric: play will be identical, except that on the second trick MAX will throw away the © 6. Again, the game will be tied at two tricks each and the lead of the & 9 is an optimal choice.

So far, so good. Now let’s hide one of MIN’s cards: MAX knows that MIN has either the first hand (with the © 4) or the second hand (with the <> 4), but has no idea which. MAX reasons as follows:

The & 9 is an optimal choice against MIN’s first hand and against MIN’s second hand, so it must be optimal now because I know that MIN has one of the two hands.

More generally, MAX is using what we might call “averaging over clairvoyancy.” The idea is to evaluate a given course of action when there are unseen cards by first computing the minimax value of that action for each possible deal of the cards, and then computing the expected value over all deals using the probability of each deal.

If you think this is reasonable (or if you have no idea because you don’t understand bridge), consider the following story:

Day 1: Road A leads to a heap of gold pieces; Road B leads to a fork. Take the left fork and you’ll find a mound of jewels, but take the right fork and you’ll be run over by a bus. Day 2: Road A leads to a heap of gold pieces; Road B leads to a fork. Take the right fork and you’ll find a mound of jewels, but take the left fork and you’ll be run over by a bus. Day 3: Road A leads to a heap of gold pieces; Road B leads to a fork. Guess correctly and you'll find a mound of jewels, but guess incorrectly and you’ll be run over by a bus.

180

Chapter 6. Adversarial Search

Obviously, it’s not unreasonable to take Road B on the first two days. No sane person, though, would take Road B on Day 3. Yet this is exactly what averaging over clairvoyancy suggests: Road B is optimal in the situations of Day 1 and Day 2; therefore it is optimal on Day 3, because one of the two previous situations must hold. Let us return to the card game: after MAX leads the & 9, MIN wins with the & 10. As before, MIN leads the # 2, and now MAX is at the fork in the road without any instructions. If MAX throws away the © 6 and MIN still has the © 4, the © 4 becomes a winner and MAX loses the game. Similarly, If MAX throws away the <> 6 and MIN still has the { 4, MAX also loses. Therefore, playing the & 9 first leads to a situation where MAX has a 50% chance of losing. (It would be much better to play the © 6 and the <> 6 first, guaranteeing a tied game.)

The lesson to be drawn from all this is that when information is missing, one must consider what information one will have at each point in the game. The problem with MAX’s algorithm is that it assumes that in each possible deal, play will proceed as if all the cards are visible. As our example shows, this leads MAX to act as if all future uncertainty will be resolved when the time comes. MAX’s algorithm will also never decide to gather information (or provide information to a partner), because within each deal there’s no need to do so; yet in games such as bridge, it is often a good idea to play a card that will help one discover things about one’s opponent’s cards or that will tell one’s partner about one’s own cards. These kinds of behaviors are generated automatically by an optimal algorithm for games of imperfect information. Such an algorithm searches not in the space of world states (hands of cards), but in the space of belief states (beliefs about who has which cards, with what probabilities). We will be able to explain the algorithm properly in Chapter 17, once we have developed the necessary probabilistic machinery. In that chapter, we will also expand on one final and very important point: in games of imperfect information, it’s best to give away as little information to the opponent as possible, and often the best way to do this is to act unpredictably. This is why restaurant hygiene inspectors do random inspection visits.

CHESS

One might say that game playing is to Al as Grand Prix motor racing is to the car industry: state-of-the-art game programs are blindingly fast, incredibly well-tuned machines that incorporate very advanced engineering techniques, but they aren’t much use for doing the shopping. Although some researchers believe that game playing is somewhat irrelevant to mainstream Al, it continues to generate both excitement and a steady stream of innovations that have been adopted by the wider community.

Chess: In 1957, Herbert Simon predicted that within 10 years computers would beat the human world champion. Forty years later, the Deep Blue program defeated Garry Kasparov in a six-game exhibition match. Simon was wrong, but only by a factor of 4. Kasparov wrote:

The decisive game of the match was Game 2, which left a scar in my memory ... we saw something that went well beyond our wildest expectations of how well a computer would be able to foresee the long-term positional consequences of its decisions. The machine

Section 6.6.

State-of-the-Art Game Programs 181

NULL MOVE

FUTILITY PRUNING

CHECKERS

refused to move to a position that had a decisive short-term advantage—showing a very human sense of danger. (Kasparov, 1997)

Deep Blue was developed by Murray Campbell, Feng-Hsiung Hsu, and Joseph Hoane at IBM (see Campbell et al., 2002), building on the Deep Thought design developed earlier by Campbell and Hsu at Carnegie Mellon. The winning machine was a parallel computer with 30 IBM RS/6000 processors running the “software search” and 480 custom VLSI chess processors that performed move generation (including move ordering), the “hardware search” for the last few levels of the tree, and the evaluation of leaf nodes. Deep Blue searched 126 million nodes per second on average, with a peak speed of 330 million nodes per second. It generated up to 30 billion positions per move, reaching depth 14 routinely. The heart of the machine is a standard iterative-deepening alpha~beta search with a transposition table, but the key to its success seems to have been its ability to generate extensions beyond the depth limit for sufficiently interesting lines of forcing/forced moves. In some cases the search reached a depth of 40 plies. The evaluation function had over 8000 features, many of them describing highly specific patterns of pieces. An “opening book” of about 4000 positions was used, as well as a database of 700,000 grandmaster games from which consensus recommendations could be extracted. The system also used a large endgame database of solved positions, containing all positions with five pieces and many with six pieces. This database has the effect of substantially extending the effective search depth, allowing Deep Blue to play perfectly in some cases even when it is many moves away from checkmate.

The success of Deep Blue reinforced the widely held belief that progress in computer game-playing has come primarily from ever-more-powerful hardware—a view encouraged by IBM. Deep Blue's creators, on the other hand, state that the search extensions and evaluation function were also critical (Campbell et al., 2002). Moreover, we know that several recent algorithmic improvements have allowed programs running on standard PCs to win every World Computer-Chess Championship since 1992, often defeating massively parallel opponents that could search 1000 times more nodes. A variety of pruning heuristics are used to reduce the effective branching factor to less than 3 (compared with the actual branching factor of about 35). The most important of these is the null move heuristic, which generates a good lower bound on the value of a position, using a shallow search in which the opponent gets to move twice at the beginning. This lower bound often allows alpha-beta pruning without the expense of a full-depth search. Also important is futility pruning, which helps decide in advance which moves will cause a beta cutoff in the successor nodes.

The Deep Blue team declined a chance for a rematch with Kasparov. Instead, the most recent major competition in 2002 featured the program FRITZ against world champion Vladimir Kramnik. The eight game match ended in a draw. The conditions of the match were much more favorable to the human, and the hardware was an ordinary PC, not a supercomputer. Still, Kramnik commented that “It is now clear that the top program and the world champion are approximately equal.”

Checkers: Beginning in 1952, Arthur Samuel of IBM, working in his spare time, developed a checkers program that learned its own evaluation function by playing itself thousands of times. We describe this idea in more detail in Chapter 21. Samuel’s program began as a

182

Chapter 6. Adversarial Search

OTHELLO

BACKGAMMON

GO

novice, but after only a few days’ self-play had improved itself beyond Samuel’s own level (although he was not a strong player). In 1962 it defeated Robert Nealy, a champion at “blind checkers,” through an error on his part. Many people felt that this meant computers were superior to people at checkers, but this was not the case. Still, when one considers that Samuel’s computing equipment (an IBM 704) had 10,000 words of main memory, magnetic tape for long-term storage, and a .000001-GHz processor, the win remains a great accomplishment.

Few other people attempted to do better until Jonathan Schaeffer and colleagues developed Chinook, which runs on regular PCs and uses alpha-beta search. Chinook uses a precomputed database of all 444 billion positions with eight or fewer pieces on the board to make its endgame play flawless. Chinook came in second in the 1990 U.S. Open and earned the right to challenge for the world championship. It then ran up against a problem, in the form of Marion Tinsley. Dr. Tinsley had been world champion for over 40 years, losing only three games in all that time. In the first match against Chinook, Tinsley suffered his fourth and fifth losses, but won the match 20.5-18.5. The world championship match in August 1994 between Tinsley and Chinook ended prematurely when Tinsley had to withdraw for health reasons. Chinook became the official world champion.

Schaeffer believes that, with enough computing power, the database of endgames could be enlarged to the point where a forward search from the initial position would always reach solved positions, i.e., checkers would be completely solved. (Chinook has announced a win as early as move 5.) This kind of exhaustive analysis can be done by hand for 3 x 3 tic-tac-toe and has been done by computer for Qubic (4 x 4 x 4 tic-tac-toe), Go-Moku (five in a row), and Nine-Men’s Morris (Gasser, 1998). Remarkable work by Ken Thompson and Lewis Stiller (1992) solved all five-piece and some six-piece chess endgames, making them available on the Internet. Stiller discovered one case where a forced mate existed but required 262 moves; this caused some consternation because the rules of chess require some “progress” to occur within 50 moves.

Othello, also called Reversi, is probably more popular as a computer game than as a board game. It has a smaller search space than chess, usually 5 to 15 legal moves, but evaluation expertise had to be developed from scratch. In 1997, the Logistello program (Buro, 2002) defeated the human world champion, Takeshi Murakami , by six games to none. It is generally acknowledged that humans are no match for computers at Othello.

Backgammon: Section 6.5 explained why the inclusion of uncertainty from dice rolls makes deep search an expensive luxury. Most work on backgammon has gone into improving the evaluation function. Gerry Tesauro (1992) combined Samuel’s reinforcement learning method with neural network techniques (Chapter 20) to develop a remarkably accurate evaluator that is used with a search to depth 2 or 3. After playing more than a million training games against itself, Tesauro’s program, TD-GAMMON, is reliably ranked among the top three players in the world. The program’s opinions on the opening moves of the game have in some cases radically altered the received wisdom.

Go is the most popular board game in Asia, requiring at least as much discipline from its professionals as chess. Because the board is 19 x 19, the branching factor starts at 361, which is too daunting for regular search methods. Up to 1997 there were no competent

Section 6.7.

Discussion 183

BRIDGE

programs at all, but now programs often play respectable moves. Most of the best programs combine pattern recognition techniques (when the following pattern of pieces appears, this move should be considered) with limited search (decide whether these pieces can be captured, staying within the local area). The strongest programs at the time of writing are probably Chen Zhixing’s Goemate and Michael Reiss’ Go4++, each rated somewhere around 10 kyu (weak amateur). Go is an area that is likely to benefit from intensive investigation using more sophisticated reasoning methods. Success may come from finding ways to integrate several lines of local reasoning about each of the many, loosely connected “subgames” into which Go can be decomposed. Such techniques would be of enormous value for intelligent systems in general.

Bridge is a game of imperfect information: a player’s cards are hidden from the other players. Bridge is also a multiplayer game with four players instead of two, although the players are paired into two teams. As we saw in Section 6.5, optimal play in bridge can include elements of information-gathering, communication, bluffing, and careful weighing of probabilities. Many of these techniques are used in the Bridge Baron IM program (Smith et al, 1998), which won the 1997 computer bridge championship. While it does not play optimally, Bridge Baron is one of the few successful game-playing systems to use complex, hierarchical plans (see Chapter 12) involving high-level ideas such as finessing and squeezing that are familiar to bridge players.

The GIB program (Ginsberg, 1999) won the 2000 championship quite decisively. GIB uses the “averaging over clairvoyancy” method, with two crucial modifications. First, rather than examining how well each choice works for every possible arrangement of the hidden cards—of which there can be up to 10 million—it examines a random sample of 100 arrangements. Second, GIB uses explanation-based generalization to compute and cache general rules for optimal play in various standard classes of situations. This enables it to solve each deal exactly. GIB’s tactical accuracy makes up for its inability to reason about information. It finished 12th in a field of 35 in the par contest (involving just play of the hand) at the 1998 human world championship, far exceeding the expectations of many human experts.

Because calculating optimal decisions in games is intractable in most cases, all algorithms must make some assumptions and approximations. The standard approach, based on minimax, evaluation functions, and alpha-beta, is just one way to do this. Probably because it was proposed so early on, the standard approach had been developed intensively and dominates other methods in tournament play. Some in the field believe that this has caused game playing to become divorced from the mainstream of Al research, because the standard approach no longer provides much room for new insight into general questions of decision making. In this section, we look at the alternatives.

First, let us consider minimax. Minimax selects an optimal move in a given search tree provided that the leaf node evaluations are exactly correct. In reality, evaluations are

184

Chapter 6. Adversarial Search

[]

METAREASONING

Figure 6.13 A two-ply game tree for which minimax may be inappropriate.

usually crude estimates of the value of a position and can be considered to have large errors associated with them. Figure 6.13 shows a two-ply game tree for which minimax seems inappropriate. Minimax suggests taking the right-hand branch, whereas it is quite likely that the true value of the left-hand branch is higher. The minimax choice relies on the assumption that all of the nodes labeled with values 100, 101, 102, and 100 are actually better than the node labeled with value 99. However, the fact that the node labeled 99 has siblings labeled 1000 suggests that in fact it might have a higher true value. One way to deal with this problem is to have an evaluation that returns a probability distribution over possible values. Then one can calculate the probability distribution for the parent’s value using standard statistical techniques. Unfortunately, the values of sibling nodes are usually highly correlated, so this can be an expensive calculation, requiring hard to obtain information.

Next, we consider the search algorithm that generates the tree. The aim of an algorithm designer is to specify a computation that runs quickly and yields a good move. The most obvious problem with the alpha—beta algorithm is that it is designed not just to select a good move, but also to calculate bounds on the values of all the legal moves. To see why this extra information is unnecessary, consider a position in which there is only one legal move. Alpha-beta search still will generate and evaluate a large, and totally useless, search tree. Of course, we can insert a test into the algorithm, but this merely hides the underlying problem: many of the calculations done by alpha-beta are largely irrelevant. Having only one legal move is not much different from having several legal moves, one of which is fine and the rest of which are obviously disastrous. In a “clear favorite” situation like this, it would be better to reach a quick decision after a small amount of search than to waste time that could be more productively used later on a more problematic position. This leads to the idea of the utility of a node expansion. A good search algorithm should select node expansions of high utility—that is, ones that are likely to lead to the discovery of a significantly better move. If there are no node expansions whose utility is higher than their cost (in terms of time), then the algorithm should stop searching and make a move. Notice that this works not only for clear-favorite situations, but also for the case of symmetrical moves, for which no amount of search will show that one move is better than another.

This kind of reasoning about what computations to do is called metareasoning (reasoning about reasoning). It applies not just to game playing, but to any kind of reasoning

Section 6.8.

Summary 185

at all. All computations are done in the service of trying to reach better decisions, all have costs, and all have some likelihood of resulting in a certain improvement in decision quality. Alpha-beta incorporates the simplest kind of metareasoning, namely, a theorem to the effect that certain branches of the tree can be ignored without loss. It is possible to do much better. In Chapter 16, we will see how these ideas can be made precise and implementable.

Finally, let us reexamine the nature of search itself. Algorithms for heuristic search and for game playing work by generating sequences of concrete states, starting from the initial state and then applying an evaluation function. Clearly, this is not how humans play games. In chess, one often has a particular goal in mind—for example, trapping the opponent’s queen— and can use this goal to selectively generate plausible plans for achieving it. This kind of goaldirected reasoning or planning sometimes eliminates combinatorial search altogether. (See Part IV.) David Wilkins’ (1980) PARADISE is the only program to have used goal-directed reasoning successfully in chess: it was capable of solving some chess problems requiring an 18-move combination. As yet there is no good understanding of how to combine the two kinds of algorithm into a robust and efficient system, although Bridge Baron might be a step in the right direction. A fully integrated system would be a significant achievement not just for game-playing research, but also for Al research in general, because it would be a good basis for a general intelligent agent.

We have looked at a variety of games to understand what optimal play means and to understand how to play well in practice. The most important ideas are as follows:

e A game can be defined by the initial state (how the board is set up), the legal actions in each state, a terminal test (which says when the game is over), and a utility function that applies to terminal states.

e In two-player zero-sum games with perfect information, the minimax algorithm can select optimal moves using a depth-first enumeration of the game tree.

e The alpha-beta search algorithm computes the same optimal move as minimax, but achieves much greater efficiency by eliminating subtrees that are provably irrelevant.

e Usually, it is not feasible to consider the whole game tree (even with alpha-beta), so we need to cut the search off at some point and apply an evaluation function that gives an estimate of the utility of a state.

e Games of chance can be handled by an extension to the minimax algorithm that evaluates a chance node by taking the average utility of all its children nodes, weighted by the probability of each child.

e Optimal play in games of imperfect information, such as bridge, requires reasoning about the current and future belief states of each player. A simple approximation can be obtained by averaging the value of an action over each possible configuration of missing information.

186 Chapter 6. Adversarial Search

e Programs can match or beat the best human players in checkers, Othello, and backgammon and are close behind in bridge. A program has beaten the world chess champion in one exhibition match. Programs remain at the amateur level in Go.

BIBLIOGRAPHICAL AND HISTORICAL NOTES

The early history of mechanical game playing was marred by numerous frauds. The most notorious of these was Baron Wolfgang von Kempelen’s (1734-1804) “The Turk,” a supposed chess-playing automaton that defeated Napoleon before being exposed as a magician’s trick cabinet housing a human chess expert (see Levitt, 2000). It played from 1769 to 1854. In 1846, Charles Babbage (who had been fascinated by the Turk) appears to have contributed the first serious discussion of the feasibility of computer chess and checkers (Morrison and Morrison, 1961). He also designed, but did not build, a special-purpose machine for playing tic-tac-toe. The first true game-playing machine was built around 1890 by the Spanish engineer Leonardo Torres y Quevedo. It specialized in the “KRK” (king and rook vs. king) chess endgame, guaranteeing a win with king and rook from any position.

The minimax algorithm is often traced to a paper published in 1912 by Ernst Zermelo, the developer of modern set theory. The paper unfortunately contained several errors and did not describe minimax correctly. A solid foundation for game theory was developed in the seminal work Theory of Games and Economic Behavior (von Neumann and Morgenstern, 1944), which included an analysis showing that some games require strategies that are randomized (or otherwise unpredictable). See Chapter 17 for more information.

Many influential figures of the early computer era were intrigued by the possibility of computer chess, Konrad Zuse (1945), the first person to design a programmable computer, developed fairly detailed ideas about how it might be done. Norbert Wiener’s (1948) influential book Cybernetics discussed one possible design for a chess program, including the ideas of minimax search, depth cutoffs, and evaluation functions. Claude Shannon (1950) laid out the basic principles of modern game-playing programs in much more detail than Wiener. He introduced the idea of quiescence search and described some ideas for selective (nonexhaustive) game-tree search. Slater (1950) and the commentators on his article also explored the possibilities for computer chess play. In particular, I. J. Good (1950) developed the notion of quiescence independently of Shannon.

In 1951, Alan Turing wrote the first computer program capable of playing a full game of chess (see Turing ef al., 1953). But Turing’s program never actually ran on a computer; it was tested by hand simulation against a very weak human player, who defeated it. Meanwhile D. G. Prinz (1952) had written, and actually run, a program that solved chess problems, although it did not play a full game. Alex Bernstein wrote the first program to play a full game of standard chess (Bernstein and Roberts, 1958; Bernstein ef al., 1958).

John McCarthy conceived the idea of alpha~beta search in 1956, although he did not publish it. The NSS chess program (Newell ef al., 1958) used a simplified version of alpha—

3 Newell et al. (1958) mention a Russian program, BESM, that may have predated Bernstein’s program.

Section 6.8.

Summary 187

beta; it was the first chess program to do so. According to Nilsson (1971), Arthur Samuel’s checkers program (Samuel, 1959, 1967) also used alpha-beta, although Samuel did not mention it in the published reports on the system. Papers describing alpha—beta were published in the early 1960s (Hart and Edwards, 1961; Brudno, 1963; Slagle, 1963b). An implementation of full alpha-beta is described by Slagle and Dixon (1969) in a program for playing the game of Kalah. Alpha-beta was also used by the “Kotok—McCarthy” chess program written by a student of John McCarthy (Kotok, 1962). Knuth and Moore (1975) provide a history of alpha—-beta, along with a proof of its correctness and a time complexity analysis. Their analysis of alpha—beta with random successor ordering showed an asymptotic complexity of O((b/log b)?), which seemed rather dismal because the effective branching factor b/log b is not much less than b itself. They then realized that the asymptotic formula is accurate only for b > 1000 or so, whereas the often-quoted O (b>) applies to the range of branching factors encountered in actual games. Pearl (1982b) shows alpha-beta to be asymptotically optimal among all fixed-depth game-tree search algorithms.

The first computer chess match featured the Kotok-McCarthy program and the “ITEP” program written in the mid-1960s at Moscow’s Institute of Theoretical and Experimental Physics (Adelson-Velsky et al., 1970). This intercontinental match was played by telegraph. It ended with a 3~1 victory for the ITEP program in 1967. The first chess program to compete successfully with humans was MacHack 6 (Greenblatt ef al., 1967). Its rating of approximately 1400 was well above the novice level of 1000, but it fell far short of the rating of 2800 or more that would have been needed to fulfill Herb Simon’s 1957 prediction that a computer program would be world chess champion within 10 years (Simon and Newell, 1958).

Beginning with the first ACM North American Computer-Chess Championship in 1970, competition among chess programs became serious. Programs in the early 1970s became extremely complicated, with various kinds of tricks for eliminating some branches of search, for generating plausible moves, and so on. In 1974, the first World Computer-Chess Championship was held in Stockholm and won by Kaissa (Adelson-Velsky et al., 1975), another program from ITEP. Kaissa used the much more straightforward approach of exhaustive alpha beta search combined with quiescence search. The dominance of this approach was confirmed by the convincing victory of CHESS 4.6 in the 1977 World Computer-Chess Championship. CHESS 4.6 examined up to 400,000 positions per move and had a rating of 1900.

A later version of Greenblatt’s MacHack 6 was the first chess program to run on custom hardware designed specifically for chess (Moussouris ef al., 1979), but the first program to achieve notable success through the use of custom hardware was Belle (Condon and Thompson, 1982). Belle’s move generation and position evaluation hardware enabled it to explore several million positions per move. Belle achieved a rating of 2250, becoming the first master-level program. The HITECH system, also a special-purpose computer, was designed by former World Correspondence Chess Champion Hans Berliner and his student Carl Ebeling at CMU to allow rapid calculation of evaluation functions (Ebeling, 1987; Berliner and Ebeling, 1989). Generating about 10 million positions per move, HITECH became North American computer champion in 1985 and was the first program to defeat a human grandmaster, in 1987. Deep Thought, which was also developed at CMU, went further in the direction of pure search speed (Hsu et al., 1990). It achieved a rating of 2551 and was the

188

Chapter 6. Adversarial Search

forerunner of Deep Blue. The Fredkin Prize, established in 1980, offered $5000 to the first program to achieve a master rating, $10,000 to the first program to achieve a USCF (United States Chess Federation) rating of 2500 (near the grandmaster level), and $100,000 for the first program to defeat the human world champion. The $5000 prize was claimed by Belle in 1983, the $10,000 prize by Deep Thought in 1989, and the $100,000 prize by Deep Blue for its victory over Garry Kasparov in 1997. It is important to remember that Deep Blue’s success was due to algorithmic improvements as well as hardware (Hsu, 1999; Campbell ef al., 2002). Techniques such as the null-move heuristic (Beal, 1990) have led to programs that are quite selective in their searches. The last three World Computer-Chess Championships in 1992, 1995, and 1999 were won by programs running on standard PCs. Probably the most complete description of a modern chess program is provided by Ernst Heinz (2000), whose DARKTHOUGHT program was the highest-ranked noncommercial PC program at the 1999 world championships.

Several attempts have been made to overcome the problems with the “standard approach” that were outlined in Section 6.7. The first selective search algorithm with some theoretical grounding was probably B\* (Berliner, 1979), which attempts to maintain interval bounds on the possible value of a node in the game tree, rather than giving it a single pointvalued estimate. Leaf nodes are selected for expansion in an attempt to refine the top-level bounds until one move is “clearly best.” Palay (1985) extends the B\* idea using probability distributions on values in place of intervals. David McAllester’s (1988) conspiracy number search expands leaf nodes that, by changing their values, could cause the program to prefer a new move at the root. MGSS\* (Russell and Wefald, 1989) uses the decision-theoretic techniques of Chapter 16 to estimate the value of expanding each leaf in terms of the expected improvement in decision quality at the root. It outplayed an alpha-beta algorithm at Othello despite searching an order of magnitude fewer nodes. The MGSS\* approach is, in principle, applicable to the control of any form of deliberation.

Alpha-beta search is in many ways the two-player analog of depth-first branch-andbound, which is dominated by A" in the single-agent case. The SSS\* algorithm (Stockman, 1979) can be viewed as a two-player A\* and never expands more nodes than alpha-beta to reach the same decision. The memory requirements and computational overhead of the queue make SSS\* in its original form impractical, but a linear-space version has been developed from the RBFS algorithm (Korf and Chickering, 1996). Plaat et al. (1996) developed a new view of SSS\* as a combination of alpha~beta and transposition tables, showing how to overcome the drawbacks of the original algorithm and developing a new variant called MTD(f) that has been adopted by a number of top programs.

D. F. Beal (1980) and Dana Nau (1980, 1983) studied the weaknesses of minimax applied to approximate evaluations. They showed that under certain independence assumptions about the distribution of leaf values in the tree, minimaxing can yield values at the root that are actually less reliable than the direct use of the evaluation function itself. Pearl’s book Heuristics (1984) partially explains this apparent paradox and analyzes many game-playing algorithms. Baum and Smith (1997) propose a probability-based replacement for minimax, showing that it results in better choices in certain games. There is still little theory of the effects of cutting off search at different levels and applying evaluation functions.

Section 6.8.

Summary 189

The expectiminimax algorithm was proposed by Donald Michie (1966), although of course it follows directly from the principles of game-tree evaluation due to von Neumann and Morgenstern. Bruce Ballard (1983) extended alpha-beta pruning to cover trees with chance nodes. The first successful backgammon program was BKG (Berliner, 1977, 1980b); it used a complex, manually constructed evaluation function and searched only to depth 1. Tt was the first program to defeat a human world champion at a major classic game (Berliner, 1980a). Berliner readily acknowledged that this was a very short exhibition match (not a world championship match) and that BKG was very lucky with the dice. Work by Gerry Tesauro, first on NEUROGAMMON (Tesauro, 1989) and later on TD-GAMMON (Tesauro, 1995), showed that much better results could be obtained via reinforcement learning, which we will cover in Chapter 21.

Checkers, rather than chess, was the first of the classic games fully played by a computer. Christopher Strachey (1952) wrote the first working program for checkers. Schaeffer (1997) gives a highly readable, “warts and all” account of the development of his Chinook world champion checkers program.

The first Go-playing programs were developed somewhat later than those for checkers and chess (Lefkovitz, 1960; Remus, 1962) and have progressed more slowly. Ryder (1971) used a pure search-based approach with a variety of selective pruning methods to overcome the enormous branching factor. Zobrist (1970) used condition—action rules to suggest plausible moves when known patterns appeared. Reitman and Wilcox (1979) combined rules and search to good effect, and most modern programs have followed this hybrid approach. Miiller (2002) summarizes the state of the art of computerized Go and provides a wealth of references. Anshelevich (2000) uses related techniques for the game of Hex. The Computer Go Newsletter, published by the Computer Go Association, describes current developments.

Papers on computer game playing appear in a variety of venues. The rather misleadingly named conference proceedings Heuristic Programming in Artificial Intelligence report on the Computer Olympiads, which include a wide variety of games. There are also several edited collections of important papers on game-playing research (Levy, 1988a, 1988b; Marsland and Schaeffer, 1990). The International Computer Chess Association (ICCA), founded in 1977, publishes the quarterly /CGA Journal (formerly the ICCA Journal). Important papers have been published in the serial anthology Advances in Computer Chess, starting with Clarke (1977). Volume 134 of the journal Artificial Intelligence (2002) contains descriptions of stateof-the-art programs for chess, Othello, Hex, shogi, Go, backgammon, poker, Scrabble. ™and other games.

EXERCISES

6.1 This problem exercises the basic concepts of game playing, using tic-tac-toe (noughts and crosses) as an example. We define X,, as the number of rows, columns, or diagonals with exactly n X's and no O’s. Similarly, O,, is the number of rows, columns, or diagonals with just n O’s. The utility function assigns +1 to any position with X53 = 1 and —1 to any

190

Chapter 6. Adversarial Search

6.2

a “?” in a circle.

if n is even and loses if n is odd.

o

1

2

3

4

Section 6.8.

Summary 191

6.4 Implement move generators and evaluation functions for one or more of the following games: Kalah, Othello, checkers, and chess. Construct a general alpha-beta game-playing agent that uses your implementation. Compare the effect of increasing search depth, improving move ordering, and improving the evaluation function. How close does your effective branching factor come to the ideal case of perfect move ordering?

6.5 Develop a formal proof of correctness for alpha-beta pruning. To do this, consider the situation shown in Figure 6.15. The question is whether to prune node n;, which is a maxnode and a descendant of node n,. The basic idea is to prune it if and only if the minimax value of m1 can be shown to be independent of the value of 7;.

a. The value of nj is given by np = min(ng, N21, ...,Nap,)

Find a similar expression for ny and hence an expression for ny in terms of n;.

b. Let; be the minimum (or maximum) value of the nodes to the left of node rn; at depth i, whose minimax value is already known. Similarly, let r; be the minimum (or maximum) value of the unexplored nodes to the right of n; at depth i. Rewrite your expression for n1 in terms of the [; and r; values.

c. Now reformulate the expression to show that in order to affect 721, n; must not exceed a certain bound derived from the /; values.

d. Repeat the process for the case where n; is a min-node.

6.6 Implement the expectiminimax algorithm and the \*-alpha~beta algorithm, which is described by Ballard (1983), for pruning game trees with chance nodes. Try them on a game such as backgammon and measure the pruning effectiveness of \*-alpha-beta.

Figure 6.15 Situation when considering whether to prune node n;.

192

Chapter 6. Adversarial Search

6.7 Prove that with a positive linear transformation of leaf values (i.e., transforming a value x to ax + b where a > 0), the choice of move remains unchanged in a game tree, even when there are chance nodes.

6.8 Consider the following procedure for choosing moves in games with chance nodes:

e Generate some die-roll sequences (say, 50) down to a suitable depth (say, 8).

e With known die rolls, the game tree becomes deterministic. For each die-roll sequence, solve the resulting deterministic game tree using alpha-beta.

e Use the results to estimate the value of each move and to choose the best.

Will this procedure work well? Why (not)?

6.9 Describe and implement a real-time, multiplayer game-playing environment, where time is part of the environment state and players are given fixed time allocations.

6.10 Describe or implement state descriptions, move generators, terminal tests, utility functions, and evaluation functions for one or more of the following games: Monopoly, Scrabble, bridge (assuming a given contract), and poker (choose your favorite variety).

6.11 Consider carefully the interplay of chance events and partial information in each of the games in Exercise 6.10.

a. For which is the standard expectiminimax model appropriate? Implement the algorithm and run it in your game-playing agent, with appropriate modifications to the gameplaying environment.

b. For which would the scheme described in Exercise 6.8 be appropriate?

c. Discuss how you might deal with the fact that in some of the games, the players do not have the same knowledge of the current state.

6.12 The minimax algorithm assumes that players take turns moving, but in card games such as whist and bridge, the winner of the previous trick plays first on the next trick.

a. Modify the algorithm to work properly for these games. You may assume that a function WINNER(s) is available that reports which player won the trick just completed (if any).

b. Draw the game tree for the first pair of hands shown on page 179.

6.13 The Chinook checkers program makes extensive use of endgame databases, which provide exact values for every position with eight or fewer pieces. How might such databases be generated efficiently?

6.14 Discuss how well the standard approach to game playing would apply to games such as tennis, pool, and croquet, which take place in a continuous physical state space.

6.15 Describe how the minimax and alpha-beta algorithms change for two-player, nonzero-sum games in which each player has his or her own utility function. You may assume that each player knows the other’s utility function. If there are no constraints on the two terminal utilities, is it possible for any node to be pruned by alpha-beta?

Section 6.8.

Summary 193

6.16 Suppose you have a chess program that can evaluate 1 million nodes per second. Decide on a compact representation of a game state for storage in a transposition table. About how many entries can you fit in a 500MB in-memory table? Will that be enough for the three minutes of search allocated for one move? How many table lookups can you do in the time it would take to do one evaluation? Now suppose the transposition table is larger than can fit in memory. About how many evaluations could you do in the time it takes to do one disk seek with standard disk hardware?

LOGICAL AGENTS

In which we design agents that can form representations of the world, use a process of inference to derive new representations about the world, and use these new representations to deduce what to do.

This chapter introduces knowledge-based agents. The concepts that we discuss—the representation of knowledge and the reasoning processes that bring knowledge to life—are central to the entire field of artificial intelligence.

Humans, it seems, know things and do reasoning. Knowledge and reasoning are also important for artificial agents because they enable successful behaviors that would be very hard to achieve otherwise. We have seen that knowledge of action outcomes enables problemsolving agents to perform well in complex environments. A reflex agents could only find its way from Arad to Bucharest by dumb luck. The knowledge of problem-solving agents is, however, very specific and inflexible. A chess program can calculate the legal moves of its king, but does not know in any useful sense that no piece can be on two different squares at the same time. Knowledge-based agents can benefit from knowledge expressed in very general forms, combining and recombining information to suit myriad purposes. Often, this process can be quite far removed from the needs of the moment—as when a mathematician proves a theorem or an astronomer calculates the earth’s life expectancy.

Knowledge and reasoning also play a crucial role in dealing with partially observable environments. A knowledge-based agent can combine general knowledge with current percepts to infer hidden aspects of the current state prior to selecting actions. For example, a physician diagnoses a patient—that is, infers a disease state that is not directly observable— prior to choosing a treatment. Some of the knowledge that the physician uses is in the form of rules learned from textbooks and teachers, and some is in the form of patterns of association that the physician may not be able to consciously describe. If it’s inside the physician’s head, it counts as knowledge.

Understanding natural language also requires inferring hidden state, namely, the intention of the speaker. When we hear, “John saw the diamond through the window and coveted it,” we know “it” refers to the diamond and not the window—we reason, perhaps unconsciously, with our knowledge of relative value. Similarly, when we hear, “John threw the brick through the window and broke it,” we know “it” refers to the window. Reasoning allows

194

Section 7.1.

Knowledge-Based Agents 195

us to cope with the virtually infinite variety of utterances using a finite store of commonsense knowledge. Problem-solving agents have difficulty with this kind of ambiguity because their representation of contingency problems is inherently exponential.

Our final reason for studying knowledge-based agents is their flexibility. They are able to accept new tasks in the form of explicitly described goals, they can achieve competence quickly by being told or learning new knowledge about the environment, and they can adapt to changes in the environment by updating the relevant knowledge.

We begin in Section 7.1 with the overall agent design. Section 7.2 introduces a simple new environment, the wumpus world, and illustrates the operation of a knowledge-based agent without going into any technical detail. Then, in Section 7.3, we explain the general principles of logic. Logic will be the primary vehicle for representing knowledge throughout Part III of the book. The knowledge of logical agents is always definite—each proposition is either true or false in the world, although the agent may be agnostic about some propositions.

Logic has the pedagogical advantage of being simple example of a representation for knowledge-based agents, but logic has some severe limitations. Clearly, a large portion of the reasoning carried out by humans and other agents in partially observable environments depends on handling knowledge that is uncertain. Logic cannot represent this uncertainty well, so in Part V we cover probability, which can. In Part VI and Part VII we cover many representations, including some based on continuous mathematics such as mixtures of Gaussians, neural networks, and other representations.

Section 7.4 of this chapter defines a simple logic called propositional logic. While much less expressive than first-order logic (Chapter 8), propositional logic serves to illustrate all the basic concepts of logic. There is also a well-developed technology for reasoning in propositional logic, which we describe in sections 7.5 and 7.6. Finally, Section 7.7 combines the concept of logical agents with the technology of propositional logic to build some simple agents for the wumpus world. Certain shortcomings in propositional logic are identified, motivating the development of more powerful logics in subsequent chapters.

KNOWLEDGE BASE SENTENCE

KNOWLEDGE REPRESENTATION LANGUAGE

INFERENCE LOGICAL AGENTS

The central component of a knowledge-based agent is its knowledge base, or KB. Informally, a knowledge base is a set of sentences. (Here “sentence” is used as a technical term. It is related but is not identical to the sentences of English and other natural languages.) Each sentence is expressed in a language called a knowledge representation language and represents some assertion about the world.

There must be a way to add new sentences to the knowledge base and a way to query what is known. The standard names for these tasks are TELL and ASK, respectively. Both tasks may involve inference—that is, deriving new sentences from old. In logical agents, which are the main subject of study in this chapter, inference must obey the fundamental requirement that when one ASKs a question of the knowledge base, the answer should follow from what has been told (or rather, TELLed) to the knowledge base previously. Later in the

196

Chapter 7. Logical Agents

BACKGROUND KNOWLEDGE

KNOWLEDGE LEVEL

IMPLEMENTATION LEVEL

function KB-AGENT( percept) returns an action static: KB, a knowledge base t, a counter, initially 0, indicating time

TELL( KB, MAKE-PERCEPT-SENTENCE( percept, t)) action «— ASK(KB,MAKE-ACTION-QUERY(t)) TELL(KB, MAKE-ACTION-SENTENCE(action, t)) t—t+1

return action

Figure 7.1 A generic knowledge-based agent.

chapter, we will be more precise about the crucial word “follow.” For now, take it to mean that the inference process should not just make things up as it goes along.

Figure 7.1 shows the outline of a knowledge-based agent program. Like all our agents, it takes a percept as input and returns an action. The agent maintains a knowledge base, KB, which may initially contain some background knowledge. Each time the agent program is called, it does three things. First, it TELLs the knowledge base what it perceives. Second, it ASKs the knowledge base what action it should perform. In the process of answering this query, extensive reasoning may be done about the current state of the world, about the outcomes of possible action sequences, and so on. Third, the agent records its choice with TELL and executes the action. The second TELL is necessary to let the knowledge base know that the hypothetical action has actually been executed.

The details of the representation language are hidden inside three functions that implement the interface between the sensors and actuators and the core representation and reasoning system. MAKE-PERCEPT-SENTENCE constructs a sentence asserting that the agent perceived the given percept at the given time. MAKEACTION-QUERY constructs a sentence that asks what action should be done at the current time. Finally, MAKEACTION-SENTENCE constructs a sentence asserting that the chosen action was executed. The details of the inference mechanisms are hidden inside TELL and ASK. Later sections will reveal these details.

The agent in Figure 7.1 appears quite similar to the agents with internal state described in Chapter 2. Because of the definitions of TELL and ASK, however, the knowledge-based agent is not an arbitrary program for calculating actions. It is amenable to a description at the knowledge level, where we need specify only what the agent knows and what its goals are, in order to fix its behavior. For example, an automated taxi might have the goal of delivering a passenger to Marin County and might know that it is in San Francisco and that the Golden Gate Bridge is the only link between the two locations. Then we can expect it to cross the Golden Gate Bridge because it knows that that will achieve its goal. Notice that this analysis is independent of how the taxi works at the implementation level. It doesn’t matter whether its geographical knowledge is implemented as linked lists or pixel maps, or whether it reasons by manipulating strings of symbols stored in registers or by propagating noisy signals in a network of neurons.

Section 7.2.

The Wumpus World 197

DECLARATIVE

As we mentioned in the introduction to the chapter, one can build a knowledge-based agent simply by TELLing it what it needs to know. The agent’s initial program, before it starts to receive percepts, is built by adding one by one the sentences that represent the designer’s knowledge of the environment. Designing the representation language to make it easy to express this knowledge in the form of sentences simplifies the construction problem enormously. This is called the declarative approach to system building. In contrast, the procedural approach encodes desired behaviors directly as program code; minimizing the role of explicit representation and reasoning can result in a much more efficient system. We will see agents of both kinds in Section 7.7. In the 1970s and 1980s, advocates of the two approaches engaged in heated debates. We now understand that a successful agent must combine both declarative and procedural elements in its design.

In addition to TELLing it what it needs to know, we can provide a knowledge-based agent with mechanisms that allow it to learn for itself. These mechanisms, which are discussed in Chapter 18, create general knowledge about the environment out of a series of percepts. This knowledge can be incorporated into the agent’s knowledge base and used for decision making. In this way, the agent can be fully autonomous.

All these capabilities—representation, reasoning, and learning—rest on the centurieslong development of the theory and technology of logic. Before explaining that theory and technology, however, we will create a simple world with which to illustrate them.

WUMPUS WORLD

The wumpus world is a cave consisting of rooms connected by passageways. Lurking somewhere in the cave is the wumpus, a beast that eats anyone who enters its room. The wumpus can be shot by an agent, but the agent has only one arrow. Some rooms contain bottomless pits that will trap anyone who wanders into these rooms (except for the wumpus, which is too big to fall in). The only mitigating feature of living in this environment is the possibility of finding a heap of gold. Although the wumpus world is rather tame by modern computer game standards, it makes an excellent testbed environment for intelligent agents. Michael Genesereth was the first to suggest this.

A sample wumpus world is shown in Figure 7.2. The precise definition of the task environment is given, as suggested in Chapter 2, by the PEAS description:

<> Performance measure: +1000 for picking up the gold, —1000 for falling into a pit or being eaten by the wumpus, —1 for each action taken and —10 for using up the arrow.

<> Environment: A 4 x 4 grid of rooms. The agent always starts in the square labeled [1,1], facing to the right. The locations of the gold and the wumpus are chosen randomly, with a uniform distribution, from the squares other than the start square. In addition, each square other than the start can be a pit, with probability 0.2.

{> Actuators: The agent can move forward, turn left by 90°, or turn right by 90°. The agent dies a miserable death if it enters a square containing a pit or a live wumpus. (It is safe, albeit smelly, to enter a square with a dead wumpus.) Moving forward has no

198 Chapter 7. Logical Agents

effect if there is a wall in front of the agent. The action Grab can be used to pick up an object that is in the same square as the agent. The action Shoot can be used to fire an arrow in a straight line in the direction the agent is facing. The arrow continues until it either hits (and hence kills) the wumpus or hits a wall. The agent only has one arrow, so only the first Shoot action has any effect.

{> Sensors: The agent has five sensors, each of which gives a single bit of information:

— In the square containing the wumpus and in the directly (not diagonally) adjacent squares the agent will perceive a stench.

— In the squares directly adjacent to a pit, the agent will perceive a breeze. — In the square where the gold is, the agent will perceive a glitter. — When an agent walks into a wall, it will perceive a bump.

— When the wumpus is killed, it emits a woeful scream that can be perceived anywhere in the cave.

The percepts will be given to the agent in the form of a list of five symbols; for example, if there is a stench and a breeze, but no glitter, bump, or scream, the agent will receive the percept Stench, Breeze, None, None, None).

Exercise 7.1 asks you to define the wumpus environment along the various dimensions given in Chapter 2. The principal difficulty for the agent is its initial ignorance of the configuration of the environment; overcoming this ignorance seems to require logical reasoning. In most instances of the wumpus world, it is possible for the agent to retrieve the gold safely. Occasionally, the agent must choose between going home empty-handed and risking death to find the gold. About 21% of the environments are utterly unfair, because the gold is in a pit or surrounded by pits.

Let us watch a knowledge-based wumpus agent exploring the environment shown in Figure 7.2. The agent’s initial knowledge base contains the rules of the environment, as listed

4 | S&S “Bee —| IN SSS 5 | Gl |= Goa SZ Br—, 2 Sener ALI 1 f Eel rir JERE START 1 2 3 4 Figure 7.2 A typical wumpus world. The agent is in the bottom left corner.

Section 7.2.

The Wumpus World 199

previously; in particular, it knows that it is in [1,1] and that [1,1] is a safe square. We will see how its knowledge evolves as new percepts arrive and actions are taken.

The first percept is [None, None, None, None, None], from which the agent can conclude that its neighboring squares are safe. Figure 7.3(a) shows the agent’s state of knowledge at this point. We list (some of) the sentences in the knowledge base using letters such as B (breezy) and OK (safe, neither pit nor wumpus) marked in the appropriate squares. Figure 7.2, on the other hand, depicts the world itself.

[]

Figure 7.3 The first step taken by the agent in the wumpus world. (a) The initial situation, after percept [ None, None, None, None, None]. (b) After one move, with percept [None, Breeze, None, None, None).

From the fact that there was no stench or breeze in [1,1], the agent can infer that [1,2] and [2,1] are free of dangers. They are marked with an OK to indicate this. A cautious agent will move only into a square that it knows is OK. Let us suppose the agent decides to move forward to [2,1], giving the scene in Figure 7.3(b).

The agent detects a breeze in [2,1], so there must be a pit in a neighboring square. The pit cannot be in [1,1], by the rules of the game, so there must be a pit in [2,2] or [3,1] or both. The notation P? in Figure 7.3(b) indicates a possible pit in those squares. At this point, there is only one known square that is OK and has not been visited yet. So the prudent agent will turn around, go back to [1,1], and then proceed to [1,2].

The new percept in [1,2] is [Stench, None, None, None, None], resulting in the state of knowledge shown in Figure 7.4(a). The stench in [1,2] means that there must be a wumpus nearby. But the wumpus cannot be in [1,1], by the rules of the game, and it cannot be in [2,2] (or the agent would have detected a stench when it was in [2,1]). Therefore, the agent can infer that the wumpus is in [1,3]. The notation W! indicates this. Moreover, the lack of a Breeze in [1,2] implies that there is no pit in [2,2]. Yet we already inferred that there must be a pit in either [2,2] or [3,1], so this means it must be in [3,1]. This is a fairly difficult inference, because it combines knowledge gained at different times in different places and

200

Chapter 7. Logical Agents

[]

1,4 2,4 34 44 = Agent 1,4 24, [34 44 B =Breeze : G = Glitter, Gold OK = Safe square 1,3 wi 2,3 3,3 4,3 P =Pit 1.3 wr 2313] 3.3 po 43 S =Stench SG V = Visited B W = Wumpus "2[3] 2.2 3,2 4,2 1,2 2.2 3,2 4,2 A s S v v OK OK OK OK 1,1 21 5 [315 [41 1,1 21 5 [31 5 [4 v v v v OK OK OK OK (a) (b) Figure 74 Two later stages in the progress of the agent. (a) After the third move, with percept [Stench, None, None, None, None]. (b) After the fifth move, with percept (Stench, Breeze, Glitter, None, None).

relies on the lack of a percept to make one crucial step. The inference is beyond the abilities of most animals, but it is typical of the kind of reasoning that a logical agent does.

The agent has now proved to itself that there is neither a pit nor a wumpus in [2,2], so it is OK to move there. We will not show the agent’s state of knowledge at [2,2]; we just assume that the agent turns and moves to [2,3], giving us Figure 7.4(b). In [2,3], the agent detects a glitter, so it should grab the gold and thereby end the game.

In each case where the agent draws a conclusion from the available information, that conclusion is guaranteed to be correct if the available information is correct. This is a fundamental property of logical reasoning. In the rest of this chapter, we describe how to build logical agents that can represent the necessary information and draw the conclusions that were described in the preceding paragraphs.

SYNTAX

This section provides an overview of all the fundamental concepts of logical representation and reasoning. We postpone the technical details of any particular form of logic until the next section. We will instead use informal examples from the wumpus world and from the familiar realm of arithmetic. We adopt this rather unusual approach because the ideas of logic are far more general and beautiful than is commonly supposed.

In Section 7.1, we said that knowledge bases consist of sentences. These sentences are expressed according to the syntax of the representation language, which specifies all the sentences that are well formed. The notion of syntax is clear enough in ordinary arithmetic: “r 4+ y = 4” is a well-formed sentence, whereas “z2y+ =" is not. The syntax of logical

Logic 201

SEMANTICS

TRUTH POSSIBLE WORLD

MODEL

ENTAILMENT

languages (and of arithmetic, for that matter) is usually designed for writing papers and books. There are literally dozens of different syntaxes, some with lots of Greek letters and exotic mathematical symbols, and some with rather visually appealing diagrams with arrows and bubbles. In all cases, however, sentences in an agent’s knowledge base are real physical configurations of (parts of) the agent. Reasoning will involve generating and manipulating those configurations.

A logic must also define the semantics of the language. Loosely speaking, semantics has to do with the “meaning” of sentences. In logic, the definition is more precise. The semantics of the language defines the truth of each sentence with respect to each possible world. For example, the usual semantics adopted for arithmetic specifies that the sentence “x + y=4" is true in a world where x is 2 and y is 2, but false in a world where x is 1 and y is 1.! In standard logics, every sentence must be either true or false in each possible world—there is no “in between.”

When we need to be precise, we will use the term model in place of “possible world.” (We will also use the phrase “m is a model of «” to mean that sentence « is true in model m.) Whereas possible worlds might be thought of as (potentially) real environments that the agent might or might not be in, models are mathematical abstractions, each of which simply fixes the truth or falsehood of every relevant sentence. Informally, we may think of x and y as the number of men and women sitting at a table playing bridge, for example, and the sentence x + y = 4 is true when there are four in total; formally, the possible models are just all possible assignments of numbers to the variables x and y. Each such assignment fixes the truth of any sentence of arithmetic whose variables are x and y.

Now that we have a notion of truth, we are ready to talk about logical reasoning. This involves the relation of logical entailment between sentences—the idea that a sentence follows logically from another sentence. In mathematical notation, we write as

op

to mean that the sentence « entails the sentence 3. The formal definition of entailment is this: a |= if and only if, in every model in which « is true, 3 is also true. Another way to say this is that if « is true, then 3 must also be true. Informally, the truth of 3 is “contained” in the truth of ce. The relation of entailment is familiar from arithmetic; we are happy with the idea that the sentence x + y = 4 entails the sentence 4 =x + y. Obviously, in any model where x + y = 4—such as the model in which z is 2 and y is 2—it is the case that 4 =x + y. We will see shortly that a knowledge base can be considered a statement, and we often talk of a knowledge base entailing a sentence.

We can apply the same kind of analysis to the wumpus-world reasoning example given in the preceding section. Consider the situation in Figure 7.3(b): the agent has detected nothing in [1,1] and a breeze in [2,1]. These percepts, combined with the agent’s knowledge of the rules of the wumpus world (the PEAS description on page 197), constitute the KB. The

1 The reader will no doubt have noticed the similarity between the notion of truth of sentences and the notion of satisfaction of constraints in Chapter 5. This is no accident—constraint languages are indeed logics and constraint solving is a form of logical reasoning.

2 Fuzzy logic, discussed in Chapter 14, allows for degrees of truth.

202

Chapter 7. Logical Agents

LOGICAL INFERENCE MODEL CHECKING

Figure 7.5 Possible models for the presence of pits in squares [1,2], [2,2], and [3,1], given observations of nothing in [1,1] and a breeze in [2,1]. (a) Models of the knowledge base and ay (no pit in [1,2]). (b) Models of the knowledge base and az (no pit in [2,2]).

agent is interested (among other things) in whether the adjacent squares [1,2], [2,2], and [3,1] contain pits. Each of the three squares might or might not contain a pit, so (for the purposes of this example) there are 2% = 8 possible models. These are shown in Figure 7.5.3

The KB is false in models that contradict what the agent knows—for example, the KB is false in any model in which [1,2] contains a pit, because there is no breeze in [1,1]. There are in fact just three models in which the KB is true, and these are shown as a subset of the models in Figure 7.5. Now let us consider two possible conclusions:

aq = “There is no pit in [1,2].”

ay = “There is no pit in [2,2]. We have marked the models of «; and «9 in Figures 7.5(a) and 7.5(b) respectively. By inspection, we see the following:

in every model in which KB is true, « is also true. Hence, KB |= a;: there is no pit in [1,2]. We can also see that in some models in which KB is true, a is false.

Hence, KB [~ ao: the agent cannot conclude that there is no pit in [2,2]. (Nor can it conclude that there is a pit in [2,2].)\*

The preceding example not only illustrates entailment, but also shows how the definition of entailment can be applied to derive conclusions—that is, to carry out logical inference, The inference algorithm illustrated in Figure 7.5 is called model checking, because it enumerates all possible models to check that « is true in all models in which KB is true.

3 Although the figure shows the models as partial wumpus worlds, they are really nothing more than assignments of true and false to the sentences “there is a pit in [1,2]” etc. Models, in the mathematical sense, do not need to have ’orrible \*airy wumpuses in them.

4 The agent can calculate the probability that there is a pit in [2,2]; Chapter 13 shows how.

Section 7.3.

Logic 203

SOUND TRUTH-PRESERVING

COMPLETENESS

In understanding entailment and inference, it might help to think of the set of all consequences of KB as a haystack and of « as a needle. Entailment is like the needle being in the haystack; inference is like finding it. This distinction is embodied in some formal notation: if an inference algorithm ¢ can derive a from KB, we write

KBbF, a,

which is pronounced “« is derived from KB by ¢” or “i derives a from KB.”

An inference algorithm that derives only entailed sentences is called sound or truthpreserving. Soundness is a highly desirable property. An unsound inference procedure essentially makes things up as it goes along—it announces the discovery of nonexistent needles. It is easy to see that model checking, when it is applicable,’ is a sound procedure.

The property of completeness is also desirable: an inference algorithm is complete if it can derive any sentence that is entailed. For real haystacks, which are finite in extent, it seems obvious that a systematic examination can always decide whether the needle is in the haystack. For many knowledge bases, however, the haystack of consequences is infinite, and completeness becomes an important issue.’ Fortunately, there are complete inference procedures for logics that are sufficiently expressive to handle many knowledge bases.

We have described a reasoning process whose conclusions are guaranteed to be true in any world in which the premises are true; in particular, if KB is true in the real world, then any sentence « derived from KB by a sound inference procedure is also true in the real world. So, while an inference process operates on “syntax”—internal physical configurations such as bits in registers or patterns of electrical blips in brains—the process corresponds to the real-world relationship whereby some aspect of the real world is the case’ by virtue of other aspects of the real world being the case. This correspondence between world and representation is illustrated in Figure 7.6.

[]

Figure 7.6 Sentences are physical configurations of the agent, and reasoning is a process of constructing new physical configurations from old ones. Logical reasoning should ensure that the new configurations represent aspects of the world that actually follow from the aspects that the old configurations represent.

5 Model checking works if the space of models is finite—for example, in wumpus worlds of fixed size. For arithmetic, on the other hand, the space of models is infinite: even if we restrict ourselves to the integers, there are infinitely many pairs of values for = and y in the sentence = + y = 4.

6 Compare with the case of infinite search spaces in Chapter 3, where depth-first search is not complete.

7 As Wittgenstein (1922) put it in his famous Tractatus: “The world is everything that is the case.”

which the agent exists. In particular, how do we know that KB is true in the real world? (After all, KB is just “syntax” inside the agent’s head.) This is a philosophical question about which many, many books have been written. (See Chapter 26.) A simple answer is that the agent’s sensors create the connection. For example, our wumpus-world agent has a smell sensor. The agent program creates a suitable sentence whenever there is a smell. Then, whenever that sentence is in the knowledge base, it is true in the real world. Thus, the meaning and truth of percept sentences are defined by the processes of sensing and sentence construction that produce them. What about the rest of the agent’s knowledge, such as its belief that wumpuses cause smells in adjacent squares? This is not a direct representation of a single percept, but a general rule—derived, perhaps, from perceptual experience but not identical to a statement of that experience. General rules like this are produced by a sentence construction process called learning, which is the subject of Part VI. Learning is fallible. It could be the case that wumpuses cause smells except on February 29 in leap years, which is when they take their baths. Thus, KB may not be true in the real world, but with good learning procedures there is reason for optimism.

PROPOSITIONAL LOGIC

ATOMIC SENTENCES PROPOSITION SYMBOL

COMPLEX SENTENCES

LOGICAL CONNECTIVES NEGATION LITERAL

We now present a very simple logic called propositional logic.® We cover the syntax of

propositional logic and its semantics—the way in which the truth of sentences is determined. Then we look at entailment—the relation between a sentence and another sentence that follows from it—and see how this leads to a simple algorithm for logical inference. Everything takes place, of course, in the wumpus world.

Syntax

The syntax of propositional logic defines the allowable sentences. The atomic sentences— the indivisible syntactic elements—consist of a single proposition symbol. Each such symbol stands for a proposition that can be true or false. We will use uppercase names for symbols: P, @}, R, and so on. The names are arbitrary but are often chosen to have some mnemonic value to the reader. For example, we might use W/ 3 to stand for the proposition that the wumpus is in [1,3]. (Remember that symbols such as Wi 3 are atomic, i.e., W, 1, and 3 are not meaningful parts of the symbol.) There are two proposition symbols with fixed meanings: True is the always-true proposition and False is the always-false proposition.

Complex sentences are constructed from simpler sentences using logical connectives. There are five connectives in common Use:

— (not). A sentence such as ~W 3 is called the negation of Wy 3. A literal is either an atomic sentence (a positive literal) or a negated atomic sentence (a negative literal).

8 Propositional logic is also called Boolean logic, after the logician George Boole (1815-1864).

Propositional Logic: A Very Simple Logic 205

CONJUNCTION

DISJUNCTION

IMPLICATION PREMISE CONCLUSION

BICONDITIONAL

A (and). A sentence whose main connective is A, such as Wy 3 A Ps 1, is called a conjunction; its parts are the conjuncts. (The A looks like an “A” for “And.”)

V (or). A sentence using V, such as (Wis A P31) VWs 9, is a disjunction of the disjuncts (Wis A P31) and Ws 5. (Historically, the V comes from the Latin “vel,” which means “or.” For most people, it is easier to remember as an upside-down A.)

= (implies). A sentence such as (Wi 3A P31) = —Wha 2 is called an implication (or conditional). Its premise or antecedent is (WW; 3A P; 1), and its conclusion or consequent is 7Wa 2. Implications are also known as rules or if-then statements. The implication symbol is sometimes written in other books as D or —.

<> (if and only if). The sentence Wy 3 < —Whs 2 is a biconditional.

Figure 7.7 gives a formal grammar of propositional logic; see page 984 if you are not familiar with the BNF notation.

Sentence — AtomicSentence | ComplexSentence

True | False | Symbol P| Q| R| ...

l

AtomicSentence Symbol

l

ComplerSentence — — Sentence | ( Sentence A Sentence) | (Sentence V Sentence) | (Sentence = Sentence)

( Sentence << Sentence)

Figure 7.7 A BNF (Backus—Naur Form) grammar of sentences in propositional logic.

Notice that the grammar is very strict about parentheses: every sentence constructed with binary connectives must be enclosed in parentheses. This ensures that the syntax is completely unambiguous. It also means that we have to write ((A A B) = C) instead of ANB = (, for example. To improve readability, we will often omit parentheses, relying instead on an order of precedence for the connectives. This is similar to the precedence used in arithmetic—for example, ab + ¢ is read as ((ab) + ¢) rather than a(b + ¢) because multiplication has higher precedence than addition. The order of precedence in propositional logic is (from highest to lowest): =, A, V, =, and <. Hence, the sentence

-PVQAR = S is equivalent to the sentence (=P)V(QAR)) = S.

Precedence does not resolve ambiguity in sentences such as A A B A C, which could be read as (ANB) AC)oras (AA (BAC)). Because these two readings mean the same thing according to the semantics given in the next section, sentences such as AA BAC are allowed. Wealso allow AV BV Cand A & B < CC. Sentencessuchas A = B = ( are not

206

Chapter 7. Logical Agents

TRUTH TABLE

allowed because the two readings have different meanings; we insist on parentheses in this case. Finally, we will sometimes use square brackets instead of parentheses when it makes the sentence clearer.

Semantics

Having specified the syntax of propositional logic, we now specify its semantics. The semantics defines the rules for determining the truth of a sentence with respect to a particular model. In propositional logic, a model simply fixes the truth value—true or false—for every proposition symbol. For example, if the sentences in the knowledge base make use of the proposition symbols Py 2, P» 2, and Pj 1, then one possible model is

my = {P12 = false, Pro = false, P31 = true} . With three proposition symbols, there are 22 = 8 possible models—exactly those depicted in Figure 7.5. Notice, however, that because we have pinned down the syntax, the models become purely mathematical objects with no necessary connection to wumpus worlds. Pj» is just a symbol; it might mean “there is a pit in [1,2]” or “I'm in Paris today and tomorrow.”

The semantics for propositional logic must specify how to compute the truth value of any sentence, given a model. This is done recursively. All sentences are constructed from atomic sentences and the five connectives; therefore, we need to specify how to compute the truth of atomic sentences and how to compute the truth of sentences formed with each of the five connectives. Atomic sentences are easy:

o True is true in every model and False is false in every model. e The truth value of every other proposition symbol must be specified directly in the model. For example, in the model m given earlier, P; 2 is false.

For complex sentences, we have rules such as

e For any sentence s and any model m, the sentence —s is true in m if and only if s is false in m.

Such rules reduce the truth of a complex sentence to the truth of simpler sentences. The rules for each connective can be summarized in a truth table that specifies the truth value of a complex sentence for each possible assignment of truth values to its components. Truth tables for the five logical connectives are given in Figure 7.8. Using these tables, the truth value of any sentence s can be computed with respect to any model m by a simple process of recursive evaluation. For example, the sentence =P» A (P22 V P31), evaluated in my, gives true A (false V true) = true A true = true. Exercise 7.3 asks you to write the algorithm PL-TRUE?(s, m), which computes the truth value of a propositional logic sentence s in a model m.

Previously we said that a knowledge base consists of a set of sentences. We can now see that a logical knowledge base is a conjunction of those sentences. That is, if we start with an empty KB and do TELL(KB, S1) ... TELL(KB, Sy) then we have KB = S1 A... A Sp. This means that we can treat knowledge bases and sentences interchangeably.

The truth tables for “and,” “or,” and “not” are in close accord with our intuitions about the English words. The main point of possible confusion is that PV Q is true when P is true

Section 7.4.

Propositional Logic: A Very Simple Logic 207

[]

Figure 7.8 Truth tables for the five logical connectives. To use the table to compute, for example, the value of P V QQ when P is true and Q is false, first look on the left for the row where P is true and @ is false (the third row). Then look in that row under the PV @ column to see the result: true. Another way to look at this is to think of each row as a model, and the entries under each column for that row as saying whether the corresponding sentence is true in that model.

or (} is true or both. There is a different connective called “exclusive or” (“xor” for short) that yields false when both disjuncts are true.” There is no consensus on the symbol for exclusive or; two choices are V and 6.

The truth table for => may seem puzzling at first, because it might not quite fit one’s intuitive understanding of “P implies (” or “if P then QQ.” For one thing, propositional logic does not require any relation of causation or relevance between P and (Q. The sentence “5 is odd implies Tokyo is the capital of Japan” is a true sentence of propositional logic (under the normal interpretation), even though it is a decidedly odd sentence of English. Another point of confusion is that any implication is true whenever its antecedent is false. For example, “5 is even implies Sam is smart” is true, regardless of whether Sam is smart. This seems bizarre, but it makes sense if you think of “P = @Q” as saying, “If P is true, then I am claiming that Q is true. Otherwise I am making no claim.” The only way for this sentence to be false is if P is true but Q is false.

The truth table for a biconditional, P << (Q, shows that it is true whenever both P = @ and Q = P are true. In English, this is often written as “P if and only if Q” or “P iff Q.” The rules of the wumpus world are best written using <>. For example, a square is breezy if a neighboring square has a pit, and a square is breezy only if a neighboring square has a pit. So we need biconditionals such as

By © (Pia2V Py), where Bj | means that there is a breeze in [1,1]. Notice that the one-way implication B11 = (Pi2V Py)

is true in the wumpus world, but incomplete. It does not rule out models in which By ; is false and Pj » is true, which would violate the rules of the wumpus world. Another way of putting it is that the implication requires the presence of pits if there is a breeze, whereas the biconditional also requires the absence of pits if there is no breeze.

9 Latin has a separate word, aut, for exclusive or.

208

Chapter 7. Logical Agents

A simple knowledge base

Now that we have defined the semantics for propositional logic, we can construct a knowledge base for the wumpus world. For simplicity, we will deal only with pits; the wumpus itself is left as an exercise. We will provide enough knowledge to carry out the inference that was done informally in Section 7.3.

First, we need to choose our vocabulary of proposition symbols. For each i, j:

e Let P; ; be true if there is a pit in [, j]. e Let B; ; be true if there is a breeze in [i, j]. The knowledge base includes the following sentences, each one labeled for convenience: e There is no pitin [1,1]: Ry: =P.

e A square is breezy if and only if there is a pit in a neighboring square. This has to be stated for each square; for now, we include just the relevant squares:

Ry: B11 ~ (P12 V Pq) . Rs: Bs 1 = (P11 V Pa V Ps) .

e The preceding sentences are true in all wumpus worlds. Now we include the breeze percepts for the first two squares visited in the specific world the agent is in, leading up to the situation in Figure 7.3(b).

Ry : -B1 . Rs : Ba 1 .

The knowledge base, then, consists of sentences Ry through Rj. It can also be considered as a single sentence—the conjunction Ry A Rs A Rs A Ry A Rs—because it asserts that all the individual sentences are true.

Inference

Recall that the aim of logical inference is to decide whether KB |= « for some sentence o. For example, is P 7 entailed? Our first algorithm for inference will be a direct implementation of the definition of entailment: enumerate the models, and check that « is true in every model in which KB is true. For propositional logic, models are assignments of true or false to every proposition symbol. Returning to our wumpus-world example, the relevant proposition symbols are By 1, Ba 1, P11, P12, P>1, P22, and P31. With seven symbols, there are 27 = 128 possible models; in three of these, KB is true (Figure 7.9). In those three models, —P 7 is true, hence there is no pit in [1,2]. On the other hand, P; 5 is true in two of the three models and false in one, so we cannot yet tell whether there is a pit in [2,2].

Figure 7.9 reproduces in a more precise form the reasoning illustrated in Figure 7.5. A general algorithm for deciding entailment in propositional logic is shown in Figure 7.10. Like the BACKTRACKING-SEARCH algorithm on page 76, TT-ENTAILS? performs a recursive enumeration of a finite space of assignments to variables. The algorithm is sound, because it

Section 7.4. Propositional Logic: A Very Simple Logic 209 Bii| Bog | Pia) Pia| Pag | Pap Pia Rr, R, Rs R, Rs KB false | false | false | false | false | false | false || true | true | true | true | false || false false | false | false | false | false | false | true || true | true | false | true | false || false false | true | false | false | false | false | false || true | true | false | true | true || false false | true | false | false | false | false | true || true | true | true | true | true || true false | true | false | false | false | true | false || true | true | true | true | true || true false | true | false | false | false | true | true || true | true | true | true | true || true false | true | false | false | true | false | false || true | false | false | true | true || false true | true | true | true | true | true | true || false | true | true | false | true || false

[]

Figure 7.9 A truth table constructed for the knowledge base given in the text. KB is true if R; through Ry are true, which occurs in just 3 of the 128 rows. In all 3 rows, P; 5 is false, so there is no pit in [1,2]. On the other hand, there might (or might not) be a pit in [2,2].

function TT-ENTAILS?(K B, a) returns true or false inputs: KB, the knowledge base, a sentence in propositional logic «, the query, a sentence in propositional logic

symbols « a list of the proposition symbols in KB and « return TT-CHECK-ALL(KB, «, symbols, [])

function TT-CHECK-ALL(KB, «x, symbols, model) returns true or false if EMPTY ?(symbols) then if PL-TRUE?(KB, model) then return PL-TRUE?(«x, model) else return true else do P — FIRST(symbols); rest «— REST(symbols) return TT-CHECK-ALL(KB, a, rest, EXTEND(P, true, model) and TT-CHECK-ALL(KB, a, rest, EXTEND(P, false, model)

Figure 7.10 A truth-table enumeration algorithm for deciding propositional entailment. TT stands for truth table. PL-TRUE? returns true if a sentence holds within a model. The variable model represents a partial model—an assignment to only some of the variables. The function call EXTEND(P, true, model) returns a new partial model in which P has the value true.

210

Chapter 7. Logical Agents

LOGICAL EQUIVALENCE

VALIDITY

TAUTOLOGY

(anB) = (BA a) commutativity of A (Vp) = (Va) commutativity of V (a AB)Av) = (aA (BAy)) associativity of A ((avpB)Vy) = (eV (BV) associativity of V —(—a) = «a double-negation elimination (a = 3) = (=8 = —a) contraposition (a = B) = (-aV 3) implication elimination (a & 08) = (a = B)A(B = «)) biconditional elimination —(a AB) = (ma V-3) DeMorgan -(aV 3) = (ma A-F) DeMorgan (an (BVA) = (aAB)V(aAy)) distributivity of A over V (aV (BAY) = ((aVP)A(aVy)) distributivity of V over A Figure 7.11 Standard logical equivalences. The symbols «, 3, and ~ stand for arbitrary sentences of propositional logic.

chapter, we will see algorithms that are much more efficient in practice. Unfortunately, every known inference algorithm for propositional logic has a worst-case complexity that is exponential in the size of the input. We do not expect to do better than this because propositional entailment is co-NP-complete. (See Appendix A.)

Equivalence, validity, and satisfiability

Before we plunge into the details of logical inference algorithms, we will need some additional concepts related to entailment. Like entailment, these concepts apply to all forms of logic, but they are best illustrated for a particular logic, such as propositional logic.

The first concept is logical equivalence: two sentences « and 3 are logically equivalent if they are true in the same set of models. We write this as « <> J. For example, we can easily show (using truth tables) that P A QQ and Q A P are logically equivalent; other equivalences are shown in Figure 7.11. They play much the same role in logic as arithmetic identities do in ordinary mathematics. An alternative definition of equivalence is as follows: for any two sentences « and 3,

a= ifandonlyif aFEfandf Fa.

(Recall that = means entailment.)

The second concept we will need is validity. A sentence is valid if it is true in all models. For example, the sentence P VV —P is valid. Valid sentences are also known as tautologies—they are necessarily true and hence vacuous. Because the sentence True is true in all models, every valid sentence is logically equivalent to True.

What good are valid sentences? From our definition of entailment, we can derive the deduction theorem, which was known to the ancient Greeks:

For any sentences « and 3, a |= 0 if and only if the sentence (ow = 3) is valid.

(Exercise 7.4 asks for a proof.) We can think of the inference algorithm in Figure 7.10 as

Reasoning Patterns in Propositional Logic 211

SATISFIABILITY

REDUCTIO AD ABSURDUM

REFUTATION

checking the validity of (KB = «). Conversely, every valid implication sentence describes a legitimate inference.

The final concept we will need is satisfiability. A sentence is satisfiable if it is true in some model. For example, the knowledge base given earlier, (R; A lo A Rg A Ry A Rs), is satisfiable because there are three models in which it is true, as shown in Figure 7.9. If a sentence « is true in a model m, then we say that m satisfies «, or that m is a model of a. Satisfiability can be checked by enumerating the possible models until one is found that satisfies the sentence. Determining the satisfiability of sentences in propositional logic was the first problem proved to be NP-complete.

Many problems in computer science are really satisfiability problems. For example, all the constraint satisfaction problems in Chapter 5 are essentially asking whether the constraints are satisfiable by some assignment. With appropriate transformations, search problems can also be solved by checking satisfiability. Validity and satisfiability are of course connected: a is valid iff —« is unsatisfiable; contrapositively, « is satisfiable iff —« is not valid. We also have the following useful result:

a |= (if and only if the sentence (a A —3) is unsatisfiable.

Proving 3 from a by checking the unsatisfiability of (aw A =) corresponds exactly to the standard mathematical proof technique of reductio ad absurdum (literally, “reduction to an absurd thing”). It is also called proof by refutation or proof by contradiction. One assumes a sentence J to be false and shows that this leads to a contradiction with known axioms a. This contradiction is exactly what is meant by saying that the sentence (a A —f3) is unsatisfiable.

INFERENCE RULES MODUS PONENS

AND-ELIMINATION

This section covers standard patterns of inference that can be applied to derive chains of conclusions that lead to the desired goal. These patterns of inference are called inference rules. The best-known rule is called Modus Ponens and is written as follows: a = 3, Qa 3 . The notation means that, whenever any sentences of the form a = 3 and «a are given, then the sentence 3 can be inferred. For example, if ( WumpusAhead A WumpusAlive) => Shoot and (WumpusAhead A WumpusAlive) are given, then Shoot can be inferred.

Another useful inference rule is And-Elimination, which says that, from a conjunction,

any of the conjuncts can be inferred: ap o For example, from ( WumpusAhead A WumpusAlive), WumpusAlive can be inferred.

By considering the possible truth values of « and 3, one can show easily that Modus Ponens and And-Elimination are sound once and for all. These rules can then be used in any particular instances where they apply, generating sound inferences without the need for enumerating models.

212

Chapter 7. Logical Agents

MONOTONICITY

All of the logical equivalences in Figure 7.11 can be used as inference rules. For exam

ple, the equivalence for biconditional elimination yields the two inference rules as (a= BAB = ao and —\_ (a = BAB = a) a <3

Not all inference rules work in both directions like this. For example, we cannot run Modus Ponens in the opposite direction to obtain « = 3 and « from S.

Let us see how these inference rules and equivalences can be used in the wumpus world. We start with the knowledge base containing [21 through Rs, and show how to prove —F 5, that is, there is no pit in [1,2]. First, we apply biconditional elimination to FR to obtain

Re: (Big = (Pi2V FP) A ((P2V Py) = Big). Then we apply And-Elimination to Rg to obtain Ry: (Pi2V Po) = Bra). Logical equivalence for contrapositives gives Rg: (=Bi1 = —(Pi2V Pa). Now we can apply Modus Ponens with Rg and the percept R4 (i.e., =Bj 1), to obtain Ry: —(PiaV Py). Finally, we apply De Morgan's rule, giving the conclusion Rig: —PiagA-Pay.

That is, neither [1,2] nor [2,1] contains a pit.

The preceding derivation—a sequence of applications of inference rules—is called a proof. Finding proofs is exactly like finding solutions to search problems. In fact, if the successor function is defined to generate all possible applications of inference rules, then all of the search algorithms in Chapters 3 and 4 can be applied to find proofs. Thus, searching for proofs is an alternative to enumerating models. The search can go forward from the initial knowledge base, applying inference rules to derive the goal sentence, or it can go backward from the goal sentence, trying to find a chain of inference rules leading from the initial knowledge base. Later in this section, we will see two families of algorithms that use these techniques.

The fact that inference in propositional logic is NP-complete suggests that, in the worst case, searching for proofs is going to be no more efficient than enumerating models. In many practical cases, however, finding a proof can be highly efficient simply because it can ignore irrelevant propositions, no matter how many of them there are. For example, the proof given earlier leading to —P; 2 A =F» 1 does not mention the propositions Bs 1, P11, Py 2, or Py 1. They can be ignored because the goal proposition, Pj 2, appears only in sentence fy; the other propositions in R4 appear only in R4 and Rs; so Rj, Rs, and Rs have no bearing on the proof. The same would hold even if we added a million more sentences to the knowledge base; the simple truth-table algorithm, on the other hand, would be overwhelmed by the exponential explosion of models.

This property of logical systems actually follows from a much more fundamental property called monotonicity. Monotonicity says that the set of entailed sentences can only in

Section 7.5.

Reasoning Patterns in Propositional Logic 213

crease as information is added to the knowledge base.!® For any sentences « and 3, if KBE=a then KBAJE a.

For example, suppose the knowledge base contains the additional assertion [3 stating that there are exactly eight pits in the world. This knowledge might help the agent draw additional conclusions, but it cannot invalidate any conclusion « already inferred—such as the conclusion that there is no pitin [1,2]. Monotonicity means that inference rules can be applied whenever suitable premises are found in the knowledge base—the conclusion of the rule must follow regardless of what else is in the knowledge base.

Resolution

We have argued that the inference rules covered so far are sound, but we have not discussed the question of completeness for the inference algorithms that use them. Search algorithms such as iterative deepening search (page 78) are complete in the sense that they will find any reachable goal, but if the available inference rules are inadequate, then the goal is not reachable—no proof exists that uses only those inference rules. For example, if we removed the biconditional elimination rule, the proof in the preceding section would not go through. The current section introduces a single inference rule, resolution, that yields a complete inference algorithm when coupled with any complete search algorithm.

We begin by using a simple version of the resolution rule in the wumpus world. Let us consider the steps leading up to Figure 7.4(a): the agent returns from [2,1] to [1,1] and then goes to [1,2], where it perceives a stench, but no breeze. We add the following facts to the knowledge base:

[]

By the same process that led to Rg earlier, we can now derive the absence of pits in [2,2] and [1,3] (remember that [1,1] is already known to be pitless):

Ris : P29 .

Ria : -Pi3 .

We can also apply biconditional elimination to [23, followed by modus ponens with Rs, to obtain the fact that there is a pitin [1,1], [2,2], or [3,1]:

Ris: PiiVPVP3;.

Now comes the first application of the resolution rule: the literal =P» » in R13 resolves with the literal P, » in R;5 to give

Rig: Pa V Ps; . In English; if there’s a pit in one of [1,1], [2,2], and [3,1], and it’s not in [2,2], then it’s in [1,1] or [3,1]. Similarly, the literal =P; ; in R; resolves with the literal P; 1 in Rig to give Ri : Ps, .

10 Nonmonotonic logics, which violate the monotonicity property, capture a common property of human reasoning: changing one’s mind. They are discussed in Section 10.7.

214 Chapter 7. Logical Agents

In English; if there’s a pit in [1,1] or [3,1], and it’s not in [1,1], then it’s in [3,1]. These last uwrResowtion two inference steps are examples of the unit resolution inference rule,

£1 VV Ag, m GV NEG ANE VV Ey ’

COMPLEMENTARY where each / is a literal and ¢; and m are complementary literals (i.e., one is the negation CLAUSE of the other). Thus, the unit resolution rule takes a clause—a disjunction of literals—and a

literal and produces a new clause. Note that a single literal can be viewed as a disjunction of UNIT CLAUSE one literal, also known as a unit clause. RESOLUTION The unit resolution rule can be generalized to the full resolution rule,

REFUTATION COMPLETENESS

lV Vy, mi V---V my OV NEL AVL Ve NV Vmy Ve Vm Vm Veo Vimy

3

where /; and m; are complementary literals. If we were dealing only with clauses of length two we could write this as Hv 4s, =f V lg {1 V {3 ’ That is, resolution takes two clauses and produces a new clause containing all the literals of the two original clauses except the two complementary literals. For example, we have

PV Py, PVP, P31V Ps

There is one more technical aspect of the resolution rule: the resulting clause should contain only one copy of each literal.!! The removal of multiple copies of literals is called factoring. For example, if we resolve (A V B) with (A V —B), we obtain (A VV A), which is reduced to just A.

The soundness of the resolution rule can be seen easily by considering the literal ¢;. If ¢; is true, then my; is false, and hence m1 V +--+ V mj\_1 Vmjq1 V +--+ V my, must be true, because mj V --V my, is given. If 4; is false, then £1 VV --V 4;\_1 V €;41 V ++ V {i must be true because £1 V --V ¥ is given. Now /; is either true or false, so one or other of these conclusions holds—exactly as the resolution rule states.

What is more surprising about the resolution rule is that it forms the basis for a family of complete inference procedures. Any complete search algorithm, applying only the resolution rule, can derive any conclusion entailed by any knowledge base in propositional logic. There is a caveat: resolution is complete in a specialized sense. Given that A is true, we cannot use resolution to automatically generate the consequence AV B. However, we can use resolution to answer the question of whether A VV B is true. This is called refutation completeness, meaning that resolution can always be used to either confirm or refute a sentence, but it cannot be used to enumerate true sentences. The next two subsections explain how resolution accomplishes this.

11 If a clause is viewed as a set of literals, then this restriction is automatically respected. Using set notation for clauses makes the resolution rule much cleaner, at the cost of introducing additional notation.

Reasoning Patterns in Propositional Logic 215

CONJUNCTIVE NORMAL FORM

K-CNF

Conjunctive normal form

The resolution rule applies only to disjunctions of literals, so it would seem to be relevant only to knowledge bases and queries consisting of such disjunctions. How, then, can it lead to a complete inference procedure for all of propositional logic? The answer is that every sentence of propositional logic is logically equivalent to a conjunction of disjunctions of literals. A sentence expressed as a conjunction of disjunctions of literals is said to be in conjunctive normal form or CNF. We will also find it useful later to consider the restricted family of k-CNF sentences. A sentence in k-CNF has exactly k literals per clause:

(av. VORA AE VV lg).

It turns out that every sentence can be transformed into a 3-CNF sentence that has an equivalent set of models.

Rather than prove these assertions (see Exercise 7.10), we describe a simple conversion procedure. We illustrate the procedure by converting Ry, the sentence By 1 < (P12 V Py), into CNF. The steps are as follows:

1. Eliminate <, replacing a < 3 with (a = SB) A (8 = a). (B11 = (P12 V Pq) A (P12 V Py) = B11) . 2. Eliminate =, replacing a = § with ~a V 3: (—B1,1 V Po V P51) A (=(P12 V Pq) V B11) . 3. CNF requires — to appear only in literals, so we “move — inwards” by repeated application of the following equivalences from Figure 7.11: —(—a) = a (double-negation elimination) =(a AB) = (-aV 4G) (De Morgan) =(aV 3) = (-aA-3) (DeMorgan) In the example, we require just one application of the last rule: (—B11 V Pa V Py) A ((=P12 A -P1) V Bi) . 4. Now we have a sentence containing nested A and V operators applied to literals. We apply the distributivity law from Figure 7.11, distributing V over A wherever possible. (=Bi11 V Po Vv Pq) A (=P12 V Bi1) A (=P Vv Bi1) . The original sentence is now in CNF, as a conjunction of three clauses. It is much harder to read, but it can be used as input to a resolution procedure.

A resolution algorithm

Inference procedures based on resolution work by using the principle of proof by contradiction discussed at the end of Section 7.4. That is, to show that KB |= a, we show that (KB A —a) is unsatisfiable. We do this by proving a contradiction.

A resolution algorithm is shown in Figure 7.12. First, (KB A —a) is converted into CNFE. Then, the resolution rule is applied to the resulting clauses. Each pair that contains complementary literals is resolved to produce a new clause, which is added to the set if it is not already present. The process continues until one of two things happens:

216

Chapter 7. Logical Agents

function PL-RESOLUTION(KB, «) returns true or false inputs: KB, the knowledge base, a sentence in propositional logic «a, the query, a sentence in propositional logic

clauses — the set of clauses in the CNF representation of KB A —a new —{ } loop do for each C;, C; in clauses do resolvents — PL-RESOLVE(C}, C;) if resolvents contains the empty clause then return true new «— new U resolvents if new C clauses then return false clauses — clauses U new

Figure 7.12 A simple resolution algorithm for propositional logic. The function PL-RESOLVE returns the set of all possible clauses obtained by resolving its two inputs.

[]

Figure 7.13 Partial application of PL-RESOLUTION to a simple inference in the wumpus world. =F; 5 is shown to follow from the first four clauses in the top row.

e there are no new clauses that can be added, in which case KB does not entail «; or, e two clauses resolve to yield the empty clause, in which case KB entails a.

The empty clause—a disjunction of no disjuncts—is equivalent to False because a disjunction is true only if at least one of its disjuncts is true. Another way to see that an empty clause represents a contradiction is to observe that it arises only from resolving two complementary unit clauses such as P and —P.

We can apply the resolution procedure to a very simple inference in the wumpus world. When the agent is in [1,1], there is no breeze, so there can be no pits in neighboring squares. The relevant knowledge base is

KB=RyANRy= (B11 = (P12 V P51) A -B11

and we wish to prove a which is, say, =P; ». When we convert (KB A —a) into CNF, we obtain the clauses shown at the top of Figure 7.13. The second row of the figure shows all the clauses obtained by resolving pairs in the first row. Then, when P; » is resolved with =P o, we obtain the empty clause, shown as a small square. Inspection of Figure 7.13 reveals that

Reasoning Patterns in Propositional Logic 217

RESOLUTION CLOSURE

GROUND RESOLUTION THEOREM

HORN CLAUSES

many resolution steps are pointless. For example, the clause B11 V Bj 1V Py 2 is equivalent to True VV Pp 2 which is equivalent to True. Deducing that True is true is not very helpful. Therefore, any clause in which two complementary literals appear can be discarded.

Completeness of resolution

To conclude our discussion of resolution, we now show why PL-RESOLUTION is complete. To do this, it will be useful to introduce the resolution closure RC(S) of a set of clauses S, which is the set of all clauses derivable by repeated application of the resolution rule to clauses in S or their derivatives. The resolution closure is what PL-RESOLUTION computes as the final value of the variable clauses. It is easy to see that RC (S) must be finite, because there are only finitely many distinct clauses that can be constructed out of the symbols Pi, ..., Px that appear in S. (Notice that this would not be true without the factoring step that removes multiple copies of literals.) Hence, PL-RESOLUTION always terminates.

The completeness theorem for resolution in propositional logic is called the ground resolution theorem:

If a set of clauses is unsatisfiable, then the resolution closure of those clauses contains the empty clause.

We prove this theorem by demonstrating its contrapositive: if the closure RC(S) does not contain the empty clause, then S is satisfiable. In fact, we can construct a model for S with suitable truth values for Py, ..., Px. The construction procedure is as follows:

Forifrom1tok,

— If there is a clause in RC (SS) containing the literal —P; such that all its other literals are false under the assignment chosen for Py, ..., P;\_1, then assign false to P,.

— Otherwise, assign true to Fj.

It remains to show that this assignment to Py, ..., Pg is a model of S, provided that RC(S) is closed under resolution and does not contain the empty clause. The proof of this is left as an exercise.

Forward and backward chaining

The completeness of resolution makes it a very important inference method. In many practical situations, however, the full power of resolution is not needed. Real-world knowledge bases often contain only clauses of a restricted kind called Horn clauses. A Horn clause is a disjunction of literals of which at most one is positive. For example, the clause (—Ly 1 V —BreezeV Bi 1), where L 1 means that the agent’s location is [1,1], is a Horn clause, whereas (=B1,1 V Pia V P51) is not.

The restriction to just one positive literal may seem somewhat arbitrary and uninteresting, but it is actually very important for three reasons:

1. Every Horn clause can be written as an implication whose premise is a conjunction of positive literals and whose conclusion is a single positive literal. (See Exercise 7.12.) For example, the Horn clause (—L1 1 V—BreezeV Bj 1) can be written as the implication

218

Chapter 7. Logical Agents

DEFINITE CLAUSES HEAD BODY

FACT

INTEGRITY CONSTRAINTS

FORWARD CHAINING

BACKWARD CHAINING

AND-OR GRAPH

FIXED POINT

(L1,1 A Breeze) = Bi 1. In the latter form, the sentence is much easier to read: it says that if the agent is in [1,1] and there is a breeze, then [1,1] is breezy. People find it easy to read and write sentences in this form for many domains of knowledge.

Horn clauses like this one with exactly one positive literal are called definite clauses. The positive literal is called the head and the negative literals form the body of the clause. A definite clause with no negative literals simply asserts a given proposition— sometimes called a fact. Definite clauses form the basis for logic programming, which is discussed in Chapter 9. A Horn clause with no positive literals can be written as an implication whose conclusion is the literal False. For example, the clause (-W11 V = Wj 2)—the wumpus cannot be in both [1,1] and [1,2]—is equivalent to Wi1 A Wi 2 = False. Such sentences are called integrity constraints in the database world, where they are used to signal errors in the data. In the algorithms that follow, we assume for simplicity that the knowledge base contains only definite clauses and no integrity constraints. We say these knowledge bases are in Horn form.

2. Inference with Horn clauses can be done through the forward chaining and backward chaining algorithms, which we explain next. Both of these algorithms are very natural, in that the inference steps are obvious and easy to follow for humans.

3. Deciding entailment with Horn clauses can be done in time that is linear in the size of the knowledge base.

This last fact is a pleasant surprise. It means that logical inference is very cheap for many propositional knowledge bases that are encountered in practice.

The forward-chaining algorithm PL-FC-ENTAILS (KB, ¢) determines whether a single proposition symbol ¢—the query—is entailed by a knowledge base of Horn clauses. It begins from known facts (positive literals) in the knowledge base. If all the premises of an implication are known, then its conclusion is added to the set of known facts. For example, if L1,1 and Breeze are known and (L1a A Breeze) = Bi 1 is in the knowledge base, then By; can be added. This process continues until the query ¢ is added or until no further inferences can be made. The detailed algorithm is shown in Figure 7.14; the main point to remember is that it runs in linear time.

The best way to understand the algorithm is through an example and a picture. Figure 7.15(a) shows a simple knowledge base of Horn clauses with A and B as known facts. Figure 7.15(b) shows the same knowledge base drawn as an AND-OR graph. In AND-OR graphs, multiple links joined by an arc indicate a conjunction—every link must be proved— while multiple links without an arc indicate a disjunction—any link can be proved. It is easy to see how forward chaining works in the graph. The known leaves (here, A and B) are set, and inference propagates up the graph as far as possible. Wherever a conjunction appears, the propagation waits until all the conjuncts are known before proceeding. The reader is encouraged to work through the example in detail.

It is easy to see that forward chaining is sound: every inference is essentially an application of Modus Ponens. Forward chaining is also complete: every entailed atomic sentence will be derived. The easiest way to see this is to consider the final state of the inferred table (after the algorithm reaches a fixed point where no new inferences are possible). The table

Section 7.5.

Reasoning Patterns in Propositional Logic 219

function PL-FC-ENTAILS?(KB, ¢) returns true or false inputs: KB, the knowledge base, a set of propositional Horn clauses q, the query, a proposition symbol local variables: count, a table, indexed by clause, initially the number of premises inferred, a table, indexed by symbol, each entry initially false agenda, a list of symbols, initially the symbols known to be true in KB

while agenda is not empty do p «— PoP(agenda) unless inferred[p] do inferred[p] « true for each Horn clause c¢ in whose premise p appears do decrement count c] if count[c] = 0 then do if HEAD[c] = ¢ then return true PusH(HEAD[c], agenda) return false

Figure 7.14 The forward-chaining algorithm for propositional logic. The agenda keeps track of symbols known to be true but not yet “processed.” The count table keeps track of how many premises of each implication are as yet unknown. Whenever a new symbol p from the agenda is processed, the count is reduced by one for each implication in whose premise p appears. (These can be identified in constant time if KB is indexed appropriately.) If a count reaches zero, all the premises of the implication are known so its conclusion can be added to the agenda. Finally, we need to keep track of which symbols have been processed; an inferred symbol need not be added to the agenda if it has been processed previously. This avoids redundant work; it also prevents infinite loops that could be caused by implications suchas P= QQ and ) = P.

contains true for each symbol inferred during the process, and false for all other symbols. We can view the table as a logical model; moreover, every definite clause in the original KB is true in this model. To see this, assume the opposite, namely that some clause a1 A... Aag = b is false in the model. Then aq A ... A ar must be true in the model and b must be false in the model. But this contradicts our assumption that the algorithm has reached a fixed point! We can conclude, therefore, that the set of atomic sentences inferred at the fixed point defines a model of the original KB. Furthermore, any atomic sentence q that is entailed by the KB must be true in all its models and in this model in particular. Hence, every entailed sentence g must be inferred by the algorithm.

Forward chaining is an example of the general concept of data-driven reasoning—that is, reasoning in which the focus of attention starts with the known data. It can be used within an agent to derive conclusions from incoming percepts, often without a specific query in mind. For example, the wumpus agent might TELL its percepts to the knowledge base using an incremental forward-chaining algorithm in which new facts can be added to the agenda to initiate new inferences. In humans, a certain amount of data-driven reasoning occurs as new

220

Chapter 7. Logical Agents

GOAL-DIRECTED REASONING

Q

P=Q

LAM = P Pp BAL = M

ANP = L M ANB = L

A B (a) (b)

Figure 7.15 (a) A simple knowledge base of Horn clauses. (b) The corresponding AND— OR graph.

information arrives. For example, if [ am indoors and hear rain starting to fall, it might occur to me that the picnic will be canceled. Yet it will probably not occur to me that the seventeenth petal on the largest rose in my neighbor’s garden will get wet; humans keep forward chaining under careful control, lest they be swamped with irrelevant consequences.

The backward-chaining algorithm, as its name suggests, works backwards from the query. If the query gq is known to be true, then no work is needed. Otherwise, the algorithm finds those implications in the knowledge base that conclude ¢. If all the premises of one of those implications can be proved true (by backward chaining), then ¢ is true. When applied to the query (J in Figure 7.15, it works back down the graph until it reaches a set of known facts that forms the basis for a proof. The detailed algorithm is left as an exercise; as with forward chaining, an efficient implementation runs in linear time.

Backward chaining is a form of goal-directed reasoning. It is useful for answering specific questions such as “What shall I do now?” and “Where are my keys?” Often, the cost of backward chaining is much less than linear in the size of the knowledge base, because the process touches only relevant facts. In general, an agent should share the work between forward and backward reasoning, limiting forward reasoning to the generation of facts that are likely to be relevant to queries that will be solved by backward chaining.

In this section, we describe two families of efficient algorithms for propositional inference based on model checking: one approach based on backtracking search, and one on hillclimbing search. These algorithms are part of the “technology” of propositional logic. This section can be skimmed on a first reading of the chapter.

Section 7.6.

Effective propositional inference 221

DAVIS-PUTNAM ALGORITHM

PURE SYMBOL

The algorithms we describe are for checking satisfiability. We have already noted the connection between finding a satisfying model for a logical sentence and finding a solution for a constraint satisfaction problem, so it is perhaps not surprising that the two families of algorithms closely resemble the backtracking algorithms of Section 5.2 and the localsearch algorithms of Section 5.3. They are, however, extremely important in their own right because so many combinatorial problems in computer science can be reduced to checking the satisfiability of a propositional sentence. Any improvement in satisfiability algorithms has huge consequences for our ability to handle complexity in general.

A complete backtracking algorithm

The first algorithm we will consider is often called the Davis—Putnam algorithm, after the seminal paper by Martin Davis and Hilary Putnam (1960). The algorithm is in fact the version described by Davis, Logemann, and Loveland (1962), so we will call it DPLL after the initials of all four authors. DPLL takes as input a sentence in conjunctive normal form—a set of clauses. Like BACKTRACKING-SEARCH and TT-ENTAILS?, it is essentially a recursive, depth-first enumeration of possible models. It embodies three improvements over the simple scheme of TT-ENTAILS?:

e Early termination: The algorithm detects whether the sentence must be true or false, even with a partially completed model. A clause is true if any literal is true, even if the other literals do not yet have truth values; hence, the sentence as a whole could be judged true even before the model is complete. For example, the sentence (A V B) A (Av C) is true if A is true, regardless of the values of B and C'. Similarly, a sentence is false if any clause is false, which occurs when each of its literals is false. Again, this can occur long before the model is complete. Early termination avoids examination of entire subtrees in the search space.

e Pure symbol heuristic: A pure symbol is a symbol that always appears with the same “sign” in all clauses. For example, in the three clauses (4 V =B), (=B V =(C), and (Cv A), the symbol A is pure because only the positive literal appears, B is pure because only the negative literal appears, and C is impure. It is easy to see that if a sentence has a model, then it has a model with the pure symbols assigned so as to make their literals ¢rue, because doing so can never make a clause false. Note that, in determining the purity of a symbol, the algorithm can ignore clauses that are already known to be true in the model constructed so far. For example, if the model contains B = false, then the clause (—B V —C)) is already true, and C becomes pure because it appears only in (C'V A).

e Unit clause heuristic: A unit clause was defined earlier as a clause with just one literal. In the context of DPLL, it also means clauses in which all literals but one are already assigned false by the model. For example, if the model contains B = false, then (BV —C) becomes a unit clause because it is equivalent to (False VV =C'), or just —C. Obviously, for this clause to be true, C' must be set to false. The unit clause heuristic assigns all such symbols before branching on the remainder. One important consequence of the heuristic is that any attempt to prove (by refutation) a literal that is

222 Chapter 7. Logical Agents function DPLL-SATISFIABLE?(s) returns true or false

inputs: s, a sentence in propositional logic

clauses « the set of clauses in the CNF representation of s

symbols — a list of the proposition symbols in s

return DPLL( clauses, symbols, [])

function DPLL( clauses, symbols, model) returns true or false

if every clause in clauses is true in model then return true

if some clause in clauses is false in model then return false

P, value — FIND-PURE-SYMBOL(symbols, clauses, model)

if P is non-null then return DPLL (clauses, symbols — P, EXTEND(P, value, model)

P, value «— FIND-UNIT-CLAUSE( clauses, model)

if P is non-null then return DPLL( clauses, symbols — P, EXTEND(P, value, model)

P — FIRST(symbols); rest — REST(symbols)

return DPLL (clauses, rest, EXTEND(P, true, model)) or

DPLL(clauses, rest, EXTEND(P, false, model))

Figure 7.16 The DPLL algorithm for checking satisfiability of a sentence in propositional

logic. FIND-PURE-SYMBOL and FIND-UNIT-CLAUSE are described in the text; each returns

a symbol (or null) and the truth value to assign to that symbol. Like TT-ENTAILS?, it operates

over partial models. already in the knowledge base will succeed immediately (Exercise 7.16). Notice also that assigning one unit clause can create another unit clause—for example, when C is set to false, (Cv A) becomes a unit clause, causing true to be assigned to A. This

UNIT PROPAGATION “cascade” of forced assignments is called unit propagation. It resembles the process

of forward chaining with Horn clauses, and indeed, if the CNF expression contains only Horn clauses then DPLL essentially replicates forward chaining. (See Exercise 7.17.)

The DPLL algorithm is shown in Figure 7.16. We have given the essential skeleton of the algorithm, which describes the search process itself. We have not described the data structures that must be maintained in order to make each search step efficient, nor the tricks that can be added to improve performance: clause learning, variable selection heuristics, and randomized restarts. When these are included DPLL is one of the fastest satisfiability algorithms yet developed, despite its antiquity. The CHAFF implementation is used to solve hardware verification problems with a million variables.

Local-search algorithms

We have seen several local-search algorithms so far in this book, including HILL-CLIMBING (page 112) and SIMULATED-ANNEALING (page 116). These algorithms can be applied directly to satisfiability problems, provided that we choose the right evaluation function. Because the goal is to find an assignment that satisfies every clause, an evaluation function that counts the number of unsatisfied clauses will do the job. In fact, this is exactly the measure

Section 7.6.

Effective propositional inference 223

function WALKS AT (clauses, p, max \_flips) returns a satisfying model or failure inputs: clauses, a set of clauses in propositional logic p, the probability of choosing to do a “random walk” move, typically around 0.5 maz \_flips, number of flips allowed before giving up

model — a random assignment of true/false to the symbols in clauses for i = 1 to max \_flips do if model satisfies clauses then return model clause «— a randomly selected clause from clauses that is false in model with probability p flip the value in model of a randomly selected symbol from clause else flip whichever symbol in clause maximizes the number of satisfied clauses return failure

Figure 7.17 The WALKSAT algorithm for checking satisfiability by randomly flipping the values of variables. Many versions of the algorithm exist.

used by the MIN-CONFLICTS algorithm for CSPs (page 151). All these algorithms take steps in the space of complete assignments, flipping the truth value of one symbol at a time. The space usually contains many local minima, to escape from which various forms of randomness are required. In recent years, there has been a great deal of experimentation to find a good balance between greediness and randomness.

One of the simplest and most effective algorithms to emerge from all this work is called WALKSAT (Figure 7.17). On every iteration, the algorithm picks an unsatisfied clause and picks a symbol in the clause to flip. It chooses randomly between two ways to pick which symbol to flip: (1) a “min-conflicts” step that minimizes the number of unsatisfied clauses in the new state, and (2) a “random walk” step that picks the symbol randomly.

Does WALKS AT actually work? Clearly, if it returns a model, then the input sentence is indeed satisfiable. What if it returns failure? Unfortunately, in that case we cannot tell whether the sentence is unsatisfiable or we need to give the algorithm more time. We could try setting maz flips to infinity. In that case, itis easy to show that WALKS AT will eventually return a model (if one exists), provided that the probability p > 0. This is because there is always a sequence of flips leading to a satisfying assignment, and eventually the random walk steps will generate that sequence. Alas, if maz\_flips is infinity and the sentence is unsatisfiable, then the algorithm never terminates!

What this suggests is that local-search algorithms such as WALKS AT are most useful when we expect a solution to exist—for example, the problems discussed in Chapters 3 and 5 usually have solutions. On the other hand, local search cannot always detect unsatisfiability, which is required for deciding entailment. For example, an agent cannot reliably use local search to prove that a square is safe in the wumpus world. Instead, it can say, “I thought about it for an hour and couldn’t come up with a possible world in which the square isn’t safe.” If the local-search algorithm is usually really fast at finding a model when one exists, the agent might be justified in assuming that failure to find a model indicates unsatisfiability. This isn’t the same as a proof, of course, and the agent should think twice before staking its life on it.

224

Chapter 7. Logical Agents

UNDERCONSTRAINED

CRITICAL POINT

Hard satisfiability problems

We now look at how DPLL and WALKS AT perform in practice. We are particularly interested in hard problems, because easy problems can be solved by any old algorithm. In Chapter 5, we saw some surprising discoveries about certain kinds of problems. For example, the n-queens problem—-thought to be quite tricky for backtracking search algorithms—turned out to be trivially easy for local-search methods, such as min-conflicts. This is because solutions are very densely distributed in the space of assignments, and any initial assignment is guaranteed to have a solution nearby. Thus, n-queens is easy because it is underconstrained.

When we look at satisfiability problems in conjunctive normal form, an underconstrained problem is one with relatively few clauses constraining the variables. For example, here is a randomly generated!'? 3-CNF sentence with five symbols and five clauses:

(DV -BVC)AN(BV-AV-C)A(-CV-BVE) AN(EV-DV B)A(BVEV-C).

16 of the 32 possible assignments are models of this sentence, so, on average, it would take just two random guesses to find a model.

So where are the hard problems? Presumably, if we increase the number of clauses, keeping the number of symbols fixed, we make the problem more constrained, and solutions become harder to find. Let m be the number of clauses and n be the number of symbols. Figure 7.18(a) shows the probability that a random 3-CNF sentence is satisfiable, as a function of the clause/symbol ratio, m /n, with n fixed at 50. As we expect, for small m /n the probability is close to 1, and at large m /n the probability is close to 0. The probability drops fairly sharply around m/n = 4.3. CNF sentences near this critical point could be described as “nearly satisfiable” or “nearly unsatisfiable.” Is this where the hard problems are?

Figure 7.18(b) shows the runtime for DPLL and WALKS AT around this point, where we have restricted attention to just the safisfiable problems. Three things are clear: First, problems near the critical point are much more difficult than other random problems. Second, even on the hardest problems, DPLL is quite effective—an average of a few thousand steps compared with 25° a 101° for truth-table enumeration. Third, WALKSAT is much faster than DPLL throughout the range.

Of course, these results are only for randomly generated problems. Real problems do not necessarily have the same structure—in terms of proportions of positive and negative literals, densities of connections among clauses, and so on—as random problems. Yet, in practice, WALKS AT and related algorithms are very good at solving real problems too—often as good as the best special-purpose algorithms for those tasks. Problems with thousands of symbols and millions of clauses are routinely handled by solvers such as CHAFF. These observations suggest that some combination of the min-conflicts heuristic and random-walk behavior provides a general-purpose capability for resolving most situations in which combinatorial reasoning is required.

12 Each clause contains three randomly selected distinct symbols, each of which is negated with 50% probability.

Section 7.7.

Agents Based on Propositional Logic 225 1 2000 1800 DPLL —+— 08 1600 | WalkSAT —--x-— tf = 1400 g 0.6 2 1200 5 2 1000 3 04 & 800 & 600 0.2 400 200 04 — : 0 +e 0 1 2 3 4 5 6 1 8 0 1 2 3 4 5 6 7 8 Clause/symbol ratio m/n Clause/symbol ratio m/n (a) (b) Figure 7.18 (a) Graph showing the probability that a random 3-CNF sentence with n= 50 symbols is satisfiable, as a function of the clanse/symbol ratio m/n. (b) Graph of the median runtime of DPLL and WALKSAT on 100 satisfiable random 3-CNF sentences with n = 50, for a narrow range of m /n around the critical point.

In this section, we bring together what we have learned so far in order to construct agents that operate using propositional logic. We will look at two kinds of agents: those which use inference algorithms and a knowledge base, like the generic knowledge-based agent in Figure 7.1, and those which evaluate logical expressions directly in the form of circuits. We will demonstrate both kinds of agents in the wumpus world, and will find that both suffer from serious drawbacks.

Finding pits and wumpuses using logical inference

Let us begin with an agent that reasons logically about the location of pits, wumpuses, and safe squares. It begins with a knowledge base that states the “physics” of the wumpus world. It knows that [1,1] does not contain a pit or a wumpus; that is, =P; ; and —-W 1. For every square [x,y], it knows a sentence stating how a breeze arises:

By Aad (Ppy+1 Vv Py Vv Priiy Vv Pr\_1y) : (7.1) For every square [z, y|, it knows a sentence stating how a stench arises:

Finally, it knows that there is exactly one wumpus. This is expressed in two parts. First, we have to say that there is ar least one wumpus:

Wi Vv Wie Veer V Was Vv Waa .

Then, we have to say that there is at most one wumpus. One way to do this is to say that for any two squares, one of them must be wumpus-free. With n squares, we get n(n — 1)/2

226 Chapter 7. Logical Agents

function PL-WUMPUSAGENT( percept) returns an action inputs: percept, a list, [stench,breeze, glitter] static: KB, a knowledge base, initially containing the “physics” of the wumpus world x, y, orientation, the agent’s position (initially 1,1) and orientation (initially right) visited, an array indicating which squares have been visited, initially false action, the agent’s most recent action, initially null plan, an action sequence, initially empty

update z,y,orientation, visited based on action

if stench then TELL(KB, S, ,) else TELL(KB,—~ 5; ,)

if breeze then TELL(KB, B; ,) else TELL(KB, ~ B, ,)

if glitter then action — grab

else if plan is nonempty then action — POP(plan)

else if for some fringe square [4,5], ASK(KB, (=P; ; A = Wj; ;)) is true or

for some fringe square [4,j], ASK(KB, (P, ; V W; ;))is false then do

plan «— A\*-GRAPH-SEARCH(ROUTE-PROBLEM([z,y], orientation, [i,7], visited)) action «— POP(plan)

else action «— a randomly chosen move

return action

Figure 7.19 A wumpus-world agent that uses propositional logic to identify pits, wumpuses, and safe squares. The subroutine ROUTE-PROBLEM constructs a search problem whose solution is a sequence of actions leading from [x,y] to [4,5] and passing through only previously visited squares.

sentences such as ~Wy 1 V ~Wj 2. For a 4 x 4 world, then, we begin with a total of 155 sentences containing 64 distinct symbols.

The agent program, shown in Figure 7.19, TELLs its knowledge base about each new breeze and stench percept. (It also updates some ordinary program variables to keep track of where it is and where it has been—more on this later.) Then, the program chooses where to look next among the fringe squares—that is, the squares adjacent to those already visited. A fringe square [1,7] is provably safe if the sentence (=P; ; A =W; ;) is entailed by the knowledge base. The next best thing is a possibly safe square, for which the agent cannot prove that there is a pit or a wumpus—that is, for which (P; ; V W; ;) is not entailed.

The entailment computation in ASK can be implemented using any of the methods described earlier in the chapter. TT-ENTAILS? (Figure 7.10) is obviously impractical, since it would have to enumerate 2%4 rows. DPLL (Figure 7.16) performs the required inferences in a few milliseconds, thanks mainly to the unit propagation heuristic. WALKS AT can also be used, with the usual caveats about incompleteness. In wumpus worlds, failures to find a model, given 10,000 flips, invariably correspond to unsatisfiability, so no errors are likely due to incompleteness.

PL-WUMPUS-AGENT works quite well in a small wumpus world. There is, however, something deeply unsatisfying about the agent’s knowledge base. KB contains “physics” sentences of the form given in Equations (7.1) and (7.2) for every single square. The larger

Section 7.7.

Agents Based on Propositional Logic 227

CIRCUIT-BASED AGENT

SEQUENTIAL CIRCUIT

GATES REGISTERS

the environment, the larger the initial knowledge base needs to be. We would much prefer to have just two sentences that say how breezes and stenches arise in all squares. These are beyond the powers of propositional logic to express. In the next chapter, we will see a more expressive logical language in which such sentences are easy to express.

Keeping track of location and orientation

The agent program in Figure 7.19 “cheats” because it keeps track of location outside the knowledge base, instead of using logical reasoning.!> To do it “properly,” we will need propositions for location. One’s first inclination might be to use a symbol such as L1 1 to mean that the agent is in [1,1]. Then the initial knowledge base might include sentences like

Li 1 A FacingRight A Forward = Lo .

Instantly, we see that this won’t work. If the agent starts in [1,1] facing right and moves forward, the knowledge base will entail both L1; (the original location) and Lo (the new location). Yet these propositions cannot both be true! The problem is that the location propositions should refer to two different times. We need Li, to mean that the agent is in [1,1] at time 1, L3, to mean that the agent is in [2,1] at time 2, and so on. The orientation and action propositions also need to depend on time. Therefore, the correct sentence is

Li, A FacingRight' A Forward" = L3

FacingRight' A TurnLeft' = FacingUp? , and so on. It turns out to be quite tricky to build a complete and correct knowledge base for keeping track of everything in the wumpus world; we will defer the full discussion until Chapter 10. The point we want to make here is that the initial knowledge base will contain sentences like the preceding two examples for every time ¢, as well as for every location. That is, for every time ¢ and location [z, y], the knowledge base contains a sentence of the form

Lt, A FacingRight' A Forward® = Lh, . (7.3)

Even if we put an upper limit on the number of time steps allowed—100, perhaps—we end up with tens of thousands of sentences. The same problem arises if we add the sentences “as needed” for each new time step. This proliferation of clauses makes the knowledge base unreadable for a human, but fast propositional solvers can still handle the 4 x 4 Wumpus world with ease (they reach their limit at around 100 x 100). The circuit-based agents in the next subsection offer a partial solution to this clause proliferation problem, but the full solution will have to wait until we have developed first-order logic in Chapter 8.

Circuit-based agents

A circuit-based agent is a particular kind of reflex agent with state, as defined in Chapter 2. The percepts are inputs to a sequential circuit—a network of gates, each of which implements a logical connective, and registers, each of which stores the truth value of a single proposition. The outputs of the circuit are registers corresponding to actions—for example,

13 The observant reader will have noticed that this allowed us to finesse the connection between the raw percepts such as Breeze and the location-specific propositions such as Bj 1.

228

Chapter 7. Logical Agents

DATAFLOW

DELAY LINE

Breeze [] Forward Stench [J Glitter [} Bump [] Scream oJ

Figure 7.20 Part of a circuit-based agent for the wumpus world, showing inputs, outputs, the circuit for grabbing the gold, and the circuit for determining whether the wumpus is alive. Registers are shown as rectangles and one-step delays are shown as small triangles.

the Grab output is set to true if the agent wants to grab something. If the Glitter input is connected directly to the Grab output, the agent will grab the goal whenever it sees it. (See Figure 7.20.)

Circuits are evaluated in a dataflow fashion: at each time step, the inputs are set and the signals propagate through the circuit. Whenever a gate has all its inputs, it produces an output. This process is closely related to the process of forward chaining in an AND-OR graph such as Figure 7.15(b).

We said in the preceding section that circuit-based agents handle time more satisfactorily than propositional inference-based agents. This is because the value stored in each register gives the truth value of the corresponding proposition symbol at the current time t, rather than having a different copy for each different time step. For example, we might have an Alive register that should contain {rue when the wumpus is alive and false when it is dead. This register corresponds to the proposition symbol Alive!, so on each time step it refers to a different proposition. The internal state of the agent—i.e., its memory—is maintained by connecting the output of a register back into the circuit through a delay line. This delivers the value of the register at the previous time step. Figure 7.20 shows an example. The value for Alive is given by the conjunction of the negation of Scream and the delayed value of Alive itself. In terms of propositions, the circuit for Alive implements the biconditional

Alive! < =Scream' A Alive! ™! 74)

which says that the wumpus is alive at time ¢ if and only if there was no scream perceived at time ¢ (from a scream at { — 1) and it was alive at { — 1. We assume that the circuit is initialized with Alive set to true. Therefore, Alive will remain true until there is a scream, whereupon it will become false and stay false. This is exactly what we want.

Section 7.7.

Agents Based on Propositional Logic 229

[>

Breeze O

Stench []

Glitter []

Bump

Figure 7.21 The circuit for determining whether the agent is at [1,1]. Every location and orientation register has a similar circuit attached.

The agent’s location can be handled in much the same way as the wumpus’s health. We need an L, , register for each = and y; its value should be true if the agent is at [x,y]. The circuit that sets the value of L; , is, however, much more complicated than the circuit for Alive. For example, the agent is at [1,1] at time £ if (a) it was there at ¢ — 1 and either didn’t move forward or tried but bumped into a wall; or (b) it was at [1,2] facing down and moved forward; or (¢) it was at [2,1] facing left and moved forward:

Li, & (Li A (~Forward"™' Vv Bump!) V (LS A (FacingDouwn'~t A Forward\*™')) (7.5) V (Ls A (FacingLeft'™' A Forward" 1)).

The circuit for Ly; is shown in Figure 7.21. Every location register has a similar circuit attached to it. Exercise 7.13(b) asks you to design a circuit for the orientation propositions.

The circuits in Figures 7.20 and 7.21 maintain the correct truth values for Alive and L, , for all time. These propositions are unusual, however, in that their correct truth values can always be ascertained. Consider instead the proposition By 4: square [4,4] is breezy. Although this proposition’s truth value remains fixed, the agent cannot learn that truth value until it has visited [4,4] (or deduced that there is an adjacent pit). Propositional and firstorder logic are designed to represent true, false, and unknown propositions automatically, but circuits are not: the register for B4 4 must contain some value, either true or false, even before the truth has been discovered. The value in the register might well be the wrong one, and this could lead the agent astray. In other words, we need to represent three possible states (By,4 is known true, known false, or unknown) and we only have one bit to do it with.

The solution to this problem is to use two bits instead of one. By 4 is represented by two registers that we will call K(By4) and K(—By 4), where K stands for “known.”. (Remember that these are still just symbols with complicated names, even though they look like structured

230

Chapter 7. Logical Agents

KNOWLEDGE PROPOSITION

expressions!) When both K (By 4) and K(—By4) are false, it means the truth value of By4 is unknown. (If both are true, there’s a bug in the knowledge base!) Now whenever we would use By 4 in some part of the circuit, we use K(By 4) instead ; and whenever we would use —By 4, we use K(—By 4). In general, we represent each potentially indeterminate proposition with two knowledge propositions that state whether the underlying proposition is known to be true and known to be false.

We will see an example of how to use knowledge propositions shortly. First, we need to work out how to determine the truth values of the knowledge propositions themselves. Notice that, whereas By 4 has a fixed truth value, K (B44) and K(—By 4) do change as the agent finds out more about the world. For example, K (By 4) starts out false and then becomes true as soon as By 4 can be determined to be true—that is, when the agent is in [4,4] and detects a breeze. It stays true thereafter. So we have

K (Bsa)! & K(Bya)'™'V (Lj, A Breeze). (7.6)

A similar equation can be written for K (By 4)".

Now that the agent knows about breezy squares, it can deal with pits. The absence of a pit in a square can be ascertained if and only if one of the neighboring squares is known not to be breezy. For example, we have

K(=Py 4) = K(—=Bj,)" Vv K(—=By3)t . 1.7)

Determining that there is a pit in a square is more difficult—there must be a breeze in an adjacent square that cannot be accounted for by another pit:

K(Py4)t © (K(B3a)' AK(=Py4)' AK(=P33)!) V (K(Bsg)! ANK(=Py2)t AK(—-Ps3)t).

While the circuits for determining the presence or absence of pits are somewhat hairy, they have only a constant number of gates for each square. This property is essential if we are to build circuit-based agents that scale up in a reasonable way. It is really a property of the wumpus world itself; we say that an environment exhibits locality if the truth of each proposition of interest can be determined looking only at a constant number of other propositions. Locality is very sensitive to the precise “physics” of the environment. For example, the minesweeper domain (Exercise 7.11) is nonlocal because determining that a mine is in a given square can involve looking at squares arbitrarily far away. For nonlocal domains, circuit-based agents are not always practical.

There is one issue around which we have tiptoed carefully: the question of acyclicity. A circuit is acyclic if every path that connects the output of a register back to its input has an intervening delay element. We require that all circuits be acyclic because cyclic circuits, as physical devices, do not work! They can go into unstable oscillations resulting in undefined values. As an example of a cyclic circuit, consider the following augmentation of Equation (7.6):

K(Bya)' © K(Bysg)t'V (Lis A Breeze!) V K(Ps4)' V K(Py3)" (7.9)

The extra disjuncts, K(P34)" and K (Ps 3), allow the agent to determine breeziness from the known presence of adjacent pits, which seems entirely reasonable. Now, unfortunately,

(7.8)

Section 7.7.

Agents Based on Propositional Logic 231

breeziness depends on adjacent pits, and pits depend on adjacent breeziness through equations such as Equation (7.8). Therefore, the complete circuit would contain cycles.

The difficulty is not that the augmented Equation (7.9) is incorrect. Rather, the problem is that the interlocking dependencies represented by these equations cannot be resolved by the simple mechanism of propagating truth values in the corresponding Boolean circuit, The acyclic version using Equation (7.6), which determines breeziness only from direct observation, is incomplete in the sense that at some points the circuit-based agent might know less than an inference-based agent using a complete inference procedure. For example, if there is a breeze in [1,1], the inference-based agent can conclude that there is also a breeze in [2,2], whereas the acyclic circuit-based agent using Equation (7.6) cannot. A complete circuit can be built—after all, sequential circuits can emulate any digital computer—but it would be significantly more complex.

A comparison

The inference-based agent and the circuit-based agent represent the declarative and procedural extremes in agent design. They can be compared along several dimensions:

o Conciseness: The circuit-based agent, unlike the inference-based agent, need not have separate copies of its “knowledge” for every time step. Instead, it refers only to the current and previous time steps. Both agents need copies of the “physics” (expressed as sentences or circuits) for every square and therefore do not scale well to larger environments. In environments with many objects related in complex ways, the number of propositions will swamp any propositional agent. Such environments require the expressive power of first-order logic. (See Chapter 8.) Propositional agents of both kinds are also poorly suited for expressing or solving the problem of finding a path to a nearby safe square. (For this reason, PL.-WUMPUSAGENT falls back on a search algorithm.)

eo Computational efficiency: In the worst case, inference can take time exponential in the number of symbols, whereas evaluating a circuit takes time linear in the size of the circuit (or linear in the depth of the circuit if realized as a physical device). In practice, however, we saw that DPLL completed the required inferences very quickly.

eo Completeness: We suggested earlier that the circuit-based agent might be incomplete because of the acyclicity restriction. The reasons for incompleteness are actually more fundamental. First, remember that a circuit executes in time linear in the circuit size. This means that, for some environments, a circuit that is complete (i.e., one that computes the truth value of every determinable proposition) must be exponentially larger than the inference-based agent’s KB. Otherwise, we would have a way to solve the propositional entailment problem in less than exponential time, which is very unlikely. A second reason is the nature of the internal state of the agent. The inference-based agent remembers every percept and knows, either implicitly or explicitly, every sentence that follows from the percepts and initial KB. For example, given Bj 1, it knows the disjunction P; 2 V Py 1, from which Bs 5 follows. The circuit-based agent, on the

14 In fact, all the inferences done by a circuit can be done in linear time by DPLL! This is because evaluating a circuit, like forward chaining, can be emulated by DPLL using the unit propagation rule.

232

Chapter 7. Logical Agents

COMPILATION

other hand, forgets all previous percepts and remembers just the individual propositions stored in registers. Thus, Py » and P, | remain individually unknown after the first percept, so no conclusion will be drawn about Bo 5.

o Ease of construction: This is a very important issue about which it is hard to be precise. Certainly, this author found it much easier to state the “physics” declaratively, whereas devising small, acyclic, not-too-incomplete circuits for direct detection of pits seemed quite difficult.

In sum, it seems there are tradeoffs among computational efficiency, conciseness, completeness, and ease of construction. When the connection between percepts and actions is simple— as in the connection between Glitter and Grab—a circuit seems optimal. For more complex connections, the declarative approach may be better. In a domain such as chess, for example, the declarative rules are concise and easily encoded (at least in first-order logic), but a circuit for computing moves directly from board states would be unimaginably vast.

We see different points on these tradeoffs in the animal kingdom. The lower animals with very simple nervous systems are probably circuit-based, whereas higher animals, including humans, seem to perform inference on explicit representations. This enables them to compute much more complex agent functions. Humans also have circuits to implement reflexes, and perhaps also compile declarative representations into circuits when certain inferences become routine. In this way, a hybrid agent design (see Chapter 2) can have the best of both worlds.

We have introduced knowledge-based agents and have shown how to define a logic with which such agents can reason about the world. The main points are as follows:

o Intelligent agents need knowledge about the world in order to reach good decisions.

e Knowledge is contained in agents in the form of sentences in a knowledge representation language that are stored in a knowledge base.

o A knowledge-based agent is composed of a knowledge base and an inference mechanism. It operates by storing sentences about the world in its knowledge base, using the inference mechanism to infer new sentences, and using these sentences to decide what action to take.

o A representation language is defined by its syntax, which specifies the structure of sentences, and its semantics, which defines the truth of each sentence in each possible world or model.

o The relationship of entailment between sentences is crucial to our understanding of reasoning. A sentence « entails another sentence 3 if 3 is true in all worlds where « is true. Equivalent definitions include the validity of the sentence « = [3 and the unsatisfiability of the sentence oo A =.

Section 7.8. Summary 233

eo Inference is the process of deriving new sentences from old ones. Sound inference algorithms derive only sentences that are entailed; complete algorithms derive all sentences that are entailed.

¢ Propositional logic is a very simple language consisting of proposition symbols and logical connectives. It can handle propositions that are known true, known false, or completely unknown.

o The set of possible models, given a fixed propositional vocabulary, is finite, so entailment can be checked by enumerating models. Efficient model-checking inference algorithms for propositional logic include backtracking and local-search methods and can often solve large problems very quickly.

o Inference rules are patterns of sound inference that can be used to find proofs. The resolution rule yields a complete inference algorithm for knowledge bases that are expressed in conjunctive normal form. Forward chaining and backward chaining are very natural reasoning algorithms for knowledge bases in Horn form.

¢ Two kinds of agents can be built on the basis of propositional logic: inference-based agents use inference algorithms to keep track of the world and deduce hidden properties, whereas circuit-based agents represent propositions as bits in registers and update them using signal propagation in logical circuits.

o Propositional logic is reasonably effective for certain tasks within an agent, but does not scale to environments of unbounded size because it lacks the expressive power to deal concisely with time, space, and universal patterns of relationships among objects.

BIBLIOGRAPHICAL AND HISTORICAL NOTES

John McCarthy’s paper “Programs with Common Sense” (McCarthy, 1958, 1968) promulgated the notion of agents that use logical reasoning to mediate between percepts and actions. It also raised the flag of declarativism, pointing out that telling an agent what it needs to know is a very elegant way to build software. Allen Newell's (1982) article “The Knowledge Level” makes the case that rational agents can be described and analyzed at an abstract level defined by the knowledge they possess rather than the programs they run. The declarative and procedural approaches to Al are compared in Boden (1977). The debate was revived by, among others, Brooks (1991) and Nilsson (1991).

Logic itself had its origins in ancient Greek philosophy and mathematics. Various logical principles—principles connecting the syntactic structure of sentences with their truth and falsity, with their meaning, or with the validity of arguments in which they figure—are scattered in the works of Plato. The first known systematic study of logic was carried out by Aristotle, whose work was assembled by his students after his death in 322 B.C. as a

SYLLOGISMS treatise called the Organon. Aristotle’s syllogisms were what we would now call inference rules. Although the syllogisms included elements of both propositional and first-order logic, the system as a whole was very weak by modern standards. It did not allow for patterns of inference that apply to sentences of arbitrary complexity, as in modern propositional logic.

234

Chapter 7. Logical Agents

The closely related Megarian and Stoic schools (originating in the fifth century B.C. and continuing for several centuries thereafter) introduced the systematic study of implication and other basic constructs still used in modern propositional logic. The use of truth tables for defining logical connectives is due to Philo of Megara. The Stoics took five basic inference rules as valid without proof, including the rule we now call Modus Ponens. They derived a number of other rules from these five, using among other principles the deduction theorem (page 210) and were much clearer about the notion of proof than Aristotle was. The Stoics claimed that their logic was complete in the sense of capturing all valid inferences, but what remains is too fragmentary to tell. A good account of the history of Megarian and Stoic logic, as far as it is known, is given by Benson Mates (1953).

The idea of reducing logical inference to a purely mechanical process applied to a formal language is due to Wilhelm Leibniz (1646-1716). Leibniz’s own mathematical logic, however, was severely defective, and he is better remembered simply for introducing these ideas as goals to be attained than for his attempts at realizing them.

George Boole (1847) introduced the first comprehensive and workable system of formal logic in his book The Mathematical Analysis of Logic. Boole’s logic was closely modeled on the ordinary algebra of real numbers and used substitution of logically equivalent expressions as its primary inference method. Although Boole’s system still fell short of full propositional logic, it was close enough that other mathematicians could quickly fill in the gaps. Schroder (1877) described conjunctive normal form, while Horn form was introduced much later by Alfred Horn (1951). The first comprehensive exposition of modern propositional logic (and first-order logic) is found in Gottlob Frege’s (1879) Begriffschrift (“Concept Writing” or “Conceptual Notation”).

The first mechanical device to carry out logical inferences was constructed by the third Earl of Stanhope (1753-1816). The Stanhope Demonstrator could handle syllogisms and certain inferences in the theory of probability. William Stanley Jevons, one of those who improved upon and extended Boole’s work, constructed his “logical piano” in 1869 to per form inferences in Boolean logic. An entertaining and instructive history of these and other early mechanical devices for reasoning is given by Martin Gardner (1968). The first published computer program for logical inference was the Logic Theorist of Newell, Shaw, and Simon (1957). This program was intended to model human thought processes. Martin Davis (1957) had actually designed a program that came up with a proof in 1954, but the Logic Theorist’s results were published slightly earlier. Both Davis’s 1954 program and the Logic Theorist were based on somewhat ad hoc methods that did not strongly influence later automated deduction.

Truth tables as a method of testing the validity or unsatisfiability of sentences in the language of propositional logic were introduced independently by Ludwig Wittgenstein (1922) and Emil Post (1921). In the 1930s, a great deal of progress was made on inference methods for first-order logic. In particular, Godel (1930) showed that a complete procedure for inference in first-order logic could be obtained via a reduction to propositional logic, using Herbrand’s theorem (Herbrand, 1930). We will take up this history again in Chapter 9; the important point here is that the development of efficient propositional algorithms in the 1960s was motivated largely by the interest of mathematicians in an effective theorem prover

Section 7.8.

Summary 235

for first-order logic. The Davis-Putnam algorithm (Davis and Putnam, 1960) was the first effective algorithm for propositional resolution but was in most cases much less efficient than the DPLL backtracking algorithm introduced two years later (1962). The full resolution rule and a proof of its completeness appeared in a seminal paper by J. A. Robinson (1965), which also showed how to do first-order reasoning without resort to propositional techniques.

Stephen Cook (1971) showed that deciding satisfiability of a sentence in propositional logic is NP-complete. Since deciding entailment is equivalent to deciding unsatisfiability, it is co-NP-complete. Many subsets of propositional logic are known for which the satisfiability problem is polynomially solvable; Horn clauses are one such subset. The linear-time forward-chaining algorithm for Horn clauses is due to Dowling and Gallier (1984), who describe their algorithm as a dataflow process similar to the propagation of signals in a circuit. Satisfiability has become one of the canonical examples for NP reductions; for example Kaye (2000) showed that the Minesweeper game (see Exercise 7.11) is NP-complete.

Local search algorithms for satisfiability were tried by various authors throughout the 1980s; all of the algorithms were based on the idea of minimizing the number of unsatisfied clauses (Hansen and Jaumard, 1990). A particularly effective algorithm was developed by Gu (1989) and independently by Selman ef al. (1992), who called it GS AT and showed that it was capable of solving a wide range of very hard problems very quickly. The WALKS AT algorithm described in the chapter is due to Selman ez al. (1996).

The “phase transition” in satisfiability of random k-SAT problems was first observed by Simon and Dubois (1989). Empirical results due to Crawford and Auton (1993) suggest that it lies at a clause/variable ratio of around 4.24 for large random 3-SAT problems; this paper also describes a very efficient implementation of DPLL. Bayardo and Schrag (1997) describe another efficient DPLL implementation using techniques from constraint satisfaction, and (Moskewicz et al., 2001) describe CHAFF, which solves million-variable hardware verification problems and was the winner of the SAT 2002 Competition. Li and Anbulagan (1997) discuss heuristics based on unit propagation that allow for fast solvers. Cheeseman et al. (1991) provide data on a number of related problems and conjecture that all NP hard problems have a phase transition. Kirkpatrick and Selman (1994) describe ways in which techniques from statistical physics might provide insight into the precise “shape” of the phase transition. Theoretical analysis of its location is still rather weak: all that can be proved is that it lies in the range [3.003,4.598] for random 3-SAT. Cook and Mitchell (1997) give an excellent survey of results on this and several other satisfiability-related topics.

Early theoretical investigations showed that DPLL has polynomial average-case complexity for certain natural distributions of problems. This potentially exciting fact became less exciting when Franco and Paull (1983) showed that the same problems could be solved in constant time simply by guessing random assignments. The random-generation method described in the chapter produces much harder problems. Motivated by the empirical success of local search on these problems, Koutsoupias and Papadimitriou (1992) showed that a simple hill-climbing algorithm can solve almost all satisfiability problem instances very quickly, suggesting that hard problems are rare. Moreover, Schoning (1999) exhibited a randomized variant of GS AT whose worst-case expected runtime on 3-SAT problems is 1.333" —still exponential, but substantially faster than previous worst-case bounds. Satisfiability algorithms

236

Chapter 7. Logical Agents

are still a very active area of research; the collection of articles in Du et al. (1999) provides a good starting point.

Circuit-based agents can be traced back to the seminal paper of McCulloch and Pitts (1943), which initiated the field of neural networks. Contrary to popular supposition, the paper was concerned with the implementation of a Boolean circuit-based agent design in the brain. Circuit-based agents have received little attention in Al, however. The most notable exception is the work of Stan Rosenschein (Rosenschein, 1985; Kaelbling and Rosenschein, 1990), who developed ways to compile circuit-based agents from declarative descriptions of the task environment. The circuits for updating propositions stored in registers are closely related to the successor-state axiom developed for first-order logic by Reiter (1991). The work of Rod Brooks (1986, 1989) demonstrates the effectiveness of circuit-based designs for controlling robots—a topic we take up in Chapter 25. Brooks (1991) argues that circuit-based designs are all that is needed for Al—that representation and reasoning are cumbersome, expensive, and unnecessary. In our view, neither approach is sufficient by itself.

The wumpus world was invented by Gregory Yob (1975). Ironically, Yob developed it because he was bored with games played on a grid: the topology of his original wumpus world was a dodecahedron; we put it back in the boring old grid. Michael Genesereth was the first to suggest that the wumpus world be used as an agent testbed.

EXERCISES

7.1 Describe the wumpus world according to the properties of task environments listed in Chapter 2.

7.2 Suppose the agent has progressed to the point shown in Figure 7.4(a), having perceived nothing in [1,1], a breeze in [2,1], and a stench in [1,2]. and is now concerned with the contents of [1,3], [2,2], and [3,1]. Each of these can contain a pit and at most one can contain a wumpus. Following the example of Figure 7.5, construct the set of possible worlds. (You should find 32 of them.) Mark the worlds in which the KB is true and those in which each of the following sentences is true:

op = “There is no pit in [2,2].” as = “There is a wumpus in [1,3].

Hence show that KB |= ap and KB |= a3.

7.3 Consider the problem of deciding whether a propositional logic sentence is true in a given model.

a. Write a recursive algorithm PL-TRUE?(s, m) that returns ¢rue if and only if the sentence s is true in the model m (where m assigns a truth value for every symbol in s). The algorithm should run in time linear in the size of the sentence. (Alternatively, use a version of this function from the online code repository.)

Section 7.8.

Summary 237

b. Give three examples of sentences that can be determined to be true or false in a partial model that does not specify a truth value for some of the symbols.

¢. Show that the truth value (if any) of a sentence in a partial model cannot be determined efficiently in general.

d. Modify your PL-TRUE? algorithm so that it can sometimes judge truth from partial models, while retaining its recursive structure and linear runtime. Give three examples of sentences whose truth in a partial model is not detected by your algorithm.

e. Investigate whether the modified algorithm makes TT-ENTAILS? more efficient.

7.4 Prove each of the following assertions:

. «is valid if and only if True | a.

. Forany «a, False = «a.

. a |= gif and only if the sentence (a = 3) is valid. . a = if and only if the sentence (a «< 3) is valid.

oe na TO

. a = Bif and only if the sentence («x A —(3) is unsatisfiable.

7.5 Consider a vocabulary with only four propositions, A, B, C', and D. How many models are there for the following sentences?

a (ANB)V (BAC)

b. AVE

c¢c As Be COC

7.6 We have defined four different binary logical connectives.

a. Are there any others that might be useful? b. How many binary connectives can there be? ¢. Why are some of them not very useful?

7.7 Using a method of your choice, verify each of the equivalences in Figure 7.11.

7.8 Decide whether each of the following sentences is valid, unsatisfiable, or neither. Verify your decisions using truth tables or the equivalence rules of Figure 7.11.

. Smoke = Smoke

. Smoke = Fire

. (Smoke = Fire) = (—Smoke = —Fire)

. Smoke V Fire V = Fire

. ((Smoke A Heat) = Fire) < ((Smoke = Fire) V (Heat = Fire))

. (Smoke = Fire) = ((Smoke A Heat) = Fire)

. Big V Dumb V (Big = Dumb)

. (Big A Dumb) V = Dumb

sow

® Lue

= ue

238 Chapter 7. Logical Agents

7.9 (Adapted from Barwise and Etchemendy (1993).) Given the following, can you prove that the unicorn is mythical? How about magical? Horned?

If the unicorn is mythical, then it is immortal, but if it is not mythical, then it is a mortal mammal. If the unicorn is either immortal or a mammal, then it is horned. The unicorn is magical if it is horned.

7.10 Any propositional logic sentence is logically equivalent to the assertion that each possible world in which it would be false is not the case. From this observation, prove that any sentence can be written in CNF.

7.11 Minesweeper, the well-known computer game, is closely related to the wumpus world. A minesweeper world is a rectangular grid of N squares with M invisible mines scattered among them. Any square may be probed by the agent; instant death follows if a mine is probed. Minesweeper indicates the presence of mines by revealing, in each probed square, the number of mines that are directly or diagonally adjacent. The goal is to have probed every unmined square.

a. Let X; ; be true iff square [¢, j] contains a mine. Write down the assertion that there are exactly two mines adjacent to [1,1] as a sentence involving some logical combination of X; ; propositions.

b. Generalize your assertion from (a) by explaining how to construct a CNF sentence asserting that k of n neighbors contain mines.

c. Explain precisely how an agent can use DPLL to prove that a given square does (or does not) contain a mine, ignoring the global constraint that there are exactly M mines in all.

d. Suppose that the global constraint is constructed via your method from part (b). How does the number of clauses depend on M and N? Suggest a way to modify DPLL so that the global constraint does not need to be represented explicitly.

e. Are any conclusions derived by the method in part (c) invalidated when the global constraint is taken into account?

f. Give examples of configurations of probe values that induce long-range dependencies such that the contents of a given unprobed square would give information about the contents of a far-distant square. [Hinz: consider an N x 1 board.]

7.12 This exercise looks into the relationship between clauses and implication sentences.

a. Show that the clause (—P; V --V =P, V Q) is logically equivalent to the implication sentence (Py A--A Pp) = Q.

b. Show that every clause (regardless of the number of positive literals) can be written in the form (Py A--A Pp) = (Q1V +++ V Qy), where the Ps and Qs are proposition symbols. A knowledge base consisting of such sentences is in implicative normal

TE form or Kowalski form.

¢. Write down the full resolution rule for sentences in implicative normal form.

Section 7.8.

Summary 239

7.13 In this exercise, you will design more of the circuit-based wumpus agent. a. Write an equation, similar to Equation (7.4), for the Arrow proposition, which should be true when the agent still has an arrow. Draw the corresponding circuit. b. Repeat part (a) for FacingRight, using Equation (7.5) as a model. c. Create versions of Equations 7.7 and 7.8 for finding the wumpus, and draw the circuit.

7.14 Discuss what is meant by optimal behavior in the wumpus world. Show that our definition of the PL-WUMPUS-AGENT is not optimal, and suggest ways to improve it.

7.15 Extend PL-WUMPUS-AGENT so that it keeps track of all relevant facts within the knowledge base.

7.16 How long does it take to prove KB |= « using DPLL when « is a literal already contained in KB? Explain.

7.17 Trace the behavior of DPLL on the knowledge base in Figure 7.15 when trying to prove (J, and compare this behavior with that of the forward chaining algorithm.

FIRST-ORDER LOGIC

FIRST-ORDER LOGIC

In which we notice that the world is blessed with many objects, some of which are related to other objects, and in which we endeavor to reason about them.

In Chapter 7, we showed how a knowledge-based agent could represent the world in which it operates and deduce what actions to take. We used propositional logic as our representation language because it sufficed to illustrate the basic concepts of logic and knowledge-based agents. Unfortunately, propositional logic is too puny a language to represent knowledge of complex environments in a concise way. In this chapter, we examine first-order logic,’ which is sufficiently expressive to represent a good deal of our commonsense knowledge. It also either subsumes or forms the foundation of many other representation languages and has been studied intensively for many decades. We begin in Section 8.1 with a discussion of representation languages in general; Section 8.2 covers the syntax and semantics of first-order logic; Sections 8.3 and 8.4 illustrate the use of first-order logic for simple representations.

In this section, we will discuss the nature of representation languages. Our discussion will motivate the development of first-order logic, a much more expressive language than the propositional logic introduced in Chapter 7. We will look at propositional logic and at other kinds of languages to understand what works and what fails. Our discussion will be cursory, compressing centuries of thought, trial, and error into a few paragraphs.

Programming languages (such as C++ or Java or Lisp) are by far the largest class of formal languages in common use. Programs themselves represent, in a direct sense, only computational processes. Data structures within programs can represent facts; for example, a program could use a 4 x 4 array to represent the contents of the wumpus world. Thus, the programming language statement World[2,2] + Pit is a fairly natural way to assert that there is a pit in square [2,2]. (Such representations might be considered ad hoc; database systems were developed precisely to provide a more general, domain-independent way to

1 Also called first-order predicate calculus, sometimes abbreviated as FOL or FOPC.

240

Section 8.1.

Representation Revisited 241

COMPOSITIONALITY

store and retrieve facts.) What programming languages lack is any general mechanism for deriving facts from other facts; each update to a data structure is done by a domain-specific procedure whose details are derived by the programmer from his or her own knowledge of the domain. This procedural approach can be contrasted with the declarative nature of propositional logic, in which knowledge and inference are separate, and inference is entirely domain-independent.

A second drawback of data structures in programs (and of databases, for that matter) is the lack of any easy way to say, for example, “There is a pit in [2,2] or [3,1]” or “If the wumpus is in [1,1] then he is not in [2,2].” Programs can store a single value for each variable, and some systems allow the value to be “unknown,” but they lack the expressiveness required to handle partial information.

Propositional logic is a declarative language because its semantics is based on a truth relation between sentences and possible worlds. It also has sufficient expressive power to deal with partial information, using disjunction and negation. Propositional logic has a third property that is desirable in representation languages, namely compositionality. In a compositional language, the meaning of a sentence is a function of the meaning of its parts. For example, “S74 A S12” is related to the meanings of “S14” and “S72.” It would be very strange if “S51 4” meant that there is a stench in square [1,4] and “St 2” meant that there is a stench in square [1,2], but “S; 4 A512” meant that France and Poland drew 1-1 in last week’s ice hockey qualifying match. Clearly, noncompositionality makes life much more difficult for the reasoning system.

As we saw in Chapter 7, propositional logic lacks the expressive power to describe an environment with many objects concisely. For example, we were forced to write a separate rule about breezes and pits for each square, such as

B11 © (PiaV Py).

In English, on the other hand, it seems easy enough to say, once and for all, “Squares adjacent to pits are breezy.” The syntax and semantics of English somehow make it possible to describe the environment concisely.

A moment’s thought suggests that natural languages (such as English or Spanish) are very expressive indeed. We managed to write almost this whole book in natural language, with only occasional lapses into other languages (including logic, mathematics, and the language of diagrams). There is a long tradition in linguistics and the philosophy of language that views natural language as essentially a declarative knowledge representation language and attempts to pin down its formal semantics. Such a research program, if successful, would be of great value to artificial intelligence because it would allow a natural language (or some derivative) to be used within representation and reasoning systems.

The modern view of natural language is that it serves a somewhat different purpose, namely as a medium for communication rather than pure representation. When a speaker points and says, “Look!” the listener comes to know that, say, Superman has finally appeared over the rooftops. Yet we would not want to say that the sentence “Look!” encoded that fact. Rather, the meaning of the sentence depends both on the sentence itself and on the context in which the sentence was spoken. Clearly, one could not store a sentence such as “Look!” in

242

Chapter 8. First-Order Logic

OBJECTS RELATIONS FUNCTIONS

PROPERTIES

ONTOLOGICAL COMMITMENT

a knowledge base and expect to recover its meaning without also storing a representation of the context—which raises the question of how the context itself can be represented. Natural languages are also noncompositional—the meaning of a sentence such as “Then she saw it” can depend on a context constructed by many preceding and succeeding sentences. Finally, natural languages suffer from ambiguity, which would cause difficulties for thinking. As Pinker (1995) puts it: “When people think about spring, surely they are not confused as to whether they are thinking about a season or something that goes boing—and if one word can correspond to two thoughts, thoughts can’t be words.”

Our approach will be to adopt the foundation of propositional logic—a declarative, compositional semantics that is context-independent and unambiguous—and build a more expressive logic on that foundation, borrowing representational ideas from natural language while avoiding its drawbacks. When we look at the syntax of natural language, the most obvious elements are nouns and noun phrases that refer to objects (squares, pits, wumpuses) and verbs and verb phrases that refer to relations among objects (is breezy, is adjacent to, shoots). Some of these relations are functions—relations in which there is only one “value” for a given “input.” It is easy to start listing examples of objects, relations, and functions:

o Objects: people, houses, numbers, theories, Ronald McDonald, colors, baseball games, wars, centuries . . .

o Relations: these can be unary relations or properties such as red, round, bogus, prime, multistoried . . ., or more general n-ary relations such as brother of, bigger than, inside, part of, has color, occurred after, owns, comes between, . . .

o Functions: father of, best friend, third inning of, one more than, beginning of . . .

Indeed, almost any assertion can be thought of as referring to objects and properties or relations. Some examples follow:

® “One plus two equals three” Objects: one, two, three, one plus two; Relation: equals; Function: plus. (“One plus two” is a name for the object that is obtained by applying the function “plus” to the objects “one” and “two.” Three is another name for this object.)

e “Squares neighboring the wumpus are smelly.” Objects: wumpus, squares; Property: smelly; Relation: neighboring.

e “Evil King John ruled England in 1200.” Objects: John, England, 1200; Relation: ruled; Properties: evil, king.

The language of first-order logic, whose syntax and semantics we will define in the next section, is built around objects and relations. It has been so important to mathematics, philosophy, and artificial intelligence precisely because those fields—and indeed, much of everyday human existence—can be usefully thought of as dealing with objects and the relations among them. First-order logic can also express facts about some or all of the objects in the universe. This enables one to represent general laws or rules, such as the statement “Squares neighboring the wumpus are smelly.”

The primary difference between propositional and first-order logic lies in the ontological commitment made by each language—that is, what it assumes about the nature of reality.

Section 8.1.

243

THE LANGUAGE OF THOUGHT

Philosophers and psychologists have long pondered how it is that humans and other animals represent knowledge. It is clear that the evolution of natural language has played an important role in developing this ability in humans. On the other hand, much psychological evidence suggests that humans do not employ language directly in their internal representations. For example, which of the following two phrases formed the opening of Section 8.17

“In this section, we will discuss the nature of representation languages . ..” “This section covers the topic of knowledge representation languages ...”

Wanner (1974) found that subjects made the right choice in such tests at chance level—about 50% of the time—but remembered the content of what they read with better than 90% accuracy. This suggests that people process the words to form some kind of nonverbal representation, which we call memory.

The exact mechanism by which language enables and shapes the representation of ideas in humans remains a fascinating question. The famous Sapir—-Wheorf hypothesis claims that the language we speak profoundly influences the way in which we think and make decisions, in particular by setting up the category structure by which we divide up the world into different sorts of objects. Whorf (1956) claimed that Eskimos have many words for snow and thus experience snow in a different way from speakers of other languages. Some linguists dispute the factual basis for this claim—Pullum (1991) argues that Inuit, Yupik, and other related languages seem to have about the same number of words for snow-related concepts as English—while others support it (Fortescue, 1984). It seems unarguably true that populations having greater familiarity with some aspects of the world develop much more detailed vocabularies—for example, field entomologists divide what most of us call beetles into hundreds of thousands of species and are personally familiar with many of these. (The evolutionary biologist J. B. S. Haldane once complained of “An inordinate fondness for beetles” on the part of the Creator.) Moreover, expert skiers have many terms for snow—powder, chowder, mashed potatoes, crud, corn, cement, crust, sugar, asphalt, corduroy, fluff, glop, and so on—that represent distinctions unfamiliar to the lay person. What is unclear is the direction of causality—do skiers become aware of the distinctions only by learning the words, or do the distinctions emerge from individual experience and become matched with the labels current in the community? This question is especially important in the study of child development. As yet, we have little understanding of the extent to which learning language and learning to think are intertwined. For example, does the knowledge of a name for a concept, such as bachelor, make it easier to construct and reason with more complex concepts that include that name, such as eligible bachelor?

244

Chapter 8. First-Order Logic

TEMPORAL LOGIC

HIGHER-ORDER LOGIC

EPISTEMOLOGICAL COMMITMENTS

For example, propositional logic assumes that there are facts that either hold or do not hold in the world. Each fact can be in one of two states: true or false. First-order logic assumes more; namely, that the world consists of objects with certain relations among them that do or do not hold. Special-purpose logics make still further ontological commitments; for example, temporal logic assumes that facts hold at particular times and that those times (which may be points or intervals) are ordered. Thus, special-purpose logics give certain kinds of objects (and the axioms about them) “first-class” status within the logic, rather than simply defining them within the knowledge base. Higher-order logic views the relations and functions referred to by first-order logic as objects in themselves. This allows one to make assertions about all relations—for example, one could wish to define what it means for a relation to be transitive. Unlike most special-purpose logics, higher-order logic is strictly more expressive than first-order logic, in the sense that some sentences of higher-order logic cannot be expressed by any finite number of first-order logic sentences.

A logic can also be characterized by its epistemological commitments—the possible states of knowledge that it allows with respect to each fact. In both propositional and firstorder logic, a sentence represents a fact and the agent either believes the sentence to be true, believes it to be false, or has no opinion. These logics therefore have three possible states of knowledge regarding any sentence. Systems using probability theory, on the other hand, can have any degree of belief, ranging from 0 (total disbelief) to 1 (total belief).® For example, a probabilistic wumpus-world agent might believe that the wumpus is in [1,3] with probability 0.75. The ontological and epistemological commitments of five different logics are summarized in Figure 8.1.

[]

In the next section, we will launch into the details of first-order logic. Just as a student of physics requires some familiarity with mathematics, a student of AI must develop a talent for working with logical notation. On the other hand, it is also important not to get too concerned with the specifics of logical notation—after all, there are dozens of different versions. The main things to keep hold of are how the language facilitates concise representations and how its semantics leads to sound reasoning procedures.

2 In contrast, facts in fuzzy logic have a degree of truth between 0 and 1. For example, the sentence “Vienna is a large city” might be true in our world only to degree 0.6.

3 It is important not to confuse the degree of belief in probability theory with the degree of truth in fuzzy logic. Indeed, some fuzzy systems allow uncertainty (degree of belief) about degrees of truth.

Section 8.2. Syntax and Semantics of First-Order Logic 245

8.2 SYNTAX AND SEMANTICS OF FIRST-ORDER LOGIC

We begin this section by specifying more precisely the way in which the possible worlds of first-order logic reflect the ontological commitment to objects and relations. Then we introduce the various elements of the language, explaining their semantics as we go along.

Models for first-order logic

Recall from Chapter 7 that the models of a logical language are the formal structures that constitute the possible worlds under consideration. Models for propositional logic are just sets of truth values for the proposition symbols. Models for first-order logic are more interesting. DOMAIN First, they have objects in them! The domain of a model is the set of objects it contains; cowan ELEMENTS these objects are sometimes called domain elements. Figure 8.2 shows a model with five objects: Richard the Lionheart, King of England from 1189 to 1199; his younger brother, the evil King John, who ruled from 1199 to 1215; the left legs of Richard and John; and a crown. The objects in the model may be related in various ways. In the figure, Richard and TUPLES John are brothers. Formally speaking, a relation is just the set of tuples of objects that are related. (A tuple is a collection of objects arranged in a fixed order and is written with angle brackets surrounding the objects.) Thus, the brotherhood relation in this model is the set

{ (Richard the Lionheart, King John), (King John, Richard the Lionheart) } . (8.1)

(Here we have named the objects in English, but you may, if you wish, mentally substitute the pictures for the names.) The crown is on King John’s head, so the “on head” relation contains

[]

Figure 8.2 A model containing five objects, two binary relations, three unary relations (indicated by labels on the objects), and one unary function, left-leg.

246

Chapter 8. First-Order Logic

TOTAL FUNCTIONS

CONSTANT SYMBOLS PREDICATE SYMBOLS

FUNCTION SYMBOLS

ARITY

INTERPRETATION

INTENDED INTERPRETATION

just one tuple, (the crown, King John). The “brother” and “on head” relations are binary relations—that is, they relate pairs of objects. The model also contains unary relations, or properties: the “person” property is true of both Richard and John; the “king” property is true only of John (presumably because Richard is dead at this point); and the “crown” property is true only of the crown.

Certain kinds of relationships are best considered as functions, in that a given object must be related to exactly one object in this way. For example, each person has one left leg, so the model has a unary “left leg” function that includes the following mappings:

(Richard the Lionheart) — Richard’s left leg

(King John) — John’s left leg . ®2)

Strictly speaking, models in first-order logic require total functions, that is, there must be a value for every input tuple. Thus, the crown must have a left leg and so must each of the left legs. There is a technical solution to this awkward problem involving an additional “invisible” object that is the left leg of everything that has no left leg, including itself. Fortunately, as long as one makes no assertions about the left legs of things that have no left legs, these technicalities are of no import.

Symbols and interpretations

We turn now to the syntax of the language. The impatient reader can obtain a complete description from the formal grammar of first-order logic in Figure 8.3.

The basic syntactic elements of first-order logic are the symbols that stand for objects, relations, and functions. The symbols, therefore, come in three kinds: constant symbols, which stand for objects; predicate symbols, which stand for relations; and function symbols, which stand for functions. We adopt the convention that these symbols will begin with uppercase letters. For example, we might use the constant symbols Richard and John; the predicate symbols Brother, OnHead, Person, King, and Crown; and the function symbol LeftLeg. As with proposition symbols, the choice of names is entirely up to the user. Each predicate and function symbol comes with an arity that fixes the number of arguments.

The semantics must relate sentences to models in order to determine truth. For this to happen, we need an interpretation that specifies exactly which objects, relations and functions are referred to by the constant, predicate, and function symbols. One possible interpretation for our example—which we will call the intended interpretation—is as follows:

o Richard refers to Richard the Lionheart and John refers to the evil King John.

e Brother refers to the brotherhood relation, that is, the set of tuples of objects given in Equation (8.1); OnHead refers to the “on head” relation that holds between the crown and King John; Person, King, and Crown refer to the sets of objects that are persons, kings, and crowns.

o LeftLeg refers to the “left leg” function, that is, the mapping given in Equation (8.2). There are many other possible interpretations relating these symbols to this particular model.

For example, one interpretation maps Richard to the crown and John to King John’s left leg. There are five objects in the model, so there are 25 possible interpretations just for the

Section 8.2.

Syntax and Semantics of First-Order Logic 247

Sentence — AtomicSentence | (Sentence Connective Sentence ) | Quantifier Variable, ... Sentence

| = Sentence AtomicSentence — Predicate(Term,...) | Term = Term Term — Function(Term,...)

| Constant | Variable

Connective — =| AN| V|& Quantifier — V| 3 Constant — A| Xi | John | --Variable — al z| s| --Predicate — Before | HasColor | Raining | --Function — Mother | LeftLeg | --

Figure 8.3 The syntax of first-order logic with equality, specified in Backus—Naur form. (See page 984 if you are not familiar with this notation.) The syntax is strict about parentheses; the comments about parentheses and operator precedence on page 205 apply equally to first-order logic.

constant symbols Richard and John. Notice that not all the objects need have a name—for example, the intended interpretation does not name the crown or the legs. It is also possible for an object to have several names; there is an interpretation under which both Richard and John refer to the crown. If you find this possibility confusing, remember that, in propositional logic, it is perfectly possible to have a model in which Cloudy and Sunny are both true; it is the job of the knowledge base to rule out models that are inconsistent with our knowledge. The truth of any sentence is determined by a model and an interpretation for the sentence’s symbols. Therefore, entailment, validity, and so on are defined in terms of all possible models and all possible interpretations. Tt is important to note that the number of domain elements in each model may be unbounded—for example, the domain elements may be integers or real numbers. Hence, the number of possible models is unbounded, as is the number of interpretations. Checking entailment by the enumeration of all possible models, which works for propositional logic, is not an option for first-order logic. Even if the number of objects is restricted, the number of combinations can be very large. With the symbols in our example, there are roughly 10% combinations for a domain with five objects. (See Exercise 8.5.)

248

Chapter 8. First-Order Logic

Terms

A term is a logical expression that refers to an object. Constant symbols are therefore terms, but it is not always convenient to have a distinct symbol to name every object. For example, in English we might use the expression “King John’s left leg” rather than giving a name to his leg. This is what function symbols are for: instead of using a constant symbol, we use LeftLeg(John). In the general case, a complex term is formed by a function symbol followed by a parenthesized list of terms as arguments to the function symbol. It is important to remember that a complex term is just a complicated kind of name. It is not a “subroutine call” that “returns a value.” There is no LeftLeg subroutine that takes a person as input and returns a leg. We can reason about left legs (e.g, stating the general rule that everyone has one and then deducing that John must have one) without ever providing a definition of LeftLeg. This is something that cannot be done with subroutines in programming languages.”

The formal semantics of terms is straightforward. Consider a term f(¢1,..., ty). The function symbol f refers to some function in the model (call it F'); the argument terms refer to objects in the domain (call them di, . . ., dy); and the term as a whole refers to the object that is the value of the function F applied to dy, .. ., d,,. For example, suppose the LeftLeg function symbol refers to the function shown in Equation (8.2) and John refers to King John, then LeftLeg( John) refers to King John's left leg. In this way, the interpretation fixes the referent of every term.

Atomic sentences

Now that we have both terms for referring to objects and predicate symbols for referring to relations, we can put them together to make atomic sentences that state facts. An atomic sentence is formed from a predicate symbol followed by a parenthesized list of terms:

Brother(Richard, John).

This states, under the intended interpretation given earlier, that Richard the Lionheart is the brother of King John.> Atomic sentences can have complex terms as arguments. Thus,

Married(Father( Richard), Mother(John))

states that Richard the Lionheart’s father is married to King John’s mother (again, under a suitable interpretation).

An atomic sentence is true in a given model, under a given interpretation, if the relation referred to by the predicate symbol holds among the objects referred to by the arguments.

4 )-expressions provide a useful notation in which new function symbols are constructed “on the fly.” For example, the function that squares its argument can be written as (Az z x z) and can be applied to arguments just like any other function symbol. A A-expression can also be defined and used as a predicate symbol. (See Chapter 22.) The lambda operator in Lisp plays exactly the same role. Notice that the use of A in this way does not increase the formal expressive power of first-order logic, because any sentence that includes a A-expression can be rewritten by “plugging in” its arguments to yield an equivalent sentence.

5 We will usually follow the argument ordering convention that P(x, y) is interpreted as “z isa P of y.”

Syntax and Semantics of First-Order Logic 249

QUANTIFIERS

VARIABLE

GROUND TERM

EXTENDED INTERPRETATION

Complex sentences

We can use logical connectives to construct more complex sentences, just as in propositional calculus. The semantics of sentences formed with logical connectives is identical to that in the propositional case. Here are four sentences that are true in the model of Figure 8.2 under our intended interpretation:

= Brother (LeftLeg( Richard), John) Brother(Richard, John) A Brother(John, Richard) King(Richard) V King(John)

= King(Richard) = King(John) .

Quantifiers

Once we have a logic that allows objects, it is only natural to want to express properties of entire collections of objects, instead of enumerating the objects by name. Quantifiers let us do this. First-order logic contains two standard quantifiers, called universal and existential.

Universal quantification (Vv)

Recall the difficulty we had in Chapter 7 with the expression of general rules in propositional logic. Rules such as “Squares neighboring the wumpus are smelly” and “All kings are persons” are the bread and butter of first-order logic. We will deal with the first of these in Section 8.3. The second rule, “All kings are persons,” is written in first-order logic as

Vz King(z) = Person(z).

VY is usually pronounced “For all ...”. (Remember that the upside-down A stands for “all.”) Thus, the sentence says, “For all x, if x is a king, then x is a person.” The symbol z is called a variable. By convention, variables are lowercase letters. A variable is a term all by itself, and as such can also serve as the argument of a function—for example, LeftLeg(z). A term with no variables is called a ground term.

Intuitively, the sentence V = P, where P is any logical expression, says that P is true for every object xz. More precisely, Vx P is true in a given model under a given interpretation if P is true in all possible extended interpretations constructed from the given interpretation, where each extended interpretation specifies a domain element to which x refers.

This sounds complicated, but it is really just a careful way of stating the intuitive meaning of universal quantification. Consider the model shown in Figure 8.2 and the intended interpretation that goes with it. We can extend the interpretation in five ways:

x — Richard the Lionheart, x — King John,

x — Richard’s left leg,

x — John’s left leg,

x — the crown.

The universally quantified sentence Va King(x) = Person(z) is true under the original interpretation if the sentence King(x) = Person(z) is true in each of the five extended inter

250

Chapter 8. First-Order Logic

pretations. That is, the universally quantified sentence is equivalent to asserting the following five sentences:

Richard the Lionheart is a king => Richard the Lionheart is a person. King John is a king = King John is a person.

Richard's left leg is a king = Richard’s left leg is a person.

John’s left leg is a king => John’s left leg is a person.

The crown is a king => the crown is a person.

Let us look carefully at this set of assertions. Since, in our model, King John is the only king, the second sentence asserts that he is a person, as we would hope. But what about the other four sentences, which appear to make claims about legs and crowns? Is that part of the meaning of “All kings are persons”? In fact, the other four assertions are true in the model, but make no claim whatsoever about the personhood qualifications of legs, crowns, or indeed Richard. This is because none of these objects is a king. Looking at the truth table for = (Figure 7.8), we see that the implication is true whenever its premise is false—regardless of the truth of the conclusion. Thus, by asserting the universally quantified sentence, which is equivalent to asserting a whole list of individual implications, we end up asserting the conclusion of the rule just for those objects for whom the premise is true and saying nothing at all about those individuals for whom the premise is false. Thus, the truth-table entries for = turn out to be perfect for writing general rules with universal quantifiers.

A common mistake, made frequently even by diligent readers who have read this paragraph several times, is to use conjunction instead of implication. The sentence

Vz King(xz) A Person(zx) would be equivalent to asserting

Richard the Lionheart is a king A Richard the Lionheart is a person, King John is a king A King John is a person, Richard's left leg is a king A Richard’s left leg is a person,

and so on. Obviously, this does not capture what we want.

Existential quantification (J)

Universal quantification makes statements about every object. Similarly, we can make a statement about some object in the universe without naming it, by using an existential quantifier. To say, for example, that King John has a crown on his head, we write

dz Crown(z) A OnHead(x, John) .

dx is pronounced “There exists an x such that...” or “For some z...”.

Intuitively, the sentence 3x P says that P is true for at least one object xz. More precisely, 3x P is true in a given model under a given interpretation if P is true in at least one extended interpretation that assigns x to a domain element. For our example, this means

Section 8.2.

Syntax and Semantics of First-Order Logic 251

that at least one of the following must be true:

Richard the Lionheart is a crown A Richard the Lionheart is on John’s head; King John is a crown A King John is on John’s head;

Richard’s left leg is a crown A Richard’s left leg is on John’s head;

John’s left leg is a crown A John’s left leg is on John’s head;

The crown is a crown A the crown is on John’s head.

The fifth assertion is true in the model, so the original existentially quantified sentence is true in the model. Notice that, by our definition, the sentence would also be true in a model in which King John was wearing two crowns. This is entirely consistent with the original sentence “King John has a crown on his head.” 6

Just as = appears to be the natural connective to use with V, A is the natural connective to use with 3. Using A as the main connective with V led to an overly strong statement in the example in the previous section; using = with 3 usually leads to a very weak statement, indeed. Consider the following sentence:

dz Crown(z) = OnHead(z, John) . On the surface, this might look like a reasonable rendition of our sentence. Applying the semantics, we see that the sentence says that at least one of the following assertions is true:

Richard the Lionheart is a crown => Richard the Lionheart is on John’s head; King John is a crown = King John is on John’s head; Richard’s left leg is a crown = Richard’s left leg is on John’s head;

and so on. Now an implication is true if both premise and conclusion are true, or if its premise is false. So if Richard the Lionheart is not a crown, then the first assertion is true and the existential is satisfied. So, an existentially quantified implication sentence is true in any model containing an object for which the premise of the implication is false; hence such sentences really do not say much at all.

Nested quantifiers

We will often want to express more complex sentences using multiple quantifiers. The simplest case is where the quantifiers are of the same type. For example, “Brothers are siblings” can be written as

Va Vy Brother(z,y) = Sibling(z,y) .

Consecutive quantifiers of the same type can be written as one quantifier with several variables. For example, to say that siblinghood is a symmetric relationship, we can write

Vz,y Sibling(z,y) < Sibling(y,x) .

In other cases we will have mixtures. “Everybody loves somebody” means that for every person, there is someone that person loves:

Vz Jy Loves(z,y) .

6 There is a variant of the existential quantifier, usually written 3' or 3!, that means “There exists exactly one.” The same meaning can be expressed using equality statements, as we show in Section 8.2.

252

Chapter 8. First-Order Logic

On the other hand, to say “There is someone who is loved by everyone,” we write Jy Vz Loves(z,y).

The order of quantification is therefore very important. It becomes clearer if we insert parentheses. Vx (Jy Loves(x,y)) says that everyone has a particular property, namely, the property that somebody loves them. On the other hand, 3 (Vy Loves(z,y)) says that someone in the world has a particular property, namely the property of being loved by everybody.

Some confusion can arise when two quantifiers are used with the same variable name. Consider the sentence

Va [Crown(z)V (3x Brother(Richard,z))] .

Here the = in Brother(Richard, x) is existentially quantified. The rule is that the variable belongs to the innermost quantifier that mentions it; then it will not be subject to any other quantification.” Another way to think of it is this: 3x Brother(Richard, zr) is a sentence about Richard (that he has a brother), not about z; so putting a V x outside it has no effect. It could equally well have been written 3 z Brother( Richard, z). Because this can be a source of confusion, we will always use different variables.

Connections between V and 3

The two quantifiers are actually intimately connected with each other, through negation. Asserting that everyone dislikes parsnips is the same as asserting there does not exist someone who likes them, and vice versa:

Vx —Likes(x, Parsnips) isequivalentto —3x Likes(z, Parsnips) .

We can go one step further: “Everyone likes ice cream” means that there is no one who does not like ice cream:

Va Likes(z, IceCream) is equivalentto —3z -Likes(z, IceCream) .

Because V is really a conjunction over the universe of objects and 3 is a disjunction, it should not be surprising that they obey De Morgan’s rules. The De Morgan rules for quantified and unquantified sentences are as follows:

Ve =P = -dz P “PA=-Q = —(PVQ) =v P = Jz =P “(PAQ) = =PV-Q

Ve P = dz =P PAQ = “(=P V -Q) dz P = -Vz =P Pva@ = (=P AQ).

Thus, we do not really need both ¥ and 3, just as we do not really need both A and Vv. Still, readability is more important than parsimony, so we will keep both of the quantifiers.

7 Ttis the potential for interference between quantifiers using the same variable name that motivates the slightly baroque mechanism of extended interpretations in the semantics of quantified sentences. The more intuitively obvious approach of substituting objects for every occurrence of x fails in our example because the x in Brother( Richard, x) would be “captured” by the substitution. Extended interpretations handle this correctly because the inner quantifier’s assignment for z overrides the outer quantifier’s.

Section 8.3.

Using First-Order Logic 253

EQUALITY SYMBOL

Equality

First-order logic includes one more way to make atomic sentences, other than using a predicate and terms as described earlier. We can use the equality symbol to make statements to the effect that two terms refer to the same object. For example,

Father(John) = Henry

says that the object referred to by Father (John) and the object referred to by Henry are the same. Because an interpretation fixes the referent of any term, determining the truth of an equality sentence is simply a matter of seeing that the referents of the two terms are the same object.

The equality symbol can be used to state facts about a given function, as we just did for the Father symbol. It can also be used with negation to insist that two terms are not the same object. To say that Richard has at least two brothers, we would write

dz,y Brother(x, Richard) A Brother (y, Richard) A =(z=1y) . The sentence dz,y Brother(x, Richard) A Brother (y, Richard) ,

does not have the intended meaning. In particular, it is true in the model of Figure 8.2, where Richard has only one brother. To see this, consider the extended interpretation in which both x and y are assigned to King John. The addition of (xz =y) rules out such models. The notation z # vy is sometimes used as an abbreviation for —(z =y).

DOMAINS

ASSERTIONS

Now that we have defined an expressive logical language, it is time to learn how to use it. The best way to do this is through examples. We have seen some simple sentences illustrating the various aspects of logical syntax; in this section, we will provide more systematic representations of some simple domains. In knowledge representation, a domain is just some part of the world about which we wish to express some knowledge.

We will begin with a brief description of the TELL/ASK interface for first-order knowledge bases. Then we will look at the domains of family relationships, numbers, sets, and lists, and at the wumpus world. The next section contains a more substantial example (electronic circuits) and Chapter 10 covers everything in the universe.

Assertions and queries in first-order logic

Sentences are added to a knowledge base using TELL, exactly as in propositional logic. Such sentences are called assertions. For example, we can assert that John is a king and that kings are persons:

TELL(KB, King(John)) . TELL(KB, Vz King(z) = Person(x)) .

254

Chapter 8. First-Order Logic

QUERIES GOALS

SUBSTITUTION BINDING LIST

We can ask questions of the knowledge base using ASK. For example, ASK(KB, King(John))

returns true. Questions asked using ASK are called queries or goals (not to be confused with goals as used to describe an agent’s desired states). Generally speaking, any query that is logically entailed by the knowledge base should be answered affirmatively. For example, given the two assertions in the preceding paragraph, the query

ASK(KB, Person(John)) should also return ¢rue. We can also ask quantified queries, such as ASK(KB, 3x Person(z)) .

The answer to this query could be true, but this is neither helpful nor amusing. (It is rather like answering “Can you tell me the time?” with “Yes.”) A query with existential variables is asking “Is there an x such that ...,” and we solve it by providing such an x. The standard form for an answer of this sort is a substitution or binding list, which is a set of variable/term pairs. In this particular case, given just the two assertions, the answer would be {x / John}. If there is more than one possible answer, a list of substitutions can be returned.

The kinship domain

The first example we consider is the domain of family relationships, or kinship. This domain includes facts such as “Elizabeth is the mother of Charles” and “Charles is the father of William” and rules such as “One’s grandmother is the mother of one’s parent.”

Clearly, the objects in our domain are people. We will have two unary predicates, Male and Female. Kinship relations—parenthood, brotherhood, marriage, and so on—will be represented by binary predicates: Parent, Sibling, Brother, Sister, Child, Daughter, Son, Spouse, Wife, Husband, Grandparent, Grandchild, Cousin, Aunt, and Uncle. We will use functions for Mother and Father, because every person has exactly one of each of these (at least according to nature’s design).

We can go through each function and predicate, writing down what we know in terms of the other symbols. For example, one’s mother is one’s female parent:

Vm, c Mother(c)=m < Female(m) A Parent(m,c) . One’s husband is one’s male spouse:

Vw, h Husband(h,w) < Male(h) A Spouse(h,w) . Male and female are disjoint categories:

Vz Male(z) & —Female(z). Parent and child are inverse relations:

Vp,c Parent(p,c) & Child(c,p) . A grandparent is a parent of one’s parent:

Vg,¢ Grandparent(g,c¢) < Ip Parent(g,p) A Parent(p,c) .

Section 8.3.

Using First-Order Logic 255

AXIOM

DEFINITION

THEOREM

A sibling is another child of one’s parents: Va,y Sibling(z,y) < = # yA3Ip Parent(p,x) A Parent(p,y) .

We could go on for several more pages like this, and Exercise 8.11 asks you to do just that.

Each of these sentences can be viewed as an axiom of the kinship domain. Axioms are commonly associated with purely mathematical domains—we will see some axioms for numbers shortly—but they are needed in all domains. They provide the basic factual information from which useful conclusions can be derived. Our kinship axioms are also definitions; they have the form Vz,y P(z,y) < .... The axioms define the Mother function and the Husband, Male, Parent, Grandparent, and Sibling predicates in terms of other predicates. Our definitions “bottom out” at a basic set of predicates (Child, Spouse, and Female) in terms of which the others are ultimately defined. This is a very natural way in which to build up the representation of a domain, and it is analogous to the way in which software packages are built up by successive definitions of subroutines from primitive library functions. Notice that there is not necessarily a unique set of primitive predicates; we could equally well have used Parent, Spouse, and Male. In some domains, as we will see, there is no clearly identifiable basic set.

Not all logical sentences about a domain are axioms. Some are theorems—that is, they are entailed by the axioms. For example, consider the assertion that siblinghood is symmetric:

Vz,y Sibling(z,y) < Sibling(y, x).

Is this an axiom or a theorem? In fact, it is a theorem that follows logically from the axiom that defines siblinghood. If we ASK the knowledge base this sentence, it should return true.

From a purely logical point of view, a knowledge base need contain only axioms and no theorems, because the theorems do not increase the set of conclusions that follow from the knowledge base. From a practical point of view, theorems are essential to reduce the computational cost of deriving new sentences. Without them, a reasoning system has to start from first principles every time, rather like a physicist having to rederive the rules of calculus for every new problem.

Not all axioms are definitions. Some provide more general information about certain predicates without constituting a definition. Indeed, some predicates have no complete definition because we do not know enough to characterize them fully. For example, there is no obvious way to complete the sentence:

Vz Person(z)

Fortunately, first-order logic allows us to make use of the Person predicate without completely defining it. Instead, we can write partial specifications of properties that every person has and properties that make something a person:

Va Person(z) = ... Ve ... = Person(z).

Axioms can also be “just plain facts,” such as Male(Jim) and Spouse(Jim, Laura). Such facts form the descriptions of specific problem instances, enabling specific questions to be answered. The answers to these questions will then be theorems that follow from the

256

Chapter 8. First-Order Logic

NATURAL NUMBERS

PEANO AXIOMS

INFIX PREFIX

SYNTACTIC SUGAR

SETS

axioms. Often, one finds that the expected answers are not forthcoming—for example, from Male( George) and Spouse( George, Laura), one expects to be able to infer Female( Laura); but this does not follow from the axioms given earlier. This is a sign that an axiom is missing. Exercise 8.8 asks you to supply it.

Numbers, sets, and lists

Numbers are perhaps the most vivid example of how a large theory can be built up from a tiny kernel of axioms. We will describe here the theory of natural numbers or nonnegative integers. We need a predicate NatNum that will be true of natural numbers; we need one constant symbol, 0; and we need one function symbol, S (successor). The Peano axioms define natural numbers and addition.® Natural numbers are defined recursively:

NatNum(0) . Vn NatNum(n) = NatNum(S(n)) .

That is, 0 is a natural number, and for every object n, if n is a natural number then S(n) is a natural number. So the natural numbers are 0, S(0), S(S(0)), and so on. We also need axioms to constrain the successor function:

Vn 0# S(n).

Vm,n m#n = S(m)# S(n). Now we can define addition in terms of the successor function:

Vm NatNum(m) = + (0,m)=m.

Vm,n NatNum(m) A NatNum(n) = + (S(m),n) = S(+(m,n)).

The first of these axioms says that adding O to any natural number m gives m itself. Notice the use of the binary function symbol “+” in the term +(m, 0); in ordinary mathematics, the term would be written m + 0 using infix notation. (The notation we have used for first-order logic is called prefix.) To make our sentences about numbers easier to read, we will allow the use of infix notation. We can also write S(n) as n + 1, so that the second axiom becomes

Vm,n NatNum(m)A NatNum(n) = (m+ 1)+n=(m+n)+1.

This axiom reduces addition to repeated application of the successor function.

The use of infix notation is an example of syntactic sugar, that is, an extension to or abbreviation of the standard syntax that does not change the semantics. Any sentence that uses sugar can be “de-sugared” to produce an equivalent sentence in ordinary first-order logic.

Once we have addition, it is straightforward to define multiplication as repeated addition, exponentiation as repeated multiplication, integer division and remainders, prime numbers, and so on. Thus, the whole of number theory (including cryptography) can be built up from one constant, one function, one predicate and four axioms.

The domain of sets is also fundamental to mathematics as well as to commonsense reasoning. (In fact, it is possible to build number theory on top of set theory.) We want to be able to represent individual sets, including the empty set. We need a way to build up sets by

8 The Peano axioms also include the principle of induction, which is a sentence of second-order logic rather than of first-order logic. The importance of this distinction is explained in Chapter 9.

Section 8.3.

Using First-Order Logic 257

LISTS

adding an element to a set or taking the union or intersection of two sets. We will want to know whether an element is a member of a set and to be able to distinguish sets from objects that are not sets.

We will use the normal vocabulary of set theory as syntactic sugar. The empty set is a constant written as { }. There is one unary predicate, Set, which is true of sets. The binary predicates are z € s (x is a member of set s) and s1 C sg (set s; is a subset, not necessarily proper, of set s2). The binary functions are s; MN sa (the intersection of two sets), $3 U so (the union of two sets), and {z|s} (the set resulting from adjoining element z to set s). One possible set of axioms is as follows:

1. The only sets are the empty set and those made by adjoining something to a set: Vs Set(s) & (s={})V (Iz, s2 Set(s2) As={z|s2}).

2. The empty set has no elements adjoined into it, in other words, there is no way to decompose EmptySet into a smaller set and an element:

-3Jz,s {z|s}={}. 3. Adjoining an element already in the set has no effect: Vz,s z€s & s={z|s}.

4. The only members of a set are the elements that were adjoined into it. We express this recursively, saying that x is a member of s if and only if s is equal to some set so adjoined with some element y, where either y is the same as x or x is a member of sg:

Vz,s z€s & [Fy,s2 (s={yls2} A(z=yVz€Es))]. 5. A set is a subset of another set if and only if all of the first set’s members are members of the second set:

Vsi,80 61 C82 & (Vo z€81 = TES).

6. Two sets are equal if and only if each is a subset of the other: Vsi,s2 (s1=s82) & (s1 Csa Asa Cost).

7. An object is in the intersection of two sets if and only if it is a member of both sets: Vi,s1,52 x€(s1Ns2) & (x€s1 ATES).

8. An object is in the union of two sets if and only if it is a member of either set: Va,81,82 z€(s1Us2) & (zE€s1VTES).

Lists are similar to sets. The differences are that lists are ordered and the same element can appear more than once in a list. We can use the vocabulary of Lisp for lists: Nil is the constant list with no elements; Cons, Append, First, and Rest are functions; and Find is the predicate that does for lists what Member does for sets. List? is a predicate that is true only of lists. As with sets, it is common to use syntactic sugar in logical sentences involving lists. The empty list is []. The term Cons(z,y), where y is a nonempty list, is written [z|y]. The term Cons(z, Nil), (i.e., the list containing the element x), is written as [z]. A list of several elements, such as [A, B,C], corresponds to the nested term Cons(A, Cons(B, Cons(C, Nil))). Exercise 8.14 asks you to write out the axioms for lists.

258

Chapter 8. First-Order Logic

SYNCHRONIC

DIACHRONIC

The wumpus world

Some propositional logic axioms for the wumpus world were given in Chapter 7. The firstorder axioms in this section are much more concise, capturing in a very natural way exactly what we want to say.

Recall that the wumpus agent receives a percept vector with five elements. The corresponding first-order sentence stored in the knowledge base must include both the percept and the time at which it occurred; otherwise the agent will get confused about when it saw what. We will use integers for time steps. A typical percept sentence would be

Percept (Stench, Breeze, Glitter, None, None], 5) .

Here, Percept is a binary predicate and Stench and so on are constants placed in a list. The actions in the wumpus world can be represented by logical terms:

Turn(Right), Turn(Left), Forward, Shoot, Grab, Release, Climb . To determine which is best, the agent program constructs a query such as Ja BestAction(a,5) .

ASK should solve this query and return a binding list such as {a/Grab}. The agent program can then return Grab as the action to take, but first it must TELL its own knowledge base that it is performing a Grab.

The raw percept data implies certain facts about the current state. For example:

Vt, s,g,m,c Percept([s, Breeze,g,m,c|,t) = Breeze(t), Vt, s,b,m,c Percept([s,b, Glitter, m,c],t) = Glitter(t) ,

and so on. These rules exhibit a trivial form of the reasoning process called perception, which we study in depth in Chapter 24. Notice the quantification over time ¢. In propositional logic, we would need copies of each sentence for each time step.

Simple “reflex” behavior can also be implemented by quantified implication sentences. For example, we have

Vt Glitter(t) = BestAction(Grab,t) .

Given the percept and rules from the preceding paragraphs, this would yield the desired conclusion BestAction(Grab,5)—that is, Grab is the right thing to do. Notice the correspondence between this rule and the direct percept—action connection in the circuit-based agent in Figure 7.20; the circuit connection implicitly quantifies over time.

So far in this section, the sentences dealing with time have been synchronic (“same time”) sentences, that is, they relate properties of a world state to other properties of the same world state. Sentences that allow reasoning “across time” are called diachronic; for example, the agent needs to know how to combine information about its previous location with information about the action just taken in order to determine its current location. We will defer discussion of diachronic sentences until Chapter 10; for now, just assume that the required inferences have been made for location and other time-dependent predicates.

We have represented the percepts and actions; now it is time to represent the environment itself. Let us begin with objects. Obvious candidates are squares, pits, and the wumpus.

Section 8.3.

Using First-Order Logic 259

DIAGNOSTIC RULES

We could name each square—Square; o and so on—but then the fact that Square; » and Square 3 are adjacent would have to be an “extra” fact, and we would need one such fact for each pair of squares. It is better to use a complex term in which the row and column appear as integers; for example, we can simply use the list term [1, 2]. Adjacency of any two squares can be defined as

Vz,y,a,b Adjacent([z,y],[a,b])

[a,b] {lz + 1,9], [x — Ly], [x,y + 1], [x,y — 1]} .

We could also name each pit, but this would be inappropriate for a different reason: there is no reason to distinguish among the pits.® It is much simpler to use a unary predicate Pit that is true of squares containing pits. Finally, since there is exactly one wumpus, a constant Wumpus is just as good as a unary predicate (and perhaps more dignified from the wumpus’s viewpoint). The wumpus lives in exactly one square, so it is a good idea to use a function such as Home( Wumpus) to name that square. This completely avoids the cumbersome set of sentences required in propositional logic to say that exactly one square contains a wumpus. (It would be even worse for propositional logic with two wumpuses.)

The agent’s location changes over time, so we will write At( Agent, s,t) to mean that the agent is at square s at time ¢. Given its current location, the agent can infer properties of the square from properties of its current percept. For example, if the agent is at a square and perceives a breeze, then that square is breezy:

Vs,t At(Agent,s,t) \ Breeze(t) = Breezy(s).

It is useful to know that a square is breezy because we know that the pits cannot move about. Notice that Breezy has no time argument.

Having discovered which places are breezy (or smelly) and, very importantly, not breezy (or not smelly), the agent can deduce where the pits are (and where the wumpus is). There are two kinds of synchronic rules that could allow such deductions:

<> Diagnostic rules: Diagnostic rules lead from observed effects to hidden causes. For finding pits, the obvious diagnostic rules say that if a square is breezy, some adjacent square must contain a pit, or

Vs Breezy(s) = Ir Adjacent(r,s) A Pit(r) ,

and that if a square is not breezy, no adjacent square contains a pit: 1

V's = Breezy(s) = —-3r Adjacent(r,s) A Pit(r) . Combining these two, we obtain the biconditional sentence

Vs Breezy(s) « Ir Adjacent(r,s) A Pit(r) . (8.3)

9 Similarly, most of us do not name each bird that flies overhead as it migrates to warmer regions in winter. An ornithologist wishing to study migration patterns, survival rates, and so on does name each bird, by means of a ring on its leg, because individual birds must be tracked.

10 There is a natural human tendency to forget to write down negative information such as this. In conversation, this tendency is entirely normal—it would be strange to say “There are two cups on the table and there are not three or more,” even though “There are two cups on the table” is, strictly speaking, still true when there are three. We will return to this topic in Chapter 10.

260

Chapter 8. First-Order Logic

CAUSAL RULES

MODEL-BASED REASONING

{> Causal rules: Causal rules reflect the assumed direction of causality in the world: some hidden property of the world causes certain percepts to be generated. For example, a pit causes all adjacent squares to be breezy:

Vr Pit(r) = [Vs Adjacent(r,s) = Breezy(s)] and if all squares adjacent to a given square are pitless, the square will not be breezy: Vs [Vr Adjacent(r,s) = =Pit(r)] = -Breezy(s).

With some work, it is possible to show that these two sentences together are logically equivalent to the biconditional sentence in Equation (8.3). The biconditional itself can also be thought of as causal, because it states how the truth value of Breezy is generated from the world state.

Systems that reason with causal rules are called model-based reasoning systems, because the causal rules form a model of how the environment operates. The distinction between model-based and diagnostic reasoning is important in many areas of Al. Medical diagnosis in particular has been an active area of research, in which approaches based on direct associations between symptoms and diseases (a diagnostic approach) have gradually been replaced by approaches using an explicit model of the disease process and how it manifests itself in symptoms. The issues come up again in Chapter 13.

Whichever kind of representation the agent uses, if the axioms correctly and completely describe the way the world works and the way that percepts are produced, then any complete logical inference procedure will infer the strongest possible description of the world state, given the available percepts. Thus, the agent designer can concentrate on getting the knowledge right, without worrying too much about the processes of deduction. Furthermore, we have seen that first-order logic can represent the wumpus world no less concisely than the original English-language description given in Chapter 7.

KNOWLEDGE ENGINEERING

The preceding section illustrated the use of first-order logic to represent knowledge in three simple domains. This section describes the general process of knowledge base construction— a process called knowledge engineering. A knowledge engineer is someone who investigates a particular domain, learns what concepts are important in that domain, and creates a formal representation of the objects and relations in the domain. We will illustrate the knowledge engineering process in an electronic circuit domain that should already be fairly familiar, so that we can concentrate on the representational issues involved. The approach we will take is suitable for developing special-purpose knowledge bases whose domain is carefully circumscribed and whose range of queries is known in advance. General-purpose knowledge bases, which are intended to support queries across the full range of human knowledge, are discussed in Chapter 10.

Section 8.4. Knowledge Engineering in First-Order Logic 261

The knowledge engineering process

Knowledge engineering projects vary widely in content, scope, and difficulty, but all such projects include the following steps:

1. Identify the task. The knowledge engineer must delineate the range of questions that the knowledge base will support and the kinds of facts that will be available for each specific problem instance. For example, does the wumpus knowledge base need to be able to choose actions or is it required to answer questions only about the contents of the environment? Will the sensor facts include the current location? The task will determine what knowledge must be represented in order to connect problem instances to answers. This step is analogous to the PEAS process for designing agents in Chapter 2.

2. Assemble the relevant knowledge. The knowledge engineer might already be an expert in the domain, or might need to work with real experts to extract what they know—a RNONIEDSE process called knowledge acquisition. At this stage, the knowledge is not represented formally. The idea is to understand the scope of the knowledge base, as determined by the task, and to understand how the domain actually works.

For the wumpus world, which is defined by an artificial set of rules, the relevant knowledge is easy to identify. (Notice, however, that the definition of adjacency was not supplied explicitly in the wumpus-world rules.) For real domains, the issue of relevance can be quite difficult—for example, a system for simulating VLSI designs might or might not need to take into account stray capacitances and skin effects.

3. Decide on a vocabulary of predicates, functions, and constants. That is, translate the important domain-level concepts into logic-level names. This involves many questions of knowledge engineering style. Like programming style, this can have a significant impact on the eventual success of the project. For example, should pits be represented by objects or by a unary predicate on squares? Should the agent’s orientation be a function or a predicate? Should the wumpus’s location depend on time? Once the

ONTOLOGY choices have been made, the result is a vocabulary that is known as the ontology of the domain. The word ontology means a particular theory of the nature of being or existence. The ontology determines what kinds of things exist, but does not determine their specific properties and interrelationships.

4. Encode general knowledge about the domain. The knowledge engineer writes down the axioms for all the vocabulary terms. This pins down (to the extent possible) the meaning of the terms, enabling the expert to check the content. Often, this step reveals misconceptions or gaps in the vocabulary that must be fixed by returning to step 3 and iterating through the process.

5. Encode a description of the specific problem instance. If the ontology is well thought out, this step will be easy. It will involve writing simple atomic sentences about instances of concepts that are already part of the ontology. For a logical agent, problem instances are supplied by the sensors, whereas a “disembodied” knowledge base is supplied with additional sentences in the same way that traditional programs are supplied with input data.

262

Chapter 8. First-Order Logic

6. Pose queries to the inference procedure and get answers. This is where the reward is: we can let the inference procedure operate on the axioms and problem-specific facts to derive the facts we are interested in knowing.

7. Debug the knowledge base. Alas, the answers to queries will seldom be correct on the first try. More precisely, the answers will be correct for the knowledge base as written, assuming that the inference procedure is sound, but they will not be the ones that the user is expecting. For example, if an axiom is missing, some queries will not be answerable from the knowledge base. A considerable debugging process could ensue. Missing axioms or axioms that are too weak can be identified easily by noticing places where the chain of reasoning stops unexpectedly. For example, if the knowledge base includes one of the diagnostic axioms for pits,

V's Breezy(s) = 3r Adjacent(r,s) A Pit(r),

but not the other, then the agent will never be able to prove the absence of pits. Incorrect axioms can be identified because they are false statements about the world. For example, the sentence

Vz NumOfLegs(z,4) = Mammal(z)

is false for reptiles, amphibians, and, more important, tables. The falsehood of this sentence can be determined independently of the rest of the knowledge base. In contrast, a typical error in a program looks like this:

offset = position + 1.

It is impossible to tell whether this statement is correct without looking at the rest of the program to see whether, for example, of fset is used to refer to the current position, or to one beyond the current position, or whether the value of position is changed by another statement and so of £set should also be changed again.

To understand this seven-step process better, we now apply it to an extended example—the domain of electronic circuits.

The electronic circuits domain

We will develop an ontology and knowledge base that allow us to reason about digital circuits of the kind shown in Figure 8.4. We follow the seven-step process for knowledge engineering.

Identify the task

There are many reasoning tasks associated with digital circuits. At the highest level, one analyzes the circuit’s functionality. For example, does the circuit in Figure 8.4 actually add properly? If all the inputs are high, what is the output of gate A2? Questions about the circuit’s structure are also interesting. For example, what are all the gates connected to the first input terminal? Does the circuit contain feedback loops? These will be our tasks in this section. There are more detailed levels of analysis, including those related to timing delays, circuit area, power consumption, production cost, and so on. Each of these levels would require additional knowledge.

Section 8.4.

Knowledge Engineering in First-Order Logic 263

[]

Figure 8.4 A digital circuit C1, purporting to be a one-bit full adder. The first two inputs are the two bits to be added and the third input is a carry bit. The first output is the sum, and the second output is a carry bit for the next adder. The circuit contains two XOR gates, two AND gates and one OR gate.

Assemble the relevant knowledge

What do we know about digital circuits? For our purposes, they are composed of wires and gates. Signals flow along wires to the input terminals of gates, and each gate produces a signal on the output terminal that flows along another wire. To determine what these signals will be, we need to know how the gates transform their input signals. There are four types of gates: AND, OR, and XOR gates have two input terminals, and NOT gates have one. All gates have one output terminal. Circuits, like gates, have input and output terminals.

To reason about functionality and connectivity, we do not need to talk about the wires themselves, the paths the wires take, or the junctions where two wires come together. All that matters is the connections between terminals—we can say that one output terminal is connected to another input terminal without having to mention the wire that actually connects them. There are many other factors of the domain that are irrelevant to our analysis, such as the size, shape, color, or cost of the various components.

If our purpose were something other than verifying designs at the gate level, the ontology would be different. For example, if we were interested in debugging faulty circuits, then it would probably be a good idea to include the wires in the ontology, because a faulty wire can corrupt the signal flowing along it. For resolving timing faults, we would need to include gate delays. If we were interested in designing a product that would be profitable, then the cost of the circuit and its speed relative to other products on the market would be important.

Decide on a vocabulary

We now know that we want to talk about circuits, terminals, signals, and gates. The next step is to choose functions, predicates, and constants to represent them. We will start from individual gates and move up to circuits.

First, we need to be able to distinguish a gate from other gates. This is handled by naming gates with constants: Xj, Xo, and so on. Although each gate is connected into the circuit in its own individual way, its behavior—the way it transforms input signals into output

264

Chapter 8. First-Order Logic

signals—depends only on its rype. We can use a function to refer to the type of the gate.!! For example, we can write Type(X1)= XOR. This introduces the constant XOR for a particular type of gate; the other constants will be called OR, AND, and NOT. The Type function is not the only way to encode the ontological distinction. We could have used a binary predicate, Type(X1, XOR), or several individual type predicates, such as XOR(X). Either of these choices would work fine, but by choosing the function Type, we avoid the need for an axiom which says that each individual gate can have only one type. The semantics of functions already guarantees this.

Next we consider terminals. A gate or circuit can have one or more input terminals and one or more output terminals. We could simply name each one with a constant, just as we named gates. Thus, gate X; could have terminals named X In, X; Ing, and X Out. The tendency to generate long compound names should be avoided, however. Calling something X1In; does not make it the first input of X; we would still need to say this using an explicit assertion. It is probably better to name the gate using a function, just as we named King John’s left leg LeftLeg(John). Thus, let In(1, X1) denote the first input terminal for gate X1. A similar function Out is used for output terminals.

The connectivity between gates can be represented by the predicate Connected, which takes two terminals as arguments, as in Connected(Out(1, X1), In(1, X2)).

Finally, we need to know whether a signal is on or off. One possibility is to use a unary predicate, On, which is true when the signal at a terminal is on. This makes it a little difficult, however, to pose questions such as “What are all the possible values of the signals at the output terminals of circuit C1 7” We will therefore introduce as objects two “signal values” 1 and 0, and a function Signal that takes a terminal as argument and denotes the signal value for that terminal.

Encode general knowledge of the domain

One sign that we have a good ontology is that there are very few general rules which need to be specified. A sign that we have a good vocabulary is that each rule can be stated clearly and concisely. With our example, we need only seven simple rules to describe everything we need to know about circuits:

1. If two terminals are connected, then they have the same signal: Viti, ta Connected (ti, ts) = Signal(t))= Signal(ts) 2. The signal at every terminal is either 1 or O (but not both): Vt Signal(t)=1V Signal(t)=0 1#0 3. Connected is a commutative predicate: Viti, ta Connected(t1,t2) < Connected(ta, tr) 4. An OR gate’s output is 1 if and only if any of its inputs is 1: Vg Type(g)=OR = Signal(Out(1,9))=1 < In Signal(In(n,g))=1

11 Note that we have used names beginning with appropriate letters—A,, X1, and so on—purely to make the example easier to read. The knowledge base must still contain type information for the gates.

Knowledge Engineering in First-Order Logic 265

CIRCUIT VERIFICATION

5. An AND gate’s output is 0 if and only if any of its inputs is 0:

Vg Type(g)=AND =

Signal(Out(1,9))=0 & In Signal(In(n,g))=0

6. An XOR gate’s output is 1 if and only if its inputs are different:

Vg Type(g)=XOR =

Signal(Out(1,g))=1 < Signal(In(1,g)) # Signal(In(2,g))

7. A NOT gate’s output is different from its input:

Vg (Type(g)=NOT) = Signal(Out(1,g)) # Signal(In(1,q))

Encode the specific problem instance

The circuit shown in Figure 8.4 is encoded as circuit C'; with the following description. First, we categorize the gates:

Type(X1)=XOR Type(X2)= XOR Type(A1)= AND Type(As)= AND Type(O1) = OR

Then, we show the connections between them:

Connected (Out (1, X1), In(1, X2)) Connected (Out(1, X1), In(2, Az)) Connected(Out(1, As), In(1,01)) Connected ) );

Connected (In(1

(Out(1. (In(1

(Out( ) (In(2,C Connected(Out(1, Ay), In(2,01)) Connected(In(2,Cy (Out(1, (In(3,C

(Out( (In(3,C

Connected

Connected(Out(1, X2), Out(1, oo Connected Connected(Out(1, 01), Out(2,C1)) Connected

Pose queries to the inference procedure

What combinations of inputs would cause the first output of C'; (the sum bit) to be 0 and the second output of C'; (the carry bit) to be 1? 3 1, 12, i3 Signal (In(1, C1) = 2 A Signal(In(2, C1) = i9 A Signal(In(3, Ch)) = 13 A Signal (Out(1,C1)) =0 A Signal(Out(2,C4)) =1 The answers are substitutions for the variables iy, ia, and i3 such that the resulting sentence is entailed by the knowledge base. There are three such substitutions:

{i1/1, i2/1, 43/0} {i1/1, i2/0, is/1} {i1/0, i2/1, i3/1} . What are the possible sets of values of all the terminals for the adder circuit?

4 2 ; 12, 13, 01,02 Signal(In(1, 1) = 1 A Signal (In(2, C1) = 12

A Signal(In(3,C1)) = 13 A Signal (Out(1,C1)) = 01 A Signal(Out(2,C1)) = 02 .

This final query will return a complete input—output table for the device, which can be used to check that it does in fact add its inputs correctly. This is a simple example of circuit verification. We can also use the definition of the circuit to build larger digital systems, for which the same kind of verification procedure can be carried out. (See Exercise 8.17.) Many domains are amenable to the same kind of structured knowledge-base development, in which more complex concepts are defined on top of simpler concepts.

266

Chapter 8. First-Order Logic

Debug the knowledge base

We can perturb the knowledge base in various ways to see what kinds of erroneous behaviors emerge. For example, suppose we omit the assertion that 1 # 0.2 Suddenly, the system will be unable to prove any outputs for the circuit, except for the input cases 000 and 110. We can pinpoint the problem by asking for the outputs of each gate. For example, we can ask

Ji1,49,0 Signal(In(1,Cy)) =i; A Signal(In(2,Cy)) =i2 A Signal(Out(1, X1)) which reveals that no outputs are known at X for the input cases 10 and 01. Then, we look at the axiom for XOR gates, as applied to X;:

Signal(Out(1,X1))=1 & Signal(In(1,X,)) # Signal(In(2, X1)) . If the inputs are known to be, say, 1 and 0, then this reduces to

Signal(Out(1,X1))=1 & 1#0.

Now the problem is apparent: the system is unable to infer that Signal( Out(1, X1)) =1, so we need to tell it that 1 # 0.

This chapter has introduced first-order logic, a representation language that is far more powerful than propositional logic. The important points are as follows:

e Knowledge representation languages should be declarative, compositional, expressive, context-independent, and unambiguous.

e Logics differ in their ontological commitments and epistemological commitments. While propositional logic commits only to the existence of facts, first-order logic commits to the existence of objects and relations and thereby gains expressive power.

\* A possible world, or model, for first-order logic is defined by a set of objects, the relations among them, and the functions that can be applied to them.

o Constant symbols name objects, predicate symbols name relations, and function symbols name functions. An interpretation specifies a mapping from symbols to the model. Complex terms apply function symbols to terms to name an object. Given an interpretation and a model, the truth of a sentence is determined.

e An atomic sentence consists of a predicate applied to one or more terms; it is true just when the relation named by the predicate holds between the objects named by the terms. Complex sentences use connectives just like propositional logic, and quantified sentences allow the expression of general rules.

» Developing a knowledge base in first-order logic requires a careful process of analyzing the domain, choosing a vocabulary, and encoding the axioms required to support the desired inferences.

12 This kind of omission is quite common because humans typically assume that different names refer to different things. Logic programming systems, described in Chapter 9, also make this assumption.

Section 8.5. Summary 267

BIBLIOGRAPHICAL AND HISTORICAL NOTES

Although even Aristotle’s logic deals with generalizations over objects, true first-order logic dates from the introduction of quantifiers in Gottlob Frege’s (1879) Begriffschrift (“Concept Writing” or “Conceptual Notation”). Frege's ability to nest quantifiers was a big step forward, but he used an awkward notation. (An example appears on the front cover of this book.) The present notation for first-order logic is due substantially to Giuseppe Peano (1889), but the semantics is virtually identical to Frege’s. Oddly enough, Peano’s axioms were due in large measure to Grassmann (1861) and Dedekind (1888).

A major barrier to the development of first-order logic had been the concentration on one-place predicates to the exclusion of many-place relational predicates. This fixation on one-place predicates had been nearly universal in logical systems from Aristotle up to and including Boole. The first systematic treatment of relations was given by Augustus De Morgan (1864), who cited the following example to show the sorts of inferences that Aristotle’s logic could not handle: “All horses are animals; therefore, the head of a horse is the head of an animal.” This inference is inaccessible to Aristotle because any valid rule that can support this inference must first analyze the sentence using the two-place predicate “z is the head of 4.” The logic of relations was studied in depth by Charles Sanders Peirce (1870), who also developed first-order logic independently of Frege, although slightly later (Peirce, 1883).

Leopold Lowenheim (1915) gave a systematic treatment of model theory for first-order logic in 1915. This paper also treated the equality symbol as an integral part of logic. Lowenheim’s results were further extended by Thoralf Skolem (1920). Alfred Tarski (1935, 1956) gave an explicit definition of truth and model-theoretic satisfaction in first-order logic, using set theory.

McCarthy (1958) was primarily responsible for the introduction of first-order logic as a tool for building AI systems. The prospects for logic-based Al were advanced significantly by Robinson’s (1965) development of resolution, a complete procedure for first-order inference described in Chapter 9. The logicist approach took root at Stanford. Cordell Green (1969a, 1969b) developed a first-order reasoning system, QA3, leading to the first attempts to build a logical robot at SRI (Fikes and Nilsson, 1971). First-order logic was applied by Zohar Manna and Richard Waldinger (1971) for reasoning about programs and later by Michael Genesereth (1984) for reasoning about circuits. In Europe, logic programming (a restricted form of first-order reasoning) was developed for linguistic analysis (Colmerauer ef al., 1973) and for general declarative systems (Kowalski, 1974). Computational logic was also well entrenched at Edinburgh through the LCF (Logic for Computable Functions) project (Gordon et al., 1979). These developments are chronicled further in Chapters 9 and 10.

There are a number of good modern introductory texts on first-order logic. Quine (1982) is one of the most readable. Enderton (1972) gives a more mathematically oriented perspective. A highly formal treatment of first-order logic, along with many more advanced topics in logic, is provided by Bell and Machover (1977). Manna and Waldinger (1985) give a readable introduction to logic from a computer science perspective. Gallier (1986) provides an extremely rigorous mathematical exposition of first-order logic, along with a great deal

268

Chapter 8. First-Order Logic

of material on its use in automated reasoning. Logical Foundations of Artificial Intelligence (Genesereth and Nilsson, 1987) provides both a solid introduction to logic and the first systematic treatment of logical agents with percepts and actions.

EXERCISES

8.1 A logical knowledge base represents the world using a set of sentences with no explicit structure. An analogical representation, on the other hand, has physical structure that corresponds directly to the structure of the thing represented. Consider a road map of your country as an analogical representation of facts about the country. The two-dimensional structure of the map corresponds to the two-dimensional surface of the area.

a. Give five examples of symbols in the map language.

b. An explicit sentence is a sentence that the creator of the representation actually writes down. An implicit sentence is a sentence that results from explicit sentences because of properties of the analogical representation. Give three examples each of implicit and explicit sentences in the map language.

¢. Give three examples of facts about the physical structure of your country that cannot be represented in the map language.

d. Give two examples of facts that are much easier to express in the map language than in first-order logic.

e. Give two other examples of useful analogical representations. What are the advantages and disadvantages of each of these languages?

8.2 Consider a knowledge base containing just two sentences: P(a) and P(b). Does this knowledge base entail Vz P(x)? Explain your answer in terms of models.

8.3 Is the sentence x,y xz =y valid? Explain.

8.4 Write down a logical sentence such that every world in which it is true contains exactly one object.

8.5 Consider a symbol vocabulary that contains ¢ constant symbols, pj, predicate symbols of each arity k, and fj, function symbols of each arity k, where 1 < k < A. Let the domain size be fixed at D. For any given interpretation—model combination, each predicate or function symbol is mapped onto a relation or function, respectively, of the same arity. You may assume that the functions in the model allow some input tuples to have no value for the function (i.e., the value is the invisible object). Derive a formula for the number of possible interpretation model combinations for a domain with D elements. Don’t worry about eliminating redundant combinations.

8.6 Represent the following sentences in first-order logic, using a consistent vocabulary (which you must define):

a. Some students took French in spring 2001.

Section 8.5.

Summary 269

. Every student who takes French passes it.

. Only one student took Greek in spring 2001.

. The best score in Greek is always higher than the best score in French. . Every person who buys a policy is smart.

. No person buys an expensive policy.

. There is an agent who sells policies only to people who are not insured.

Sam oem 0 a =

There is a barber who shaves all men in town who do not shave themselves.

—

. A person born in the UK, each of whose parents is a UK citizen or a UK resident, is a UK citizen by birth.

jA person born outside the UK, one of whose parents is a UK citizen by birth, is a UK citizen by descent.

k. Politicians can fool some of the people all of the time, and they can fool all of the people some of the time, but they can’t fool all of the people all of the time.

8.7 Represent the sentence “All Germans speak the same languages” in predicate calculus. Use Speaks(z, 1), meaning that person x speaks language [.

8.8 What axiom is needed to infer the fact Femnale(Laura) given the facts Male( Jim) and Spouse (Jim, Laura)?

8.9 Write a general set of facts and axioms to represent the assertion “Wellington heard about Napoleon's death” and to correctly answer the question “Did Napoleon hear about Wellingtons death?”

8.10 Rewrite the propositional wumpus world facts from Section 7.5 into first-order logic. How much more compact is this version?

8.11 Write axioms describing the predicates Grand Child, GreatGrandparent, Brother, Sister, Daughter, Son, Aunt, Uncle, BrotherInLaw, SisterInLow, and FirstCousin. Find out the proper definition of mth cousin n times removed, and write the definition in first-order logic.

Now write down the basic facts depicted in the family tree in Figure 8.5. Using a suitable logical reasoning system, TELL it all the sentences you have written down, and ASK it who are Elizabeth’s grandchildren, Diana’s brothers-in-law, and Zara’s great-grandparents.

8.12 Write down a sentence asserting that + is a commutative function. Does your sentence follow from the Peano axioms? If so, explain whys; if not, give a model in which the axioms are true and your sentence is false.

8.13 Explain what is wrong with the following proposed definition of the set membership predicate €:

Vz,s z€{x|s}

Vz,s z€s = Vy xz€{y|s}.

270

Chapter 8. First-Order Logic

George MX Mum

Spencer Md Kydd Elizabeth ) Philip Margaret

Diana Charles Anne Mark Andrew 4 Sarah Edward

William Harry Peter Zara Beatrice Eugenie

>

Figure 8.5 A typical family tree. The symbol “=" connects spouses and arrows point to children.

8.14 Using the set axioms as examples, write axioms for the list domain, including all the constants, functions, and predicates mentioned in the chapter.

8.15 Explain what is wrong with the following proposed definition of adjacent squares in the wumpus world:

Va,y Adjacent(z,y], [z+ 1,y]) A Adjacent([z,y], [z,y + 1])

8.16 Write out the axioms required for reasoning about the wumpus’s location, using a constant symbol Wumpus and a binary predicate In( Wumpus, Location). Remember that there is only one wumpus.

8.17 Extend the vocabulary from Section 8.4 to define addition for 7-bit binary numbers. Then encode the description of the four-bit adder in Figure 8.6, and pose the queries needed to verify that it is in fact correct.

8.18 The circuit representation in the chapter is more detailed than necessary if we care only about circuit functionality. A simpler formulation describes any m-input, n-output gate or circuit using a predicate with m +n arguments, such that the predicate is true exactly when

Xx, 4 Z, 7] Ady X, z XX, X; Xo Y, H Adi

+ 3 hon XY X, Z Ys d 44: RH : 7, Zy 7, 7, Zy X3 yA Ys H A 17,

Figure 8.6 A four-bit adder.

Section 8.5.

Summary 271

the inputs and outputs are consistent. For example, NOT-gates are described by the binary predicate NOT (i, 0), for which NOT(0,1) and NOT(1,0) are known. Compositions of gates are defined by conjunctions of gate predicates in which shared variables indicate direct connections. For example, a NAND circuit can be composed from ANDs and NOT:

Vii, i2,0q,0 NAND(i1,i2,0) <= AND(i1,i2,04) A NOT (04,0) .

Using this representation, define the one-bit adder in Figure 8.4 and the four-bit adder in Figure 8.6, and explain what queries you would use to verify the designs. What kinds of queries are not supported by this representation that are supported by the representation in Section 8.47?

8.19 Obtain a passport application for your country, identify the rules determining eligibility for a passport, and translate them into first-order logic, following the steps outlined in Section 8.4.

INFERENCE IN FIRST-ORDER LOGIC

In which we define effective procedures for answering questions posed in firstorder logic.

Chapter 7 defined the notion of inference and showed how sound and complete inference can be achieved for propositional logic. In this chapter, we extend those results to obtain algorithms that can answer any answerable question stated in first-order logic. This is significant, because more or less anything can be stated in first-order logic if you work hard enough at it.

Section 9.1 introduces inference rules for quantifiers and shows how to reduce firstorder inference to propositional inference, albeit at great expense. Section 9.2 describes the idea of unification, showing how it can be used to construct inference rules that work directly with first-order sentences. We then discuss three major families of first-order inference algorithms: forward chaining and its applications to deductive databases and production systems are covered in Section 9.3; backward chaining and logic programming systems are developed in Section 9.4; and resolution-based theorem-proving systems are described in Section 9.5. In general, one tries to use the most efficient method that can accommodate the facts and axioms that need to be expressed. Reasoning with fully general first-order sentences using resolution is usually less efficient than reasoning with definite clauses using forward or backward chaining.

0.1 PROPOSITIONAL VS. FIRST-ORDER INFERENCE

This section and the next introduce the ideas underlying modern logical inference systems. We begin with some simple inference rules that can be applied to sentences with quantifiers to obtain sentences without quantifiers. These rules lead naturally to the idea that first-order inference can be done by converting the knowledge base to propositional logic and using propositional inference, which we already know how to do. The next section points out an obvious shortcut, leading to inference methods that manipulate first-order sentences directly.

272

Propositional vs. First-Order Inference 273

UNIVERSAL INSTANTIATION

EXISTENTIAL INSTANTIATION

SKOLEM CONSTANT

Inference rules for quantifiers

Let us begin with universal quantifiers. Suppose our knowledge base contains the standard folkloric axiom stating that all greedy kings are evil:

Va King(z) A Greedy(xz) = Ewvil(z). Then it seems quite permissible to infer any of the following sentences:

King(John) \ Greedy(John) = Euvil(John) . King(Richard) N\ Greedy( Richard) = Euvil(Richard) . King(Father(John)) A Greedy(Father(John)) = Evil(Father(John)) .

The rule of Universal Instantiation (UI for short) says that we can infer any sentence obtained by substituting a ground term (a term without variables) for the variable. ! To write out the inference rule formally, we use the notion of substitutions introduced in Section 8.3. Let SUBST(, cv) denote the result of applying the substitution § to the sentence ov. Then the rule is written

Yu a

SuBsT({v/g}, a) for any variable v and ground term ¢g. For example, the three sentences given earlier are obtained with the substitutions {z/ John}, {x / Richard}, and {x /Father(John)}. The corresponding Existential Instantiation rule for the existential quantifier is slightly more complicated. For any sentence «, variable v, and constant symbol k that does not appear elsewhere in the knowledge base,

dv «a SuBsT({v/k},a) For example, from the sentence dz Crown(z) A OnHead(z, John) we can infer the sentence

Crown(Cy) A OnHead (C1, John)

as long as Cy does not appear elsewhere in the knowledge base. Basically, the existential sentence says there is some object satisfying a condition, and the instantiation process is just giving a name to that object. Naturally, that name must not already belong to another object. Mathematics provides a nice example: suppose we discover that there is a number that is a little bigger than 2.71828 and that satisfies the equation d(z¥) /dy = z¥ for z. We can give this number a name, such as e, but it would be a mistake to give it the name of an existing object, such as 7. In logic, the new name is called a Skolem constant. Existential Instantiation is a special case of a more general process called skolemization, which we cover in Section 9.5.

1 Do not confuse these substitutions with the extended interpretations used to define the semantics of quantifiers. The substitution replaces a variable with a term (a piece of syntax) to produce a new sentence, whereas an interpretation maps a variable to an object in the domain.

274

Chapter 9. Inference in First-Order Logic

INFERENTIAL EQUIVALENCE

PROPOSITIONALIZATION

As well as being more complicated than Universal Instantiation, Existential Instantiation plays a slightly different role in inference. Whereas Universal Instantiation can be applied many times to produce many different consequences, Existential Instantiation can be applied once, and then the existentially quantified sentence can be discarded. For example, once we have added the sentence Kill( Murderer, Victim), we no longer need the sentence Jz Kill(x, Victim). Strictly speaking, the new knowledge base is not logically equivalent to the old, but it can be shown to be inferentially equivalent in the sense that it is satisfiable exactly when the original knowledge base is satisfiable.

Reduction to propositional inference

Once we have rules for inferring nonquantified sentences from quantified sentences, it becomes possible to reduce first-order inference to propositional inference. In this section we will give the main ideas; the details are given in Section 9.5.

The first idea is that, just as an existentially quantified sentence can be replaced by one instantiation, a universally quantified sentence can be replaced by the set of all possible instantiations. For example, suppose our knowledge base contains just the sentences

Va King(xz) A Greedy(z) = Fuil(x) King(John)

Greedy( John)

Brother(Richard, John) .

Then we apply UI to the first sentence using all possible ground term substitutions from the vocabulary of the knowledge base—in this case, {x/ John} and {x/ Richard}. We obtain

King(John) N Greedy(John) = Evil(John) , King (Richard) A Greedy(Richard) = Ewvil( Richard) ,

and we discard the universally quantified sentence. Now, the knowledge base is essentially propositional if we view the ground atomic sentences—King(John), Greedy(John), and so on—as proposition symbols. Therefore, we can apply any of the complete propositional algorithms in Chapter 7 to obtain conclusions such as Evil( John).

This technique of propositionalization can be made completely general, as we show in Section 9.5; that is, every first-order knowledge base and query can be propositionalized in such a way that entailment is preserved. Thus, we have a complete decision procedure for entailment . .. or perhaps not. There is a problem: When the knowledge base includes a function symbol, the set of possible ground term substitutions is infinite! For example, if the knowledge base mentions the Father symbol, then infinitely many nested terms such as Father (Father(Father(John))) can be constructed. Our propositional algorithms will have difficulty with an infinitely large set of sentences.

Fortunately, there is a famous theorem due to Jacques Herbrand (1930) to the effect that if a sentence is entailed by the original, first-order knowledge base, then there is a proof involving just a finite subset of the propositionalized knowledge base. Since any such subset has a maximum depth of nesting among its ground terms, we can find the subset by first generating all the instantiations with constant symbols ( Richard and John), then all terms of

©.)

Section 9.2.

Unification and Lifting 275

depth 1 (Father(Richard) and Father(John)), then all terms of depth 2, and so on, until we are able to construct a propositional proof of the entailed sentence.

We have sketched an approach to first-order inference via propositionalization that is complete—that is, any entailed sentence can be proved. This is a major achievement, given that the space of possible models is infinite. On the other hand, we do not know until the proof is done that the sentence is entailed! What happens when the sentence is not entailed? Can we tell? Well, for first-order logic, it turns out that we cannot. Our proof procedure can go on and on, generating more and more deeply nested terms, but we will not know whether it is stuck in a hopeless loop or whether the proof is just about to pop out. This is very much like the halting problem for Turing machines. Alan Turing (1936) and Alonzo Church (1936) both proved, in rather different ways, the inevitability of this state of affairs. The question of entailment for first-order logic is semidecidable—that is, algorithms exist that say yes to every entailed sentence, but no algorithm exists that also says no to every nonentailed sentence.

The preceding section described the understanding of first-order inference that existed up to the early 1960s. The sharp-eyed reader (and certainly the computational logicians of the early 1960s) will have noticed that the propositionalization approach is rather inefficient. For example, given the query Evil(x) and the knowledge base in Equation (9.1), it seems perverse to generate sentences such as King( Richard) A Greedy(Richard) = Evil( Richard). Indeed, the inference of Evil( John) from the sentences

Vz King(z) A Greedy(z) = FEuvil(z) King (John) Greedy(John)

seems completely obvious to a human being. We now show how to make it completely obvious to a computer.

A first-order inference rule

The inference that John is evil works like this: find some z such that x is a king and zx is greedy, and then infer that this x is evil. More generally, if there is some substitution 0 that makes the premise of the implication identical to sentences already in the knowledge base, then we can assert the conclusion of the implication, after applying 6. In this case, the substitution {x /John} achieves that aim.

We can actually make the inference step do even more work. Suppose that instead of knowing Greedy( John), we know that everyone is greedy:

Vy Greedy(y) . (9.2)

Then we would still like to be able to conclude that Ewvil(John), because we know that John is a king (given) and John is greedy (because everyone is greedy). What we need for this to work is find a substitution both for the variables in the implication sentence

276

Chapter 9. Inference in First-Order Logic

GENERALIZED MODUS PONENS

LIFTING

UNIFICATION

and for the variables in the sentences to be matched. In this case, applying the substitution {z/John,y/ John} to the implication premises King(z) and Greedy(x) and the knowledge base sentences King( John) and Greedy(y) will make them identical. Thus, we can infer the conclusion of the implication.

This inference process can be captured as a single inference rule that we call Generalized Modus Ponens: For atomic sentences p;, p;’, and gq, where there is a substitution # such that SUBST(#, p;") = SUBST(0, p;), for all i,

pi’, pos oo ps (MAPA Ap = q) SuBST(f, q) There are n + 1 premises to this rule: the nn atomic sentences p;” and the one implication. The conclusion is the result of applying the substitution € to the consequent g. For our example:

pi’ is King (John) p1 is King(z)

po is Greedy(y) po is Greedy(x)

6 is {«/ John, y/ John} q is Evil(x)

SuUBST(#, q) is Evil(John) . It is easy to show that Generalized Modus Ponens is a sound inference rule. First, we observe that, for any sentence p (whose variables are assumed to be universally quantified) and for any substitution 0,

p |= SUBST(A,p) .

This holds for the same reasons that the Universal Instantiation rule holds. It holds in particular for a # that satisfies the conditions of the Generalized Modus Ponens rule. Thus, from pt'y..., pn we can infer

SUBST(#,p1") A... A SUBST(0, py) and from the implication p; A ... A py, = q we can infer SUBST(#,p1) A... ASUBST(6,p,) = SUBST(0,q) .

Now, 0 in Generalized Modus Ponens is defined so that SUBST(8, p;') = SUBST(6, p;), for all i; therefore the first of these two sentences matches the premise of the second exactly. Hence, SuBST(, gq) follows by Modus Ponens.

Generalized Modus Ponens is a lifted version of Modus Ponens—it raises Modus Ponens from propositional to first-order logic. We will see in the rest of the chapter that we can develop lifted versions of the forward chaining, backward chaining, and resolution algorithms introduced in Chapter 7. The key advantage of lifted inference rules over propositionalization is that they make only those substitutions which are required to allow particular inferences to proceed. One potentially confusing point is that in one sense Generalized Modus Ponens is less general than Modus Ponens (page 211): Modus Ponens allows any single « on the left-hand side of the implication, while Generalized Modus Ponens requires a special format for this sentence. It is generalized in the sense that it allows any number of P.

Unification

Lifted inference rules require finding substitutions that make different logical expressions look identical. This process is called unification and is a key component of all first-order

Section 9.2.

Unification and Lifting 277

UNIFIER

STANDARDIZING APART

MOST GENERAL UNIFIER

OCCUR CHECK

inference algorithms. The UNIFY algorithm takes two sentences and returns a unifier for them if one exists:

UNIFY(p, q¢) = 6 where SUBST(¢,p) = SUBST(6, q) .

Let us look at some examples of how UNIFY should behave. Suppose we have a query Knows(John, x): whom does John know? Some answers to this query can be found by finding all sentences in the knowledge base that unify with Knows(John, x). Here are the results of unification with four different sentences that might be in the knowledge base.

UNIFY(Knows(John, x), Knows(John, Jane)) = {x/ Jane}

UNIFY (Knows(John, x), Knows(y, Bill)) = {x/Bill,y/ John} UNIFY(Knows(John, zx), Knows(y, Mother(y))) = {y/ John, x/ Mother (John)} UNIFY(Knows(John, x), Knows(x, Elizabeth) = fail .

The last unification fails because x cannot take on the values John and Elizabeth at the same time. Now, remember that Knows (x, Elizabeth) means “Everyone knows Elizabeth,” so we should be able to infer that John knows Elizabeth. The problem arises only because the two sentences happen to use the same variable name, x. The problem can be avoided by standardizing apart one of the two sentences being unified, which means renaming its variables to avoid name clashes. For example, we can rename x in Knows(z, Elizabeth) to 217 (a new variable name) without changing its meaning. Now the unification will work:

UNIFY(Knows(John, x), Knows(z17, Elizabeth)) = {x/ Elizabeth, z17/ John} .

Exercise 9.7 delves further into the need for standardizing apart.

There is one more complication: we said that UNIFY should return a substitution that makes the two arguments look the same. But there could be more than one such unifier. For example, UNIFY (Knows(John, x), Knows(y, z)) could return {y/John,z/z} or {y/John,xz/ John, z/ John}. The first unifier gives Knows(John, z) as the result of unification, whereas the second gives Knows( John, John). The second result could be obtained from the first by an additional substitution {z/John}; we say that the first unifier is more general than the second, because it places fewer restrictions on the values of the variables. It turns out that, for every unifiable pair of expressions, there is a single most general unifier (or MGU) that is unique up to renaming of variables. In this case it is {y/ John, z/z}.

An algorithm for computing most general unifiers is shown in Figure 9.1. The process is very simple: recursively explore the two expressions simultaneously “side by side,” building up a unifier along the way, but failing if two corresponding points in the structures do not match. There is one expensive step: when matching a variable against a complex term, one must check whether the variable itself occurs inside the term; if it does, the match fails because no consistent unifier can be constructed. This so-called occur check makes the complexity of the entire algorithm quadratic in the size of the expressions being unified. Some systems, including all logic programming systems, simply omit the occur check and sometimes make unsound inferences as a result; other systems use more complex algorithms with linear-time complexity.

278

Chapter 9. Inference in First-Order Logic

function UNIFY(z, y, 8) returns a substitution to make z and y identical inputs: z, a variable, constant, list, or compound y, a variable, constant, list, or compound 6, the substitution built up so far (optional, defaults to empty)

if 0 = failure then return failure else if = = y then return 6 else if VARIABLE?(z) then return UNIFY-VAR(z, y, 0) else if VARIABLE?(y) then return UNIFY-VAR(y, z, 0) else if CoMPOUND?(z) and COMPOUND?(y) then return UNTFY(ARGS[z], ARGS[y], UNIFY(OP[z], OP[y], 0)) else if LIST?(z) and LIST?(y) then return UNIFY(REST[z], REST[y], UNIFY(FIRST[z], FIRST[y], 0)) else return failure

function UNIFYVAR(var, z, 6) returns a substitution inputs: var, a variable x, any expression 6, the substitution built up so far

if {var /val} € 6 then return UNIFY(val, z, 6) else if {z/val} € 6 then return UNIFY(var, val, 6) else if OCCUR-CHECK ?(var, x) then return failure else return add {var/z} to 0

Figure 9.1 The unification algorithm. The algorithm works by comparing the structures of the inputs, element by element. The substitution § that is the argument to UNIFY is built up along the way and is used to make sure that later comparisons are consistent with bindings that were established earlier. In a compound expression, such as F(A, B), the function OP picks out the function symbol F and the function ARGS picks out the argument list (A, B).

Storage and retrieval

Underlying the TELL and ASK functions used to inform and interrogate a knowledge base are the more primitive STORE and FETCH functions. STORE(S) stores a sentence s into the knowledge base and FETCH(q) returns all unifiers such that the query ¢ unifies with some sentence in the knowledge base. The problem we used to illustrate unification—finding all facts that unify with Knows (John, x)—is an instance of FETCHing.

The simplest way to implement STORE and FETCH is to keep all the facts in the knowledge base in one long list; then, given a query g, call UNIFY(q, s) for every sentence s in the list. Such a process is inefficient, but it works, and it’s all you need to understand the rest of the chapter. The remainder of this section outlines ways to make retrieval more efficient, and can be skipped on first reading.

We can make FETCH more efficient by ensuring that unifications are attempted only with sentences that have some chance of unifying. For example, there is no point in trying

Section 9.2.

Unification and Lifting 279

INDEXING

PREDICATE INDEXING

SUBSUMPTION LATTICE

to unify Knows(John, x) with Brother(Richard, John). We can avoid such unifications by indexing the facts in the knowledge base. A simple scheme called predicate indexing puts all the Knows facts in one bucket and all the Brother facts in another. The buckets can be stored in a hash table? for efficient access.

Predicate indexing is useful when there are many predicate symbols but only a few clauses for each symbol. In some applications, however, there are many clauses for a given predicate symbol. For example, suppose that the tax authorities want to keep track of who employs whom, using a predicate Employs(z,y). This would be a very large bucket with perhaps millions of employers and tens of millions of employees. Answering a query such as Employs(x, Richard) with predicate indexing would require scanning the entire bucket.

For this particular query, it would help if facts were indexed both by predicate and by second argument, perhaps using a combined hash table key. Then we could simply construct the key from the query and retrieve exactly those facts that unify with the query. For other queries, such as Employs(AIMA.org,y), we would need to have indexed the facts by combining the predicate with the first argument. Therefore, facts can be stored under multiple index keys, rendering them instantly accessible to various queries that they might unify with.

Given a sentence to be stored, it is possible to construct indices for all possible queries that unify with it. For the fact Employs(AIMA.org, Richard), the queries are

Employs(AIMA.org, Richard) Does AIMA org employ Richard?

Employs(z, Richard) Who employs Richard? Employs(AIMA.org, y) Whom does AIMA org employ? Employs(z,y) Who employs whom?

These queries form a subsumption lattice, as shown in Figure 9.2(a). The lattice has some interesting properties. For example, the child of any node in the lattice is obtained from its parent by a single substitution; and the “highest” common descendant of any two nodes is the result of applying their most general unifier. The portion of the lattice above any ground fact can be constructed systematically (Exercise 9.5). A sentence with repeated constants has a slightly different lattice, as shown in Figure 9.2(b). Function symbols and variables in the sentences to be stored introduce still more interesting lattice structures.

The scheme we have described works very well whenever the lattice contains a small number of nodes. For a predicate with n arguments, the lattice contains O(2") nodes. If function symbols are allowed, the number of nodes is also exponential in the size of the terms in the sentence to be stored. This can lead to a huge number of indices. At some point, the benefits of indexing are outweighed by the costs of storing and maintaining all the indices. We can respond by adopting a fixed policy, such as maintaining indices only on keys composed of a predicate plus each argument, or by using an adaptive policy that creates indices to meet the demands of the kinds of queries being asked. For most Al systems, the number of facts to be stored is small enough that efficient indexing is considered a solved problem. For industrial and commercial databases, the problem has received substantial technology development.

2 A hash table is a data structure for storing and retrieving information indexed by fixed keys. For practical purposes, a hash table can be considered to have constant storage and retrieval times, even when the table contains a very large number of items.

280

Chapter 9. Inference in First-Order Logic

Employs(x,y) Employs(x,y) Employs(x,Richard) Employs(AIMA.org,y) Employs(x,John) Employs(x,x) Employs(John,y) Employs(AIMA.org, Richard) Employs(John,John)

(@ (®)

Figure 9.2 (a) The subsumption lattice whose lowest node is the sentence Employs(AIMA. org, Richard). (b) The subsumption lattice for the sentence Employs(John, John).

A forward-chaining algorithm for propositional definite clauses was given in Section 7.5. The idea is simple: start with the atomic sentences in the knowledge base and apply Modus Ponens in the forward direction, adding new atomic sentences, until no further inferences can be made. Here, we explain how the algorithm is applied to first-order definite clauses and how it can be implemented efficiently. Definite clauses such as Situation = Response are especially useful for systems that make inferences in response to newly arrived information. Many systems can be defined this way, and reasoning with forward chaining can be much more efficient than resolution theorem proving. Therefore it is often worthwhile to try to build a knowledge base using only definite clauses so that the cost of resolution can be avoided.

First-order definite clauses

First-order definite clauses closely resemble propositional definite clauses (page 217): they are disjunctions of literals of which exactly one is positive. A definite clause either is atomic or is an implication whose antecedent is a conjunction of positive literals and whose consequent is a single positive literal. The following are first-order definite clauses:

King(z) \ Greedy(z) = FEvil(z) . King(John) . Greedy(y) .

Unlike propositional literals, first-order literals can include variables, in which case those variables are assumed to be universally quantified. (Typically, we omit universal quantifiers when writing definite clauses.) Definite clauses are a suitable normal form for use with Generalized Modus Ponens.

Not every knowledge base can be converted into a set of definite clauses, because of the single-positive-literal restriction, but many can. Consider the following problem:

The law says that it is a crime for an American to sell weapons to hostile nations. The country Nono, an enemy of America, has some missiles, and all of its missiles were sold to it by Colonel West, who is American.

Section 9.3.

Forward Chaining 281

DATALOG

RENAMING

We will prove that West is a criminal, First, we will represent these facts as first-order definite clauses. The next section shows how the forward-chaining algorithm solves the problem.

“...itis a crime for an American to sell weapons to hostile nations”: American(x) A Weapon(y) A Sells(x,y, z) A Hostile(z) = Criminal(x) . 9.3)

“Nono . .. has some missiles.” The sentence 3 x Owns(Nono, x) A Missile(x) is transformed into two definite clauses by Existential Elimination, introducing a new constant A:

Owns(Nono, My) 9.4) Missile(M;) 9.5)

“All of its missiles were sold to it by Colonel West”: Missile(z) A Owns(Nono, xz) = Sells( West, xz, Nono) . (9.6)

We will also need to know that missiles are weapons:

Missile(x) = Weapon(z) 9.7) and we must know that an enemy of America counts as “hostile”:

Enemy(z, America) = Hostile(x) . 9.8) “West, who is American ...”:

American( West) . 9.9) “The country Nono, an enemy of America ...”:

Enemy(Nono, America) . 9.10)

This knowledge base contains no function symbols and is therefore an instance of the class of Datalog knowledge bases—that is, sets of first-order definite clauses with no function symbols. We will see that the absence of function symbols makes inference much easier.

A simple forward-chaining algorithm

The first forward chaining algorithm we will consider is a very simple one, as shown in Figure 9.3. Starting from the known facts, it triggers all the rules whose premises are satisfied, adding their conclusions to the known facts. The process repeats until the query is answered (assuming that just one answer is required) or no new facts are added. Notice that a fact is not “new” if it is just a renaming of a known fact. One sentence is a renaming of another if they are identical except for the names of the variables. For example, Likes(x, IceCream) and Likes (y, IceCream) are renamings of each other because they differ only in the choice of x or y; their meanings are identical: everyone likes ice cream.

We will use our crime problem to illustrate how FOL-FC-ASK works. The implication sentences are (9.3), (9.6), (9.7), and (9.8). Two iterations are required:

e On the first iteration, rule (9.3) has unsatisfied premises. Rule (9.6) is satisfied with {z:/M; }, and Sells( West, My, Nono) is added. Rule (9.7) is satisfied with {z/M }, and Weapon (M,) is added. Rule (9.8) is satisfied with {z/ Nono}, and Hostile( Nono) is added.

282

Chapter 9. Inference in First-Order Logic

function FOL-FC-ASK (KB, a) returns a substitution or false inputs: KB, the knowledge base, a set of first-order definite clauses «, the query, an atomic sentence local variables: new, the new sentences inferred on each iteration

repeat until new is empty new —{ } for each sentence r in KB do (pLA...A pp = gq) STANDARDIZE-APART(r) for each 6 such that SUBST(0,p1 A ... A p,)=SUBST(G,p; A ... A pl) for some pi,...,p, in KB q' — SUBST, q) if ¢ is not a renaming of some sentence already in KB or new then do add ¢' to new ¢ — UNIFY(¢, 0) if ¢ is not fail then return ¢ add new to KB return false

Figure 9.3 A conceptually straightforward, but very inefficient, forward-chaining algorithm. On each iteration, it adds to KB all the atomic sentences that can be inferred in one step from the implication sentences and the atomic sentences already in KB.

[]

Figure 9.4 The proof tree generated by forward chaining on the crime example. The initial facts appear at the bottom level, facts inferred on the first iteration in the middle level, and facts inferred on the second iteration at the top level.

e On the second iteration, rule (9.3) is satisfied with {x/ West, y/M, z/Nono}, and Criminal( West) is added.

Figure 9.4 shows the proof tree that is generated. Notice that no new inferences are possible at this point because every sentence that could be concluded by forward chaining is already contained explicitly in the KB. Such a knowledge base is called a fixed point of the inference process. Fixed points reached by forward chaining with first-order definite clauses are similar

Section 9.3.

Forward Chaining 283

PATTERN MATCHING

to those for propositional forward chaining (page 219); the principal difference is that a firstorder fixed point can include universally quantified atomic sentences.

FOL-FC-ASK is easy to analyze. First, it is sound, because every inference is just an application of Generalized Modus Ponens, which is sound. Second, it is complete for definite clause knowledge bases; that is, it answers every query whose answers are entailed by any knowledge base of definite clauses. For Datalog knowledge bases, which contain no function symbols, the proof of completeness is fairly easy. We begin by counting the number of possible facts that can be added, which determines the maximum number of iterations. Let be the maximum arity (number of arguments) of any predicate, p be the number of predicates, and n be the number of constant symbols. Clearly, there can be no more than pn” distinct ground facts, so after this many iterations the algorithm must have reached a fixed point. Then we can make an argument very similar to the proof of completeness for propositional forward chaining. (See page 219.) The details of how to make the transition from propositional to first-order completeness are given for the resolution algorithm in Section 9.5.

For general definite clauses with function symbols, FOL-FC-ASK can generate infinitely many new facts, so we need to be more careful. For the case in which an answer to the query sentence q is entailed, we must appeal to Herbrand’s theorem to establish that the algorithm will find a proof. (See Section 9.5 for the resolution case.) If the query has no answer, the algorithm could fail to terminate in some cases. For example, if the knowledge base includes the Peano axioms

NatNum(0) Vn NatNum(n) = NatNum(S(n))

then forward chaining adds NatNum(S(0)), NatNum(S(S(0))), NatNum(S(S(S(0)))), and so on. This problem is unavoidable in general. As with general first-order logic, entailment with definite clauses is semidecidable.

Efficient forward chaining

The forward chaining algorithm in Figure 9.3 is designed for ease of understanding rather than for efficiency of operation. There are three possible sources of complexity. First, the “inner loop” of the algorithm involves finding all possible unifiers such that the premise of a rule unifies with a suitable set of facts in the knowledge base. This is often called pattern matching and can be very expensive. Second, the algorithm rechecks every rule on every iteration to see whether its premises are satisfied, even if very few additions are made to the knowledge base on each iteration. Finally, the algorithm might generate many facts that are irrelevant to the goal. We will address each of these sources in turn.

Matching rules against known facts

The problem of matching the premise of a rule against the facts in the knowledge base might seem simple enough. For example, suppose we want to apply the rule

Missile(x) = Weapon(z) .

284 Chapter 9. Inference in First-Order Logic Diff (wa, nt) A Diff (wa, sa) A Diff (nt, q) A Diff (nt, sa) A Diff (gq, nsw) A Diff (q, sa) A Diff (nsw, v) A Diff (nsw, sa) A NW, Diff (v, sa) = Colorable() (v) Diff (Red, Blue) Diff (Red, Green) Diff (Green, Red) Diff (Green, Blue) (1) Diff (Blue, Red) Diff (Blue, Green) (a) (b) Figure 9.5 (a) Constraint graph for coloring the map of Australia (from Figure5.1). (b) The map-coloring CSP expressed as a single definite clause. Note that the domains of the variables are defined implicitly by the constants given in the ground facts for Diff. Then we need to find all the facts that unify with Missile(x); in a suitably indexed knowledge base, this can be done in constant time per fact. Now consider a rule such as Missile(x) A Owns(Nono, x) = Sells( West, xz, Nono) . Again, we can find all the objects owned by Nono in constant time per object; then, for each object, we could check whether it is a missile. If the knowledge base contains many objects owned by Nono and very few missiles, however, it would be better to find all the missiles first Sondunet and then check whether they are owned by Nono. This is the conjunct ordering problem:

find an ordering to solve the conjuncts of the rule premise so that the total cost is minimized. It turns out that finding the optimal ordering is NP-hard, but good heuristics are available. For example, the most constrained variable heuristic used for CSPs in Chapter 5 would suggest ordering the conjuncts to look for missiles first if there are fewer missiles than objects that are owned by Nono.

The connection between pattern matching and constraint satisfaction is actually very close. We can view each conjunct as a constraint on the variables that it contains—for example, Missile(x) is a unary constraint on x. Extending this idea, we can express every finite-domain CSP as a single definite clause together with some associated ground facts. Consider the map-coloring problem from Figure 5.1, shown again in Figure 9.5(a). An equivalent formulation as a single definite clause is given in Figure 9.5(b). Clearly, the conclusion Colorable() can be inferred only if the CSP has a solution. Because CSPs in general include 3SAT problems as special cases, we can conclude that matching a definite clause against a set of facts is NP-hard.

It might seem rather depressing that forward chaining has an NP-hard matching problem in its inner loop. There are three ways to cheer ourselves up:

Section 9.3. Forward Chaining 285

e We can remind ourselves that most rules in real-world knowledge bases are small and simple (like the rules in our crime example) rather than large and complex (like the CSP formulation in Figure 9.5). It is common in the database world to assume that both the sizes of rules and the arities of predicates are bounded by a constant and to worry DATA COMPLEXITY only about data complexity—that is, the complexity of inference as a function of the number of ground facts in the database. It is easy to show that the data complexity of forward chaining is polynomial.

e We can consider subclasses of rules for which matching is efficient. Essentially every Datalog clause can be viewed as defining a CSP, so matching will be tractable just when the corresponding CSP is tractable. Chapter 5 describes several tractable families of CSPs. For example, if the constraint graph (the graph whose nodes are variables and whose links are constraints) forms a tree, then the CSP can be solved in linear time. Exactly the same result holds for rule matching. For instance, if we remove South Australia from the map in Figure 9.5, the resulting clause is

Diff (wa, nt) A Diff (nt, q) A Diff (q, nsw) A Diff (nsw,v) = Colorable()

which corresponds to the reduced CSP shown in Figure 5.11. Algorithms for solving tree-structured CSPs can be applied directly to the problem of rule matching.

e We can work hard to eliminate redundant rule matching attempts in the forward chaining algorithm, which is the subject of the next section.

Incremental forward chaining

When we showed how forward chaining works on the crime example, we cheated; in particular, we omitted some of the rule matching done by the algorithm shown in Figure 9.3. For example, on the second iteration, the rule

Missile(z) = Weapon(x)

matches against Missile( M1) (again), and of course the conclusion Weapon (AM) is already known so nothing happens. Such redundant rule matching can be avoided if we make the following observation: Every new fact inferred on iteration t must be derived from at least one new fact inferred on iteration t — 1. This is true because any inference that does not require a new fact from iteration ¢ — 1 could have been done at iteration ¢ — 1 already.

This observation leads naturally to an incremental forward chaining algorithm where, at iteration ¢, we check a rule only if its premise includes a conjunct p; that unifies with a fact p; newly inferred at iteration ¢ — 1. The rule matching step then fixes p; to match with pj, but allows the other conjuncts of the rule to match with facts from any previous iteration. This algorithm generates exactly the same facts at each iteration as the algorithm in Figure 9.3, but is much more efficient.

With suitable indexing, it is easy to identify all the rules that can be triggered by any given fact, and indeed many real systems operate in an “update” mode wherein forward chaining occurs in response to each new fact that is TELLed to the system. Inferences cascade through the set of rules until the fixed point is reached, and then the process begins again for the next new fact.

286

Chapter 9. Inference in First-Order Logic

RETE

PRODUCTION SYSTEMS

COGNITIVE ARCHITECTURES

Typically, only a small fraction of the rules in the knowledge base are actually triggered by the addition of a given fact. This means that a great deal of redundant work is done in constructing partial matches repeatedly that have some unsatisfied premises. Our crime example is rather too small to show this effectively, but notice that a partial match is constructed on the first iteration between the rule

American{x) A Weapon(y) A Sells(x,y, z) A Hostile(z) = Criminal(z)

and the fact American( West). This partial match is then discarded and rebuilt on the second iteration (when the rule succeeds). It would be better to retain and gradually complete the partial matches as new facts arrive, rather than discarding them.

The rete algorithm? was the first to address this problem seriously. The algorithm preprocesses the set of rules in the knowledge base to construct a sort of dataflow network in which each node is a literal from a rule premise. Variable bindings flow through the network and are filtered out when they fail to match a literal. If two literals in a rule share a variable— for example, Sells(x,y, z) A Hostile(z) in the crime example—then the bindings from each literal are filtered through an equality node. A variable binding reaching a node for an nary literal such as Sells(z, y, z) might have to wait for bindings for the other variables to be established before the process can continue. At any given point, the state of a rete network captures all the partial matches of the rules, avoiding a great deal of recomputation.

Rete networks, and various improvements thereon, have been a key component of socalled production systems, which were among the earliest forward chaining systems in widespread use.\* The XCON system (originally called R1, McDermott, 1982) was built using a production system architecture. XCON contained several thousand rules for designing configurations of computer components for customers of the Digital Equipment Corporation. It was one of the first clear commercial successes in the emerging field of expert systems. Many other similar systems have been built using the same underlying technology, which has been implemented in the general-purpose language OPS-5.

Production systems are also popular in cognitive architectures—that is, models of human reasoning—such as ACT (Anderson, 1983) and SOAR (Laird et al., 1987). In such systems, the “working memory” of the system models human short-term memory, and the productions are part of long-term memory. On each cycle of operation, productions are matched against the working memory of facts. A production whose conditions are satisfied can add or delete facts in working memory. In contrast to the typical situation in databases, production systems often have many rules and relatively few facts. With suitably optimized matching technology, some modern systems can operate in real time with over a million rules.

Irrelevant facts

The final source of inefficiency in forward chaining appears to be intrinsic to the approach and also arises in the propositional context. (See Section 7.5.) Forward chaining makes all allowable inferences based on the known facts, even if they are irrelevant to the goal at hand. In our crime example, there were no rules capable of drawing irrelevant conclusions,

3 Rete is Latin for net. The English pronunciation rhymes with treaty. 4 The word production in production systems denotes a condition—action rule.

Section 9.4.

Backward Chaining 287

MAGIC SET

so the lack of directedness was not a problem. In other cases (e.g., if we have several rules describing the eating habits of Americans and the prices of missiles), FOL-FC-AsK will generate many irrelevant conclusions.

One way to avoid drawing irrelevant conclusions is to use backward chaining, as described in Section 9.4. Another solution is to restrict forward chaining to a selected subset of rules; this approach was discussed in the propositional context. A third approach has emerged in the deductive database community, where forward chaining is the standard tool. The idea is to rewrite the rule set, using information from the goal, so that only relevant variable bindings—those belonging to a so-called magic set—are considered during forward inference. For example, if the goal is Criminal( West), the rule that concludes Criminal(x) will be rewritten to include an extra conjunct that constrains the value of x:

Magic(z) A American{x) A Weapon(y) A Sells(x,y, z) A Hostile(z) = Criminal(z) .

The fact Magic( West) is also added to the KB. In this way, even if the knowledge base contains facts about millions of Americans, only Colonel West will be considered during the forward inference process. The complete process for defining magic sets and rewriting the knowledge base is too complex to go into here, but the basic idea is to perform a sort of “generic” backward inference from the goal in order to work out which variable bindings need to be constrained. The magic sets approach can therefore be thought of as a kind of hybrid between forward inference and backward preprocessing.

The second major family of logical inference algorithms uses the backward chaining approach introduced in Section 7.5. These algorithms work backward from the goal, chaining through rules to find known facts that support the proof. We describe the basic algorithm, and then we describe how it is used in logic programming, which is the most widely used form of automated reasoning. We will also see that backward chaining has some disadvantages compared with forward chaining, and we look at ways to overcome them. Finally, we will look at the close connection between logic programming and constraint satisfaction problems.

A backward chaining algorithm

Figure 9.6 shows a simple backward-chaining algorithm, FOL-BC-AsK. It is called with a list of goals containing a single element, the original query, and returns the set of all substitutions satisfying the query. The list of goals can be thought of as a “stack” waiting to be worked on; if all of them can be satisfied, then the current branch of the proof succeeds. The algorithm takes the first goal in the list and finds every clause in the knowledge base whose positive literal, or head, unifies with the goal. Each such clause creates a new recursive call in which the premise, or body, of the clause is added to the goal stack. Remember that facts are clauses with a head but no body, so when a goal unifies with a known fact, no new subgoals are added to the stack and the goal is solved. Figure 9.7 is the proof tree for deriving Criminal( West) from sentences (9.3) through (9.10).

288 Chapter 9. Inference in First-Order Logic

function FOL-BC-ASK(KB, goals, f) returns a set of substitutions inputs: KB, a knowledge base goals, a list of conjuncts forming a query (# already applied) ¢, the current substitution, initially the empty substitution { } local variables: answers, a set of substitutions, initially empty

if goals is empty then return {6} q' + SUBST(#, FIRST(goals)) for each sentence r in KB where STANDARDIZE-APART(7) = (p1 A ... A pp = q) and 6’ — UNIFY(q, ¢’) succeeds new-\_goals «— [p1, . . ., pn |REST(goals)] answers «+ FOL-BC-ASK(KB, new \_goals, COMPOSE(#’,0)) U answers return answers

Figure 9.6 A simple backward-chaining algorithm.

Criminal(West)

| American(Wesr)) Weapon(y) | Sells(West, M2) | Hostile(Nono)

{1 {2/Nono}

| Missile(y) || Missite(m)) | | Owns(Nono,M,) | | Enemy(Nono, America) | {yiM1} {} {} {}

Figure 9.7 Proof tree constructed by backward chaining to prove that West is a criminal. The tree should be read depth first, left to right. To prove Criminal( West), we have to prove the four conjuncts below it. Some of these are in the knowledge base, and others require further backward chaining. Bindings for each successful unification are shown next to the corresponding subgoal. Note that once one subgoal in a conjunction succeeds, its substitution is applied to subsequent subgoals. Thus, by the time FOL-BC-ASK gets to the last conjunct, originally Hostile(z), z is already bound to Nono.

COMPOSITION The algorithm uses composition of substitutions. COMPOSE(f1, #2) is the substitution whose effect is identical to the effect of applying each substitution in turn. That is,

SUBST(COMPOSE(f1, 65), p) = SUBST(f2, SUBST(61,p)) .

In the algorithm, the current variable bindings, which are stored in 6, are composed with the bindings resulting from unifying the goal with the clause head, giving a new set of current bindings for the recursive call.

Section 9.4.

Backward Chaining 289

PROLOG

Backward chaining, as we have written it, is clearly a depth-first search algorithm. This means that its space requirements are linear in the size of the proof (neglecting, for now, the space required to accumulate the solutions). It also means that backward chaining (unlike forward chaining) suffers from problems with repeated states and incompleteness. We will discuss these problems and some potential solutions, but first we will see how backward chaining is used in logic programming systems.

Logic programming

Logic programming is a technology that comes fairly close to embodying the declarative ideal described in Chapter 7: that systems should be constructed by expressing knowledge in a formal language and that problems should be solved by running inference processes on that knowledge. The ideal is summed up in Robert Kowalski’s equation,

Algorithm = Logic + Control .

Prolog is by far the most widely used logic programming language. Its users number in the hundreds of thousands. It is used primarily as a rapid-prototyping language and for symbolmanipulation tasks such as writing compilers (Van Roy, 1990) and parsing natural language (Pereira and Warren, 1980). Many expert systems have been written in Prolog for legal, medical, financial, and other domains.

Prolog programs are sets of definite clauses written in a notation somewhat different from standard first-order logic. Prolog uses uppercase letters for variables and lowercase for constants. Clauses are written with the head preceding the body; “: -” is used for leftimplication, commas separate literals in the body, and a period marks the end of a sentence:

criminal (X) :american(X), weapon(Y), sells(X,Y,Z), hostile(z).

Prolog includes “syntactic sugar” for list notation and arithmetic. As an example, here is a Prolog program for append (X, Y, Z), which succeeds if list Z is the result of appending lists X and Y:

append ([],Y,Y). append ([A|X],Y,[A]|2Z2]) :append(X,Y,Z).

In English, we can read these clauses as (1) appending an empty list with a list Y produces the same list Y and (2) [A|Z] is the result of appending [A |X] onto Y, provided that Z is the result of appending X onto Y. This definition of append appears fairly similar to the corresponding definition in Lisp, but is actually much more powerful. For example, we can ask the query append (A,B, [1,2]): what two lists can be appended to give [1,2]7 We get back the solutions

A=[] B=[1,2] A=[1] B=[2] A=[1,2] B=[] The execution of Prolog programs is done via depth-first backward chaining, where

clauses are tried in the order in which they are written in the knowledge base. Some aspects of Prolog fall outside standard logical inference:

290

Chapter 9. Inference in First-Order Logic

CHOICE POINT

e There is a set of built-in functions for arithmetic. Literals using these function symbols are “proved” by executing code rather than doing further inference. For example, the goal “X is 4+3” succeeds with X bound to 7. On the other hand, the goal “5 is X+Y” fails, because the built-in functions do not do arbitrary equation solving.’

e There are built-in predicates that have side effects when executed. These include inputoutput predicates and the assert/retract predicates for modifying the knowledge base. Such predicates have no counterpart in logic and can produce some confusing effects—for example, if facts are asserted in a branch of the proof tree that eventually fails.

e Prolog allows a form of negation called negation as failure. A negated goal not P is considered proved if the system fails to prove P. Thus, the sentence

alive(X) :not dead (X).

can be read as “Everyone is alive if not provably dead.”

e Prolog has an equality operator, =, but it lacks the full power of logical equality. An equality goal succeeds if the two terms are unifiable and fails otherwise. So X+Y=2+3 succeeds with X bound to 2 and Y¥ bound to 3, but morningstar=eveningstar fails. (In classical logic, the latter equality might or might not be true.) No facts or rules about equality can be asserted.

e The occur check is omitted from Prolog’s unification algorithm. This means that some unsound inferences can be made; these are seldom a problem except when using Prolog for mathematical theorem proving.

The decisions made in the design of Prolog represent a compromise between declarativeness and execution efficiency—inasmuch as efficiency was understood at the time Prolog was designed. We will return to this subject after looking at how Prolog is implemented.

Efficient implementation of logic programs

The execution of a Prolog program can happen in two modes: interpreted and compiled. Interpretation essentially amounts to running the FOL-BCASK algorithm from Figure 9.6, with the program as the knowledge base. We say “essentially,” because Prolog interpreters contain a variety of improvements designed to maximize speed. Here we consider only two.

First, instead of constructing the list of all possible answers for each subgoal before continuing to the next, Prolog interpreters generate one answer and a “promise” to generate the rest when the current answer has been fully explored. This promise is called a choice point. When the depth-first search completes its exploration of the possible solutions arising from the current answer and backs up to the choice point, the choice point is expanded to yield a new answer for the subgoal and a new choice point. This approach saves both time and space. It also provides a very simple interface for debugging because at all times there is only a single solution path under consideration.

Second, our simple implementation of FOL-BC-ASK spends a good deal of time generating and composing substitutions. Prolog implements substitutions using logic variables

5 Note that if the Peano axioms are provided, such goals can be solved by inference within a Prolog program.

Section 9.4.

Backward Chaining 291

TRAIL

OPEN-CODE

procedure APPEND(azx, i, az, continuation)

trail — GLOBAL-TRAIL-POINTER()

if ax = [] and UNIFY (y, az) then CALL(continuation)

RESET-TRAIL (trail)

a +— NEW-VARIABLE(); & <— NEW-VARIABLE(); z « NEW-VARIABLE()

if UNIFY(az, [a | z]) and UNIFY (az, [a | z]) then APPEND(z, v, 2, continuation)

Figure 9.8 Pseudocode representing the result of compiling the Append predicate. The function NEwVARIABLE returns a new variable, distinct from all other variables so far used. The procedure CALL( continuation) continues execution with the specified continuation.

that can remember their current binding. At any point in time, every variable in the program either is unbound or is bound to some value. Together, these variables and values implicitly define the substitution for the current branch of the proof. Extending the path can only add new variable bindings, because an attempt to add a different binding for an already bound variable results in a failure of unification. When a path in the search fails, Prolog will back up to a previous choice point, and then it might have to unbind some variables. This is done by keeping track of all the variables that have been bound in a stack called the trail. As each new variable is bound by UNIFYVAR, the variable is pushed onto the trail. When a goal fails and it is time to back up to a previous choice point, each of the variables is unbound as it is removed from the trail.

Even the most efficient Prolog interpreters require several thousand machine instructions per inference step because of the cost of index lookup, unification, and building the recursive call stack. In effect, the interpreter always behaves as if it has never seen the program before; for example, it has to find clauses that match the goal. A compiled Prolog program, on the other hand, is an inference procedure for a specific set of clauses, so it knows what clauses match the goal. Prolog basically generates a miniature theorem prover for each different predicate, thereby eliminating much of the overhead of interpretation. It is also possible to open-code the unification routine for each different call, thereby avoiding explicit analysis of term structure. (For details of open-coded unification, see Warren et al. (1977).)

The instruction sets of today’s computers give a poor match with Prolog’s semantics, so most Prolog compilers compile into an intermediate language rather than directly into machine language. The most popular intermediate language is the Warren Abstract Machine, or WAM, named after David H. D. Warren, one of the implementors of the first Prolog compiler. The WAM is an abstract instruction set that is suitable for Prolog and can be either interpreted or translated into machine language. Other compilers translate Prolog into a highlevel language such as Lisp or C and then use that language’s compiler to translate to machine language. For example, the definition of the Append predicate can be compiled into the code shown in Figure 9.8. There are several points worth mentioning:

e Rather than having to search the knowledge base for Append clauses, the clauses become a procedure and the inferences are carried out simply by calling the procedure.

292

Chapter 9. Inference in First-Order Logic

CONTINUATIONS

OR-PARALLELISM

AND-PARALLELISM

e As described earlier, the current variable bindings are kept on a trail. The first step of the procedure saves the current state of the trail, so that it can be restored by RESET-TRAIL if the first clause fails. This will undo any bindings generated by the first call to UNIFY.

e The trickiest part is the use of continuations to implement choice points. You can think of a continuation as packaging up a procedure and a list of arguments that together define what should be done next whenever the current goal succeeds. It would not do just to return from a procedure like APPEND when the goal succeeds, because it could succeed in several ways, and each of them has to be explored. The continuation argument solves this problem because it can be called each time the goal succeeds. In the APPEND code, if the first argument is empty, then the APPEND predicate has succeeded. We then CALL the continuation, with the appropriate bindings on the trail, to do whatever should be done next. For example, if the call to APPEND were at the top level, the continuation would print the bindings of the variables.

Before Warren’s work on the compilation of inference in Prolog, logic programming was too slow for general use. Compilers by Warren and others allowed Prolog code to achieve speeds that are competitive with C on a variety of standard benchmarks (Van Roy, 1990). Of course, the fact that one can write a planner or natural language parser in a few dozen lines of Prolog makes it somewhat more desirable than C for prototyping most small-scale Al research projects.

Parallelization can also provide substantial speedup. There are two principal sources of parallelism. The first, called OR-parallelism, comes from the possibility of a goal unifying with many different clauses in the knowledge base. Each gives rise to an independent branch in the search space that can lead to a potential solution, and all such branches can be solved in parallel. The second, called AND-parallelism, comes from the possibility of solving each conjunct in the body of an implication in parallel. AND-parallelism is more difficult to achieve, because solutions for the whole conjunction require consistent bindings for all the variables. Each conjunctive branch must communicate with the other branches to ensure a global solution.

Redundant inference and infinite loops

We now turn to the Achilles heel of Prolog: the mismatch between depth-first search and search trees that include repeated states and infinite paths. Consider the following logic program that decides if a path exists between two points on a directed graph:

path(X,z) :1ink(X,Z). path(X,z2) :path(X,Y), link(Y,Z).

A simple three-node graph, described by the facts 1ink (a,b) and 1ink (b,c), is shown in Figure 9.9(a). With this program, the query path (a, c) generates the proof tree shown in Figure 9.10(a). On the other hand, if we put the two clauses in the order

path(X,Z) :path(X,Y), link(Y,Z). path(X,z) :1link(X,Z).

Section 9.4.

Backward Chaining 293

[]

[]

DYNAMIC PROGRAMMING

A B C G—@—0 (a) (b)

Figure 9.9 (a) Finding a path from A to C' can lead Prolog into an infinite loop. (b) A graph in which each node is connected to two random successors in the next layer. Finding a path from A; to .J4 requires 877 inferences.

{¥/b)

(a) (b)

Figure 9.10 (a) Proof that a path exists from A to C. (b) Infinite proof tree generated when the clauses are in the “wrong” order.

then Prolog follows the infinite path shown in Figure 9.10(b). Prolog is therefore incomplete as a theorem prover for definite clauses—even for Datalog programs, as this example shows— because, for some knowledge bases, it fails to prove sentences that are entailed. Notice that forward chaining does not suffer from this problem: once path (a,b), path (b,c), and path (a, c) are inferred, forward chaining halts.

Depth-first backward chaining also has problems with redundant computations. For example, when finding a path from A; to J, in Figure 9.9(b), Prolog performs 877 inferences, most of which involve finding all possible paths to nodes from which the goal is unreachable. This is similar to the repeated-state problem discussed in Chapter 3. The total amount of inference can be exponential in the number of ground facts that are generated. If we apply forward chaining instead, at most n2 path (X,Y) facts can be generated linking n nodes. For the problem in Figure 9.9(b), only 62 inferences are needed.

Forward chaining on graph search problems is an example of dynamic programming, in which the solutions to subproblems are constructed incrementally from those of smaller subproblems and are cached to avoid recomputation. We can obtain a similar effect in a backward chaining system using memoization—that is, caching solutions to subgoals as they are

294 Chapter 9. Inference in First-Order Logic

found and then reusing those solutions when the subgoal recurs, rather than repeating the

ria previous computation. This is the approach taken by tabled logic programming systems, which use efficient storage and retrieval mechanisms to perform memoization. Tabled logic programming combines the goal-directedness of backward chaining with the dynamic programming efficiency of forward chaining. It is also complete for Datalog programs, which means that the programmer need worry less about infinite loops.

Constraint logic programming

In our discussion of forward chaining (Section 9.3), we showed how constraint satisfaction problems (CSPs) can be encoded as definite clauses. Standard Prolog solves such problems in exactly the same way as the backtracking algorithm given in Figure 5.3.

Because backtracking enumerates the domains of the variables, it works only for finite domain CSPs. In Prolog terms, there must be a finite number of solutions for any goal with unbound variables. (For example, the goal diff (q, sa), which says that Queensland and South Australia must be different colors, has six solutions if three colors are allowed.) Infinite-domain CSPs—for example with integer or real-valued variables—require quite different algorithms, such as bounds propagation or linear programming.

The following clause succeeds if three numbers satisfy the triangle inequality:

triangle(X,Y,z) :X>=0, ¥Y>=0, Z>=0, X+¥>=7, Y+7Z>=X, X+7Z>=Y.

If we ask Prolog the query triangle (3,4,5), this works fine. On the other hand, if we ask triangle (3,4, 2), no solution will be found, because the subgoal zZ>=0 cannot be handled by Prolog. The difficulty is that variables in Prolog must be in one of two states: unbound or bound to a particular term.

Binding a variable to a particular term can be viewed as an extreme form of constraint,

SONSTRANIGOSIC namely an equality constraint. Constraint logic programming (CLP) allows variables to be constrained rather than bound. A solution to a constraint logic program is the most specific set of constraints on the query variables that can be derived from the knowledge base. For example, the solution to the triangle (3,4,7) query is the constraint 7 >= 7Z >= 1. Standard logic programs are just a special case of CLP in which the solution constraints must be equality constraints—that is bindings.

CLP systems incorporate various constraint-solving algorithms for the constraints allowed in the language. For example, a system that allows linear inequalities on real-valued variables might include a linear programming algorithm for solving those constraints. CLP systems also adopt a much more flexible approach to solving standard logic programming queries. For example, instead of depth-first, left-to-right backtracking, they might use any of the more efficient algorithms discussed in Chapter 5, including heuristic conjunct ordering, backjumping, cutset conditioning, and so on. CLP systems therefore combine elements of constraint satisfaction algorithms, logic programming, and deductive databases.

CLP systems can also take advantage of the variety of CSP search optimizations described in Chapter 5, such as variable and value ordering, forward checking, and intelligent backtracking. Several systems have been defined that allow the programmer more control

Section 9.5.

Resolution 295

METARULES

over the search order for inference. For example, the MRS Language (Genesereth and Smith, 1981; Russell, 1985) allows the programmer to write metarules to determine which conjuncts are tried first. The user could write a rule saying that the goal with the fewest variables should be tried first or could write domain-specific rules for particular predicates.

COMPLETENESS THEOREM

INCOMPLETENESS THEOREM

The last of our three families of logical systems is based on resolution. We saw in Chapter 7 that propositional resolution is a refutation complete inference procedure for propositional logic. In this section, we will see how to extend resolution to first-order logic.

The question of the existence of complete proof procedures is of direct concern to mathematicians. If a complete proof procedure can be found for mathematical statements, two things follow: first, all conjectures can be established mechanically; second, all of mathematics can be established as the logical consequence of a set of fundamental axioms. The question of completeness has therefore generated some of the most important mathematical work of the 20th century. In 1930, the German mathematician Kurt Godel proved the first completeness theorem for first-order logic, showing that any entailed sentence has a finite proof. (No really practical proof procedure was found until J. A. Robinson published the resolution algorithm in 1965.) In 1931, G6del proved an even more famous incompleteness theorem. The theorem states that a logical system that includes the principle of induction— without which very little of discrete mathematics can be constructed—is necessarily incomplete. Hence, there are sentences that are entailed, but have no finite proof within the system. The needle may be in the metaphorical haystack, but no procedure can guarantee that it will be found.

Despite Godel’s theorem, resolution-based theorem provers have been applied widely to derive mathematical theorems, including several for which no proof was known previously. Theorem provers have also been used to verify hardware designs and to generate logically correct programs, among other applications.

Conjunctive normal form for first-order logic

As in the propositional case, first-order resolution requires that sentences be in conjunctive normal form (CNF)—that is, a conjunction of clauses, where each clause is a disjunction of literals. Literals can contain variables, which are assumed to be universally quantified. For example, the sentence

Va American(z) A Weapon(y) A Sells(z,y, z) A Hostile(z) = Criminal(x) becomes, in CNF, —American(z) V ~ Weapon (y) V ~Sells(z,y, z) V —Hostile(z) V Criminal(x) .

6 A clause can also be represented as an implication with a conjunction of atoms on the left and a disjunction of atoms on the right, as shown in Exercise 7.12, This form, sometimes called Kowalski form when written with a right-to-left implication symbol (Kowalski, 1979b), is often much easier to read.

296 Chapter 9. Inference in First-Order Logic

Every sentence of first-order logic can be converted into an inferentially equivalent CNF sentence. In particular, the CNF sentence will be unsatisfiable just when the original sentence is unsatisfiable, so we have a basis for doing proofs by contradiction on the CNF sentences.

The procedure for conversion to CNF is very similar to the propositional case, which we saw on page 215. The principal difference arises from the need to eliminate existential quantifiers. We will illustrate the procedure by translating the sentence “Everyone who loves all animals is loved by someone,” or

Va [Vy Animal(y) = Loves(z,y)] = [By Loves(y,z)] . The steps are as follows: ¢> Eliminate implications: Va [Vy —Animal(y)V Loves(z,y)| V [Jy Loves(y,z)] .

{> Move — inwards: In addition to the usual rules for negated connectives, we need rules for negated quantifiers. Thus, we have

Vz p becomes dz —p —3x p becomes Ye —p.

Our sentence goes through the following transformations:

Va [By —(-Animal(y) V Loves(z,y))] V [By Loves(y,x)] . Va [3y ——Animal(y) A —Loves(z,y)| V [By Loves(y,z)] . Va [3y Animal(y) A —Loves(z,y)] V [3y Loves(y,z)].

Notice how a universal quantifier (Vy) in the premise of the implication has become an existential quantifier. The sentence now reads “Either there is some animal that x doesn’t love, or (if this is not the case) someone loves x.” Clearly, the meaning of the original sentence has been preserved.

¢ Standardize variables: For sentences like (Vz P(z))V (3x Q(z)) which use the same variable name twice, change the name of one of the variables. This avoids confusion later when we drop the quantifiers. Thus, we have

Va [By Animal(y) A —Loves(z,y)] V [32 Loves(z,x)].

SKOLEMIZATION ¢ Skolemize: Skolemization is the process of removing existential quantifiers by elimination. In the simple case, it is just like the Existential Instantiation rule of Section 9.1: translate 32 P(x) into P(A), where A is a new constant. If we apply this rule to our sample sentence, however, we obtain

Va [Animal(A) A —Loves(z, A)] V Loves(B, x) which has the wrong meaning entirely: it says that everyone either fails to love a particular animal A or is loved by some particular entity B. In fact, our original sentence

allows each person to fail to love a different animal or to be loved by a different person. Thus, we want the Skolem entities to depend on x:

Vx [Animal(F(x)) A =Loves(z, F(x))] V Loves(G(z), x) .

SKOLEM FUNCTION Here F' and G are Skolem functions. The general rule is that the arguments of the

Section 9.5.

Resolution 297

BINARY RESOLUTION

Skolem function are all the universally quantified variables in whose scope the existential quantifier appears. As with Existential Instantiation, the Skolemized sentence is satisfiable exactly when the original sentence is satisfiable.

¢> Drop universal quantifiers: At this point, all remaining variables must be universally quantified. Moreover, the sentence is equivalent to one in which all the universal quantifiers have been moved to the left. We can therefore drop the universal quantifiers:

[Animal(F(z)) A —Loves(z, F(z))] V Loves(G(z), x) . ¢ Distribute V over A: [Animal(F(z)) V Loves(G(z),x)] A [~Loves(z, F(x)) V Loves(G(z), z)] . This step may also require flattening out nested conjunctions and disjunctions.

The sentence is now in CNF and consists of two clauses. It is quite unreadable. (It may help to explain that the Skolem function F(x) refers to the animal potentially unloved by x, whereas G(x) refers to someone who might love x.) Fortunately, humans seldom need look at CNF sentences—the translation process is easily automated.

The resolution inference rule

The resolution rule for first-order clauses is simply a lifted version of the propositional resolution rule given on page 214. Two clauses, which are assumed to be standardized apart so that they share no variables, can be resolved if they contain complementary literals. Propositional literals are complementary if one is the negation of the other; first-order literals are complementary if one unifies with the negation of the other. Thus we have ON Ey, mi V---Vmy

SUBST(6,41V ---V £4; VEG Ve VL Vm Ve Vm Vm Ve Vimy) where UNIFY (¥;, =m) = 6. For example, we can resolve the two clauses

[Animal(F(z)) V Loves(G(z),z)] and [-Loves(u,v)V —Kills(u,v)]

by eliminating the complementary literals Loves(G(z), x) and —Loves(u,v), with unifier 6 ={u/G(z),v/z}, to produce the resolvent clause

[Animal (F(z) V ~Kills(G(z), z)] .

The rule we have just given is the binary resolution rule, because it resolves exactly two literals. The binary resolution rule by itself does not yield a complete inference procedure. The full resolution rule resolves subsets of literals in each clause that are unifiable. An alternative approach is to extend factoring—the removal of redundant literals—to the first-order case. Propositional factoring reduces two literals to one if they are identical; first-order factoring reduces two literals to one if they are unifiable. The unifier must be applied to the entire clause. The combination of binary resolution and factoring is complete.

Example proofs

Resolution proves that KB |= « by proving KB A —« unsatisfiable, i.e., by deriving the empty clause. The algorithmic approach is identical to the propositional case, described in

298

Chapter 9. Inference in First-Order Logic

[]

Figure 9.11 A resolution proof that West is a criminal.

Figure 7.12, so we will not repeat it here. Instead, we will give two example proofs. The first is the crime example from Section 9.3. The sentences in CNF are

— American (z) V = Weapon (y) V —=Sells(z,y, z) V —Hostile(z) V Criminal(z) . —Missile(z) V = Owns(Nono, x) V Sells( West, x, Nono) .

—Enemy(z, America) V Hostile(x) .

—Missile(x) V Weapon(z) .

Owns(Nono, My). Missile(My) .

American West) . Enemy(Nono, America) .

We also include the negated goal —Criminal( West). The resolution proof is shown in Figure 9.11. Notice the structure: single “spine” beginning with the goal clause, resolving against clauses from the knowledge base until the empty clause is generated. This is characteristic of resolution on Horn clause knowledge bases. In fact, the clauses along the main spine correspond exactly to the consecutive values of the goals variable in the backward chaining algorithm of Figure 9.6. This is because we always chose to resolve with a clause whose positive literal unified with the leftmost literal of the “current” clause on the spine; this is exactly what happens in backward chaining. Thus, backward chaining is really just a special case of resolution with a particular control strategy to decide which resolution to perform next.

Our second example makes use of Skolemization and involves clauses that are not detinite clauses. This results in a somewhat more complex proof structure. In English, the problem is as follows:

Everyone who loves all animals is loved by someone. Anyone who kills an animal is loved by no one.

Jack loves all animals.

Either Jack or Curiosity killed the cat, who is named Tuna. Did Curiosity kill the cat?

Section 9.5. Resolution 299

First, we express the original sentences, some background knowledge, and the negated goal G in first-order logic:

A. Vz [Vy Animal(y) = Loves(z,y)] = [Jy Loves(y,z)] Va [3y Animal(y) A Kills(z,y)] = [Vz —Loves(z,x)] Vz Animal(z) = Loves(Jack, x) Kills(Jack, Tuna) V Kills( Curiosity, Tuna) Cat( Tuna) Va Cot(z) = Animal(z) -G. —Kills( Curiosity, Tuna) Now we apply the conversion procedure to convert each sentence to CNF: Al. Animal(F(z))V Loves(G(z), x) A2. —Loves(z, F(z))V Loves(G(z), x) B. —Animal(y) Vv ~Kills(z,y) V = Loves(z, x) C. —Animal(x)V Loves(Jack, z) D. Kills(Jack, Tuna) V Kills( Curiosity, Tuna) E F

mmUnNnw

Cat (Tuna) . =Cat(z)V Animal(x) -G. —Kills( Curiosity, Tuna)

The resolution proof that Curiosity killed the cat is given in Figure 9.12. In English, the proof could be paraphrased as follows:

Suppose Curiosity did not kill Tuna. We know that either Jack or Curiosity did; thus Jack must have. Now, Tuna is a cat and cats are animals, so Tuna is an animal. Because anyone who kills an animal is loved by no one, we know that no one loves Jack. On the other hand, Jack loves all animals, so someone loves him; so we have a contradiction. Therefore, Curiosity killed the cat.

The proof answers the question “Did Curiosity kill the cat?” but often we want to pose more general questions, such as “Who killed the cat?” Resolution can do this, but it takes a little

[]

Figure 9.12 A resolution proof that Curiosity killed the cat. Notice the use of factoring in the derivation of the clause Loves(G (Jack), Jack).

300

Chapter 9. Inference in First-Order Logic

NONCONSTRUCTIVE PROOF

ANSWER LITERAL

REFUTATION COMPLETENESS

more work to obtain the answer. The goal is 3w Kills(w, Tuna), which, when negated, becomes —Kills(w, Tuna) in CNE. Repeating the proof in Figure 9.12 with the new negated goal, we obtain a similar proof tree, but with the substitution {w/ Curiosity} in one of the steps. So, in this case, finding out who killed the cat is just a matter of keeping track of the bindings for the query variables in the proof.

Unfortunately, resolution can produce nonconstructive proofs for existential goals. For example, —Kills(w, Tuna) resolves with Kills(Jack, Tuna) V Kills( Curiosity, Tuna) to give Kills(Jack, Tuna), which resolves again with —Kills(w, Tuna) to yield the empty clause. Notice that w has two different bindings in this proof; resolution is telling us that, yes, someone killed Tuna—either Jack or Curiosity. This is no great surprise! One solution is to restrict the allowed resolution steps so that the query variables can be bound only once in a given proof; then we need to be able to backtrack over the possible bindings. Another solution is to add a special answer literal to the negated goal, which becomes —Kills(w, Tuna) V Answer(w). Now, the resolution process generates an answer whenever a clause is generated containing just a single answer literal. For the proof in Figure 9.12, this is Answer (Curiosity). The nonconstructive proof would generate the clause Answer (Curiosity) V Answer (Jack), which does not constitute an answer.

Completeness of resolution

are willing to take it on faith.

We will show that resolution is refutation-complete, which means that if a set of sentences is unsatisfiable, then resolution will always be able to derive a contradiction. Resolution cannot be used to generate all logical consequences of a set of sentences, but it can be used to establish that a given sentence is entailed by the set of sentences. Hence, it can be used to find all answers to a given question, using the negated-goal method that we described earlier in the Chapter.

We will take it as given that any sentence in first-order logic (without equality) can be rewritten as a set of clauses in CNF. This can be proved by induction on the form of the sentence, using atomic sentences as the base case (Davis and Putnam, 1960). Our goal therefore is to prove the following: if S is an unsatisfiable set of clauses, then the application of a finite number of resolution steps to S will yield a contradiction.

Our proof sketch follows the original proof due to Robinson, with some simplifications from Genesereth and Nilsson (1987). The basic structure of the proof is shown in Figure 9.13; it proceeds as follows:

1. First, we observe that if S is unsatisfiable, then there exists a particular set of ground instances of the clauses of .S such that this set is also unsatisfiable (Herbrand’s theorem).

2. We then appeal to the ground resolution theorem given in Chapter 7, which states that propositional resolution is complete for ground sentences.

3. We then use a lifting lemma to show that, for any propositional resolution proof using the set of ground sentences, there is a corresponding first-order resolution proof using the first-order sentences from which the ground sentences were obtained.

Section 9.5. Resolution 301

Any set of sentences S is [a in clausal form

Assume § is unsatisfiable, and in clausal form

-— Herbrand’s theorem Some set S’ of ground instances is unsatisfiable

CL Ground resolution

theorem

Resolution can find a contradiction in §”

cette

-t Lifting lemma

There is a resolution proof for the contradiction in S"

Figure 9.13 Structure of a completeness proof for resolution.

To carry out the first step, we will need three new concepts:

HERBRAND {> Herbrand universe: If S is a set of clauses, then Hg, the Herbrand universe of 5, is the set of all ground terms constructible from the following:

a. The function symbols in S, if any. b. The constant symbols in 5, if any; if none, then the constant symbol A. For example, if S contains just the clause P(x, F(z, A)) V =Q(z, A) V R(x, B), then Hyg is the following infinite set of ground terms: {A,B,F(A,A),F(A,B),F(B,A),F(B,B),F(A,F(A,A)),...}.

SATURATION $ Saturation: If S is a set of clauses and P is a set of ground terms, then P(S), the saturation of S with respect to P, is the set of all ground clauses obtained by applying all possible consistent substitutions of ground terms in P with variables in S.

HERBRAND BASE {> Herbrand base: The saturation of a set S of clauses with respect to its Herbrand universe is called the Herbrand base of S, written as Hg(S). For example, if S contains solely the clause just given, then Hg(S) is the infinite set of clauses

[]

~P(F(A, B), F(F(A,B), A)) Vv ~Q(F(A, B), A) V R(F(4, B), B), ...}

HERBRAND'S These definitions allow us to state a form of Herbrand’s theorem (Herbrand, 1930):

If a set S of clauses is unsatisfiable, then there exists a finite subset of Hg(.S) that is also unsatisfiable.

Let S’ be this finite subset of ground sentences. Now, we can appeal to the ground resolution theorem (page 217) to show that the resolution closure RC (S’) contains the empty clause. That is, running propositional resolution to completion on S’ will derive a contradiction. Now that we have established that there is always a resolution proof involving some finite subset of the Herbrand base of 5, the next step is to show that there is a resolution

302 Chapter 9. Inference in First-Order Logic

GODEL’S INCOMPLETENESS THEOREM

By slightly extending the language of first-order logic to allow for the mathematical induction schema in arithmetic, Godel was able to show, in his incompleteness theorem, that there are true arithmetic sentences that cannot be proved.

The proof of the incompleteness theorem is somewhat beyond the scope of this book, occupying, as it does, at least 30 pages, but we can give a hint here. We begin with the logical theory of numbers. In this theory, there is a single constant, 0, and a single function, S (the successor function). In the intended model, S(0) denotes 1, S(.S(0)) denotes 2, and so on; the language therefore has names for all the natural numbers. The vocabulary also includes the function symbols +, x, and Expt (exponentiation) and the usual set of logical connectives and quantifiers. The first step is to notice that the set of sentences that we can write in this language can be enumerated. (Imagine defining an alphabetical order on the symbols and then arranging, in alphabetical order, each of the sets of sentences of length 1, 2, and so on.) We can then number each sentence o with a unique natural number #c (the Godel number). This is crucial: number theory contains a name for each of its own sentences. Similarly, we can number each possible proof PP with a Godel number G(P), because a proof is simply a finite sequence of sentences.

Now suppose we have a recursively enumerable set A of sentences that are true statements about the natural numbers. Recalling that A can be named by a given set of integers, we can imagine writing in our language a sentence «(j, A) of the following sort:

Vi¢ 4 is not the Godel number of a proof of the sentence whose Godel number is j, where the proof uses only premises in A.

Then let o be the sentence a(#0, A), that is, a sentence that states its own unprovability from A. (That this sentence always exists is true but not entirely obvious.)

Now we make the following ingenious argument: Suppose that ¢ is provable from A; then o is false (because o says it cannot be proved). But then we have a false sentence that is provable from A, so A cannot consist of only true sentences— a violation of our premise. Therefore o is not provable from A. But this is exactly what ¢ itself claims; hence ¢ is a true sentence.

So, we have shown (barring 291 pages) that for any set of true sentences of number theory, and in particular any set of basic axioms, there are other true sentences that cannot be proved from those axioms. This establishes, among other things, that we can never prove all the theorems of mathematics within any given system of axioms. Clearly, this was an important discovery for mathematics. Its significance for Al has been widely debated, beginning with speculations by Godel himself. We take up the debate in Chapter 26.

Resolution 303

LIFTING LEMMA

proof using the clauses of S itself, which are not necessarily ground clauses. We start by considering a single application of the resolution rule. Robinson’s basic lemma implies the following fact:

Let C; and C5 be two clauses with no shared variables, and let C{ and C7 be ground instances of C1 and Cs. If C’ is a resolvent of C] and CY, then there exists a clause C such that (1) C is a resolvent of C; and Cs and (2) C’ is a ground instance of C'.

This is called a lifting lemma, because it lifts a proof step from ground clauses up to general first-order clauses. In order to prove his basic lifting lemma, Robinson had to invent unification and derive all of the properties of most general unifiers. Rather than repeat the proof here, we simply illustrate the lemma:

C1 = —P(z, F(z, A) Vv ~Q(z, A) V R(z, B) Cz = =N(G(y), 2) v P(H(y

( );2) Cl = ~P(H(B),F(H(B), A) Vv ~Q(H(B), A) v R(H(B), B) Cy = ~N(G(B),F(H(B), A) v P(H(B), F(H(B), 4)) C' = ~N(G(B),F(H(B),A)) Vv ~Q(H(B), A) v R(H(B), B) C = -N(G(y), F(H(y), A) Vv -Q(H(y), A) V R(H(y), B)

We see that indeed C’ is a ground instance of C. In general, for C] and C to have any resolvents, they must be constructed by first applying to C; and Cs the most general unifier of a pair of complementary literals in C; and C5. From the lifting lemma, it is easy to derive a similar statement about any sequence of applications of the resolution rule:

For any clause C’ in the resolution closure of S’ there is a clause C in the resolution closure of S, such that C’ is a ground instance of C and the derivation of C is the same length as the derivation of C".

From this fact, it follows that if the empty clause appears in the resolution closure of 5’, it must also appear in the resolution closure of S. This is because the empty clause cannot be a ground instance of any other clause. To recap: we have shown that if .S is unsatisfiable, then there is a finite derivation of the empty clause using the resolution rule.

The lifting of theorem proving from ground clauses to first-order clauses provides a vast increase in power. This increase comes from the fact that the first-order proof need instantiate variables only as far as necessary for the proof, whereas the ground-clause methods were required to examine a huge number of arbitrary instantiations.

Dealing with equality

None of the inference methods described so far in this chapter handle equality. There are three distinct approaches that can be taken. The first approach is to axiomatize equality—to write down sentences about the equality relation in the knowledge base. We need to say that equality is reflexive, symmetric, and transitive, and we also have to say that we can substitute

304

Chapter 9. Inference in First-Order Logic

DEMODULATION

PARAMODULATION

EQUATIONAL UNIFICATION

equals for equals in any predicate or function. So we need three basic axioms, and then one for each predicate and function:

Yr z==x

Ye, y t=y = y==z

Ya, y,2 T=yNy=2 => x==2

Vo,y z=y = (P(x) & Pi(y))

Vo,y e=y = (P(x) © Py)

Vw,z,y, 2 wW=yANr=z = (Fi(w,z)=Fi(y, 2) Vw, z,y,2 w=yAz=z = (Fa(w,z)=F(y, 2))

Given these sentences, a standard inference procedure such as resolution can perform tasks requiring equality reasoning, such as solving mathematical equations.

Another way to deal with equality is with an additional inference rule. The simplest rule, demodulation, takes a unit clause x = y and substitutes y for any term that unifies with x in some other clause. More formally, we have

¢& Demodulation: For any terms x, y, and 2, where UNIFY (2,2) = 6 and my[2] is a

literal containing 2:

z=vy, m1 VV mg[2]

my VV my, [SUBST(6,y)] Demodulation is typically used for simplifying expressions using collections of assertions such as z + 0=z, 2! =z, and so on. The rule can also be extended to handle non-unit clauses in which an equality literal appears:

¢ Paramodulation: For any terms z, y, and 2, where UNIFY(z, 2) = 6, LV Vi Vr=y, mi VV impylz] SUBST(8,£1V --VL VmiV---Vmyly]) Unlike demodulation, paramodulation yields a complete inference procedure for first-order logic with equality.

A third approach handles equality reasoning entirely within an extended unification algorithm. That is, terms are unifiable if they are provably equal under some substitution, where “provably” allows for some amount of equality reasoning. For example, the terms 1 + 2 and 2 + 1 normally are not unifiable, but a unification algorithm that knows that x + y =v + x could unify them with the empty substitution. Equational unification of this kind can be done with efficient algorithms designed for the particular axioms used (commutativity, associativity, and so on), rather than through explicit inference with those axioms. Theorem provers using this technique are closely related to the constraint logic programming systems described in Section 9.4.

Resolution strategies

We know that repeated applications of the resolution inference rule will eventually find a proof if one exists. In this subsection, we examine strategies that help find proofs efficiently.

Section 9.5.

Resolution 305

UNIT RESOLUTION

SET OF SUPPORT

INPUT RESOLUTION

LINEAR RESOLUTION

Unit preference

This strategy prefers to do resolutions where one of the sentences is a single literal (also known as a unit clause). The idea behind the strategy is that we are trying to produce an empty clause, so it might be a good idea to prefer inferences that produce shorter clauses. Resolving a unit sentence (such as P) with any other sentence (such as =P Vv —-Q V R) always yields a clause (in this case, =} V R) that is shorter than the other clause. When the unit preference strategy was first tried for propositional inference in 1964, it led to a dramatic speedup, making it feasible to prove theorems that could not be handled without the preference. Unit preference by itself does not, however, reduce the branching factor in medium-sized problems enough to make them solvable by resolution. It is, nonetheless, a useful heuristic that can be combined with other strategies.

Unit resolution is a restricted form of resolution in which every resolution step must involve a unit clause. Unit resolution is incomplete in general, but complete for Horn knowledge bases. Unit resolution proofs on Horn knowledge bases resemble forward chaining.

Set of support

Preferences that try certain resolutions first are helpful, but in general it is more effective to try to eliminate some potential resolutions altogether. The set-of-support strategy does just that. It starts by identifying a subset of the sentences called the set of support. Every resolution combines a sentence from the set of support with another sentence and adds the resolvent into the set of support. If the set of support is small relative to the whole knowledge base, the search space will be reduced dramatically.

We have to be careful with this approach, because a bad choice for the set of support will make the algorithm incomplete. However, if we choose the set of support .S so that the remainder of the sentences are jointly satisfiable, then set-of-support resolution will be complete. A common approach is to use the negated query as the set of support, on the assumption that the original knowledge base is consistent. (After all, if it is not consistent, then the fact that the query follows from it is vacuous.) The set-of-support strategy has the additional advantage of generating proof trees that are often easy for humans to understand, because they are goal-directed.

Input resolution

In the input resolution strategy, every resolution combines one of the input sentences (from the KB or the query) with some other sentence. The proof in Figure 9.11 uses only input resolutions and has the characteristic shape of a single “spine” with single sentences combining onto the spine. Clearly, the space of proof trees of this shape is smaller than the space of all proof graphs. In Horn knowledge bases, Modus Ponens is a kind of input resolution strategy, because it combines an implication from the original KB with some other sentences. Thus, it is no surprise that input resolution is complete for knowledge bases that are in Horn form, but incomplete in the general case. The linear resolution strategy is a slight generalization that allows P and @) to be resolved together either if P is in the original KB or if P is an ancestor of (J in the proof tree. Linear resolution is complete.

306

Chapter 9. Inference in First-Order Logic

SUBSUMPTION

Subsumption

The subsumption method eliminates all sentences that are subsumed by (i.e., more specific than) an existing sentence in the KB. For example, if P(z) is in the KB, then there is no sense in adding P(A) and even less sense in adding P(A) V Q(B). Subsumption helps keep the KB small, and thus helps keep the search space small.

Theorem provers

Theorem provers (also known as automated reasoners) differ from logic programming languages in two ways. First, most logic programming languages handle only Horn clauses, whereas theorem provers accept full first-order logic. Second, Prolog programs intertwine logic and control. The programmer’s choice A :B, C instead of A :C, B affects the execution of the program. In most theorem provers, the syntactic form chosen for sentences does not affect the results. Theorem provers still need control information to operate efficiently, but that information is usually kept distinct from the knowledge base, rather than being part of the knowledge representation itself. Most of the research in theorem provers involves finding control strategies that are generally useful, as well as increasing the speed.

Design of a theorem prover

In this section, we describe the theorem prover OTTER (Organized Techniques for Theoremproving and Effective Research) (McCune, 1992), with particular attention to its control strategy. In preparing a problem for OTTER, the user must divide the knowledge into four parts:

e A set of clauses known as the set of support (or sos), which defines the important facts about the problem. Every resolution step resolves a member of the set of support against another axiom, so the search is focused on the set of support.

e A set of usable axioms that are outside the set of support. These provide background knowledge about the problem area. The boundary between what is part of the problem (and thus in sos) and what is background (and thus in the usable axioms) is up to the user’s judgment.

e A set of equations known as rewrites or demodulators. Although demodulators are equations, they are always applied in the left to right direction. Thus, they define a canonical form into which all terms will be simplified. For example, the demodulator z + 0 =z says that every term of the form x + 0 should be replaced by the term z.

e A set of parameters and clauses that defines the control strategy. In particular, the user specifies a heuristic function to control the search and a filtering function to eliminate some subgoals as uninteresting.

OTTER works by continually resolving an element of the set of support against one of the usable axioms. Unlike Prolog, it uses a form of best-first search. Its heuristic function measures the “weight” of each clause, where lighter clauses are preferred. The exact choice of heuristic is up to the user, but generally, the weight of a clause should be correlated with its size or difficulty. Unit clauses are treated as light; the search can thus be seen as a generalization of the unit preference strategy. At each step, OTTER moves the “lightest” clause in the

Section 9.5.

Resolution 307

of support to the usable list and adds to the set of support some immediate consequences of resolving the lightest clause with elements of the usable list. OTTER halts when it has found a refutation or when there are no more clauses in the set of support. The algorithm is shown in more detail in Figure 9.14.

procedure OTTER(sos, usable) inputs: sos, a set of support—clauses defining the problem (a global variable) usable, background knowledge potentially relevant to the problem

repeat clause «+ the lightest member of sos move clause from sos to usable PROCESS(INFER (clause, usable), sos) until sos = [] or a refutation has been found

function INFER( clause, usable) returns clauses

resolve clause with each member of usable return the resulting clauses after applying FILTER

procedure PROCESS( clauses, $08)

for each clause in clauses do clause — SIMPLIFY (clause) merge identical literals discard clause if it is a tautology sos «— [clause | sos] if clause has no literals then a refutation has been found if clause has one literal then look for unit refutation

Figure 9.14 Sketch of the OTTER theorem prover. Heuristic control is applied in the selection of the “lightest” clause and in the FILTER function that eliminates uninteresting clauses from consideration.

Extending Prolog

An alternative way to build a theorem prover is to start with a Prolog compiler and extend it to get a sound and complete reasoner for full first-order logic. This was the approach taken in the Prolog Technology Theorem Prover, or PTTP (Stickel, 1988). PTTP includes five significant changes to Prolog to restore completeness and expressiveness:

e The occurs check is put back into the unification routine to make it sound.

o The depth-first search is replaced by an iterative deepening search. This makes the search strategy complete and takes only a constant factor more time.

eo Negated literals (such as —P(z)) are allowed. In the implementation, there are two separate routines, one trying to prove FP and one trying to prove —P.

308

Chapter 9. Inference in First-Order Logic

LOCKING

PROOF-CHECKER

SOCRATIC REASONER

o A clause with n atoms is stored as n different rules. For example, A <= B A C would also be stored as = B <= C' A —A and as -C «<= B A —A. This technique, known as locking, means that the current goal need be unified with only the head of each clause, yet it still allows for proper handling of negation.

e Inference is made complete (even for non-Horn clauses) by the addition of the linear input resolution rule: If the current goal unifies with the negation of one of the goals on the stack, then that goal can be considered solved. This is a way of reasoning by contradiction. Suppose the original goal is P and this is reduced by a series of inferences to the goal =P. This establishes that =F = P, which is logically equivalent to P.

Despite these changes, PTTP retains the features that make Prolog fast. Unifications are still done by modifying variables directly, with unbinding done by unwinding the trail during backtracking. The search strategy is still based on input resolution, meaning that every resolution is against one of the clauses given in the original statement of the problem (rather than a derived clause). This makes it feasible to compile all the clauses in the original statement of the problem.

The main drawback of PTTP is that the user has to relinquish all control over the search for solutions. Each inference rule is used by the system both in its original form and in the contrapositive form. This can lead to unintuitive searches. For example, consider the rule

(f(z. y) = fla.b)) = (x=a) A (y=b).

As a Prolog rule, this is a reasonable way to prove that two f terms are equal. But PTTP would also generate the contrapositive:

(x # a) < (f(z,y) # fla. 0) Ay =D) .

It seems that this is a wasteful way to prove that any two terms x and a are different.

Theorem provers as assistants

So far, we have thought of a reasoning system as an independent agent that has to make decisions and act on its own. Another use of theorem provers is as an assistant, providing advice to, say, a mathematician. In this mode the mathematician acts as a supervisor, mapping out the strategy for determining what to do next and asking the theorem prover to fill in the details. This alleviates the problem of semi-decidability to some extent, because the supervisor can cancel a query and try another approach if the query is taking too much time. A theorem prover can also act as a proof-checker, where the proof is given by a human as a series of fairly large steps; the individual inferences required to show that each step is sound are filled in by the system.

A Socratic reasoner is a theorem prover whose ASK function is incomplete, but which can always arrive at a solution if asked the right series of questions. Thus, Socratic reasoners make good assistants, provided that there is a supervisor to make the right series of calls to ASK. ONTIC (McAllester, 1989) is a Socratic reasoning system for mathematics.

Resolution 309

ROBBINS ALGEBRA

VERIFICATION SYNTHESIS

DEDUCTIVE SYNTHESIS

Practical uses of theorem provers

Theorem provers have come up with novel mathematical results. The SAM (Semi-Automated Mathematics) program was the first, proving a lemma in lattice theory (Guard ef al., 1969). The AURA program has also answered open questions in several areas of mathematics (Wos and Winker, 1983). The Boyer—Moore theorem prover (Boyer and Moore, 1979) has been used and extended over many years and was used by Natarajan Shankar to give the first fully rigorous formal proof of Godel’s Incompleteness Theorem (Shankar, 1986). The OTTER program is one of the strongest theorem provers; it has been used to solve several open questions in combinatorial logic. The most famous of these concerns Robbins algebra. In 1933, Herbert Robbins proposed a simple set of axioms that appeared to define Boolean algebra, but no proof of this could be found (despite serious work by several mathematicians including Alfred Tarski himself). On October 10, 1996, after eight days of computation, EQP (a version of OTTER) found a proof (McCune, 1997).

Theorem provers can be applied to the problems involved in the verification and synthesis of both hardware and software, because both domains can be given correct axiomatizations. Thus, theorem proving research is carried out in the fields of hardware design, programming languages, and software engineering—not just in Al. In the case of software, the axioms state the properties of each syntactic element of the programming language. (Reasoning about programs is quite similar to reasoning about actions in the situation calculus.) An algorithm is verified by showing that its outputs meet the specifications for all inputs. The RSA public key encryption algorithm and the Boyer—-Moore string-matching algorithm have been verified this way (Boyer and Moore, 1984). In the case of hardware, the axioms describe the interactions between signals and circuit elements. (See Chapter 8 for an example.) The design of a 16-bit adder has been verified by AURA (Wojcik, 1983). Logical reasoners designed specially for verification have been able to verify entire CPUs, including their timing properties (Srivas and Bickford, 1990).

The formal synthesis of algorithms was one of the first uses of theorem provers, as outlined by Cordell Green (1969a), who built on earlier ideas by Simon (1963). The idea is to prove a theorem to the effect that “there exists a program p satisfying a certain specification.” If the proof is constrained to be constructive, the program can be extracted. Although fully automated deductive synthesis, as it is called, has not yet become feasible for general-purpose programming, hand-guided deductive synthesis has been successful in designing several novel and sophisticated algorithms. Synthesis of special-purpose programs is also an active area of research. In the area of hardware synthesis, the AURA theorem prover has been applied to design circuits that are more compact than any previous design (Wojciechowski and Wojcik, 1983). For many circuit designs, propositional logic is sufficient because the set of interesting propositions is fixed by the set of circuit elements. The application of propositional inference in hardware synthesis is now a standard technique having many large-scale deployments (see, e.g., Nowick ef al. (1993)).

These same techniques are now starting to be applied to software verification as well, by systems such as the SPIN model checker (Holzmann, 1997). For example, the Remote Agent spacecraft control program was verified before and after flight (Havelund et al., 2000).

310

Chapter 9. Inference in First-Order Logic

We have presented an analysis of logical inference in first-order logic and a number of algorithms for doing it.

e A first approach uses inference rules for instantiating quantifiers in order to propositionalize the inference problem. Typically, this approach is very slow.

e The use of unification to identify appropriate substitutions for variables eliminates the instantiation step in first-order proofs, making the process much more efficient.

e A lifted version of Modus Ponens uses unification to provide a natural and powerful inference rule, generalized Modus Ponens. The forward chaining and backward chaining algorithms apply this rule to sets of definite clauses.

e Generalized Modus Ponens is complete for definite clauses, although the entailment problem is semidecidable. For Datalog programs consisting of function-free definite clauses, entailment is decidable.

e Forward chaining is used in deductive databases, where it can be combined with relational database operations. It is also used in production systems, which perform efficient updates with very large rule sets.

e Forward chaining is complete for Datalog programs and runs in polynomial time.

e Backward chaining is used in logic programming systems such as Prolog, which employ sophisticated compiler technology to provide very fast inference.

e Backward chaining suffers from redundant inferences and infinite loops; these can be alleviated by memoization.

e The generalized resolution inference rule provides a complete proof system for firstorder logic, using knowledge bases in conjunctive normal form.

e Several strategies exist for reducing the search space of a resolution system without compromising completeness. Efficient resolution-based theorem provers have been used to prove interesting mathematical theorems and to verity and synthesize software and hardware.

SYLLOGISM

Logical inference was studied extensively in Greek mathematics. The type of inference most carefully studied by Aristotle was the syllogism, which is a kind of inference rule. Aristotle’s syllogisms did include elements of first-order logic, such as quantification, but were restricted to unary predicates. Syllogisms were categorized by “figures” and “moods,” depending on the order of the terms (which we would call predicates) in the sentences, the degree of generality (which we would today interpret through quantifiers) applied to each term, and whether each term is negated. The most fundamental syllogism is that of the first mood of the first figure:

Section 9.6.

Summary 311

All S are M. All M are P. Therefore, all S are P.

Aristotle tried to prove the validity of other syllogisms by “reducing” them to those of the first figure. He was much less precise in describing what this “reduction” should involve than he was in characterizing the syllogistic figures and moods themselves.

Gottlob Frege, who developed full first-order logic in 1879, based his system of inference on a large collection of logically valid schemas plus a single inference rule, Modus Ponens. Frege took advantage of the fact that the effect of an inference rule of the form “From P, infer ()” can be simulated by applying Modus Ponens to P along with a logically valid schema P = (). This “axiomatic” style of exposition, using Modus Ponens plus a number of logically valid schemas, was employed by a number of logicians after Frege; most notably, it was used in Principia Mathematica (Whitehead and Russell, 1910).

Inference rules, as distinct from axiom schemas, were the focus of the natural deduction approach, introduced by Gerhard Gentzen (1934) and by Stanislaw Jaskowski (1934). Natural deduction is called “natural” because it does not require conversion to (unreadable) normal form and because its inference rules are intended to appear natural to humans. Prawitz (1965) offers a book-length treatment of natural deduction. Gallier (1986) uses Gentzen’s approach to expound the theoretical underpinnings of automated deduction.

The invention of clausal form was a crucial step in the development of a deep mathematical analysis of first-order logic. Whitehead and Russell (1910) expounded the so-called rules of passage (the actual term is from Herbrand (1930)) that are used to move quantifiers to the front of formulas. Skolem constants and Skolem functions were introduced, appropriately enough, by Thoralf Skolem (1920). The general procedure for skolemization is given by Skolem (1928), along with the important notion of the Herbrand universe.

Herbrand’s theorem, named after the French logician Jacques Herbrand (1930), has played a vital role in the development of automated reasoning methods, both before and after Robinson’s introduction of resolution. This is reflected in our reference to the “Herbrand universe” rather than the “Skolem universe,” even though Skolem really invented the concept. Herbrand can also be regarded as the inventor of unification. Godel (1930) built on the ideas of Skolem and Herbrand to show that first-order logic has a complete proof procedure. Alan Turing (1936) and Alonzo Church (1936) simultaneously showed, using very different proofs, that validity in first-order logic was not decidable. The excellent text by Enderton (1972) explains all of these results in a rigorous yet moderately understandable fashion.

Although McCarthy (1958) had suggested the use of first-order logic for representation and reasoning in Al, the first such systems were developed by logicians interested in mathematical theorem proving. It was Abraham Robinson who proposed the use of propositionalization and Herbrand’s theorem, and Gilmore (1960) who wrote the first program based on this approach. Davis and Putnam (1960) used clausal form and produced a program that attempted to find refutations by substituting members of the Herbrand universe for variables to produce ground clauses and then looking for propositional inconsistencies among the ground clauses. Prawitz (1960) developed the key idea of letting the quest for propositional

312

Chapter 9. Inference in First-Order Logic

inconsistency drive the search process, and generating terms from the Herbrand universe only when it was necessary to do so in order to establish propositional inconsistency. After further development by other researchers, this idea led J. A. Robinson (no relation) to develop the resolution method (Robinson, 1965). The so-called inverse method developed at about the same time by the Soviet researcher S. Maslov (1964, 1967), based on somewhat different principles, offers similar computational advantages over propositionalization. Wolfgang Bibel’s (1981) connection method can be viewed as an extension of this approach.

After the development of resolution, work on first-order inference proceeded in several different directions. In Al, resolution was adopted for question-answering systems by Cordell Green and Bertram Raphael (1968). A somewhat less formal approach was taken by Carl Hewitt (1969). His PLANNER language, although never fully implemented, was a precursor to logic programming and included directives for forward and backward chaining and for negation as failure. A subset known as MICRO-PLANNER (Sussman and Winograd, 1970) was implemented and used in the SHRDLU natural language understanding system (Winograd, 1972). Early Al implementations put a good deal of effort into data structures that would allow efficient retrieval of facts; this work is covered in Al programming texts (Charniak et al., 1987; Norvig, 1992; Forbus and de Kleer, 1993).

By the early 1970s, forward chaining was well established in Al as an easily understandable alternative to resolution. It was used in a wide variety of systems, ranging from Nevins’s geometry theorem prover (Nevins, 1975) to the R1 expert system for VAX configuration (McDermott, 1982). AI applications typically involved large numbers of rules, so it was important to develop efficient rule-matching technology, particularly for incremental updates. The technology for production systems was developed to support such applications. The production system language OPS-5 (Forgy, 1981; Brownston et al., 1985) was used for R1 and for the SOAR cognitive architecture (Laird ef al., 1987). OPS-5 incorporated the rete match process (Forgy, 1982). SOAR, which generates new rules to cache the results of previous computations, can create very large rule sets—over 1,000,000 rules in the case of the TACAIR-SOAR system for controlling simulated fighter aircraft (Jones et al., 1998). CLIPS (Wygant, 1989) was a C-based production system language developed at NASA that allowed better integration with other software, hardware, and sensor systems and was used for spacecraft automation and several military applications.

The area of research known as deductive databases has also contributed a great deal to our understanding of forward inference. It began with a workshop in Toulouse in 1977, organized by Jack Minker, that brought together experts in logical inference and database systems (Gallaire and Minker, 1978). A recent historical survey (Ramakrishnan and Ullman, 1995) says, “Deductive [database] systems are an attempt to adapt Prolog, which has a ‘small data’ view of the world, to a ‘large data’ world.” Thus, it aims to meld relational database technology, which is designed for retrieving large sets of facts, with Prolog-based inference technology, which typically retrieves one fact at a time. Texts on deductive databases include Ullman (1989) and Ceri ef al. (1990).

Influential work by Chandra and Harel (1980) and Ullman (1985) led to the adoption of Datalog as a standard language for deductive databases. “Bottom-up” inference, or forward chaining, also became the standard—partly because it avoids the problems with nontermi

Summary 313

CONJUNCTIVE QUERY

DATA COMPLEXITY

SL-RESOLUTION

SLD-RESOLUTION

nation and redundant computation that occur with backward chaining and partly because it has a more natural implementation in terms of the basic relational database operations. The development of the magic sets technique for rule rewriting by Bancilhon ef al. (1986) allowed forward chaining to borrow the advantage of goal-directedness from backward chaining. Equalizing the arms race, tabled logic programming methods (see page 313) borrow the advantage of dynamic programming from forward chaining.

Much of our understanding of the complexity of logical inference has come from the deductive database community. Chandra and Merlin (1977) first showed that matching a single nonrecursive rule (a conjunctive query in database terminology) can be NP-hard. Kuper and Vardi (1993) proposed data complexity—that is, complexity as a function of database size, viewing rule size as constant—as a suitable measure for query answering. Gottlob et al. (1999b) discuss the connection between conjunctive queries and constraint satisfaction, showing how hypertree decomposition can optimize the matching process.

As mentioned earlier, backward chaining for logical inference appeared in Hewitt’s PLANNER language (1969). Logic programming per se evolved independently of this effort. A restricted form of linear resolution called SL-resolution was developed by Kowalski and Kuehner (1971), building on Loveland’s model elimination technique (1968); when applied to definite clauses, it becomes SLD-resolution, which lends itself to the interpretation of definite clauses as programs (Kowalski, 1974, 1979a, 1979b). Meanwhile, in 1972, the French researcher Alain Colmerauer had developed and implemented Prolog for the purpose of parsing natural language—Prolog’s clauses were intended initially as context-free grammar rules (Roussel, 1975; Colmerauer et al., 1973). Much of the theoretical background for logic programming was developed by Kowalski, working with Colmerauer. The semantic definition using least fixed points is due to Van Emden and Kowalski (1976). Kowalski (1988) and Cohen (1988) provide good historical overviews of the origins of Prolog. Foundations of Logic Programming (Lloyd, 1987) is a theoretical analysis of the underpinnings of Prolog and other logic programming languages.

Efficient Prolog compilers are generally based on the Warren Abstract Machine (WAM) model of computation developed by David H. D. Warren (1983). Van Roy (1990) showed that the application of additional compiler techniques, such as type inference, made Prolog programs competitive with C programs in terms of speed. The Japanese Fifth Generation project, a 10-year research effort beginning in 1982, was based completely on Prolog as the means to develop intelligent systems.

Methods for avoiding unnecessary looping in recursive logic programs were developed independently by Smith et al. (1986) and Tamaki and Sato (1986). The latter paper also included memoization for logic programs, a method developed extensively as tabled logic programming by David S. Warren. Swift and Warren (1994) show how to extend the WAM to handle tabling, enabling Datalog programs to execute an order of magnitude faster than forward-chaining deductive database systems.

Early theoretical work on constraint logic programming was done by Jaffar and Lassez (1987). Jaffar et al. (1992a) developed the CLP(R) system for handling real-valued constraints. Jaffar ef al. (1992b) generalized the WAM to produce the CLAM (Constraint Logic Abstract Machine) for specifying implementations of CLP. Ait-Kaci and Podelski (1993)

314

Chapter 9. Inference in First-Order Logic

describe a sophisticated language called LIFE, which combines CLP with functional programming and with inheritance reasoning. Kohn (1991) describes an ambitious project to use constraint logic programming as the foundation for a real-time control architecture, with applications to fully automatic pilots.

There are a number of textbooks on logic programming and Prolog. Logic for Problem Solving (Kowalski, 1979b) is an early text on logic programming in general. Prolog texts include Clocksin and Mellish (1994), Shoham (1994), and Bratko (2001). Marriott and Stuckey (1998) provide excellent coverage of CLP. Until its demise in 2000, the Journal of Logic Programming was the journal of record; it has now been replaced by Theory and Practice of Logic Programming. Logic programming conferences include the International Conference on Logic Programming (ICLP) and the International Logic Programming Symposium (ILPS).

Research into mathematical theorem proving began even before the first complete first-order systems were developed. Herbert Gelernter’s Geometry Theorem Prover (Gelernter, 1959) used heuristic search methods combined with diagrams for pruning false subgoals and was able to prove some quite intricate results in Euclidean geometry. Since that time, however, there has not been very much interaction between theorem proving and Al.

Early work concentrated on completeness. Following Robinson’s seminal paper, the demodulation and paramodulation rules for equality reasoning were introduced by Wos ef al. (1967) and Wos and Robinson (1968), respectively. These rules were also developed independently in the context of term rewriting systems (Knuth and Bendix, 1970). The incorporation of equality reasoning into the unification algorithm is due to Gordon Plotkin (1972); it was also a feature of QLISP (Sacerdoti et al., 1976). Jouannaud and Kirchner (1991) survey equational unification from a term rewriting perspective. Efficient algorithms for standard unification were developed by Martelli and Montanari (1976) and Paterson and Wegman (1978).

In addition to equality reasoning, theorem provers have incorporated a variety of specialpurpose decision procedures. Nelson and Oppen (1979) proposed an influential scheme for integrating such procedures into a general reasoning system; other methods include Stickel’s (1985) “theory resolution” and Manna and Waldinger’s (1986) “special relations.”

A number of control strategies have been proposed for resolution, beginning with the unit preference strategy (Wos ef al., 1964). The set of support strategy was proposed by Wos et al. (1965), to provide a degree of goal-directedness in resolution. Linear resolution first appeared in Loveland (1970). Genesereth and Nilsson (1987, Chapter 5) provide a short but thorough analysis of a wide variety of control strategies.

Guard ef al. (1969) describe the early SAM theorem prover, which helped to solve an open problem in lattice theory. Wos and Winker (1983) give an overview of the contributions of the AURA theorem prover toward solving open problems in various areas of mathematics and logic. McCune (1992) follows up on this, recounting the accomplishments of AURA’s successor, OTTER, in solving open problems. Weidenbach (2001) describes SPASS, one of the strongest current theorem provers. A Computational Logic (Boyer and Moore, 1979) is the basic reference on the Boyer-Moore theorem prover. Stickel (1988) covers the Prolog Technology Theorem Prover (PTTP), which combines the advantages of Prolog compilation with the completeness of model elimination (Loveland, 1968). SETHEO (Letz ert al., 1992) is another widely used theorem prover based on this approach; it can perform several million

Section 9.6. Summary 315

inferences per second on 2000-model workstations. LEANTAP (Beckert and Posegga, 1995) is an efficient theorem prover implemented in only 25 lines of Prolog.

Early work in automated program synthesis was done by Simon (1963), Green (1969a), and Manna and Waldinger (1971). The transformational system of Burstall and Darlington (1977) used equational reasoning for recursive program synthesis. KIDS (Smith, 1990, 1996) is one of the strongest modern systems; it operates as an expert assistant. Manna and Waldinger (1992) give a tutorial introduction to the current state of the art, with emphasis on their own deductive approach. Automating Software Design (Lowry and McCartney, 1991) collects a number of papers in the area. The use of logic in hardware design is surveyed by Kern and Greenstreet (1999); Clarke ef al. (1999) cover model checking for hardware verification.

Computability and Logic (Boolos and Jeffrey, 1989) is a good reference on completeness and undecidability. Many early papers in mathematical logic are to be found in From Frege to Godel: A Source Book in Mathematical Logic (van Heijenoort, 1967). The journal of record for the field of pure mathematical logic (as opposed to automated deduction) is The Journal of Symbolic Logic. Textbooks geared toward automated deduction include the classic Symbolic Logic and Mechanical Theorem Proving (Chang and Lee, 1973), as well as more recent works by Wos et al. (1992), Bibel (1993), and Kaufmann ef al. (2000). The anthology Automation of Reasoning (Siekmann and Wrightson, 1983) includes many important early papers on automated deduction. Other historical surveys have been written by Loveland (1984) and Bundy (1999). The principal journal for the field of theorem proving is the Journal of Automated Reasoning; the main conference is the annual Conference on Automated Deduction (CADE). Research in theorem proving is also strongly related to the use of logic in analyzing programs and programming languages, for which the principal conference is Logic in Computer Science.

EXERCISES

9.1 Prove from first principles that Universal Instantiation is sound and that Existential Instantiation produces an inferentially equivalent knowledge base.

9.2 From Likes(Jerry, IceCream) it seems reasonable to infer 3x Likes(z, IceCream). RAN Write down a general inference rule, Existential Introduction, that sanctions this inference. State carefully the conditions that must be satisfied by the variables and terms involved.

9.3 Suppose a knowledge base contains just one sentence, Ix AsHighAs(x, Everest). Which of the following are legitimate results of applying Existential Instantiation?

a. AsHighAs(Fverest, Everest).

b. AsHighAs(Kilimanjaro, Everest).

c. AsHighAs(Kilimanjaro, Everest) A AsHighAs(BenNeuvis, Everest) (after two applications).

316

Chapter 9. Inference in First-Order Logic

9.4 For each pair of atomic sentences, give the most general unifier if it exists: P(A,B,B), P(z,y,z).

Qy, G(4, B)), Q(G(z, x), y).

Older (Father(y),y), Older(Father(z), John).

. Knows(Father(y),y), Knows(x, x).

Bap Tw

9.5 Consider the subsumption lattices shown in Figure 9.2.

a. Construct the lattice for the sentence Employs(Mother (John), Father (Richard)).

b. Construct the lattice for the sentence Employs(IBM ,y) (“Everyone works for IBM”). Remember to include every kind of query that unifies with the sentence.

¢. Assume that STORE indexes each sentence under every node in its subsumption lattice. Explain how FETCH should work when some of these sentences contain variables; use as examples the sentences in (a) and (b) and the query Employs(z, Father(x)).

9.6 Suppose we put into a logical database a segment of the U.S. census data listing the age, city of residence, date of birth, and mother of every person, using social security numbers as identifying constants for each person. Thus, George's age is given by Age(443-65-1282, 56). Which of the indexing schemes S1-S5 following enable an efficient solution for which of the queries Q1-Q4 (assuming normal backward chaining)?

¢ S1: an index for each atom in each position.

¢ S82: an index for each first argument.

¢ 83: an index for each predicate atom.

> S4: an index for each combination of predicate and first argument.

{> S85: an index for each combination of predicate and second argument and an index for

each first argument (nonstandard).

& Ql: Age(443-44-4321, x)

$ Q2: ResidesIn(x, Houston)

$ Q3: Mother(z,y)

$ Q4: Age(x, 34) A ResidesIn(z, Tiny TounUSA)

9.7 One might suppose that we can avoid the problem of variable conflict in unification during backward chaining by standardizing apart all of the sentences in the knowledge base once and for all. Show that, for some sentences, this approach cannot work. (Hint: Consider a sentence, one part of which unifies with another.)

9.8 Explain how to write any given 3-SAT problem of arbitrary size using a single first-order definite clause and no more than 30 ground facts.

9.9 Write down logical representations for the following sentences, suitable for use with Generalized Modus Ponens:

Section 9.6.

a. b. c. d e. f.

9.10

Horses, cows, and pigs are mammals.

An offspring of a horse is a horse. Bluebeard is a horse.

Bluebeard is Charlie’s parent.

Offspring and parent are inverse relations.

Every mammal has a parent.

In this question we will use the sentences you wrote in Exercise 9.9 to answer a ques

a.

ao

9.11

Draw the proof tree generated by an exhaustive backward-chaining algorithm for the query 3h Horse(h), where clauses are matched in the order given.

What do you notice about this domain? How many solutions for h actually follow from your sentences?

et al. (1986).)

A popular children’s riddle is “Brothers and sisters have I none, but that man’s father

9.12

Trace the execution of the backward chaining algorithm in Figure 9.6 when it is applied

9.13

The following Prolog code defines a predicate P:

P(X, [X]|Y]). P(X, [Y|2]) :P(X,2Z).

In this exercise, we will look at sorting in Prolog.

list L is sorted in ascending order.

is a permutation of M.

ity of your program? Write a faster sorting algorithm, such as insertion sort or quicksort, in Prolog.

318

Chapter 9. Inference in First-Order Logic

9.15 In this exercise, we will look at the recursive application of rewrite rules, using logic programming. A rewrite rule (or demodulator in OTTER terminology) is an equation with a specified direction. For example, the rewrite rule z+0 — x suggests replacing any expression that matches x + 0 with the expression z. The application of rewrite rules is a central part of equational reasoning systems. We will use the predicate rewrite (X,Y) to represent rewrite rules. For example, the earlier rewrite rule is written as rewrite (X+0, X). Some terms are primitive and cannot be further simplified; thus, we will write primitive (0) to say that 0 is a primitive term.

a. Write a definition of a predicate simplify (X,Y), that is true when Y is a simplified version of X—that is, when no further rewrite rules are applicable to any subexpression of Y.

b. Write a collection of rules for the simplification of expressions involving arithmetic operators, and apply your simplification algorithm to some sample expressions.

¢. Write a collection of rewrite rules for symbolic differentiation, and use them along with your simplification rules to differentiate and simplify expressions involving arithmetic expressions, including exponentiation.

9.16 In this exercise, we will consider the implementation of search algorithms in Prolog. Suppose that successor (X,Y) is true when state Y is a successor of state X; and that goal (X) is true when X is a goal state. Write a definition for solve (X, P), which means that P is a path (list of states) beginning with X, ending in a goal state, and consisting of a sequence of legal steps as defined by successor. You will find that depth-first search is the easiest way to do this. How easy would it be to add heuristic search control?

9.17 How can resolution be used to show that a sentence is valid? Unsatisfiable?

9.18 From “Horses are animals,” it follows that “The head of a horse is the head of an animal.” Demonstrate that this inference is valid by carrying out the following steps:

a. Translate the premise and the conclusion into the language of first-order logic. Use three predicates: Head Of (h, z) (meaning “h is the head of 2), Horse(z), and Animal(x).

b. Negate the conclusion, and convert the premise and the negated conclusion into conjunctive normal form.

¢. Use resolution to show that the conclusion follows from the premise.

9.19 Here are two sentences in the language of first-order logic: (A:Vz Jy (z>y) B):3y Vz (z2y) a. Assume that the variables range over all the natural numbers 0, 1,2, .. ., oo and that the “>” predicate means “is greater than or equal to.” Under this interpretation, translate (A) and (B) into English. b. Is (A) true under this interpretation?

Section 9.6.

Summary 319

¢. Is (B) true under this interpretation?

d. Does (A) logically entail (B)?

e. Does (B) logically entail (A)?

f. Using resolution, try to prove that (A) follows from (B). Do this even if you think that (B) does not logically entail (A); continue until the proof breaks down and you cannot proceed (if it does break down). Show the unifying substitution for each resolution step. If the proof fails, explain exactly where, how, and why it breaks down.

g. Now try to prove that (B) follows from (A).

9.20 Resolution can produce nonconstructive proofs for queries with variables, so we had to introduce special mechanisms to extract definite answers. Explain why this issue does not arise with knowledge bases containing only definite clauses.

9.21 We said in this chapter that resolution cannot be used to generate all logical consequences of a set of sentences. Can any algorithm do this?

1 () KNOWLEDGE REPRESENTATION

10.1

ONTOLOGICAL ENGINEERING

In which we show how to use first-order logic to represent the most important aspects of the real world, such as action, space, time, mental events, and shopping.

The last three chapters described the technology for knowledge-based agents: the syntax, semantics, and proof theory of propositional and first-order logic, and the implementation of agents that use these logics. In this chapter we address the question of what content to put into such an agent’s knowledge base—how to represent facts about the world.

Section 10.1 introduces the idea of a general ontology, which organizes everything in the world into a hierarchy of categories. Section 10.2 covers the basic categories of objects, substances, and measures. Section 10.3 discusses representations for actions, which are central to the construction of knowledge-based agents, and also explains the more general notion of events, or space—time chunks. Section 10.4 discusses knowledge about beliefs, and Section 10.5 brings all the knowledge together in the context of an Internet shopping environment. Sections 10.6 and 10.7 cover specialized reasoning systems for representing uncertain and changing knowledge.

In “toy” domains, the choice of representation is not that important; it is easy to come up with a consistent vocabulary. On the other hand, complex domains such as shopping on the Internet or controlling a robot in a changing physical environment require more general and flexible representations. This chapter shows how to create these representations, concentrating on general concepts—such as Actions, Time, Physical Objects, and Beliefs—that occur in many different domains. Representing these abstract concepts is sometimes called ontological engineering—it is related to the knowledge engineering process described in Section 8.4, but operates on a grander scale.

The prospect of representing everything in the world is daunting. Of course, we won’t actually write a complete description of everything—that would be far too much for even a 1000—page textbook—but we will leave placeholders where new knowledge for any domain can fit in. For example, we will define what it means to be a physical object, and the details

320

Section 10.1.

Ontological Engineering 321

UPPER ONTOLOGY

Anything AbstractObjects GeneralizedEvents TT TT Sets Numbers RepresentationalObjects Interval Places PhysicalObjects Processes NN | Categories Sentences Measurements Moments Things Stuff SN SN Times Weights Animals Agents Solid Liquid Gas

Humans

Figure 10.1 ~~ The upper ontology of the world, showing the topics to be covered later in the chapter. Each arc indicates that the lower concept is a specialization of the upper one.

of different types of objects—robots, televisions, books, or whatever—can be filled in later. The general framework of concepts is called an upper ontology, because of the convention of drawing graphs with the general concepts at the top and the more specific concepts below them, as in Figure 10.1.

Before considering the ontology further, we should state one important caveat. We have elected to use first-order logic to discuss the content and organization of knowledge. Certain aspects of the real world are hard to capture in FOL. The principal difficulty is that almost all generalizations have exceptions, or hold only to a degree. For example, although “tomatoes are red” is a useful rule, some tomatoes are green, yellow, or orange. Similar exceptions can be found to almost all the general statements in this chapter. The ability to handle exceptions and uncertainty is extremely important, but is orthogonal to the task of understanding the general ontology. For this reason, we will delay the discussion of exceptions until Section 10.6, and the more general topic of uncertain information until Chapter 13.

Of what use is an upper ontology? Consider again the ontology for circuits in Section 8.4. It makes a large number of simplifying assumptions. For example, time is omitted completely. Signals are fixed, and do not propagate. The structure of the circuit remains constant. If we wanted to make this more general, consider signals at particular times, and include the wire lengths and propagation delays. This would allow us to simulate the timing properties of the circuit, and indeed such simulations are often carried out by circuit designers. We could also introduce more interesting classes of gates, for example by describing the technology (TTL, MOS, CMOS, and so on) as well as the input/output specification. If we wanted to discuss reliability or diagnosis, we would include the possibility that the structure of the circuit or the properties of the gates might change spontaneously. To account for stray capacitances, we would need to move from a purely topological representation of connectivity to a more realistic description of geometric properties.

322

Chapter 10. Knowledge Representation

If we look at the wumpus world, similar considerations apply. Although we do include time, it has a very simple structure: Nothing happens except when the agent acts, and all changes are instantaneous. A more general ontology, better suited for the real world, would allow for simultaneous changes extended over time. We also used a Pit predicate to say which squares have pits. We could have allowed for different kinds of pits by having several individuals belonging to the class of pits, each having different properties. Similarly, we might want to allow for other animals besides wumpuses. It might not be possible to pin down the exact species from the available percepts, so we would need to build up a wumpusworld biological taxonomy to help the agent predict behavior from scanty clues.

For any special-purpose ontology, it is possible to make changes like these to move toward greater generality. An obvious question then arises: do all these ontologies converge on a general-purpose ontology? After centuries of philosophical and computational investigation, the answer is “Possibly.” In this section, we will present one version, representing a synthesis of ideas from those centuries. There are two major characteristics of generalpurpose ontologies that distinguish them from collections of special-purpose ontologies:

e A general-purpose ontology should be applicable in more or less any special-purpose domain (with the addition of domain-specific axioms). This means that, as far as possible, no representational issue can be finessed or brushed under the carpet.

¢ In any sufficiently demanding domain, different areas of knowledge must be unified, because reasoning and problem solving could involve several areas simultaneously. A robot circuit-repair system, for instance, needs to reason about circuits in terms of electrical connectivity and physical layout, and about time, both for circuit timing analysis and estimating labor costs. The sentences describing time therefore must be capable of being combined with those describing spatial layout and must work equally well for nanoseconds and minutes and for angstroms and meters.

After we present the general ontology we use it to describe the Internet shopping domain. This domain is more than adequate to exercise our ontology, and leaves plenty of scope for the reader to do some creative knowledge representation of his or her own. Consider for example that the Internet shopping agent must know about myriad subjects and authors to buy books at Amazon.com, about all sorts of foods to buy groceries at Peapod.com, and about everything one might find at a garage sale to hunt for bargains at Ebay.com.

CATEGORIES

The organization of objects into categories is a vital part of knowledge representation. Although interaction with the world takes place at the level of individual objects, much reasoning takes place at the level of categories. For example, a shopper might have the goal of buying a basketball, rather than a particular basketball such as BBg. Categories also serve to make predictions about objects once they are classified. One infers the presence of certain

1 We apologize if, due to circumstances beyond our control, some of these online stores are no longer functioning by the time you read this.

Section 10.2.

Categories and Objects 323

INHERITANCE

TAXONOMY

objects from perceptual input, infers category membership from the perceived properties of the objects, and then uses category information to make predictions about the objects. For example, from its green, mottled skin, large size, and ovoid shape, one can infer that an object is a watermelon; from this, one infers that it would be useful for fruit salad.

There are two choices for representing categories in first-order logic: predicates and objects. That is, we can use the predicate Basketball(b), or we can reify the category as an object, Basketballs. We could then say Member(b, Basketballs) (which we will abbreviate as b € Basketballs) to say that b is a member of the category of basketballs. We say Subset( Basketballs, Balls) (abbreviated as Basketballs C Balls) to say that Basketballs is a subcategory, or subset, of Balls. So you can think of a category as being the set of its members, or you can think of it as a more complex object that just happens to have the Member and Subset relations defined for it.

Categories serve to organize and simplify the knowledge base through inheritance. If we say that all instances of the category Food are edible, and if we assert that Fruit is a subclass of Food and Apples is a subclass of Fruit, then we know that every apple is edible. We say that the individual apples inherit the property of edibility, in this case from their membership in the Food category.

Subclass relations organize categories into a taxonomy, or taxonomic hierarchy. Taxonomies have been used explicitly for centuries in technical fields. For example, systematic biology aims to provide a taxonomy of all living and extinct species; library science has developed a taxonomy of all fields of knowledge, encoded as the Dewey Decimal system; and tax authorities and other government departments have developed extensive taxonomies of occupations and commercial products. Taxonomies are also an important aspect of general commonsense knowledge.

First-order logic makes it easy to state facts about categories, either by relating objects to categories or by quantifying over their members:

e An object is a member of a category. For example: BBgy € Basketballs e A category is a subclass of another category. For example: Basketballs C Balls e All members of a category have some properties. For example: x € Basketballs = Round(z) e Members of a category can be recognized by some properties. For example: Orange(z) A Round(x) A Diameter(x)=9.5" A x € Balls = x € Basketballs e A category as a whole has some properties. For example: Dogs € DomesticatedSpecies

Notice that because Dogs is a category and is a member of DomesticatedSpecies, the latter must be a category of categories. One can even have categories of categories of categories, but they are not much use.

Although subclass and member relations are the most important ones for categories, we also want to be able to state relations between categories that are not subclasses of each other. For example, if we just say that Males and Females are subclasses of Animals, then

324 Chapter 10. Knowledge Representation we have not said that a male cannot be a female. We say that two or more categories are DISJOINT disjoint if they have no members in common. And even if we know that males and females are disjoint, we will not know that an animal that is not a male must be a female, unless BON we say that males and females constitute an exhaustive decomposition of the animals. A PARTITION disjoint exhaustive decomposition is known as a partition. The following examples illustrate these three concepts: Disjoint({ Animals, Vegetables}) Exhaustive Decomposition({ Americans, Canadians, Mexicans}, NorthAmericans) Partition({ Males, Females}, Animals) . (Note that the EzhaustiveDecomposition of NorthAmericans is not a Partition, because some people have dual citizenship.) The three predicates are defined as follows: Disjoint(s) & (Veci,c0 c1€sAca€s Aer # ca = Intersection(ci,c2)={}) Exhaustive Decomposition(s,c) < (Vi i€c < Teco ca€sNi€cs) Partition(s,c) < Disjoint(s) A ExhaustiveDecomposition(s,c) . Categories can also be defined by providing necessary and sufficient conditions for membership. For example, a bachelor is an unmarried adult male: z € Bachelors < Unmarried(z) A x € Adults A x € Males . As we discuss in the sidebar on natural kinds, strict logical definitions for categories are neither always possible nor always necessary. Physical composition The idea that one object can be part of another is a familiar one. One’s nose is part of one’s head, Romania is part of Europe, and this chapter is part of this book. We use the general PartOf relation to say that one thing is part of another. Objects can be grouped into PartOf hierarchies, reminiscent of the Subset hierarchy: PartOf (Bucharest, Romania) PartOf (Romania, EasternEurope) PartOf (Eastern Europe, Europe) PartOf (Europe, Earth) . The PartOf relation is transitive and reflexive; that is, PartOf (x,y) A PartOf (y,z) = PartOf (x, 2) . PartOf (x, x) . Therefore, we can conclude PartOf (Bucharest, Earth). COMPOSITE OBJECT Categories of composite objects are often characterized by structural relations among

parts. For example, a biped has two legs attached to a body: Biped(a) = 3li,lo,b Leg(li) A Leg(l2) A Body(b) A PartOf (11, a) A PartOf (I2,a) A PartOf (b,a) A Attached(l1,b) A Attached (lz, b) A li #1ls N[Vig Leg(l3) A PartOf(l3,a) = (Is=1; Vig=1s)].

Section 10.2.

Categories and Objects 325

BUNCH

LOGICAL MINIMIZATION

MEASURES

UNITS FUNCTION

The notation for “exactly two” is a little awkward; we are forced to say that there are two legs, that they are not the same, and that if anyone proposes a third leg, it must be the same as one of the other two. In Section 10.6, we will see how a formalism called description logic makes it easier to represent constraints like “exactly two.”

We can define a PartPartition relation analogous to the Partition relation for categories. (See Exercise 10.6.) An object is composed of the parts in its PartPartition and can be viewed as deriving some properties from those parts. For example, the mass of a composite object is the sum of the masses of the parts. Notice that this is not the case with categories, which have no mass, even though their elements might.

It is also useful to define composite objects with definite parts but no particular structure. For example, we might want to say, “The apples in this bag weigh two pounds.” The temptation would be to ascribe this weight to the set of apples in the bag, but this would be a mistake because the set is an abstract mathematical concept that has elements but does not have weight. Instead, we need a new concept, which we will call a bunch. For example, if the apples are Apple, Apples, and Apples, then

BunchOf ({ Apple, Apples, Apples})

denotes the composite object with the three apples as parts (not elements). We can then use the bunch as a normal, albeit unstructured, object. Notice that BunchOf ({z}) = z. Furthermore, BunchOf (Apples) is the composite object consisting of all apples—not to be confused with Apples, the category or set of all apples.

We can define BunchOf in terms of the PartOf relation. Obviously, each element of s is part of BunchOf (s):

Va z€s = PartOf (xz, BunchOf(s)) .

Furthermore, BunchOf (s) is the smallest object satisfying this condition. In other words, BunchOf (s) must be part of any object that has all the elements of s as parts:

Vy Vz z€s = PartOf(z,y)] = PartOf (BunchOf(s),y) .

These axioms are an example of a general technique called logical minimization, which means defining an object as the smallest one satisfying certain conditions.

Measurements

In both scientific and commonsense theories of the world, objects have height, mass, cost, and so on. The values that we assign for these properties are called measures. Ordinary quantitative measures are quite easy to represent. We imagine that the universe includes abstract “measure objects,” such as the length that is the length of this line segment: 1 We can call this length 1.5 inches or 3.81 centimeters. Thus, the same length has different names in our language. Logically, this can be done by combining a units function with a number. (An alternative scheme is explored in Exercise 10.8.) If the line segment is called Lj, we can write

Length(L1) = Inches(1.5) = Centimeters(3.81) . Conversion between units is done by equating multiples of one unit to another: Centimeters(2.54 x d) = Inches(d) .

326 Chapter 10. Knowledge Representation

NATURAL KINDS

Some categories have strict definitions: an object is a triangle if and only if it is a polygon with three sides. On the other hand, most categories in the real world have no clear-cut definition; these are called natural kind categories. For example, tomatoes tend to be a dull scarlet; roughly spherical; with an indentation at the top where the stem was; about two to four inches in diameter; with a thin but tough skin; and with flesh, seeds, and juice inside. There is, however, variation: some tomatoes are orange, unripe tomatoes are green, some are smaller or larger than average, and cherry tomatoes are uniformly small. Rather than having a complete definition of tomatoes, we have a set of features that serves to identify objects that are clearly typical tomatoes, but might not be able to decide for other objects. (Could there be a tomato that is furry, like a peach?)

This poses a problem for a logical agent. The agent cannot be sure that an object it has perceived is a tomato, and even if it were sure, it could not be certain which of the properties of typical tomatoes this one has. This problem is an

fob fort ci inevitable consequence of operating in partially observable environments.

REE ANE One useful approach is to separate what is true of all instances of a cateapa bans gory from what is true only of typical instances. So in addition to the category SERPS 2 Tomatoes, we will also have the category Typical( Tomatoes). Here, the Typical Toe 21g: function maps a category to the subclass that contains only typical instances:

Typical(c) Cc. Most knowledge about natural kinds will actually be about their typical instances: x € Typical( Tomatoes) => Red(x) A Round(z) .

Thus, we can write down useful facts about categories without exact definitions.

The difficulty of providing exact definitions for most natural categories was explained in depth by Wittgenstein (1953), in his book Philosophical Investigations. He used the example of games to show that members of a category shared “family resemblances” rather than necessary and sufficient characteristics.

The utility of the notion of strict definition was also challenged by Quine (1953). He pointed out that even the definition of “bachelor” as an unmarried adult male is suspect; one might, for example, question a statement such as “the Pope is a bachelor.” While not strictly false, this usage is certainly infelicitous because it induces unintended inferences on the part of the listener. The tension could perhaps be resolved by distinguishing between logical definitions suitable for internal knowledge representation and the more nuanced criteria for felicitous linguistic usage. The latter may be achieved by “filtering” the assertions derived from the former. It is also possible that failures of linguistic usage serve as feedback for modifying internal definitions, so that filtering becomes unnecessary.

Section 10.2.

Categories and Objects 327

INDIVIDUATION STUFF

Similar axioms can be written for pounds and kilograms, seconds and days, and dollars and cents. Measures can be used to describe objects as follows:

Diameter(Basketball12) = Inches(9.5) . ListPrice(Basketballi2) = $(19) . d € Days = Duration(d) = Hours(24) .

Note that $(1) is nor a dollar bill! One can have two dollar bills, but there is only one object named $(1). Note also that, while Inches(0) and Centimeters(0) refer to the same zero length, they are not identical to other zero measures, such as Seconds(0).

Simple, quantitative measures are easy to represent. Other measures present more of a problem, because they have no agreed scale of values. Exercises have difficulty, desserts have deliciousness, and poems have beauty, yet numbers cannot be assigned to these qualities. One might, in a moment of pure accountancy, dismiss such properties as useless for the purpose of logical reasoning; or, still worse, attempt to impose a numerical scale on beauty. This would be a grave mistake, because it is unnecessary. The most important aspect of measures is not the particular numerical values, but the fact that measures can be ordered.

Although measures are not numbers, we can still compare them using an ordering symbol such as >. For example, we might well believe that Norvig’s exercises are tougher than Russell’s, and that one scores less on tougher exercises:

e1 € Exercises A ea € Exercises A Wrote(Norvig, e1) A Wrote( Russell, ex) = Difficulty(e1) > Difficulty (ez) .

e1 € Exercises A ex € Exercises A Difficulty (ey) > Difficulty (ez) = EzpectedScore(ey) < ExpectedScore(es) .

This is enough to allow one to decide which exercises to do, even though no numerical values for difficulty were ever used. (One does, however, have to discover who wrote which exercises.) These sorts of monotonic relationships among measures form the basis for the field of qualitative physics, a subfield of Al that investigates how to reason about physical systems without plunging into detailed equations and numerical simulations. Qualitative physics is discussed in the historical notes section.

Substances and objects

The real world can be seen as consisting of primitive objects (particles) and composite objects built from them. By reasoning at the level of large objects such as apples and cars, we can overcome the complexity involved in dealing with vast numbers of primitive objects individually. There is, however, a significant portion of reality that seems to defy any obvious individuation—division into distinct objects. We give this portion the generic name stuff. For example, suppose I have some butter and an aardvark in front of me. I can say there is one aardvark, but there is no obvious number of “butter-objects,” because any part of a butter-object is also a butter-object, at least until we get to very small parts indeed. This is the major distinction between stuff and things. If we cut an aardvark in half, we do not get two aardvarks (unfortunately).

The English language distinguishes clearly between stuff and things. We say “an aardvark,” but, except in pretentious California restaurants, one cannot say “a butter.” Linguists

328

Chapter 10. Knowledge Representation

COUNT NOUNS MASS NOUNS

INTRINSIC

EXTRINSIC

distinguish between count nouns, such as aardvarks, holes, and theorems, and mass nouns, such as butter, water, and energy. Several competing ontologies claim to handle this distinction. We will describe just one; the others are covered in the historical notes section.

To represent stuff properly, we begin with the obvious. We will need to have as objects in our ontology at least the gross “lumps” of stuff we interact with. For example, we might recognize a lump of butter as the same butter that was left on the table the night before; we might pick it up, weigh it, sell it, or whatever. In these senses, it is an object just like the aardvark. Let us call it Butters. We will also define the category Butter. Informally, its elements will be all those things of which one might say “It’s butter,” including Butters. With some caveats about very small parts that we will omit for now, any part of a butter-object is also a butter-object:

x € Butter A PartOf (y,x) = y € Butter . We can now say that butter melts at around 30 degrees centigrade: x € Butter = MeltingPoint(z, Centigrade(30)) .

We could go on to say that butter is yellow, is less dense than water, is soft at room temperature, has a high fat content, and so on. On the other hand, butter has no particular size, shape, or weight. We can define more specialized categories of butter such as Unsalted Butter, which is also a kind of stuff. On the other hand, the category PoundOfButter, which includes as members all butter-objects weighing one pound, is not a substance! If we cuta pound of butter in half, we do not, alas, get two pounds of butter.

What is actually going on is this: there are some properties that are intrinsic: they belong to the very substance of the object, rather than to the object as a whole. When you cut a substance in half, the two pieces retain the same set of intrinsic properties—things like density, boiling point, flavor, color, ownership, and so on. On the other hand, extrinsic properties are the opposite: properties such as weight, length, shape, function, and so on are not retained under subdivision.

A class of objects that includes in its definition only intrinsic properties is then a substance, or mass noun; a class that includes any extrinsic properties in its definition is a count noun. The category Stuff is the most general substance category, specifying no intrinsic properties. The category Thing is the most general discrete object category, specifying no extrinsic properties. All physical objects belong to both categories, so the categories are coextensive—they refer to the same entities.

Reasoning about the results of actions is central to the operation of a knowledge-based agent. Chapter 7 gave examples of propositional sentences describing how actions affect the wumpus world—for example, Equation (7.3) on page 227 states how the agent’s location is changed by forward motion. One drawback of propositional logic is the need to have a different copy of the action description for each time at which the action might be executed. This section describes a representation method that uses first-order logic to avoid that problem,

Section 10.3.

Actions, Situations, and Events 329

SITUATION CALCULUS

SITUATIONS

FLUENTS

The ontology of situation calculus

One obvious way to avoid multiple copies of axioms is simply to quantify over time—to say, “Vt, such-and-such is the result at ¢ + 1 of doing the action at ¢.” Instead of dealing with explicit times like ¢ + 1, we will concentrate in this section on situations, which denote the states resulting from executing actions. This approach is called situation calculus and involves the following ontology:

e Asin Chapter 8, actions are logical terms such as Forward and Turn(Right). For now, we will assume that the environment contains only one agent. (If there is more than one, an additional argument can be inserted to say which agent is doing the action.)

e Situations are logical terms consisting of the initial situation (usually called So) and all situations that are generated by applying an action to a situation. The function Result(a, s) (sometimes called Do) names the situation that results when action a is executed in situation s. Figure 10.2 illustrates this idea.

e Fluents are functions and predicates that vary from one situation to the next, such as the location of the agent or the aliveness of the wumpus. The dictionary says a fluent is something that flows, like a liquid. In this use, it means flowing or changing across situations. By convention, the situation is always the last argument of a fluent. For example, ~Holding(G1, Sp) says that the agent is not holding the gold Gi; in the initial situation Sp. Age( Wumpus, Sp) refers to the wumpus’s age in So.

e Atemporal or eternal predicates and functions are also allowed. Examples include the predicate Gold(G1) and the function LeftLegOf ( Wumpus).

Result(Turn(Right), Result(Forward, Sy) Turn(Right) Result(Forward, Sp) Forward 0

S

Figure 10.2 In situation calculus, each situation (except Sp) is the result of an action.

330

Chapter 10. Knowledge Representation

PROJECTION PLANNING

POSSIBILITY AXIOM EFFECT AXIOM

In addition to single actions, it is also helpful to reason about action sequences. We can define the results of sequences in terms of the results of individual actions. First, we say that executing an empty sequence leaves the situation unchanged:

Result([],s) = s.

Executing a nonempty sequence is the same as executing the first action and then executing the rest in the resulting situation:

Result([a] seq], s) = Result(seq, Result(a,s)) .

A situation calculus agent should be able to deduce the outcome of a given sequence of actions; this is the projection task. With a suitable constructive inference algorithm, it should also be able to find a sequence that achieves a desired effect; this is the planning task.

We will use an example from a modified version of the wumpus world where we do not worry about the agent’s orientation and where the agent can Go from one location to an adjacent one. Suppose the agent is at [1, 1] and the gold is at [1, 2]. The aim is to have the gold in [1, 1]. The fluent predicates are At(o,z, s) and Holding (o, s). Then the initial knowledge base might include the following description:

At (Agent, 1, 1], So) AN At(Gh, 1, 2], So) This is not quite enough, however, because it doesn’t say what what isn’t true in Sp. (See page 355 for further discussion of this point.) The complete description is as follows: At(o,z,5) © [(o=AgentAz=[1,1))V (c=G1 Az=]1,2])]. ~ Holding (0, Sp) . We also need to state that G is gold and that [1,1] and [1, 2] are adjacent: Gold(G1) A Adjacent ([1,1],[1,2]) A Adjacent([1,2],[1,1]) . One would like to be able to prove that the agent achieves its aim by going to [1, 2], grabbing the gold, and returning to [1, 1]. That is, At(G1, [1,1], Result([Go([1,1],[1,2]), Grab(G1), Go([1,2],[1,1])], So))

More interesting is the possibility of constructing a plan to get the gold, which is achieved by answering the query “what sequence of actions results in the gold being at [1,1]7”

dseq At(Gr, [1,1], Result(seq, So))

Let us see what has to go into the knowledge base for these queries to be answered.

Describing actions in situation calculus

In the simplest version of situation calculus, each action is described by two axioms: a possibility axiom that says when it is possible to execute the action, and an effect axiom that says what happens when a possible action is executed. We will use Poss(a, s) to mean that it is possible to execute action ¢ in situation s. The axioms have the following form:

POSSIBILITY AXIOM: Preconditions = Poss(a,s) . EFFECT AXIOM: Poss(a,s) = Changes that result from taking action.

Section 10.3.

Actions, Situations, and Events 331

FRAME PROBLEM

FRAME AXIOM

REPRESENTATIONAL FRAME PROBLEM

INFERENTIAL FRAME PROBLEM

QUALIFICATION PROBLEM

We will present these axioms for the modified wumpus world. To shorten our sentences, we will omit universal quantifiers whose scope is the entire sentence. We assume that the variable s ranges over situations, a ranges over actions, o over objects (including agents), g over gold, and z and y over locations.

The possibility axioms for this world state that an agent can go between adjacent locations, grab a piece of gold in the current location, and release some gold that it is holding:

At(Agent, x, s) AN Adjacent(z,y) = Poss(Go(z,y),s) . Gold(g) N At(Agent,z,s) A At(g,xz,s) = Poss(Grab(g),s) . Holding(g, 3) = Poss(Release(g), s) .

The effect axioms state that, if an action is possible, then certain properties (fluents) will hold in the situation that results from executing the action. Going from x to y results in being at y, grabbing the gold results in holding the gold, and releasing the gold results in not holding it:

Poss(Go(x,y),s) = At(Agent,y, Result(Go(z,y),s)) .

Poss(Grab(g), s) = Holding(g, Result(Grab(g), s)) .

Poss(Release(g),s) = —Holding(g, Result(Release(g), s)) . Having stated these axioms, can we prove that our little plan achieves the goal? Unfortunately not! At first, everything works fine; Go([1,1],[1, 2]) is indeed possible in Sy and the effect axiom for Go allows us to conclude that the agent reaches [1,2]:

At(Agent, [1,2], Result(Go([1,1],[1,2]), So))

Now we consider the Grab(Gy) action. We have to show that it is possible in the new situation—that is,

At(Gh, [1,2], Result(Go([1,1],[1,2]),S0))

Alas, nothing in the knowledge base justifies such a conclusion. Intuitively, we understand that the agent’s Go action should have no effect on the gold’s location, so it should still be at [1,2], where it was in Sy. The problem is that the effect axioms say what changes, but don’t say what stays the same.

Representing all the things that stay the same is called the frame problem. We must find an efficient solution to the frame problem because, in the real world, almost everything stays the same almost all the time. Each action affects only a tiny fraction of all fluents.

One approach is to write explicit frame axioms that do say what stays the same. For example, the agent’s movements leave other objects stationary unless they are held:

At(o,z, 8) A (0 # Agent) A ~Holding(o,s) = At(o,z, Result(Go(y, z),s)) .

If there are F' fluent predicates and A actions, then we will need O( AF") frame axioms. On the other hand, if each action has at most E effects, where FE is typically much less than F, then we should be able to represent what happens with a much smaller knowledge base of size O(AE). This is the representational frame problem. The closely related inferential frame problem is to project the results of a ¢-step sequence of actions in time O(E?), rather than time O(Ft) or O(AEt). We will address each problem in turn. Even then, another problem remains—that of ensuring that all necessary conditions for an action’s success have been specified. For example, Go fails if the agent dies en route. This is the qualification problem, for which there is no complete solution.

332

Chapter 10. Knowledge Representation

SUCCESSOR-STATE AXIOM

IMPLICIT EFFECT

RAMIFICATION PROBLEM

Solving the representational frame problem

The solution to the representational frame problem involves just a slight change in viewpoint on how to write the axioms. Instead of writing out the effects of each action, we consider how each fluent predicate evolves over time.> The axioms we use are called successor-state axioms. They have the following form:

SUCCESSOR-STATE AXIOM: Action is possible = (Fluent is true in result state <> Action’s effect made it true V It was true before and action left it alone) .

After the qualification that we are not considering impossible actions, notice that this definition uses <>, not =-. This means that the axiom says that the fluent will be true if and only if the right-hand side holds. Put another way, we are specifying the truth value of each fluent in the next state as a function of the action and the truth value in the current state. This means that the next state is completely specified from the current state and hence that there are no additional frame axioms needed.

The successor-state axiom for the agent’s location says that the agent is at y after executing an action either if the action is possible and consists of moving to y or if the agent was already at y and the action is not a move to somewhere else:

Poss(a,s) = (At(Agent,y, Result(a,s)) < a= Go(z,y) V (At(Agent,y,s) Na # Go(y, 2))) .

The axiom for Holding says that the agent is holding g after executing an action if the action was a grab of g and the grab is possible or if the agent was already holding g, and the action is not releasing it:

Poss(a,s) = (Holding(g, Result(a,s)) << a= Grab(yg) V (Holding(g, s) A a # Release(g))) .

Successor-state axioms solve the representational frame problem because the total size of the axioms is O( AF) literals: each of the E effects of each of the A actions is mentioned exactly once. The literals are spread over F' different axioms, so the axioms have average size AE/F.

The astute reader will have noticed that these axioms handle the At fluent for the agent, but not for the gold; thus, we still cannot prove that the three-step plan achieves the goal of having the gold in [1, 1]. We need to say that an implicit effect of an agent moving from z to y is that any gold it is carrying will move too (as will any ants on the gold, any bacteria on the ants, etc.). Dealing with implicit effects is called the ramification problem. We will discuss the problem in general later, but for this specific domain, it can be solved by writing a more general successor-state axiom for Af. The new axiom, which subsumes the previous version, says that an object o is at y if the agent went to y and o is the agent or something the

3 This is essentially the approach we took in building the Boolean circuit-based agent in Chapter 7. Indeed, axioms such as Equation (7.4) and Equation (7.5) can be viewed as successor-state axioms.

Section 10.3.

Actions, Situations, and Events 333

UNIQUE NAMES AXIOM

UNIQUE ACTION AXIOMS

agent was holding; or if 0 was already at y and the agent didn’t go elsewhere, with o being the agent or something the agent was holding.

Poss(a,s) = At(o,y, Result(a,s)) < (a= Go(z,y) A (o= Agent V Holding(o, s))) V (At(o,y,s) A=(Fz y#2zANa=Go(y,z) A (o= Agent Vv Holding (o, s)))) .

There is one more technicality: an inference process that uses these axioms must be able to prove nonidentities. The simplest kind of nonidentity is between constants—for example, Agent # G1. The general semantics of first-order logic allows distinct constants to refer to the same object, so the knowledge base must include an axiom to prevent this. The unique names axiom states a disequality for every pair of constants in the knowledge base. When this is assumed by the theorem prover, rather than written down in the knowledge base, it is called a unique names assumption. We also need to state disequalities between action terms: Go([1,1],[1,2]) is a different action from Go([1,2],[1,1]) or Grab(G1). First, we say that each type of action is distinct—that no Go action is a Grab action. For each pair of action names A and B, we would have

Alz1,... zm) # Bly, ..-, yn)

Next, we say that two action terms with the same action name refer to the same action only if they involve all the same objects:

Alzi, om) =AY1, Um) © Z1=UL Ac. ATi = Ym

These are called, collectively, the unique action axioms. The combination of initial state description, successor-state axioms, unique name axiom, and unique action axioms suffices to prove that the proposed plan achieves the goal.

Solving the inferential frame problem

Successor-state axioms solve the representational frame problem, but not the inferential frame problem. Consider a ¢-step plan p such that S; = Result(p, Sp). To decide which fluents are true in .S;, we need to consider each of the F' frame axioms on each of the ¢ time steps. Because the axioms have average size AE/F, this gives us O( AE?) inferential work. Most of the work involves copying fluents unchanged from one situation to the next.

To solve the inferential frame problem, we have two possibilities. First, we could discard situation calculus and invent a new formalism for writing axioms. This has been done with formalisms such as the fluent calculus. Second, we could alter the inference mechanism to handle frame axioms more efficiently. A hint that this should be possible is that the simple approach is O( AEt); why should it depend on the number of actions, A, when we know exactly which one action is executed at each time step? To see how to improve matters, we first look at the format of the frame axioms:

Poss(a,s) =

Fi(Result(a,s)) & (a=A1Va=A43...) V Fi(s)N(a+# Ag) A (a# Ag)...

334

Chapter 10. Knowledge Representation

EVENT CALCULUS

That is, each axiom mentions several actions that can make the fluent true and several actions that can make it false. We can formalize this by introducing the predicate PosEffect(a, F;), meaning that action a causes F; to become true, and NegEffect(a, F;) meaning that a causes Fj; to become false. Then we can rewrite the foregoing axiom schema as:

Poss(a,s) =

F;(Result(a, s)) < PosEffect(a,F;) V [Fi(s) A "NegEffect(a, F;)] fi 3) PosEffect( As, F;) NegEffect( As, F;) NegEffect(Aq, F;) .

Whether this can be done automatically depends on the exact format of the frame axioms. To make an efficient inference procedure using axioms like this, we need to do three things:

1. Index the PosEffect and NegEffect predicates by their first argument so that when we are given an action that occurs at time ¢, we can find its effects in O(1) time.

2. Index the axioms so that once you know that Fj is an effect of an action, you can find the axiom for F; in O(1) time. Then you need not even consider the axioms for fluents that are not an effect of the action.

3. Represent each situation as a previous situation plus a delta. Thus, if nothing changes from one step to the next, we need do no work at all. In the old approach, we would need to do O(F') work in generating an assertion for each fluent F;( Result(a, s)) from the preceding F;(s) assertions.

Thus at each time step, we look at the current action, fetch its effects, and update the set of true fluents. Each time step will have an average of E' of these updates, for a total complexity of O(Et). This constitutes a solution to the inferential frame problem.

Time and event calculus

Situation calculus works well when there is a single agent performing instantaneous, discrete actions. When actions have duration and can overlap with each other, situation calculus becomes somewhat awkward. Therefore, we will cover those topics with an alternative formalism known as event calculus, which is based on points in time rather than on situations. (The terms “event” and “action” may be used interchangeably. Informally, “event” connotes a wider class of actions, including ones with no explicit agent. These are easier to handle in event calculus than in situation calculus.)

In event calculus, fluents hold at points in time rather than at situations, and the calculus is designed to allow reasoning over intervals of time. The event calculus axiom says that a fluent is true at a point in time if the fluent was initiated by an event at some time in the past and was not terminated by an intervening event. The Initiates and Terminates relations play a role similar to the Result relation in situation calculus; Initiates(e, f,t) means that the occurrence of event e at time ¢ causes fluent f to become true, while Terminates(w, f,?) means that f ceases to be true. We use Happens(e, t) to mean that event e happens at time 2,

Section 10.3.

Actions, Situations, and Events 335

GENERALIZED EVENT

SUBEVENTS

and we use Clipped (f,t,t2) to mean that f is terminated by some event sometime between ¢ and to. Formally, the axiom is:

EVENT CALCULUS AXIOM: T(f ta) © Fe, t Happens(e,t) A Initiates(e, f, 1) A(t < ta) A Clipped(f,t,ts) Clipped (f,t, ta) < Fe,t1 Happens(e, ty) A Terminates(e, f,t1) A(t

This gives us functionality that is similar to situation calculus, but with the ability to talk about time points and intervals, so we can say Happens( TurnOff ( LightSwitch,), 1:00) to say that a lightswitch was turned off at exactly 1:00.

Many extensions to event calculus have been made to address problems of indirect effects, events with duration, concurrent events, continuously changing events, nondeterministic effects, causal constraints, and other complications. We will revisit some of these issues in the next subsection. It is fair to say that, at present, completely satisfactory solutions are not yet available for most of them, but no insuperable obstacles have been encountered.

Generalized events

So far, we have looked at two main concepts: actions and objects. Now it is time to see how they fit into an encompassing ontology in which both actions and objects can be thought of as aspects of a physical universe. We think of a particular universe as having both a spatial and a temporal dimension. The wumpus world has its spatial component laid out in a two-dimensional grid and has discrete time; our world has three spatial dimensions and one temporal dimension, all continuous. A generalized event is composed from aspects of some “space-time chunk” —a piece of this multidimensional space-time universe. This abstraction generalizes most of the concepts we have seen so far, including actions, locations, times, fluents, and physical objects. Figure 10.3 gives the general idea. From now on, we will use the simple term “event” to refer to generalized events.

For example, World War II is an event that took place at various points in space-time, as indicated by the irregularly shaped grey patch. We can break it down into subevents:\*

SubEvent( Battle OfBritain, WorldWarlII) . Similarly, World War II is a subevent of the 20th century: SubEvent( WorldWarll, TwentiethCentury) .

The 20th century is an interval of time. Intervals are chunks of space-time that include all of space between two time points. The function Period(e) denotes the smallest interval enclosing the event e. Duration(z) is the length of time occupied by an interval, so we can say Duration(Period( WorldWarlII)) > Years(5).

3 Some physicists studying string theory argue for 10 dimensions or more, and some argue for a discrete world, but 4-D continuous space-time is an adequate representation for commonsense reasoning purposes. 4 Note that SubEvent is a special case of the PartOf relation and is also transitive and reflexive.

336

Chapter 10. Knowledge Representation

“space”

WorldWarl]

i

Australia

ol |

~ time

FE FE A —, mmm mda mmen ee

TwentiethCentury

Figure 10.3 Generalized events. A universe has spatial and temporal dimensions; in this figure we show only a single spatial dimension. All events are PartOf the universe. An event, such as WorldWarll, occurs in a portion of space-time with somewhat arbitrary and timevarying borders. An Interval, such as the TwentiethCentury, has a fixed, limited temporal extent and maximal spatial extent, and a Place, such as Australia, has a roughly fixed spatial extent and maximal temporal extent.

Australia is a place; a chunk with some fixed spatial borders. The borders can vary over time, due to geological or political changes. We use the predicate In to denote the subevent relation that holds when one event’s spatial projection is PartOf of anothers:

In(Sydney, Australia) .

The function Location(e) denotes the smallest place that encloses the event e.

Like any other sort of object, events can be grouped into categories. For example, WorldWarII belongs to the category Wars. To say that a civil war occurred in England in the 1640s, we would say

Jw we CiwilWars A SubEvent(w, 1640s) A In(Location(w), England) .

The notion of a category of events answers a question that we avoided when we described the effects of actions in Section 10.3: what exactly do logical terms such as Go([1, 1], [1, 2]) refer to? Are they events? The answer, perhaps surprisingly, is no. We can see this by considering a plan with two “identical” actions, such as

[Go([1,1],(1,2]), Go([1,2], [1,1]), Go([1, 1], [1,2])] . In this plan, Go([1, 1], [1, 2]) cannot be the name of an event, because there are two different events occurring at different times. Instead, Go([1,1],[1,2]) is the name of a category of

events—all those events where the agent goes from [1,1] to [1,2]. The three-step plan says that instances of these three event categories will occur.

Section 10.3.

Actions, Situations, and Events 337

DISCRETE EVENTS

PROCESS LIQUID EVENT

TEMPORAL SUBSTANCES SPATIAL SUBSTANCES STATES

Notice that this is the first time we have seen categories named by complex terms rather than just constant symbols. This presents no new difficulties; in fact, we can use the argument structure to our advantage. Eliminating arguments creates a more general category:

Go(z,y) C GoTo(y) Go(z,y) C GoFrom(x) .

Similarly, we can add arguments to create more specific categories. For example, to describe actions by other agents, we can add an agent argument. Thus, to say that Shankar flew from New York to New Delhi yesterday, we would write:

Je e€ Fly(Shankar, NewYork, NewDelhi) A SubEvent(e, Yesterday) .

The form of this formula is so common that we will create an abbreviation for it: F(c, 1) will mean that an element of the category of events c is a subevent of the event or interval i:

E(c,i) © de e€c A SubBuvent(e,i) . Thus, we have: E(Fly(Shankar, NewYork, NewDelhi), Yesterday) .

Processes

The events we have seen so far are what we call discrete events—they have a definite structure. Shankar’s trip has a beginning, middle, and end. If interrupted halfway, the event would be different—it would not be a trip from New York to New Delhi, but instead a trip from New York to somewhere over Europe. On the other hand, the category of events denoted by Flying(Shankar) has a different quality. If we take a small interval of Shankar’s flight, say, the third 20-minute segment (while he waits anxiously for a second bag of peanuts), that event is still a member of Flying( Shankar). In fact, this is true for any subinterval.

Categories of events with this property are called process categories or liquid event categories. Any subinterval of a process is also a member of the same process category. We can employ the same notation used for discrete events to say that, for example, Shankar was flying at some time yesterday:

E(Flying(Shankar), Yesterday) .

We often want to say that some process was going on throughout some interval, rather than just in some subinterval of it. To do this, we use the predicate 1":

T(Working( Stuart), TodayLunchHour) .

T'(c, i) means that some event of type c occurred over exactly the interval i—that is, the event begins and ends at the same time as the interval.

The distinction between liquid and nonliquid events is exactly analogous to the difference between substances, or stuff, and individual objects. In fact, some have called liquid event types temporal substances, whereas things like butter are spatial substances.

As well as describing processes of continuous change, liquid events can describe processes of continuous non-change. These are often called states. For example, “Shankar being in New York” is a category of states that we denote by In(Shankar, NewYork). To say he was in New York all day, we would write

T(In(Shankar, NewYork), Today) .

338

Chapter 10. Knowledge Representation

FLUENT CALCULUS

We can form more complex states and events by combining primitive ones. This approach is called fluent calculus. Fluent calculus reifies combinations of fluents, not just individual fluents. We have already seen a way of representing the event of two things happening at once, namely, the function Both{e1, ez). In fluent calculus, this is usually abbreviated with the infix notation e; o ez. For example, to say that someone walked and chewed gum at the same time, we can write

dp,i (p€ People) AN T(Walk(p) o ChewGum(p),i) .

The “o” function is commutative and associative, just like logical conjunction. We can also define analogs of disjunction and negation, but we have to be more careful—there are two reasonable ways of interpreting disjunction. When we say “the agent was either walking or chewing gum for the last two minutes” we might mean that the agent was doing one of the actions for the whole interval, or we might mean that the agent was alternating between the two actions. We will use OneOf and Either to indicate these two possibilities. Figure 10.4 diagrams the complex events.

-— -— — -— —— p Fo fw HE q ETE IT IE (a) (b) ©

Figure 10.4 A depiction of complex events. (a) T'(Both(p. ¢). 1), also denoted as T'(p o q,%). (0) T(OneOf (p, ), 1). (c) T(Either(p.q). 1).

Intervals

Time is important to any agent that takes action, and there has been much work on the representation of time intervals. We will consider two kinds: moments and extended intervals. The distinction is that only moments have zero duration:

Partition({ Moments, ExtendedIntervals}, Intervals) i € Moments < Duration(i) = Seconds(0) .

Next we invent a time scale and associate points on that scale with moments, giving us absolute times. The time scale is arbitrary; we will measure it in seconds and say that the moment at midnight (GMT) on January 1, 1900, has time 0. The functions Start and End pick out the earliest and latest moments in an interval, and the function Time delivers the point on the time scale for a moment. The function Duration gives the difference between the end time and the start time.

Interval(i) = Duration(i) = ( Tiéme(End(3)) — Time(Start(i)))

Time(Start(AD1900)) = Seconds(0) .

Time(Start(AD2001)) = Seconds(3187324800) .

Time(End(AD2001)) = Seconds(3218860800) .

Duration( AD2001) = Seconds(31536000) .

Section 10.3.

Actions, Situations, and Events 339

To make these numbers easier to read, we also introduce a function Date, which takes

six arguments (hours, minutes, seconds, day, month, and year) and returns a time point: Time(Start(AD2001)) = Date(0,0,0, 1, Jan, 2001) Date(0,20,21,24,1,1995) = Seconds(3000000000) .

Two intervals Meet if the end time of the first equals the start time of the second. It is possible to define predicates such as Before, After, During, and Overlap solely in terms of Meet, but it is more intuitive to define them in terms of points on the time scale. (See Figure 10.5 for a graphical representation.)

Meet(i, j) & Time(End(i)) = Time(Start(j)) .

Before(i,j) << Time(End(i)) < Time(Start(j)) .

After(4,1) & Before(i, j)

During(i,j) << Time(Start(j)) < Time(Start(i))

A Time(End(i)) < Time(End(j)) . Overlap(i,j) << 3k During(k,i) A During(k, 7) .

Meet(ij)y ~~ F———

Before(i,j) — . After(j,i) Ly

During(i,j) —

Overlap(ij) —p——— Overlap(j,i) I

Figure 10.5 Predicates on time intervals.

For example, to say that the reign of Elizabeth II followed that of George VI, and the reign of Elvis overlapped with the 1950s, we can write the following:

After(ReignOf ( Elizabethll), ReignOf (George VI)) . Overlap (Fifties, ReignOf (Elvis)) .

Start( Fifties) = Start(AD1950) .

End( Fifties) = End(AD1959) .

Fluents and objects

We mentioned that physical objects can be viewed as generalized events, in the sense that a physical object is a chunk of space-time. For example, USA can be thought of as an event that began in, say, 1776 as a union of 13 states and is still in progress today as a union of 50.

340

Chapter 10. Knowledge Representation

We can describe the changing properties of USA using state fluents. For example, we can say that at some point in 1999 its population was 271 million:

E(Population( USA, 271000000), AD1999) .

Another property of the USA that changes every four or eight years, barring mishaps, is its president. One might propose that President( USA) is a logical term that denotes a different object at different times. Unfortunately, this is not possible, because a term denotes exactly one object in a given model structure. (The term President( USA, t) can denote different objects, depending on the value of , but our ontology keeps time indices separate from fluents.) The only possibility is that President( USA) denotes a single object that consists of different people at different times. It is the object that is George Washington from 1789 to 1796, John Adams from 1796 to 1800, and so on, as in Figure 10.6.

1800 time

1789

Figure 10.6 A schematic view of the object President USA) for the first 15 years of its existence.

To say that George Washington was president throughout 1790, we can write

T(President( USA) = George Washington, AD1790) .

9

We need to be careful, however. In this sentence, must be a function symbol rather than the standard logical operator. The interpretation is not that George Washington and President( USA) are logically identical in 1790; logical identity is not something that can change over time. The logical identity exists between the subevents of each object that are defined by the period 1790.

Don’t confuse the physical object George Washington with a collections of atoms. George Washington is not logically identical to any specific collection of atoms, because the set of atoms of which he is constituted varies considerably over time. He has his short lifetime, and each atom has its own very long lifetime. They intersect for some period, during which the temporal slice of the atom is PartOf George, and then they go their separate ways.

Section 10.4.

Mental Events and Mental Objects 341

PROPOSITIONAL ATTITUDE

REIFICATION

The agents we have constructed so far have beliefs and can deduce new beliefs. Yet none of them has any knowledge about beliefs or about deduction. For single-agent domains, knowledge about one’s own knowledge and reasoning processes is useful for controlling inference. For example, if one knows that one does not know anything about Romanian geography, then one need not expend enormous computational effort trying to calculate the shortest path from Arad to Bucharest. One can also reason about one’s own knowledge in order to construct plans that will change it—for example by buying a map of Romania. In multiagent domains, it becomes important for an agent to reason about the mental states of the other agents. For example, a Romanian police officer might well know the best way to get to Bucharest, so the agent might ask for help.

In essence, what we need is a model of the mental objects that are in someone’s head (or something’s knowledge base) and of the mental processes that manipulate those mental objects. The model should be faithful, but it does not have to be detailed. We do not have to be able to predict how many milliseconds it will take for a particular agent to make a deduction, nor do we have to predict what neurons will fire when an animal is faced with a particular visual stimulus. We will be happy to conclude that the Romanian police officer will tell us how to get to Bucharest if he or she knows the way and believes we are lost.

A formal theory of beliefs

We begin with the relationships between agents and “mental objects” —relationships such as Believes, Knows, and Wants. Relations of this kind are called propositional attitudes, because they describe an attitude that an agent can take toward a proposition. Suppose that Lois believes something—that is, Believes(Lois, xz). What kind of thing is #? Clearly, x cannot be a logical sentence. If Flies(Superman) is a logical sentence, we can’t say Believes(Lois, Flies(Superman)), because only terms (not sentences) can be arguments of predicates. But if Flies is a function, then Flies( Superman) is a candidate for being a mental object, and Believes can be a relation between an agent and a propositional fluent. Turning a proposition into an object is called reification.

This appears to give us what we want: the ability for an agent to reason about the beliefs of agents. Unfortunately, there is a problem with that approach: If Clark and Superman are one and the same (i.e., Clark = Superman) then Clark’s flying and Superman’s flying are one and the same event category, i.e., Flies( Clark) = Flies(Superman). Hence, we must conclude that if Lois believes that Superman can fly, she also believes that Clark can fly, even if she doesn’t believe that Clark is Superman. That is,

(Superman = Clark) |= (Believes(Lois, Flies(Superman)) < Believes(Lois, Flies( Clark))) .

There is a sense in which this is right: Lois does believe of a certain person, who happens

6 The term “reification” comes from the Latin word res, or thing. John McCarthy proposed the term “thingification,” but it never caught on.

342

Chapter 10. Knowledge Representation

REFERENTIAL TRANSPARENCY

OPAQUE MODAL LOGIC MODAL OPERATOR

SYNTACTIC THEORY STRINGS

UNIQUE STRING AXIOM

UNQUOTED

to be called Clark sometimes, that that person can fly. But there is another sense in which it is wrong: if you asked Lois “Can Clark fly?” she would certainly say no. Reified objects and events work fine for the first sense of Believes, but for the second sense we need to reify descriptions of those objects and events, so that Clark and Superman can be different descriptions (even though they refer to the same object).

Technically, the property of being able to substitute a term freely for an equal term is called referential transparency. In first-order logic, every relation is referentially transparent. We would like to define Believes (and the other propositional attitudes) as relations whose second argument is referentially opaque—that is, one cannot substitute an equal term for the second argument without changing the meaning.

There are two ways to achieve this. The first is to use a different form of logic called modal logic, in which propositional attitudes such as Believes and Knows become modal operators that are referentially opaque. This approach is covered in the historical notes section. The second approach, which we will pursue, is to achieve effective opacity within a referentially transparent language using a syntactic theory of mental objects. This means that mental objects are represented by strings. The result is a crude model of an agent’s knowledge base as consisting of strings that represent sentences believed by the agent. A string is just a complex term denoting a list of symbols, so the event Flies( Clark) can be represented by the list of characters [F,[,4,e,s,(,C,l,a,r, k,)], which we will abbreviate as a quoted string, “Flies(Clark)”. The syntactic theory includes a unique string axiom stating that strings are identical if and only if they consist of identical characters. In this way, even if Clark = Superman, we still have “Clark” # “Superman”.

Now all we have to do is provide a syntax, semantics, and proof theory for the string representation language, just as we did in Chapter 7. The difference is that we have to define them all in first-order logic. We start by defining Den as the function that maps a string to the object that it denotes and Name as a function that maps an object to a string that is the name of a constant that denotes the object. For example, the denotation of both “Clark” and “Superman” is the object referred to by the constant symbol Man OfSteel, and the name of that object within the knowledge base could be either “Superman”, “Clark”, or some other constant, such as “X11”:

Den (“Clark”) = ManOfSteel A Den(“Superman”) = ManOfSteel . Name(ManOfSteel) =X”. The next step is to define inference rules for logical agents. For example, we might want to say that a logical agent can do Modus Ponens: if it believes p and believes p = ¢, then it will also believe q. The first attempt at writing this axiom is

Logical Agent(a) A Believes(a,p) \ Believes(a,“p=q”) = Believes(a,q) . But this is not right because the string “p = ¢” contains the letters ‘p’ and ‘q’ but has nothing to do with the strings that are the values of the variables p and ¢. The correct formulation is LogicalAgent(a) A Believes(a,p) A Believes(a, Concat(p, “=",q)) = Believes(a,q) .

where Concat is a function on strings that concatenates their elements. We will abbreviate Concat(p,\* =", q) as “p=>¢”. That is, an occurrence of x within a string is unquoted,

Section 10.4.

Mental Events and Mental Objects 343

LOGICAL OMNISCIENCE

meaning that we are to substitute in the value of the variable z. Lisp programmers will recognize this as the comma/backquote operator, and Perl programmers will recognize it as $-variable interpolation.

Once we add in the other inference rules besides Modus Ponens, we will be able to answer questions of the form “given that a logical agent knows these premises, can it draw that conclusion?” Besides the normal inference rules, we need some rules that are specific to belief. For example, the following rule says that if a logical agent believes something, then it believes that it believes it.

LogicalAgent(a) A Believes(a,p) = Believes(a, “Believes(Name(a), p)”) . Now, according to our axioms, an agent can deduce any consequence of its beliefs infallibly. This is called logical omniscience. A variety of attempts have been made to define limited rational agents, which can make a limited number of deductions in a limited time.

None is completely satisfactory, but these formulations do allow a highly restricted range of predictions about limited agents.

Knowledge and belief

The relation between believing and knowing has been studied extensively in philosophy. It is commonly said that knowledge is justified true belief. That is, if you believe something for an unassailably good reason, and if it is actually true, then you know it. The “unassailably good reason” is necessary to prevent you from saying “I know this coin flip will come up heads” and being right half the time.

Let Knows{a, p) mean that agent a knows that proposition p is true. It is also possible to define other kinds of knowing. For example, here is a definition of “knowing whether’:

Knows Whether(a,p) © Knows(a,p)V Knows(a,“—p”).

Continuing our example, Lois knows whether Clark can fly if she either knows that Clark can fly or knows that he cannot.

The concept of “knowing what” is more complicated. One is tempted to say that an agent knows what Bob’s phone number is if there is some x for which the agent knows PhoneNumber(Bob) = x. But that is not enough, because the agent might know that Alice and Bob have the same number (i.e., PhoneNumber( Bob) = Phone Number(Alice)), but if Alice’s number is unknown, that isn’t much help. A better definition of “knowing what” says that the agent has to be aware of some z that is a string of digits and that is Bob’s number:

Knows What(a, “Phone Number(b)”) dz Knows(a, “xz = PhoneNumber(b)”) A x € DigitStrings . Of course, for other questions we have different criteria for what is an acceptable answer. For the question “what is the capital of New York,” an acceptable answer is a proper name, “Albany,” not something like “the city where the state house is.” To handle this, we will make Knows What a three-place relation: it takes an agent, a string representing a term, and a category to which the answer must belong. For example, we might have the following:

Knows What( Agent, “Capital(NewY ork)”, Proper Names) . Knows What( Agent, “Phone Number(Bob)”, DigitStrings) .

344

Chapter 10. Knowledge Representation

KNOWLEDGE PRECONDITIONS KNOWLEDGE EFFECTS

RUNTIME VARIABLES

Knowledge, time, and action

In most real situations, an agent will be dealing with beliefs—its own or those of other agents—that change over time. The agent will also have to make plans that involve changes to its own beliefs, such as buying a map to find out how to get to Bucharest. As with other predicates, we can reify Believes and talk about beliefs occurring over some period. For example, to say that Lois believes today that Superman can fly, we write

T'(Believes( Lois, “Flies{(Superman)”), Today) .

If the object of belief is a proposition that can change over time, then it too can be described using the 7" operator within the string. For example, Lois might believe today that that Superman could fly yesterday:

T'(Believes( Lois, “T(Flies(Superman), Yesterday)”, Today) .

Given a way to describe beliefs over time, we can use the machinery of event calculus to make plans involving beliefs. Actions can have knowledge preconditions and knowledge effects. For example, the action of dialing a person’s number has the precondition of knowing the number, and the action of looking up the number has the effect of knowing the number. We can describe the latter action using the machinery of event calculus:

Initiates(Lookup(a, “ Phone Number(b)”), Knows What (a, “Phone Number(b)”, DigitStrings),t) . Plans to gather and use information are often represented using a shorthand notation called

runtime variables, which is closely related to the unquoted-variable convention described earlier. For example, the plan to look up Bob’s number and then dial it can be written as

[Lookup( Agent, “Phone Number( Bob)”, n), Dial(n)] .

Here, n is a runtime variable whose value will be bound by the Lookup action and can then be used by the Dial action. Plans of this kind occur frequently in partially observable domains. We will see examples in the next section and in Chapter 12,

In this section we will encode some knowledge related to shopping on the Internet. We will create a shopping research agent that helps a buyer find product offers on the Internet. The shopping agent is given a product description by the buyer and has the task of producing a list of Web pages that offer such a product for sale. In some cases the buyer's product description will be precise, as in Coolpix 995 digital camera, and the task is then to find the store(s) with the best offer. In other cases the description will be only partially specified, as in digital camera for under $300, and the agent will have to compare different products.

The shopping agent’s environment is the entire World Wide Web—mnot a toy simulated environment, but the same complex, constantly evolving environment that is used by millions of people every day. The agent's percepts are Web pages, but whereas a human Web user would see pages displayed as an array of pixels on a screen, the shopping agent will perceive

Section 10.5.

The Internet Shopping World 345

Generic Online Store Select from our fine line of products: e Computers e Cameras e Books ® Videos e Music

Generic Online Store

Select from our fine line of products:

Computers Cameras Books Videos Music

Figure 10.7 A Web page from a generic online store in the form perceived by the human user of a browser (top), and the corresponding HTML string as perceived by the browser or the shopping agent (bottom). In HTML, characters between < and > are markup directives that specify how the page is displayed. For example, the string Select means to switch to italic font, display the word Select, and then end the use of italic font. A page identifier such as http: //gen-store.com/books is called a uniform resource locator (URL) or URL. The markup anchor means to create a hypertext link to url with the anchor text anchor.

a page as a character string consisting of ordinary words interspersed with formatting commands in the HTML markup language. Figure 10.7 shows a Web page and a corresponding HTML character string. The perception problem for the shopping agent involves extracting useful information from percepts of this kind.

Clearly, perception on Web pages is easier than, say, perception while driving a taxi in Cairo. Nonetheless, there are complications to the Internet perception task. The web page in Figure 10.7 is very simple compared to real shopping sites, which include cookies, Java, Javascript, Flash, robot exclusion protocols, malformed HTML, sound files, movies, and text that appears only as part of a JPEG image. An agent that can deal with all of the Internet is almost as complex as a robot that can move in the real world. We will concentrate on a simple agent that ignores most of these complications.

The agent's first task is to find relevant product offers (we’ll see later how to choose the best of the relevant offers). Let query be the product description that the user types in (e.g., “laptops”); then a page is a relevant offer for query if the page is relevant and the page is indeed an offer. We will also keep track of the URL associated with the page:

RelevantOffer(page, url, query) < Relevant(page, url, query) A Offer(page) .

346

Chapter 10. Knowledge Representation

A page with a review of the latest high-end laptop would be relevant, but if it doesn’t provide a way to buy, it isn’t an offer. For now, we can say a page is an offer if it contains the word “buy” or “price” within an HTML link or form on the page. In other words, if the page contains a string of the form “ “price” instead of “buy” or use “form” instead of “a”. We can write axioms for this: “price” instead of “buy” or use “form” instead of “a”. We can write axioms for this:

Offer (page) & (InTag(“a”, str, page) V InTag(“ form”, str, page))

A (In(“buy”, str) V In(“price”, str)) .

InTag(tag, str, page) << In(“<” + tag + str + “< /” + tag, page) .

In(sub, str) & Ji stiri: i+ Length(sub)] = sub . Now we need to find relevant pages. The strategy is to start at the home page of an online store and consider all pages that can be reached by following relevant links.” The agent will have knowledge of a number of stores, for example:

Amazon € OnlineStores A Homepage( Amazon, “amazon.com” . Ebay € OnlineStores A Homepage (Ebay, “ebay.com™) . GenStore € OnlineStores AN Homepage GenStore, “gen-store.com™) .

These stores classify their goods into product categories, and provide links to the major categories from their home page. Minor categories can be reached by following a chain of relevant links, and eventually we will reach offers. In other words, a page is relevant to the query if it can be reached by a chain of relevant category links from a store’s home page, and then following one more link to the product offer:

Relevant (page, url, query) < d store, home store € OnlineStores A Homepage store, home) A Jurly RelevantChain{home, urls, query) A Link (urls, url) A page = GetPage(url) .

Here the predicate Link({ from, to) means that there is a hyperlink from the from URL to the to URL. (See Exercise 10.13.) To define what counts as a RelevantChain, we need to follow not just any old hyperlinks, but only those links whose associated anchor text indicates that the link is relevant to the product query. For this, we will use LinkText( from, to, text) to mean that there is a link between from and fo with text as the anchor text. A chain of links between two URLS, start and end, is relevant to a description d if the anchor text of each link is a relevant category name for d. The existence of the chain itself is determined by a recursive definition, with the empty chain (start = end) as the base case:

Relevant Chain (start, end, query) < (start = end) V (Fu, text LinkText (start, u, text) A RelevantCategoryName( query, text) A RelevantChain(u, end, query)) .

Now we must define what it means for text to be a RelevantCategoryName for query. First, we need to relate strings to the categories they name. This is done using the predicate Name(s, c), which says that string s is a name for category c—for example, we might assert that Name (“laptops”, LaptopComputers). Some more examples of the Name predicate

7 An alternative to the link-following strategy is to use an Internet search engine; the technology behind Internet search. information retrieval. will be covered in Section 23.2.

Section 10.5.

The Internet Shopping World 347

Books C Products Name(“books”, Books)

MusicRecordings C Products Name(“music”’, MusicRecordings) MusicCDs C MusicRecordings Name(“CDs”, MusicCDs)

MusicTapes C MusicRecordings Name(“tapes”, Music Tapes)

Electronics C Products Name(“electronics”, Electronics) DigitalCameras C Electronics Name(“digital cameras”, DigitalCameras) StereoEquipment C Electronics Name(“stereos”, Stereo Equipment) Computers C Electronics Name (“computers”, Computers)

Laptop Computers C Computers Name(“laptops”, LaptopComputers) DesktopComputers C Computers Name(“desktops”, DesktopComputers) (a) (b) Figure 10.8 (a) Taxonomy of product categories. (b) Referring words for those categories.

appear in Figure 10.8(b). Next, we define relevance. Suppose that query is “laptops.” Then Relevant CategoryName(query, text) is true when one of the following holds:

e The text and query name the same category—e.g., “laptop computers” and “laptops.” e The text names a supercategory such as “computers.” o The text names a subcategory such as “ultralight notebooks.”

The logical definition of RelevantCategoryName is as follows:

RelevantCategoryName( query, text) dei1,ca Name(query, ci) A Name(text, ca) A (c; CcaVea Cay). (10.1)

Otherwise, the anchor text is irrelevant because it names a category outside this line, such as “mainframe computers” or “lawn & garden.”

To follow relevant links, then, it is essential to have a rich hierarchy of product categories. The top part of this hierarchy might look like Figure 10.8(a). It will not be feasible to list all possible shopping categories, because a buyer could always come up with some new desire and manufacturers will always come out with new products to satisfy them (electric kneecap warmers?). Nonetheless, an ontology of about a thousand categories will serve as a very useful tool for most buyers.

In addition to the product hierarchy itself, we also need to have a rich vocabulary of names for categories. Life would be much easier if there were a one-to-one correspondence between categories and the character strings that name them. We have already seen the problem of synonymy—two names for the same category, such as “laptop computers” and “laptops.” There is also the problem of ambiguity—one name for two or more different categories. For example, if we add the sentence

Name(“C Ds”, CertificatesOfDeposit)

to the knowledge base in Figure 10.8(b), then “CDs” will name two different categories. Synonymy and ambiguity can cause a significant increase in the number of paths that the agent has to follow, and can sometimes make it difficult to determine whether a given

348

Chapter 10. Knowledge Representation

PROCEDURAL ATTACHMENT

WRAPPER

page is indeed relevant. A much more serious problem is that there is a very broad range of descriptions that a user can type, or category names that a store can use. For example, the link might say “laptop” when the knowledge base has only “laptops;” or the user might ask for “a computer I can fit on the tray table of an economy-class seat in a Boeing 737.” It is impossible to enumerate in advance all the ways a category can be named, so the agent will have to be able to do additional reasoning in some cases to determine if the Name relation holds. In the worst case, this requires full natural language understanding, a topic that we will defer to Chapter 22. In practice, a few simple rules—such as allowing “laptop” to match a category named “laptops”—go a long way. Exercise 10.15 asks you to develop a set of such rules after doing some research into online stores.

Given the logical definitions from the preceding paragraphs and suitable knowledge bases of product categories and naming conventions, are we ready to apply an inference algorithm to obtain a set of relevant offers for our query? Not quite! The missing element is the GetPage(url) function, which refers to the HTML page at a given URL. The agent doesn’t have the page contents of every URL in its knowledge base; nor does it have explicit rules for deducing what those contents might be. Instead, we can arrange for the right HTTP procedure to be executed whenever a subgoal involves the GerPage function. In this way, it appears to the inference engine as if the entire Web is inside the knowledge base. This is an example of a general technique called procedural attachment, whereby particular predicates and functions can be handled by special-purpose methods.

Comparing offers

Let us assume that the reasoning processes of the preceding section have produced a set of offer pages for our “laptops” query. To compare those offers, the agent must extract the relevant information—price, speed, disk size, weight, and so on—from the offer pages. This can be a difficult task with real web pages, for all the reasons mentioned previously. A common way of dealing with this problem is to use programs called wrappers to extract information from a page. The technology of information extraction is discussed in Section 23.3. For now we assume that wrappers exist, and when given a page and a knowledge base, they add assertions to the knowledge base. Typically a hierarchy of wrappers would be applied to a page: a very general one to extract dates and prices, a more specific one to extract attributes for computer-related products, and if necessary a site-specific one that knows the format of a particular store. Given a page on the gen-store.com site with the text

YVM ThinkBook 970. Our price: $1449.00

followed by various technical specifications, we would like a wrapper to extract information such as the following:

dlc, offer lc € LaptopComputers A offer € ProductOffers A ScreenSize(lc, Inches(14)) A Screen Type(lc, ColorLCD) A MemorySize(lc, Megabytes(512)) AN CPUSpeed (lc, GHz(2.4)) A Offered Product (offer, ic) A Store( offer, GenStore) A URL( offer, “genstore.com/comps/34356. html”) A Price( offer, $(449)) A Date(offer, Today) .

Section 10.6.

Reasoning Systems for Categories 349

This example illustrates several issues that arise when we take seriously the task of knowledge engineering for commercial transactions. For example, notice that the price is an attribute of the offer, not the product itself. This is important because the offer at a given store may change from day to day even for the same individual laptop; for some categories—such as houses and paintings—the same individual object may even be offered simultaneously by different intermediaries at different prices. There are still more complications that we have not handled, such as the possibility that the price depends on method of payment and on the buyer’s qualifications for certain discounts. All in all, there is much interesting work to do.

The final task is to compare the offers that have been extracted. For example, consider these three offers:

A: 2.4GHz CPU, 512MB RAM, 80 GB disk, DVD, CDRW, $1695 . B : 2.0 GHz CPU, 1GB RAM, 120 GB disk, DVD, CDRW, $1800 . C : 2.2 GHz CPU, 512MB RAM, 80 GB disk, DVD, CDRW, $1800 .

C is dominated by A; that is, A is cheaper and faster, and they are otherwise the same. In general, X dominates Y if X has a better value on at least one attribute, and is not worse on any attribute. But neither A nor B dominates the other. To decide which is better we need to know how the buyer weighs CPU speed and price against memory and disk space. The general topic of preferences among multiple attributes is addressed in Section 16.4; for now, our shopping agent will simply return a list of all undominated offers that meet the buyer's description. In this example, both A and B are undominated. Notice that this outcome relies on the assumption that everyone prefers cheaper prices, faster processors, and more storage. Some attributes, such as screen size on a notebook, depend on the user’s particular preference (portability versus visibility); for these, the shopping agent will just have to ask the user.

The shopping agent we have described here is a simple one; many refinements are possible. Still, it has enough capability that with the right domain-specific knowledge it can actually be of use to a shopper. Because of its declarative construction, it extends easily to more complex applications. The main point of this section is to show that some knowledge representation—in particular, the product hierarchy—is necessary for an agent like this, and that once we have some knowledge in this form, it is not too hard to do the rest as a knowledge-based agent.

We have seen that categories are the primary building blocks of any large-scale knowledge representation scheme. This section describes systems specially designed for organizing and reasoning with categories. There are two closely related families of systems: semantic networks provide graphical aids for visualizing a knowledge base and efficient algorithms for inferring properties of an object on the basis of its category membership; and description logics provide a formal language for constructing and combining category definitions and efficient algorithms for deciding subset and superset relationships between categories.

350

Chapter 10. Knowledge Representation

EXISTENTIAL GRAPHS

MULTIPLE INHERITANCE

Semantic networks

In 1909, Charles Peirce proposed a graphical notation of nodes and arcs called existential graphs that he called “the logic of the future.” Thus began a long-running debate between advocates of “logic” and advocates of “semantic networks.” Unfortunately, the debate obscured the fact that semantics networks—at least those with well-defined semantics—are a form of logic. The notation that semantic networks provide for certain kinds of sentences is often more convenient, but if we strip away the “human interface” issues, the underlying concepts—objects, relations, quantification, and so on—are the same.

There are many variants of semantic networks, but all are capable of representing individual objects, categories of objects, and relations among objects. A typical graphical notation displays object or category names in ovals or boxes, and connects them with labeled arcs. For example, Figure 10.9 has a MemberOf link between Mary and FemalePersons, corresponding to the logical assertion Mary € Female Persons; similarly, the SisterOf link between Mary and John corresponds to the assertion SisterOf (Mary, John). We can connect categories using SubsetOf links, and so on. It is such fun drawing bubbles and arrows that one can get carried away. For example, we know that persons have female persons as mothers, so can we draw a HasMother link from Persons to FemalePersons? The answer isno, because HasMother is arelation between a person and his or her mother, and categories do not have mothers. For this reason, we have used a special notation—the double-boxed link—in Figure 10.9. This link asserts that

Vx x & Persons = [Vy HasMother(z,y) = y¢& FemalePersons] . We might also want to assert that persons have two legs—that is, Vx x € Persons = Legs(z,2) .

As before, we need to be careful not to assert that a category has legs; the single-boxed link in Figure 10.9 is used to assert properties of every member of a category.

The semantic network notation makes it very convenient to perform inheritance reasoning of the kind introduced in Section 10.2. For example, by virtue of being a person, Mary inherits the property of having two legs. Thus, to find out how many legs Mary has, the inheritance algorithm follows the MemberOf link from Mary to the category she belongs to, and then follows SubsetOf links up the hierarchy until it finds a category for which there is a boxed Legs link—in this case, the Persons category. The simplicity and efficiency of this inference mechanism, compared with logical theorem proving, has been one of the main attractions of semantic networks.

Inheritance becomes complicated when an object can belong to more than one category or when a category can be a subset of more than one other category; this is called multiple inheritance. In such cases, the inheritance algorithm might find two or more conflicting values

8 Several early systems failed to distinguish between properties of members of a category and properties of the category as a whole. This can lead directly to inconsistencies, as pointed out by Drew McDermott (1976) in his article “Artificial Intelligence Meets Natural Stupidity.” Another common problem was the use of IsA links for both subset and membership relations, in correspondence with English usage: “a cat is a mammal” and “Fifi is a cat.” See Exercise 10.25 for more on these issues.

Section 10.6.

Reasoning Systems for Categories 351

[]

INVERSE LINK

Figure 10.9 A semantic network with four objects (John, Mary, 1, and 2) and four categories. Relations are denoted by labeled links.

answering the query. For this reason, multiple inheritance is banned in some object-oriented programming (OOP) languages, such as Java, that use inheritance in a class hierarchy. It is usually allowed in semantic networks, but we defer discussion of that until Section 10.7.

Another common form of inference is the use of inverse links. For example, Has Sister is the inverse of SisterOf , which means that

Vp,s HasSister(p,s) < SisterOf(s,p) .

This sentence can be asserted in a semantic network if links are reified—that is, made into objects in their own right. For example, we could have a SisterOf object, connected by an Inverse link to HasSister. Given a query asking who is a SisterOf John, the inference algorithm can discover that HasSister is the inverse of SisterOf and can therefore answer the query by following the HasSister link from John to Mary. Without the inverse information, it might be necessary to check every female person to see whether that person has a SisterOf link to John. This is because semantic networks provide direct indexing only for objects, categories, and the links emanating from them; in the vocabulary of first-order logic, it is as if the knowledge base were indexed only on the first argument of each predicate.

The reader might have noticed an obvious drawback of semantic network notation, compared to first-order logic: the fact that links between bubbles represent only binary relations. For example, the sentence Fly(Shankar, NewYork, NewDelhi, Yesterday) cannot be asserted directly in a semantic network. Nonetheless, we can obtain the effect of n-ary assertions by reifying the proposition itself as an event (see Section 10.3) belonging to an appropriate event category. Figure 10.10 shows the semantic network structure for this particular event. Notice that the restriction to binary relations forces the creation of a rich ontology of reified concepts; indeed, much of the ontology developed in this chapter originated in semantic network systems.

Reification of propositions makes it possible to represent every ground, function-free atomic sentence of first-order logic in the semantic network notation. Certain kinds of universally quantified sentences can be asserted using inverse links and the singly boxed and doubly boxed arrows applied to categories, but that still leaves us a long way short of full first-order

352

Chapter 10. Knowledge Representation

[]

DEFAULT VALUES

OVERRIDING

Figure 10.10 A fragment of a semantic network showing the representation of the logical assertion Fly(Shankar, NewYork, NewDelhi, Yesterday).

logic. Negation, disjunction, nested function symbols, and existential quantification are all missing. Now it is possible to extend the notation to make it equivalent to first-order logic—as in Peirce’s existential graphs or Hendrix's (1975) partitioned semantic networks—but doing so negates one of the main advantages of semantic networks, which is the simplicity and transparency of the inference processes. Designers can build a large network and still have a good idea about what queries will be efficient, because (a) it is easy to visualize the steps that the inference procedure will go through and (b) in some cases the query language is so simple that difficult queries cannot be posed. In cases where the expressive power proves to be too limiting, many semantic network systems provide for procedural attachment to fill in the gaps. Procedural attachment is a technique whereby a query about (or sometimes an assertion of) a certain relation results in a call to a special procedure designed for that relation rather than a general inference algorithm.

One of the most important aspects of semantic networks is their ability to represent default values for categories. Examining Figure 10.9 carefully, one notices that John has one leg, despite the fact that he is a person and all persons have two legs. In a strictly logical KB, this would be a contradiction, but in a semantic network, the assertion that all persons have two legs has only default status; that is, a person is assumed to have two legs unless this is contradicted by more specific information. The default semantics is enforced naturally by the inheritance algorithm, because it follows links upwards from the object itself (John in this case) and stops as soon as it finds a value. We say that the default is overridden by the more specific value. Notice that we could also override the default number of legs by creating a category of OneLeggedPersons, a subset of Persons of which John is a member.

We can retain a strictly logical semantics for the network if we say that the Legs assertion for Persons includes an exception for John:

Vz x € Persons Nx # John = Legs(z,2) .

For a fixed network, this is semantically adequate, but will be much less concise than the network notation itself if there are lots of exceptions. For a network that will be updated with more assertions, however, such an approach fails—we really want to say that any persons as yet unknown with one leg are exceptions too. Section 10.7 goes into more depth on this issue and on default reasoning in general.

Section 10.6.

Reasoning Systems for Categories 353

DESCRIPTION LOGICS

SUBSUMPTION CLASSIFICATION

Description logics

The syntax of first-order logic is designed to make it easy to say things about objects. Description logics are notations that are designed to make it easier to describe definitions and properties of categories. Description logic systems evolved from semantic networks in response to pressure to formalize what the networks mean while retaining the emphasis on taxonomic structure as an organizing principle.

The principal inference tasks for description logics are subsumption—checking if one category is a subset of another by comparing their definitions—and classification—checking whether an object belongs to a category. Some systems also include consistency of a category definition—whether the membership criteria are logically satisfiable.

The CLASSIC language (Borgida ef al., 1989) is a typical description logic. The syntax of CLASSIC descriptions is shown in Figure 10.11.° For example, to say that bachelors are unmarried adult males we would write

Bachelor = And( Unmarried, Adult, Male) . The equivalent in first-order logic would be Bachelor(z) < Unmarried(xz) A Adult(z) A Male(z) .

Notice that the description logic effectively allows direct logical operations on predicates, rather than having to first create sentences to be joined by connectives. Any description in CLASSIC can be written in first-order logic, but some descriptions are more straightforward in CLASSIC. For example, to describe the set of men with at least three sons who are all unemployed and married to doctors and at most two daughters who are all professors in physics or math departments, we would use

And(Man, AtLeast(3, Son), AtMost(2, Daughter), All(Son, And( Unemployed, Married, All(Spouse, Doctor))), All( Daughter, And( Professor, Fills( Department, Physics, Math)))) .

We leave it as an exercise to translate this into first-order logic.

Perhaps the most important aspect of description logics is their emphasis on tractability of inference. A problem instance is solved by describing it and then asking if it is subsumed by one of several possible solution categories. In standard first-order logic systems, predicting the solution time is often impossible. It is frequently left to the user to engineer the representation to detour around sets of sentences that seem to be causing the system to take several weeks to solve a problem. The thrust in description logics, on the other hand, is to ensure that subsumption-testing can be solved in time polynomial in the size of the descriptions."

This sounds wonderful in principle, until one realizes that it can only have one of two consequences: either hard problems cannot be stated at all, or they require exponentially large descriptions! However, the tractability results do shed light on what sorts of constructs cause problems and thus help the user to understand how different representations behave.

9 Notice that the language does not allow one to simply state that one concept, or category, is a subset of another. This is a deliberate policy: subsumption between categories must be derivable from some aspects of the descriptions of the categories. If not, then something is missing from the descriptions.

10 CLASSIC provides efficient subsumption testing in practice, but the worst-case runtime is exponential.

354

Chapter 10. Knowledge Representation

Concept — Thing | ConceptName | And(Concept,...) | All(RoleName, Concept) | AtLeast(Integer, RoleName) | AtMost( Integer, RoleName) | Fills(RoleName, IndividualName, . . .) | SameAs(Path, Path) | OneOf(IndividualName, ...) Path — [RoleName,.. | Figure 10.11 The syntax of descriptions in a subset of the CLASSIC language.

For example, description logics usually lack negation and disjunction. Each forces first order logical systems to go through a potentially exponential case analysis in order to ensure completeness. For the same reason, they are excluded from Prolog. CLASSIC allows only a limited form of disjunction in the Fills and OneOf constructs, which permit disjunction over explicitly enumerated individuals but not over descriptions. With disjunctive descriptions, nested definitions can lead easily to an exponential number of alternative routes by which one category can subsume another.

In the preceding section, we saw a simple example of an assertion with default status: people have two legs. This default can be overridden by more specific information, such as that Long John Silver has one leg. We saw that the inheritance mechanism in semantic networks implements the overriding of defaults in a simple and natural way. In this section, we study defaults more generally, with a view toward understanding the semantics of defaults rather than just providing a procedural mechanism.

Open and closed worlds

Suppose you were looking at a bulletin board in a university computer science department and saw a notice saying, “The following courses will be offered: CS 101, CS 102, CS 106, EE 101.” Now, how many courses will be offered? If you answered “Four,” you would be in agreement with a typical database system. Given a relational database with the equivalent of the four assertions

Course(CS,101), Course(CS,102), Course(CS,106), Course(FE,101), (10.2)

the SQL query count \* from Course returns 4. On the other hand, a first-order logical system would answer “Somewhere between one and infinity,” not “four.” The reason is that

Section 10.7.

Reasoning with Default Information 355

CLOSED-WORLD ASSUMPTION

COMPLETION

CLARK NORMAL FORM

the Course assertions do not deny the possibility that other unmentioned courses are also offered, nor do they say that the courses mentioned are different from each other.

This example shows that database systems and human communication conventions differ from first-order logic in at least two ways. First, databases (and people) assume that the information provided is complete, so that ground atomic sentences not asserted to be true are assumed to be false. This is called the closed-world assumption, or CWA. Second, we usually assume that distinct names refer to distinct objects. This is the unique names assumption, or UNA, which we introduced first in the context of action names in Section 10.3.

First-order logic does not assume these conventions, and thus needs to be more explicit. To say that only the four distinct courses are offered, we would write:

Course(d,n) << [d,n]=[CS,101]V [d,n] =[CS,102] V [d,n]=[CS,106] V [d,n] = EE, 101] . (10.3) Equation 10.3 is called the completion'! of 10.2. In general, the completion will contain a definition—an if-and-only-if sentence—for each predicate, and each definition will contain a

disjunct for each definite clause having that predicate in its head.!? In general, the completion is constructed as follows:

1. Gather up all the clauses with the same predicate name (P) and the same arity (n). 2. Translate each clause to Clark Normal Form: replace P(ty,...,tn) « Body, where £; are terms, with Pv, ...,vp) — Jwi... wm [v1,...,05) = [t1,...,tx] A Body,

where v; are newly invented variables and w; are the variables that appear in the original clause. Use the same set of v; for every clause. This gives us a set of clauses

P(vi,...,vp) «— By

P(vi,...,v5) «— Bg . 3. Combine these together into one big disjunctive clause: P(vy,..., vn) — Biv...vB. 4. Form the completion by replacing the « with an equivalence: P(v1,...,vn) & Biv...v Bg. Figure 10.12 shows an example of the Clark completion for a knowledge base with both ground facts and rules. To add in the unique names assumption, we simply construct the Clark completion for the equality relation, where the only known facts are that CS = CS, 101 = 101, and so on. This is left as an exercise.

The closed-world assumption allows us to find a minimal model of a relation. That is, we can find the model of the relation Course with the fewest elements. In Equation (10.2)

11 Sometimes called “Clark Completion” after the inventor, Keith Clark. 12 Notice that this is also the form of the successor-state axioms given in Section 10.3.

[]

356

Figure 10.12 The Clark Completion of a set of Horn clauses. The original Horn program (left) lists four courses explicitly and also asserts that there is a math class for every integer from 101 to 130, and that for every CS class in the 100 (undergraduate) series, there is a corresponding class in the 200 (graduate) series. The Clark completion (right) says that there are no other classes. With the completion and the unique names assumption (and the obvious definition of the Integer predicate), we get the desired conclusion that there are exactly 36 courses: 30 math courses and 6 CS courses.

the minimal model of Course has four elements; any less and we’d have a contradiction. For Horn knowledge bases, there is always a unique minimal model. Notice that, with the unique names assumption, this applies to the equality relation too: each term is equal only to itself. Paradoxically, this means that minimal models are maximal in the sense of having as many objects as possible.

It is possible to take a Horn program, generate the Clark completion, and hand that to a theorem prover to do inference. But it is usually more efficient to use a special-purpose inference mechanism such as Prolog, which has the closed world and unique names assumptions built into the inference mechanism.

Those who make the closed-world assumption must be careful about what kind of reasoning they will be doing. For example, in a census database it would be reasonable to make the CWA when reasoning about the current population of cities, but it would be wrong to conclude that no baby will ever be born in the future just because the database contains no entries with future birthdates. The CWA makes the database complete, in the sense that every atomic query is answered either positively or negatively; when we are genuinely ignorant of facts (such as future births) we cannot use the CWA. A more sophisticated knowledge representation system might allow the user to specify rules for when to apply the CWA.

Negation as failure and stable model semantics

We saw in Chapters 7 and 9 than Horn-form knowledge bases have desirable computational properties. In many applications, however, the requirement that every literal in the body of a clause be positive is rather inconvenient. We would like to say “You can go outside if it’s not raining,” without having to concoct predicates such as NotRaining. In this section, we explore the addition of a form of explicit negation to Horn clauses using the idea using of

Section 10.7.

Reasoning with Default Information 357

NEGATION AS FAILURE

STABLE MODEL JUSTIFICATION

REDUCT

ANSWER SET PROGRAMMING

ANSWER SETS

negation as failure. The idea is that a negative literal, not P, can be “proved” true just in case the proof of P fails. This is a form of default reasoning closely related to the closed world assumption: we assume something is false if it cannot be proved true. We use “not” to distinguish negation as failure from the logical “—” operator.

Prolog allows the not operator in the body of a clause. For example, consider the following Prolog program:

IDFEdrive « Drive A not SCSIdrive . SCSIdrive «— Drive A not IDEdrive . SCSIcontroller «+ SCSIdrive .

Drive .

(10.4)

The first rule says that if we have a hard drive on a computer and it is not SCSI, then it must be IDE. The second says if it is not IDE it must be SCSI. The third says that having a SCSI drive implies having a SCSI controller, and the fourth says that we do indeed have a drive. This program has fwo minimal models:

My = { Drive, IDEdrive} , My = { Drive, SCSIdrive, SCSIcontroller} .

Minimal models do not capture the intended semantics of programs with negation as failure. Consider the program

P — not Q. (10.5)

This has two minimal models, {P} and {Q}. From an FOL point of view this makes sense, since P «< —() is equivalent to P V (). But from a Prolog point of view it is worrisome: () never appears on the left hand side of an arrow, so how can it be a consequence?

An alternative is the idea of a stable model, which is a minimal model where every atom in the model has a justification: a rule where the head is the atom and where every literal in the body is satisfied. Technically, we say that M is a stable model of a program H if M is the unique minimal model of the reduct of H with respect to M. The reduct of a program H is defined by first deleting from H any rule that has a literal not A in the body, where A is in the model, and then deleting any negative literals in the remaining rules. Since the reduct of H is now a list of Horn clauses, it must have a unique minimal model.

The reduct of P « not @Q with respect to { P} is P, which has minimal model { P}. Therefore { P} is a stable model. The reduct with respect to {@Q} is the empty program, which has minimal model {}. Therefore {Q} is not a stable model because () has no justification in Equation (10.5). As another example, the reduct of 10.4 with respect to Mj is as follows:

IDEdrive — Drive . SCSIcontroller — SCSIdrive . Drive .

This has minimal model Ay, so M; is a stable model. Answer set programming is a kind of logic programming with negation as failure that works by translating the logic program into ground form and then searching for stable models (also known as answer sets) using propositional model checking techniques. Thus answer set programming is a descendant both of Prolog and of the fast propositional satisfiability provers such as WALKS AT. Indeed,

358

Chapter 10. Knowledge Representation

NONMONOTONICITY NONMONOTONIC LOGICS

CIRCUMSCRIPTION

MODEL PREFERENCE

answer set programming has been successfully applied to problems in planning just as the propositional satisfiability provers have. The advantage of answer set planning over other planners is the degree of flexibility: the planning operators and constraints can be expressed as logic programs and are not bound to the restricted format of a particular planning formalism. The disadvantage of answer set planning is the same as for other propositional techniques: if there are very many objects in the universe, then there can be an exponential slow-down.

Circumscription and default logic

We have seen two examples where apparently natural reasoning processes violate the monotonicity property of logic that was proved in Chapter 7.13 In the first example, a property inherited by all members of a category in a semantic network could be overridden by more specific information for a subcategory. In the second example, negated literals derived from a closed-world assumption could be overridden by the addition of positive literals.

Simple introspection suggests that these failures of monotonicity are widespread in commonsense reasoning. It seems that humans often “jump to conclusions.” For example, when one sees a car parked on the street, one is normally willing to believe that it has four wheels even though only three are visible. (If you feel that the existence of the fourth wheel is dubious, consider also the question as to whether the three visible wheels are real or merely cardboard facsimiles.) Now, probability theory can certainly provide a conclusion that the fourth wheel exists with high probability, yet, for most people, the possibility of the car’s not having four wheels does not arise unless some new evidence presents itself. Thus, it seems that the four-wheel conclusion is reached by default, in the absence of any reason to doubt it. If new evidence arrives—for example, if one sees the owner carrying a wheel and notices that the car is jacked up—then the conclusion can be retracted. This kind of reasoning is said to exhibit nonmonotonicity, because the set of beliefs does not grow monotonically over time as new evidence arrives. Nonmonotonic logics have been devised with modified notions of truth and entailment in order to capture such behavior. We will look at two such logics that have been studied extensively: circumscription and default logic.

Circumscription can be seen as a more powerful and precise version of the closedworld assumption. The idea is to specify particular predicates that are assumed to be “as false as possible”—that is, false for every object except those for which they are known to be true. For example, suppose we want to assert the default rule that birds fly. We would introduce a predicate, say Abnormali(x), and write

Bird(z) A = Abnormal; (x) = Flies(x) .

If we say that Abnormal; is to be circumscribed, a circumscriptive reasoner is entitled to assume — Abnormal; (x) unless Abnormali(x) is known to be true. This allows the conclusion Flies( Tweety) to be drawn from the premise Bird( Tweety), but the conclusion no longer holds if Abnormal, ( Tweety) is asserted.

Circumscription can be viewed as an example of a model preference logic. In such logics, a sentence is entailed (with default status) if it is true in all preferred models of the KB,

13 Recall that monotonicity requires all entailed sentences to remain entailed after new sentences are added to the KB. That is, if KB |= a then KB A 8 |= c.

Section 10.7.

Reasoning with Default Information 359

PRIORITIZED CIRCUMSCRIPTION

DEFAULT LOGIC DEFAULT RULES

EXTENSION

as opposed to the requirement of truth in all models in classical logic. For circumscription, one model is preferred to another if it has fewer abnormal objects. Let us see how this idea works in the context of multiple inheritance in semantic networks. The standard example for which multiple inheritance is problematic is called the “Nixon diamond.” It arises from the observation that Richard Nixon was both a Quaker (and hence by default a pacifist) and a Republican (and hence by default not a pacifist). We can write this as follows:

Republican(Nizon) A Quaker( Nizon) .

Republican(z) A —=Abnormala(x) = —Pacifist(z) .

Quaker(z) A =Abnormalz(z) = Pacifist(z) .

If we circumscribe Abnormals and Abnormal, there are two preferred models: one in which Abnormals( Nizon) and Pacifist( Nizon) hold and one in which Abnormals(Nizon) and — Pacifist Nizon) hold. Thus, the circumscriptive reasoner remains properly agnostic as to whether Nixon is a pacifist. If we wish, in addition, to assert that religious beliefs take precedence over political beliefs, we can use a formalism called prioritized circumscription to give preference to models where Abnormals is minimized.

Default logic is a formalism in which default rules can be written to generate contingent, nonmonotonic conclusions. A default rule looks like this:

Bird(z) : Flies(x)/Flies(z) .

This rule means that if Bird(z) is true, and if Flies(x) is consistent with the knowledge base, then Flies(z) may be concluded by default. In general, a default rule has the form

P:J,....,Jn/C

where P is called the prerequisite, C' is the conclusion, and J; are the justifications—if any one of them can be proven false, then the conclusion cannot be drawn. Any variable that appears in J; or C must also appear in P. The Nixon-diamond example can be represented in default logic with one fact and two default rules:

Republican(Nizon) A Quaker(Nizon) .

Republican(z) : ~Pacifist(x)/—Pacifist(x) .

Quaker(x) : Pacifist(z)/Pacifist(x) . To interpret what the default rules mean, we define the notion of an extension of a default theory to be a maximal set of consequences of the theory. That is, an extension .S consists of the original known facts and a set of conclusions from the default rules, such that no additional conclusions can be drawn from .S and the justifications of every default conclusion in .S are consistent with S. As in the case of the preferred models in circumscription, we have two possible extensions for the Nixon diamond: one wherein he is a pacifist and one wherein he is not. Prioritized schemes exist in which some default rules can be given precedence over others, allowing some ambiguities to be resolved.

Since 1980, when nonmonotonic logics were first proposed, a great deal of progress has been made in understanding their mathematical properties. Beginning in the late 1990s, 14 For the closed-world assumption, one model is preferred to another if it has fewer true atoms—that is, preferred

models are minimal models. There is a natural connection between the CWA and definite clause KBs, because the fixed point reached by forward chaining on such KBs is the unique minimal model. (See page 219.)

360

Chapter 10. Knowledge Representation

practical systems based on logic programming have shown promise as knowledge representation tools. There are still unresolved questions, however. For example, if “Cars have four wheels” is false, what does it mean to have it in one’s knowledge base? What is a good set of default rules to have? If we cannot decide, for each rule separately, whether it belongs in our knowledge base, then we have a serious problem of nonmodularity. Finally, how can beliefs that have default status be used to make decisions? This is probably the hardest issue for default reasoning. Decisions often involve tradeoffs, and one therefore needs to compare the strengths of belief in the outcomes of different actions. In cases where the same kinds of decisions are being made repeatedly, it is possible to interpret default rules as “threshold probability” statements. For example, the default rule “My brakes are always OK” really means “The probability that my brakes are OK, given no other information, is sufficiently high that the optimal decision is for me to drive without checking them.” When the decision context changes—for example, when one is driving a heavily laden truck down a steep mountain road—the default rule suddenly becomes inappropriate, even though there is no new evidence to suggest that the brakes are faulty. These considerations have led some researchers to consider how to embed default reasoning in probability theory.

BELIEF REVISION

TRUTH MAINTENANCE SYSTEM

The previous section argued that many of the inferences drawn by a knowledge representation system will have only default status, rather than being absolutely certain. Inevitably, some of these inferred facts will turn out to be wrong and will have to be retracted in the face of new information. This process is called belief revision.!> Suppose that a knowledge base KB contains a sentence P—perhaps a default conclusion recorded by a forward-chaining algorithm, or perhaps just an incorrect assertion—and we want to execute TELL(KB, —P). To avoid creating a contradiction, we must first execute RETRACT(KB, P). This sounds easy enough. Problems arise, however, if any additional sentences were inferred from P and asserted in the KB. For example, the implication P = (J) might have been used to add 2. The obvious “solution”—retracting all sentences inferred from P—fails because such sentences may have other justifications besides P. For example, if KR and BR = ( are also in the KB, then ¢) does not have to be removed after all. Truth maintenance systems, or TMSs, are designed to handle exactly these kinds of complications.

One very simple approach to truth maintenance is to keep track of the order in which sentences are told to the knowledge base by numbering them from P; to P,. When the call RETRACT(KB, F;) is made, the system reverts to the state just before P; was added, thereby removing both FP; and any inferences that were derived from P;. The sentences FP; through P,, can then be added again. This is simple, and it guarantees that the knowledge base will be consistent, but retracting F; requires retracting and reasserting n — 7 sentences as well as

15 Belief revision is often contrasted with belief update, which occurs when a knowledge base is revised to reflect a change in the world rather than new information about a fixed world. Belief update combines belief revision with reasoning about time and change; it is also related to the process of filtering described in Chapter 15.

Section 10.8.

Truth Maintenance Systems 361

JTMS JUSTIFICATION

ATMS

EXPLANATIONS

undoing and redoing all the inferences drawn from those sentences. For systems to which many facts are being added—such as large commercial databases—this is impractical.

A more efficient approach is the justification-based truth maintenance system, or JTMS. In a JTMS, each sentence in the knowledge base is annotated with a justification consisting of the set of sentences from which it was inferred. For example, if the knowledge base already contains P = @), then TELL(P) will cause @ to be added with the justification {P, P = @Q}. In general, a sentence can have any number of justifications. Justifications are used to make retraction efficient. Given the call RETRACT(P), the JTMS will delete exactly those sentences for which P is a member of every justification. So, if a sentence @ had the single justification {P, P = Q} it would be removed, if it had the additional justification {P, PV R = Q} it would still be removed, but if it also had the justification {R, PVR = Q}, then it would be spared. In this way, the time required for retraction of P depends only on the number of sentences derived from P rather than on the number of other sentences added since P entered the knowledge base.

The JTMS assumes that sentences that are considered once will probably be considered again, so rather than deleting a sentence from the knowledge base entirely when it loses all justifications, we merely mark the sentence as being out of the knowledge base. If a subsequent assertion restores one of the justifications, then we mark the sentence as being back in. In this way, the JTMS retains all of the inference chains that it uses and need not rederive sentences when a justification becomes valid again.

In addition to handling the retraction of incorrect information, TMSs can be used to speed up the analysis of multiple hypothetical situations. Suppose, for example, that the Romanian Olympic Committee is choosing sites for the swimming, athletics, and equestrian events at the 2048 Games to be held in Romania. For example, let the first hypothesis be Site( Swimming, Pitesti), Site( Athletics, Bucharest), and Site( Equestrian, Arad). A great deal of reasoning must then be done to work out the logistical consequences and hence the desirability of this selection. If we want to consider Site( Athletics, Sibiu) instead, the TMS avoids the need to start again from scratch. Instead, we simply retract Site( Athletics, Bucharest) and assert Site( Athletics, Sibiu) and the TMS takes care of the necessary revisions. Inference chains generated from the choice of Bucharest can be reused with Sibiu, provided that the conclusions are the same.

An assumption-based truth maintenance system, or ATMS, is designed to make this type of context-switching between hypothetical worlds particularly efficient. In a JTMS, the maintenance of justifications allows you to move quickly from one state to another by making a few retractions and assertions, but at any time only one state is represented. An ATMS represents all the states that have ever been considered at the same time. Whereas a JTMS simply labels each sentence as being in or out, an ATMS keeps track, for each sentence, of which assumptions would cause the sentence to be true. In other words, each sentence has a label that consists of a set of assumption sets. The sentence holds just in those cases where all the assumptions in one of the assumption sets hold.

Truth maintenance systems also provide a mechanism for generating explanations. Technically, an explanation of a sentence P is a set of sentences F such that F entails P. If the sentences in F are already known to be true, then F simply provides a sufficient ba

362

Chapter 10. Knowledge Representation

ASSUMPTIONS

sis for proving that P must be the case. But explanations can also include assumptions— sentences that are not known to be true, but would suffice to prove P if they were true. For example, one might not have enough information to prove that one’s car won’t start, but a reasonable explanation might include the assumption that the battery is dead. This, combined with knowledge of how cars operate, explains the observed nonbehavior. In most cases, we will prefer an explanation F that is minimal, meaning that there is no proper subset of E that is also an explanation. An ATMS can generate explanations for the “car won’t start” problem by making assumptions (such as “gas in car” or “battery dead”) in any order we like, even if some assumptions are contradictory. Then we look at the label for the sentence “car won't start” to read off the sets of assumptions that would justify the sentence.

The exact algorithms used to implement truth maintenance systems are a little complicated, and we do not cover them here. The computational complexity of the truth maintenance problem is at least as great as that of propositional inference—that is, NP-hard. Therefore, you should not expect truth maintenance to be a panacea. When used carefully, however, a TMS can provide a substantial increase in the ability of a logical system to handle complex environments and hypotheses.

This has been the most detailed chapter of the book so far. By delving into the details of how one represents a variety of knowledge, we hope we have given the reader a sense of how real knowledge bases are constructed. The major points are as follows:

e Large-scale knowledge representation requires a general-purpose ontology to organize and tie together the various specific domains of knowledge.

e A general-purpose ontology needs to cover a wide variety of knowledge and should be capable, in principle, of handling any domain.

e We presented an upper ontology based on categories and the event calculus. We covered structured objects, time and space, change, processes, substances, and beliefs.

e Actions, events, and time can be represented either in situation calculus or in more expressive representations such as event calculus and fluent calculus. Such representations enable an agent to construct plans by logical inference.

e The mental states of agents can be represented by strings that denote beliefs.

e We presented a detailed analysis of the Internet shopping domain, exercising the general ontology and showing how the domain knowledge can be used by a shopping agent.

e Special-purpose representation systems, such as semantic networks and description logics, have been devised to help in organizing a hierarchy of categories. Inheritance is an important form of inference, allowing the properties of objects to be deduced from their membership in categories.

e The closed-world assumption, as implemented in logic programs, provides a simple way to avoid having to specify lots of negative information. It is best interpreted as a default that can be overridden by additional information.

Section 10.9.

Summary 363

e Nonmoneotonic logics, such as circumscription and default logic, are intended to capture default reasoning in general. Answer set programming speeds up nonmonotonic inference, much as WALKS AT speeds up propositional inference.

e Truth maintenance systems handle knowledge updates and revisions efficiently.

There are plausible claims (Briggs, 1985) that formal knowledge representation research began with classical Indian theorizing about the grammar of Shastric Sanskrit, which dates back to the first millennium B.C. In the West, the use of definitions of terms in ancient Greek mathematics can be regarded as the earliest instance. Indeed, the development of technical terminology in any field can be regarded as a form of knowledge representation.

Early discussions of representation in Al tended to focus on “problem representation” rather than “knowledge representation.” (See, for example, Amarel’s (1968) discussion of the Missionaries and Cannibals problem.) In the 1970s, AI emphasized the development of “‘expert systems” (also called “knowledge-based systems”) that could, if given the appropriate domain knowledge, match or exceed the performance of human experts on narrowly defined tasks. For example, the first expert system, DENDRAL (Feigenbaum et al, 1971; Lindsay et al., 1980), interpreted the output of a mass spectrometer (a type of instrument used to analyze the structure of organic chemical compounds) as accurately as expert chemists. Although the success of DENDRAL was instrumental in convincing the Al research community of the importance of knowledge representation, the representational formalisms used in DENDRAL are highly specific to the domain of chemistry. Over time, researchers became interested in standardized knowledge representation formalisms and ontologies that could streamline the process of creating new expert systems. In so doing, they ventured into territory previously explored by philosophers of science and of language. The discipline imposed in Al by the need for one’s theories to “work’ has led to more rapid and deeper progress than was the case when these problems were the exclusive domain of philosophy (although it has at times also led to the repeated reinvention of the wheel).

The creation of comprehensive taxonomies or classifications dates back to ancient times. Aristotle (384-322 B.C.) strongly emphasized classification and categorization schemes. His Organon, a collection of works on logic assembled by his students after his death, included a treatise called Categories in which he attempted to construct what we would now call an upper ontology. He also introduced the notions of genus and species for lower-level classification, although not with their modern, specifically biological meaning. Our present system of biological classification, including the use of “binomial nomenclature” (classification via genus and species in the technical sense), was invented by the Swedish biologist Carolus Linnaeus, or Carl von Linne (1707-1778). The problems associated with natural kinds and inexact category boundaries have been addressed by Wittgenstein (1953), Quine (1953), Lakoff (1987), and Schwartz (1977), among others.

Interest in larger-scale ontologies is increasing. The CYC project (Lenat, 1995; Lenat and Guha, 1990) has released a 6,000-concept upper ontology with 60,000 facts, and licenses

364

Chapter 10. Knowledge Representation

a much larger global ontology. The IEEE has established subcommittee P1600.1, the Standard Upper Ontology Working Group, and the Open Mind Initiative has enlisted over 7,000 Internet users to enter more than 400,000 facts about commonsense concepts. On the Web, standards such as RDF, XML, and the Semantic Web (Berners-Lee et al., 2001) are emerging, but are not yet widely used. The conferences on Formal Ontology in Information Systems (FOIS) contain many interesting papers on both general and domain-specific ontologies.

The taxonomy used in this chapter was developed by the authors and is based in part on their experience in the CYC project and in part on work by Hwang and Schubert (1993) and Davis (1990). An inspirational discussion of the general project of commonsense knowledge representation appears in Hayes’s (1978, 1985b) “The Naive Physics Manifesto.”

The representation of time, change, actions, and events has been studied extensively in philosophy and theoretical computer science as well as in Al. The oldest approach is temporal logic, which is a specialized logic in which each model describes a complete trajectory through time (usually either linear or branching), rather than just a static relational structure. The logic includes modal operators that are applied to formulas; Op means “p will be true at all times in the future,” and {p means “p will be true at some time in the future.” The study of temporal logic was initiated by Aristotle and the Megarian and Stoic schools in ancient Greece. In modern times, Findlay (1941) was the first to suggest a formal calculus for reasoning about time, but the work of Arthur Prior (1967) is considered the most influential. Textbooks on temporal logic include those by Rescher and Urquhart (1971) and van Benthem (1983).

Theoretical computer scientists have long been interested in formalizing the properties of programs, viewed as sequences of computational actions. Burstall (1974) introduced the idea of using modal operators to reason about computer programs. Soon thereafter, Vaughan Pratt (1976) designed dynamic logic, in which modal operators indicate the effects of programs or other actions (see also Harel, 1984). For instance, in dynamic logic, if « is the name of a program, then “[a]p” means “p would be true in all world states resulting from executing program « in the current world state”, and “(«)p” means “p would be true in at least one world state resulting from executing program « in the current world state.” Dynamic logic was applied to the actual analysis of programs by Fischer and Ladner (1977). Pnueli (1977) introduced the idea of using classical temporal logic to reason about programs.

Whereas temporal logic puts time directly into the model theory of the language, representations of time in Al have tended to incorporate axioms about times and events explicitly in the knowledge base, giving time no special status in the logic. This approach can allow for greater clarity and flexibility in some cases. Also, temporal knowledge expressed in firstorder logic can be more easily integrated with other knowledge that has been accumulated in that notation.

The earliest treatment of time and action in Al was John McCarthy’s (1963) situation calculus. The first Al system to make substantial use of general-purpose reasoning about actions in first-order logic was QA3 (Green, 1969b). Kowalski (1979b) developed the idea of reifying propositions within situation calculus.

The frame problem was first recognized by McCarthy and Hayes (1969). Many researchers considered the problem insoluble within first-order logic, and it spurred a great

Section 10.9.

Summary 365

COGNITIVE ROBOTICS

MEREOLOGY

deal of research into nonmonotonic logics. Philosophers from Dreyfus (1972) to Crockett (1994) have cited the frame problem as one symptom of the inevitable failure of the entire Al enterprise. The partial solution of the representational frame problem using successorstate axioms is due to Ray Reiter (1991); a solution of the inferential frame problem can be traced to work by Holldobler and Schneeberger (1990) on what became known as fluent calculus (Thielscher, 1999). The discussion in this chapter is based partly on the analyses by Lin and Reiter (1997) and Thielscher (1999). Books by Shanahan (1997) and Reiter (2001b) give complete, modern treatments of reasoning about action in situation calculus.

The partial resolution of the frame problem has rekindled interest in the declarative approach to reasoning about actions, which had been eclipsed by special-purpose planning systems since the early 1970s. (See Chapter 11.) Under the banner of cognitive robotics, much progress has been made on logical representations of action and time. The GOLOG language uses the full expressive power of logic programming to describe actions and plans (Levesque et al., 1997a) and has been extended to handle concurrent actions (Giacomo et al., 2000), stochastic environments (Boutilier ef al., 2000), and sensing (Reiter, 2001a).

The event calculus was introduced by Kowalski and Sergot (1986) to handle continuous time, and there have been several variations (Sadri and Kowalski, 1995). Shanahan (1999) presents a good short overview. James Allen introduced time intervals for the same reason (Allen, 1983, 1984), arguing that intervals were much more natural than situations for reasoning about extended and concurrent events. Peter Ladkin (1986a, 1986b) introduced “concave” time intervals (intervals with gaps; essentially, unions of ordinary “convex” time intervals) and applied the techniques of mathematical abstract algebra to time representation. Allen (1991) systematically investigates the wide variety of techniques available for time representation. Shoham (1987) describes the reification of events and sets forth a novel scheme of his own for the purpose. There are significant commonalities between the eventbased ontology given in this chapter and an analysis of events due to the philosopher Donald Davidson (1980). The histories in Pat Hayes’s (1985a) ontology of liquids also have much the same flavor.

The question of the ontological status of substances has a long history. Plato proposed that substances were abstract entities entirely distinct from physical objects; he would say MadeOf ( Butters, Butter) rather than Butters € Butter. This leads to a substance hierarchy in which, for example, Unsalted Butter is a more specific substance than Butter. The position adopted in this chapter, in which substances are categories of objects, was championed by Richard Montague (1973). It has also been adopted in the CYC project. Copeland (1993) mounts a serious, but not invincible, attack. The alternative approach mentioned in the chapter, in which butter is one object consisting of all buttery objects in the universe, was proposed originally by the Polish logician Le$niewski (1916). His mereology (the name is derived from the Greek word for “part”) used the part—whole relation as a substitute for mathematical set theory, with the aim of eliminating abstract entities such as sets. A more readable exposition of these ideas is given by Leonard and Goodman (1940), and Goodman’s The Structure of Appearance (1977) applies the ideas to various problems in knowledge representation. While some aspects of the mereological approach are awkward—for example, the need for a separate inheritance mechanism based on part—-whole relations—the approach gained the

366

Chapter 10. Knowledge Representation

MODAL LOGIC

support of Quine (1960). Harry Bunt (1985) has provided an extensive analysis of its use in knowledge representation.

Mental objects and states have been the subject of intensive study in philosophy and Al Modal logic is the classical method for reasoning about knowledge in philosophy. Modal logic augments first-order logic with modal operators, such as B (believes) and K (knows), that take sentences rather than terms as arguments. The proof theory for modal logic restricts substitution within modal contexts, thereby achieving referential opacity. The modal logic of knowledge was invented by Jaakko Hintikka (1962). Saul Kripke (1963) defined the semantics of the modal logic of knowledge in terms of possible worlds. Roughly speaking, a world is possible for an agent if it is consistent with everything the agent knows. From this, one can derive rules of inference involving the K operator. Robert C. Moore relates the modal logic of knowledge to a style of reasoning about knowledge that refers directly to possible worlds in first-order logic (Moore, 1980, 1985). Modal logic can be an intimidatingly arcane field, but it has found significant applications in reasoning about information in distributed computer systems. The book Reasoning about Knowledge by Fagin et al. (1995) provides a thorough introduction to the modal approach. The biennial conference on Theoretical Aspects of Reasoning About Knowledge (TARK) covers applications of the theory of knowledge in Al, economics, and distributed systems.

The syntactic theory of mental objects was first studied in depth by Kaplan and Montague (1960), who showed that it led to paradoxes if not handled carefully. Because it has a natural model in terms of beliefs as physical configurations of a computer or a brain, it has been popular in Al in recent years. Konolige (1982) and Haas (1986) used it to describe inference engines of limited power, and Morgenstern (1987) showed how it could be used to describe knowledge preconditions in planning. The methods for planning observation actions in Chapter 12 are based on the syntactic theory. Ernie Davis (1990) gives an excellent comparison of the syntactic and modal theories of knowledge.

The Greek philosopher Porphyry (c. 234-305 A.D.), commenting on Aristotle’s Categories, drew what might qualify as the first semantic network. Charles S. Peirce (1909) developed existential graphs as the first semantic network formalism using modern logic. Ross Quillian (1961), driven by an interest in human memory and language processing, initiated work on semantic networks within Al. An influential paper by Marvin Minsky (1975) presented a version of semantic networks called frames; a frame was a representation of an object or category, with attributes and relations to other objects or categories. Although the paper served to initiate interest in the field of knowledge representation per se, it was criticized as a recycling of earlier ideas developed in object-oriented programming, such as inheritance and the use of default values (Dahl ef al., 1970; Birtwistle et al., 1973). It is not clear to what extent the latter papers on object-oriented programming were influenced in turn by early AI work on semantic networks.

The question of semantics arose quite acutely with respect to Quillian’s semantic networks (and those of others who followed his approach), with their ubiquitous and very vague “IS-A links,” as well as other early knowledge representation formalisms such as that of MERLIN (Moore and Newell, 1973) with its mysterious “flat” and “cover” operations. Woods’ (1975) famous article “What’s In a Link?” drew the attention of Al researchers to the

Section 10.9.

Summary 367

need for precise semantics in knowledge representation formalisms. Brachman (1979) elaborated on this point and proposed solutions. Patrick Hayes’s (1979) “The Logic of Frames” cut even deeper, claiming that “Most of ‘frames’ is just a new syntax for parts of first-order logic” Drew McDermott’s (1978b) “Tarskian Semantics, or, No Notation without Denotation!” argued that the model-theoretic approach to semantics used in first-order logic should be applied to all knowledge representation formalisms. This remains a controversial idea; notably, McDermott himself has reversed his position in “A Critique of Pure Reason” (McDermott, 1987). NETL (Fahlman, 1979) was a sophisticated semantic network system whose IS-A links (called “virtual copy,” or VC, links) were based more on the notion of “inheritance” characteristic of frame systems or of object-oriented programming languages than on the subset relation and were much more precisely defined than Quillian’s links from the pre-Woods era. NETL is particularly intriguing because it was intended to be implemented in parallel hardware to overcome the difficulty of retrieving information from large semantic networks. David Touretzky (1986) subjects inheritance to rigorous mathematical analysis. Selman and Levesque (1993) discuss the complexity of inheritance with exceptions, showing that in most formulations it is NP-complete.

The development of description logics is the most recent stage in a long line of research aimed at finding useful subsets of first-order logic for which inference is computationally tractable. Hector Levesque and Ron Brachman (1987) showed that certain logical constructs—notably, certain uses of disjunction and negation—were primarily responsible for the intractability of logical inference. Building on the KL-ONE system (Schmolze and Lipkis, 1983), a number of systems have been developed whose designs incorporate the results of theoretical complexity analysis, most notably KRYPTON (Brachman et al., 1983) and Classic (Borgida et al., 1989). The result has been a marked increase in the speed of inference and a much better understanding of the interaction between complexity and expressiveness in reasoning systems. Calvanese et al. (1999) summarize the state of the art. Against this trend, Doyle and Patil (1991) have argued that restricting the expressiveness of a language either makes it impossible to solve certain problems or encourages the user to circumvent the language restrictions through nonlogical means.

The three main formalisms for dealing with nonmonotonic inference—circumscription (McCarthy, 1980), default logic (Reiter, 1980), and modal nonmonotonic logic (McDermott and Doyle, 1980)—were all introduced in one special issue of the Al Journal. Answer set programming can be seen as an extension of negation as failure or as a refinement of circumscription; the underlying theory of stable model semantics was introduced by Gelfond and Lifschitz (1988) and the leading answer set programming systems are DLV (Eiter et al., 1998) and SMODELS (Niemeli et al., 2000). The disk drive example comes from the SMODELS user manual (Syrjanen, 2000). Lifschitz (2001) discusses the use of answer set programming for planning. Brewka er al. (1997) give a good overview of the various approaches to nonmonotonic logic. Clark (1978) covers the negation-as-failure approach to logic programming and Clark completion. Van Emden and Kowalski (1976) show that every Prolog program without negation has a unique minimal model. Recent years have seen renewed interest in applications of nonmonotonic logics to large-scale knowledge representation systems. The BENINQ systems for handling insurance benefits inquiries was perhaps the first commercially success

368 Chapter 10. Knowledge Representation

ful application of a nonmonotonic inheritance system (Morgenstern, 1998). Lifschitz (2001) discusses the application of answer set programming to planning. A variety of nonmonotonic reasoning systems based on logic programming are documented in the proceedings of the conferences on Logic Programming and Nonmonotonic Reasoning (LPNMR).

The study of truth maintenance systems began with the TMS (Doyle, 1979) and RUP (McAllester, 1980) systems, both of which were essentially JTMSs. The ATMS approach was described in a series of papers by Johan de Kleer (1986a, 1986b, 1986¢). Building Problem Solvers (Forbus and de Kleer, 1993) explains in depth how TMSs can be used in Al applications. Nayak and Williams (1997) show how an efficient TMS makes it feasible to plan the operations of a NASA spacecraft in real time.

For obvious reasons, this chapter does not cover every area of knowledge representation in depth. The three principal topics omitted are the following:

QUALITATIVE ¢ Qualitative physics: Qualitative physics is a subfield of knowledge representation concerned specifically with constructing a logical, nonnumeric theory of physical objects and processes. The term was coined by Johan de Kleer (1975), although the enterprise could be said to have started in Fahlman’s (1974) BUILD, a sophisticated planner for constructing complex towers of blocks. Fahlman discovered in the process of designing it that most of the effort (80%, by his estimate) went into modeling the physics of the blocks world to calculate the stability of various subassemblies of blocks, rather than into planning per se. He sketches a hypothetical naive-physics-like process to explain why young children can solve BUILD-like problems without access to the highspeed floating-point arithmetic used in BUILD’s physical modeling. Hayes (1985a) uses “histories” —four-dimensional slices of space-time similar to Davidson’s events— to construct a fairly complex naive physics of liquids. Hayes was the first to prove that a bath with the plug in will eventually overflow if the tap keeps running and that a person who falls into a lake will get wet all over. De Kleer and Brown (1985) and Ken Forbus (1985) attempted to construct something like a general-purpose theory of the physical world, based on qualitative abstractions of physical equations. In recent years, qualitative physics has developed to the point where it is possible to analyze an impressive variety of complex physical systems (Sacks and Joskowicz, 1993; Yip, 1991). Qualitative techniques have been used to construct novel designs for clocks, windscreen wipers, and six-legged walkers (Subramanian, 1993; Subramanian and Wang, 1994). The collection Readings in Qualitative Reasoning about Physical Systems (Weld and de Kleer, 1990) provides a good introduction to the field.

SPATIAL REASONING {> Spatial reasoning: The reasoning necessary to navigate in the wumpus world and shopping world is trivial in comparison to the rich spatial structure of the real world. The earliest serious attempt to capture commonsense reasoning about space appears in the work of Ernest Davis (1986, 1990). The region connection calculus of Cohn et al. (1997) supports a form of qualitative spatial reasoning and has led to new kinds of geographical information system. As with qualitative physics, an agent can go a long way, so to speak, without resorting to a full metric representation. When such a representation is necessary, techniques developed in robotics (Chapter 25) can be used.

Section 10.9.

Summary 369

PSYCHOLOGICAL REASONING

¢& Psychological reasoning: Psychological reasoning involves the development of a working psychology for artificial agents to use in reasoning about themselves and other agents. This is often based on so-called folk psychology, the theory that humans in general are believed to use in reasoning about themselves and other humans. When Al researchers provide their artificial agents with psychological theories for reasoning about other agents, the theories are frequently based on the researchers’ description of the logical agents’ own design. Psychological reasoning is currently most useful within the context of natural language understanding, where divining the speaker’s intentions is of paramount importance.

The proceedings of the international conferences on Principles of Knowledge Representation and Reasoning provide the most up-to-date sources for work in this area. Readings in Knowledge Representation (Brachman and Levesque, 1985) and Formal Theories of the Commonsense World (Hobbs and Moore, 1985) are excellent anthologies on knowledge representation; the former focuses more on historically important papers in representation languages and formalisms, the latter on the accumulation of the knowledge itself. Davis (1990), Stefik (1995), and Sowa (1999) provide textbook introductions to knowledge representation.

EXERCISES

10.1 Write sentences to define the effects of the Shoot action in the wumpus world. Describe its effects on the wumpus and remember that shooting uses the agent’s arrow.

10.2 Within situation calculus, write an axiom to associate time O with the situation Sy and another axiom to associate the time ¢ with any situation that is derived from Sy by a sequence of ¢ actions.

10.3 In this exercise, we will consider the problem of planning a route for a robot to take

from one city to another. The basic action taken by the robot is Go(x, y), which takes it from

city x to city y if there is a direct route between those cities. DirectRoute(x,y) is true if and

only if there is a direct route from x to y; you can assume that all such facts are already in the

KB. (See the map on page 63.) The robot begins in Arad and must reach Bucharest.

. Write a suitable logical description of the initial situation of the robot.

Write a suitable logical query whose solutions will provide possible paths to the goal.

Write a sentence describing the Go action.

Now suppose that following the direct route between two cities consumes an amount of

fuel equal to the distance between the cities. The robot starts with fuel at full capacity.

Augment your representation to include these considerations. Your action description

should be such that the query you specified earlier will still result in feasible plans.

e. Describe the initial situation, and write a new rule or rules describing the Go action.

f. Now suppose some of the vertices are also gas stations, at which the robot can fill its tank. Extend your representation and write all the rules needed to describe gas stations, including the F'illup action.

Be Tw

370

Chapter 10. Knowledge Representation

EE

10.4 Investigate ways to extend the event calculus to handle simultaneous events. Is it possible to avoid a combinatorial explosion of axioms?

10.5 Represent the following seven sentences using and extending the representations developed in the chapter: a. Water is a liquid between 0 and 100 degrees. b. Water boils at 100 degrees. . The water in John’s water bottle is frozen. . Perrier is a kind of water. John has Perrier in his water bottle.

=e 8&0

All liquids have a freezing point. g. A liter of water weighs more than a liter of alcohol.

Now repeat the exercise using a representation based on the mereological approach, in which, for example, Water is an object containing as parts all the water in the world.

10.6 Write definitions for the following:

a. ExhaustivePartDecomposition b. PartPartition ¢. PartwiseDisjoint

These should be analogous to the definitions for Exhaustive Decomposition, Partition, and Disjoint. Ts it the case that PartPartition(s, BunchOf(s))? If so, prove it; if not, give a counterexample and define sufficient conditions under which it does hold.

10.7 Write a set of sentences that allows one to calculate the price of an individual tomato (or other object), given the price per pound. Extend the theory to allow the price of a bag of tomatoes to be calculated.

10.8 An alternative scheme for representing measures involves applying the units function to an abstract length object. In such a scheme, one would write Inches(Length(L1)) = 1.5. How does this scheme compare with the one in the chapter? Issues include conversion axioms, names for abstract quantities (such as “50 dollars”), and comparisons of abstract measures in different units (50 inches is more than 50 centimeters).

10.9 Construct a representation for exchange rates between currencies that allows fluctuations on a daily basis.

10.10 This exercise concerns the relationships between event categories and the time intervals in which they occur. a. Define the predicate T'(c, 7) in terms of During and €. b. Explain precisely why we do not need two different notations to describe conjunctive event categories. c. Give a formal definition for 7'(OneOf (p,q), 1) and T'( Either(p, q), 1). d. Explain why it makes sense to have two forms of negation of events, analogous to the two forms of disjunction. Call them Not and Never and give them formal definitions.

Section 10.9.

Summary 371

10.11 Define the predicate Fized, where Fixed (Location(x)) means that the location of object x is fixed over time.

10.12 Define the predicates Before, After, During, and Overlap, using the predicate Meet and the functions Start and End, but not the function Time or the predicate <.

10.13 Section 10.5 used the predicates Link and LinkText to describe connections between web pages. Using the InTag and GetPage predicates, among others, write definitions for Link and LinkText.

10.14 One part of the shopping process that was not covered in this chapter is checking for compatibility between items. For example, if a customer orders a computer, is it matched with the right peripherals? If a digital camera is ordered, does it have the right memory card and batteries? Write a knowledge base that will decide whether a set of items is compatible and that can be used to suggest replacements or additional items if they are not compatible. Make sure that the knowledge base works with at least one line of products, and is easily extensible to other lines.

10.15 Add rules to extend the definition of the predicate Name(s, ¢) so that a string such as “laptop computer” matches against the appropriate category names from a variety of stores. Try to make your definition general. Test it by looking at ten online stores, and at the category names they give for three different categories. For example, for the category of laptops, we found the names “Notebooks,” “Laptops,” “Notebook Computers,” “Notebook,” “Laptops and Notebooks,” and “Notebook PCs.” Some of these can be covered by explicit Name facts, while others could be covered by rules for handling plurals, conjunctions, etc.

10.16 A complete solution to the problem of inexact matches to the buyer’s description in shopping is very difficult and requires a full array of natural language processing and information retrieval techniques. (See Chapters 22 and 23.) One small step is to allow the user to specify minimum and maximum values for various attributes. We will insist that the buyer use the following grammar for product descriptions:

Description — Category [Connector Modifier]x

Connector —— “with” | “and” | “” Modifier — Attribute | Attribute Op Value Op — “\_" | “>” | “ee”

Here, Category names a product category, Attribute is some feature such as “CPU” or “price,” and Value is the target value for the attribute. So the query “computer with at least a 2.5-GHz CPU for under $1000” must be re-expressed as “computer with CPU > 2.5 GHz and price < $1000.” Implement a shopping agent that accepts descriptions in this language.

10.17 Our description of Internet shopping omitted the all-important step of actually buying the product. Provide a formal logical description of buying, using event calculus. That is, define the sequence of events that occurs when a buyer submits a credit card purchase and then eventually gets billed and receives the product.

10.18 Describe the event of trading something for something else. Describe buying as a kind of trading in which one of the objects traded is a sum of money.

372

Chapter 10. Knowledge Representation

10.19 The two preceding exercises assume a fairly primitive notion of ownership. For example, the buyer starts by owning the dollar bills. This picture begins to break down when, for example, one’s money is in the bank, because there is no longer any specific collection of dollar bills that one owns. The picture is complicated still further by borrowing, leasing, renting, and bailment. Investigate the various commonsense and legal concepts of ownership, and propose a scheme by which they can be represented formally.

10.20 You are to create a system for advising computer science undergraduates on what courses to take over an extended period in order to satisfy the program requirements. (Use whatever requirements are appropriate for your institution.) First, decide on a vocabulary for representing all the information, and then represent it; then use an appropriate query to the system, that will return a legal program of study as a solution. You should allow for some tailoring to individual students, in that your system should ask what courses or equivalents the student has already taken, and not generate programs that repeat those courses.

Suggest ways in which your system could be improved—for example to take into account knowledge about student preferences, the workload, good and bad instructors, and so on. For each kind of knowledge, explain how it could be expressed logically. Could your system easily incorporate this information to find the best program of study for a student?

10.21 Figure 10.1 shows the top levels of a hierarchy for everything. Extend it to include as many real categories as possible. A good way to do this is to cover all the things in your everyday life. This includes objects and events. Start with waking up, and proceed in an orderly fashion noting everything that you see, touch, do, and think about. For example, a random sampling produces music, news, milk, walking, driving, gas, Soda Hall, carpet, talking, Professor Fateman, chicken curry, tongue, $7, sun, the daily newspaper, and so on.

You should produce both a single hierarchy chart (on a large sheet of paper) and a listing of objects and categories with the relations satisfied by members of each category. Every object should be in a category, and every category should be in the hierarchy.

10.22 (Adapted from an example by Doug Lenat.) Your mission is to capture, in logical form, enough knowledge to answer a series of questions about the following simple sentence: Yesterday John went to the North Berkeley Safeway supermarket and bought two

pounds of tomatoes and a pound of ground beef. Start by trying to represent the content of the sentence as a series of assertions. You should write sentences that have straightforward logical structure (e.g., statements that objects have certain properties, that objects are related in certain ways, that all objects satisfying one property satisfy another). The following might help you get started: e Which classes, objects, and relations would you need? What are their parents, siblings and so on? (You will need events and temporal ordering, among other things.) e Where would they fit in a more general hierarchy? e What are the constraints and interrelationships among them? e How detailed must you be about each of the various concepts? The knowledge base you construct must be capable of answering a list of questions that we will give shortly. Some of the questions deal with the material stated explicitly in the story,

Section 10.9.

Summary 373

but most of them require one to have other background knowledge—to read between the lines. You’ll have to deal with what kind of things are at a supermarket, what is involved with purchasing the things one selects, what will purchases be used for, and so on. Try to make your representation as general as possible. To give a trivial example: don’t say “People buy food from Safeway,” because that won’t help you with those who shop at another supermarket. Don’t say “Joe made spaghetti with the tomatoes and ground beef,” because that won’t help you with anything else at all. Also, don’t turn the questions into answers; for example, question (c) asks “Did John buy any meat?”—not “Did John buy a pound of ground beef?”

Sketch the chains of reasoning that would answer the questions. In the process of doing so, you will no doubt need to create additional concepts, make additional assertions, and so on. If possible, use a logical reasoning system to demonstrate the sufficiency of your knowledge base. Many of the things you write might be only approximately correct in reality, but don’t worry too much; the idea is to extract the common sense that lets you answer these questions at all. A truly complete answer to this question is extremely difficult, probably beyond the state of the art of current knowledge representation. But you should be able to put together a consistent set of axioms for the limited questions posed here.

a. Is John a child or an adult? [Adult]

b. Does John now have at least two tomatoes? [Yes]

Did John buy any meat? [Yes]

If Mary was buying tomatoes at the same time as John, did he see her? [Yes] Are the tomatoes made in the supermarket? [No]

What is John going to do with the tomatoes? [Eat them]

Does Safeway sell deodorant? [Yes]

SR ome oe

Did John bring any money to the supermarket? [Yes]

tie

Does John have less money after going to the supermarket? [Yes]

10.23 Make the necessary additions or changes to your knowledge base from the previous exercise so that the questions that follow can be answered. Show that they can indeed be answered by the KB, and include in your report a discussion of the fixes, explaining why they were needed, whether they were minor or major, and so on.

Are there other people in Safeway while John is there? [Yes—staff!]

Is John a vegetarian? [No]

Who owns the deodorant in Safeway? [Safeway Corporation]

Did John have an ounce of ground beef? [Yes]

Does the Shell station next door have any gas? [Yes]

0 fn Toe

Do the tomatoes fit in John’s car trunk? [Yes]

10.24 Recall that inheritance information in semantic networks can be captured logically by suitable implication sentences. In this exercise, we will consider the efficiency of using such sentences for inheritance.

374

Chapter 10. Knowledge Representation

a. Consider the information content in a used-car catalog such as Kelly’s Blue Book— for example, that 1973 Dodge Vans are worth $575. Suppose all this information (for 11,000 models) is encoded as logical rules, as suggested in the chapter. Write down three such rules, including that for 1973 Dodge Vans. How would you use the rules to find the value of a particular car (e.g., JB, which is a 1973 Dodge Van), given a backward-chaining theorem prover such as Prolog?

b. Compare the time efficiency of the backward-chaining method for solving this problem with the inheritance method used in semantic nets.

c¢. Explain how forward chaining allows a logic-based system to solve the same problem efficiently, assuming that the KB contains only the 11,000 rules about prices.

d. Describe a situation in which neither forward nor backward chaining on the rules will allow the price query for an individual car to be handled efficiently.

e. Can you suggest a solution enabling this type of query to be solved efficiently in all cases in logic systems? [Hint: Remember that two cars of the same category have the same price. ]

10.25 One might suppose that the syntactic distinction between unboxed links and singly boxed links in semantic networks is unnecessary, because singly boxed links are always attached to categories; an inheritance algorithm could simply assume that an unboxed link attached to a category is intended to apply to all members of that category. Show that this argument is fallacious, giving examples of errors that would arise.

1 PLANNING

CLA; PLA

SSICAL NNING

In which we see how an agent can take advantage of the structure of a problem to construct complex plans of action.

The task of coming up with a sequence of actions that will achieve a goal is called planning. We have seen two examples of planning agents so far; the search-based problem-solving agent of Chapter 3 and the logical planning agent of Chapter 10. This chapter is concerned primarily with scaling up to complex planning problems that defeat the approaches we have seen so far.

Section 11.1 develops an expressive yet carefully constrained language for representing planning problems, including actions and states. The language is closely related to the propositional and first-order representations of actions in Chapters 7 and 10. Section 11.2 shows how forward and backward search algorithms can take advantage of this representation, primarily through accurate heuristics that can be derived automatically from the structure of the representation. (This is analogous to the way in which effective heuristics were constructed for constraint satisfaction problems in Chapter 5.) Sections 11.3 through 11.5 describe planning algorithms that go beyond forward and backward search, taking advantage of the representation of the problem. In particular, we explore approaches that are not constrained to consider only totally ordered sequences of actions.

For this chapter, we consider only environments that are fully observable, deterministic, finite, static (change happens only when the agent acts), and discrete (in time, action, objects, and effects). These are called classical planning environments. In contrast, nonclassical planning is for partially observable or stochastic environments and involves a different set of algorithms and agent designs, outlined in Chapters 12 and 17.

Let us consider what can happen when an ordinary problem-solving agent using standard search algorithms—depth-first, A\*, and so on—comes up against large, real-world problems. That will help us design better planning agents.

375

376

Chapter 11. Planning

PROBLEM DECOMPOSITION

NEARLY DECOMPOSABLE

The most obvious difficulty is that the problem-solving agent can be overwhelmed by irrelevant actions. Consider the task of buying a copy of Al: A Modern Approach from an online bookseller. Suppose there is one buying action for each 10-digit ISBN number, for a total of 10 billion actions. The search algorithm would have to examine the outcome states of all 10 billion actions to find one that satisfies the goal, which is to own a copy of ISBN 0137903952. A sensible planning agent, on the other hand, should be able to work back from an explicit goal description such as Have(ISBN0137903952) and generate the action Buy (ISBN 0137903952) directly. To do this, the agent simply needs the general knowledge that Buy(z) results in Have(z). Given this knowledge and the goal, the planner can decide in a single unification step that Buy (ISBN 0137903952) is the right action.

The next difficulty is finding a good heuristic function. Suppose the agent’s goal is to buy four different books online. Then there will be 10%° plans of just four steps, so searching without an accurate heuristic is out of the question. It is obvious to a human that a good heuristic estimate for the cost of a state is the number of books that remain to be bought; unfortunately, this insight is not obvious to a problem-solving agent, because it sees the goal test only as a black box that returns true or false for each state. Therefore, the problemsolving agent lacks autonomy; it requires a human to supply a heuristic function for each new problem. On the other hand, if a planning agent has access to an explicit representation of the goal as a conjunction of subgoals, then it can use a single domain-independent heuristic: the number of unsatisfied conjuncts. For the book-buying problem, the goal would be Have( A) A Have(B) A Have(C) A Have(D), and a state containing Have( A) A Have(C') would have cost 2. Thus, the agent automatically gets the right heuristic for this problem, and for many others. We shall see later in the chapter how to construct more sophisticated heuristics that examine the available actions as well as the structure of the goal.

Finally, the problem solver might be inefficient because it cannot take advantage of problem decomposition. Consider the problem of delivering a set of overnight packages to their respective destinations, which are scattered across Australia. It makes sense to find out the nearest airport for each destination and divide the overall problem into several subproblems, one for each airport. Within the set of packages routed through a given airport, whether further decomposition is possible depends on the destination city. We saw in Chapter 5 that the ability to do this kind of decomposition contributes to the efficiency of constraint satisfaction problem solvers. The same holds true for planners: in the worst case, it can take O(n!) time to find the best plan to deliver n packages, but only O((n/k)! x k) time if the problem can be decomposed into k equal parts.

As we noted in Chapter 5, perfectly decomposable problems are delicious but rare. The design of many planning systems—particularly the partial-order planners described in Section 11.3—is based on the assumption that most real-world problems are nearly decomposable. That is, the planner can work on subgoals independently, but might need to do some additional work to combine the resulting subplans. For some problems, this assump

1

1 Notice that even the delivery of a package is not perfectly decomposable. There may be cases in which it is better to assign packages to a more distant airport if that renders a flight to the nearest airport unnecessary. Nevertheless, most delivery companies prefer the computational and organizational simplicity of sticking with decomposed solutions.

Section 11.1.

The Planning Problem 377

GOAL SATISFACTION

ACTION SCHEMA

PRECONDITION

EFFECT

tion breaks down because working on one subgoal is likely to undo another subgoal. These interactions among subgoals are what makes puzzles (like the 8-puzzle) puzzling.

The language of planning problems

The preceding discussion suggests that the representation of planning problems—states, actions, and goals—should make it possible for planning algorithms to take advantage of the logical structure of the problem. The key is to find a language that is expressive enough to describe a wide variety of problems, but restrictive enough to allow efficient algorithms to operate over it. In this section, we first outline the basic representation language of classical planners, known as the STRIPS language.” Later, we point out some of the many possible variations in STRIPS-like languages.

Representation of states. Planners decompose the world into logical conditions and represent a state as a conjunction of positive literals. We will consider propositional literals; for example, Poor A Unknown might represent the state of a hapless agent. We will also use first-order literals; for example, At(Planeq, Melbourne) A At( Planes, Sydney) might represent a state in the package delivery problem. Literals in first-order state descriptions must be ground and function-free. Literals such as At(x, y) or At(Father(Fred), Sydney) are not allowed. The closed-world assumption is used, meaning that any conditions that are not mentioned in a state are assumed false.

Representation of goals. A goal is a partially specified state, represented as a conjunction of positive ground literals, such as Rich A Famous or At( Ps, Tahiti). A propositional state s satisfies a goal g if s contains all the atoms in g (and possibly others). For example, the state Rich A Famous A Miserable satisfies the goal Rich A Famous.

Representation of actions. An action is specified in terms of the preconditions that must hold before it can be executed and the effects that ensue when it is executed. For example, an action for flying a plane from one location to another is:

Action( Fly(p, from, to), PRECOND: At (p, from) A Plane(p) A Airport(from) A Airport(to) EFFECT: = At (p, from) A At(p, to))

This is more properly called an action schema, meaning that it represents a number of different actions that can be derived by instantiating the variables p, from, and fo to different constants. In general, an action schema consists of three parts:

o The action name and parameter list—for example, Fly(p, from, to)—serves to identify the action.

e The precondition is a conjunction of function-free positive literals stating what must be true in a state before the action can be executed. Any variables in the precondition must also appear in the action’s parameter list.

o The effect is a conjunction of function-free literals describing how the state changes when the action is executed. A positive literal P in the effect is asserted to be true in

2 STRIPS stands for STanford Research Institute Problem Solver.

378

Chapter 11. Planning

ADD LIST DELETE LIST

APPLICABLE

RESULT

STRIPS ASSUMPTION

SOLUTION

the state resulting from the action, whereas a negative literal — P is asserted to be false. Variables in the effect must also appear in the action’s parameter list.

To improve readability, some planning systems divide the effect into the add list for positive literals and the delete list for negative literals.

Having defined the syntax for representations of planning problems, we can now define the semantics. The most straightforward way to do this is to describe how actions affect states. (An alternative method is to specify a direct translation into successor-state axioms, whose semantics comes from first-order logic; see Exercise 11.3.) First, we say that an action is applicable in any state that satisfies the precondition; otherwise, the action has no effect. For a first-order action schema, establishing applicability will involve a substitution 8 for the variables in the precondition. For example, suppose the current state is described by

At(Py, JFK) A At(Ps, SFO) A Plane(P;) A Plane(P,) A Airport(JFK) A Airport(SFO) .

This state satisfies the precondition At(p, from) A Plane(p) A Atrport(from) A Airport (to)

with substitution {p/ Py, from/JFK , to/SFO} (among others—see Exercise 11.2). Thus, the concrete action Fly(Py, JFK, SFO) is applicable.

Starting in state s, the result of executing an applicable action a is a state s’ that is the same as s except that any positive literal P in the effect of a is added to s’ and any negative literal =P is removed from s'. Thus, after Fly (Py, JFK, SFO), the current state becomes

At(Py, SFO) A At(P2, SFO) A Plane(Pr) A Plane(Py)

A Airport(JFK) A Airport(SFO) . Note that if a positive effect is already in s it is not added twice, and if a negative effect is not in s, then that part of the effect is ignored. This definition embodies the so-called STRIPS assumption: that every literal not mentioned in the effect remains unchanged. In this way, STRIPS avoids the representational frame problem described in Chapter 10.

Finally, we can define the solution for a planning problem. In its simplest form, this is just an action sequence that, when executed in the initial state, results in a state that satisfies the goal. Later in the chapter, we will allow solutions to be partially ordered sets of actions, provided that every action sequence that respects the partial order is a solution.

Expressiveness and extensions

The various restrictions imposed by the STRIPS representation were chosen in the hope of making planning algorithms simpler and more efficient, without making it too difficult to describe real problems. One of the most important restrictions is that literals be functionfree. With this restriction, we can be sure that any action schema for a given problem can be propositionalized—that is, turned into a finite collection of purely propositional action representations with no variables. (See Chapter 9 for more on this topic.) For example, in the air cargo domain for a problem with 10 planes and five airports, we could translate the Fly(p, from, to) schema into 10 x 5 x 5 = 250 purely propositional actions. The planners

Section 11.1.

The Planning Problem 379

[]

ADL

in Sections 11.4 and 11.5 work directly with propositionalized descriptions. If we allow function symbols, then infinitely many states and actions can be constructed.

In recent years, it has become clear that STRIPS is insufficiently expressive for some real domains. As a result, many language variants have been developed. Figure 11.1 briefly describes one important one, the Action Description Language or ADL, by comparing it with the basic STRIPS language. In ADL, the Fly action could be written as

Action(Fly(p : Plane, from : Airport, to : Airport), PRECOND: At(p, from) A (from # to) EFFECT: —At(p, from) A At(p, to)) .

The notation p : Plane in the parameter list is an abbreviation for Plane(p) in the precondition; this adds no expressive power, but can be easier to read. (It also cuts down on the number of possible propositional actions that can be constructed.) The precondition (from # to) expresses the fact that a flight cannot be made from an airport to itself. This could not be expressed succinctly in STRIPS.

The various planning formalisms used in Al have been systematized within a standard syntax called the Planning Domain Definition Language, or PDDL. This language allows researchers to exchange benchmark problems and compare results. PDDL includes sublanguages for STRIPS, ADL, and the hierarchical task networks we will see in Chapter 12.

380

Chapter 11. Planning

STATE CONSTRAINTS

Init(At(Cy, SFO) A At(Cay, JFK) A At(Py, SFO) A At(P,, JFK) A Cargo(Cy) A Cargo(C2) A Plane(Py) A Plane(Py) A Airport(JFK) A Airport(SFO)) Goal(At(Cy, JFK) A At(C2, SFO)) Action(Load(c, p, a), PRECOND: At(c, a) A At(p, a) A Cargo(c) A Plane(p) A Airport(a) EFFECT: — At(c, a) A In(c, p)) Action(Unload(c, p, a), PRECOND: In(c, p) A At(p, a) A Cargo(c) A Plane(p) A Airport(a) EFFECT: At(c, a) A = In(c, p)) Action (Fly(p, from, to), PRECOND: At(p, from) A Plane(p) AN Airport(from) A Airport(to) EFFECT: — At(p, from) A At(p, to))

Figure 11.2 A STRIPS problem involving transportation of air cargo between airports.

The STRIPS and ADL notations are adequate for many real domains. The subsections that follow show some simple examples. There are still some significant restrictions, however. The most obvious is that they cannot represent in a natural way the ramifications of actions. For example, if there are people, packages, or dust motes in the airplane, then they too change location when the plane flies. We can represent these changes as the direct effects of flying, whereas it seems more natural to represent the location of the plane’s contents as a logical consequence of the location of the plane. We will see more examples of such state constraints in Section 11.5. Classical planning systems do not even attempt to address the qualification problem: the problem of unrepresented circumstances that could cause an action to fail. We will see how to address qualifications in Chapter 12.

Example: Air cargo transport

Figure 11.2 shows an air cargo transport problem involving loading and unloading cargo onto and off of planes and flying it from place to place. The problem can be defined with three actions: Load, Unload, and Fly. The actions affect two predicates: In(c, p) means that cargo c is inside plane p, and At(x, a) means that object x (either plane or cargo) is at airport a. Note that cargo is not A¢{ anywhere when it is In a plane, so At really means “available for use at a given location.” It takes some experience with action definitions to handle such details consistently. The following plan is a solution to the problem:

[Load (Cy, Py, SFO), Fly(P1, SFO, JFK), Unload(C1, Py, JFK), Load(C>, Ps, JFK), Fly( Pa, JFK, SFO), Unload (Cs, Py, SFO)] .

Our representation is pure STRIPS. In particular, it allows a plane to fly to and from the same airport. Inequality literals in ADL could prevent this.

Section 11.1.

The Planning Problem 381

BLOCKS WORLD

Example: The spare tire problem

Consider the problem of changing a flat tire. More precisely, the goal is to have a good spare tire properly mounted onto the car’s axle, where the initial state has a flat tire on the axle and a good spare tire in the trunk. To keep it simple, our version of the problem is a very abstract one, with no sticky lug nuts or other complications. There are just four actions: removing the spare from the trunk, removing the flat tire from the axle, putting the spare on the axle, and leaving the car unattended overnight. We assume that the car is in a particularly bad neighborhood, so that the effect of leaving it overnight is that the tires disappear.

The ADL description of the problem is shown in Figure 11.3. Notice that it is purely propositional. It goes beyond STRIPS in that it uses a negated precondition, —At( Flat, Axle), for the PutOn( Spare, Axle) action. This could be avoided by using Clear( Azle) instead, as we will see in the next example.

Init (At(Flat, Azle) A At(Spare, Trunk)) Goal( At(Spare, Azle)) Action( Remove (Spare, Trunk),

PRECOND: At(Spare, Trunk)

EFFECT: — At(Spare, Trunk) A At(Spare, Ground)) Action(Remove( Flat, Azle),

PRECOND: At(Flat, Azle)

EFFECT: = At(Flat, Azle) AN At(Flat, Ground)) Action PutOn (Spare, Axle),

PRECOND: At(Spare, Ground) A — At(Flat, Azle)

EFFECT: — At(Spare, Ground) AN At(Spare, Azle)) Action(LeaveQOuvernight,

PRECOND:

EFFECT: — At(Spare, Ground) A — At(Spare, Axle) N — At(Spare, Trunk)

A = At(Flat, Ground) A — At(Flat, Azle))

Figure 11.3 The simple spare tire problem.

Example: The blocks world

One of the most famous planning domains is known as the blocks world. This domain consists of a set of cube-shaped blocks sitting on a table.® The blocks can be stacked, but only one block can fit directly on top of another. A robot arm can pick up a block and move it to another position, either on the table or on top of another block. The arm can pick up only one block at a time, so it cannot pick up a block that has another one on it. The goal will always be to build one or more stacks of blocks, specified in terms of what blocks are on top of what other blocks. For example, a goal might be to get block A on B and block C' on D.

3 The blocks world used in planning research is much simpler than SHRDLU’s version, shown on page 20.

382

Chapter 11. Planning

We will use On(b, z) to indicate that block b is on x, where x is either another block or the table. The action for moving block & from the top of x to the top of y will be Move(b, x,y). Now, one of the preconditions on moving & is that no other block be on it. In first-order logic, this would be =3 = On(z, b) or, alternatively, V x —=On(x, b). These could be stated as preconditions in ADL. We can stay within the STRIPS language, however, by introducing a new predicate, Clear (x), that is true when nothing is on z.

The action Move moves a block b from x to y if both b and y are clear. After the move is made, « is clear but y is not. A formal description of Move in STRIPS is

Action(Move(b, x,y), PRECOND:On(b, xz) A Clear(b) A Clear(y), EFFECT: On(b, y) A Clear(z) A —=On(b,z) A =Clear(y)) . Unfortunately, this action does not maintain Clear properly when x or y is the table. When x = Table, this action has the effect Clear (Table), but the table should not become clear, and when y = Table, it has the precondition Clear( Table), but the table does not have to be clear to move a block onto it. To fix this, we do two things. First, we introduce another action to move a block b from z to the table: Action(MoveToTable(b, x), PRECOND:On(b, x) A Clear(b)), EFFECT: On(b, Table) A Clear(xz) A =On(b,x)) . Second, we take the interpretation of Clear (b) to be “there is a clear space on b to hold a block.” Under this interpretation, Clear( Table) will always be true. The only problem is that nothing prevents the planner from using Move (b, z:, Table) instead of MoveToTable(b, x). We could live with this problem—it will lead to a larger-than-necessary search space, but will not lead to incorrect answers—or we could introduce the predicate Block and add Block (b) A Block(y) to the precondition of Move.

Finally, there is the problem of spurious actions such as Move (B,C, C'), which should be a no-op, but which has contradictory effects. It is common to ignore such problems, because they seldom cause incorrect plans to be produced. The correct approach is add inequality preconditions as shown in Figure 11.4.

PROGRESSION

Now we turn our attention to planning algorithms. The most straightforward approach is to use state-space search. Because the descriptions of actions in a planning problem specify both preconditions and effects, it is possible to search in either direction: either forward from the initial state or backward from the goal, as shown in Figure 11.5. We can also use the explicit action and goal representations to derive effective heuristics automatically.

Forward state-space search

Planning with forward state-space search is similar to the problem-solving approach of Chapter 3. It is sometimes called progression planning, because it moves in the forward direction.

Section 11.2. Planning with State-Space Search 383

Init(On(A, Table) A On(B, Table) A On(C, Table) A Block(A) A Block(B) A Block(C) A Clear(A) A Clear(B) A Clear(C)) Goal(On(A,B) A On(B, C)) Action(Move(b, z, y), PRECOND: On(bz) A Clear(b) A Clear(y) A Block(b) A (b#x)Nb#y) AN (@#yY), EFFECT: On(b,y) A Clear(z) A = On(b,z) A = Clear(y)) Action(MoveToTable(b, x), PRECOND: On(b,z) A Clear(b) A Block(b) A (b # z), EFFECT: On(b, Table) A Clear(z) A — On(b,x))

Figure 11.4 A planning problem in the blocks world: building a three-block tower. One solution is the sequence [Move (B, Table, C), Move(A, Table, B)].

(a) Fly(P,, A, B) (b)

Fiy(P, A, B)

Figure 11.5 Two approaches to searching for a plan. (a) Forward (progression) state-space search, starting in the initial state and using the problem’s actions to search forward for the goal state. (b) Backward (regression) state-space search: a belief-state search (see page 84) starting at the goal state(s) and using the inverse of the actions to search backward for the initial state.

We start in the problem’s initial state, considering sequences of actions until we find a sequence that reaches a goal state. The formulation of planning problems as state-space search problems is as follows:

o The initial state of the search is the initial state from the planning problem. In general, each state will be a set of positive ground literals; literals not appearing are false.

384

Chapter 11. Planning

RELEVANCE

o The actions that are applicable to a state are all those whose preconditions are satisfied. The successor state resulting from an action is generated by adding the positive effect literals and deleting the negative effect literals. (In the first-order case, we must apply the unifier from the preconditions to the effect literals.) Note that a single successor function works for all planning problems—a consequence of using an explicit action representation.

o The goal test checks whether the state satisfies the goal of the planning problem.

o The step cost of each action is typically 1. Although it would be easy to allow different costs for different actions, this is seldom done by STRIPS planners.

Recall that, in the absence of function symbols, the state space of a planning problem is finite. Therefore, any graph search algorithm that is complete—for example, A\*—will be a complete planning algorithm,

From the earliest days of planning research (around 1961) until recently (around 1998) it was assumed that forward state-space search was too inefficient to be practical. It is not hard to come up with reasons why-—just refer back to the start of Section 11.1. First, forward search does not address the irrelevant action problem—all applicable actions are considered from each state. Second, the approach quickly bogs down without a good heuristic. Consider an air cargo problem with 10 airports, where each airport has 5 planes and 20 pieces of cargo. The goal is to move all the cargo at airport A to airport B. There is a simple solution to the problem: load the 20 pieces of cargo into one of the planes at A, fly the plane to B, and unload the cargo. But finding the solution can be difficult because the average branching factor is huge: each of the 50 planes can fly to 9 other airports, and each of the 200 packages can be either unloaded (if it is loaded), or loaded into any plane at its airport (if it is unloaded). On average, let’s say there are about 1000 possible actions, so the search tree up to the depth of the obvious solution has about 1000\*! nodes. It is clear that a very accurate heuristic will be needed to make this kind of search efficient. We will discuss some possible heuristics after looking at backward search.

Backward state-space search

Backward state-space search was described briefly as part of bidirectional search in Chapter 3. We noted there that backward search can be difficult to implement when the goal states are described by a set of constraints rather than being listed explicitly. In particular, it is not always obvious how to generate a description of the possible predecessors of the set of goal states. We will see that the STRIPS representation makes this quite easy because sets of states can be described by the literals that must be true in those states.

The main advantage of backward search is that it allows us to consider only relevant actions. An action is relevant to a conjunctive goal if it achieves one of the conjuncts of the goal. For example, the goal in our 10-airport air cargo problem is to have 20 pieces of cargo at airport B, or more precisely,

At(Ch, B) A At(Cy, B) AN. A At (Cy, B) . Now consider the conjunct At(C1, B). Working backwards, we can seek actions that have this as an effect. There is only one: Unload(C1,p, B), where plane p is unspecified.

Section 11.2.

Planning with State-Space Search 385

REGRESSION

CONSISTENCY

Notice that there are many irrelevant actions that can also lead to a goal state. For example, we can fly an empty plane from JFK to SFO; this action reaches a goal state from a predecessor state in which the plane is at JFK and all the goal conjuncts are satisfied. A backward search that allows irrelevant actions will still be complete, but it will be much less efficient. If a solution exists, it will be found by a backward search that allows only relevant actions. The restriction to relevant actions means that backward search often has a much lower branching factor than forward search. For example, our air cargo problem has about 1000 actions leading forward from the initial state, but only 20 actions working backward from the goal.

Searching backwards is sometimes called regression planning. The principal question in regression planning is this: what are the states from which applying a given action leads to the goal? Computing the description of these states is called regressing the goal through the action. To see how to do it, consider the air cargo example. We have the goal

At(Ch, B) A At(Cy, BY A... A At(Cy, B)

and the relevant action Unload (Cy, p, B), which achieves the first conjunct. The action will work only if its preconditions are satisfied. Therefore, any predecessor state must include these preconditions: In(Cy,p) A At(p, B). Moreover, the subgoal A¢(C}, B) should not be true in the predecessor state.\* Thus, the predecessor description is

In(Ch,p) A At(p, B) A At(Co, BYAN...A At(Cy, B) .

In addition to insisting that actions achieve some desired literal, we must insist that the actions not undo any desired literals. An action that satisfies this restriction is called consistent. For example, the action Load (C2, p) would not be consistent with the current goal, because it would negate the literal At(Cy, B).

Given definitions of relevance and consistency, we can describe the general process of constructing predecessors for backward search. Given a goal description G, let A be an action that is relevant and consistent. The corresponding predecessor is as follows:

e Any positive effects of A that appear in G are deleted.

e Each precondition literal of A is added, unless it already appears. Any of the standard search algorithms can be used to carry out the search. Termination occurs when a predecessor description is generated that is satisfied by the initial state of the planning problem. In the first-order case, satisfaction might require a substitution for variables in the

predecessor description. For example, the predecessor description in the preceding paragraph is satisfied by the initial state

In(C1, P12) A At( Pra, B) A At(Co, BYA...A At(Ca, B)

with substitution {p/ P12}. The substitution must be applied to the actions leading from the state to the goal, producing the solution [Unload (C1, P12, B)).

4 If the subgoal were true in the predecessor state, the action would still lead to a goal state. On the other hand, such actions are irrelevant because they do not make the goal true.

386

Chapter 11. Planning

SUBGOAL INDEPENDENCE

Heuristics for state-space search

It turns out that neither forward nor backward search is efficient without a good heuristic function. Recall from Chapter 4 that a heuristic function estimates the distance from a state to the goal; in STRIPS planning, the cost of each action is 1, so the distance is the number of actions. The basic idea is to look at the effects of the actions and at the goals that must be achieved and to guess how many actions are needed to achieve all the goals. Finding the exact number is NP hard, but it is possible to find reasonable estimates most of the time without too much computation. We might also be able to derive an admissible heuristic—one that does not overestimate. This could be used with A\* search to find optimal solutions.

There are two approaches that can be tried. The first is to derive a relaxed problem from the given problem specification, as described in Chapter 4. The optimal solution cost for the relaxed problem—which we hope is very easy to solve—gives an admissible heuristic for the original problem. The second approach is to pretend that a pure divide-and-conquer algorithm will work. This is called the subgoal independence assumption: the cost of solving a conjunction of subgoals is approximated by the sum of the costs of solving each subgoal independently. The subgoal independence assumption can be optimistic or pessimistic. It is optimistic when there are negative interactions between the subplans for each subgoal— for example, when an action in one subplan deletes a goal achieved by another subplan. It is pessimistic, and therefore inadmissible, when subplans contain redundant actions—for instance, two actions that could be replaced by a single action in the merged plan.

Let us consider how to derive relaxed planning problems. Since explicit representations of preconditions and effects are available, the process will work by modifying those representations. (Compare this approach with search problems, where the successor function is a black box.) The simplest idea is to relax the problem by removing all preconditions from the actions. Then every action will always be applicable, and any literal can be achieved in one step (if there is an applicable action—if not, the goal is impossible). This almost implies that the number of steps required to solve a conjunction of goals is the number of unsatisfied goals—almost but not quite, because (1) there may be two actions, each of which deletes the goal literal achieved by the other, and (2) some action may achieve multiple goals. If we combine our relaxed problem with the subgoal independence assumption, both of these issues are assumed away and the resulting heuristic is exactly the number of unsatisfied goals.

In many cases, a more accurate heuristic is obtained by considering at least the positive interactions arising from actions that achieve multiple goals. First, we relax the problem further by removing negative effects (see Exercise 11.6). Then, we count the minimum number of actions required such that the union of those actions’ positive effects satisfies the goal. For example, consider

Goal(AAN BAC)

Action(X, EFFECT:A A P) Action(Y,EFFECT:BAC AQ) Action(Z,EFFECT:BA P AQ) .

The minimal set cover of the goal { A, B, C'} is given by the actions {X,Y}, so the set cover heuristic returns a cost of 2. This improves on the subgoal independence assumption, which

Section 11.3.

Partial-Order Planning 387

EMPTY-DELETE-LIST

gives a heuristic value of 3. There is one minor irritation: the set cover problem is NPhard. A simple greedy set-covering algorithm is guaranteed to return a value that is within a factor of log n of the true minimum value, where 7 is the number of literals in the goal, and usually works much better than this in practice. Unfortunately, the greedy algorithm loses the guarantee of admissibility for the heuristic.

It is also possible to generate relaxed problems by removing negative effects without removing preconditions. That is, if an action has the effect A A =B in the original problem, it will have the effect A in the relaxed problem. This means that we need not worry about negative interactions between subplans, because no action can delete the literals achieved by another action. The solution cost of the resulting relaxed problem gives what is called the empty-delete-list heuristic. The heuristic is quite accurate, but computing it involves actually running a (simple) planning algorithm. In practice, the search in the relaxed problem is often fast enough that the cost is worthwhile.

The heuristics described here can be used in either the progression or the regression direction. At the time of writing, progression planners using the empty-delete-list heuristic hold the lead. That is likely to change as new heuristics and new search techniques are explored. Since planning is exponentially hard,’ no algorithm will be efficient for all problems, but many practical problems can be solved with the heuristic methods in this chapter—far more than could be solved just a few years ago.

LEAST COMMITMENT

Forward and backward state-space search are particular forms of rotally ordered plan search. They explore only strictly linear sequences of actions directly connected to the start or goal. This means that they cannot take advantage of problem decomposition. Rather than work on each subproblem separately, they must always make decisions about how to sequence actions from all the subproblems. We would prefer an approach that works on several subgoals independently, solves them with several subplans, and then combines the subplans.

Such an approach also has the advantage of flexibility in the order in which it constructs the plan. That is, the planner can work on “obvious” or “important” decisions first, rather than being forced to work on steps in chronological order. For example, a planning agent that is in Berkeley and wishes to be in Monte Carlo might first try to find a flight from San Francisco to Paris; given information about the departure and arrival times, it can then work on ways to get to and from the airports.

The general strategy of delaying a choice during search is called a least commitment strategy. There is no formal definition of least commitment, and clearly some degree of commitment is necessary, lest the search would make no progress. Despite the informality, least commitment is a useful concept for analyzing when decisions should be made in any search problem.

5 Technically, STRIPS-style planning is PSPACE-complete unless actions have only positive preconditions and only one effect literal (Bylander, 1994).

388

Chapter 11. Planning

PARTIAL-ORDER PLANNER

LINEARIZATION

Our first concrete example will be much simpler than planning a vacation. Consider the simple problem of putting on a pair of shoes. We can describe this as a formal planning problem as follows:

Goal (RightShoeOn A LeftShoeOn)

Init()

Action(RightShoe, PRECOND: RightSockOn, EFFECT: RightShoeOn) Action (RightSock, EFFECT: RightSockOn)

Action (LeftShoe, PRECOND: LeftSockOn, EFFECT: Left Shoe On) Action (LeftSock, EFFECT: LeftSockOn) .

A planner should be able to come up with the two-action sequence RightSock followed by RightShoe to achieve the first conjunct of the goal and the sequence LeftSock followed by LeftShoe for the second conjunct. Then the two sequences can be combined to yield the final plan. In doing this, the planner will be manipulating the two subsequences independently, without committing to whether an action in one sequence is before or after an action in the other. Any planning algorithm that can place two actions into a plan without specifying which comes first is called a partial-order planner. Figure 11.6 shows the partial-order plan that is the solution to the shoes and socks problem. Note that the solution is represented as a graph of actions, not a sequence. Note also the “dummy” actions called Start and Finish, which mark the beginning and end of the plan. Calling them actions symplifies things, because now every step of a plan is an action. The partial-order solution corresponds to six possible total-order plans; each of these is called a linearization of the partial-order plan.

Partial-order planning can be implemented as a search in the space of partial-order plans. (From now on, we will just call them “plans.”) That is, we start with an empty plan. Then we consider ways of refining the plan until we come up with a complete plan that solves the problem. The actions in this search are not actions in the world, but actions on plans: adding a step to the plan, imposing an ordering that puts one action before another, and so on.

We will define the POP algorithm for partial-order planning. It is traditional to write out the POP algorithm as a stand-alone program, but we will instead formulate partial-order planning as an instance of a search problem. This allows us to focus on the plan refinement steps that can be applied, rather than worrying about how the algorithm explores the space. In fact, a wide variety of uninformed or heuristic search methods can be applied once the search problem is formulated.

Remember that the states of our search problem will be (mostly unfinished) plans. To avoid confusion with the states of the world, we will talk about plans rather than states. Each plan has the following four components, where the first two define the steps of the plan and the last two serve a bookkeeping function to determine how plans can be extended:

e A set of actions that make up the steps of the plan. These are taken from the set of actions in the planning problem. The “empty” plan contains just the Start and Finish actions. Start has no preconditions and has as its effect all the literals in the initial state of the planning problem. Finish has no effects and has as its preconditions the goal literals of the planning problem.

389

ORDERING CONSTRAINTS

CAUSAL LINKS ACHIEVES

CONFLICTS

OPEN PRECONDITIONS

Partial-Order Plan: Total-Order Plans: Start Start Start Start Start Start Start J \ Right Right Left Left Right Left Sock Sock Sock Sock Sock Sock Left Right Sock Sock 1 1 1 Y Y Left Left Right Right Right Left Sock Sock Sock Sock Shoe Shoe LeftSockOn RightSockOn 1 1 1 1 y : Right Left Right Left Left Right ft Right or oro = v = Shoe or Left Right Left Right Left Right Shoe Shoe Shoe Shoe Shoe Shoe LeftShoeOn, RightShoeOn { { 7 yo Finish Finish Finish Finish Finish Finish Finish Figure 11.6 A partial-order plan for putting on shoes and socks, and the six corresponding linearizations into total-order plans.

read as “A before B” and means that action A must be executed sometime before action B, but not necessarily immediately before. The ordering constraints must describe a proper partial order. Any cycle—such as A < B and B < A—represents a contradiction, so an ordering constraint cannot be added to the plan if it creates a cycle.

as A \_P, Band is read as “A achieves p for B.” For example, the causal link RightSock ®9P45okOn pian i Shoe

asserts that RightSockOn is an effect of the RightSock action and a precondition of RightShoe. 1t also asserts that RightSockOn must remain true from the time of action RightSock to the time of action RighiShoe. In other words, the plan may not be extended by adding a new action C that conflicts with the causal link. An action C conflicts with A \_P, B if C has the effect —p and if C could (according to the ordering constraints) come after A and before B. Some authors call causal links protection intervals, because the link A \_?, B protects p from being negated over the interval from Ato B.

in the plan. Planners will work to reduce the set of open preconditions to the empty set, without introducing a contradiction.

390

Chapter 11. Planning

CONSISTENT PLAN

For example, the final plan in Figure 11.6 has the following components (not shown are the ordering constraints that put every other action after Start and before Finish):

Actions: { RightSock, RightShoe, LeftSock, LeftShoe, Start, Finish}

Orderings:{ RightSock < RightShoe, LeftSock < LeftShoe}

Links: { RightSock RightSockOn: piohtShoe, LeftSock Ft50%kO™ LeftShoe, RightShoe Fi9ht5hoeOn pinish LeftShoe LeftShoeOn Binish}

Open Preconditions:{ } .

We define a consistent plan as a plan in which there are no cycles in the ordering constraints and no conflicts with the causal links. A consistent plan with no open preconditions is a solution. A moment’s thought should convince the reader of the following fact: every linearization of a partial-order solution is a total-order solution whose execution from the initial state will reach a goal state. This means that we can extend the notion of “executing a plan” from total-order to partial-order plans. A partial-order plan is executed by repeatedly choosing any of the possible next actions. We will see in Chapter 12 that the flexibility available to the agent as it executes the plan can be very useful when the world fails to cooperate. The flexible ordering also makes it easier to combine smaller plans into larger ones, because each of the small plans can reorder its actions to avoid conflict with the other plans.

Now we are ready to formulate the search problem that POP solves. We will begin with a formulation suitable for propositional planning problems, leaving the first-order complications for later. As usual, the definition includes the initial state, actions, and goal test.

e The initial plan contains Start and Finish, the ordering constraint Start < Finish, and no causal links and has all the preconditions in Finish as open preconditions.

e The successor function arbitrarily picks one open precondition p on an action B and generates a successor plan for every possible consistent way of choosing an action A that achieves p. Consistency is enforced as follows:

1. The causal link A \_\*, B and the ordering constraint A < B are added to the plan. Action A may be an existing action in the plan or a new one. If it is new, add it to the plan and also add Start < A and A < Finish.

2. We resolve conflicts between the new causal link and all existing actions and between the action A (if it is new) and all existing causal links. A conflict between A \_\*, Band C is resolved by making C occur at some time outside the protection interval, either by adding B < C or C < A. We add successor states for either or both if they result in consistent plans.

e The goal test checks whether a plan is a solution to the original planning problem. Because only consistent plans are generated, the goal test just needs to check that there are no open preconditions.

Remember that the actions considered by the search algorithms under this formulation are plan refinement steps rather than the real actions from the domain itself. The path cost is therefore irrelevant, strictly speaking, because the only thing that matters is the total cost of the real actions in the plan to which the path leads. Nonetheless, it is possible to specify a path cost function that reflects the real plan costs: we charge 1 for each real action added to

Section 11.3.

Partial-Order Planning 391

the plan and O for all other refinement steps. In this way, g(n), where n is a plan, will be equal to the number of real actions in the plan. A heuristic estimate h(n) can also be used.

At first glance, one might think that the successor function should include successors for every open p, not just for one of them. This would be redundant and inefficient, however, for the same reason that constraint satisfaction algorithms don’t include successors for every possible variable: the order in which we consider open preconditions (like the order in which we consider CSP variables) is commutative. (See page 141.) Thus, we can choose an arbitrary ordering and still have a complete algorithm. Choosing the right ordering can lead to a faster search, but all orderings end up with the same set of candidate solutions.

A partial-order planning example

Now let’s look at how POP solves the spare tire problem from Section 11.1. The problem description is repeated in Figure 11.7.

Init(At(Flat, Azle) A At(Spare, Trunk)) Goal(At(Spare, Azle)) Action(Remove (Spare, Trunk),

PRECOND: At(Spare, Trunk)

EFFECT: — At(Spare, Trunk) A At(Spare, Ground)) Action (Remove (Flat, Azle),

PRECOND: At(Flat, Azle)

EFFECT: = At(Flat, Azle) A At(Flat, Ground)) Action(PutOn(Spare, Axle),

PRECOND: At(Spare, Ground) A — At(Flat, Azle)

EFFECT: = At(Spare, Ground) A At(Spare, Azle)) Action (Leave Overnight,

PRECOND:

EFFECT: = At(Spare, Ground) A — At(Spare, Azle) A — At(Spare, Trunk)

A = At( Flat, Ground) A — At(Flat, Azle))

Figure 11.7 The simple flat tire problem description.

The search for a solution begins with the initial plan, containing a Start action with the effect At(Spare, Trunk) A At(Flat, Azle) and a Finish action with the sole precondition At(Spare, Azle). Then we generate successors by picking an open precondition to work on (irrevocably) and choosing among the possible actions to achieve it. For now, we will not worry about a heuristic function to help with these decisions; we will make seemingly arbitrary choices. The sequence of events is as follows:

1. Pick the only open precondition, At(Spare, Azle) of Finish. Choose the only applicable action, PutOn(Spare, Azle). 2. Pick the At(Spare, Ground) precondition of PutOn (Spare, Azle). Choose the only

applicable action, Remove (Spare, Trunk) to achieve it. The resulting plan is shown in Figure 11.8.

392

Chapter 11. Planning

At(Spare, Trunk)] Remove(Spare, Trunk) Al Spare, Trunk) At(Spare, Ground) Start PutOn(Spare,Axle At(Spare,Axie)| Finish | | Start | ce axe —Al( Flat, Axle) (Sp ) [ Finisn\_}

Figure 11.8 The incomplete partial-order plan for the tire problem, after choosing actions for the first two open preconditions. Boxes represent actions, with preconditions on the left and effects on the right. (Effects are omitted, except for that of the Start action.) Dark arrows represent causal links protecting the proposition at the head of the arrow.

3. Pick the —At(Flat, Azle) precondition of PutOn (Spare, Azle). Just to be contrary, choose the LeaveQuvernight action rather than the Remove (Flat, Azle) action. Notice that Leave Overnight also has the effect ~ At (Spare, Ground), which means it conflicts with the causal link

Remove (Spare, Trunk) At(Spare, Ground) PutOn(Spare, Azle) .

To resolve the conflict we add an ordering constraint putting LeaveQvernight before Remove (Spare, Trunk). The resulting plan is shown in Figure 11.9. (Why does this resolve the conflict, and why is there no other way to resolve it?)

At(Spare, Trunk)] Remove(Spare, Trunk)

[]

Figure 11.9 The plan after choosing LeaveQuernight as the action for achieving —At(Flat, Azle). To avoid a conflict with the causal link from Remove(Spare, Trunk) that protects At(Spare, Ground), LeaveOvernight is constrained to occur before Remove (Spare, Trunk), as shown by the dashed arrow.

4. The only remaining open precondition at this point is the A¢( Spare, Trunk) precondition of the action Remove (Spare, Trunk). The only action that can achieve it is the existing Start action, but the causal link from Start to Remove (Spare, Trunk) is in conflict with the — At (Spare, Trunk) effect of LeaveOvernight. This time there is no way to resolve the conflict with Leave Overnight: we cannot order it before Start (because nothing can come before Start), and we cannot order it after Remove(Spare, Trunk) (because there is already a constraint ordering it before Remove (Spare, Trunk)). So we are forced to back up, remove the LeaveOQuernight action and the last two causal links, and return to the state in Figure 11.8. In essence, the planner has proved that LeaveQOvernight doesn’t work as a way to change a tire.

Section 11.3.

Partial-Order Planning 393

5. Consider again the — At (Flat, Azle) precondition of PutOn (Spare, Azle). This time, we choose Remove(Flat, Azle).

6. Once again, pick the A¢(Spare, Trunk) precondition of Remove (Spare, Trunk) and choose Start to achieve it. This time there are no conflicts.

7. Pick the At(Flat, Azle) precondition of Remove(Flat, Azle), and choose Start to achieve it. This gives us a complete, consistent plan—in other words a solution—as shown in Figure 11.10.

[]

Figure 11.10 The final solution to the tire problem. Note that Remove (Spare, Trunk) and Remove(Flat, Azle) can be done in either order, as long as they are completed before the PutOn(Spare, Azle) action.

Although this example is very simple, it illustrates some of the strengths of partial-order planning. First, the causal links lead to early pruning of portions of the search space that, because of irresolvable conflicts, contain no solutions. Second, the solution in Figure 11.10 is a partial-order plan. In this case the advantage is small, because there are only two possible linearizations; nonetheless, an agent might welcome the flexibility—for example, if the tire has to be changed in the middle of heavy traffic.

The example also points to some possible improvements that could be made. For example, there is duplication of effort: Start is linked to Remove(Spare, Trunk) before the conflict causes a backtrack and is then unlinked by backtracking even though it is not involved in the conflict. It is then relinked as the search continues. This is typical of chronological backtracking and might be mitigated by dependency-directed backtracking.

Partial-order planning with unbound variables

In this section, we consider the complications that can arise when POP is used with firstorder action representations that include variables. Suppose we have a blocks world problem (Figure 11.4) with the open precondition On(A, B) and the action

Action(Move(b, x,y), PRECOND:On(b, z) A Clear(b) A Clear(y), EFFECT: On(b,y) A Clear(z) A =On(b,z) A =Clear(y)) .

394 Chapter 11. Planning

This action achieves On(A, B) because the effect On(b, y) unifies with On (A, B) with the substitution {b/A, y/B}. We then apply this substitution to the action, yielding

Action(Move(A, z, B), PRECOND:On(A, z) A Clear(A) A Clear(B), EFFECT: On (A, B) A Clear(z) A ~On(A,z) A =Clear(B)) .

This leaves the variable z unbound. That is, the action says to move block A from somewhere, without yet saying whence. This is another example of the least commitment principle: we can delay making the choice until some other step in the plan makes it for us. For example, suppose we have On (A, D) in the initial state. Then the Start action can be used to achieve On(A, z), binding z to D. This strategy of waiting for more information before choosing z is often more efficient than trying every possible value of z and backtracking for each one that fails.

The presence of variables in preconditions and actions complicates the process of detecting and resolving conflicts. For example, when Move(A, z, B) is added to the plan, we will need a causal link

Move(A, z, B) OnAB) Binish |

If there is another action M> with effect ~On(A, z), then My; conflicts only if z is B. To accommodate this possibility, we extend the representation of plans to include a set of inequalTs ity constraints of the form z # X where z is a variable and X is either another variable or a constant symbol. In this case, we would resolve the conflict by adding z # B, which means that future extensions to the plan can instantiate z to any value except B. Anytime we apply a substitution to a plan, we must check that the inequalities do not contradict the substitution. For example, a substitution that includes x /y conflicts with the inequality constraint z # y. Such conflicts cannot be resolved, so the planner must backtrack. A more extensive example of POP planning with variables in the blocks world is given in Section 12.6.

Heuristics for partial-order planning

Compared with total-order planning, partial-order planning has a clear advantage in being able to decompose problems into subproblems. It also has a disadvantage in that it does not represent states directly, so it is harder to estimate how far a partial-order plan is from achieving a goal. At present, there is less understanding of how to compute accurate heuristics for partial-order planning than for total-order planning.

The most obvious heuristic is to count the number of distinct open preconditions. This can be improved by subtracting the number of open preconditions that match literals in the Start state. As in the total-order case, this overestimates the cost when there are actions that achieve multiple goals and underestimates the cost when there are negative interactions between plan steps. The next section presents an approach that allows us to get much more accurate heuristics from a relaxed problem.

The heuristic function is used to choose which plan to refine. Given this choice, the algorithm generates successors based on the selection of a single open precondition to work

Section 11.4.

Planning Graphs 395

on. As in the case of variable selection on constraint satisfaction algorithms, this selection has a large impact on efficiency. The most-constrained-variable heuristic from CSPs can be adapted for planning algorithms and seems to work well. The idea is to select the open condition that can be satisfied in the fewest number of ways. There are two special cases of this heuristic. First, if an open condition cannot be achieved by any action, the heuristic will select it; this is a good idea because early detection of impossibility can save a great deal of work. Second, if an open condition can be achieved in only one way, then it should be selected because the decision is unavoidable and could provide additional constraints on other choices still to be made. Although full computation of the number of ways to satisfy each open condition is expensive and not always worthwhile, experiments show that handling the two special cases provides very substantial speedups.

PLANNING GRAPH

LEVELS

All of the heuristics we have suggested for total-order and partial-order planning can suffer from inaccuracies. This section shows how a special data structure called a planning graph can be used to give better heuristic estimates. These heuristics can be applied to any of the search techniques we have seen so far. Alternatively, we can extract a solution directly from the planning graph, using a specialized algorithm such as the one called GRAPHPLAN.

A planning graph consists of a sequence of levels that correspond to time steps in the plan, where level 0 is the initial state. Each level contains a set of literals and a set of actions. Roughly speaking, the literals are all those that could be true at that time step, depending on the actions executed at preceding time steps. Also roughly speaking, the actions are all those actions that could have their preconditions satisfied at that time step, depending on which of the literals actually hold. We say “roughly speaking” because the planning graph records only a restricted subset of the possible negative interactions among actions; therefore, it might be optimistic about the minimum number of time steps required for a literal to become true. Nonetheless, this number of steps in the planning graph provides a good estimate of how difficult it is to achieve a given literal from the initial state. More importantly, the planning graph is defined in such a way that it can be constructed very efficiently.

Planning graphs work only for propositional planning problems—ones with no variables. As we mentioned in Section 11.1, both STRIPS and ADL representations can be propositionalized. For problems with large numbers of objects, this could result in a very substantial blowup in the number of action schemata. Despite this, planning graphs have proved to be effective tools for solving hard planning problems.

We will illustrate planning graphs with a simple example. (More complex examples lead to graphs that won’t fit on the page.) Figure 11.11 shows a problem, and Figure 11.12 shows its planning graph. We start with state level Sp, which represents the problem’s initial state. We follow that with action level Ag, in which we place all the actions whose preconditions are satisfied in the previous level. Each action is connected to its preconditions in Sy and its effects in 57, in this case introducing new literals into 57 that were not in Sp.

396

Chapter 11. Planning

PERSISTENCE ACTIONS

MUTUAL EXCLUSION MUTEX

Ingt(Have(Cake)) Goal(Have( Cake) A Eaten(Cake)) Action(Eat( Cake)

PRECOND: Have (Cake)

EFFECT: ~ Have( Cake) A Eaten(Cake)) Action(Bake(Cake)

PRECOND: — Have( Cake)

EFFECT: Have( Cake)

Figure 11.11 The “have cake and eat cake too” problem.

ke(Cake

A Bake(Cako)

Have(Cake) + Have(Cake) +, Have(Cake) 3 = Have{Cake) {1 —Have(Cake)\*. | Ea

| Su Eaten(Cake) [} Eaten(Cake)

Eaten(Cake) —\_— — Eaten(Cake) ere eee — Eaten(Cake) &

Figure 11.12 The planning graph for the “have cake and eat cake too” problem up to level So. Rectangles indicate actions (small squares indicate persistence actions) and straight lines indicate preconditions and effects. Mutex links are shown as curved gray lines.

The planning graph needs a way to represent inaction as well as action. That is, it needs the equivalent of the frame axioms in situation calculus that allow a literal to remain true from one situation to the next if no action alters it. In a planning graph this is done with a set of persistence actions. For every positive and negative literal C, we add to the problem a persistence action with precondition C' and effect C'. Figure 11.12 shows one “real” action, Eat(Cake) in Ag, along with two persistence actions drawn as small square boxes.

Level Ag contains all the actions that could occur in state Sy, but just as importantly it records conflicts between actions that would prevent them from occurring together. The gray lines in Figure 11.12 indicate mutual exclusion (or mutex) links. For example, Eat (Cake) is mutually exclusive with the persistence of either Have( Cake) or = Eaten(Cake). We shall see shortly how mutex links are computed.

Level S| contains all the literals that could result from picking any subset of the actions in Ag. It also contains mutex links (gray lines) indicating literals that could not appear together, regardless of the choice of actions. For example, Have(Cake) and Eaten(Cake) are mutex: depending on the choice of actions in Ag, one or the other, but not both, could be the result. In other words, Sp represents multiple states, just as regression state-space search does, and the mutex links are constraints that define the set of possible states.

We continue in this way, alternating between state level S; and action level A; until we reach a level where two consecutive levels are identical. At this point, we say that the graph

Section 11.4.

Planning Graphs 397

LEVELED OFF

LEVEL COST

SERIAL PLANNING GRAPH

has leveled off. Every subsequent level will be identical, so further expansion is unnecessary.

What we end up with is a structure where every A; level contains all the actions that are applicable in S;, along with constraints saying which pairs of actions cannot both be executed. Every 5; level contains all the literals that could result from any possible choice of actions in A;\_1, along with constraints saying which pairs of literals are not possible. It is important to note that the process of constructing the planning graph does not require choosing among actions, which would entail combinatorial search. Instead, it just records the impossibility of certain choices using mutex links. The complexity of constructing the planning graph is a low-order polynomial in the number of actions and literals, whereas the state space is exponential in the number of literals.

We now define mutex links for both actions and literals. A mutex relation holds between two actions at a given level if any of the following three conditions holds:

eo Inconsistent effects: one action negates an effect of the other. For example Fat( Cake) and the persistence of Have( Cake) have inconsistent effects because they disagree on the effect Have( Cake).

eo Interference: one of the effects of one action is the negation of a precondition of the other. For example Eat( Cake) interferes with the persistence of Have (Cake) by negating its precondition.

e Competing needs: one of the preconditions of one action is mutually exclusive with a precondition of the other. For example, Bake (Cake) and Eat( Cake) are mutex because they compete on the value of the Have(Cake) precondition.

A mutex relation holds between two literals at the same level if one is the negation of the other or if each possible pair of actions that could achieve the two literals is mutually exclusive. This condition is called inconsistent support. For example, Have( Cake) and Eaten(Cake) are mutex in S; because the only way of achieving Have( Cake), the persistence action, is mutex with the only way of achieving Faten(Cake), namely Fat(Cake). In Sy the two literals are not mutex because there are new ways of achieving them, such as Bake( Cake) and the persistence of Eaten (Cake), that are not mutex.

Planning graphs for heuristic estimation

A planning graph, once constructed, is a rich source of information about the problem. For example, a literal that does not appear in the final level of the graph cannot be achieved by any plan. This observation can be used in backward search as follows: any state containing an unachievable literal has a cost h(n) = occ. Similarly, in partial-order planning, any plan with an unachievable open condition has h(n) = co.

This idea can be made more general. We can estimate the cost of achieving any goal literal as the level at which it first appears in the planning graph. We will call this the level cost of the goal. In Figure 11.12, Have( Cake) has level cost 0 and Eaten (Cake) has level cost 1. It is easy to show (Exercise 11.9) that these estimates are admissible for the individual goals. The estimate might not be very good, however, because planning graphs allow several actions at each level whereas the heuristic counts just the level and not the number of actions. For this reason, it is common to use a serial planning graph for computing heuristics. A

398

Chapter 11. Planning

MAX-LEVEL

LEVEL SUM

SET-LEVEL

serial graph insists that only one action can actually occur at any given time step; this is done by adding mutex links between every pair of actions except persistence actions. Level costs extracted from serial graphs are often quite reasonable estimates of actual costs.

To estimate the cost of a conjunction of goals, there are three simple approaches. The max-level heuristic simply takes the maximum level cost of any of the goals; this is admissible, but not necessarily very accurate. The level sum heuristic, following the subgoal independence assumption, returns the sum of the level costs of the goals; this is inadmissible but works very well in practice for problems that are largely decomposable. It is much more accurate than the number-of-unsatisfied-goals heuristic from Section 11.2. For our problem, the heuristic estimate for the conjunctive goal Have( Cake) A Eaten( Cake) willbe 0+1 = 1, whereas the correct answer is 2. Moreover, if we eliminated the Bake (Cake) action, the estimate would still be 1, but the conjunctive goal would be impossible. Finally, the set-level heuristic finds the level at which all the literals in the conjunctive goal appear in the planning graph without any pair of them being mutually exclusive. This heuristic gives the correct values of 2 for our original problem and infinity for the problem without Bake( Cake). It dominates the max-level heuristic and works extremely well on tasks in which there is a good deal of interaction among subplans.

As a tool for generating accurate heuristics, we can view the planning graph as a relaxed problem that is efficiently soluble. To understand the nature of the relaxed problem, we need to understand exactly what it means for a literal g to appear at level \S; in the planning graph. Ideally, we would like it to be a guarantee that there exists a plan with ¢ action levels that achieves g, and also that if g does not appear that there is no such plan. Unfortunately, making that guarantee is as difficult as solving the original planning problem. So the planning graph makes the second half of the guarantee (if g does not appear, there is no plan), but if g does appear, then all the planning graph promises is that there is a plan that possibly achieves ¢g and has no “obvious” flaws. An obvious flaw is defined as a flaw that can be detected by considering two actions or two literals at a time—in other words, by looking at the mutex relations. There could be more subtle flaws involving three, four, or more actions, but experience has shown that it is not worth the computational effort to keep track of these possible flaws. This is similar to the lesson learned from constraint satisfaction problems that it is often worthwhile to compute 2-consistency before searching for a solution, but less often worthwhile to compute 3-consistency or higher. (See Section 5.2.)

The GRAPHPLAN algorithm

This subsection shows how to extract a plan directly from the planning graph, rather than just using the graph to provide a heuristic. The GRAPHPLAN algorithm (Figure 11.13) has two main steps, which alternate within a loop. First, it checks whether all the goal literals are present in the current level with no mutex links between any pair of them. If this is the case, then a solution might exist within the current graph, so the algorithm tries to extract that solution. Otherwise, it expands the graph by adding the actions for the current level and the state literals for the next level. The process continues until either a solution is found or it is learned that no solution exists.

Planning Graphs 399

function GRAPHPLAN( problem) returns solution or failure

graph «— INITIAL-PLANNING-GRAPH( problem) goals — GOALS| problem] loop do if goals all non-mutex in last level of graph then do solution «+ EXTRACT-SOLUTION(graph, goals, LENGTH{graph)) if solution # failure then return solution else if NO-SOLUTION-POSSIBLE(graph) then return failure graph + EXPAND-GRAPH(graph, problem)

Figure 11.13 The GRAPHPLAN algorithm. GRAPHPLAN alternates between a solution extraction step and a graph expansion step. EXTRACT-SOLUTION looks for whether a plan can be found, starting at the end and searching backwards. EXPAND-GRAPH adds the actions for the current level and the state literals for the next level.

Let us now trace the operation of GRAPHPLAN on the spare tire problem from Section 11.1. The entire graph is shown in Figure 11.14. The first line of GRAPHPLAN initializes the planning graph to a one-level (5g) graph consisting of the five literals from the initial state. The goal literal At(Spare, Azle) is not present in Sy, so we need not call EXTRACT-SOLUTION—we are certain that there is no solution yet. Instead, EXPANDGRAPH adds the three actions whose preconditions exist at level Sy (i.e., all the actions except PutOn(Spare, Azle)), along with persistence actions for all the literals in Sp. The effects of the actions are added at level S;. EXPAND-GRAPH then looks for mutex relations and adds them to the graph.

[]

Figure 11.14 The planning graph for the spare tire problem after expansion to level Ss. Mutex links are shown as gray lines. Only some representative mutexes are shown, because the graph would be too cluttered if we showed them all. The solution is indicated by bold lines and outlines.

400 Chapter 11. Planning

At (Spare, Azle) is still not present in Si, so again we do not call EXTRACT-SOLUTION. The call to EXPAND-GRAPH gives us the planning graph shown in Figure 11.14. Now that we have the full complement of actions, it is worthwhile to look at some of the examples of mutex relations and their causes:

e Inconsistent effects: Remove(Spare, Trunk) is mutex with Leave Overnight because one has the effect A¢(Spare, Ground) and the other has its negation.

eo Interference: Remove(Flat, Azle) is mutex with Leave Overnight because one has the precondition At(Flat, Azle) and the other has its negation as an effect.

o Competing needs: PutOn(Spare, Azle) is mutex with Remove(Flat, Azle) because one has At( Flat, Azle) as a precondition and the other has its negation.

e Inconsistent support: At(Spare, Azle) is mutex with At(Flat, Azle) in Sy because the only way of achieving At(Spare, Azle) is by PutOn(Spare, Azle), and that is mutex with the persistence action that is the only way of achieving A¢( Flat, Azle). Thus, the mutex relations detect the immediate conflict that arises from trying to put two objects in the same place at the same time.

This time, when we go back to the start of the loop, all the literals from the goal are present in Sy, and none of them is mutex with any other. That means that a solution might exist, and EXTRACT-SOLUTION will try to find it. In essence, EXTRACT-SOLUTION solves a Boolean CSP whose variables are the actions at each level, and the values for each variable are in or out of the plan. We can use standard CSP algorithms for this, or we can define EXTRACT-SOLUTION as a search problem, where each state in the search contains a pointer to a level in the planning graph and a set of unsatisfied goals. We define this search problem as follows:

o The initial state is the last level of the planning graph, S;,, along with the set of goals from the planning problem.

e The actions available in a state at level S; are to select any conflict-free subset of the actions in A;\_; whose effects cover the goals in the state. The resulting state has level Si—1 and has as its set of goals the preconditions for the selected set of actions. By “conflict-free,” we mean a set of actions such that no two of them are mutex, and no two of their preconditions are mutex.

e The goal is to reach a state at level Sy such that all the goals are satisfied. e The cost of each action is 1.

For this particular problem, we start at So with the goal A¢(Spare, Azle). The only choice we have for achieving the goal set is PutOn( Spare, Azle). That brings us to a search state at 5; with goals At(Spare, Ground) and —At(Flat, Azle). The former can be achieved only by Remove(Spare, Trunk), and the latter by either Remove( Flat, Azle) or Leave Overnight. But Leave Overnight is mutex with Remove (Spare, Trunk), so the only solution is to choose Remove( Spare, Trunk) and Remove( Flat, Azle). That brings us to a search state at Sy with the goals At( Spare, Trunk) and At( Flat, Azle). Both of these are present in the state, so we have a solution: the actions Remove(Spare, Trunk) and Remove(Flat, Azle) in level Ay, followed by PutOn (Spare, Azle) in Aj.

Section 11.4.

Planning Graphs 401

We know that planning is PSPACE-complete and that constructing the planning graph takes polynomial time, so it must be the case that solution extraction is intractable in the worst case. Therefore, we will need some heuristic guidance for choosing among actions during the backward search. One approach that works well in practice is a greedy algorithm based on the level cost of the literals. For any set of goals, we proceed in the following order:

1. Pick first the literal with the highest level cost.

2. To achieve that literal, choose the action with the easiest preconditions first. That is, choose an action such that the sum (or maximum) of the level costs of its preconditions is smallest.

Termination of GRAPHPLAN

So far, we have skated over the question of termination. If a problem has no solution, can we be sure that GRAPHPLAN will not loop forever, extending the planning graph at each iteration? The answer is yes, but the full proof is beyond the scope of this book. Here, we outline just the main ideas, particularly the ones that shed light on planning graphs in general.

The first step is to notice that certain properties of planning graphs are monotonically increasing or decreasing. “X increases monotonically” means that the set of Xs at level 7 + 1 is a superset (not necessarily proper) of the set at level i. The properties are as follows:

e Literals increase monotonically: Once a literal appears at a given level, it will appear at all subsequent levels. This is because of the persistence actions; once a literal shows up, persistence actions cause it to stay forever.

e Actions increase monotonically: Once an action appears at a given level, it will appear at all subsequent levels. This is a consequence of literals’ increasing; if the preconditions of an action appear at one level, they will appear at subsequent levels, and thus so will the action.

® Mutexes decrease monotonically: If two actions are mutex at a given level A;, then they will also be mutex for all previous levels at which they both appear. The same holds for mutexes between literals. It might not always appear that way in the figures, because the figures have a simplification: they display neither literals that cannot hold at level 5; nor actions that cannot be executed at level A;. We can see that “mutexes decrease monotonically” is true if you consider that these invisible literals and actions are mutex with everything.

The proof is a little complex, but can be handled by cases: if actions A and B are mutex at level A;, it must be because of one of the three types of mutex. The first two, inconsistent effects and interference, are properties of the actions themselves, so if the actions are mutex at A;, they will be mutex at every level. The third case, competing needs, depends on conditions at level .S;: that level must contain a precondition of A that is mutex with a precondition of B. Now, these two preconditions can be mutex if they are negations of each other (in which case they would be mutex in every level) or if all actions for achieving one are mutex with all actions for achieving the other. But we already know that the available actions are increasing monotonically, so by induction, the mutexes must be decreasing.

402 Chapter 11. Planning

Because the actions and literals increase and the mutexes decrease, and because there are only a finite number of actions and literals, every planning graph will eventually level off— all subsequent levels will be identical. Once a graph has leveled off, if it is missing one of the goals of the problem, or if two of the goals are mutex, then the problem can never be solved, and we can stop the GRAPHPLAN algorithm and return failure. If the graph levels off with all goals present and nonmutex, but EXTRACT-SOLUTION fails to find a solution, then we might have to extend the graph again a finite number of times, but eventually we can stop. This aspect of termination is more complex and is not covered here.

11.5 PLANNING WITH PROPOSITIONAL LOGIC

We saw in Chapter 10 that planning can be done by proving a theorem in situation calculus. That theorem says that, given the initial state and the successor-state axioms that describe the effects of actions, the goal will be true in a situation that results from a certain action sequence. As early as 1969, this approach was thought to be too inefficient for finding interesting plans. Recent developments in efficient reasoning algorithms for propositional logic (see Chapter 7) have generated renewed interest in planning as logical reasoning.

The approach we take in this section is based on testing the satisfiability of a logical sentence rather than on proving a theorem. We will be finding models of propositional sentences that look like this:

initial state N\ all possible action descriptions A goal .

The sentence will contain proposition symbols corresponding to every possible action occurrence; a model that satisfies the sentence will assign #rue to the actions that are part of a correct plan and false to the others. An assignment that corresponds to an incorrect plan will not be a model, because it will be inconsistent with the assertion that the goal is true. If the planning problem is unsolvable, then the sentence will be unsatisfiable.

Describing planning problems in propositional logic

The process we will follow to translate STRIPS problems into propositional logic is a textbook example (so to speak) of the knowledge representation cycle: We will begin with what seems to be a reasonable set of axioms, we will find that these axioms allow for spurious unintended models, and we will write more axioms.

Let us begin with a very simple air transport problem. In the initial state (time 0), plane P; is at SFO and plane Ps is at JFK. The goal is to have P; at JFK and Ps at SFO; that is, the planes are to change places. First, we will need distinct proposition symbols for assertions about each time step. We will use superscripts to denote the time step, as in Chapter 7. Thus, the initial state will be written as

At(Py, SFO)? A At(Py, JFK)°

(Remember that A¢(P;, SFO)? is an atomic symbol.) Because propositional logic has no closed-world assumption, we must also specify the propositions that are not true in the initial

Section 11.5.

Planning with Propositional Logic 403

state. If some propositions are unknown in the initial state, then they can be left unspecified (the open world assumption). In this example we specify:

—At(Py, JFK)? A ~At(Py, SFO) .

The goal itself must be associated with a particular time step. Since we do not know a priori how many steps it takes to achieve the goal, we can try asserting that the goal is true in the initial state, time 7" =0. That is, we assert At(Py, JFK)? A At(Py, SFO). If that fails, we try again with 7"=1, and so on until we reach the minimum feasible plan length. For each value of 7°, the knowledge base will include only sentences covering the time steps from 0 up to 7". To ensure termination, an arbitrary upper limit, 77,,x, is imposed. This algorithm is shown in Figure 11.15. An alternative approach that avoids multiple solution attempts is discussed in Exercise 11.17.

function SATPLAN( problem, T 1.5) returns solution or failure inputs: problem, a planning problem T max, an upper limit for plan length

for T=0to T ,.x do enf, mapping — TRANSLATE-TO-SAT (problem, T) assignment «— SAT-SOLVER(cnf) if assignment is not null then return EXTRACT-SOLUTION(assignment, mapping) return failure

Figure 11.15 The SATPLAN algorithm. The planning problem is translated into a CNF sentence in which the goal is asserted to hold at a fixed time step I" and axioms are included for each time step up to 7". (Details of the translation are given in the text.) If the satisfiability algorithm finds a model, then a plan is extracted by looking at those proposition symbols that refer to actions and are assigned true in the model. If no model exists, then the process is repeated with the goal moved one step later.

The next issue is how to encode action descriptions in propositional logic. The most straightforward approach is to have one proposition symbol for each action occurrence; for example, Fly(Py, SFO, JFK)? is true if plane P; flies from SFO to JFK at time 0. As in Chapter 7, we write propositional versions of the successor-state axioms developed for the situation calculus in Chapter 10. For example, we have

At(P, JFK)! & (At(Py, JFK)? A =(Fly(Py, JFK, SFO)° A At(Py, JFK )?)) V (Fly(Py, SFO, JFK) A At(Py, SFO)?) . That is, plane P; will be at JFK at time 1 if it was at JFK at time 0 and didn’t fly away, or it was at SFO at time 0 and flew to JFK. We need one such axiom for each plane, airport, and time step. Moreover, each additional airport adds another way to travel to or from a given airport and hence adds more disjuncts to the right-hand side of each axiom.

With these axioms in place, we can run the satisfiability algorithm to find a plan. There

ought to be a plan that achieves the goal at time 7" = 1, namely, the plan in which the two

(11.1)

404

Chapter 11. Planning

PRECONDITION AXIOMS

ACTION EXCLUSION AXIOMS

planes swap places. Now, suppose the KB is initial state A successor-state axioms A goal , (11.2) which asserts that the goal is true at time 7" = 1. You can check that the assignment in which Fly(P1, SFO, JFK)? and Fly(Py, JFK, SFO)°

are true and all other action symbols are false is a model of the KB. So far, so good. Are there other possible models that the satisfiability algorithm might return? Indeed, yes. Are all these other models satisfactory plans? Alas, no. Consider the rather silly plan specified by the action symbols

Fly(Py, SFO, JFK)® and Fly(P,, JFK , SFO)? and Fly(P, JFK, SFO) .

This plan is silly because plane P; starts at SFO, so the action Fly(Py, JFK, SFO)? is infeasible. Nonetheless, the plan is a model of the sentence in Equation (11.2)! That is, it is consistent with everything we have said so far about the problem. To understand why, we need to look more carefully at what the successor-state axioms (such as Equation (11.1)) say about actions whose preconditions are not satisfied. The axioms do predict correctly that nothing will happen when such an action is executed (see Exercise 11.15), but they do not say that the action cannot be executed! To avoid generating plans with illegal actions, we must add precondition axioms stating that an action occurrence requires the preconditions to be satisfied.® For example, we need Fly(P;, JFK,SFO)° = At(P, JFK).

Because At(P1, JFK)? is stated to be false in the initial state, this axiom ensures that every model also has Fly(Py, JFK , SFO) set to false. With the addition of precondition axioms, there is exactly one model that satisfies all of the axioms when the goal is to be achieved at time 1, namely the model in which plane P; flies to JFK and plane P flies to SFO. Notice that this solution has two parallel actions, just as with GRAPHPLAN or POP.

More surprises emerge when we add a third airport, LAX. Now, each plane has two actions that are legal in each state. When we run the satisfiability algorithm, we find that a model with Fly(Py, SFO, JFK)® and Fly(P, JFK, SFO)? and Fly(P,, JFK,LAX)° satisfies all the axioms. That is, the successor-state and precondition axioms allow a plane to fly to two destinations at once! The preconditions for the two flights by P, are satisfied in the initial state; the successor-state axioms say that P, will be at SFO and LAX at time 1; so the goal is satisfied. Clearly, we must add more axioms to eliminate these spurious solutions. One approach is to add action exclusion axioms that prevent simultaneous actions. For example, we can insist on complete exclusion by adding all possible axioms of the form

—~(Fly(Py, JFK, SFO)° A Fly(Py, JFK, LAX") . These axioms ensure that no two actions can occur at the same time. They eliminate all spurious plans, but also force every plan to be totally ordered. This loses the flexibility of partially ordered plans; also, by increasing the number of time steps in the plan, computation time may be lengthened.

6 Notice that the addition of precondition axioms means that we need not include preconditions for actions in the successor-state axioms.

Section 11.5.

Planning with Propositional Logic 405

STATE CONSTRAINTS

Instead of complete exclusion, we can require only partial exclusion—that is, rule out simultaneous actions only if they interfere with each other. The conditions are the same as those for mutex actions: two actions cannot occur simultaneously if one negates a precondition or effect of the other. For example, Fly(P, JFK, SFO)? and Fly(P, JFK, LAX)? cannot both occur, because each negates the precondition of the other; on the other hand, Fly(Py, SFO, JFK)? and Fly(P, JFK, SFO)° can occur together because the two planes do not interfere. Partial exclusion eliminates spurious plans without forcing a total ordering.

Exclusion axioms sometimes seem a rather blunt instrument. Instead of saying that a plane cannot fly to two airports at the same time, we might simply insist that no object can be in two places at once:

Vp, z,y,t x #y = —(At(p, 2)" A At(p,y)').

This fact, combined with the successor-state axioms, implies that a plane cannot fly to two airports at the same time. Facts such as this are called state constraints. In propositional logic, of course, we have to write out all the ground instances of each state constraint. For the airport problem, the state constraint suffices to rule out all spurious plans. State constraints are often much more compact than action exclusion axioms, but they are not always easy to derive from the original STRIPS description of a problem.

To summarize, planning as satisfiability involves finding models for a sentence containing the initial state, the goal, the successor-state axioms, the precondition axioms, and either the action exclusion axioms or the state constraints. It can be shown that this collection of axioms is sufficient, in the sense that there are no longer any spurious “solutions.” Any model satisfying the propositional sentence will be a valid plan for the original problem—that is, every linearization of the plan is a legal sequence of actions that reaches the goal.

Complexity of propositional encodings

The principal drawback of the propositional approach is the sheer size of the propositional knowledge base that is generated from the original planning problem. For example, the action schema Fly(p, a1, az) becomes T' x | Planes| x | Airports|? different proposition symbols. In general, the total number of action symbols is bounded by T x |Act| x |O|F, where | Act| is the number of action schemata, |O| is the number of objects in the domain, and P is the maximum arity (number of arguments) of any action schema. The number of clauses is larger still. For example, with 10 time steps, 12 planes, and 30 airports, the complete action exclusion axiom has 583 million clauses.

Because the number of action symbols is exponential in the arity of the action schema, one answer might be to try to reduce the arity. We can do this by borrowing an idea from semantic networks (Chapter 10). Semantic networks use only binary predicates; predicates with more arguments are reduced to a set of binary predicates that describe each argument separately. Applying this idea to an action symbol such as Fly(P;, SFO, JFK), we obtain three new symbols:

Fly; (Py)? : plane Pj flew at time 0

Flys (SFO)? : the origin of the flight was SFO Fly3(JFK)? : the destination of the flight was JFK .

406

Chapter 11. Planning

SYMBOL SPLITTING

This process, called symbol splitting, eliminates the need for an exponential number of symbols. Now we only need T° x |Act| x P x |O).

Symbol splitting by itself can reduce the number of symbols, but does not automatically reduce the number of axioms in the KB. That is, if each action symbol in each clause were simply replaced by a conjunction of three symbols, then the total size of the KB would remain roughly the same. Symbol splitting actually does reduce the size of the KB because some of the split symbols will be irrelevant to certain axioms and can be omitted. For example, consider the successor-state axiom in Equation (11.1), modified to include LAX and to omit action preconditions (which will be covered by separate precondition axioms):

At(P, JFK)! & (At(Py, JFK)° A —Fly(Py, JFK, SFO)? A ~Fly(Py, JFK, LAX)°) V Fly(Py, SFO, JFK)° v Fly(P,, LAX, JFK)° .

The first condition says that P; will be at JFK if it was there at time 0 and didn’t fly from JFK to any other city, no matter which one; the second says it will be there if it flew to JFK from another city, no matter which one. Using the split symbols, we can simply omit the argument whose value does not matter:

At(Py, JFK)! < (At(P1, JFK)? A (Fly (P)° A Fly2(JFK)?)) V (Fly: (P1)° A Flys (JFK)Y) .

Notice that SFO and LAX are no longer mentioned in the axiom. More generally, the split action symbols now allow the size of each successor-state axiom to be independent of the number of airports. Similar reductions occur with the precondition axioms and action exclusion axioms (see Exercise 11.16). For the case described earlier with 10 time steps, 12 planes, and 30 airports, the complete action exclusion axiom is reduced from 583 million clauses to 9,360 clauses.

There is one drawback: the split-symbol representation does not allow for parallel actions. Consider the two parallel actions Fly(P;, SFO, JFK)? and Fly(Py, JFK, SFO). Converting to the split representation, we have

Fly1(Py)° A Flys (SFO) A Flys(JFK)® A Flyy(P)° A Flys (JFK)? A Flys(SFO)0

It is no longer possible to determine what happened! We know that P; and P» flew, but we cannot identify the origin and destination of each flight. This means that a complete action exclusion axiom must be used, with the drawbacks noted previously.

Planners based on satisfiability can handle large planning problems—for example, finding optimal 30-step solutions to blocks-world planning problems with dozens of blocks. The size of the propositional encoding and the cost of solution are highly problem-dependent, but in most cases the memory required to store the propositional axioms is the bottleneck. One interesting finding from this work has been that backtracking algorithms such as DPLL are often better at solving planning problems than local search algorithms such as WALKS AT. This is because the majority of the propositional axioms are Horn clauses, which are handled efficiently by the unit propagation technique. This observation has led to the development of hybrid algorithms combining some random search with backtracking and unit propagation.

Section 11.6. Analysis of Planning Approaches 407

11.6 ANALYSIS OF PLANNING APPROACHES

Planning is an area of great current interest within Al. One reason for this is that it combines the two major areas of Al we have covered so far: search and logic. That is, a planner can be seen either as a program that searches for a solution or as one that (constructively) proves the existence of a solution. The cross-fertilization of ideas from the two areas has led to both improvements in performance amounting to several orders of magnitude in the last decade and an increased use of planners in industrial applications. Unfortunately, we do not yet have a clear understanding of which techniques work best on which kinds of problems. Quite possibly, new techniques will emerge that dominate existing methods.

Planning is foremost an exercise in controlling combinatorial explosion. If there are p primitive propositions in a domain, then there are 2? states. For complex domains, p can grow quite large. Consider that objects in the domain have properties (Location, Color, etc.) and relations (At, On, Between, etc.). With d objects in a domain with ternary relations, we get 24° states. We might conclude that, in the worst case, planning is hopeless.

Against such pessimism, the divide-and-conquer approach can be a powerful weapon. In the best case—full decomposability of the problem—divide-and-conquer offers an exponential speedup. Decomposability is destroyed, however, by negative interactions between actions. Partial-order planners deal with this with causal links, a powerful representational approach, but unfortunately each conflict must be resolved with a choice (put the conflicting action before or after the link), and the choices can multiply exponentially. GRAPHPLAN avoids these choices during the graph construction phase, using mutex links to record conflicts without actually making a choice as to how to resolve them. SATPLAN represents a similar range of mutex relations, but does so by using the general CNF form rather than a specific data structure. How well this works depends on the SAT solver used.

Sometimes it is possible to solve a problem efficiently by recognizing that negative

SERIALIZABLE interactions can be ruled out. We say that a problem has serializable subgoals if there exists an order of subgoals such that the planner can achieve them in that order, without having to undo any of the previously achieved subgoals. For example, in the blocks world, if the goal is to build a tower (e.g., A on B, which in turn is on C, which in turn is on the Table), then the subgoals are serializable bottom to top: if we first achieve C on Table, we will never have to undo it while we are achieving the other subgoals. A planner that uses the bottom-to-top trick can solve any problem in the blocks world domain without backtracking (although it might not always find the shortest plan).

As a more complex example, for the Remote Agent planner which commanded NASA’s Deep Space One spacecraft, it was determined that the propositions involved in commanding a spacecraft are serializable. This is perhaps not too surprising, because a spacecraft is designed by its engineers to be as easy as possible to control (subject to other constraints). Taking advantage of the serialized ordering of goals, the Remote Agent planner was able to eliminate most of the search. This meant that it was fast enough to control the spacecraft in real time, something previously considered impossible.

408

Chapter 11. Planning

There is more than one way to control combinatorial explosions. We saw in Chapter 5 that there are many techniques for controlling backtracking in constraint satisfaction problems (CSPs), such as dependency-directed backtracking. All of these techniques can be applied to planning. For example, extracting a solution from a planning graph can be formulated as a Boolean CSP whose variables state whether a given action should occur at a given time. The CSP can be solved using any of the algorithms in Chapter 5, such as min-conflicts. A closely related method, used in the BLACKBOX system, is to convert the planning graph into a CNF expression and then extract a plan by using a SAT solver. This approach seems to work better than SATPLAN, presumably because the planning graph has already eliminated many of the impossible states and actions from the problem. It also works better than GRAPHPLAN, presumably because a satisfiability search such as WALKS AT has much greater flexibility than the strict backtracking search that GRAPHPLAN uses.

There is no doubt that planners such as GRAPHPLAN, SATPLAN, and BLACKBOX have moved the field of planning forward, both by raising the level of performance of planning systems and by clarifying the representational and combinatorial issues involved. These methods are, however, inherently propositional and thus are limited in the domains they can express. (For example, logistics problems with a few dozen objects and locations can require gigabytes of storage for the corresponding CNF expressions.) It seems likely that first-order representations and algorithms will be required if further progress is to occur, although structures such as planning graphs will continue to be useful as a source of heuristics.

In this chapter, we defined the problem of planning in deterministic, fully observable environments. We described the principal representations used for planning problems and several algorithmic approaches for solving them. The points to remember are:

e Planning systems are problem-solving algorithms that operate on explicit propositional (or first-order) representations of states and actions. These representations make possible the derivation of effective heuristics and the development of powerful and flexible algorithms for solving problems.

e The STRIPS language describes actions in terms of their preconditions and effects and describes the initial and goal states as conjunctions of positive literals. The ADL language relaxes some of these constraints, allowing disjunction, negation, and quantifiers.

e State-space search can operate in the forward direction (progression) or the backward direction (regression). Effective heuristics can be derived by making a subgoal independence assumption and by various relaxations of the planning problem.

e Partial-order planning (POP) algorithms explore the space of plans without committing to a totally ordered sequence of actions. They work back from the goal, adding actions to the plan to achieve each subgoal. They are particularly effective on problems amenable to a divide-and-conquer approach.

Section 11.7. Summary 409

o A planning graph can be constructed incrementally, starting from the initial state. Each layer contains a superset of all the literals or actions that could occur at that time step and encodes mutual exclusion, or mutex, relations among literals or actions that cannot cooccur. Planning graphs yield useful heuristics for state-space and partial-order planners and can be used directly in the GRAPHPLAN algorithm.

e The GRAPHPLAN algorithm processes the planning graph, using a backward search to extract a plan. It allows for some partial ordering among actions.

eo The SATPLAN algorithm translates a planning problem into propositional axioms and applies a satisfiability algorithm to find a model that corresponds to a valid plan. Several different propositional representations have been developed, with varying degrees of compactness and efficiency.

o Each of the major approaches to planning has its adherents, and there is as yet no consensus on which is best. Competition and cross-fertilization among the approaches have resulted in significant gains in efficiency for planning systems.

BIBLIOGRAPHICAL AND HISTORICAL NOTES

Al planning arose from investigations into state-space search, theorem proving, and control theory and from the practical needs of robotics, scheduling, and other domains. STRIPS (Fikes and Nilsson, 1971), the first major planning system, illustrates the interaction of these influences. STRIPS was designed as the planning component of the software for the Shakey robot project at SRI. Its overall control structure was modeled on that of GPS, the General Problem Solver (Newell and Simon, 1961), a state-space search system that used means—ends analysis. STRIPS used a version of the Q A3 theorem proving system (Green, 1969b) as a subroutine for establishing the truth of preconditions for actions. Lifschitz (1986) offers precise definitions and an analysis of the STRIPS language. Bylander (1992) shows simple STRIPS planning to be PSPACE-complete. Fikes and Nilsson (1993) give a historical retrospective on the STRIPS project and a survey of its relationship to more recent planning efforts.

The action representation used by STRIPS has been far more influential than its algorithmic approach. Almost all planning systems since then have used one variant or another of the STRIPS language. Unfortunately, the proliferation of variants has made comparisons needlessly difficult. With time came a better understanding of the limitations and tradeoffs among formalisms. The Action Description Language, or ADL, (Pednault, 1986) relaxed some of the restrictions in the STRIPS language and made it possible to encode more realistic problems. Nebel (2000) explores schemes for compiling ADL into STRIPS. The Problem Domain Description Language or PDDL (Ghallab et al., 1998) was introduced as a computer-parsable, standardized syntax for representing STRIPS, ADL, and other languages. PDDL has been used as the standard language for the planning competitions at the AIPS conference, beginning in 1998.

Planners in the early 1970s generally worked with totally ordered action sequences. Problem decomposition was achieved by computing a subplan for each subgoal and then

410

Chapter 11. Planning

LINEAR PLANNING

INTERLEAVING

stringing the subplans together in some order. This approach, called linear planning by Sacerdoti (1975), was soon discovered to be incomplete. It cannot solve some very simple problems, such as the Sussman anomaly (see Exercise 11.11), found by Allen Brown during experimentation with the HACKER system (Sussman, 1975). A complete planner must allow for interleaving of actions from different subplans within a single sequence. The notion of serializable subgoals (Korf, 1987) corresponds exactly to the set of problems for which noninterleaved planners are complete.

One solution to the interleaving problem was goal regression planning, a technique in which steps in a totally ordered plan are reordered so as to avoid conflict between subgoals. This was introduced by Waldinger (1975) and also used by Warren’s (1974) WARPLAN. WARPLAN is also notable in that it was the first planner to be written in a logic programming language (Prolog) and is one of the best examples of the remarkable economy that can sometimes be gained by using logic programming: WARPLAN is only 100 lines of code, a small fraction of the size of comparable planners of the time. INTERPLAN (Tate, 1975a, 1975b) also allowed arbitrary interleaving of plan steps to overcome the Sussman anomaly and related problems.

The ideas underlying partial-order planning include the detection of conflicts (Tate, 1975a) and the protection of achieved conditions from interference (Sussman, 1975). The construction of partially ordered plans (then called task networks) was pioneered by the NOAH planner (Sacerdoti, 1975, 1977) and by Tate’s (1975b, 1977) NONLIN system.’

Partial-order planning dominated the next 20 years of research, yet for much of that time, the field was not widely understood. TWEAK (Chapman, 1987) was a logical reconstruction and simplification of planning work of this time; his formulation was clear enough to allow proofs of completeness and intractability (NP-hardness and undecidability) of various formulations of the planning problem. Chapman’s work led to what was arguably the first simple and readable description of a complete partial-order planner (McAllester and Rosenblitt, 1991). An implementation of McAllester and Rosenblitt’s algorithm called SNLP (Soderland and Weld, 1991) was widely distributed and allowed many researchers to understand and experiment with partial-order planning for the first time. The POP algorithm described in this chapter is based on SNLP.

Weld’s group also developed UCPOP (Penberthy and Weld, 1992), the first planner for problems expressed in ADL. UCPOP incorporated the number-of-unsatisfied-goals heuristic. It ran somewhat faster than SNLP, but was seldom able to find plans with more than a dozen or so steps. Although improved heuristics were developed for UCPOP (Joslin and Pollack, 1994; Gerevini and Schubert, 1996), partial-order planning fell into disrepute in the 1990s as faster methods emerged. Nguyen and Kambhampati (2001) suggest that a rehabilitation is merited: with accurate heuristics derived from a planning graph, their REPOP planner scales up much better than GRAPHPLAN and is competitive with the fastest state-space planners.

Avrim Blum and Merrick Furst (1995, 1997) revitalized the field of planning with their GRAPHPLAN system, which was orders of magnitude faster than the partial-order planners of

7 Some confusion exists over terminology. Many authors use the term nonlinear to mean partially ordered. This is slightly different from Sacerdoti’s original usage referring to interleaved plans.

Section 11.7.

Summary 411

BINARY DECISION DIAGRAMS

the time. Other graph planning systems, such as IPP (Koehler ef al., 1997), STAN (Fox and Long, 1998) and SGP (Weld ef al., 1998), soon followed. A data structure closely resembling the planning graph had been developed slightly earlier by Ghallab and Laruelle (1994), whose IXTET partial-order planner used it to derive accurate heuristics to guide searches. Nguyen et al. (2001) give a very thorough analysis of heuristics derived from planning graphs. Our discussion of planning graphs is based partly on this work and on lecture notes by Subbarao Kambhampati. As mentioned in the chapter, a planning graph can be used in many different ways to guide the search for a solution. The winner of the 2002 AIPS planning competition, LPG (Gerevini and Serina, 2002), searched planning graphs using a local search technique inspired by WALKSAT.

Planning as satisfiability and the SATPLAN algorithm were proposed by Kautz and Selman (1992), who were inspired by the surprising success of greedy local search for satisfiability problems. (See Chapter 7.) Kautz et al. (1996) also investigated various forms of propositional representations for STRIPS axioms, finding that the most compact forms did not necessarily lead to the fastest solution times. A systematic analysis was carried out by Ernst et al. (1997), who also developed an automatic “compiler” for generating propositional representations from PDDL problems. The BLACKBOX planner, which combines ideas from GRAPHPLAN and SATPLAN, was developed by Kautz and Selman (1998).

The resurgence of interest in state-space planning was pioneered by Drew McDermott’s UNPOP program (1996), which was the first to suggest a distance heuristic based on a relaxed problem with delete lists ignored. The name UNPOP was a reaction to the overwhelming concentration on partial-order planning at the time; McDermott suspected that other approaches were not getting the attention they deserved. Bonet and Geffner’s Heuristic Search Planner (HSP) and its later derivatives (Bonet and Geffner, 1999) were the first to make state-space search practical for large planning problems. The most successful state-space searcher to date is Hoffmann’s (2000) FASTFORWARD or FF, winner of the ATPS 2000 planning competition. FF uses a simplified planning graph heuristic with a very fast search algorithm that combines forward and local search in a novel way.

Most recently, there has been interest in the representation of plans as binary decision diagrams, a compact description of finite automata widely studied in the hardware verification community (Clarke and Grumberg, 1987; McMillan, 1993). There are techniques for proving properties of binary decision diagrams, including the property of being a solution to a planning problem. Cimatti ef al. (1998) present a planner based on this approach. Other representations have also been used; for example, Vossen et al. (2001) survey the use of integer programming for planning.

The jury is still out, but there are now some interesting comparisons of the various approaches to planning. Helmert (2001) analyzes several classes of planning problems, and shows that constraint-based approaches, such as GRAPHPLAN and SATPLAN are best for NPhard domains, while search-based approaches do better in domains where feasible solutions can be found without backtracking. GRAPHPLAN and SATPLAN have trouble in domains with many objects, because that means they must create many actions. In some cases the problem can be delayed or avoided by generating the propositionalized actions dynamically, only as needed, rather than instantiating them all before the search begins.

412

Chapter 11. Planning

Weld (1994, 1999) provides two excellent surveys of modern planning algorithms. It is interesting to see the change in the five years between the two surveys: the first concentrates on partial-order planning, and the second introduces GRAPHPLAN and SATPLAN. Readings in Planning (Allen et al., 1990) is a comprehensive anthology of many of the best earlier articles in the field, including several good surveys. Yang (1997) provides a book-length overview of partial-order planning techniques.

Planning research has been central to Al since its inception, and papers on planning are a staple of mainstream Al journals and conferences. There are also specialized conferences such as the International Conference on Al Planning Systems (AIPS), the International Workshop on Planning and Scheduling for Space, and the European Conference on Planning.

EXERCISES

11.1 Describe the differences and similarities between problem solving and planning.

11.2 Given the axioms from Figure 11.2, what are all the applicable concrete instances of Fly(p, from, to) in the state described by

At( Py, JFK) A At (Py, SFO) A Plane( Py) A Plane(P,) A Airport (JFK) A Airport(SFO) ?

11.3 Let us consider how we might translate a set of STRIPS schemata into the successorstate axioms of situation calculus. (See Chapter 10.)

e Consider the schema for Fly(p, from, to). Write a logical definition for the predicate FlyPrecond(p, from, to, s), which is true if the preconditions for Fly(p, from, to) are satisfied in situation s.

e Next, assuming that Fly(p, from, to) is the only action schema available to the agent, write down a successor-state axiom for At(p, x, s) that captures the same information as the action schema.

e Now suppose there is an additional method of travel: Teleport(p, from, to). It has the additional precondition — Warped (p) and the additional effect Warped (p). Explain how the situation calculus knowledge base must be modified.

o Finally, develop a general and precisely specified procedure for carrying out the translation from a set of STRIPS schemata to a set of successor-state axioms.

11.4 The monkey-and-bananas problem is faced by a monkey in a laboratory with some bananas hanging out of reach from the ceiling. A box is available that will enable the monkey to reach the bananas if he climbs on it. Initially, the monkey is at A, the bananas at B, and the box at C'. The monkey and box have height Low, but if the monkey climbs onto the box he will have height High, the same as the bananas. The actions available to the monkey include Go from one place to another, Push an object from one place to another, ClimbUp onto or

Section 11.7. Summary 413

ClimbDown from an object, and Grasp or Ungrasp an object. Grasping results in holding the object if the monkey and object are in the same place at the same height.

a. Write down the initial state description. b. Write down STRIPS-style definitions of the six actions.

¢. Suppose the monkey wants to fool the scientists, who are off to tea, by grabbing the bananas, but leaving the box in its original place. Write this as a general goal (i.e., not assuming that the box is necessarily at C) in the language of situation calculus. Can this goal be solved by a STRIPS-style system?

d. Your axiom for pushing is probably incorrect, because if the object is too heavy, its position will remain the same when the Push operator is applied. Is this an example of the ramification problem or the qualification problem? Fix your problem description to account for heavy objects.

11.5 Explain why the process for generating predecessors in backward search does not need to add the literals that are negative effects of the action.

11.6 Explain why dropping negative effects from every action schema in a STRIPS problem results in a relaxed problem.

11.7 Examine the definition of bidirectional search in Chapter 3.

a. Would bidirectional state-space search be a good idea for planning?

b. What about bidirectional search in the space of partial-order plans?

¢. Devise a version of partial-order planning in which an action can be added to a plan if its preconditions can be achieved by the effects of actions already in the plan. Explain how to deal with conflicts and ordering constraints. Is the algorithm essentially identical to forward state-space search?

d. Consider a partial-order planner that combines the method in part (c) with the standard

method of adding actions to achieve open conditions. Would the resulting algorithm be the same as part (b)?

11.8 Construct levels 0, 1, and 2 of the planning graph for the problem in Figure 11.2.

11.9 Prove the following assertions about planning graphs:

o A literal that does not appear in the final level of the graph cannot be achieved.

o The level cost of a literal in a serial graph is no greater than the actual cost of an optimal plan for achieving it.

11.10 We contrasted forward and backward state-space search planners with partial-order planners, saying that the latter is a plan-space searcher. Explain how forward and backward state-space search can also be considered plan-space searchers, and say what the plan refinement operators are.

414

SUSSMAN ANOMALY

Chapter 11. Planning

11.11 Figure 11.16 shows a blocks-world problem known as the Sussman anomaly. The problem was considered anomalous because the noninterleaved planners of the early 1970s could not solve it. Write a definition of the problem in STRIPS notation and solve it, either by hand or with a planning program. A noninterleaved planner is a planner that, when given two subgoals G1 and Gy, produces either a plan for G; concatenated with a plan for Go, or vice-versa. Explain why a noninterleaved planner cannot solve this problem.

C

[BA

Start State Goal State

| aE

Figure 11.16 The “Sussman anomaly” blocks-world planning problem.

11.12 Consider the problem of putting on one’s shoes and socks, as defined in Section 11.3. Apply GRAPHPLAN to this problem and show the solution obtained. Now add actions for putting on a coat and a hat. Show the partial order plan that is a solution, and show that there are 180 different linearizations of the partial-order plan. What is the minimum number of different planning graph solutions needed to represent all 180 linearizations?

11.13 The original STRIPS program was designed to control Shakey the robot. Figure 11.17 shows a version of Shakey’s world consisting of four rooms lined up along a corridor, where each room has a door and a light switch.

The actions in Shakey’s world include moving from place to place, pushing movable objects (such as boxes), climbing onto and down from rigid objects (such as boxes), and turning light switches on and off. The robot itself was never dexterous enough to climb on a box or toggle a switch, but the STRIPS planner was capable of finding and printing out plans that were beyond the robot’s abilities. Shakey’s six actions are the following:

e Go(z,y), which requires that Shakey be at = and that = and y are locations in the same room. By convention a door between two rooms is in both of them.

e Push a box b from location x to location y within the same room: Push(b, x,y). We will need the predicate Boz and constants for the boxes.

e Climb onto a box: ClimbUp(b); climb down from a box: ClimbDown(b). We will need the predicate On and the constant Floor.

e Turn a light switch on: TurnOn(s); turn it off: TurnOff(s). To turn a light on or off, Shakey must be on top of a box at the light switch’s location.

Section 11.7.

Summary 415

Switch d Door 4 Room 4 mY Door 3 Room 3 Shakey Swich ; Corridor Door 2 Room 2 Switch 1 Box 3 Box 2 q Door 1 Room 1 Box 4 Box 1 Figure 11.17 Shakey’s world. Shakey can move between landmarks within a room, can pass through the door between rooms, can climb climbable objects and push pushable objects, and can flip light switches.

Describe Shakey’s six actions and the initial state from Figure 11.17 in STRIPS notation. Construct a plan for Shakey to get Boxs into Rooms.

11.14 We saw that planning graphs can handle only propositional actions. What if we want to use planning graphs for a problem with variables in the goal, such as At(Py, 2) A At (Py, z), where x ranges over a finite domain of locations? How could you encode such a problem to work with planning graphs? (Hint: remember the Finish action from POP planning. What preconditions should it have?)

11.15 Up to now we have assumed that actions are only executed in the appropriate situations. Let us see what propositional successor-state axioms such as Equation (11.1) have to say about actions whose preconditions are not satisfied.

a. Show that the axioms predict that nothing will happen when an action is executed in a state where its preconditions are not satisifed.

416 Chapter 11. Planning

b. Consider a plan p that contains the actions required to achieve a goal but also includes illegal actions. Is it the case that

initial state N\ successor-state axioms \ p |= goal ?

¢. With first-order successor-state axioms in situation calculus (as in Chapter 10), is it possible to prove that a plan containing illegal actions will achieve the goal?

11.16 Giving examples from the airport domain, explain how symbol-splitting reduces the size of the precondition axioms and the action exclusion axioms. Derive a general formula for the size of each axiom set in terms of the number of time steps, the number of action schemata, their arities, and the number of objects.

11.17 In the SATPLAN algorithm in Figure 11.15, each call to the satisfiability algorithm asserts a goal gt, where T ranges from 0 to Tyax. Suppose instead that the satisfiability algorithm is called only once, with the goal ¢" V g' Vv -.. vv gTmax,

a. Will this always return a plan if one exists with length less than or equal to Tay?

b. Does this approach introduce any new spurious “solutions”?

c¢. Discuss how one might modify a satisfiability algorithm such as WALKSAT so that it finds short solutions (if they exist) when given a disjunctive goal of this form.

1 ) PLANNING AND ACTING IN THE REAL WORLD

In which we see how more expressive representations and more interactive agent architectures lead to planners that are useful in the real world.

The previous chapter introduced the most basic concepts, representations, and algorithms for planning. Planners that are are used in the real world for tasks such as scheduling Hubble Space Telescope observations, operating factories, and handling the logistics for military campaigns are more complex; they extend the basics in terms both of the representation language and of the way the planner interacts with the environment. This chapter shows how. Section 12.1 describes planning and scheduling with time and resource constraints, and Section 12.2 describes planning with predefined subplans. Sections 12.3 to 12.6 present a series of agent architectures designed to deal with uncertain environments. Section 12.7 shows how to plan when the environment contains other agents.

12.1 TIME, SCHEDULES, AND RESOURCES

The STRIPS representation talks about what actions do, but, because the representation is based on situation calculus, it cannot talk about sow long an action takes or even about when an action occurs, except to say that it is before or after another action. For some domains, we would like to talk about when actions begin and end. For example, in the cargo delivery domain, we might like to know when the plane carrying some cargo will arrive, not just that it will arrive when it is done flying. 0B SHOP Time is of the essence in the general family of applications called job shop scheduling. Such tasks require completing a set of jobs, each of which consists of a sequence of actions, where each action has a given duration and might require some resources. The problem is to determine a schedule that minimizes the total time required to complete all the jobs, while respecting the resource constraints. An example of a job shop scheduling problem is given in Figure 12.1. This is a highly simplified automobile assembly problem. There are two jobs: assembling cars Cj and Co. Each job consists of three actions: adding the engine, adding the wheels, and inspecting the

417

418

Chapter 12. Planning and Acting in the Real World

CRITICAL PATH METHOD

CRITICAL PATH

SLACK

SCHEDULE

Init(Chassis(C1) A Chassis(Ca) A Engine (Eq, C1, 30) A Engine(E2, Co, 60) A Wheels(W1, C1, 30) A Wheels(Ws, Ca, 15)) Goal(Done(C1) A Done(C2))

Action(AddEngine(e, c), PRECOND: Engine(e, c,d) A Chassis(c) A —Engineln(c), EFFECT: Engineln(c) A Duration(d)) Action(AddWheels(w, ¢), PRECOND: Wheels(w, c,d) A Chassis(c) A Engineln(c), EFFECT: WheelsOn(c) A Duration(d)) Action (Inspect(c), PRECOND: Engineln(c) A WheelsOn(c) A Chassis(c), EFFECT: Done(c) A Duration(10))

Figure 12.1 A job shop scheduling problem for assembling two cars. The notation Duration(d) means that an action takes d minutes to execute. Engine(E,, Cy, 60) means that £4 1s an engine that fits into chassis C1 and takes 60 minutes to install.

results. The engine must be put in first (because having the front wheels on would inhibit access to the engine compartment) and of course the inspection must be done last.

The problem in Figure 12.1 can be solved by any of the planners we have already seen. Figure 12.2 (if you ignore the numbers) shows the solution that the partial-order planner POP would come up with. To make this a scheduling problem rather than a planning problem, we must now determine when each action should begin and end, based on the durations of actions as well as their ordering. The notation Duration (d) in the effect of an action (where d must be bound to a number) means that the action takes d minutes to complete.

Given a partial ordering of actions with durations, as in Figure 12.2, we can apply the critical path method (CPM) to determine the possible start and end times of each action. A path through a partial-order plan is a linearly ordered sequence of actions beginning with Start and ending with Finish. (For example, there are two paths in the partial-order plan in Figure 12.2.)

The critical path is that path whose total duration is longest; the path is “critical” because it determines the duration of the entire plan—shortening other paths doesn’t shorten the plan as a whole, but delaying the start of any action on the critical path slows down the whole plan. In the figure, the critical path is shown with bold lines. To complete the whole plan in the minimal total time, the actions on the critical path must be executed with no delay between them. Actions that are off the critical path have some leeway—a window of time in which they can be executed. The window is specified in terms of an earliest possible start time, ES, and a latest possible start time, LS. The quantity LS — ES is known as the slack of an action. We can see in Figure 12.2 that the whole plan will take 80 minutes, that each action on the critical path has 0 slack (this will always be the case) and that each of the actions in the assembly of C; have a 10-minute window in which they can be started. Together the ES and LS times for all the actions constitute a schedule for the problem.

Section 12.1.

Time, Schedules, and Resources 419

[]

Figure 12.2 A solution to the job shop scheduling problem from Figure 12.1. At the top, the solution is given as a partial-order plan. The duration of each action is given at the bottom of each rectangle, with the earliest and latest start time listed as [ES, LS] in the upper left. The difference between these two numbers is the slack of an action; actions with zero slack are on the critical path, shown with bold arrows. At the bottom of the figure, the same solution is shown as a timeline. Grey rectangles represent time intervals during which an action may be executed, provided that the ordering constraints are respected. The unoccupied portion of

a gray rectangle indicates the slack.

The following formulas serve as a definition for ES and LS and also as the outline of a dynamic programming algorithm to compute them:

ES(Start) =0.

ES(B) = max

LS(Finish) = ES(Finish) .

LS(A) = ming gp LS(B) — Duration(A) . The idea is that we start by assigning ES (Start) to be 0. Then as soon as we get an action B such that all the actions that come immediately before B have ES values assigned, we set £S(B) to be the maximum of the earliest finish times of those immediately preceding actions, where the earliest finish time of an action is defined as the earliest start time plus the duration. This process repeats until every action has been assigned an ES value. The LS values are computed in a similar manner, working backwards from the Finish action. The details are left as an exercise.

The complexity of the critical path algorithm is just O(Nb), where IV is the number of actions and b is the maximum branching factor into or out of an action. (To see this,

420

Chapter 12. Planning and Acting in the Real World

RESOURCES

REUSABLE RESOURCE

AGGREGATION

note that the LS and ES computations are done once for each action, and each computation iterates over at most b other actions.) Therefore, the problem of finding a minimum-duration schedule, given a partial ordering on the actions, is quite easy to solve.

Scheduling with resource constraints

Real scheduling problems are complicated by the presence of constraints on resources. For example, adding an engine to a car requires an engine hoist. If there is only one hoist, then we cannot simultaneously add engine Fj to car C1 and engine Es to car C2; hence, the schedule shown in Figure 12.2 would be infeasible. The engine hoist is an example of a reusable resource—a resource that is “occupied” during the action but that becomes available again when the action is finished. Notice that reusable resources cannot be handled in our standard description of actions in terms of preconditions and effects, because the amount of resource available is unchanged after the action is completed.! For this reason, we augment our representation to include a field of the form RESOURCE: R(k), which means that k units of resource R are required by the action. The resource requirement is both a prerequisite—the action cannot be performed if the resource is unavailable—and a temporary effect, in the sense that the availability of resource r is reduced by k for the duration of the action. Figure 12.3 shows how to extend the engine assembly problem to include three resources: an engine hoist for installing engines, a wheel station for putting on the wheels, and two inspectors. Figure 12.4 shows the solution with the fastest completion time, 115 minutes. This is longer than the 80 minutes required for a schedule without resource constraints. Notice that there is no time at which both inspectors are required, so we can immediately move one of our two inspectors to a more productive position.

The representation of resources as numerical quantities, such as Inspectors(2), rather than as named entities, such as Inspector(I1) and Inspector(I2), is an example of a very general technique called aggregation. The central idea of aggregation is to group individual objects into quantities when the objects are all indistinguishable with respect to the purpose at hand. In our assembly problem, it does not matter which inspector inspects the car, so there is no need to make the distinction. (The same idea works in the missionaries-and-cannibals problem in Exercise 3.9.) Aggregation is essential for reducing complexity. Consider what happens when a schedule is proposed that has 10 concurrent Inspect actions but only 9 inspectors are available. With inspectors represented as quantities, a failure is detected immediately and the algorithm backtracks to try another schedule. With inspectors represented as individuals, the algorithm backtracks to try all 10! ways of assigning inspectors to Inspect actions, to no avail.

Despite their advantages, resource constraints make scheduling problems more complicated by introducing additional interactions among actions. Whereas unconstrained scheduling using the critical-path method is easy, finding a resource-constrained schedule with the earliest possible completion time is NP-hard. This complexity is often seen in practice as well as in theory. A challenge problem posed in 1963—to find the optimal schedule for a

I In contrast, consumable resources, such as screws for assembling the engine, can be handled within the standard framework: see Exercise 12.2.

Section 12.1.

Time, Schedules, and Resources 421

MINIMUM SLACK

Init(Chassis(C1) A Chassis(Cs)

A Engine(En, C1,30) A Engine( Ey, Ca, 60)

A Wheels(W1, C1,30) A Wheels(Wa, C2, 15)

A EngineHoists(1) A WheelStations(1) A Inspectors(2)) Goal(Done(C1) A Done(C2))

Action( AddFEngine(e, c), PRECOND: Engine(e, c,d) A Chassis(c) A ~Engineln(c), EFFECT: Engineln(c) A Duration(d), RESOURCE: EngineHoists(1)) Action( Add Wheels(w, ¢), PRECOND: Wheels(w, c,d) A Chassis(c) A Engineln(c), EFFECT: WheelsOn(c) A Duration(d), RESOURCE: WheelStations(1)) Action (Inspect(c), PRECOND: Engineln(c) A WheelsOn(c), EFFECT: Done(c) A Duration(10), RESOURCE: Inspectors(1))

Figure 12.3 Job shop scheduling problem for assembling two cars, with resources. The available resources are one engine assembly station, one wheel assembly station, and two inspectors. The notation RESOURCE: means that the resource r is used during execution of an action, but becomes free again when the action is complete.

EngineHoists(1) | AddEngine1 2 AddEngne2 D

AddWheets? N Acowneeis2 N == 7 1] T 1 T I T

T r T 1 0 10 0 0 40 50 0 70 0 0 100 110 120

WheelStations{1)

Inspectors(2)

Figure 12.4 A solution to the job shop scheduling problem with resources from Figure 12.3. The left-hand margin lists the three resources, and actions are shown aligned horizontally with the resources they consume. There are two possible schedules, depending on which assembly uses the engine station first; we’ve shown the optimal solution, which

takes 115 minutes.

problem involving just 10 machines and 10 jobs of 100 actions each—went unsolved for 23 years (Lawler et al., 1993). Many approaches have been tried. including branch-and-bound, simulated annealing, tabu search, constraint satisfaction, and other techniques from Part II. One simple but popular heuristic is the minimum slack algorithm. It schedules actions in a greedy fashion. On each iteration, it considers the unscheduled actions that have had all their predecessors scheduled and schedules the one with the least slack for the earliest possible start. It then updates the ES and LS times for each affected action and repeats. The heuristic

422

Chapter 12. Planning and Acting in the Real World

is based on the same principle as the most-constrained-variable heuristic in constraint satisfaction. It often works well in practice, but for our assembly problem it yields a 130-minute solution, not the 115-minute solution of Figure 12.4.

The approach we have taken in this section is “plan first, schedule later”: that is, we divided the overall problem into a planning phase in which actions are selected and partially ordered to meet the goals of the problem, and a later scheduling phase, in which temporal information is added to the plan to ensure that it meets resource and deadline constraints. This approach is common in real-world manufacturing and logistical settings, where the planning phase is often performed by human experts. When there are severe resource constraints, however, it could be that some legal plans will lead to much better schedules than others. In that case, it makes sense to integrate planning and scheduling by taking into account durations and overlaps during the construction of a partial-order plan. Several of the planning algorithms in Chapter 11 can be augmented to handle this information. For example, partial-order planners can detect resource constraint violations in much the same way that they detect conflicts with causal links. Heuristics can be modified to estimate the total completion time of a plan, rather than just the total cost of the actions. This is currently an active area of research.

HIERARCHICAL DECOMPOSITION

HIERARCHICAL TASK NETWORK

ACTION DECOMPOSITION

PRIMITIVE ACTION

One of the most pervasive ideas for dealing with complexity is hierarchical decomposition. Complex software is created from a hierarchy of subroutines or object classes, armies operate as a hierarchy of units, governments and corporations have hierarchies of departments, subsidiaries, and branch offices. The key benefit of hierarchical structure is that, at each level of the hierarchy, a computational task, military mission, or administrative function is reduced to a small number of activities at the next lower level, so that the computational cost of finding the correct way to arrange those activities for the current problem is small. Nonhierarchical methods, on the other hand, reduce a task to a large number of individual actions; for largescale problems, this is completely impractical. In the best case—when high-level solutions always turn out to have satisfactory low-level implementations—hierarchical methods can result in linear-time instead of exponential-time planning algorithms.

This section describes a planning method based on hierarchical task networks or HTNs. The approach we take combines ideas from both partial-order planning (Section 11.3) and the area known as “HTN planning.” In HTN planning, the initial plan, which describes the problem, is viewed as a very high-level description of what is to be done—for example, building a house. Plans are refined by applying action decompositions. Each action decomposition reduces a high-level action to a partially ordered set of lower-level actions. Action decompositions, therefore, embody knowledge about how to implement actions. For example, building a house might be reduced to obtaining a permit, hiring a contractor, doing the construction, and paying the contractor. (Figure 12.5 shows such a decomposition.) The process continues until only primitive actions remain in the plan. Typically, the primitive actions will be actions that the agent can execute automatically. For a general contractor,

Section 12.2.

Hierarchical Task Network Planning 423

PLAN LIBRARY

EXTERNAL PRECONDITIONS

EXTERNAL EFFECT

PRIMARY EFFECT

SECONDARY EFFECT

“install landscaping” might be primitive because it simply involves calling the landscaping contractor. For the landscaping contractor, actions such as “plant rhododendrons here” might be considered primitive.

In “pure” HTN planning, plans are generated only by successive action decompositions. The HTN therefore views planning as a process of making an activity description more concrete, rather than (as in the case of state-space and partial-order planning) a process of constructing an activity description, starting from the empty activity. It turns out that every STRIPS action description can be turned into an action decomposition (see Exercise 12.6), and that partial-order planning can be viewed as a special case of pure HTN planning. For certain tasks, however—especially “novel” conjunctive goals—the pure HTN viewpoint is rather unnatural, and we prefer to take a hybrid approach in which action decompositions are used as plan refinements in partial-order planning, in addition to the standard operations of establishing an open condition and resolving conflicts by adding ordering constraints. (Viewing HTN planning as an extension of partial-order planning has the additional advantage that we can use the same notational conventions instead of introducing a whole new set.) We begin by describing action decomposition in more detail. Then we explain how the partial-order planning algorithm must be modified to handle decompositions, and finally we discuss issues of completeness, complexity, and practicality.

Representing action decompositions

General descriptions of action decomposition methods are stored in a plan library, from which they are extracted and instantiated to fit the needs of the plan being constructed. Each method is an expression of the form Decompose(a,d). This says that an action a can be decomposed into the plan d, which is represented as a partial-order plan, as described in Section 11.3.

Building a house is a nice, concrete example, so we will use it to illustrate the concept of action decomposition. Figure 12.5 depicts one possible decomposition of the BuildHouse action into four lower-level actions. Figure 12.6 shows some of the action descriptions for the domain, as well as the decomposition for BuildHouse as it would appear in the plan library. There might be other possible decompositions in the library.

The Start action of the decomposition supplies all those preconditions of actions in the plan that are not supplied by other actions. We call these the external preconditions. In our example, the external preconditions of the decomposition are Land and Money. Similarly, the external effects, which are the preconditions of Finish, are all those effects of actions in the plan that are not negated by other actions. In our example, the external effects of BuildHouse are House and ~Money. Some HTN planners also distinguish between primary effects, such as House, and secondary effects, such as ~Money. Only primary effects may be used to achieve goals, whereas both kinds of effects might cause conflicts with other actions; this can greatly reduce the search space.’

2 Itcould also prevent the discovery of unexpected plans. For example, a person facing bankruptcy proceedings can eliminate all liquid assets (i.e., achieve ~ Money) by buying or building a house. This plan is useful because current law precludes the seizure of a primary residence by creditors.

424

Chapter 12. Planning and Acting in the Real World

[]

Figure 12.5 One possible decomposition for the BuildHouse action.

Action(BuyLand, PRECOND: Money, EFFECT: Land A — Money)

Action(GetLoan, PRECOND: GoodCredit, EFFECT: Money A Mortgage) Action(BuildHouse, PRECOND: Land, EFFECT: House) ( (

Action( GetPermit, PRECOND: Land, EFFECT: Permit)

Action (Hire Builder, EFFECT: Contract)

Action( Construction, PRECOND: Permit A Contract, EFFECT: HouseBuilt A — Permit)

Action(PayBuilder, PRECOND: Money A HouseBuilt, EFFECT: = Money A House A — Contract)

Decompose( BuildHouse, Plan(STEPS: {S71 : GetPermit, Sy: HireBuilder, Sz: Construction, Sy: PayBuilder} ORDERINGS: {Start < S; < S3 < Sy < Finish, Start < So < S3}, LINKS: {Start L2"¢ S), Start Momev g,, S) Permit Ss, Ss Contract Ss, Ss HouseBuilt Sy, Sq louse Finish, Sy ~ MTV Finish}))

Figure 12.6 Action descriptions for the house-building problem and a detailed decomposition for the Build House action. The descriptions adopt a simplified view of money and an optimistic view of builders.

A decomposition should be a correct implementation of the action. A plan d implements an action a correctly if d is a complete and consistent partial-order plan for the problem of achieving the effects of a given the preconditions of a. Obviously, a decomposition will be correct if it is the result of running a sound partial-order planner.

A plan library could contain several decompositions for any given high-level action; for example, there might be another decomposition for Build House that describes a process whereby the agent builds a house from rocks and turf with its own bare hands. Each de

Section 12.2.

Hierarchical Task Network Planning 425

INTERNAL EFFECT

composition should be a correct plan, but it could have additional preconditions and effects beyond those stated in the high-level action description. For example, the decomposition for BuildHouse in Figure 12.5 requires Money in addition to Land and has the effect = Money. The self-build option, on the other hand, doesn’t require money, but does require a ready supply of Rocks and Turf, and could result in a BadBack.

Given that a high-level action, such as BuildHouse, may have several possible decompositions, it is inevitable that its STRIPS action description will hide some of the preconditions and effects of those decompositions. The preconditions of the high-level action should be the intersection of the external preconditions of its decompositions, and the effects should be the intersection of the external effects of the decompositions. Put another way, the high-level preconditions and effects are guaranteed to be a subset of the true preconditions and effects of every primitive implementation.

Two other forms of information hiding should be noted. First, the high-level description completely ignores all internal effects of the decompositions. For example, our BuildHouse decomposition has temporary internal effects Permit and Contract.® Second, the high-level description does not specify the intervals “inside” the activity during which the high-level preconditions and effects must hold. For example, the Land precondition needs to be true (in our very approximate model) only until GetPermit is performed, and House is true only after PayBuilder is performed.

Information hiding of this kind is essential if hierarchical planning is to reduce complexity; we need to be able to reason about high-level actions without worrying about myriad details of the implementations. There is, however, a price to pay. For example, conflicts might exist between internal conditions of one high-level action and internal actions of another, but there is no way to detect this from the high-level descriptions. This issue has significant implications for HTN planning algorithms. In a nutshell, whereas primitive actions can be treated as point events by the planning algorithm, high-level actions have temporal extent within which all sorts of things can be going on.

Modifying the planner for decompositions

We now show how to modify POP to incorporate HTN planning. We do that by modifying the POP successor function (page 390) to allow decomposition methods to be applied to the current partial plan P. The new successor plans are formed by first selecting some nonprimitive action a’ in P and then, for any Decompose(a, d) method from the plan library such that a and &’ unify with substitution 6, replacing a’ with d' = SUBST(#, d).

Figure 12.7 shows an example. At the top, there is a plan P for getting a house. The high-level action, a’ = BuildHouse, is selected for decomposition. The decomposition d is selected from Figure 12.5, and Build House is replaced by this decomposition, An additional step, GetLoan, is then introduced to resolve the new open condition, Money, that is created by the decomposition step. Replacing an action with its decomposition is a bit like transplant surgery: we have to take the new subplan out of its packaging (the Start and Finish steps),

3 Construction negates the Permit, otherwise the same permit could be used to build many houses, Unfortunately, Construction does not terminate the Contract because we have to PayBuilder first.

426

Chapter 12. Planning and Acting in the Real World

[]

SUBTASK SHARING

Figure 12.7 Decomposition of a high-level action within an existing plan. The BuildHouse action is replaced by the decomposition from Figure 12.5. The external precondition Land is supplied by the existing causal link from BuyLand. The external precondition Money remains open after the decomposition step, so we add a new action, GetLoan.

we need to choose an action to fill the role of s and add it to the plan. It can be either a new instantiation of s or an existing step s’ from P that unifies with s. For example, the decomposition of a Make Wine action might suggest that we BuyLand; possibly, we can use the same BuyLand action that we already have in the plan. We call this subtask sharing.

In Figure 12.7, there are no sharing opportunities, so new instances of the actions are created. Once the actions have been chosen, all the internal constraints from d’ are copied over—for example, that GetPermit is ordered before Construction and that there is a causal link between these two steps supplying the Permit precondition of Construction. This completes the task of replacing a’ with the instantiation of df.

steps in d’. First, consider an ordering constraint in P of the form B B be ordered with respect to the steps in d’? The most obvious solution is that B should come before every step in d’, and that can be achieved by replacing every constraint of the form Start < sin d’ with a constraint B < s. On the other hand, this approach might be too strict! For example, BuyLand has to come before BuildHouse, but there is no need for BuyLand to come before HireBuilder in the expanded plan. Imposing an overly strict ordering might prevent some solutions from being found. Therefore, the best solution is for each ordering constraint to record the reason for the constraint; then, when a high-level action is expanded, the new ordering constraints can be as relaxed as possible, consistent with the reason for the original constraint. Exactly the same considerations apply when we are replacing constraints of the form a’ < C. B be ordered with respect to the steps in d’? The most obvious solution is that B should come before every step in d’, and that can be achieved by replacing every constraint of the form Start < sin d’ with a constraint B < s. On the other hand, this approach might be too strict! For example, BuyLand has to come before BuildHouse, but there is no need for BuyLand to come before HireBuilder in the expanded plan. Imposing an overly strict ordering might prevent some solutions from being found. Therefore, the best solution is for each ordering constraint to record the reason for the constraint; then, when a high-level action is expanded, the new ordering constraints can be as relaxed as possible, consistent with the reason for the original constraint. Exactly the same considerations apply when we are replacing constraints of the form a’ < C.

Section 12.2.

Hierarchical Task Network Planning 427

3. The final step is to hook up causal links. If B \_, a’ was a causal link in the original plan, replace it by a set of causal links from B to all the steps in d’ with preconditions p that were supplied by the Start step in the decomposition d (i.e., to all the steps in d’ for which p is an external precondition). In the example, the causal link BuyLand “24 BuildHouse is replaced by the link BuyLand “®¢ Permit. (The Money precondition for PayBuilder in the decomposition becomes an open condition, because no action in the original plan supplies Money to BuildHouse.) Similarly, for each causal link a’ \_2, C in the plan, replace it with a set of causal links to C' from whichever step in d’ supplies p to the Finish step in the decomposition d (i.e., from the step in d’ that has p as an external effect). In the example, we replace the link BuildHouse f94$¢ Finish with the link PayBuilder 7°%¢ Finish.

This completes the additions required for generating decompositions in the context of the POP planner.

Additional modifications to the POP algorithm are required because of the fact that high-level actions Aide information about their final primitive implementations. In particular, the original POP algorithm backtracks with failure if the current plan contains an irresolvable conflict—that is, if an action conflicts with a causal link but cannot be ordered either before or after the link. (Figure 11.9 shows an example.) With high-level actions, on the other hand, apparently irresolvable conflicts can sometimes be resolved by decomposing the conflicting actions and interleaving their steps. An example is given in Figure 12.8. Thus, it may be the case that a complete and consistent primitive plan can be obtained by decomposition even when no complete and consistent high-level plan exists. This possibility means that a complete HTN planner must forgo many pruning opportunities that are available to a standard POP planner. Alternatively, we can prune anyway and hope that no solution is missed.

Discussion

Let’s begin with the bad news: pure HTN planning (where the only allowable plan refinement is decomposition) is undecidable, even though the underlying state space is finite! This might seem very depressing, since the point of HTN planning is to gain efficiency. The difficulty arises because action decompositions can be recursive—for example, going for a walk can be implemented by taking a step and then going for a walk—so HTN plans can be arbitrarily long. In particular, the shortest HTN solution could be arbitrarily long, so that there is no way to terminate the search after any fixed time. There are, however, at least three ways to look on the bright side:

1. We can rule out recursion, which very few domains require. In that case, all HTN plans are of finite length and can be enumerated.

2. We can bound the length of solutions we care about. Because the state space is finite, a plan that has more steps than there are states in the state space must include a loop that visits the same state twice. We would lose little by ruling out HTN solutions of this kind, and we would control the search.

4 There are some additional minor modifications required for handling conflict resolution with high-level actions; the interested reader can consult the papers cited at the end of the chapter.

428

Chapter 12. Planning and Acting in the Real World

[]

(c) Decomposition of (b) into a Consistent Solution

Figure 12.8 The Gift of the Magi problem, taken from the O. Henry story, shows an inconsistent abstract plan that nevertheless can be decomposed into a consistent solution, Part (a) shows the problem: A poor couple has only two prized possessions—he a gold watch and she her beautiful long hair. Each plans to buy a present to make the other happy. He decides to trade his watch to buy a silver comb for her hair, and she decides to sell her hair to get a gold chain for his watch. In (b) the partial plan is inconsistent, because there is no way to order the “Give Comb” and “Give Chain” abstract steps without a conflict. (We assume that the “Give Comb” action has the precondition Hair, because if the wife doesn’t have her long hair, the action won’t have the intended effect of making her happy, and similarly for the “Give Chain” action.) In (c) we decompose the “Give Comb” step with an “installment plan” method. In the first step of the decomposition, the husband takes possession of the comb and gives it to his wife, while agreeing to deliver the watch in payment at a later date, In the second step, the watch is handed over and the obligation is fulfilled. A similar method decomposes the “Give Chain” step. As long as both giving steps are ordered before the delivery steps, this decomposition solves the problem. (Note that it relies on the problem being defined so that the happiness of using the chain with the watch or the comb with the hair persists even after the possessions are surrendered.)

3. We can adopt the hybrid approach that combines POP and HTN planning. Partial-order planning by itself suffices to decide whether a plan exists, so the hybrid problem is clearly decidable.

We need to be a little bit careful with the third answer. POP can string together primitive actions in arbitrary ways, so we might find ourselves with solutions that are very hard to understand and do not have the nice, hierarchical organization of HTN plans. An appropriate compromise is to control the hybrid search so that action decompositions are preferred over adding new actions, although not to such an extent that arbitrarily long HTN plans are generated before any primitive actions can be added. One way to do this is to use a cost function

Section 12.2.

Hierarchical Task Network Planning 429

that gives a discount for actions introduced by decomposition; the larger the discount, the more the search will resemble pure HTN planning and the more hierarchical the solution will be. Hierarchical plans are usually much easier to execute in realistic settings, and are easier to fix when things go wrong.

Another important characteristic of HTN plans is the possibility of subtask sharing. Recall that subtask sharing means using the same action to implement two different steps in plan decompositions. If we disallow subtask sharing, then each instantiation of a decomposition d’ can be done in only one way, rather than many, thereby greatly pruning the search space. Usually, this pruning saves some time and at worst leads to a solution that is slightly longer than optimal. In some cases, however, it can be more problematic. For example, consider the goal “enjoy a honeymoon and raise a family.” The plan library might come up with “get married and go to Hawaii” for the first subgoal and “get married and have two children” for the second. Without subtask sharing, the plan will include two distinct marriage actions, often considered highly undesirable.

An interesting example of the costs and benefits of subtask sharing occurs in optimizing compilers. Consider the problem of compiling the expression tan(z) — sin(x). Most compilers accomplish this by merging two separate subroutine calls in a trivial way: all the steps of tan come before any of the steps of sin. But consider the following Taylor series approximations for sin and tan:

tang ~ x + 5 4 22 ial, sinz ar — 4 In LL An HTN planner with subtask sharing could generate a more efficient solution, because it could choose to implement many steps of the sin computation with existing steps from tan. Most compilers do not do this kind of interprocedural sharing because it would take too much time to consider all the possible shared plans. Instead, most compilers generate each subplan independently, and then perhaps modify the result with a peephole optimizer.

Given all the additional complications caused by the introduction of action decompositions, why do we believe that HTN planning can be efficient? The actual sources of complexity are hard to analyze in practice, so we consider an idealized case. Suppose, for example, that we want to construct a plan with n actions. For a nonhierarchical, forward state-space planner with b allowable actions at each state, the cost is O(b™). For an HTN planner, let us suppose a very regular decomposition structure: each nonprimitive action has d possible decompositions, each into k actions at the next lower level. We want to know how many different decomposition trees there are with this structure. Now, if there are n actions at the primitive level, then the number of levels below the root is logy, n2, so the number of internal decomposition nodes is 1 + k + k2 + --+ kl°8 "=! = (n — 1) /(k — 1). Each internal node has d possible decompositions, so there are d(n=1/(k=1) possible regular decomposition trees that could be constructed. Examining this formula, we see that keeping d small and % large can result in huge savings: essentially we are taking the kth root of the nonhierarchical cost, if b and d are comparable. On the other hand, constructing a plan library that has a small number of long decompositions, but nonetheless allows us to solve any problem, is not always possible. Another way of saying this is that long macros that are usable across a wide range of problems are extremely precious.

430

Chapter 12. Planning and Acting in the Real World

Another, and perhaps better, reason for believing that HTN planning is efficient is that it works in practice. Almost all planners for large-scale applications are HTN planners, because HTN planning allows the human expert to provide the crucial knowledge about how to perform complex tasks so that large plans can be constructed with little computational effort. For example, O-PLAN (Bell and Tate, 1985), which combines HTN planning with scheduling, has been used to develop production plans for Hitachi. A typical problem involves a product line of 350 different products, 35 assembly machines, and over 2000 different operations. The planner generates a 30-day schedule with three 8-hour shifts a day, involving millions of steps.

The key to HTN planning, then, is the construction of a plan library containing known methods for implementing complex, high-level actions. One method of constructing the 1ibrary is to learn the methods from problem-solving experience. After the excruciating experience of constructing a plan from scratch, the agent can save the plan in the library as a method for implementing the high-level action defined by the task. In this way, the agent can become more and more competent over time as new methods are built on top of old methods. One important aspect of this learning process is the ability to generalize the methods that are constructed, eliminating detail that is specific to the problem instance (e.g., the name of the builder or the address of the plot of land) and keeping just the key elements of the plan. Methods for achieving this kind of generalization are described in Chapter 19. It seems to us inconceivable that humans could be as competent as they are without some such mechanism.

BOUNDED INDETERMINACY

So far we have considered only classical planning domains that are fully observable, static, and deterministic. Furthermore, we have assumed that the action descriptions are correct and complete. In these circumstances, an agent can plan first and then execute the plan “with its eyes closed.” In an uncertain environment, on the other hand, an agent must use its percepts to discover what is happening while the plan is being executed and possibly modify or replace the plan if something unexpected happens.

Agents have to deal with both incomplete and incorrect information. Incompleteness arises because the world is partially observable, nondeterministic, or both. For example, the door to the office supply cabinet might or might not be locked; one of my keys might or might not open the door if it is locked; and I might or might not be aware of these kinds of incompleteness in my knowledge. Thus, my model of the world is weak, but correct. On the other hand, incorrectness arises because the world does not necessarily match my model of the world; for example, I might believe that my key opens the supply cabinet, but I could be wrong if the locks have been changed. Without the ability to handle incorrect information, an agent can end up being as unintelligent as the dung beetle (page 37), which attempts to plug up its nest with a ball of dung even after the ball has been removed from its grasp.

The possibility of having complete or correct knowledge depends on how much indeterminacy there is in the world. With bounded indeterminacy, actions can have unpredictable

Planning and Acting in Nondeterministic Domains 431

UNBOUNDED INDETERMINACY

SENSORLESS PLANNING

CONDITIONAL PLANNING

SENSING ACTIONS

EXECUTION MONITORING AND REPLANNING

CONTINUOUS PLANNING

effects, but the possible effects can be listed in the action description axioms. For example, when we flip a coin, it is reasonable to say that the outcome will be Heads or Tails. An agent can cope with bounded indeterminacy by making plans that work in all possible circumstances. With unbounded indeterminacy, on the other hand, the set of possible preconditions or effects either is unknown or is too large to be enumerated completely. This would be the case in very complex or dynamic domains such as driving, economic planning, and military strategy. An agent can cope with unbounded indeterminacy only if it is prepared to revise its plans and/or its knowledge base. Unbounded indeterminacy is closely related to the qualification problem discussed in Chapter 10—the impossibility of listing all the preconditions required for a real-world action to have its intended effect.

There are four planning methods for handling indeterminacy. The first two are suitable for bounded indeterminacy, and the second two for unbounded indeterminacy:

¢& Sensorless planning: Also called conformant planning, this method constructs standard, sequential plans that are to be executed without perception. The sensorless planning algorithm must ensure that the plan achieves the goal in all possible circumstances, regardless of the true initial state and the actual action outcomes. Sensorless planning relies on coercion—the idea that the world can be forced into a given state even when the agent has only partial information about the current state. Coercion is not always possible, so sensorless planning is often inapplicable. Sensorless problem solving, involving search in belief state space, was described in Chapter 3.

{> Conditional planning: Also known as contingency planning, this approach deals with bounded indeterminacy by constructing a conditional plan with different branches for the different contingencies that could arise. Just as in classical planning, the agent plans first and then executes the plan that was produced. The agent finds out which part of the plan to execute by including sensing actions in the plan to test for the appropriate conditions. In the air transport domain, for example, we could have plans that say “check whether SFO airport is operational. If so, fly there; otherwise, fly to Oakland.” Conditional planning is covered in Section 12.4.

{> Execution monitoring and replanning: In this approach, the agent can use any of the preceding planning techniques (classical, sensorless, or conditional) to construct a plan, but it also uses execution monitoring to judge whether the plan has a provision for the actual current situation or need to be revised. Replanning occurs when something goes wrong. In this way, the agent can handle unbounded indeterminacy. For example, even if a replanning agent did not envision the possibility of SFO’s being closed, it can recognize that situation when it occurs and call the planner again to find a new path to the goal. Replanning agents are covered in Section 12.5.

¢$ Continuous planning: All the planners we have seen so far are designed to achieve a goal and then stop. A continuous planner is designed to persist over a lifetime. It can handle unexpected circumstances in the environment, even if these occur while the agent is in the middle of constructing a plan. It can also handle the abandonment of goals and the creation of additional goals by goal formulation. Continuous planning is described in Section 12.6.

432

Chapter 12. Planning and Acting in the Real World

Let’s consider an example to clarify the differences among the various kinds of agents. The problem is this: given an initial state with a chair, a table, and some cans of paint, with everything of unknown color, achieve the state where the chair and table have the same color.

A classical planning agent could not handle this problem, because the initial state is not fully specified—we don’t know what color the furniture is.

A sensorless planning agent must find a plan that works without requiring any sensors during plan execution. The solution is to open any can of paint and apply it to both chair and table, thus coercing them to be the same color (even though the agent doesn’t know what the color is). Coercion is most appropriate when propositions are expensive or impossible to perceive. For example, doctors often prescribe a broad-spectrum antibiotic rather than using the conditional plan of doing a blood test, then waiting for the results to come back, and then prescribing a more specific antibiotic. They do this because the delays and costs involved in performing the blood tests are usually too great.

A conditional planning agent can generate a better plan: first sense the color of the table and chair; if they are already the same then the plan is done. If not, sense the labels on the paint cans; if there is a can that is the same color as one piece of furniture, then apply the paint to the other piece. Otherwise paint both pieces with any color.

A replanning agent could generate the same plan as the conditional planner, or it could generate fewer branches at first and fill in the others at execution time as needed. It could also deal with incorrectness of its action descriptions. For example, suppose that the Paint(obj, color) action is believed to have the deterministic effect Color (obj, color). A conditional planner would just assume that the effect has occurred once the action has been executed, but a replanning agent would check for the effect, and if it were not true (perhaps because the agent was careless and missed a spot), it could then replan to repaint the spot. We will return to this example on page 441.

A continuous planning agent, in addition to handling unexpected events, can revise its plans appropriately if, say, we add the goal of having dinner on the table, so that the painting plan must be postponed.

In the real world, agents use a combination of approaches. Car manufacturers sell spare tires and air bags, which are physical embodiments of conditional plan branches designed to handle punctures or crashes; on the other hand, most car drivers never consider these possibilities, so they respond to punctures and crashes as replanning agents. In general, agents create conditional plans only for those contingencies that have important consequences and a nonnegligible chance of going wrong. Thus, a car driver contemplating a trip across the Sahara desert might do well to consider explicitly the possibility of breakdowns, whereas a trip to the supermarket requires less advance planning.

The agents we describe in this chapter are designed to handle indeterminacy, but are not capable of making tradeoffs between the probability of success and the cost of plan construction. Chapter 16 provides additional tools for dealing with these issues.

Section 12.4. Conditional Planning 433

12.4 CONDITIONAL PLANNING

Conditional planning is a way to deal with uncertainty by checking what is actually happening in the environment at predetermined points in the plan. Conditional planning is simplest to explain for fully observable environments, so we will begin with that case. The partially observable case is more difficult, but more interesting.

Conditional planning in fully observable environments

Full observability means that the agent always knows the current state. If the environment is nondeterministic, however, the agent will not be able to predict the outcome of its actions. A conditional planning agent handles nondeterminism by building into the plan (at planning time) conditional steps that will check the state of the environment (at execution time) to decide what to do next. The problem, then, is how to construct these conditional plans.

We will use as our example domain the venerable vacuum world, whose state space, for the deterministic case, is laid out on page 65. Recall that the available actions are Left, Right, and Suck. We will need some propositions to define the states: let AtL (AtR) be true if the agent is in the left (right) state,® and let Clean (CleanR) be true if the left (right) state is clean. The first thing we need to do is augment the STRIPS language to allow for

DISANCTIVE nondeterminism. To do this, we allow actions to have disjunctive effects, meaning that the action can have two or more different outcomes whenever it is executed. For example, suppose that moving Left sometimes fails. Then the normal action description

Action(Left, PRECOND: AtR, EFFECT: AtL A — AtR) must be modified to include a disjunctive effect: Action(Left, PRECOND: AtR, EFFECT: AtL V AtR) . (12.1)

CONDITIONAL We will also find it useful to allow actions to have conditional effects, wherein the effect of the action depends on the state in which it is executed. Conditional effects appear in the EFFECT slot of an action, and have the syntax “when : " For example, to model the Suck action, we would write

Action(Suck, PRECOND:, EFFECT: (when AtL: CleanL) A (when AtR: CleanR))].

Conditional effects do not introduce indeterminacy, but they can help to model it. For example, suppose we have a devious vacuum cleaner that sometimes dumps dirt on the destination square when it moves, but only if that square is clean. This can be modeled with a description such as

Action (Left, PRECOND: AtR, EFFECT: AtL V (AtL A when CleanL: —CleanL)), which is both disjunctive and conditional.® To create conditional plans, we need conditional

conomonaLsTErs steps. We will write these using the syntax “if then plan\_A else plan\_B,” where

5 Obviously, AtR is true iff ~AtL is true, and vice versa. We use two propositions mainly to improve readability. 6 The conditional effect when CleanL: — Clean may look a little odd. Remember, however, that here Clean refers to the situation before the action and —~ Clean refers to the situation after the action.

434

Chapter 12. Planning and Acting in the Real World

GAMES AGAINST NATURE

is a Boolean function of the state variables. For example, a conditional step for the vacuum world might be, “if A¢L A Clean then Right else Suck.” The execution of such a step proceeds in the obvious way. By nesting conditional steps, plans become trees.

We want conditional plans that work regardless of which action outcomes actually occur. We have seen this problem before, in a different guise. In two-player games (Chapter 6), we want moves that will win regardless of which moves the opponent makes. For this reason, nondeterministic planning problems are often called games against nature.

Let us consider a specific example in the vacuum world. The initial state has the robot in the right square of a clean world; because the environment is fully observable, the agent knows the full state description, AtR A Clean A CleanR. The goal state has the robot in the left square of a clean world. This would be quite trivial, were it not for the “double Murphy” vacuum cleaner that sometimes deposits dirt when it moves to a clean destination square and sometimes deposits dirt if Suck is applied to a clean square.

A “game tree” for this environment is shown in Figure 12.9. Actions are taken by the robot in the “state” nodes of the tree, and nature decides what the outcome will be at the “chance” nodes, shown as circles. A solution is a subtree that (1) has a goal node at every leaf, (2) specifies one action at each of its “state” nodes, and (3) includes every outcome branch at each of its “chance” nodes. The solution is shown in bold lines in the figure; it corresponds to the plan [Left,if AtL A CleanL A CleanR then [] else Suck]. (For now, because we are using a state-space planner, the tests in conditional steps will be complete state descriptions.)

For exact solutions of games, we use the minimax algorithm (Figure 6.3). For conditional planning, there are typically two modifications. First, MAX and MIN nodes can become

[]

Figure 12.9 The first two levels of the search tree for the “double Murphy” vacuum world. State nodes are OR nodes where some action must be chosen. Chance nodes, shown as circles, are AND nodes where every outcome must be handled, as indicated by the arc linking the outgoing branches. The solution is shown in bold lines.

Section 12.4.

Conditional Planning 435

function AND-OR-GRAPH-SEARCH(problem) returns a conditional plan, or failure OR-SEARCH(INITIAL-STATE[ problem], problem, [])

function OR-SEARCH(state, problem, path) returns a conditional plan, or failure if GOAL-TEST[ problem](state) then return the empty plan if state is on path then return failure for each action, state\_set in SUCCESSORS] problem](state) do plan — AND-SEARCH(state-set, problem, [state | path]) if plan # failure then return [action | plan] return failure

function AND-SEARCH(state\_set, problem, path) returns a conditional plan, or failure for each s; in state\_set do plan; — OR-SEARCH(s;, problem, path) if plan = failure then return failure return [if s; then plan, elseif s; then plan, else . ..if s,\_; then plan, \_, else plan,]

Figure 12.10 ~~ An algorithm for searching AND—OR graphs generated by nondeterministic environments. We assume that SUCCESSORS returns a list of actions, each associated with a set of possible outcomes. The aim is to find a conditional plan that reaches a goal state in all circumstances.

OR and AND nodes. Intuitively, the plan needs to take some action at every state it reaches, but must handle every outcome for the action it takes. Second, the algorithm needs to return a conditional plan rather than just a single move. At an OR node, the plan is just the action selected, followed by whatever comes next. At an AND node, the plan is a nested series of if— then—else steps specifying subplans for each outcome; the tests in these steps are the complete state descriptions.’

Formally speaking, the search space we have defined is an AND—OR graph. In Chapter 7, AND—OR graphs showed up in propositional Horn clause inference. Here, the branches are actions rather than logical inference steps, but the algorithm is the same. Figure 12.10 gives a recursive, depth-first algorithm for AND—OR graph search.

One key aspect of the algorithm is the way in which it deals with cycles, which often arise in nondeterministic planning problems (e.g., if an action sometimes has no effect, or if an unintended effect can be corrected). If the current state is identical to a state on the path from the root, then it returns with failure. This doesn’t mean that there is no solution from the current state; it simply means that if there is a noncyclic solution, it must be reachable from the earlier incarnation of the current state, so the new incarnation can be discarded. With this check, we ensure that the algorithm terminates in every finite state space, because every path must reach a goal, a dead end, or a repeated state. Notice that the algorithm does not check whether the current state is a repetition of a state on some other path from the root. Exercise 12.15 investigates this issue.

7 Such plans could also be written using a case construct.

436

Chapter 12, Planning and Acting in the Real World

CYCLIC SOLUTION LABEL

GOAL

Figure 12.11 The first level of the search graph for the “triple Murphy” vacuum world, where we have shown cycles explicitly. All solutions for this problem are cyclic plans.

The plans returned by AND-OR-GRAPH-SEARCH contain conditional steps that test the entire state description to decide on a branch. In many cases, we can get away with less exhaustive tests. For example, the solution plan in Figure 12.9 could be written simply as [Left, if Clean then [] else Suck]. This is because the single test, Clean, suffices to divide the states at the AND-node into two singleton sets, so that after the test the agent knows exactly what state it is in. In fact, a series of if-then—else tests of single variables always suffices to divide a set of states into singletons, provided that the state is fully observable. We could, therefore, restrict the tests to be of single variables without loss of generality.

There is one final complication that often arises in nondeterministic domains: things don’t always work the first time, and one has to try again. For example, consider the “triple Murphy” vacuum cleaner, which (in addition to its previously stated habits) sometimes fails to move when commanded—for example, Left can have the disjunctive effect AtL Vv AtR, as in Equation (12.1). Now the plan [Left, if Clean then [] else Suck] is no longer guaranteed to work. Figure 12.11 shows part of the the search graph; clearly, there are no longer any acyclic solutions, and AND-OR-GRAPH-SEARCH would return with failure. There is, however, a cyclic solution, which is to keep trying Left until it works. We can express this solution by adding a label to denote some portion of the plan and using that label later instead of repeating the plan itself. Thus, our cyclic solution is

[Ly : Left,if AtR then L, else if Clean then [] else Suck] .

(A better syntax for the looping part of this plan would be “while AtR do Left”) The modifications needed to AND-OR-GRAPH-SEARCH are covered in Exercise 12.16. The key realization is that a loop in the state space back to a state L translates to a loop in the plan back to the point where the subplan for state L is executed.

Now we have the ability to synthesize complex plans that look more like programs, with conditionals and loops. Unfortunately, these loops are, potentially, infinite loops. For example, nothing in the action representation for the triple Murphy world says that Left will eventually succeed. Cyclic plans are therefore less desirable than acyclic plans, but they may be considered solutions, provided that every leaf is a goal state and a leaf is reachable from every point in the plan.

Section 12.4.

Conditional Planning 437

Conditional planning in partially observable environments

The preceding section dealt with fully observable environments, which have the advantage that conditional tests can ask any question at all and be sure of getting an answer. In the real world, partial observability is much more common. In the initial state of a partially observable planning problem, the agent knows only a certain amount about the actual state. The simplest way to model this situation is to say that the initial state belongs to a state set; the state set is a way of describing the agents initial belief state.

Suppose that a vacuum-world agent knows that it is in the right-hand square and that the square is clean, but it cannot sense the presence or absence of dirt in other squares. Then as far as it knows it could be in one of two states: the left-hand square might be either clean or dirty. This belief state is marked A in Figure 12.12. The figure shows part of the ANDOR graph for the “alternate double Murphy” vacuum world, in which dirt can sometimes be left behind when the agent leaves a clean square.’ If the world were fully observable, the agent could construct a cyclic solution of the form “Keep moving left and right, sucking up dirt whenever it appears, until both squares are clean and I’m in the left square.” (See Exercise 12.16.) Unfortunately, with local dirt sensing, this plan is unexecutable, because the truth value of the test “both squares are clean” cannot be determined.

Let us look at how the AND—OR graph is constructed. From belief state A, we show the outcome of moving Left. (The other actions make no sense.) Because the agent can leave dirt behind, the two possible initial worlds become four possible worlds, as shown in B and C. The worlds form two distinct belief states, classified by the available sensor information. 10 B, the agent knows CleanL; in C' it knows — Clean. From C, cleaning up the dirt moves the agent to B. From B, moving Right might or might not leave dirt behind, so there are again four possible worlds, divided according to the agent’s knowledge of CleanR (back to A) or = CleanR (belief state D).

In sum, nondeterministic, partially observable environments give us an AND—OR graph of belief states. Conditional plans can be found, therefore, using exactly the same algorithm as in the fully observable case, namely AND-OR-GRAPH-SEARCH. Another way to understand what’s going on is to see that the agent’s belief state is always fully observable—it always knows what it knows. “Standard” fully observable problem solving is just a special case in which every belief state is a singleton set containing exactly one physical state.

Are we done? Not quite! We still need to decide how belief states should be represented, how sensing works, and how action descriptions should be written in this new setting.

There are three basic choices for belief states:

1. Sets of full state descriptions. For example, the initial belief state in Figure 12.12 is { (AtR A CleanR A CleanL), (AtR A CleanR A = Clean) } . This representation is simple to work with, but very expensive: if there are n Boolean

8 These concepts are introduced in Section 3.6, which the reader might wish to consult before proceeding.

9 Parents with young children will be familiar with this phenomenon. Apologies to others, as usual.

10 Notice that they are not classified by whether there is dirt left behind when the agent moves. Branching in belief-state space is caused by alternative knowledge outcomes, not alternative physical outcomes.

438

Chapter 12. Planning and Acting in the Real World

[]

=] = =) 8 4] ox Left CleanL — Clean B [= o[<4] || a0 Suck Right CleanR —CleanR =) : =) 2| og js | 6 oof Figure 12.12 Part of the AND—OR graph for the “alternate double Murphy” vacuum world, in which dirt can sometimes be left behind when the agent leaves a clean square. The agent cannot sense dirt in other squares.

propositions defining the state, then a belief state can contain O(2") physical state descriptions, each of size O(n). Exponentially large belief states will occur whenever the agent knows only a fraction of the propositions—the less it knows, the more possible states it might be in.

example, the initial state can be written as AtR A CleanR .

Clearly, every belief state can be captured exactly by a single logical sentence; if we have to, we can use the disjunction of all the conjunctive state descriptions, but our example shows that more compact sentences could exist.

One drawback with general logical sentences is that, because there are many different, logically equivalent sentences that describe the same belief state, repeated state checking in the graph search algorithm can require general theorem proving. For this

Section 12.4, Conditional Planning 439

reason, we would like a canonical representation for sentences in which every belief state corresponds to exactly one sentence.!! One such representation uses a conjunction of literals ordered by proposition name—AtR A CleanR is an example. This is just the standard state representation under the open-world assumption from Chapter 11. Not all logical sentences can be written in such form—for example, there is no way to represent AtL V Clean B—but many domains can be handled.

3. Knowledge propositions describing the agent’s knowledge. (See also Section 7.7.) For the initial state, we have

K(AtR) AN K(CleanR) .

Here, K stands for “knows that” and K (P) means that the agent knows that P is true.!? With knowledge propositions, we use the closed-world assumption—if a knowledge proposition does not appear in the list, it is assumed false. For example, ~K ( Cleanl) and ~K (—~CleanL) are implicit in the sentence above, so it captures the fact that the agent is ignorant of the truth value of CleanlL.

It turns out that the second and third options are roughly equivalent, but we will use the third option, knowledge propositions, because it gives a more vivid description of sensing and because we already know how to write STRIPS descriptions with the closed-world assumption. In both options, each proposition symbol can appear in one of three ways: positive, negative, or unknown. Therefore, there are exactly 3" possible belief states that can be described this way. Now, the set of belief states is the powerset (set of all subsets) of the set of physical states. There are 2” physical states, so there are 22" belief states—far more than 3", so options 2 and 3 are quite restricted as representations of belief states. This currently is believed to be inevitable, because any scheme capable of representing every possible belief state will require O(log,(22")) = O(2™) bits to represent each one in the worst case. Our simple schemes require only O(n) bits to represent each belief state, trading expressiveness for compactness. In particular, if an action occurs, one of whose preconditions is unknown, then the resulting belief state will not be exactly representable and the action outcome becomes unknown. Now we need to decide how sensing works. There are two choices here. We can auTomATic sensing have automatic sensing, which means that at every time step the agent gets all the available percepts. The example in Figure 12.12 assumes automatic sensing of location and local dirt. ACTIVE SENSING Alternatively, we can insist on active sensing, which means that percepts are obtained only sensoRy AcTIoNs by executing specific sensory actions such as CheckDirt and CheckLocation. We will treat each kind of sensing in turn.

Let us now write an action description using knowledge propositions. Suppose the agent moves Left in the alternate-double-Murphy world with automatic local dirt sensing; according to the rules for that world, the agent might or might not leave dirt behind if the square was clean. As a physical effect, this would be disjunctive; but as a knowledge effect, it

11 The best-known canonical representation for a general propositional sentence is the binary decision diagram, or BDD (Bryant, 1992).

12 This is the same notation used for circuit-based agents in Chapter 7. Some authors use it to mean “knows whether P is true.” Translating between the two representations is straightforward.

440

Chapter 12. Planning and Acting in the Real World

simply deletes the agent’s knowledge of CleanR. The agent will also know whether Clean is true, one way or the other, because of local dirt sensing, and it will know that it is A¢L:

Action(Left, PRECOND: AtR, EFFECT: K (AtL) A ~K(AtR) A when CleanR: —K (CleanR) A when Cleanl: K(CleanL) A when —CleanL: K(—CleanL)) .

Notice that the preconditions and when conditions are plain propositions, not knowledge propositions. This is as it should be, because the outcomes of actions do depend on the actual world, but how do we check the truth of those conditions when all we have is the belief state? If the agent knows a proposition, say K(AtR), in the current belief state, then the proposition must be true in the current physical state, and indeed the action is applicable. If the agent doesn’t know a proposition—for example, the when condition Cleanl—then the belief state must include worlds in which Clean is true and worlds in which Cleanl. is false. It is this that gives rise to multiple belief states resulting from the action. Thus, if the initial state is (K (AtR) A K(CleanR)), then after the move Left, the two outcome belief states are (K(AtL) AN K(CleanL)) and (K(AtL) A K(~CleanL)). In both cases, the truth value of Cleanl is known, so the Cleanl test can be used in the plan.

With active sensing (as opposed to automatic sensing), the agent gets new percepts only by asking for them. Thus, after moving Left, the agent will not know whether the left-hand square is dirty, so the last two conditional effects no longer appear in the action description in Equation (12.2). To find out whether the square is dirty, the agent can CheckDirt:

Action(CheckDirt, EFFECT:when AtL A CleanL: K(CleanL) N when AtL A —~CleanL: K(—CleanL) A when AtR A CleanR: K(CleanR) A when AtR A ~CleanR: K(—CleanR)) .

It is easy to show that Left followed by CheckDirt in the active sensing setting results in the same two belief states as Left did in the automatic sensing setting. With active sensing, it is always the case that physical actions map a belief state into a single successor belief state. Multiple belief states can be introduced only by sensory actions, which provide specific knowledge and hence allow conditional tests to be used in plans.

We have described a general approach to conditional planning based on state-space AND-OR search. The approach has proved to be quite effective on some test problems, but other problems are intractable. Theoretically, it can be shown that conditional planning belongs to a harder complexity class than classical planning. Recall that the definition of the class NP is that a candidate solution can be checked to see whether it really is a solution in polynomial time. This is true for classical plans (at least, for those of polynomial size) so the problem of classical planning is in NP. But in conditional planning a candidate must be checked to see whether, for all possible states, there exists some path through the plan that satisfies the goal. Checking the “all/some” combination cannot be done in polynomial time, so conditional planning is harder than NP. The only way out is to ignore some of the possible contingencies during the planning phase and to handle them only when they actually occur. This is the approach we pursue in the next section.

(12.2)

(12.3)

Section 12.5. Execution Monitoring and Replanning 441

12.5 EXECUTION MONITORING AND REPLANNING

An execution monitoring agent checks its percepts to see whether everything is going according to plan. Murphy's law tells us that even the best-laid plans of mice, men, and conditional planning agents frequently fail. The problem is unbounded indeterminacy—some unanticipated circumstance will always arise for which the agent’s actions descriptions are incorrect. Therefore, execution monitoring is a necessity in realistic environments. We will consider two kinds of execution monitoring: a simple, but weak form called action monitoring, whereby the agent checks the environment to verify that the next action will work, and a more complex but more effective form called plan monitoring, in which the agent verifies the entire remaining plan.

A replanning agent knows what to do when something unexpected happens: call a planner again to come up with a new plan to reach the goal. To avoid spending too much time planning, this is usually done by trying to repair the old plan—to find a way from the current unexpected state back onto the plan.

As an example, let us return to the double Murphy vacuum world in Figure 12.9. In this world, moving into a clean square sometimes deposits dirt in that square; but what if the agent doesn’t know that or doesn’t worry about it? Then it will come up with a very simple solution: [Left]. If no dirt is dumped on arrival when the plan is actually executed, then the agent will detect the achievement of the goal. Otherwise, because the Clean precondition of the implicit Finish step is not satisfied, the agent will generate a new plan: [Suck]. Execution of this plan always succeeds.

Together, execution monitoring and replanning form a general strategy that can be applied to both fully and partially observable environments, and to a variety of planning representations including state-space, partial-order, and conditional plans. One simple approach to state-space planning is shown in Figure 12.13. The planning agent starts with a goal and creates an initial plan to achieve it. The agent then starts executing actions one by one. The replanning agent, unlike our other planning agents, keeps track of both the remaining unexecuted plan segment plan and the complete original plan whole\_plan. It uses action

acTionmonToRING monitoring: before carrying out the next action of plan, the agent examines its percepts to see whether any preconditions of the plan have unexpectedly become unsatisfied. If they have, the agent will try to get back on track by replanning a sequence of actions that should take it back to some point in the whole\_plan.

Figure 12.14 provides a schematic illustration of the process. The replanner notices that the preconditions of the first action in plan are not satisfied by the current state. It then calls the planner to come up with a new subplan called repair that will get from the current situation to some state s on whole\_plan. In this example, the state s happens to be one step back from the current remaining plan. (That is why we keep track of the whole plan, rather than just the remaining plan.) In general, we choose s to be as close as possible to the current state. The concatenation of repair and the portion of whole\_plan from s onward, which we call continuation, makes up the new plan, and the agent is ready to resume execution.

442

function REPLANNING-AGENT( percept) returns an action static: KB, a knowledge base (includes action descriptions) plan, a plan, initially [] whole\_plan, a plan, initially [] goal, a goal

TELL(KB, MAKE-PERCEPT-SENTENCE( percept, t))

current «— STATE-DESCRIPTION(KB, t)

if plan = [] then whole\_plan — plan + PLANNER( current, goal, KB)

if PRECONDITIONS( FIRST( plan)) not currently true in KB then candidates — SORT(whole\_plan, ordered by distance to current) find state s in candidates such that

failure # repair «— PLANNER( current, s, KB)

continuation + the tail of whole\_plan starting at s whole\_plan — plan «— APPEND(repair, continuation)

return Pop( plan)

Figure 12.13 An agent that does action monitoring and replanning. It uses a complete state-space planning algorithm called PLANNER as a subroutine. If the preconditions of the next action are not met, the agent loops through the possible points p in whole\_plan, trying to find one that PLANNER can plan a path to. This path is called repair. If PLANNER succeeds in finding a repair, the agent appends repair and the tail of the plan after p, to create the new plan. The agent then returns the first step in the plan.

Figure 12.14 Before execution, the planner comes up with a plan, here called whole\_plan, to get from S to GG. The agent executes the plan until the point marked FE. Before executing the remaining plan, it checks preconditions as usual and finds that it is actually in state O rather than state FE. It then calls its planning algorithm to come up with repair, which is a plan to get from O to some point P on the original whole\_plan. The new plan now becomes the concatenation of repair and continuation (the resumption of the original whole\_plan).

Section 12.5.

Execution Monitoring and Replanning 443

PLAN MONITORING

Now let’s return to the example problem of achieving a chair and table of matching color, this time via replanning. We’ll assume a fully observable environment. In the initial state the chair is blue, the table is green, and there is a can of blue paint and a can of red paint. That gives us the following problem definition:

Init( Color( Chair, Blue) A Color( Table, Green) A ContainsColor (BC, Blue) A PaintCan(BC)) A ContainsColor (RC, Red) A PaintCan(RC') Goal(Color( Chair, x) A Color( Table, x)) Action (Paint (object, color), PRECOND: HavePaint( color) EFFECT: Color( object, color)) Action(Open(can), PRECOND: PaintCan(can) A ContainsColor (can, color) EFFECT: Have Paint (color) The agent’s PLANNER should come up with the following plan: [Start; Open(BC); Paint( Table, Blue); Finish] Now the agent is ready to execute the plan. Assume that all goes well as the agent opens the blue paint and applies it to the table. The agents from previous sections would declare victory at this point, having completed the steps in the plan. But the execution monitoring agent must first check the precondition of the Finish step, which says that the two pieces must have the same color. Suppose the agent perceives that they do not have the same color, because it missed a spot of green on the table. The agent then needs to figure out a position in whole\_plan to aim for and a repair action sequence to get there. The agent notices that the current state is identical to the precondition before the Paint action, so the agent chooses the empty sequence for repair and makes its plan be the same | Paint, Finish] sequence that it just attempted. With this new plan in place, execution monitoring resumes, and the Paint action is retried. This behavior will loop until the table is perceived to be completely painted. But notice that the loop is created by a process of plan—execute-replan, rather than by an explicit loop in a plan.

Action monitoring is a very simple method of execution monitoring but it can sometimes lead to less than intelligent behavior. For example, suppose that the agent constructs a plan to solve the painting problem by painting the chair and table red. Then it opens the can of red paint and finds that there is only enough paint for the chair. Action monitoring would not detect failure until after the chair has been painted, at which point Have Paint( Red) becomes false. What we really need to do is detect failure whenever the state is such that the remaining plan no longer works. Plan monitoring achieves this by checking the preconditions for success of the entire remaining plan—that is, the preconditions of each step in the plan, except those preconditions that are achieved by another step in the remaining plan. Plan monitoring cuts off execution of a doomed plan as soon as possible, rather than continuing until the failure actually occurs.!® In some cases, it can rescue the agent from disaster when the doomed plan would have led to a dead end from which the goal would be unachievable.

13 Plan monitoring makes our agent smarter than a dung beetle. (See page 37.) Our agent would notice that the

444

Chapter 12. Planning and Acting in the Real World

It is relatively straightforward to modify a planning algorithm so that it annotates the plan at each point with the preconditions for success of the remaining plan. If we extend plan monitoring to check whether the current state satisfies the plan preconditions at any future point, rather than just the current point, then plan monitoring will also be able to take advantage of serendipity—that is, accidental success. If someone comes along and paints the table red at the same time that the agent is painting the chair red, then the final plan preconditions are satisfied (the goal has been achieved), and the agent can go home early.

So far, we have described monitoring and replanning in fully observable environments. Things can become much more complicated when the environment is partially observable. First, things can go wrong without the agent’s being able to detect it. Second, “checking preconditions” could require the execution of sensing actions, which have to be planned for— either at planning time, which takes us back to conditional planning, or at execution time. In the worst case, the execution of a sensing action could require a complex plan that itself requires monitoring and hence further sensing actions, and so on. If the agent insists on checking every precondition, it might never get around to actually doing anything. The agent should prefer to check those variables that are important, have a good chance of going wrong, and are not too expensive to perceive. This allows the agent to respond appropriately to important threats, but not waste time checking to see whether the sky is falling.

Now that we have described a method for monitoring and replanning, we need to ask, “Does it work?” This is a surprisingly tricky question. If we mean, “Can we guarantee that the agent will always achieve the goal, even with unbounded indeterminacy?” then the answer is no, because the agent could inadvertently arrive at a dead end, as described for online search in Section 4.5. For example, the vacuum agent might not know that its batteries can run out. Let’s rule out dead ends; that is, let’s assume that the agent can construct a plan to reach the goal from any state in the environment. If we assume that the environment is really nondeterministic, in the sense that such a plan always has some chance of success on any given execution attempt, then the agent will eventually reach the goal. The replanning agent therefore has capabilities analogous to those of the conditional planning agent. In fact, we can modify a conditional planner so that it constructs only a partial solution plan that includes steps of the form “if then plan\_A else replan.” Under the assumptions we have made, such a plan can be a correct solution to the original problem; it might also be much cheaper to construct than a full conditional plan.

Trouble occurs when the agent’s repeated attempts to reach the goal are futile—when they are blocked by some precondition or effect that it doesn’t know about. For example, if the agent has the wrong card key to its hotel room, no amount of inserting and removing it is going to open the door.'\* One solution is to choose randomly from among the set of possible repair plans, rather than trying the same one each time. In this case, the repair plan of going to the front desk and getting a card key to the room would be a useful alternative. Given that the agent might not be able to distinguish between the truly nondeterministic case and the futile case, some variation in repairs is a good idea in general.

dung ball was missing from its grasp and would replan to get another ball and plug its hole. 14 Futile repetition of a plan repair is exactly the behavior exhibited by the sphex wasp. (See page 37.)

Section 12.6.

Continuous Planning 445

Another solution to the problem of incorrect action descriptions is learning. After a few tries, a learning agent should be able to modify the action description that says that the key opens the door. At that point, the replanner will automatically come up with an alternative plan, such as getting a new key. This kind of learning is described in Chapter 21.

Even with all these potential improvements, the replanning agent still has a few shortcomings. It cannot perform in real-time environments, and there is no bound on the amount of time it will spend replanning and thus no bound on the time it takes to decide on an action. Also, it cannot formulate new goals of its own or accept new goals in addition to its current goals, so it cannot be a long-lived agent in a complex environment. These shortcomings will be addressed in the next section.

CONTINUOUS PLANNING AGENT

In this section, we design an agent that persists indefinitely in an environment. Thus it is not a “problem solver” that is given a single goal and then plans and acts until the goal is achieved; rather, it lives through a series of ever-changing goal formulation, planning, and acting phases. Rather than thinking of the planner and execution monitor as separate processes, one of which passes its results to the other, we can think of them as a single process in a continuous planning agent.

The agent is thought of as always being part of the way through executing a plan—the grand plan of living its life. Its activities include executing some steps of the plan that are ready to be executed, refining the plan to satisfy open preconditions or resolve conflicts, and modifying the plan in the light of additional information obtained during execution. Obviously, when it first formulates a new goal, the agent will have no actions ready to execute, so it will spend a while generating a partial plan. It is quite possible, however, for the agent to begin execution before the plan is complete, especially when it has independent subgoals to achieve. The continuous planning agent monitors the world continuously, updating its world model from new percepts even if its deliberations are still continuing.

We will first go through an example and then describe the agent program, which we will call CONTINUOUS-POP-AGENT because it uses partial-order plans to represent its intended activities. To simplify the presentation, we will assume a fully observable environment. The same techniques can be extended to the partially observable case.

The example we will use is a problem from the blocks world domain (Section 11.1). The start state is shown in Figure 12.15(a). The action we will need is Move (x,y), which moves block x onto block y, provided that both are clear. Its action schema is

Action(Move(x,y), PRECOND: Clear(z) A Clear(y) A On(z, 2), EFFECT: On(z,y) A Clear(z) A ~On(z, z) A = Clear(y)) .

The agent first needs to formulate a goal for itself. We won’t discuss goal formulation here, but instead we will assume that somehow the agent was told (or decided on its own) to achieve the goal On(C, D) A On(D, B). The agent starts planning for this goal. Unlike

446

Chapter 12. Planning and Acting in the Real World

0] 8] a

F |

(©) (@

=e n

Figure 12.15 The sequence of states as the continuous planning agent tries to reach the goal state On(C, D) A On(D, B), as shown in (d). The start state is (a). At (b), another agent has interfered, putting DD on B. At (c), the agent has executed Move(C, D) but has failed, dropping C on A instead. It retries Move(C, D), reaching the goal state (d).

all our other agents, which would shut off their percepts until the planner returns a complete solution to this problem, the continuous planning agent builds the plan incrementally, with each increment taking a bounded amount of time. After each increment, the agent returns NoOp as its action and checks its percepts again. We assume that the percepts don’t change and the agent quickly constructs the plan shown in Figure 12.16. Notice that although the preconditions of both actions are satisfied by Start, there is an ordering constraint putting Move(D, B) before Move(C, D). This is needed to ensure that Clear(D) remains true until Move(D, B) is completed. Throughout the continuous planning process, Start is always used as the label for the current state. The agent updates the state after each action.

Gear) | Move(C,D

lear ove(C,

Ontable(A) Clear(D) ( )

On(B,E)

onba on(c,b)

Start on(D,G) Clear(A) Clear(C)

On(D,B) Finish Clear(D) On(D,G) Clear(B) mm Geary Move(D,B)

Figure 12.16 The initial plan constructed by the continuous planning agent. The plan is indistinguishable, so far, from that produced by a normal partial-order planner.

The plan is now ready to be executed, but before the agent can take action, nature intervenes. An external agent (perhaps the agent’s teacher getting impatient) moves D onto B and the world is now in the state shown in Figure 12.15(b). The agent perceives this, recognizes that Clear(B) and On(D,G) are no longer true in the current state, and updates its model of the current state accordingly. The causal links that were supplying the preconditions Clear(B) and On(D, G) for the Move(D, B) action become invalid and must be removed from the plan. The new plan is shown in Figure 12.17. At all times, Start represents the current state, so this Start is different from the one in the previous figure. Notice that the plan is now incomplete: two of the preconditions for Move(D, B) are open, and its precondition On(D,y) is now uninstantiated, because there is no longer any reason to assume the that move will be from G.

Section 12.6. Continuous Planning 447 Cie Cid car | Move(C,D

Ontable(A) lear(D) (©.0)

fee

n(C, Cc Lo Start kd onsg Finish

Clear(C)

Clear(D) oni D 5)

Clear(G) Clean) Move(D,B) Figure 12.17 After someone else moves D onto B, the unsupported links supplying Clear(B) and On(D, G) are dropped, producing this plan,

EXTENSION

REDUNDANT STEP

Now the agent can take advantage of the “helpful” interference by noticing that the causal link Move(D, B) OnD:B) pinish can be replaced by a direct link from Start to Finish. This process is called extending a causal link and is done whenever a condition can be supplied by an earlier step instead of a later one without causing a new conflict.

Once the old causal link from Move(D, B) to Finish is removed, Move(D, B) no longer supplies any causal links at all. It is now a redundant step. All redundant steps, and

any links supplying them, are dropped from the plan. This gives the plan in Figure 12.18.

Qn(C.F) Clear(C) Ontable(; EE OnBE) On(C,F)

Clear(G)

Move(C,D)

Oon(C,D Start onze | Finish

Figure 12.18 The link supplied by Move(D, B) has been replaced by one from Start, and the now-redundant step Move(D, B) has been dropped.

Now the step Move(C, D) is ready to be executed, because all of its preconditions are satisfied by the Start step, no other steps are necessarily before it, and it does not conflict with any other link in the plan. The step is removed from the plan and executed. Unfortunately, the agent is clumsy and drops C' onto A instead of D, giving the state shown in Figure 12.15(c). The new plan state is shown in Figure 12.19. Notice that although there are now no actions in the plan, there is still an open condition for the Finish step.

Ontable(A) On(B,E) On(C,A) On(D,B) Glear(F) Clear(C) Ciear(D) Clear(G)

on(Cc,D — one)

Start

Finish

Figure 12.19 After Move(C, D) is executed and removed from the plan, the effects of the Start step reflect the fact that C' ended up on A instead of the intended D. The goal

precondition On (C, D) is still open.

448

Chapter 12. Planning and Acting in the Real World

PLAN FLAW

On(C,A nea

Move(C,D Ontable(A) Clear(D) ve(C,D) On(B,E) ono) on(c.0)

On(D,B, a Start pei 2 On(D.B) Finish Clear(C) Clear(D) Clear(G)

Figure 12.20 The open condition is resolved by adding Move(C, D) back in. Notice the new bindings for the preconditions.

The agent decides to plan for the open condition. Once again, Move(C, D) will satisfy the goal condition. Its preconditions are satisfied by new causal links from the Start step. The new plan appears in Figure 12.20.

Once again, Move(C, D) is ready for execution. This time it works, resulting in the goal state shown in Figure 12.15(d). Once the step is dropped from the plan, the goal condition On(C, D) becomes open again. Because the Start step is updated to reflect the new world state, however, the goal condition can be satisfied immediately by a link from the Start step. This is the normal course of events when an action is successful. The final plan state is shown in Figure 12.21. Because all the goal conditions are satisfied by the Start step and there are no remaining actions, the agent is now free to remove the goals from Finish and formulate a new goal.

Ontable(A) oe n(C, Oon(C,D) . Start Gar? onCE) | Finish I Glearid) Clear(G)

Figure 12.21 After Move(C, D) is executed and dropped from the plan, the remaining open condition On(C, D) is resolved by adding a causal link from the new Start step. The plan is now completed.

From this example, we can see that continuous planning is quite similar to partial-order planning. On each iteration, the algorithm finds something about the plan that needs fixing— a so-called plan flaw—and fixes it. The POP algorithm can be seen as a flaw-removal algorithm where the two flaws are open preconditions and causal conflicts. The continuous planning agent, on the other hand, addresses a much broader range of flaws:

o Missing goal: The agent can decide to add a new goal or goals to the Finish state. (Under continuous planning, it might make more sense to change the name of Finish to Infinity, and of Start to Current, but we will stick with tradition.)

o Open precondition: Add a causal link to an open precondition, choosing either a new or an existing action (as in POP).

o Causal Conflict: Given a causal link A \_£, B and an action C with effect =p, choose an ordering constraint or variable constraint to resolve the conflict (as in POP).

Section 12.7.

Unsupported link: If there is a causal link Start \_2, A where p is no longer true in Start, then remove the link. (This prevents us from executing an action whose preconditions are false.)

Redundant action: If an action A supplies no causal links, remove it and its links. (This allows us to take advantage of serendipitous events.)

Unexecuted action: If an action A (other than Finish) has its preconditions satisfied in Start, has no other actions (besides Start) ordered before it, and conflicts with no causal links, then remove A and its causal links and return it as the action to be executed. Unnecessary historical goal: If there are no open preconditions and no actions in the plan (so that all causal links go directly from Start to Finish), then we have achieved the current goal set. Remove the goals and the links to them to allow for new goals.

static: plan, a plan, initially with just Start, Finish

action «— NoQOp (the default)

EFFECTS[ Start] = UPDATE(EFFECTS[ Start], percept) REMOVE-FLAW( plan) // possibly updating action return action

Figure 12.22 CONTINUOUS-POP-AGENT, a continuous partial-order planning agent. After receiving a percept, the agent removes a flaw from its constantly updated plan and then returns an action. Often it will take many steps of flaw-removal planning, during which it returns NoQOp, before it is ready to take a real action.

450 Chapter 12. Planning and Acting in the Real World

Agents(A, B) Init(At(A, [Left, Baseline]) A At(B, [Right, Net]) A

Approaching( Ball, [Right, Baseline])) A Partner(A, B) A Partner(B, A) Goal( Returned(Ball) A At(agent, [z, Net])) Action (Hit( agent, Ball),

PRECOND: Approaching( Ball, [z, y]) A At{agent, [z, y]) A

Partner(agent, partner) A — At(partner,|z,y])

EFFECT: Returned(Ball)) Action(Go(agent, [z, y]),

PRECOND: At(agent, [a, b]),

EFFECT: At(agent, [x,y]) A — At{agent, [a, 8])) Figure 12.23 The doubles tennis problem. Two agents are playing together and can be in one of four locations: [Left, Baseline], [Right, Baseline], [Left, Net], and [Right, Net]. The ball can be returned if exactly one player is in the right place.

We saw in Chapter 2 that multiagent environments can be cooperative or competitive. We will begin with a simple cooperative example: team planning in doubles tennis. Plans can be constructed that specify actions for both players on the team; we will describe techniques for constructing such plans efficiently. Efficient plan construction is useful, but does not guarantee success; the agents have to agree to use the same plan! This requires some form of coordination, possibly achieved by communication.

Cooperation: Joint goals and plans

Two agents playing on a doubles tennis team have the joint goal of winning the match, which

gives rise to various subgoals. Let’s suppose that at one point in the game, they have the joint

goal of returning the ball that has been hit to them and ensuring that at least one of them is MU ASENT covering the net. We can represent this notion as a multiagent planning problem, as shown

in Figure 12.23.

This notation introduces two new features. First, Agents(A, B) declares that there are two agents, A and B, who are participating in the plan. (For this problem the opposing players are not considered agents.) Second, each action explicitly mentions the agent as a parameter, because we need to keep track of which agent does what.

JOINT PLAN A solution to a multiagent planning problem is a joint plan consisting of actions for

each agent. A joint plan is a solution if the goal will be achieved when each agent performs its assigned actions. The following plan is a solution to the tennis problem:

PLAN 1: A: [Go(A, Right, Baseline]), Hit(A, Ball) B: [NoOp(B), NoOp(B)] . If both agents have the same knowledge base, and if this is the only solution, then everything would be fine; the agents could each determine the solution and then jointly execute it. Unfortunately for the agents (and we will soon see why it’s unfortunate), there is another plan

MultiA gent Planning 451

COORDINATION

MULTIBODY PLANNING

SYNCHRONIZATION

JOINT ACTION

CONCURRENT ACTION LIST

that satisfies the goal just as well as the first:

PLAN 2: A: [Go(A, Left, Net]), NoOp(A)] B : [Go(B, Right, baseline)), Hit(B, Ball)] .

If A chooses plan 2 and B chooses plan 1, then nobody will return the ball. Conversely, if A chooses 1 and B chooses 2, then they will probably collide with each other; no one returns the ball and the net may remain uncovered. Hence, the existence of correct joint plans does not mean that the goal will be achieved. The agents need a mechanism for coordination to reach the same joint plan; moreover, it must be common knowledge (see Chapter 10) among the agents that some particular joint plan will be executed.

Multibody planning

This section concentrates on the construction of correct joint plans, deferring the coordination issue for the time being. We call this multibody planning; it is essentially the planning problem faced by a single centralized agent that can dictate actions to each of several physical entities. In the truly multiagent case, it enables each agent to figure out what the possible joint plans are that would succeed if executed jointly.

Our approach to multibody planning will be based on partial-order planning, as described in Section 11.3. We will assume full observability, to keep things simple. There is one additional issue that doesn’t arise in the single-agent case: the environment is no longer truly static, because other agents could act while any particular agent is deliberating. Therefore, we need to be concerned about synchronization. For simplicity, we will assume that each action takes the same amount of time and that actions at each point in the joint plan are simultaneous.

At any point in time, each agent is executing exactly one action (perhaps including NoOp). This set of concurrent actions is called a joint action. For example, a joint action in the tennis domain (page 450) with two agents A and B is (NoOp(A), Hit(B, Ball)). A joint plan consists of a partially ordered graph of joint actions. For example, Plan 2 for the tennis problem can be represented as this sequence of joint actions:

(Go(A, Left, Net), Go(B, [Right, baseline])) (NoOp(A), Hit(B, Ball)

We could do planning using the regular POP algorithm, applied to the set of all possible joint actions. The only problem is the size of this set: with 10 actions and 5 agents we get 10° joint actions. It would be tedious to specify the preconditions and effects of each action correctly, and inefficient to do planning with such a large set.

An alternative is to define joint actions implicitly, by describing how each individual action interacts with other possible actions. This will be simpler, because most actions are independent of most others; we need list only the few actions that actually interact. We can do that by augmenting the usual STRIPS or ADL action descriptions with one new feature: a concurrent action list. This is similar to the precondition of an action description except that rather than describing state variables, it describes actions that must or must not be executed

452

Chapter 12. Planning and Acting in the Real World

CONVENTION

SOCIAL LAWS

concurrently. For example, the Hit action could be described as follows:

Action(Hit(A, Ball), CONCURRENT: Hit(B, Ball) PRECOND:Approaching( Ball, [x,y]) A At(A, [z,y]) EFFECT: Returned (Ball) .

Here, we have the prohibited-concurrency constraint that, during the execution of the Hit action, there can be no other Hit action by another agent. We can also require concurrent action, for example when two agents are needed to carry a cooler full of beverages to the tennis court. The description for this action says that agent A cannot execute a Carry action unless there is another agent B who is simultaneously executing a Carry of the same cooler:

Action(Carry(A, cooler, here, there), CONCURRENT: Carry(B, cooler, here, there) PRECOND:At(A, here) A At(cooler, here) A Cooler (cooler) EFFECT: At(A, there) A At(cooler, there) A —At(A, here) A =At(cooler, here)).

With this representation, it is possible to create a planner that is very close to the POP partialorder planner. There are three differences:

1. In addition to the temporal ordering relation A < B, we allow A = B and A meaning “concurrent” and “before or concurrent,” respectively. meaning “concurrent” and “before or concurrent,” respectively.

2. When a new action has required concurrent actions, we must instantiate those actions, using new or existing actions in the plan.

3. Prohibited concurrent actions are an additional source of constraints. Each constraint must be resolved by constraining conflicting actions to be before or after.

This representation gives us the equivalent of POP for multibody domains. We could extend this approach with the refinements of the last two chapters—HTNss, partial observability, conditionals, execution monitoring, and replanning—but that is beyond the scope of this book.

Coordination mechanisms

The simplest method by which a group of agents can ensure agreement on a joint plan is to adopt a convention prior to engaging in joint activity. A convention is any constraint on the selection of joint plans, beyond the basic constraint that the joint plan must work if all agents adopt it. For example, the convention “stick to your side of the court” would cause the doubles partners to select plan 2, whereas the convention “one player always stays at the net” would lead them to plan 1. Some conventions, such as driving on the proper side of the road, are so widely adopted that they are considered social laws. Human languages can also be viewed as conventions.

The conventions in the preceding paragraph are domain-specific and can be implemented by constraining the action descriptions to rule out violations of the convention. A more general approach is to use domain-independent conventions. For example, if each agent runs the same multibody planning algorithm with the same inputs, it can follow the convention of executing the first feasible joint plan found, confident that the other agents will come

Section 12.7.

MultiAgent Planning 453

BOID

EMERGENT BEHAVIOR

PLAN RECOGNITION

to the same choice. A more robust but more expensive strategy would be to generate all joint plans and then pick the one, say, whose printed representation is alphabetically first.

Conventions can also arise through evolutionary processes. For example, colonies of social insects execute very elaborate joint plans, which are facilitated by the common genetic makeup of the individuals in the colony. Conformity can also be enforced by the fact that deviation from conventions reduces evolutionary fitness, so that any feasible joint plan can become a stable equilibrium. Similar considerations apply to the development of human language, where the important thing is not which language each individual should speak, but the fact that all individuals speak the same language. One final example appears in the flocking behavior of birds. We can obtain a reasonable simulation if each bird agent (sometimes called a birdoid or boid) executes the following three rules with some method of combination:

1. Separation: Steer away from neighbors when you start to get too close. 2. Cohesion: Steer towards the average position of the neighbors. 3. Alignment: Steer towards the average orientation (heading) of the neighbors.

If all the birds execute the same policy, the flock exhibits the emergent behavior of flying as a pseudo-rigid body with roughly constant density that does not disperse over time. As with insects, there is no need for each agent to possess the joint plan that models the actions of other agents.

Typically, conventions are adopted to cover a universe of individual multiagent planning problems, rather than being developed anew for each problem. This can lead to inflexibility and breakdown, as can be seen sometimes in doubles tennis when the ball is roughly equidistant between the two partners. In the absence of an applicable convention, agents can use communication to achieve common knowledge of a feasible joint plan. For example, a doubles tennis player could shout “Mine!” or “Yours!” to indicate a preferred joint plan. We cover mechanisms for communication in more depth in Chapter 22, where we observe that communication does not necessarily involve a verbal exchange. For example, one player can communicate a preferred joint plan to the other simply by executing the first part of it. In our tennis problem, if agent A heads for the net, then agent B is obliged to go back to the baseline to hit the ball, because plan 2 is the only joint plan that begins with A’s heading for the net. This approach to coordination, sometimes called plan recognition, works when a single action (or short sequence of actions) is enough to determine a joint plan unambiguously.

The burden for ensuring that the agents arrive at a successful joint plan can be placed either on the agent designers or on the agents themselves. In the former case, before the agents begin to plan, the agent designer should prove that the agents’ policies and strategies will be successful. The agents themselves can be reactive if that works for the environment they exist in, and they need not have explicit models about the other agents. In the latter case, the agents are deliberative; they must prove or otherwise demonstrate that their plan will be effective, taking the other agents’ reasoning into account. For example, in an environment with two logical agents A and B, they could both have the following definition:

Vp,s Feasible(p,s) & CommonKnowledge({A, B}, Achieves(p, s, Goal))

This says that in any situation s, the plan p is a feasible joint plan in that situation if it is common knowledge among the agents that p will achieve the goal. We need further axioms

454

Chapter 12. Planning and Acting in the Real World

JOINT INTENTION

COMPETITION

to establish common knowledge of a joint intention to execute a particular joint plan; only then can agents begin to act.

Competition

Not all multiagent environments involve cooperative agents. Agents with conflicting utility functions are in competition with each other. One example of this is two-player zero-sum games, such as chess. We saw in Chapter 6 that a chess-playing agent needs to consider the opponent’s possible moves for several steps into the future. That is, an agent in a competitive environment must (a) recognize that there are other agents, (b) compute some of the other agent’s possible plans, (c) compute how the other agent’s plans interact with its own plans, and (d) decide on the best action in view of these interactions. So competition, like cooperation, requires a model of the other agent’s plans. On the other hand, there is no commitment to a joint plan in a competitive environment.

Section 12.4 drew the analogy between games and conditional planning problems. The conditional planning algorithm in Figure 12.10 constructs plans that work under worst-case assumptions about the environment, so it can be applied in competitive situations where the agent is concerned only with success and failure. When the agent and its opponents are concerned about the cost of a plan, then minimax is appropriate. As yet, there has been little work on combining minimax with methods, such as POP and HTN planning, that go beyond the state-space search model used in Chapter 6. We will return to the question of competition in Section 17.6, which covers game theory.

This chapter has addressed some of the complications of planning and acting in the real world. The main points are:

e Many actions consume resources, such as money, gas, or raw materials. It is convenient to treat these resources as numeric measures in a pool rather than try to reason about, say, each individual coin and bill in the world. Actions can generate and consume resources, and it is usually cheap and effective to check partial plans for satisfaction of resource constraints before attempting further refinements.

o Time is one of the most important resources. It can be handled by specialized scheduling algorithms, or scheduling can be integrated with planning.

o Hierarchical task network (HTN) planning allows the agent to take advice from the domain designer in the form of decomposition rules. This makes it feasible to create the very large plans required by many real-world applications.

o Standard planning algorithms assume complete and correct information and deterministic, fully observable environments. Many domains violate this assumption.

o Incomplete information can be dealt with by planning to use sensing actions to obtain the information needed. Conditional plans allow the agent to sense the world during

Section 12.8.

Summary 455

execution to decide what branch of the plan to follow. In some cases, sensorless or conformant planning can be used to construct a plan that works without the need for perception. Both sensorless and conditional plans can be constructed by search in the space of belief states.

eo Incorrect information results in unsatisfied preconditions for actions and plans. Execution monitoring detects violations of the preconditions for successful completion of the plan.

o A replanning agent uses execution monitoring and splices in repairs as needed.

e A continuous planning agent creates new goals as it goes and reacts in real time.

o Multiagent planning is necessary when there are other agents in the environment with which to cooperate, compete, or coordinate. Multibody planning constructs joint plans, using an efficient decomposition of joint action descriptions, but must be augmented with some form of coordination if two cooperative agents are to agree on which joint plan to execute.

Planning with continuous time was first dealt with by DEVISER (Vere, 1983). The issue of systematic representation of time in plans was addressed by Dean et al. (1990) in the FORBIN system. NONLIN+ (Tate and Whiter, 1984) and S1PE (Wilkins, 1988, 1990) could reason about the allocation of limited resources to various plan steps. O-PLAN (Bell and Tate, 1985), an HTN planner, had a uniform, general representation for constraints on time and resources. In addition to the Hitachi application mentioned in the text, O-PLAN has been applied to software procurement planning at Price Waterhouse and back-axle assembly planning at Jaguar Cars. A number of hybrid planning-and-scheduling systems have been deployed: Isis (Fox er al., 1982; Fox, 1990) has been used for job shop scheduling at Westinghouse, GARI (Descotte and Latombe, 1985) planned the machining and construction of mechanical parts, FORBIN was used for factory control, and NONLIN+ was used for naval logistics planning.

After an initial flurry of theoretical work in the late 1980s, temporal planning made a comeback recently, when new algorithms and increased processing power made it feasible to attack practical applications. The two planners SAPA (Do and Kambhampati, 2001) and T4 (Haslum and Geffner, 2001) both used forward state-space search with sophisticated heuristics to handle actions with durations and resources. An alternative is to use very expressive action languages, but guide them by human-written domain-specific heuristics, as is done by ASPEN (Fukunaga et al., 1997), HSTS (Jonsson er al., 2000), and IxTeT (Ghallab and Laruelle, 1994).

There is a long history of scheduling in aerospace. T-SCHED (Drabble, 1990) was used to schedule mission-command sequences for the UOSAT-II satellite. OPTIMUM-AIV (Aarup et al., 1994) and PLAN-ERS1 (Fuchs et al., 1990), both based on O-PLAN, were used for spacecraft assembly and observation planning, respectively, at the European Space Agency.

456

Chapter 12. Planning and Acting in the Real World

MACROPS

ABSTRACTION HIERARCHY

CASE-BASED PLANNING

SPIKE (Johnston and Adorf, 1992) was used for observation planning at NASA for the Hubble Space Telescope, while the Space Shuttle Ground Processing Scheduling System (Deale et al., 1994) does job-shop scheduling of up to 16,000 worker-shifts. Remote Agent (Muscettola et al., 1998) became the first autonomous planner-scheduler to control a spacecraft when it flew onboard the Deep Space One probe in 1999. The literature on job-shop scheduling in operations research is surveyed by Vaessens et al. (1996); theoretical results are presented by Martin and Shmoys (1996).

The facility in the STRIPS program for learning macrops—“macro-operators” consisting of a sequence of primitive steps—could be considered the first mechanism for hierarchical planning (Fikes ef al., 1972). Hierarchy was also used in the LAWALY system (Siklossy and Dreussi, 1973). The ABSTRIPS system (Sacerdoti, 1974) introduced the idea of an abstraction hierarchy, whereby planning at higher levels was permitted to ignore lower-level preconditions of actions in order to derive the general structure of a working plan. Austin Tate’s Ph.D. thesis (1975b) and work by Earl Sacerdoti (1977) developed the basic ideas of HTN planning in its modern form. Many practical planners, including O-PLAN and SIPE, are HTN planners. Yang (1990) discusses properties of actions that make HTN planning efficient. Erol, Hendler, and Nau (1994, 1996) present a complete hierarchical decomposition planner as well as a range of complexity results for pure HTN planners. Other authors (AmbrosIngerson and Steel, 1988; Young er al., 1994; Barrett and Weld, 1994; Kambhampati ef al., 1998) have proposed the hybrid approach taken in this chapter, in which decompositions are just another form of refinement that can be used in partial-order planning.

Beginning with the work on macro-operators in STRIPS, one of the goals of hierarchical planning has been the reuse of previous planning experience in the form of generalized plans. The technique of explanation-based learning, described in depth in Chapter 19, has been applied in several systems as a means of generalizing previously computed plans, including SOAR (Laird et al., 1986) and PRODIGY (Carbonell ef al., 1989). An alternative approach is to store previously computed plans in their original form and then reuse them to solve new, similar problems by analogy to the original problem. This is the approach taken by the field called case-based planning (Carbonell, 1983; Alterman, 1988; Hammond, 1989). Kambhampati (1994) argues that case-based planning should be analyzed as a form of refinement planning and provides a formal foundation for case-based partial-order planning.

The unpredictability and partial observability of real environments was recognized early on in robotics projects that used planning techniques, including Shakey (Fikes et al., 1972) and FREDDY (Michie, 1974). The problem received more attention after the publication of McDermott’s (1978a) influential article, Planning and Acting.

Early planners, which lacked conditionals and loops, did not explicitly recognize the concept of conditional planning; but nevertheless they sometimes resorted to a coercive style in response to environmental uncertainty. Sacerdoti’s NOAH used coercion in its solution to the “keys and boxes” problem, a planning challenge problem in which the planner knows little about the initial state. Mason (1993) argued that sensing often can and should be dispensed with in robotic planning, and described a sensorless plan that can move a tool into a specific position on a table by a sequence of tilting actions, regardless of the initial position. We describe this idea in the context of robotics. (See Figure 25.17.)

Section 12.8.

Summary 457

Goldman and Boddy (1996) introduced the term conformant planning for sensorless planners that handle uncertainty by coercing the world into known states, noting that sensorless plans are often effective even if the agent has sensors. The first moderately efficient conformant planner was Smith and Weld’s (1998) Conformant Graphplan or CGP. Ferraris and Giunchiglia (2000) and Rintanen (1999) independently developed SATplan-based conformant planners. Bonet and Geftner (2000) describe a conformant planner based on heuristic search in the space of belief states, drawing on ideas first developed in the 1960s for partially observable Markov decision processes, or POMDPs (see Chapter 17). Currently, the fastest belief-state conformant planners, such as HSCP (Bertoli et al., 2001a), use binary decision diagrams (BDDs) (Bryant, 1992) to represent belief states and are up to five orders of magnitude faster than CGP.

WARPLAN-C (Warren, 1976), a variant of WARPLAN, was one of the earliest planners to use conditional actions. Olawski and Gini (1990) lay out the major issues involved in conditional planning.

The conditional planning approach described in the chapter is based on the efficient search algorithms for cyclic AND—OR graphs developed by Jimenez and Torras (2000) and Hansen and Zilberstein (2001). Bertoli et al. (2001b) describe a BDD-based approach that constructs conditional plans with loops. C-BURIDAN (Draper ef al., 1994) handles conditional planning for actions with probabilistic outcomes, a problem also addressed under the heading of POMDPs (Chapter 17).

There is a close relation between conditional planning and automated program synthesis; a number of references appear in Chapter 9. The two fields have been pursued separately, because of the enormous difference in cost between execution of machine instructions and execution of actions by robot vehicles or manipulators. Linden (1991) attempts explicit cross-fertilization between the two fields.

In retrospect, it is now possible to see how the major classical planning algorithms led to extended versions for domains involving uncertainty. Search-based techniques led to search in belief space (Bonet and Geffner, 2000); SATPLAN led to stochastic SATPLAN (Majercik and Littman, 1999) and to planning using quantified Boolean logic (Rintanen, 1999); partial order planning led to UWL (Etzioni et al., 1992), CNLP (Peot and Smith, 1992), and CASSANDRA (Pryor and Collins, 1996). GRAPHPLAN led to Sensory Graphplan or SGP (Weld et al., 1998), but a full probabilistic GRAPHPLAN has yet to be developed.

The earliest major treatment of execution monitoring was PLANEX (Fikes et al., 1972), which worked with the STRIPS planner to control the robot Shakey. PLANEX used triangle tables—essentially an efficient storage mechanism for the plan preconditions at each point in the plan—to allow recovery from partial execution failure without complete replanning. Shakey’s model of execution is discussed further in Chapter 25. The NASL planner (McDermott, 1978a) treated a planning problem simply as a specification for carrying out a complex action, so that execution and planning were completely unified. It used theorem proving to reason about these complex actions.

SPE (System for Interactive Planning and Execution monitoring) (Wilkins, 1988, 1990) was the first planner to deal systematically with the problem of replanning. It has been used in

458

Chapter 12. Planning and Acting in the Real World

REACTIVE PLANNING

POLICIES

demonstration projects in several domains, including planning operations on the flight deck of an aircraft carrier and job-shop scheduling for an Australian beer factory. Another study used SIPE to plan the construction of multistory buildings, one of the most complex domains ever tackled by a planner.

IPEM (Integrated Planning, Execution, and Monitoring) (Ambros-Ingerson and Steel, 1988) was the first system to integrate partial-order planning and execution to yield a continuous planning agent. Our CONTINUOUS-POP-AGENT combines ideas from IPEM, the PUCCINI planner (Golden, 1998), and the CYPRESS system (Wilkins et al., 1995).

In the mid-1980s, it was believed by some that partial-order planning and related techniques could never run fast enough to generate effective behavior for an agent in the real world (Agre and Chapman, 1987). Instead, reactive planning systems were proposed; in their basic form, these are reflex agents, possibly with internal state, that can be implemented with any of a variety of representations for condition—action rules. Brooks’s (1986) subsumption architecture (see Chapters 7 and 25) used layered finite-state machines in legged and wheeled robots to control their locomotion and avoid obstacles. Pengi (Agre and Chapman, 1987) was able to play a (fully observable) video game using Boolean circuits combined with a “visual” representation of current goals and the agent’s internal state.

“Universal plans” (Schoppers, 1987) were developed as a lookup-table method for reactive planning, but turned out to be a rediscovery of the idea of policies that had long been used in Markov decision processes. A universal plan (or a policy) contains a mapping from any state to the action that should be taken in that state. Ginsberg (1989) made a spirited attack on universal plans, including intractability results for some formulations of the reactive planning problem. Schoppers (1989) made an equally spirited reply.

As is often the case, a hybrid approach resolves the controversy. Using well-designed hierarchies, HTN planners, such as PRS (Georgeff and Lansky, 1987) and RAP (Firby, 1996), as well as continuous planning agents, can achieve reactive response times and complex long-range planning behavior in many problem domains.

Multiagent planning has leaped in popularity in recent years, although it does have a long history. Konolige (1982) provided a formalization of multiagent planning in first-order logic, while Pednault (1986) gave a STRIPS-style description. The notion of joint intention, which is essential if agents are to execute a joint plan, comes from work on communicative acts (Cohen and Levesque, 1990; Cohen et al., 1990). Our presentation of multibody partialorder planning is based on the work of Boutilier and Brafman (2001).

We have barely skimmed the surface of work on negotiation in multiagent planning. Durfee and Lesser (1989) discuss how tasks can be shared out among agents by negotiation. Kraus et al. (1991) describe a system for playing Diplomacy, a board game requiring negotiation, coalition formation and dissolution, and dishonesty. Stone (2000) shows how agents can cooperate as teammates in the competitive, dynamic, partially observable environment of robotic soccer. (Weiss, 1999) is a book-length overview of multiagent systems.

The boid model on page 453 is due to Reynolds (1987), who won an Academy Award for its application to flocks of bats and swarms of penguins in Batman Returns.

Section 12.8.

Summary 459

EXERCISES

CONSUMABLE RESOURCE

12.1 Examine carefully the representation of time and resources in Section 12.1.

a. Why is it a good idea to have Duration(d) be an effect of an action, rather than having a separate field in the action of the form DURATION: d? (Hint: Consider conditional effects and disjunctive effects.)

b. Why is RESOURCE: m a separate field in the action, rather than being an effect?

12.2 A consumable resource is a resource that is (partially) used up by an action. For example, attaching engines to cars requires screws. The screws, once used, are not available for other attachments.

a. Explain how to modify the representation in Figure 12.3 so that there are 100 screws initially, engine E'| requires 40 screws, and engine FE requires 50 screws. The + and — function symbols may be used in effect literals for resources.

b. Explain how the definition of conflict between causal links and actions in partial-order planning must be modified to handle consumable resources.

¢. Some actions—for example, resupplying the factory with screws or refueling a car— can increase the availability of resources. A resource is monotonically non-increasing if no action increases it. Explain how to use this property to prune the search space.

12.3 Give decompositions for the HireBuilder and GetPermit steps in Figure 12.7, and show how the decomposed subplans connect into the overall plan.

12.4 Give an example in the house-building domain of two abstract subplans that cannot be merged into a consistent plan without sharing steps. (Hint: Places where two physical parts of the house come together are also places where two subplans tend to interact.)

12.5 Some people say an advantage of HTN planning is that it can solve problems like “take around trip from Los Angeles to New York and back” that are hard to express in non-HTN notations because the start and goal states would be the same (A¢(LA)). Can you think of a way to represent and solve this problem without HTNs?

12.6 Show how a standard STRIPS action description can be rewritten as an HTN decomposition, using the notation Achieve(p) to denote the activity of achieving the condition p.

12.7 Some of the operations in standard programming languages can be modeled as actions that change the state of the world. For example, the assignment operation copies the contents of a memory location, while the print operation changes the state of the output stream. A program consisting of these operations can also be considered as a plan, whose goal is given by the specification of the program. Therefore, planning algorithms can be used to construct programs that achieve a given specification.

a. Write an operator schema for the assignment operator (assigning the value of one variable to another). Remember that the original value will be overwritten!

460

Chapter 12. Planning and Acting in the Real World

b. Show how object creation can be used by a planner to produce a plan for exchanging the values of two variables using a temporary variable.

12.8 Consider the following argument: In a framework that allows uncertain initial states, disjunctive effects are just a notational convenience, not a source of additional representational power. For any action schema a with disjunctive effect PV (J, we could always replace it with the conditional effects when R: P A when —R: (), which in turn can be reduced to two regular actions. The proposition R stands for a random proposition that is unknown in the initial state and for which there are no sensing actions. Is this argument correct? Consider separately two cases, one in which only one instance of action schema a is in the plan, the other in which more than one instance is.

12.9 Why can’t conditional planning deal with unbounded indeterminacy?

12.10 In the blocks world we were forced to introduce two STRIPS actions, Move and MoveToTable, in order to maintain the Clear predicate properly. Show how conditional effects can be used to represent both of these cases with a single action.

12.11 Conditional effects were illustrated for the Suck action in the vacuum world—which square becomes clean depends on which square the robot is in. Can you think of a new set of propositional variables to define states of the vacuum world, such that Suck has an unconditional description? Write out the descriptions of Suck, Left, and Right, using your propositions, and demonstrate that they suffice to describe all possible states of the world.

12.12 Write out the full description of Suck for the double Murphy vacuum cleaner that sometimes deposits dirt when it moves to a clean destination square and sometimes deposits dirt if Suck is applied to a clean square.

12.13 Find a suitably dirty carpet, free of obstacles, and vacuum it. Draw the path taken by the vacuum cleaner as accurately as you can. Explain it, with reference to the forms of planning discussed in this chapter.

12.14 The following quotes are from the backs of shampoo bottles. Identify each as an unconditional, conditional, or execution monitoring plan. (a) “Lather. Rinse. Repeat.” (b) “Apply shampoo to scalp and let it remain for several minutes. Rinse and repeat if necessary.” (c) “See a doctor if problems persist.”

12.15 The AND-OR-GRAPH-SEARCH algorithm in Figure 12.10 checks for repeated states only on the path from the root to the current state. Suppose that, in addition, the algorithm were to store every visited state and check against that list. (See GRAPH-SEARCH in Figure 3.19 for an example.) Determine the information that should be stored and how the algorithm should use that information when a repeated state is found. (Hint: You will need to distinguish at least between states for which a successful subplan was constructed previously and states for which no subplan could be found.) Explain how to use labels to avoid having multiple copies of subplans.

12.16 Explain precisely how to modify the AND-OR-GRAPH-SEARCH algorithm to generate a cyclic plan if no acyclic plan exists. You will need to deal with three issues: labeling

Section 12.8.

Summary 461

the plan steps so that a cyclic plan can point back to an earlier part of the plan, modifying OR-SEARCH so that it continues to look for acyclic plans after finding a cyclic plan, and augmenting the plan representation to indicate whether a plan is cyclic. Show how your algorithm works on (a) the triple Murphy vacuum world, and (b) the alternate double Murphy vacuum world. You might wish to use a computer implementation to check your results. Can the plan for case (b) be written using standard loop syntax?

12.17 Specify in full the belief state update procedure for partially observable environments. That is, the method for computing the new belief state representation (as a list of knowledge propositions) from the current belief-state representation and an action description with conditional effects.

12.18 Write action descriptions, analogous to Equation (12.2), for the Right and Suck actions. Also write a description for CheckLocation, analogous to Equation (12.3). Repeat using the alternative set of propositions from Exercise 12.11.

12.19 Look at the list on page 445 of things that the replanning agent can’t do. Sketch an algorithm that can handle one or more of them.

12.20 Consider the following problem: A patient arrives at the doctor’s office with symptoms that could have been caused either by dehydration or by disease D (but not both). There are two possible actions: Drink, which unconditionally cures dehydration, and Medicate, which cures disease D, but has an undesirable side-effect if taken when the patient is dehydrated. Write the problem description in PDDL,, and diagram a sensorless plan that solves the problem, enumerating all relevant possible worlds.

12.21 To the medication problem in the previous exercise, add a Test action that has the conditional effect CultureGrowth when Disease is true and in any case has the perceptual effect Known (CultureGrowth). Diagram a conditional plan that solves the problem and minimizes the use of the Medicate action.

3 UNCERTAINTY

In which we see what an agent should do when not all is crystal clear.

13.1 ACTING UNDER UNCERTAINTY

The logical agents described in Parts III and TV make the epistemological commitment that

propositions are true, false, or unknown. When an agent knows enough facts about its en

vironment, the logical approach enables it to derive plans that are guaranteed to work. This

is a good thing. Unfortunately, agents almost never have access to the whole truth about

their environment. Agents must, therefore, act under uncertainty. For example, an agent in

UNCERTAINTY the wumpus world of Chapter 7 has sensors that report only local information; most of the world is not immediately observable. A wumpus agent often will find itself unable to discover which of two squares contains a pit. If those squares are en route to the gold, then the agent might have to take a chance and enter one of the two squares.

The real world is far more complex than the wampus world. For a logical agent, it might be impossible to construct a complete and correct description of how its actions will work. Suppose, for example, that the agent wants to drive someone to the airport to catch a flight and is considering a plan, Ag, that involves leaving home 90 minutes before the flight departs and driving at a reasonable speed. Even though the airport is only about 15 miles away, the agent will not be able to conclude with certainty that “Plan Agg will get us to the airport in time.” Instead, it reaches the weaker conclusion “Plan Aggy will get us to the airport in time, as long as my car doesn’t break down or run out of gas, and I don’t get into an accident, and there are no accidents on the bridge, and the plane doesn’t leave early, and ... ” None of these conditions can be deduced, so the plan’s success cannot be inferred. This is an example of the qualification problem mentioned in Chapter 10,

If a logical agent cannot conclude that any particular course of action achieves its goal, then it will be unable to act. Conditional planning can overcome uncertainty to some extent, but only if the agent’s sensing actions can obtain the required information and only if there are not too many different contingencies. Another possible solution would be to endow the agent with a simple but incorrect theory of the world that does enable it to derive a plan;

462

Section 13.1.

Acting under Uncertainty 463

presumably, such plans will work most of the time, but problems arise when events contradict the agent’s theory. Moreover, handling the tradeoff between the accuracy and usefulness of the agent’s theory seems itself to require reasoning about uncertainty. In sum, no purely logical agent will be able to conclude that plan Ag is the right thing to do.

Nonetheless, let us suppose that Agg is in fact the right thing to do. What do we mean by saying this? As we discussed in Chapter 2, we mean that out of all the plans that could be executed, Agg is expected to maximize the agent's performance measure, given the information it has about the environment. The performance measure includes getting to the airport in time for the flight, avoiding a long, unproductive wait at the airport, and avoiding speeding tickets along the way. The information the agent has cannot guarantee any of these outcomes for Ago, but it can provide some degree of belief that they will be achieved. Other plans, such as Ajo, might increase the agent’s belief that it will get to the airport on time, but also increase the likelihood of a long wait. The right thing to do—the rational decision—therefore depends on both the relative importance of various goals and the likelihood that, and degree to which, they will be achieved. The remainder of this section hones these ideas, in preparation for the development of the general theories of uncertain reasoning and rational decisions that we present in this and subsequent chapters.

Handling uncertain knowledge

In this section, we look more closely at the nature of uncertain knowledge. We will use a simple diagnosis example to illustrate the concepts involved. Diagnosis—whether for medicine, automobile repair, or whatever—is a task that almost always involves uncertainty. Let us try to write rules for dental diagnosis using first-order logic, so that we can see how the logical approach breaks down. Consider the following rule:

Vp Symptom(p, Toothache) = Disease(p, Cavity) .

The problem is that this rule is wrong. Not all patients with toothaches have cavities; some of them have gum disease, an abscess, or one of several other problems:

Vp Symptom(p, Toothache) = Disease(p, Cavity) V Disease(p, GumDisease) V Disease(p, Abscess)...

Unfortunately, in order to make the rule true, we have to add an almost unlimited list of possible causes. We could try turning the rule into a causal rule:

Vp Disease(p, Cavity) = Symptom(p, Toothache) .

But this rule is not right either; not all cavities cause pain. The only way to fix the rule is to make it logically exhaustive: to augment the left-hand side with all the qualifications required for a cavity to cause a toothache. Even then, for the purposes of diagnosis, one must also take into account the possibility that the patient might have a toothache and a cavity that are unconnected.

Trying to use first-order logic to cope with a domain like medical diagnosis thus fails for three main reasons:

¢& Laziness: It is too much work to list the complete set of antecedents or consequents needed to ensure an exceptionless rule and too hard to use such rules.

464

Chapter 13. Uncertainty

THEORETICAL IGNORANCE

PRACTICAL IGNORANCE

DEGREE OF BELIEF PROBABILITY THEORY

¢ Theoretical ignorance: Medical science has no complete theory for the domain.

¢ Practical ignorance: Even if we know all the rules, we might be uncertain about a particular patient because not all the necessary tests have been or can be run.

The connection between toothaches and cavities is just not a logical consequence in either direction. This is typical of the medical domain, as well as most other judgmental domains: law, business, design, automobile repair, gardening, dating, and so on. The agent’s knowledge can at best provide only a degree of belief in the relevant sentences. Our main tool for dealing with degrees of belief will be probability theory, which assigns to each sentence a numerical degree of belief between 0 and 1. (Some alternative methods for uncertain reasoning are covered in Section 14.7.)

Probability provides a way of summarizing the uncertainty that comes from our laziness and ignorance. We might not know for sure what afflicts a particular patient, but we believe that there is, say, an 80% chance—that is, a probability of 0.8—that the patient has a cavity if he or she has a toothache. That is, we expect that out of all the situations that are indistinguishable from the current situation as far as the agent’s knowledge goes, the patient will have a cavity in 80% of them. This belief could be derived from statistical data—80% of the toothache patients seen so far have had cavities—or from some general rules, or from a combination of evidence sources. The 80% summarizes those cases in which all the factors needed for a cavity to cause a toothache are present and other cases in which the patient has both toothache and cavity but the two are unconnected. The missing 20% summarizes all the other possible causes of toothache that we are too lazy or ignorant to confirm or deny.

Assigning a probability of 0 to a given sentence corresponds to an unequivocal belief that the sentence is false, while assigning a probability of 1 corresponds to an unequivocal belief that the sentence is true. Probabilities between 0 and 1 correspond to intermediate degrees of belief in the truth of the sentence. The sentence itself is in fact either true or false. It is important to note that a degree of belief is different from a degree of truth. A probability of 0.8 does not mean “80% true” but rather an 80% degree of belief—that is, a fairly strong expectation. Thus, probability theory makes the same ontological commitment as logic— namely, that facts either do or do not hold in the world. Degree of truth, as opposed to degree of belief, is the subject of fuzzy logic, which is covered in Section 14.7.

In logic, a sentence such as “The patient has a cavity” is true or false depending on the interpretation and the world; it is true just when the fact it refers to is the case. In probability theory, a sentence such as “The probability that the patient has a cavity is 0.8” is about the agent’s beliefs, not directly about the world. These beliefs depend on the percepts that the agent has received to date. These percepts constitute the evidence on which probability assertions are based. For example, suppose that the agent has drawn a card from a shuffled pack. Before looking at the card, the agent might assign a probability of 1/52 to its being the ace of spades. After looking at the card, an appropriate probability for the same proposition would be 0 or 1. Thus, an assignment of probability to a proposition is analogous to saying whether a given logical sentence (or its negation) is entailed by the knowledge base, rather than whether or not it is true. Just as entailment status can change when more sentences are

Section 13.1.

Acting under Uncertainty 465

PREFERENCES OUTCOMES

UTILITY THEORY

DECISION THEORY

added to the knowledge base, probabilities can change when more evidence is acquired.

All probability statements must therefore indicate the evidence with respect to which the probability is being assessed. As the agent receives new percepts, its probability assessments are updated to reflect the new evidence. Before the evidence is obtained, we talk about prior or unconditional probability; after the evidence is obtained, we talk about posterior or conditional probability. In most cases, an agent will have some evidence from its percepts and will be interested in computing the posterior probabilities of the outcomes it cares about.

Uncertainty and rational decisions

The presence of uncertainty radically changes the way an agent makes decisions. A logical agent typically has a goal and executes any plan that is guaranteed to achieve it. An action can be selected or rejected on the basis of whether it achieves the goal, regardless of what other actions might achieve. When uncertainty enters the picture, this is no longer the case. Consider again the Ago plan for getting to the airport. Suppose it has a 95% chance of succeeding. Does this mean it is a rational choice? Not necessarily: There might be other plans, such as Aja, with higher probabilities of success. If it is vital not to miss the flight, then it is worth risking the longer wait at the airport. What about Aj449, a plan that involves leaving home 24 hours in advance? In most circumstances, this is not a good choice, because, although it almost guarantees getting there on time, it involves an intolerable wait.

To make such choices, an agent must first have preferences between the different possible outcomes of the various plans. A particular outcome is a completely specified state, including such factors as whether the agent arrives on time and the length of the wait at the airport. We will be using utility theory to represent and reason with preferences. (The term utility is used here in the sense of “the quality of being useful,” not in the sense of the electric company or water works.) Utility theory says that every state has a degree of usefulness, or utility, to an agent and that the agent will prefer states with higher utility.

The utility of a state is relative to the agent whose preferences the utility function is supposed to represent. For example, the payoff functions for games in Chapter 6 are utility functions. The utility of a state in which White has won a game of chess is obviously high for the agent playing White, but low for the agent playing Black. Or again, some players (including the authors) might be happy with a draw against the world champion, whereas other players (including the former world champion) might not. There is no accounting for taste or preferences: you might think that an agent who prefers jalapefio bubble-gum ice cream to chocolate chocolate chip is odd or even misguided, but you could not say the agent is irrational. A utility function can even account for altruistic behavior, simply by including the welfare of others as one of the factors contributing to the agent’s own utility.

Preferences, as expressed by utilities, are combined with probabilities in the general theory of rational decisions called decision theory:

Decision theory = probability theory + utility theory .

1 This is quite different from a sentence’s becoming true or false as the world changes. Handling a changing world via probabilities requires the same kinds of mechanisms—situations, intervals, and events—that we used in Chapter 10 for logical representations. These mechanisms are discussed in Chapter 15.

466

Chapter 13. Uncertainty

BELIEF STATE

The fundamental idea of decision theory is that an agent is rational if and only if it chooses the action that yields the highest expected utility, averaged over all the possible outcomes of the action. This is called the principle of Maximum Expected Utility (MEU). We saw this principle in action in Chapter 6 when we touched briefly on optimal decisions in backgammon. We will see that it is in fact a completely general principle.

Design for a decision-theoretic agent

Figure 13.1 sketches the structure of an agent that uses decision theory to select actions. The agent is identical, at an abstract level, to the logical agent described in Chapter 7. The primary difference is that the decision-theoretic agent’s knowledge of the current state is uncertain; the agent’s belief state is a representation of the probabilities of all possible actual states of the world. As time passes, the agent accumulates more evidence and its belief state changes. Given the belief state, the agent can make probabilistic predictions of action outcomes and hence select the action with highest expected utility. This chapter and the next concentrate on the task of representing and computing with probabilistic information in general. Chapter 15 deals with methods for the specific tasks of representing and updating the belief state and predicting the environment. Chapter 16 covers utility theory in more depth, and Chapter 17 develops algorithms for making complex decisions.

function DT-AGENT( percept) returns an action static: belief\_state, probabilistic beliefs about the current state of the world action, the agent’s action

update belief \_state based on action and percept calculate outcome probabilities for actions,

given action descriptions and current belief\_state select action with highest expected utility

given probabilities of outcomes and utility information return action

Figure 13.1 A decision-theoretic agent that selects rational actions. The steps will be fleshed out in the next five chapters.

Now that we have set up the general framework for a rational agent, we will need a formal language for representing and reasoning with uncertain knowledge. Any notation for describing degrees of belief must be able to deal with two main issues: the nature of the sentences to which degrees of belief are assigned and the dependence of the degree of belief on the agent’s experience. The version of probability theory we present uses an extension of propositional

Basic Probability Notation 467

RANDOM VARIABLE

DOMAIN

BOOLEAN RANDOM VARIABLES

DISCRETE RANDOM VARIABLES

CONTINUOUS RANDOM VARIABLES

logic for its sentences. The dependence on experience is reflected in the syntactic distinction between prior probability statements, which apply before any evidence is obtained, and conditional probability statements, which include the evidence explicitly.

Propositions

Degrees of belief are always applied to propositions—assertions that such-and-such is the case. So far we have seen two formal languages—propositional logic and first-order logic— for stating propositions. Probability theory typically uses a language that is slightly more expressive than propositional logic. This section describes that language. (Section 14.6 discusses ways to ascribe degrees of belief to assertions in first-order logic.)

The basic element of the language is the random variable, which can be thought of as referring to a “part” of the world whose “status” is initially unknown. For example, Cavity might refer to whether my lower left wisdom tooth has a cavity. Random variables play a role similar to that of CSP variables in constraint satisfaction problems and that of proposition symbols in propositional logic. We will always capitalize the names of random variables. (However, we still use lowercase, single-letter names to represent an unknown random variable, for example: P(a)=1— P(~a).)

Each random variable has a domain of values that it can take on. For example, the domain of Cavity might be (true, false). (We will use lowercase for the names of values.) The simplest kind of proposition asserts that a random variable has a particular value drawn from its domain. For example, Cavity = true might represent the proposition that I do in fact have a cavity in my lower left wisdom tooth.

As with CSP variables, random variables are typically divided into three kinds, depending on the type of the domain:

¢ Boolean random variables, such as Cavity, have the domain (true, false). We will often abbreviate a proposition such as Cavity = true simply by the lowercase name cavity. Similarly, Cavity = false would be abbreviated by —cavity.

¢ Discrete random variables, which include Boolean random variables as a special case, take on values from a countable domain. For example, the domain of Weather might be (sunny, rainy, cloudy, snow). The values in the domain must be mutually exclusive and exhaustive. Where no confusion arises, we will use, for example, snow as an abbreviation for Weather = snow.

{> Continuous random variables take on values from the real numbers. The domain can be either the entire real line or some subset such as the interval [0,1]. For example, the proposition X = 4.02 asserts that the random variable X has the exact value 4.02. Propositions concerning continuous random variables can also be inequalities, such as X £4.02

With some exceptions, we will be concentrating on the discrete case.

Elementary propositions, such as Cavity = true and Toothache = false, can be combined to form complex propositions using all the standard logical connectives. For example,

2 One might expect the domain to be written as a set: { true, false}. We write it as a tuple because it will be convenient later to impose an ordering on the values.

468

Chapter 13. Uncertainty

ATOMIC EVENT

UNCONDITIONAL PRIOR PROBABILITY

Cavity = true A Toothache = false is a proposition to which one may ascribe a degree of (dis)belief. As explained in the previous paragraph, this proposition may also be written as cavity A —toothache.

Atomic events

The notion of an atomic event is useful in understanding the foundations of probability theory. An atomic event is a complete specification of the state of the world about which the agent is uncertain. It can be thought of as an assignment of particular values to all the variables of which the world is composed. For example, if my world consists of only the Boolean variables Cavity and Toothache, then there are just four distinct atomic events; the proposition Cavity = false A Toothache = true is one such event.

Atomic events have some important properties:

e They are mutually exclusive—at most one can actually be the case. For example, cavity A toothache and cavity N\ —~toothache cannot both be the case.

o The set of all possible atomic events is exhaustive—at least one must be the case. That is, the disjunction of all atomic events is logically equivalent to true.

e Any particular atomic event entails the truth or falsehood of every proposition, whether simple or complex. This can be seen by using the standard semantics for logical connectives (Chapter 7). For example, the atomic event cavity A —toothache entails the truth of cavity and the falsehood of cavity = toothache.

e Any proposition is logically equivalent to the disjunction of all atomic events that entail the truth of the proposition. For example, the proposition cavity is equivalent to disjunction of the atomic events cavity A toothache and cavity A toothache.

Exercise 13.4 asks you to prove some of these properties.

Prior probability

The unconditional or prior probability associated with a proposition a is the degree of belief accorded to it in the absence of any other information; it is written as P(a). For example, if the prior probability that I have a cavity is 0.1, then we would write

P(Cavity =true) = 0.1 or P(cavity) =0.1.

It is important to remember that P(a) can be used only when there is no other information. As soon as some new information is known, we must reason with the conditional probability of a given that new information. Conditional probabilities are covered in the next section. Sometimes, we will want to talk about the probabilities of all the possible values of a random variable. In that case, we will use an expression such as P( Weather), which denotes a vector of values for the probabilities of each individual state of the weather. Thus, instead

3 Many standard formulations of probability theory take atomic events, also known as sample points, as primitive and define a random variable as a function taking an atomic event as input and returning a value from the appropriate domain. Such an approach is perhaps more general, but also less intuitive.

Basic Probability Notation 469

PROBABILITY DISTRIBUTION

JOINT PROBABILITY DISTRIBUTION

FULL JOINT PROBABILITY DISTRIBUTION

PROBABILITY DENSITY FUNCTIONS

of writing the four equations

P( Weather = sunny) = 0.7 P(Weather = rain) = 0.2

P( Weather = cloudy) = 0.08 P( Weather = snow) = 0.02.

we may simply write P( Weather) = (0.7,0.2,0.08, 0.02) .

This statement defines a prior probability distribution for the random variable Weather.

We will also use expressions such as P( Weather, Cavity) to denote the probabilities of all combinations of the values of a set of random variables.\* In that case, P( Weather, Cavity) can be represented by a 4 x 2 table of probabilities. This is called the joint probability distribution of Weather and Cavity.

Sometimes it will be useful to think about the complete set of random variables used to describe the world. A joint probability distribution that covers this complete set is called the full joint probability distribution. For example, if the world consists of just the variables Cavity, Toothache, and Weather, then the full joint distribution is given by

P( Cavity, Toothache, Weather).

This joint distribution can be represented as a 2 x 2 x 4 table with 16 entries. A full joint distribution specifies the probability of every atomic event and is therefore a complete specification of one’s uncertainty about the world in question. We will see in Section 13.4 that any probabilistic query can be answered from the full joint distribution.

For continuous variables, it is not possible to write out the entire distribution as a table, because there are infinitely many values. Instead, one usually defines the probability that a random variable takes on some value x as a parameterized function of z. For example, let the random variable X denote tomorrow’s maximum temperature in Berkeley. Then the sentence

P(X =x) = U[18,26)(x)

expresses the belief that X is distributed uniformly between 18 and 26 degrees Celsius. (Several useful continuous distributions are defined in Appendix A.) Probability distributions for continuous variables are called probability density functions. Density functions differ in meaning from discrete distributions. For example, using the temperature distribution given earlier, we find that P(X =20.5) =U[18,26](20.5) =0.125/C. This does not mean that there’s a 12.5% chance that the maximum temperature will be exactly 20.5 degrees tomorrow; the probability that this will happen is of course zero. The technical meaning is that the probability that the temperature is in a small region around 20.5 degrees is equal, in the limit, to 0.125 divided by the width of the region in degrees Celsius:

lim P(20.5 < X < 20.5 + dz)/dz = 0.125/C . dzx—0 4 The general notational rule is that the distribution covers all values of the variables that are capitalized. Thus,

the expression P( Weather, cavity) is a four-element vector of probabilities for the conjunction of each weather type with Cavity = true.

470

Chapter 13. Uncertainty

CONDITIONAL PROBABILITY

POSTERIOR PROBABILITY

PRODUCT RULE

Some authors use different symbols for discrete distributions and density functions; we use P in both cases, since confusion seldom arises and the equations are usually identical. Note that probabilities are unitless numbers, whereas density functions are measured with a unit, in this case reciprocal degrees.

Conditional probability

Once the agent has obtained some evidence concerning the previously unknown random variables making up the domain, prior probabilities are no longer applicable. Instead, we use conditional or posterior probabilities. The notation used is P(a|b), where a and b are any propositions.> This is read as “the probability of a, given that all we know is b.” For example,

P(cavity|toothache) = 0.8

indicates that if a patient is observed to have a toothache and no other information is yet available, then the probability of the patient’s having a cavity will be 0.8. A prior probability, such as P(cavity), can be thought of as a special case of the conditional probability P(cavity| ), where the probability is conditioned on no evidence. Conditional probabilities can be defined in terms of unconditional probabilities. The defining equation is P(a nb) P(b) which holds whenever P(b) > 0. This equation can also be written as P(a Ab) = P(alb)P(b) which is called the product rule. The product rule is perhaps easier to remember: it comes from the fact that, for a and b to be true, we need b to be true, and we also need a to be true given b. We can also have it the other way around: P(a nb) = P(bla)P(a) . In some cases, it is easier to reason in terms of prior probabilities of conjunctions, but for the most part, we will use conditional probabilities as our vehicle for probabilistic inference. We can also use the P notation for conditional distributions. P(X|Y") gives the values of P(X =u;|Y =y;) for each possible i, j. As an example of how this makes our notation more concise, consider applying the product rule to each case where the propositions ¢ and b assert particular values of X and Y respectively. We obtain the following equations: P(X=21 AY =y1) = P(X =a, [V =y) P(Y = 1) P(X =m A Y =y9) = P(X =x1|Y =y2) P(Y =y2) .

P(alb) = (13.1)

We can combine all these into the single equation

P(X,Y)=P(X|Y)P(Y). Remember that this denotes a set of equations relating the corresponding individual entries in the tables, not a matrix multiplication of the tables.

5 The “|” operator has the lowest possible precedence, so P{a A blc V d) means P{(a Ab)|(cV d)).

Section 13.3.

The Axioms of Probability 471

It is tempting, but wrong, to view conditional probabilities as if they were logical implications with uncertainty added. For example, the sentence P(a|b) = 0.8 cannot be interpreted to mean “whenever b holds, conclude that P(a) is 0.8.” Such an interpretation would be wrong on two counts: first, P(a) always denotes the prior probability of a, not the posterior probability given some evidence; second, the statement P(a|b) = 0.8 is immediately relevant just when b is the only available evidence. When additional information c is available, the degree of belief in a is P(a|b A ¢), which might have little relation to P(al|b). For example, ¢ might tell us directly whether a is true or false. If we examine a patient who complains of toothache, and discover a cavity, then we have additional evidence cavity, and we conclude (trivially) that P(cavity|toothache A cavity) = 1.0.

KOLMOGOROV'S AXIOMS

So far, we have defined a syntax for propositions and for prior and conditional probability statements about those propositions. Now we must provide some sort of semantics for probability statements. We begin with the basic axioms that serve to define the probability scale and its endpoints:

1. All probabilities are between 0 and 1. For any proposition a, 0

2. Necessarily true (i.e., valid) propositions have probability 1, and necessarily false (i.e., unsatisfiable) propositions have probability 0.

P(true) =1 P(false) =0.

Next, we need an axiom that connects the probabilities of logically related propositions. The simplest way to do this is to define the probability of a disjunction as follows:

3. The probability of a disjunction is given by P(aVb)= Pa) +P(b)— Planbd).

This rule is easily remembered by noting that the cases where a holds, together with the cases where b holds, certainly cover all the cases where a V b holds; but summing the two sets of cases counts their intersection twice, so we need to subtract P(a A b).

These three axioms are often called Kolmogorov’s axioms in honor of the Russian mathematician Andrei Kolmogorov, who showed how to build up the rest of probability theory from this simple foundation. Notice that the axioms deal only with prior probabilities rather than conditional probabilities; this is because we have already defined the latter in terms of the former via Equation (13.1).

472 Chapter 13. Uncertainty

WHERE DO PROBABILITIES COME FROM?

There has been endless debate over the source and status of probability numbers. The frequentist position is that the numbers can come only from experiments: if we test 100 people and find that 10 of them have a cavity, then we can say that the probability of a cavity is approximately 0.1. In this view, the assertion “the probability of a cavity is 0.1” means that 0.1 is the fraction that would be observed in the limit of infinitely many samples. From any finite sample, we can estimate the true fraction and also calculate how accurate our estimate is likely to be.

The objectivist view is that probabilities are real aspects of the universe— propensities of objects to behave in certain ways—rather than being just descrip

L tions of an observer’s degree of belief. For example, that a fair coin comes up heads with probability 0.5 is a propensity of the coin itself. In this view, frequentist measurements are attempts to observe these propensities. Most physicists agree that quantum phenomena are objectively probabilistic, but uncertainty at the macroscopic scale—e.g., in coin tossing—usually arises from ignorance of initial conditions and does not seem consistent with the propensity view.

The subjectivist view describes probabilities as a way of characterizing an agent’s beliefs, rather than as having any external physical significance. This allows the doctor or analyst to make the numbers up—to say, “In my opinion, [ expect the probability of a cavity to be about 0.1.” Several more reliable techniques, such as the betting systems described on page 474, have also been developed for eliciting probability assessments from humans.

In the end, even a strict frequentist position involves subjective analysis, so the difference probably has little practical importance. The reference class problem illustrates the intrusion of subjectivity. Suppose that a frequentist doctor wants to know the chances that a patient has a particular disease. The doctor wants to consider other patients who are similar in important ways—age, symptoms, perhaps sex—and see what proportion of them had the disease. But if the doctor considered everything that is known about the patient—weight to the nearest gram, hair color, mother’s maiden name, etc.—the result would be that there are no other patients who are exactly the same and thus no reference class from which to collect experimental data. This has been a vexing problem in the philosophy of science.

Laplace’s principle of indifference (1816) states that propositions that are syntactically “symmetric” with respect to the evidence should be accorded equal probability. Various refinements have been proposed, culminating in the attempt by Carnap and others to develop a rigorous inductive logic, capable of computing the correct probability for any proposition from any collection of observations. Currently, it is believed that no unique inductive logic exists; rather, any such logic rests on a subjective prior probability distribution whose effect is diminished as more observations are collected.

Section 13.3.

The Axioms of Probability 473

Using the axioms of probability

We can derive a variety of useful facts from the basic axioms. For example, the familiar rule for negation follows by substituting —a for b in axiom 3, giving us: P(aV —a) = P(a) + P(-a) — P(a A —a) (by axiom 3 with b= —a) P(true) = P(a)+ P(—a) — P(false) (by logical equivalence)

1 = P(a)+ P(—a) (by axiom 2)

P(-a) = 1— P(a) (by algebra). The third line of this derivation is itself a useful fact and can be extended from the Boolean case to the general discrete case. Let the discrete variable D have the domain (dy, ...,d,).

Then it is easy to show (Exercise 13.2) that n > P(D=d;)=1. i=1

That is, any probability distribution on a single variable must sum to 1.% It is also true that any joint probability distribution on any set of variables must sum to 1: this can be seen simply by creating a single megavariable whose domain is the cross product of the domains of the the original variables.

Recall that any proposition a is equivalent to the disjunction of all the atomic events in which a holds; call this set of events e(a). Recall also that atomic events are mutually exclusive, so the probability of any conjunction of atomic events is zero, by axiom 2. Hence, from axiom 3, we can derive the following simple relationship: The probability of a proposition is equal to the sum of the probabilities of the atomic events in which it holds; that is,

Pla)= > Ple). (13.2)

e; €e(a) This equation provides a simple method for computing the probability of any proposition, given a full joint distribution that specifies the probabilities of all atomic events. (See Section 13.4.) In subsequent sections we will derive additional rules for manipulating probabilities. First, however, we will examine the foundation for the axioms themselves.

Why the axioms of probability are reasonable

The axioms of probability can be seen as restricting the set of probabilistic beliefs that an agent can hold. This is somewhat analogous to the logical case, where a logical agent cannot simultaneously believe A, B, and =(A A B), for example. There is, however, an additional complication. In the logical case, the semantic definition of conjunction means that at least one of the three beliefs just mentioned must be false in the world, so it is unreasonable for an agent to believe all three. With probabilities, on the other hand, statements refer not to the world directly, but to the agent’s own state of knowledge. Why, then, can an agent not hold the following set of beliefs, which clearly violates axiom 3? Pla) =04 PlaAnb)=0.0

P(b)=0.3 P(aVvb)=0.8 (13.3)

6 For continuous variables, the summation is replaced by an integral: {~ P(X =z)dz = 1. p y Ce

474

Chapter 13. Uncertainty

This kind of question has been the subject of decades of intense debate between those who advocate the use of probabilities as the only legitimate form for degrees of belief and those who advocate alternative approaches. Here, we give one argument for the axioms of probability, first stated in 1931 by Bruno de Finetti.

The key to de Finetti’s argument is the connection between degree of belief and actions. The idea is that if an agent has some degree of belief in a proposition a, then the agent should be able to state odds at which it is indifferent to a bet for or against a. Think of it as a game between two agents: Agent 1 states “my degree of belief in event a is 0.4.” Agent 2 is then free to choose whether to bet for or against a, at stakes that are consistent with the stated degree of belief. That is, Agent 2 could choose to bet that a will occur, betting $4 against Agent 1's $6. Or Agent 2 could bet $6 against $4 that A will not occur.” If an agent’s degrees of belief do not accurately reflect the world, then you would expect that it would tend to lose money over the long run to an opposing agent whose beliefs more accurately reflect the state of the world.

But de Finetti proved something much stronger: If Agent 1 expresses a set of degrees of belief that violate the axioms of probability theory then there is a combination of bets by Agent 2 that guarantees that Agent 1 will lose money every time. So if you accept the idea that an agent should be willing to “put its money where its probabilities are,” then you should accept that it is irrational to have beliefs that violate the axioms of probability.

One might think that this betting game is rather contrived. For example, what if one refuses to bet? Does that end the argument? The answer is that the betting game is an abstract model for the decision-making situation in which every agent is unavoidably involved at every moment. Every action (including inaction) is a kind of bet, and every outcome can be seen as a payoff of the bet. Refusing to bet is like refusing to allow time to pass.

We will not provide the proof of de Finetti’s theorem, but we will show an example. Suppose that Agent 1 has the set of degrees of belief from Equation (13.3). Figure 13.2 shows that if Agent 2 chooses to bet $4 on a, $3 on b, and $2 on —(a V b), then Agent 1 always loses money, regardless of the outcomes for a and b.

[]

Figure 13.2 Because Agent 1 has inconsistent beliefs, Agent 2 is able to devise a set of

bets that guarantees a loss for Agent 1, no matter what the outcome of ¢ and b.

7 One might argue that the agent’s preferences for different bank balances are such that the possibility of losing $1 is not counterbalanced by an equal possibility of winning $1. One possible response is to make the bet amounts small enough to avoid this problem. Savage’s analysis (1954) circumvents the issue altogether.

Section 13.4,

Inference Using Full Joint Distributions 475

Other strong philosophical arguments have been put forward for the use of probabilities, most notably those of Cox (1946) and Carnap (1950). The world being the way it is, however, practical demonstrations sometimes speak louder than proofs. The success of reasoning systems based on probability theory has been much more effective in making converts, We now look at how the axioms can be deployed to make inferences.

PROBABILISTIC INFERENCE

[]

MARGINAL PROBABILITY

MARGINALIZATION

In this section we will describe a simple method for probabilistic inference—that is, the computation from observed evidence of posterior probabilities for query propositions. We will use the full joint distribution as the “knowledge base” from which answers to all questions may be derived. Along the way we will also introduce several useful techniques for manipulating equations involving probabilities.

We begin with a very simple example: a domain consisting of just the three Boolean variables Toothache, Cavity, and Catch (the dentist’s nasty steel probe catches in my tooth). The full joint distribution is a 2 x 2 x 2 table as shown in Figure 13.3.

toothache —toothache catch —catch catch catch cavity 0.108 0.012 0.072 0.008 — cavity 0.016 0.064 0.144 0.576 Figure 13.3 A full joint distribution for the Toothache, Cavity, Catch world.

Notice that the probabilities in the joint distribution sum to 1, as required by the axioms of probability. Notice also that Equation (13.2) gives us a direct way to calculate the probability of any proposition, simple or complex: We simply identify those atomic events in which the proposition is true and add up their probabilities. For example, there are six atomic events in which cavity V toothache holds:

P(cavity V toothache) = 0.108 + 0.012 + 0.072 + 0.008 + 0.016 + 0.064 = 0.28 , One particularly common task is to extract the distribution over some subset of variables or a single variable. For example, adding the entries in the first row gives the unconditional or marginal probability? of cavity:

P(cavity) = 0.108 + 0.012 + 0.072 + 0.008 = 0.2. This process is called marginalization, or summing out—because the variables other than

Cavity are summed out. We can write the following general marginalization rule for any sets of variables Y and Z:

P(Y)=) P(Y,z). (13.4)

8 So called because of a common practice among actuaries of writing the sums of observed frequencies in the margins of insurance tables.

476 Chapter 13. Uncertainty That is, a distribution over Y can be obtained by summing out all the other variables from any joint distribution containing Y. A variant of this rule involves conditional probabilities instead of joint probabilities, using the product rule:

P(Y) => P(Y|z)P(z). (13.5)

z

CONDITIONING This rule is called conditioning. Marginalization and conditioning will turn out to be useful rules for all kinds of derivations involving probability expressions.

In most cases, we will be interested in computing conditional probabilities of some variables, given evidence about others. Conditional probabilities can be found by first using Equation (13.1) to obtain an expression in terms of unconditional probabilities and then evaluating the expression from the full joint distribution. For example, we can compute the probability of a cavity, given evidence of a toothache, as follows:

[]

P (cavity A toothache) p ity| toothache) = ——7———F—— (cavity|toothache) P(toothache) \_ 0.108 + 0.012 ~ 06 © 0.108 +0.012 + 0.016 + 0.064 Just to check, we can also compute the probability that there is no cavity, given a toothache: P(— cavity A toothache) P(—cavity|toothache) = ——————F—— (eavity| toothache) P(toothache) \_ 0.016 + 0.064 04 0.108 + 0.012 4 0.016 + 0.064 Notice that in these two calculations the term 1/P (toothache) remains constant, no matter NORMALIZATION which value of Cavity we calculate. In fact, it can be viewed as a normalization constant for

the distribution P( Cavity| toothache), ensuring that it adds up to 1. Throughout the chapters dealing with probability, we will use « to denote such constants. With this notation, we can write the two preceding equations in one:

P(Cavity| toothache) = a P( Cavity, toothache) = a [P(Cavity, toothache, catch) + P( Cavity, toothache, —catch)) = @[(0.108,0.016) + (0.012, 0.064)] = a (0.12,0.08) = (0.6, 0.4) .

Normalization will turn out to be a useful shortcut in many probability calculations.

From the example, we can extract a general inference procedure. We will stick to the case in which the query involves a single variable. We will need some notation: let X be the query variable (Cavity in the example), let E be the set of evidence variables (just Toothache in the example), let e be the observed values for them, and let Y be the remaining unobserved variables (just Catch in the example). The query is P(X |e) and can be evaluated as

P(Xle) = aP(X,e) =a) P(X,ey), (13.6) y

where the summation is over all possible ys (i.e., all possible combinations of values of the unobserved variables Y). Notice that together the variables X, E, and Y constitute the complete set of variables for the domain, so P(X, e,y) is simply a subset of probabilities from the full joint distribution. The algorithm is shown in Figure 13.4. Tt loops over the values

Section 13.5.

Independence 477

function ENUMERATE-JOINT-ASK(X, e, P) returns a distribution over X inputs: X, the query variable e, observed values for variables E P, a joint distribution on variables {X} UE UY /\* Y = hidden variables \* /

Q(X) — a distribution over X, initially empty for each value x; of X do

Q(z;) « ENUMERATE-JOINT(z;,€,Y,[],P) return NORMALIZE(Q( X))

function ENUMERATE-JOINT(x, @, vars, values, P) returns a real number if EMPTY ?(vars) then return P(z, e, values) Y «— FIRST(wars) return > ENUMERATE-JOINT(z, e, REST(vars), [y| values}, P)

Figure 13.4 An algorithm for probabilistic inference by enumeration of the entries in a

full joint distribution.

of X and the values of ¥ to enumerate all possible atomic events with e fixed, adds up their probabilities from the joint table, and normalizes the results.

Given the full joint distribution to work with, ENUMERATE-JOINT-ASK is a complete algorithm for answering probabilistic queries for discrete variables. It does not scale well, however: For a domain described by n Boolean variables, it requires an input table of size O(2") and takes O(2") time to process the table. In a realistic problem, there might be hundreds or thousands of random variables to consider, not just three. It quickly becomes completely impractical to define the vast numbers of probabilities required—the experience needed in order to estimate each of the table entries separately simply cannot exist.

For these reasons, the full joint distribution in tabular form is not a practical tool for building reasoning systems (although the historical notes at the end of the chapter includes one real-world application of this method). Instead, it should be viewed as the theoretical foundation on which more effective approaches may be built. The remainder of this chapter introduces some of the basic ideas required in preparation for the development of realistic systems in Chapter 14.

Let us expand the full joint distribution in Figure 13.3 by adding a fourth variable, Weather. The full joint distribution then becomes P( Toothache, Catch, Cavity, Weather), which has 32 entries (because Weather has four values). It contains four “editions” of the table shown in Figure 13.3, one for each kind of weather. It seems natural to ask what relationship these editions have to each other and to the original three-variable table. For example, how are P(toothache, catch, cavity, Weather = cloudy) and P(toothache, catch, cavity) related?

478

Chapter 13. Uncertainty

INDEPENDENCE

One way to answer this question is to use the product rule: P (toothache, catch, cavity, Weather = cloudy) = P(Weather = cloudy| toothache, catch, cavity) P (toothache, catch, cavity) .

Now, unless one is in the deity business, one should not imagine that one’s dental problems influence the weather. Therefore, the following assertion seems reasonable:

P( Weather = cloudy|toothache, catch, cavity) = P( Weather = cloudy) . (13.7) From this, we can deduce

P(toothache, catch, cavity, Weather = cloudy) = P(Weather = cloudy) P(toothache, catch, cavity) .

A similar equation exists for every entry in P( Toothache, Catch, Cavity, Weather). In fact, we can write the general equation

P( Toothache, Catch, Cavity, Weather) = P( Toothache, Catch, Cavity)P( Weather) . Thus, the 32-element table for four variables can be constructed from one 8-element table and one four-element table. This decomposition is illustrated schematically in Figure 13.5(a).

The property we used in writing Equation (13.7) is called independence (also marginal independence and absolute independence). In particular, the weather is independent of one’s dental problems. Independence between propositions a and b can be written as

P(alb)=P(a) or P(bla)=P(b) or Planb)=P(a)P(b). (13.8) All these forms are equivalent (Exercise 13.7). Independence between variables X and Y can be written as follows (again, these are all equivalent):

P(X|Y)=P(X) or P(Y|X)=P(Y) or P(X,Y)=P(X)P(Y).

[]

Toothache Figure 13.5 Two examples of factoring a large joint distribution into smaller distributions, using absolute independence. (a) Weather and dental problems are independent. (b) Coin flips are independent.

Section 13.6.

Bayes’ Rule and Its Use 479

Independence assertions are usually based on knowledge of the domain. As we have seen, they can dramatically reduce the amount of information necessary to specify the full joint distribution. If the complete set of variables can be divided into independent subsets, then the full joint can be factored into separate joint distributions on those subsets. For example, the joint distribution on the outcome of n independent coin flips, P(C1,..., Cy), can be represented as the product of n single-variable distributions P(C;). In a more practical vein, the independence of dentistry and meteorology is a good thing, because otherwise the practice of dentistry might require intimate knowledge of meteorology and vice versa.

When they are available, then, independence assertions can help in reducing the size of the domain representation and the complexity of the inference problem. Unfortunately, clean separation of entire sets of variables by independence is quite rare. Whenever a connection, however indirect, exists between two variables, independence will fail to hold. Moreover, even independent subsets can be quite large—for example, dentistry might involve dozens of diseases and hundreds of symptoms, all of which are interrelated. To handle such problems, we will need more subtle methods than the straightforward concept of independence.

BAYES’ RULE

On page 470, we defined the product rule and pointed out that it can be written in two forms because of the commutativity of conjunction:

Pa Ab) = P(alb)P(b) P(a Ab) = P(bla)P(a) .

Equating the two right-hand sides and dividing by P(a), we get

P(alb)P(b)

P(bla) = 5;

(13.9) This equation is known as Bayes’ rule (also Bayes’ law or Bayes’ theorem).” This simple equation underlies all modern Al systems for probabilistic inference. The more general case of multivalued variables can be written in the P notation as

P(X|Y)P(Y)

P(Y|X) = POX)

where again this is to be taken as representing a set of equations, each dealing with specific values of the variables. We will also have occasion to use a more general version conditionalized on some background evidence e:

P(X[Y,e)P(Y]e)

P(Y|X,e) = P(X)

(13.10)

9 According to rule 1 on page 1 of Strunk and White’s The Elements of Style, it should be Bayes’s rather than Bayes’. The latter is, however, more commonly used.

480

Chapter 13. Uncertainty

Applying Bayes’ rule: The simple case

On the surface, Bayes’ rule does not seem very useful. It requires three terms—a conditional probability and two unconditional probabilities—just to compute one conditional probability.

Bayes’ rule is useful in practice because there are many cases where we do have good probability estimates for these three numbers and need to compute the fourth. In a task such as medical diagnosis, we often have conditional probabilities on causal relationships and want to derive a diagnosis. A doctor knows that the disease meningitis causes the patient to have a stiff neck, say, 50% of the time. The doctor also knows some unconditional facts: the prior probability that a patient has meningitis is 1/50,000, and the prior probability that any patient has a stiff neck is 1/20. Letting s be the proposition that the patient has a stiff neck and m be the proposition that the patient has meningitis, we have

P(slm) = 0.5 P(m) = 1/50000 P(s) = 1/20 P(m]s) = en) \_ 05 sh — 0.0002.

That is, we expect only 1 in 5000 patients with a stiff neck to have meningitis. Notice that, even though a stiff neck is quite strongly indicated by meningitis (with probability 0.5), the probability of meningitis in the patient remains small. This is because the prior probability on stiff necks is much higher than that on meningitis.

Section 13.4 illustrated a process by which one can avoid assessing the probability of the evidence (here, P(s)) by instead computing a posterior probability for each value of the query variable (here, m and —m) and then normalizing the results. The same process can be applied when using Bayes’ rule. We have

P(M|s) = a (P(s|m)P(m), P(s|—=m)P(—-m)) .

Thus, in order to use this approach we need to estimate P(s|—m) instead of P(s). There is no free lunch—sometimes this is easier, sometimes it is harder. The general form of Bayes’ rule with normalization is

P(Y|X) = aP(X|Y)P(Y), (13.11)

where « is the normalization constant needed to make the entries in P(Y'|X') sum to 1.

One obvious question to ask about Bayes’ rule is why one might have available the conditional probability in one direction, but not the other. In the meningitis domain, perhaps the doctor knows that a stiff neck implies meningitis in 1 out of 5000 cases; that is, the doctor has quantitative information in the diagnostic direction from symptoms to causes. Such a doctor has no need to use Bayes’ rule. Unfortunately, diagnostic knowledge is often more fragile than causal knowledge. If there is a sudden epidemic of meningitis, the unconditional probability of meningitis, P(m), will go up. The doctor who derived the diagnostic probability P(m/|s) directly from statistical observation of patients before the epidemic will have no idea how to update the value, but the doctor who computes P(m|s) from the other three values will see that P(m/|s) should go up proportionately with P(m). Most importantly, the

Section 13.6. Bayes’ Rule and Its Use 481

causal information P(s|m) is unaffected by the epidemic, because it simply reflects the way meningitis works. The use of this kind of direct causal or model-based knowledge provides the crucial robustness needed to make probabilistic systems feasible in the real world.

Using Bayes’ rule: Combining evidence

We have seen that Bayes’ rule can be useful for answering probabilistic queries conditioned on one piece of evidence—for example, the stiff neck. In particular, we have argued that probabilistic information is often available in the form P(effect|cause). What happens when we have two or more pieces of evidence? For example, what can a dentist conclude if her nasty steel probe catches in the aching tooth of a patient? If we know the full joint distribution (Figure 13.3), one can read off the answer:

P(Cavity|toothache A catch) = a (0.108, 0.016) ~ (0.871,0.129) .

We know, however, that such an approach will not scale up to larger numbers of variables. We can try using Bayes’ rule to reformulate the problem:

P(Cavity|toothache A catch) = aP(toothache A catch|Cavity)P( Cavity) . (13.12)

For this reformulation to work, we need to know the conditional probabilities of the conjunction toothache A catch for each value of Cavity. That might be feasible for just two evidence variables, but again it will not scale up. If there are n possible evidence variables (X rays, diet, oral hygiene, etc.), then there are 2" possible combinations of observed values for which we would need to know conditional probabilities. We might as well go back to using the full joint distribution. This is what first led researchers away from probability theory toward approximate methods for evidence combination that, while giving incorrect answers, require fewer numbers to give any answer at all.

Rather than taking this route, we need to find some additional assertions about the domain that will enable us to simplify the expressions. The notion of independence in Section 13.5 provides a clue, but needs refining. Tt would be nice if Toothache and Catch were independent, but they are not: if the probe catches in the tooth, it probably has a cavity and that probably causes a toothache. These variables are independent, however, given the presence or the absence of a cavity. Each is directly caused by the cavity, but neither has a direct effect on the other: toothache depends on the state of the nerves in the tooth, whereas the probe’s accuracy depends on the dentist’s skill, to which the toothache is irrelevant. '? Mathematically, this property is written as

P(toothache A catch| Cavity) = P(toothache| Cavity)P(catch| Cavity) . (13.13)

OD OA This equation expresses the conditional independence of toothache and catch given Cavity. We can plug it into Equation (13.12) to obtain the probability of a cavity:

P(Cavity|toothache A catch) = a P(toothache| Cavity)P(catch| Cavity )P( Cavity).

Now the information requirements are the same as for inference using each piece of evidence separately: the prior probability P( Cavity) for the query variable and the conditional probability of each effect, given its cause.

10 We assume that the patient and dentist are distinct individuals.

Chapter 13. Uncertainty

SEPARATION

NAIVE BAYES

IDIOT BAYES

The general definition of conditional independence of two variables X and Y', given a third variable Z is

P(X, YZ) =P X|Z2)P(Y |Z). In the dentist domain, for example, it seems reasonable to assert conditional independence of the variables Toothache and Catch, given Cavity:

P( Toothache, Catch| Cavity) = P(Toothache| Cavity )P(Catch| Cavity) . (13.14)

Notice that this assertion is somewhat stronger than Equation (13.13), which asserts independence only for specific values of Toothache and Catch. As with absolute independence in Equation (13.8), the equivalent forms

P(X|Y,Z)=P(X|Z) and P(Y|X,Z)=P(Y |Z) can also be used.

Section 13.5 showed that absolute independence assertions allow a decomposition of the full joint distribution into much smaller pieces. It turns out that the same is true for conditional independence assertions. For example, given the assertion in Equation (13.14), we can derive a decomposition as follows:

P( Toothache, Catch, Cavity) = P(Toothache, Catch| Cavity)P( Cavity) (product rule) = P(Toothache|Cavity)P( Catch| Cavity)P( Cavity) [using (13.14)].

In this way, the original large table is decomposed into three smaller tables. The original table has seven independent numbers (2° — 1, because the numbers must sum to 1). The smaller tables contain five independent numbers (2 x (2! — 1) for each conditional probability distribution and 2! — 1 for the prior on Cavity). This might not seem to be a major triumph, but the point is that, for n symptoms that are all conditionally independent given Cavity, the size of the representation grows as O(n) instead of O(2"). Thus, conditional independence assertions can allow probabilistic systems to scale up; moreover, they are mich more commonly available than absolute independence assertions. Conceptually, Cavity separates Toothache and Catch because it is a direct cause of both of them. The decomposition of large probabilistic domains into weakly connected subsets via conditional independence is one of the most important developments in the recent history of Al

The dentistry example illustrates a commonly occurring pattern in which a single cause directly influences a number of effects, all of which are conditionally independent, given the cause. The full joint distribution can be written as

P(Cause, Effect, ..., Effect) = P(Cause) II P(Effect;| Cause) .

Such a probability distribution is called a naive Bayes model—“naive” because it is often used (as a simplifying assumption) in cases where the “effect” variables are not conditionally independent given the cause variable. (The naive Bayes model is sometimes called a Bayesian classifier, a somewhat careless usage that has prompted true Bayesians to call it the idiot Bayes model.) In practice, naive Bayes systems can work surprisingly well, even when the independence assumption is not true. Chapter 20 describes methods for learning naive Bayes distributions from observations.

Section 13.7. The Wumpus World Revisited 483

13.7 THE WUMPUS WORLD REVISITED

We can combine many of the ideas in this chapter to solve probabilistic reasoning problems in the wumpus world. (See Chapter 7 for a complete description of the wumpus world.) Uncertainty arises in the wumpus world because the agent’s sensors give only partial, local information about the world. For example, Figure 13.6 shows a situation in which each of the three reachable squares—{1,3], [2,2], and [3,1]—might contain a pit. Pure logical inference can conclude nothing about which square is most likely to be safe, so a logical agent might be forced to choose randomly. We will see that a probabilistic agent can do much better than the logical agent.

Our aim will be to calculate the probability that each of the three squares contains a pit. (For the purposes of this example, we will ignore the wumpus and the gold.) The relevant properties of the wumpus world are that (1) a pit causes breezes in all neighboring squares, and (2) each square other than [1,1] contains a pit with probability 0.2. The first step is to identify the set of random variables we need:

¢ As in the propositional logic case, we want one Boolean variable F;; for each square, which is true iff square [Z, j] actually contains a pit. e We also have Boolean variables B;; that are true iff square [, j] is breezy; we include these variables only for the observed squares—in this case, [1,1], [1,2], and [2,1]. The next step is to specify the full joint distribution, P(P, 1, ..., Py 4, B11, B12, B2,1). Applying the product rule, we have P(Pi1,...,Py4,B11,B12,B21) = P(B11,B12,Bai | Pip, Pia)P(Pra,..., Pag).

Figure 13.6 (a) After finding a breeze in both [1,2] and [2,1], the agent is stuck—there is no safe place to explore. (b) Division of the squares into Known, Fringe, and Other, for a . query about [1,3].

484

Chapter 13. Uncertainty

This decomposition makes it very easy to see what the joint probability values should be. The first term is the conditional probability of a breeze configuration, given a pit configuration; this is 1 if the breezes are adjacent to the pits and 0 otherwise. The second term is the prior probability of a pit configuration. Each square contains a pit with probability 0.2, independently of the other squares; hence,

44 P(Pii,.... Pq) = J] P(Piy)(13.15) ij=11 For a configuration with n pits, this is just 0.2" x 0.816—",

In the situation in Figure 13.6(a), the evidence consists of the observed breeze (or its absence) in each square that is visited, combined with the fact that each such square contains no pit. We'll abbreviate these facts as b= —by 1Ab1 2Ab2 1 and known = —p1 1 A—p12A—Pa 1. We are interested in answering queries such as P(P; 3|known, b): how likely is it that [1,3] contains a pit, given the observations so far?

To answer this query, we can follow the standard approach suggested by Equation (13.6) and implemented in the ENUMERATE-JOINTASK, namely, summing over entries from the full joint distribution. Let Unknown be a composite variable consisting of the F; ; variables for squares other than the Known squares and the query square [1,3]. Then, by Equation (13.6), we have

P(Pyslknown,b) =a > P(Py3, unknown, known,b). unknown

The full joint probabilities have already been specified, so we are done—that is, unless we care about computation. There are 12 unknown squares; hence the summation contains 212 — 4096 terms. In general, the summation grows exponentially with the number of squares.

Intuition suggests that we are missing something here. Surely, one might ask, aren’t the other squares irrelevant? The contents of [4,4] don’t affect whether [1,3] has a pit! Indeed, this intuition is correct. Let Fringe be the variables (other than the query variable) that are adjacent to visited squares, in this case just [2,2] and [3,1]. Also, let Other be the variables for the other unknown squares; in this case, there are 10 other squares, as shown in Figure 13.6(b). The key insight is that the observed breezes are conditionally independent of the other variables, given the known, fringe, and query variables. The rest is, as they say, a small matter of algebra.

To use the insight, we manipulate the query formula into a form in which the breezes are conditioned on all the other variables, and then we simplify using conditional independence:

P(P, 3|known, b) = > P(b| P13, known, unknown )P(Py 3, known, unknown)

unknoun

(by the product rule) = a > > P(blknown, Py 3, fringe, other)P(P 3, known, fringe, other)

fringe other

= a > > P(blknown, Py 3, fringe)P(Py 3, known, fringe, other)

fringe other

Section 13.7.

The Wumpus World Revisited

15 B] 12 13] 1.3 112g |E8 12, 2s [BB 12 [BR 2 (28 OK OK OK OK OK Lian, a 11721, | 121 127 1\_ 8N OK OK OK OK K | OK OK | OK OK = OK 0.2x0.2=0.04 0.2x0.8=0.16 0.8x0.2=0.16 02x02=0.04 02x0.8=0.16 (a) (b) Figure 13.7 Consistent models for the fringe variables Po and Ps 1, showing P(fringe) for each model: (a) three models with P; 3 = true showing two or three pits, and (b) two models with P; 3 = false showing one or two pits.

where the final step uses conditional independence. Now, the first term in this expression does not depend on the other variables, so we can move the summation inwards:

P(P, 3|known, b) = « 3 P(b|known, Py 3, fringe) > P(P, 3, known, fringe, other) .

fringe other By independence, as in Equation (13.15), the prior term can be factored, and then the terms can be reordered: P(P, 3|known, b)

= « > P(b|knoun, Py 3, fringe) > P(P, 3) P (known) P (fringe) P(other)

fringe other = a P(known)P(P 3) > P(b|known, Py 3, fringe) P(fringe) > P(other) fringe other = o' P(P 3) 3 P(blknown, Py 3, fringe) P(fringe) ,

fringe where the last step folds P(known) into the normalizing constant and uses the fact that > other P(other) equals 1. Now), there are just four terms in the summation over the fringe variables 1% 5 and P31. The use of independence and conditional independence has completely eliminated the other squares from consideration. Notice that the expression P(b|known, Py 3, fringe) is 1 when the fringe is consistent with the breeze observations and 0 otherwise. Thus, for each value of Py 3, we sum over the logical models for the fringe variables that are consistent with the known facts. (Compare with the enumeration over models in Figure 7.5.) The models and their associated prior probabilities—P( fringe)—are shown in Figure 13.7. We have

P( Py 3|known, b) = o/ (0.2(0.04 + 0.16 + 0.16), 0.8(0.04 + 0.16)) ~ (0.31, 0.69) .

That is, [1,3] (and [3,1] by symmetry) contains a pit with roughly 31% probability. A similar calculation, which the reader might wish to perform, shows that [2,2] contains a pit with roughly 86% probability. The wumpus agent should definitely avoid [2,2]!

What this section has shown is that even seemingly complicated problems can be formulated precisely in probability theory and solved using simple algorithms. To get efficient

486

Chapter 13. Uncertainty

Uncertainty arises because of both laziness and ignorance. It is inescapable in complex, dynamic, or inaccessible worlds.

Uncertainty means that many of the simplifications that are possible with deductive inference are no longer valid.

Probabilities express the agent's inability to reach a definite decision regarding the truth of a sentence. Probabilities summarize the agent’s beliefs.

Basic probability statements include prior probabilities and conditional probabilities over simple and complex propositions.

The full joint probability distribution specifies the probability of each complete assignment of values to random variables. It is usually too large to create or use in its explicit form.

The axioms of probability constrain the possible assignments of probabilities to propositions. An agent that violates the axioms will behave irrationally in some circumstances.

When the full joint distribution is available, it can be used to answer queries simply by adding up entries for the atomic events corresponding to the query propositions.

Absolute independence between subsets of random variables might allow the full joint distribution to be factored into smaller joint distributions. This could greatly reduce complexity, but seldom occurs in practice.

Bayes’ rule allows unknown probabilities to be computed from known conditional probabilities, usually in the causal direction. Applying Bayes’ rule with many pieces of evidence will in general run into the same scaling problems as does the full joint distribution.

Conditional independence brought about by direct causal relationships in the domain might allow the full joint distribution to be factored into smaller, conditional distributions. The naive Bayes model assumes the conditional independence of all effect variables, given a single cause variable, and grows linearly with the number of effects.

A wumpus-world agent can calculate probabilities for unobserved aspects of the world and use them to make better decisions than a purely logical agent makes.

Section 13.8. Summary 487

BIBLIOGRAPHICAL AND HISTORICAL NOTES

Although games of chance date back at least to around 300 B.C., the mathematical analysis of odds and probability appears to be much more recent. Some work done by Mahaviracarya in India is dated to roughly the ninth century A.D. In Europe, the first attempts date only to the Italian Renaissance, beginning around 1500 A.D. The first significant systematic analyses were produced by Girolamo Cardano around 1565, but they remained unpublished until 1663. By that time, the discovery by Blaise Pascal (in correspondence with Pierre Fermat in 1654) of a systematic way of calculating probabilities had for the first time established probability as a mathematical discipline. The first published textbook on probability was De Ratiociniis in Ludo Aleae (Huygens, 1657). Pascal also introduced conditional probability, which is covered in Huygens’s textbook. The Rev. Thomas Bayes (1702-1761) introduced the rule for reasoning about conditional probabilities that was named after him. It was published posthumously (Bayes, 1763). Kolmogorov (1950, first published in German in 1933) presented probability theory in a rigorously axiomatic framework for the first time. Rényi (1970) later gave an axiomatic presentation that took conditional probability, rather than absolute probability, as primitive,

Pascal used probability in ways that required both the objective interpretation, as a property of the world based on symmetry or relative frequency, and the subjective interpretation, based on degree of belief—the former in his analyses of probabilities in games of chance, the latter in the famous “Pascal’s wager” argument about the possible existence of God. However, Pascal did not clearly realize the distinction between these two interpretations. The distinction was first drawn clearly by James Bernoulli (1654-1705).

Leibniz introduced the “classical” notion of probability as a proportion of enumerated, equally probable cases, which was also used by Bernoulli, although it was brought to prominence by Laplace (1749-1827). This notion is ambiguous between the frequency interpretation and the subjective interpretation. The cases can be thought to be equally probable either because of a natural, physical symmetry between them, or simply because we do not have any knowledge that would lead us to consider one more probable than another. The use of this latter, subjective consideration to justify assigning equal probabilities is known as the principle of indifference (Keynes, 1921).

The debate between objectivists and subjectivists became sharper in the 20th century. Kolmogorov (1963), R. A. Fisher (1922), and Richard von Mises (1928) were advocates of the relative frequency interpretation. Karl Popper’s (1959, first published in German in 1934) “propensity” interpretation traces relative frequencies to an underlying physical symmetry. Frank Ramsey (1931), Bruno de Finetti (1937), R. T. Cox (1946), Leonard Savage (1954), and Richard Jeffrey (1983) interpreted probabilities as the degrees of belief of specific individuals. Their analyses of degree of belief were closely tied to utilities and to behavior—specifically, to the willingness to place bets. Rudolf Carnap, following Leibniz and Laplace, offered a different kind of subjective interpretation of probability—not as any actual individual’s degree of belief, but as the degree of belief that an idealized individual should have in a particular proposition a, given a particular body of evidence e. Carnap attempted to go further

488

Chapter 13. Uncertainty

CONFIRMATION

INDUCTIVE LOGIC

than Leibniz or Laplace by making this notion of degree of confirmation mathematically precise, as a logical relation between a and e. The study of this relation was intended to constitute a mathematical discipline called inductive logic, analogous to ordinary deductive logic (Carnap, 1948, 1950). Carnap was not able to extend his inductive logic much beyond the propositional case, and Putnam (1963) showed that some fundamental difficulties would prevent a strict extension to languages capable of expressing arithmetic.

The question of reference classes is closely tied to the attempt to find an inductive logic. The approach of choosing the “most specific” reference class of sufficient size was formally proposed by Reichenbach (1949). Various attempts have been made, notably by Henry Kyburg (1977, 1983), to formulate more sophisticated policies in order to avoid some obvious fallacies that arise with Reichenbach’s rule, but such approaches remain somewhat ad hoc. More recent work by Bacchus, Grove, Halpern, and Koller (1992) extends Carnap’s methods to first-order theories, thereby avoiding many of the difficulties associated with the straightfoward reference-class method.

Bayesian probabilistic reasoning has been used in Al since the 1960s, especially in medical diagnosis. It was used not only to make a diagnosis from available evidence, but also to select further questions and tests using the theory of information value (Section 16.6) when available evidence was inconclusive (Gorry, 1968; Gorry ef al., 1973). One system outperformed human experts in the diagnosis of acute abdominal illnesses (de Dombal et al., 1974). These early Bayesian systems suffered from a number of problems, however. Because they lacked any theoretical model of the conditions they were diagnosing, they were vulnerable to unrepresentative data occurring in situations for which only a small sample was available (de Dombal ef al., 1981). Even more fundamentally, because they lacked a concise formalism (such as the one to be described in Chapter 14) for representing and using conditional independence information, they depended on the acquisition, storage, and processing of enormous tables of probabilistic data. Because of these difficulties, probabilistic methods for coping with uncertainty fell out of favor in AI from the 1970s to the mid-1980s. Developments since the late 1980s are described in the next chapter.

The naive Bayes representation for joint distributions has been studied extensively in the pattern recognition literature since the 1950s (Duda and Hart, 1973). It has also been used, often unwittingly, in text retrieval, beginning with the work of Maron (1961). The probabilistic foundations of this technique, described further in Exercise 13.18, were elucidated by Robertson and Sparck Jones (1976). Domingos and Pazzani (1997) provide an explanation for the surprising success of naive Bayesian reasoning even in domains where the independence assumptions are clearly violated.

There are many good introductory textbooks on probability theory, including those by Chung (1979) and Ross (1988). Morris DeGroot (1989) offers a combined introduction to probability and statistics from a Bayesian standpoint, as well as a more advanced text (1970). Richard Hamming’s (1991) textbook gives a mathematically sophisticated introduction to probability theory from the standpoint of a propensity interpretation based on physical symmetry. Hacking (1975) and Hald (1990) cover the early history of the concept of probability. Bernstein (1996) gives an entertaining popular account of the story of risk.

Section 13.8.

Summary 489

EXERCISES

13.1 Show from first principles that P(a|b Aa) = 1.

13.2 Using the axioms of probability, prove that any probability distribution on a discrete random variable must sum to 1.

13.3 Would it be rational for an agent to hold the three beliefs P(A) = 0.4, P(B) = 0.3, and P(AV B)=0.5? If so, what range of probabilities would be rational for the agent to hold for A A B? Make up a table like the one in Figure 13.2, and show how it supports your argument about rationality. Then draw another version of the table where P(A Vv B)=0.7. Explain why it is rational to have this probability, even though the table shows one case that is a loss and three that just break even. (Hint: what is Agent 1 committed to about the probability of each of the four cases, especially the case that is a loss?)

13.4 This question deals with the properties of atomic events, as discussed on page 468. a. Prove that the disjunction of all possible atomic events is logically equivalent to true. [Hint: Use a proof by induction on the number of random variables.]

b. Prove that any proposition is logically equivalent to the disjunction of the atomic events that entail its truth.

13.5 Consider the domain of dealing 5-card poker hands from a standard deck of 52 cards, under the assumption that the dealer is fair.

a. How many atomic events are there in the joint probability distribution (i.e., how many 5-card hands are there)?

b. What is the probability of each atomic event?

¢. What is the probability of being dealt a royal straight flush? Four of a kind?

13.6 Given the full joint distribution shown in Figure 13.3, calculate the following:

P(toothache)

P( Cavity)

P(Toothache| cavity) P(Cavity| toothache V catch).

13.7 Show that the three forms of independence in Equation (13.8) are equivalent.

13.8 After your yearly checkup, the doctor has bad news and good news. The bad news is that you tested positive for a serious disease and that the test is 99% accurate (i.e., the probability of testing positive when you do have the disease is 0.99, as is the probability of testing negative when you don’t have the disease). The good news is that this is a rare disease, striking only 1 in 10,000 people of your age. Why is it good news that the disease is rare? What are the chances that you actually have the disease?

490

Chapter 13. Uncertainty

13.9 tis quite often useful to consider the effect of some specific propositions in the context of some general background evidence that remains fixed, rather than in the complete absence of information. The following questions ask you to prove more general versions of the product rule and Bayes’ rule, with respect to some background evidence e:

a. Prove the conditionalized version of the general product rule: P(X,Y|e) =P(X|Y,e)P(Y]e) .

b. Prove the conditionalized version of Bayes’ rule in Equation (13.10).

13.10 Show that the statement P(A, B|C) = P(A|C)P(B|C)

is equivalent to either of the statements P(A|B,C)=P(A|C) and P(B|A,C)=P(B|C).

13.11 Suppose you are given a bag containing n unbiased coins. You are told that n — 1 of these coins are normal, with heads on one side and tails on the other, whereas one coin is a fake, with heads on both sides. a. Suppose you reach into the bag, pick out a coin uniformly at random, flip it, and get a head. What is the (conditional) probability that the coin you chose is the fake coin? b. Suppose you continue flipping the coin for a total of k times after picking it and see k heads. Now what is the conditional probability that you picked the fake coin? ¢. Suppose you wanted to decide whether the chosen coin was fake by flipping it & times. The decision procedure returns FAKE if all k flips come up heads, otherwise it returns NORMAL. What is the (unconditional) probability that this procedure makes an error?

13.12 In this exercise, you will complete the normalization calculation for the meningitis example. First, make up a suitable value for P(S|—-M), and use it to calculate unnormalized values for P(M|S) and P(—~M |S) (i.e., ignoring the P(S) term in the Bayes’ rule expression). Now normalize these values so that they add to 1.

13.13 This exercise investigates the way in which conditional independence relationships affect the amount of information needed for probabilistic calculations.

a. Suppose we wish to calculate P(hle1,e2) and we have no conditional independence information. Which of the following sets of numbers are sufficient for the calculation? (i) P(Ey, Ey), P(H), P(E1|H), P(E, |H) (ii) P(E, Eo), P(H), P(E, EH) (iii) P(H), P(Ex|H), P(E, |H) b. Suppose we know that P(E) |H, Ey) = P(E; |H) for all values of H, Fy, Es. Now which of the three sets are sufficient?

13.14 Let X,Y, Z be Boolean random variables. Label the eight entries in the joint distribution P(X, Y, Z) as a through h. Express the statement that X and Y are conditionally

Section 13.8.

Summary 491

independent given Z as a set of equations relating a through h. How many nonredundant equations are there?

13.15 (Adapted from Pearl (1988).) Suppose you are a witness to a nighttime hit-and-run accident involving a taxi in Athens. All taxis in Athens are blue or green. You swear, under oath, that the taxi was blue. Extensive testing shows that, under the dim lighting conditions, discrimination between blue and green is 75% reliable. Is it possible to calculate the most likely color for the taxi? (Hint: distinguish carefully between the proposition that the taxi is blue and the proposition that it appears blue.)

What about now, given that 9 out of 10 Athenian taxis are green?

13.16 (Adapted from Pearl (1988).) Three prisoners, A, B, and C, are locked in their cells. It is common knowledge that one of them will be executed the next day and the others pardoned. Only the governor knows which one will be executed. Prisoner A asks the guard a favor: “Please ask the governor who will be executed, and then take a message to one of my friends Bor C to let him know that he will be pardoned in the morning.” The guard agrees, and comes back later and tells A that he gave the pardon message to B.

What are A’s chances of being executed, given this information? (Answer this mathematically, not by energetic waving of hands.)

13.17 Write out a general algorithm for answering queries of the form P(Cause|e), using a naive Bayes distribution. You should assume that the evidence e may assign values to any subset of the effect variables.

13.18 Text categorization is the task of assigning a given document to one of a fixed set of categories, on the basis of the text it contains. Naive Bayes models are often used for this task. In these models, the query variable is the document category, and the “effect” variables are the presence or absence of each word in the language; the assumption is that words occur independently in documents, with frequencies determined by the document category.

a. Explain precisely how such a model can be constructed, given as “training data” a set of documents that have been assigned to categories.

b. Explain precisely how to categorize a new document.

c. Is the independence assumption reasonable? Discuss.

13.19 In our analysis of the wumpus world, we used the fact that each square contains a pit with probability 0.2, independently of the contents of the other squares. Suppose instead that exactly V/5 pits are scattered uniformly at random among the N squares other than [1,1]. Are the variables F; ; and Py; still independent? What is the joint distribution P(Py 1,..., Py4) now? Redo the calculation for the probabilities of pits in [1,3] and [2,2].

1 4 PROBABILISTIC REASONING

In which we explain how to build network models to reason under uncertainty according to the laws of probability theory.

Chapter 13 gave the syntax and semantics of probability theory. We remarked on the importance of independence and conditional independence relationships in simplifying probabilistic representations of the world. This chapter introduces a systematic way to represent such relationships explicitly in the form of Bayesian networks. We define the syntax and semantics of these networks and show how they can be used to capture uncertain knowledge in a natural and efficient way. We then show how probabilistic inference, although computationally intractable in the worst case, can be done efficiently in many practical situations. We also describe a variety of approximate inference algorithms that are often applicable when exact inference is infeasible. We explore ways in which probability theory can be applied to worlds with objects and relations—that is, to first-order, as opposed to propositional, representations. Finally, we survey alternative approaches to uncertain reasoning.

BAYESIAN NETWORK

In Chapter 13, we saw that the full joint probability distribution can answer any question about the domain, but can become intractably large as the number of variables grows. Furthermore, specifying probabilities for atomic events is rather unnatural and can be very difficult unless a large amount of data is available from which to gather statistical estimates.

We also saw that independence and conditional independence relationships among variables can greatly reduce the number of probabilities that need to be specified in order to define the full joint distribution. This section introduces a data structure called a Bayesian network’ to represent the dependencies among variables and to give a concise specification of any full joint probability distribution.

1 This is the most common name, but there are many others, including belief network, probabilistic network, causal network, and knowledge map. In statistics, the term graphical model refers to a somewhat broader class

that includes Bayesian networks. An extension of Bayesian networks called a decision network or influence diagram will be covered in Chapter 16.

492

Section 14.1. Representing Knowledge in an Uncertain Domain 493

A Bayesian network is a directed graph in which each node is annotated with quantitative probability information. The full specification is as follows:

1. A set of random variables makes up the nodes of the network. Variables may be discrete or continuous.

2. A set of directed links or arrows connects pairs of nodes. If there is an arrow from node X tonode Y, X is said to be a parent of Y.

3. Each node X; has a conditional probability distribution P(X;|Parents(X;)) that quantifies the effect of the parents on the node.

4. The graph has no directed cycles (and hence is a directed, acyclic graph, or DAG).

The topology of the network—the set of nodes and links—specifies the conditional independence relationships that hold in the domain, in a way that will be made precise shortly. The intuitive meaning of an arrow in a properly constructed network is usually that X has a direct influence on Y. It is usually easy for a domain expert to decide what direct influences exist in the domain—much easier, in fact, than actually specifying the probabilities themselves. Once the topology of the Bayesian network is laid out, we need only specify a conditional probability distribution for each variable, given its parents. We will see that the combination of the topology and the conditional distributions suffices to specify (implicitly) the full joint distribution for all the variables.

Recall the simple world described in Chapter 13, consisting of the variables Toothache, Cavity, Catch, and Weather. We argued that Weather is independent of the other variables; furthermore, we argued that Toothache and Catch are conditionally independent, given Cavity. These relationships are represented by the Bayesian network structure shown in Figure 14.1. Formally, the conditional independence of Toothache and Catch given Cavity is indicated by the absence of a link between Toothache and Catch. Intuitively, the network represents the fact that Cavity is a direct cause of Toothache and Catch, whereas no direct causal relationship exists between Toothache and Catch.

Now consider the following example, which is just a little more complex. You have a new burglar alarm installed at home. It is fairly reliable at detecting a burglary, but also responds on occasion to minor earthquakes. (This example is due to Judea Pearl, a resident of Los Angeles—hence the acute interest in earthquakes.) You also have two neighbors, John and Mary, who have promised to call you at work when they hear the alarm. John always calls

Goo (=D

Figure 14.1 A simple Bayesian network in which Weather is independent of the other three variables and Toothache and Catch are conditionally independent, given Cavity.

494

Chapter 14, Probabilistic Reasoning

CONDITIONAL PROBABILITY TABLE

CONDITIONING CASE

Burglary yo Earthquake =

A PM

t .70 f 01

Figure 14.2 A typical Bayesian network, showing both the topology and the conditional probability tables (CPTs). In the CPTs, the letters B, E, A, J, and M stand for Burglary, Earthquake, Alarm, John Calls, and MaryCalls, respectively.

when he hears the alarm, but sometimes confuses the telephone ringing with the alarm and calls then, too. Mary, on the other hand, likes rather loud music and sometimes misses the alarm altogether. Given the evidence of who has or has not called, we would like to estimate the probability of a burglary. The Bayesian network for this domain appears in Figure 14.2.

For the moment, let us ignore the conditional distributions in the figure and concentrate on the topology of the network. In the case of the burglary network, the topology shows that burglary and earthquakes directly affect the probability of the alarm’s going off, but whether John and Mary call depends only on the alarm. The network thus represents our assumptions that they do not perceive any burglaries directly, they do not notice the minor earthquakes, and they do not confer before calling.

Notice that the network does not have nodes corresponding to Mary’s currently listening to loud music or to the telephone ringing and confusing John. These factors are summarized in the uncertainty associated with the links from Alarm to JohnCalls and MaryCalls. This shows both laziness and ignorance in operation: it would be a lot of work to find out why those factors would be more or less likely in any particular case, and we have no reasonable way to obtain the relevant information anyway. The probabilities actually summarize a potentially infinite set of circumstances in which the alarm might fail to go off (high humidity, power failure, dead battery, cut wires, a dead mouse stuck inside the bell, etc.) or John or Mary might fail to call and report it (out to lunch, on vacation, temporarily deaf, passing helicopter, etc.). In this way, a small agent can cope with a very large world, at least approximately. The degree of approximation can be improved if we introduce additional relevant information.

Now let us turn to the conditional distributions shown in Figure 14.2. In the figure, each distribution is shown as a conditional probability table, or CPT. (This form of table can be used for discrete variables; other representations, including those suitable for continuous variables, are described in Section 14.2.) Each row in a CPT contains the conditional probability of each node value for a conditioning case. A conditioning case is just a possible

Section 14.2.

The Semantics of Bayesian Networks 495

combination of values for the parent nodes—a miniature atomic event, if you like. Each row must sum to 1, because the entries represent an exhaustive set of cases for the variable. For Boolean variables, once you know that the probability of a true value is p, the probability of false must be 1 — p, so we often omit the second number, as in Figure 14.2. In general, a table for a Boolean variable with k£ Boolean parents contains 2¥ independently specifiable probabilities. A node with no parents has only one row, representing the prior probabilities of each possible value of the variable.

The previous section described what a network is, but not what it means. There are two ways in which one can understand the semantics of Bayesian networks. The first is to see the network as a representation of the joint probability distribution. The second is to view it as an encoding of a collection of conditional independence statements. The two views are equivalent, but the first turns out to be helpful in understanding how to construct networks, whereas the second is helpful in designing inference procedures.

Representing the full joint distribution

A Bayesian network provides a complete description of the domain. Every entry in the full joint probability distribution (hereafter abbreviated as “joint”) can be calculated from the information in the network. A generic entry in the joint distribution is the probability of a conjunction of particular assignments to each variable, such as P(X; =x1 A... AX, = xy). We use the notation P(zi, ..., Zn) as an abbreviation for this. The value of this entry is given by the formula n P(z1, ..., zp) = IT P(z;|parents(X;)) , (14.1) i=1

where parents(X;) denotes the specific values of the variables in Parents(X;). Thus, each entry in the joint distribution is represented by the product of the appropriate elements of the conditional probability tables (CPTs) in the Bayesian network. The CPTs therefore provide a decomposed representation of the joint distribution.

To illustrate this, we can calculate the probability that the alarm has sounded, but neither a burglary nor an earthquake has occurred, and both John and Mary call. We use single-letter names for the variables:

P(jAmAaAN—bA—e) = P(j|a)P(mla) P(a|=b A ~¢) P(~b) P(~e) = 0.90 x 0.70 x 0.001 x 0.999 x 0.998 = 0.00062 .

Section 13.4 explained that the full joint distribution can be used to answer any query

about the domain. If a Bayesian network is a representation of the joint distribution, then it

too can be used to answer any query, by summing all the relevant joint entries. Section 14.4 explains how to do this, but also describes methods that are much more efficient.

496

Chapter 14. Probabilistic Reasoning

CHAIN RULE

A method for constructing Bayesian networks

Equation (14.1) defines what a given Bayesian network means. It does not, however, explain how to construct a Bayesian network in such a way that the resulting joint distribution is a good representation of a given domain. We will now show that Equation (14.1) implies certain conditional independence relationships that can be used to guide the knowledge engineer in constructing the topology of the network. First, we rewrite the joint distribution in terms of a conditional probability, using the product rule (see Chapter 13):

P(x, ce Zn) = P(zn|xn-1, cee x1) P(xn-1, cee x1) .

Then we repeat the process, reducing each conjunctive probability to a conditional probability and a smaller conjunction. We end up with one big product:

P(z1, ..., xn) = P(xp|Zn-1,---, 21)P(Xn-1|Tn-2, ..., x1) ++ P(xa|21)P(x1)

n = IT P(xzilxi-1, Cee x1) .

i=1 This identity holds true for any set of random variables and is called the chain rule. Comparing it with Equation (14.1), we see that the specification of the joint distribution is equivalent to the general assertion that, for every variable X; in the network,

P(X;| Xi 1, ..., X1) = P(X;|Parents(X;)) , (14.2)

provided that Parents(X;) C {X;-1, ..., X1}. This last condition is satisfied by labeling the nodes in any order that is consistent with the partial order implicit in the graph structure.

What Equation (14.2) says is that the Bayesian network is a correct representation of the domain only if each node is conditionally independent of its predecessors in the node ordering, given its parents. Hence, in order to construct a Bayesian network with the correct structure for the domain, we need to choose parents for each node such that this property holds. Intuitively, the parents of node X; should contain all those nodes in Xq, ..., X;\_ that directly influence X;. For example, suppose we have completed the network in Figure 14.2 except for the choice of parents for MaryCalls. MaryCalls is certainly influenced by whether there is a Burglary or an Earthquake, but not directly influenced. Intuitively, our knowledge of the domain tells us that these events influence Mary’s calling behavior only through their effect on the alarm. Also, given the state of the alarm, whether John calls has no influence on Mary’s calling. Formally speaking, we believe that the following conditional independence statement holds;

P(MaryCalls| John Calls, Alarm, Earthquake, Burglary) = P(MaryCalls| Alarm) .

Compactness and node ordering

As well as being a complete and nonredundant representation of the domain, a Bayesian network can often be far more compact than the full joint distribution. This property is what makes it feasible to handle domains with many variables. The compactness of Bayesian networks is an example of a very general property of locally structured (also called sparse) systems. In a locally structured system, each subcomponent interacts directly with only a bounded number of other components, regardless of the total number of components. Local

Section 14.2.

The Semantics of Bayesian Networks 497

structure is usually associated with linear rather than exponential growth in complexity. In the case of Bayesian networks, it is reasonable to suppose that in most domains each random variable is directly influenced by at most k others, for some constant k. If we assume n Boolean variables for simplicity, then the amount of information needed to specify each conditional probability table will be at most 2¥ numbers, and the complete network can be specified by n2% numbers. In contrast, the joint distribution contains 2™ numbers. To make this concrete, suppose we have n = 30 nodes, each with five parents (k= 5). Then the Bayesian network requires 960 numbers, but the full joint distribution requires over a billion.

There are domains in which each variable can be influenced directly by all the others, so that the network is fully connected. Then specifying the conditional probability tables requires the same amount of information as specifying the joint distribution. In some domains, there will be slight dependencies that should strictly be included by adding a new link. But if these dependencies are very tenuous, then it may not be worth the additional complexity in the network for the small gain in accuracy. For example, one might object to our burglary network on the grounds that if there is an earthquake, then John and Mary would not call even if they heard the alarm, because they assume that the earthquake is the cause. Whether to add the link from Earthquake to JohnCalls and MaryCalls (and thus enlarge the tables) depends on comparing the importance of getting more accurate probabilities with the cost of specifying the extra information.

Even in a locally structured domain, constructing a locally structured Bayesian network is not a trivial problem. We require not only that each variable be directly influenced by only a few others, but also that the network topology actually reflect those direct influences with the appropriate set of parents. Because of the way that the construction procedure works, the “direct influencers” will have to be added to the network first if they are to become parents of the node they influence. Therefore, the correct order in which to add nodes is to add the “root causes” first, then the variables they influence, and so on, until we reach the “leaves,” which have no direct causal influence on the other variables.

What happens if we happen to choose the wrong order? Let us consider the burglary example again. Suppose we decide to add the nodes in the order MaryCalls, JohnCalls, Alarm, Burglary, Earthquake. Then we get the somewhat more complicated network shown in Figure 14.3(a). The process goes as follows:

e Adding MaryCalls: No parents.

e Adding JohnCalls: If Mary calls, that probably means the alarm has gone off, which of course would make it more likely that John calls. Therefore, JohnCalls needs MaryCalls as a parent

o Adding Alarm: Clearly, if both call, it is more likely that the alarm has gone off than if just one or neither call, so we need both MaryCalls and JohnCalls as parents.

e Adding Burglary: If we know the alarm state, then the call from John or Mary might give us information about our phone ringing or Mary’s music, but not about burglary:

P(Burglary| Alarm, JohnCalls, MaryCalls) = P(Burglary| Alarm) .

Hence we need just Alarm as parent.

498

Chapter 14. Probabilistic Reasoning

Earthquake

(

Burglary Earthquake

(a) (b)

Burglary

Figure 14.3 Network structure depends on order of introduction. In each network, we have introduced nodes in top-to-bottom order.

e Adding Earthquake: if the alarm is on, it is more likely that there has been an earthquake. (The alarm is an earthquake detector of sorts.) But if we know that there has been a burglary, then that explains the alarm, and the probability of an earthquake would be only slightly above normal. Hence, we need both Alarm and Burglary as parents.

The resulting network has two more links than the original network in Figure 14.2 and requires three more probabilities to be specified. What's worse, some of the links represent tenuous relationships that require difficult and unnatural probability judgments, such as assessing the probability of Earthquake, given Burglary and Alarm. This phenomenon is quite general and is related to the distinction between causal and diagnostic models introduced in Chapter 8. If we try to build a diagnostic model with links from symptoms to causes (as from MaryCalls to Alarm or Alarm to Burglary), we end up having to specify additional dependencies between otherwise independent causes (and often between separately occurring symptoms as well). If we stick to a causal model, we end up having to specify fewer numbers, and the numbers will often be easier to come up with. In the domain of medicine, for example, it has been shown by Tversky and Kahneman (1982) that expert physicians prefer to give probability judgments for causal rules rather than for diagnostic ones.

Figure 14.3(b) shows a very bad node ordering: MaryCalls, JohnCalls, Earthquake, Burglary, Alarm. This network requires 31 distinct probabilities to be specified—exactly the same as the full joint distribution. Tt is important to realize, however, that any of the three networks can represent exactly the same joint distribution. The last two versions simply fail to represent all the conditional independence relationships and hence end up specifying a lot of unnecessary numbers instead.

Section 14.2. The Semantics of Bayesian Networks 499

Conditional independence relations in Bayesian networks

We have provided a “numerical” semantics for Bayesian networks in terms of the representation of the full joint distribution, as in Equation (14.1). Using this semantics to derive a method for constructing Bayesian networks, we were led to the consequence that a node is conditionally independent of its predecessors, given its parents. It turns out that we can also go in the other direction. We can start from a “topological” semantics that specifies the conditional independence relationships encoded by the graph structure, and from these we can derive the “numerical” semantics. The topological semantics is given by either of the following specifications, which are equivalent?

DESCENDANTS 1. A node is conditionally independent of its non-descendants, given its parents. For example, in Figure 14.2, JohnCalls is independent of Burglary and Earthquake, given the value of Alarm.

2. A node is conditionally independent of all other nodes in the network, given its parMARKOV BLANKET ents, children, and children’s parents—that is, given its Markov blanket. For example, Burglary is independent of JohnCalls and MaryCalls, given Alarm and Earthquake.

These specifications are illustrated in Figure 14.4. From these conditional independence assertions and the CPTs, the full joint distribution can be reconstructed; thus, the “numerical” semantics and the “topological” semantics are equivalent.

[]

Figure 14.4 (2) A node X is conditionally independent of its non-descendants (e.g., the Z;;s) given its parents (the U;s shown in the gray area). (b) A node X is conditionally independent of all other nodes in the network given its Markov blanket (the gray area).

2 There is also a general topological criterion called d-separation for deciding whether a set of nodes X is independent of another set Y, given a third set Z. The criterion is rather complicated and is not needed for deriving the algorithms in this chapter, so we omit it. Details may be found in Russell and Norvig (1995) or Pearl (1988). Shachter (1998) gives a more intuitive method of ascertaining d-separation.

500

Chapter 14. Probabilistic Reasoning

CANONICAL DISTRIBUTION

DETERMINISTIC NODES

NOISY-OR

LEAK NODE

Even if the maximum number of parents k is smallish, filling in the CPT for a node requires up to O(2\*) numbers and perhaps a great deal of experience with all the possible conditioning cases. In fact, this is a worst-case scenario in which the relationship between the parents and the child is completely arbitrary. Usually, such relationships are describable by a canonical distribution that fits some standard pattern. In such cases, the complete table can be specified by naming the pattern and perhaps supplying a few parameters—much easier than supplying an exponential number of parameters.

The simplest example is provided by deterministic nodes. A deterministic node has its value specified exactly by the values of its parents, with no uncertainty. The relationship can be a logical one: for example, the relationship between the parent nodes Canadian, US, Mexican and the child node NorthAmerican is simply that the child is the disjunction of the parents. The relationship can also be numerical: for example, if the parent nodes are the prices of a particular model of car at several dealers, and the child node is the price that a bargain hunter ends up paying, then the child node is the minimum of the parent values; or if the parent nodes are the inflows (rivers, runoff, precipitation) into a lake and the outflows (rivers, evaporation, seepage) from the lake and the child is the change in the water level of the lake, then the value of the child is the difference between the inflow parents and the outflow parents.

Uncertain relationships can often be characterized by so-called “noisy” logical relationships. The standard example is the noisy-OR relation, which is a generalization of the logical OR. In propositional logic, we might say that Fever is true if and only if Cold, Flu, or Malaria is true. The noisy-OR model allows for uncertainty about the ability of each parent to cause the child to be true—the causal relationship between parent and child may be inhibited, and so a patient could have a cold, but not exhibit a fever. The model makes two assumptions. First, it assumes that all the possible causes are listed. (This is not as strict as it seems, because we can always add a so-called leak node that covers “miscellaneous causes.”) Second, it assumes that inhibition of each parent is independent of inhibition of any other parents: for example, whatever inhibits Malaria from causing a fever is independent of whatever inhibits Flu from causing a fever. Given these assumptions, Fever is false if and only if all its true parents are inhibited, and the probability of this is the product of the inhibition probabilities for each parent. Let us suppose these individual inhibition probabilities are as follows:

P(—fever|cold, = flu, —malaria) = 0.6 , P(—=fever|—cold, flu, ~malaria) = 0.2 , P(—fever|—cold, —flu, malaria) = 0.1.

Then, from this information and the noisy-OR assumptions, the entire CPT can be built. The following table shows how:

Section 14.3.

Efficient Representation of Conditional Distributions 501

[]

DISCRETIZATION

PARAMETERS

HYBRID BAYESIAN NETWORK

In general, noisy logical relationships in which a variable depends on k parents can be described using O(k) parameters instead of O(2\*) for the full conditional probability table. This makes assessment and learning much easier. For example, the CPCS network (Pradhan et al., 1994) uses noisy-OR and noisy-MAX distributions to model relationships among diseases and symptoms in internal medicine. With 448 nodes and 906 links, it requires only 8,254 values instead of 133,931,430 for a network with full CPTs.

Bayesian nets with continuous variables

Many real-world problems involve continuous quantities, such as height, mass, temperature, and money; in fact, much of statistics deals with random variables whose domains are continuous. By definition, continuous variables have an infinite number of possible values, so it is impossible to specify conditional probabilities explicitly for each value. One possible way to handle continuous variables is to avoid them by using discretization—that is, dividing up the possible values into a fixed set of intervals. For example, temperatures could be divided into (<0°0), (0°C—-100°C), and (>100°C). Discretization is sometimes an adequate solution, but often results in a considerable loss of accuracy and very large CPTs. The other solution is to define standard families of probability density functions (see Appendix A) that are specified by a finite number of parameters. For example, a Gaussian (or normal) distribution N{(u,0?)(x) has the mean yu and the variance o2 as parameters.

A network with both discrete and continuous variables is called a hybrid Bayesian network. To specify a hybrid network, we have to specify two new kinds of distributions: the conditional distribution for a continuous variable given discrete or continuous parents; and the conditional distribution for a discrete variable given continuous parents. Consider the simple example in Figure 14.5, in which a customer buys some fruit depending on its cost, which depends in turn on the size of the harvest and whether the government's subsidy scheme is operating. The variable Cost is continuous and has continuous and discrete parents; the variable Buys is discrete and has a continuous parent.

For the Cost variable, we need to specify P( Cost| Harvest, Subsidy). The discrete parent is handled by explicit enumeration—that is, specifying both P(Cost| Harvest, subsidy) and P(Cost| Harvest, —subsidy). To handle Harvest, we specify how the distribution over the cost ¢ depends on the continuous value h of Harvest. In other words, we specify the parameters of the cost distribution as a function of h. The most common choice is the linear

502

Chapter 14. Probabilistic Reasoning

LINEAR GAUSSIAN

Guid) Garves)

Figure 14.5 A simple network with discrete variables (Subsidy and Buys) and continuous variables (Harvest and Cost).

P(c | h, subsidy) & 0.4

P(c | h, —subsidy) 04

Figure 14.6 The graphs in (a) and (b) show the probability distribution over Cost as a function of Harvest size, with Subsidy true and false respectively. Graph (c) shows the

distribution P(Cost| Harvest), obtained by summing over the two subsidy cases.

Gaussian distribution, in which the child has a Gaussian distribution whose mean : varies linearly with the value of the parent and whose standard deviation ¢ is fixed. We need two distributions, one for subsidy and one for —subsidy, with different parameters:

5 1 -1 (tegtrea” P(c|h, subsidy) = N{agh + be, 07 )(c) = e ot (eh, subsidy) = Nach + bi,of)(e) = ——

1 pegs’

— \_\_¢ of on For this example, then, the conditional distribution for Cost is specified by naming the linear Gaussian distribution and providing the parameters a, by, 0¢, af, by, and oy. Figures 14.6(a) and (b) show these two relationships. Notice that in each case the slope is negative, because price decreases as supply increases. (Of course, the assumption of linearity implies that the price becomes negative at some point; the linear model is reasonable only if the harvest size is limited to a narrow range.) Figure 14.6(c) shows the distribution P(c|h), averaging over the two possible values of Subsidy and assuming that each has prior probability 0.5. This shows that even with very simple models, quite interesting distributions can be represented. The linear Gaussian conditional distribution has some special properties. A network containing only continuous variables with linear Gaussian distributions has a joint distribu

P(c|h, ~subsidy) = N(agh+bg,0%)(c) =

Section 14.3.

CONDITIONAL GAUSSIAN

PROBIT DISTRIBUTION

LOGIT DISTRIBUTION SIGMOID FUNCTION

Efficient Representation of Conditional Distributions 503

1 1 0.8 0.8 506 506 1 g g < 8 x 04 5 04 0.2 0.2 0 r v \ 0 v + T \ 0 2 4 6 8 10 12 0 2 4 6 8 10 12 Cost ¢ Cost c¢ (a) (b) Figure 14.7 (a) A probit distribution for the probability of Buys given Cost, with 1 =6.0 and ¢ = 1.0. (b) A logit distribution with the same parameters.

tion that is a multivariate Gaussian distribution over all the variables (Exercise 14.5).3 (A multivariate Gaussian distribution is a surface in more than one dimension that has a peak at the mean (in n dimensions) and drops off on all sides.) When discrete variables are added (provided that no discrete variable is a child of a continuous variable), the network defines a conditional Gaussian, or CG, distribution: given any assignment to the discrete variables, the distribution over the continuous variables is a multivariate Gaussian.

Now we turn to the distributions for discrete variables with continuous parents. Consider, for example, the Buys node in Figure 14.5. It seems reasonable to assume that the customer will buy if the cost is low and will not buy if it is high and that the probability of buying varies smoothly in some intermediate region. In other words, the conditional distribution is like a “soft” threshold function. One way to make soft thresholds is to use the integral of the standard normal distribution:

x B(x) = / NO, 1)(2)dz . —0o0 Then the probability of Buys given Cost might be

P(buys | Cost=c) = ®((—c+ p)/o) which means that the cost threshold occurs around pu, the width of the threshold region is proportional to o, and the probability of buying decreases as cost increases.

This probit distribution is illustrated in Figure 14.7(a). The form can be justified by proposing that the underlying decision process has a hard threshold, but that the precise location of the threshold is subject to random Gaussian noise. An alternative to the probit model is the logit distribution, which uses the sigmoid function to produce a soft threshold:

1

Plbuys | Cost=¢) = ——m————. ( uy ) 1 + ezp(—2=E)

3 It follows that inference in linear Gaussian networks takes only O(n?) time in the worst case, regardless of the network topology. In Section 14.4, we will see that inference for networks of discrete variables is NP-hard.

504

Chapter 14. Probabilistic Reasoning

This is illustrated in Figure 14.7(b). The two distributions look similar, but the logit actually has much longer “tails.” The probit is often a better fit to real situations, but the logit is sometimes easier to deal with mathematically. It is used widely in neural networks (Chapter 20). Both probit and logit can be generalized to handle multiple continuous parents by taking a linear combination of the parent values. Extensions for a multivalued discrete child are explored in Exercise 14.6.

EVENT

HIDDEN VARIABLES

The basic task for any probabilistic inference system is to compute the posterior probability distribution for a set of query variables, given some observed event-—that is, some assignment of values to a set of evidence variables. We will use the notation introduced in Chapter 13: X denotes the query variable; E denotes the set of evidence variables E1,. .., Ey, and e is a particular observed event; Y will denote the nonevidence variables Yi, ...,Y; (sometimes called the hidden variables). Thus, the complete set of variables X={X}UEUY. A typical query asks for the posterior probability distribution P(X|e).\*

In the burglary network, we might observe the event in which JohnCalls = true and Mary Calls = true. We could then ask for, say, the probability that a burglary has occurred:

P(Burglary| John Calls = true, MaryCalls = true) = (0.284,0.716) .

In this section we will discuss exact algorithms for computing posterior probabilities and will consider the complexity of this task. It turns out that the general case is intractable, so Section 14.5 covers methods for approximate inference.

Inference by enumeration

Chapter 13 explained that any conditional probability can be computed by summing terms from the full joint distribution. More specifically, a query P(X|e) can be answered using Equation (13.6), which we repeat here for convenience:

P(X|e) = aP(X,e) =a > P(X.ey) . y

Now, a Bayesian network gives a complete representation of the full joint distribution. More specifically, Equation (14.1) shows that the terms P(x, e,y) in the joint distribution can be written as products of conditional probabilities from the network. Therefore, a query can be answered using a Bayesian network by computing sums of products of conditional probabilities from the network.

In Figure 13.4, an algorithm, ENUMERATE-JOINTASK, was given for inference by enumeration from the full joint distribution. The algorithm takes as input a full joint distribution P and looks up values therein. Itis a simple matter to modify the algorithm so that it takes

4 We will assume that the query variable is not among the evidence variables; if it is, then the posterior distribution for X simply gives probability 1 to the observed value. For simplicity, we have also assumed that the query is just a single variable. Our algorithms can be extended easily to handle a joint query over several variables.

Section 14.4.

Exact Inference in Bayesian Networks 505

as input a Bayesian network bn and “looks up” joint entries by multiplying the corresponding CPT entries from bn.

Consider the query P(Burglary|JohnCalls = true, MaryCalls = true). The hidden variables for this query are Earthquake and Alarm. From Equation (13.6), using initial letters for the variables in order to shorten the expressions, we have’

P(B|j,m) =aP(B,j,m) =a > > P(B,e.a.j.m). e a

The semantics of Bayesian networks (Equation (14.1)) then gives us an expression in terms of CPT entries. For simplicity, we will do this just for Burglary = true:

Pljm) =a > Pb)P(e)P(alb,e)P(jla)P(mla).

To compute this expression, we have to add four terms, each computed by multiplying five numbers. In the worst case, where we have to sum out almost all the variables, the complexity of the algorithm for a network with n Boolean variables is O(n2").

An improvement can be obtained from the following simple observations: the P(b) term is a constant and can be moved outside the summations over a and e, and the P(e) term can be moved outside the summation over a. Hence, we have

P(blj,m) = a P(b) > P(e) >" P(alb, e)P(jla)P(mla) . (14.3)

This expression can be evaluated by looping through the variables in order, multiplying CPT entries as we go. For each summation, we also need to loop over the variable’s possible values. The structure of this computation is shown in Figure 14.8. Using the numbers from Figure 14.2, we obtain P(b|j, m) = a x 0.00059224. The corresponding computation for —b yields a x 0.0014919; hence

P(B|j,m) = a (0.00059224,0.0014919) ~ (0.284,0.716) .

That is, the chance of a burglary, given calls from both neighbors, is about 28%.

The evaluation process for the expression in Equation (14.3) is shown as an expression tree in Figure 14.8. The ENUMERATIONASK algorithm in Figure 14.9 evaluates such trees using depth-first recursion. Thus, the space complexity of ENUMERATION-ASK is only linear in the number of variables—effectively, the algorithm sums over the full joint distribution without ever constructing it explicitly. Unfortunately, its time complexity for a network with n Boolean variables is always O(2")—better than the O(n2") for the simple approach described earlier, but still rather grim. One thing to note about the tree in Figure 14.8 is that it makes explicit the repeated subexpressions that are evaluated by the algorithm. The products P(jla)P(m|a) and P(j|—a)P(m|-a) are computed twice, once for each value of e. The next section describes a general method that avoids such wasted computations.

5 An expression such as )°\_ P(a, e) means to sum P(A = a, E = e) for all possible values of e. There is an ambiguity in that P(e) is used to mean both P(E = true) and P(E = e), but it should be clear from context which is intended; in particular, in the context of a sum the latter is intended.

506 Chapter 14. Probabilistic Reasoning

P(-alb,—e)

P(-alb,e) P(alb,—e) 05 94

Figure 14.8 The structure of the expression shown in Equation (14.3). The evaluation proceeds top-down, multiplying values along each path and summing at the “+” nodes. Notice the repetition of the paths for j and m.

function ENUMERATION-ASK(X, e, bn) returns a distribution over X inputs: X, the query variable e, observed values for variables E bn, a Bayes net with variables {X} U E UY /\*Y = hidden variables \* /

Q(X) «a distribution over X, initially empty for each value x; of X do

extend e with value x; for X

Q(x;) — ENUMERATE-ALL(VARS[bn], e) return NORMALIZE(Q(X))

function ENUMERATE-ALL(vars, e) returns a real number if EMPTY ?(vars) then return 1.0 Y «— FIRST(vars) if Y has value yin e then return P(y | parents(Y)) x ENUMERATE-ALL(REST(vars),e€) elsereturn > P(y | parents(Y)) x ENUMERATE-ALL(REST(vars), ey) where e, is e extended with Y = y

Figure 14.9 The enumeration algorithm for answering queries on Bayesian networks.

Section 14.4. Exact Inference in Bayesian Networks 507

The variable elimination algorithm

The enumeration algorithm can be improved substantially by eliminating repeated calculations of the kind illustrated in Figure 14.8. The idea is simple: do the calculation once and save the results for later use. This is a form of dynamic programming. There are several ver

BE sions of this approach; we present the variable elimination algorithm, which is the simplest. Variable elimination works by evaluating expressions such as Equation (14.3) in right-to-left order (that is, bottom-up in Figure 14.8). Intermediate results are stored, and summations over each variable are done only for those portions of the expression that depend on the variable.

Let us illustrate this process for the burglary network. We evaluate the expression

P(B|j,m) =a P(B) > P(e) Pla|B,e) P(jla) P(m|a) . —— ~~ ————— B E A J M Notice that we have annotated each part of the expression with the name of the associated FACTORS variable; these parts are called factors. The steps are as follows;

e The factor for M, P(m|a), does not require summing over M (because M’s value is already fixed). We store the probability, given each value of a, in a two-element vector,

P(m|a far(4) = Pont ) (The f3s means that M was used to produce f.) e Similarly, we store the factor for J as the two-element vector f;( A). eo The factor for A is P(a|B, e), which will be a 2 x 2 x 2 matrix f4(A, B, E).

e Now we must sum out A from the product of these three factors. This will give us a 2 x 2 matrix whose indices range over just B and E. We put a bar over A in the name of the matrix to indicate that A has been summed out:

fi;0(B.E) = > faa, B,E) x f;(a) x fy (a)

= fa(a, B, E) x f(a) x frr(a) + fa(—a, B, E) X f;(—a) X fr1(—a) . EOINTWISE The multiplication process used here is called a peintwise product and will be described shortly. e We process FE in the same way: sum out E from the product of fg (FE) and f 5 ;,,(B, E): fpasm(B) = fe(e) xf55(B.e) + fe(-e) x f3,,/(B, —e) . e Now we can compute the answer simply by multiplying the factor for B (i.e., fp(B) = P(B)), by the accumulated matrix fz 175, (B): P(B|j,m) = afp(B) xfgi;m(B) Exercise 14.7(a) asks you to check that this process yields the correct answer. Examining this sequence of steps, we see that there are two basic computational oper

ations required: pointwise product of a pair of factors, and summing out a variable from a product of factors.

508

Chapter 14. Probabilistic Reasoning

The pointwise product is not matrix multiplication, nor is it element-by-element multiplication. The pointwise product of two factors f; and f; yields a new factor f whose variables are the union of the variables in f; and fa. Suppose the two factors have variables Y7,..., Yy in common. Then we have

(XX; V1... YZ... 2) =tu(X1... X;, Yi... Yi) (Vi... Yi, 2... Z)).

If all the variables are binary, then f; and fa have 207% and 2F+! entries respectively, and the pointwise product has 275+! entries. For example, given two factors fi (A, B) and f2(B, C) with probability distributions shown below, the pointwise product f; x f2 is given as f3(A, B,C):

[]

Summing out a variable from a product of factors is also a straightforward computation. The only trick is to notice that any factor that does not depend on the variable to be summed out can be moved outside the summation process. For example,

> fee) xfa(A, Be) x f5(A) x far (A) = f,(A) x far (A) x doe fr(e) x f1(A, B, e) . Now the pointwise product inside the summation is computed, and the variable is summed out of the ey matrix:

f;(A) x far (A <2 tse) yx fa( A, B ce) =f£;(A) x far (A) x fz 4(A, B).

Notice that matrices are not multiplied until we need to sum out a variable from the accumulated product. At that point, we multiply just those matrices that include the variable to be summed out. Given routines for pointwise product and summing out, the variable elimination algorithm itself can be written quite simply, as shown in Figure 14.10.

Let us consider one more query: P(JohnCalls|Burglary = true). As usual, the first step is to write out the nested summation:

P(J|b) =a P(b )2 Pe )> P(alb,e)P(J]a) > P(m|a).

If we evaluate this expression from right to left, we notice something interesting: >\_,, P(m|a) is equal to 1 by definition! Hence, there was no need to include it in the first place; the variable M is irrelevant to this query. Another way of saying this is that the result of the query P(JohnCalls| Burglary = true) is unchanged if we remove MaryCalls from the network altogether. In general, we can remove any leaf node that is not a query variable or an evidence

Section 14.4. Exact Inference in Bayesian Networks 509

function ELIMINATION-ASK(X,e, bn) returns a distribution over X inputs: X, the query variable e, evidence specified as an event bn, a Bayesian network specifying joint distribution P(X, ..., X,,)

factors — |]; vars — REVERSE(VARS[bn]) for each var in vars do

factors — [MAKE-FACTOR (var, €)|factors]

if var is a hidden variable then factors «< SUM-OQUT (var, factors) return NORMALIZE(POINTWISE-PRODUCT(factors))

Figure 14.10 The variable elimination algorithm for answering queries on Bayesian networks.

variable. After its removal, there may be some more leaf nodes, and these too may be irrelevant. Continuing this process, we eventually find that every variable that is not an ancestor of a query variable or evidence variable is irrelevant to the query. A variable elimination algorithm can therefore remove all these variables before evaluating the query.

The complexity of exact inference

We have argued that variable elimination is more efficient than enumeration because it avoids repeated computations (as well as dropping irrelevant variables). The time and space requirements of variable elimination are dominated by the size of the largest factor constructed during the operation of the algorithm. This in turn is determined by the order of elimination of variables and by the structure of the network.

The burglary network of Figure 14.2 belongs to the family of networks in which there is at most one undirected path between any two nodes in the network. These are called singly sively connecter connected networks or polytrees, and they have a particularly nice property: The time and POLYTREES space complexity of exact inference in polytrees is linear in the size of the network. Here, the size is defined as the number of CPT entries; if the number of parents of each node is bounded by a constant, then the complexity will also be linear in the number of nodes. These results hold for any ordering consistent with the topological ordering of the network (Exercise 14.7). ED For multiply connected networks, such as that of Figure 14.11(a), variable elimination can have exponential time and space complexity in the worst case, even when the number of parents per node is bounded. This is not surprising when one considers that, because it includes inference in propositional logic as a special case, inference in Bayesian networks is NP-hard. In fact, it can be shown (Exercise 14.8) that the problem is as hard as that of computing the number of satisfying assignments for a propositional logic formula. This means

that it is #P-hard (“number-P hard”)—that is, strictly harder than NP-complete problems. There is a close connection between the complexity of Bayesian network inference and the complexity of constraint satisfaction problems (CSPs). As we discussed in Chapter 5, the difficulty of solving a discrete CSP is related to how “tree-like” its constraint graph is.

510

Chapter 14. Probabilistic Reasoning

CLUSTERING JOIN TREE

Measures such as hypertree width, which bound the complexity of solving a CSP, can also be applied directly to Bayesian networks. Moreover, the variable elimination algorithm can be generalized to solve CSPs as well as Bayesian networks.

Clustering algorithms

The variable elimination algorithm is simple and efficient for answering individual queries. If we want to compute posterior probabilities for all the variables in a network, however, it can be less efficient. For example, in a polytree network, one would need to issue O(n) queries costing O(n) each, for a total of O(n?) time. Using clustering algorithms (also known as join tree algorithms), the time can be reduced to O(n). For this reason, these algorithms are widely used in commercial Bayesian network tools.

The basic idea of clustering is to join individual nodes of the network to form cluster nodes in such a way that the resulting network is a polytree. For example, the multiply connected network shown in Figure 14.11(a) can be converted into a polytree by combining the Sprinkler and Rain node into a cluster node called Sprinkler+ Rain, as shown in Figure 14.11(b). The two Boolean nodes are replaced by a meganode that takes on four possible values: TT, TF, FT, and FF. The meganode has only one parent, the Boolean variable Cloudy, so there are two conditioning cases.

Once the network is in polytree form, a special-purpose inference algorithm is applied. Essentially, the algorithm is a form of constraint propagation (see Chapter 5) where the constraints ensure that neighboring clusters agree on the posterior probability of any variables that they have in common. With careful bookkeeping, this algorithm is able to compute posterior probabilities for all the nonevidence nodes in the network in time O(n), where n is now the size of the modified network. However, the NP-hardness of the problem has not disappeared: if a network requires exponential time and space with variable elimination, then the CPTs in the clustered network will require exponential time and space to construct.

ID C | P(S C | PRR (cto) t| .10 Csprinkier) (Rain) t| .80 P(S+R=2)

£1.50 £1.20 cl FoF sprain) i] 0s 02.72 18 RE) S+R | POW) fl .10 40.10 40 tt] 99 rt 99 tr Ff .90 tf | 90 (ga) fot} 90 ft 90 f fl .00 ff 00 (a) (b)

Figure 14.11 (a) A multiply connected network with conditional probability tables. (b) A clustered equivalent of the multiply connected network.

Section 14.5.

Approximate Inference in Bayesian Networks 511

MONTE CARLO

Given the intractability of exact inference in large, multiply connected networks, it is essential to consider approximate inference methods. This section describes randomized sampling algorithms, also called Monte Carlo algorithms, that provide approximate answers whose accuracy depends on the number of samples generated. In recent years, Monte Carlo algorithms have become widely used in computer science to estimate quantities that are difficult to calculate exactly. For example, the simulated annealing algorithm described in Chapter 4 is a Monte Carlo method for optimization problems. In this section, we are interested in sampling applied to the computation of posterior probabilities. We describe two families of algorithms: direct sampling and Markov chain sampling. Two other approaches—uvariational methods and loopy propagation—are mentioned in the notes at the end of the chapter.

Direct sampling methods

The primitive element in any sampling algorithm is the generation of samples from a known probability distribution. For example, an unbiased coin can be thought of as a random variable Coin with values (heads, tails) and a prior distribution P(Coin) = (0.5,0.5). Sampling from this distribution is exactly like flipping the coin: with probability 0.5 it will return heads, and with probability 0.5 it will return tails. Given a source of random numbers in the range [0, 1], it is a simple matter to sample any distribution on a single variable. (See Exercise 14.9.)

The simplest kind of random sampling process for Bayesian networks generates events from a network that has no evidence associated with it. The idea is to sample each variable in turn, in topological order. The probability distribution from which the value is sampled is conditioned on the values already assigned to the variable’s parents. This algorithm is shown in Figure 14.12. We can illustrate its operation on the network in Figure 14.11(a), assuming an ordering [ Cloudy, Sprinkler, Rain, WetGrass|:

Sample from P( Cloudy) = (0.5,0.5); suppose this returns true.

Sample from P(Sprinkler| Cloudy = true) = (0.1,0.9); suppose this returns false.

1.

2.

3. Sample from P(Rain| Cloudy = true) = (0.8, 0.2); suppose this returns true. 4.

~~~

Sample from P returns true.

WetGrass|Sprinkler = false, Rain = true) = (0.9, 0.1); suppose this

In this case, PRIOR-SAMPLE returns the event [true, false, true, true).

It is easy to see that PRIOR-SAMPLE generates samples from the prior joint distribution specified by the network. First, let Spg(z1,. .., zy) be the probability that a specific event is generated by the PRIOR-SAMPLE algorithm. Just looking at the sampling process, we have

n Sps(x1...2n) = II P(x;|parents(X;)) i=1

because each sampling step depends only on the parent values. This expression should look

512 Chapter 14. Probabilistic Reasoning function PRIOR-SAMPLE(bn) returns an event sampled from the prior specified by bn

inputs: bn, a Bayesian network specifying joint distribution P(X, ..., X,,)

X «+ an event with n elements

fori = 1tondo

2; + a random sample from P(X; | parents(X;))

return x

Figure 14.12 A sampling algorithm that generates events from a Bayesian network. familiar, because it is also the probability of the event according to the Bayesian net’s representation of the joint distribution, as stated in Equation (14.1). That is, we have

Sps(zy...zy) = P(xy.. Ty) .

This simple fact makes it very easy to answer questions by using samples.

In any sampling algorithm, the answers are computed by counting the actual samples generated. Suppose there are N total samples, and let N(x1,..., zy) be the frequency of the specific event x1, ..., z,. We expect this frequency to converge, in the limit, to its expected value according to the sampling probability:

. Npglxzg,...,z lim NPSL: Tn) = Sps(@1s... an) = P(z1,..., 70). (14.4)

N-xo N For example, consider the event produced earlier: [true, false, true, true]. The sampling probability for this event is

Sps (true, false, true, true) = 0.5 x 0.9 x 0.8 x 0.9 = 0.324 .

Hence, in the limit of large IV, we expect 32.4% of the samples to be of this event.

Whenever we use an approximate equality (“~”") in what follows, we mean it in exactly this sense—that the estimated probability becomes exact in the large-sample limit. Such an

CONSISTENT estimate is called consistent. For example, one can produce a consistent estimate of the probability of any partially specified event 21, ..., Zs, where m < n, as follows:

P(x1,...,%m) ~ Nps(z1,...,2m)/N . (14.5) That is, the probability of the event can be estimated as the fraction of all complete events generated by the sampling process that match the partially specified event. For example, if we generate 1000 samples from the sprinkler network, and 511 of them have Rain = true, then the estimated probability of rain, written as P(Rain = true), is 0.511.

Rejection sampling in Bayesian networks REECTION Rejection sampling is a general method for producing samples from a hard-to-sample distri

bution given an easy-to-sample distribution. In its simplest form, it can be used to compute conditional probabilities—that is, to determine P(X |e). The REJECTION-SAMPLING algorithm is shown in Figure 14.13. First, it generates samples from the prior distribution specified

Section 14.5.

Approximate Inference in Bayesian Networks 513

[]

Let P(X |e) be the estimated distribution that the algorithm returns. From the definition of the algorithm, we have

P(X|e) = aNps(X,e) = lt

From Equation (14.5), this becomes P(X. e) P(e)

That is, rejection sampling produces a consistent estimate of the true probability.

Continuing with our example from Figure 14.11(a), let us assume that we wish to estimate P(Rain|Sprinkler = true), using 100 samples. Of the 100 that we generate, suppose that 73 have Sprinkler = false and are rejected, while 27 have Sprinkler = true; of the 27, 8 have Rain = true and 19 have Rain = false. Hence,

P(Rain|Sprinkler = true) ~ NORMALIZE((8, 19)) = (0.296,0.704) .

P(Xle) ~ =P(Xle).

The true answer is (0.3,0.7). As more samples are collected, the estimate will converge to the true answer. The standard deviation of the error in each probability will be proportional to 1/4/n, where n is the number of samples used in the estimate.

The biggest problem with rejection sampling is that it rejects so many samples! The fraction of samples consistent with the evidence e drops exponentially as the number of evidence variables grows, so the procedure is simply unusable for complex problems.

Notice that rejection sampling is very similar to the estimation of conditional probabilities directly from the real world. For example, to estimate P(Rain|RedSkyAtNight = true), one can simply count how often it rains after a red sky is observed the previous evening— ignoring those evenings when the sky is not red. (Here, the world itself plays the role of the

514 Chapter 14. Probabilistic Reasoning

sample generation algorithm.) Obviously, this could take a long time if the sky is very seldom red, and that is the weakness of rejection sampling.

Likelihood weighting

PKELIHOOD Likelihood weighting avoids the inefficiency of rejection sampling by generating only events that are consistent with the evidence e. We begin by describing how the algorithm works; then we show that it works correctly—that is, generates consistent probability estimates.

LIKELIHOOD-WEIGHTING (see Figure 14.14) fixes the values for the evidence variables E and samples only the remaining variables X and Y. This guarantees that each event generated is consistent with the evidence. Not all events are equal, however. Before tallying the counts in the distribution for the query variable, each event is weighted by the likelihood that the event accords to the evidence, as measured by the product of the conditional probabilities for each evidence variable, given its parents. Intuitively, events in which the actual evidence appears unlikely should be given less weight.

Let us apply the algorithm to the network shown in Figure 14.11(a), with the query P(Rain| Sprinkler = true, WetGrass = true). The process goes as follows: First, the weight w is set to 1.0. Then an event is generated:

1. Sample from P( Cloudy) = (0.5, 0.5); suppose this returns true. 2. Sprinkler is an evidence variable with value true. Therefore, we set

w «— w x P(Sprinkler = true| Cloudy = true) = 0.1. 3. Sample from P(Rain| Cloudy = true) = (0.8, 0.2); suppose this returns true. 4. WetGrass is an evidence variable with value true. Therefore, we set

w + w xX P(WetGrass = true| Sprinkler = true, Rain = true) = 0.099 .

Here WEIGHTED-S AMPLE returns the event [true, true, true, true] with weight 0.099, and this is tallied under Rain = true. The weight is low because the event describes a cloudy day, which makes the sprinkler unlikely to be on.

To understand why likelihood weighting works, we start by examining the sampling distribution Sg for WEIGHTED-SAMPLE. Remember that the evidence variables E are fixed with values e. We will call the other variables Z, that is, Z ={X } UY. The algorithm samples each variable in Z given its parent values:

! Sws(z,e) = II P(zi|parents(Z;)) . (14.6) i=1 Notice that Parents(Z;) can include both hidden variables and evidence variables. Unlike the prior distribution P(z), the distribution Sys pays some attention to the evidence: the sampled values for each Z; will be influenced by evidence among Z;’s ancestors. On the other hand, Sg pays less attention to the evidence than does the true posterior distribution P(zle), because the sampled values for each Z; ignore evidence among Z;’s non-ancestors.

6 Ideally, we would like use a sampling distribution equal to the true posterior P(z]e), to take all the evidence into account. This cannot be done efficiently, however. If it could, then we could approximate the desired probability to arbitrary accuracy with a polynomial number of samples. It can be shown that no such polynomialtime approximation scheme can exist.

Section 14.5.

Approximate Inference in Bayesian Networks 515

function LIKELIHOOD-WEIGHTING(X, e, bn, N) returns an estimate of P(X |e) inputs: X, the query variable e, evidence specified as an event bn, a Bayesian network N, the total number of samples to be generated local variables: W, a vector of weighted counts over X, initially zero

forj=1to N do

X, w + WEIGHTED-SAMPLE(bn)

Wz] — Wz] + w where z is the value of X in x return NORMALIZE(W[X])

function WEIGHTED-S AMPLE(bn, €) returns an event and a weight

X «an event with n elements; w «— 1 fori=1tondo if X; has a value z; ine then w «—w x P(X;= =z; | parents(X;}) else z; «— a random sample from P(X; | parents(X;)) return x, w

Figure 14.14 The likelihood weighting algorithm for inference in Bayesian networks.

The likelihood weight w makes up for the difference between the actual and desired sampling distributions. The weight for a given sample x, composed from z and e, is the product of the likelihoods for each evidence variable given its parents (some or all of which may be among the Z;s):

m w(z,e) = II P(e;|parents(E;)) . (14.7) i=1 Multiplying Equations (14.6) and (14.7), we see that the weighted probability of a sample has the particularly convenient form

l m Sws(z,e)w(z,e) = II P(zi|lparents(Z;)) II P(e;|parents(E;)) i=1 i=1 = P(z,e), (14.8)

because the two products cover all the variables in the network, allowing us to use Equation (14.1) for the joint probability.

Now it is easy to show that likelihood weighting estimates are consistent. For any particular value x of X, the estimated posterior probability can be calculated as follows:

P(zle) = a > Nws (x,y, e)w(x,y,e) from LIKELIHOOD-WEIGHTING y

~ ao > Sws(z,y,e)w(z,y,e) for large N y

516

Chapter 14. Probabilistic Reasoning

MARKOV CHAIN MONTE CARLO

= od > P(z,y,e) by Equation (14.8) y

= oP(x,e) = P(zle).

Hence, likelihood weighting returns consistent estimates.

Because likelihood weighting uses all the samples generated, it can be much more efficient than rejection sampling. It will, however, suffer a degradation in performance as the number of evidence variables increases. This is because most samples will have very low weights and hence the weighted estimate will be dominated by the tiny fraction of samples that accord more than an infinitesimal likelihood to the evidence. The problem is exacerbated if the evidence variables occur late in the variable ordering, because then the samples will be simulations that bear little resemblance to the reality suggested by the evidence.

Inference by Markov chain simulation

In this section, we describe the Markov chain Monte Carlo (MCMC) algorithm for inference in Bayesian networks. We will first describe what the algorithm does, then we will explain why it works and why it has such a complicated name.

The MCMC algorithm

Unlike the other two sampling algorithms, which generate each event from scratch, MCMC generates each event by making a random change to the preceding event. It is therefore helpful to think of the network as being in a particular current state specifying a value for every variable. The next state is generated by randomly sampling a value for one of the nonevidence variables X;, conditioned on the current values of the variables in the Markov blanket of X;. (Recall from page 499 that the Markov blanket of a variable consists of its parents, children, and children’s parents.) MCMC therefore wanders randomly around the state space—the space of possible complete assignments—flipping one variable at a time, but keeping the evidence variables fixed.

Consider the query P(Rain|Sprinkler = true, Wet Grass = true) applied to the network in Figure 14.11(a). The evidence variables Sprinkler and WetGrass are fixed to their observed values and the hidden variables Cloudy and Rain are initialized randomly—Iet us say to true and false respectively. Thus, the initial state is [true, true, false, true]. Now the following steps are executed repeatedly:

1. Cloudy is sampled, given the current values of its Markov blanket variables: in this case, we sample from P(Cloudy| Sprinkler = true, Rain = false). (Shortly, we will show how to calculate this distribution.) Suppose the result is Cloudy = false. Then the new current state is [false, true, false, true].

2. Rain is sampled, given the current values of its Markov blanket variables: in this case, we sample from P(Rain| Cloudy = false, Sprinkler = true, WetGrass = true). Suppose this yields Rain = true. The new current state is [false, true, true, true].

Each state visited during this process is a sample that contributes to the estimate for the query variable Rain. If the process visits 20 states where Rain is true and 60 states where Rain is

Section 14.5.

Approximate Inference in Bayesian Networks 517

| = Ro

TRANSITION PROBABILITY

MARKOV CHAIN

STATIONARY DISTRIBUTION

function MCMC-ASK(X,e, bn, N) returns an estimate of P(X |e) local variables: N[X], a vector of counts over X, initially zero Z, the nonevidence variables in bn x, the current state of the network, initially copied from e

initialize x with random values for the variables in Z for j=1to N do for each Z; in Z do sample the value of Z; in x from P(Z;|mb(Z;)) given the values of M B(Z;)} in x Niz] «N[z] + 1 where z is the value of X in x return NORMALIZE(N[X])

Figure 14.15 The MCMC algorithm for approximate inference in Bayesian networks.

false, then the answer to the query is NORMALIZE((20, 60)) = (0.25,0.75). The complete algorithm is shown in Figure 14.15.

Why MCMC works

We will now show that MCMC returns consistent estimates for posterior probabilities. The material in this section is quite technical, but the basic claim is straightforward: the sampling process settles into a “dynamic equilibrium” in which the long-run fraction of time spent in each state is exactly proportional to its posterior probability. This remarkable property follows from the specific transition probability with which the process moves from one state to another, as defined by the conditional distribution given the Markov blanket of the variable being sampled.

Let g(x — x’) be the probability that the process makes a transition from state X to state x’. This transition probability defines what is called a Markov chain on the state space. (Markov chains will also figure prominently in Chapters 15 and 17.) Now suppose that we run the Markov chain for ¢ steps, and let 7; (x) be the probability that the system is in state x at time ¢. Similarly, let 7:41 (x) be the probability of being in state x’ at time ¢ + 1. Given 7:(X), we can calculate 741 (x") by summing, for all states the system could be in at time ¢, the probability of being in that state times the probability of making the transition to x’:

me1(X) =) m(x)g(x — x).

We will say that the chain has reached its stationary distribution if 7; =m |. Let us call this stationary distribution 7; its defining equation is therefore n(x) = > w(x)a(x — x) forall x’. (14.9) X Under certain standard assumptions about the transition probability distribution g,” there is exactly one distribution 7 satisfying this equation for any given gq.

7 The Markov chain defined by ¢ must be ergodic—that is, essentially, every state must be reachable from every other, and there can be no strictly periodic cycles.

518

Chapter 14. Probabilistic Reasoning

DETAILED BALANCE

[]

GIBBS SAMPLER

Equation (14.9) can be read as saying that the expected “outflow” from each state (i.e., its current “population”) is equal to the expected “inflow” from all the states. One obvious way to satisfy this relationship is if the expected flow between any pair of states is the same in both directions. This is the property of detailed balance:

r(x)g(x — x) = 1(xX)g(x = x) forallx, x . (14.10)

We can show that detailed balance implies stationarity simply by summing over x in Equation (14.10). We have

X X X where the last step follows because a transition from x’ is guaranteed to occur.

Now we will show that the transition probability ¢(x — x’) defined by the sampling step in MCMCAsK satisfies the detailed balance equation with a stationary distribution equal to P(x|e), (the true posterior distribution on the hidden variables). We will do this in two steps. First, we will define a Markov chain in which each variable is sampled conditionally on the current values of all the other variables, and we will show that this satisfies detailed balance. Then, we will simply observe that, for Bayesian networks, doing that is equivalent to sampling conditionally on the variable’s Markov blanket (see page 499).

Let X; be the variable to be sampled, and let X; be all the hidden variables other than X;. Their values in the current state are x; and X;. If we sample a new value x for X; conditionally on all the other variables, including the evidence, we have

q(x — x’) = q(x, Xi) Id (2,%7)) = P (xxi, e) :

This transition probability is called the Gibbs sampler and is a particularly convenient form of MCMC. Now we show that the Gibbs sampler is in detailed balance with the true posterior:

r(x)a(x — X) = P(x|e) P(a/[%5, ¢) = P(a;, Kile) P(a/%, ©) = P(x;|%;, e)P(X;|e)P(z}|x;,e) (using the chain rule on the first term) = P(x;|%;,e)P(z},X;le) (using the chain rule backwards) = 7(x)g(x — x). As stated on page 499, a variable is independent of all other variables given its Markov blanket; hence,

P(x}[xi, €) = P(x;lmb(X:)) where mb(X;) denotes the values of the variables in X;’s Markov blanket, M B(X;). As shown in Exercise 14.10, the probability of a variable given its Markov blanket is proportional to the probability of the variable given its parents times the probability of each child given its respective parents:

P(zflmb(X;)) = a P(z}|parents(X;)) x II P(yjlparents(Y;)) . (14.11)

Y;eChildren(X;)

Hence, to flip each variable X;, the number of multiplications required is equal to the number of X;’s children.

Section 14.6.

Extending Probability to First-Order Representations 519

We have discussed here only one simple variant of MCMC, namely the Gibbs sampler. In its most general form, MCMC is a powerful method for computing with probability models and many variants have been developed, including the simulated annealing algorithm presented in Chapter 4, the stochastic satisfiability algorithms in Chapter 7, and the Metropolis—Hastings sampler in Chapter 15.

RELATIONAL PROBABILITY MODEL

In Chapter 8, we explained the representational advantages possessed by first-order logic in comparison to propositional logic. First-order logic commits to the existence of objects and relations among them and can express facts about some or all of the objects in a domain. This often results in representations that are vastly more concise than the equivalent propositional descriptions. Now, Bayesian networks are essentially propositional: the set of variables is fixed and finite, and each has a fixed domain of possible values. This fact limits the applicability of Bayesian networks. If we can find a way to combine probability theory with the expressive power of first-order representations, we expect to be able to increase dramatically the range of problems that can be handled.

The basic insight required to achieve this goal is the following: In the propositional context, a Bayesian network specifies probabilities over atomic events, each of which specifies a value for each variable in the network. Thus, an atomic event is a model or possible world, in the terminology of propositional logic. In the first-order context, a model (with its interpretation) specifies a domain of objects, the relations that hold among those objects, and a mapping from the constants and predicates of the knowledge base to the objects and relations in the model. Therefore, a first-order probabilistic knowledge base should specify probabilities for all possible first-order models. Let p1(M) be the probability assigned to model M by the knowledge base. For any first-order sentence ¢, the probability P(¢) is given in the usual way by summing over the possible worlds where ¢ is true:

Pig)= YY. uM). (14.12) M:¢ is true in M So far, so good. There is, however, a problem: the set of first-order models is infinite. This means that (1) the summation could be infeasible, and (2) specifying a complete, consistent distribution over an infinite set of worlds could be very difficult.

Let us scale back our ambition, at least temporarily. In particular, let us devise a restricted language for which there are only finitely many models of interest. There are several ways to do this. Here, we present relational probability models, or RPMs, which borrow ideas from semantic networks (Chapter 10) and from object-relational databases. Other approaches are discussed in the bibliographical and historical notes.

RPMs allow constant symbols that name objects. For example, let ProfSmith be the name of a professor, and let Jones be the name of a student. Each object is an instance of a class; for example, ProfSmith is a Professor and Jones is a Student. We assume that the class of every constant symbol is known.

520

Chapter 14. Probabilistic Reasoning

SIMPLE FUNCTION

COMPLEX FUNCTION

Our function symbols will be divided into two kinds. The first kind, simple functions, maps an object not to another structured object, but to a value from a fixed domain of values, just like a random variable. For example, Intelligence(Jones) and Funding(ProfSmith) might be hi or lo; Success(Jones) and Fame(ProfSmith) may be true or false. Function symbols must not be applied to values such as true and false, so it is not possible to have nesting of simple functions. In this way, we avoid one source of infinities. The value of a simple function applied to a given object may be observed or unknown; these will be the basic random variables of our representation ?

We also allow complex functions, which map objects to other objects. For example, Advisor(Jones) may be ProfSmith. Each complex function has a specified domain and range, which are classes. For example, the domain of Advisor is Student and the range is Professor. Functions apply only to objects of the right class; for instance, the Advisor of ProfSmith is undefined. Complex functions may be nested: DeptHead( Advisor(Jones)) could be ProfMoore. We will assume (for now) that the values of all complex functions are known for all constant symbols. Because the KB is finite, this implies that every chain of complex function applications leads to one of a finite number of objects.’

The last element we need is the probabilistic information. For each simple function, we specify a set of parents, just as in Bayesian networks. The parents can be other simple functions of the same object; for example, the Funding of a Professor might depend on his or her Fame. The parents can also be simple functions of related objects—for example, the Success of a student could depend on the Intelligence of the student and the Fame of the student’s advisor. These are really universally quantified assertions about the parents of all the objects in a class. Thus, we could write

Vz x € Student = Parents(Success(x)) = {Intelligence(x), Fame (Advisor(zx))} .

(Less formally, we can draw diagrams like Figure 14.16(a).) Now we specify the conditional probability distribution for the child, given its parents. For example, we might say that

Vz x € Student = P(Success(x) = true|Intelligence(x) = hi, Fame(Advisor(z)) = true) = 0.95 .

Just as in semantic networks, we can attach the conditional distribution to the class itself, so that the instances inherit the dependencies and conditional probabilities from the class.

The semantics for the RPM language assumes that every constant symbol refers to a distinct object—the unique names assumption described in Chapter 10. Given this assumption and the restrictions listed previously, it can be shown that every RPM generates a fixed, finite set of random variables, each of which is a simple function applied to a constant symbol. Then, provided that the parent—child dependencies are acyclic, we can construct an equivalent Bayesian network. That is, the RPM and the Bayesian network specify identical probabili

8 They play a role very similar to that of the ground atomic sentences generated in the propositionalization process described in Section 9.1.

9 This restriction means that we cannot use complex functions such as Father and Mother, which lead to potentially infinite chains that would have to end with an unknown object. We revisit this restriction later.

Section 14.6. Extending Probability to First-Order Representations 521 Professor t«— ProfSmith Funding Advisor Student ) le Jones Intelligence(Bloggs) Intelligence(Jones) Intelligence =— Bloggs

[]

RECURSIVE DEPENDENCY

RELATIONAL UNCERTAINTY

IDENTITY UNCERTAINTY

(a) (b)

Figure 14.16 (a) An RPM describing two classes: Professor and Student. There are two professors and two students, and ProfSmith is the advisor of both students. (b) The Bayesian network equivalent to the RPM in (a).

ties for each possible world. Figure 14.16(b) shows the Bayesian network corresponding to the RPM in Figure 14.16(a). Notice that the Advisor links in the RPM are absent in the Bayesian network. This is because they are fixed and known. They appear implicitly in the network topology, however; for example, Success( Jones) has Fame ( ProfSmith) as a parent because Advisor(Jones) is ProfSmith. In general, the relations that hold among the objects determine the pattern of dependencies among the properties of those objects.

There are several ways to increase the expressive power of RPMs. We can allow recursive dependencies among variables to capture certain kinds of recurring relationships. For example, suppose that addiction to fast food is caused by the McGene. Then, for any x, McGene(x) depends on McGene(Father(x)) and McGene(Mother(z)), which depend in turn on McGene(Father(Father(x))), McGene(Mother(Father(z))), and so on. Even though such knowledge bases correspond to Bayesian networks with infinitely many random variables, solutions can sometimes be obtained from fixed-point equations. For example, the equilibrium distribution of the McGene can be calculated, given the conditional probability of inheritance. Another very important family of recursive knowledge bases consists of the temporal probability models described in Chapter 15. In these models, properties of the state at time ¢ depend on properties of the state at time ¢ — 1, and so on.

RPMs can also be extended to allow for relational uncertainty—that is, uncertainty about the values of complex functions. For example, we may not know who Advisor(Jones) is. Advisor(Jones) then becomes a random variable, with possible values ProfSmith and ProfMoore. The corresponding network is shown in Figure 14.17.

There can also be identity uncertainty; for example, we might not know whether Mary and ProfSmith are the same person. With identity uncertainty, the number of objects and propositions can vary across possible worlds. A world where Mary and ProfSmith are the same person has one fewer object than a world in which they are different people. This makes the inference process more complicated, but the basic principle established in Equation (14.12) still holds: the probability of any sentence is well defined and can be calculated.

522

Chapter 14. Probabilistic Reasoning

[]

Success(Jones) BN

Figure 14.17 Part of the Bayesian network corresponding to an RPM in which Advisor (Jones) is unknown, but is either ProfSmith or ProfMoore. The choice of advisor depends on how much funding each professor has. Notice that Success(Jones) will now depend on the Fame of both professors, although the value of Advisor(Jones) determines which one actually has an influence.

Identity uncertainty is particularly important for robots and for embedded sensor systems that must keep track of multiple objects. We return to this problem in Chapter 15.

Let us now examine the question of inference. Clearly, inference can be done in the equivalent Bayesian network, provided that we restrict the RPM language so that the equivalent network is finite and has a fixed structure. This is analogous to the way in which first-order logical inference can be done via propositional inference on the equivalent propositional knowledge base. (See Section 9.1.) As in the logical case, the equivalent network could be too large to construct, let alone evaluate. Dense interconnections are also a problem. (See Exercise 14.12.) Approximation algorithms. such as MCMC (Section 14.5), are therefore very useful for RPM inference.

When MCMC is applied to the equivalent Bayesian network for a simple RPM knowledge base with no relational or identity uncertainty, the algorithm samples from the space of possible worlds defined by the values of simple functions of the objects. It is easy to see that this approach can be extended to handle relational and identity uncertainty as well. In that case, a transition between possible worlds might change the value of a simple function or it might change a complex function, and so lead to a change in the dependency structure. Transitions might also change the identity relations among the constant symbols. Thus, MCMC seems to be an elegant way to handle inference for quite expressive first-order probabilistic knowledge bases.

Research in this area is still at an early stage, but already it is becoming clear that firstorder probabilistic reasoning yields a tremendous increase in the effectiveness of Al systems at handling uncertain information. Potential applications include computer vision, natural language understanding, information retrieval, and situation assessment. In all of these areas, the set of objects——and hence the set of random variables—is not known in advance, so purely “propositional” methods, such as Bayesian networks, are incapable of representing the situation completely. They have been augmented by search over the space of model, but RPMs allow reasoning about this uncertainty in a single model.

Section 14.7. Other Approaches to Uncertain Reasoning 523

14.7 OTHER APPROACHES TO UNCERTAIN REASONING

Other sciences (e.g., physics, genetics, and economics) have long favored probability as a model for uncertainty. In 1819, Pierre Laplace said “Probability theory is nothing but common sense reduced to calculation.” In 1850, James Maxwell said “the true logic for this world is the calculus of Probabilities, which takes account of the magnitude of the probability which is, or ought to be, in a reasonable man’s mind.”

Given this long tradition, it is perhaps surprising that Al has considered many alternatives to probability. The earliest expert systems of the 1970s ignored uncertainty and used strict logical reasoning, but it soon became clear that this was impractical for most real-world domains. The next generation of expert systems (especially in medical domains) used probabilistic techniques. Initial results were promising, but they did not scale up because of the exponential number of probabilities required in the full joint distribution. (Efficient Bayesian network algorithms were unknown then.) As a result, probabilistic approaches fell out of favor from roughly 1975 to 1988, and a variety of alternatives to probability were tried for a variety of reasons:

e One common view is that probability theory is essentially numerical, whereas human judgmental reasoning is more “qualitative.” Certainly, we are not consciously aware of doing numerical calculations of degrees of belief. (Neither are we aware of doing unification, yet we seem to be capable of some kind of logical reasoning.) It might be that we have some kind of numerical degrees of belief encoded directly in strengths of connections and activations in our neurons. In that case, the difficulty of conscious access to those strengths is not surprising. One should also note that qualitative reasoning mechanisms can be built directly on top of probability theory, so that the “no numbers” argument against probability has little force. Nonetheless, some qualitative schemes have a good deal of appeal in their own right. One of the best studied is default reasoning, which treats conclusions not as “believed to a certain degree,” but as “believed until a better reason is found to believe something else.” Default reasoning is covered in Chapter 10.

o Rule-based approaches to uncertainty also have been tried. Such approaches hope to build on the success of logical rule-based systems, but add a sort of “fudge factor” to each rule to accommodate uncertainty. These methods were developed in the mid-1970s and formed the basis for a large number of expert systems in medicine and other areas.

e One area that we have not addressed so far is the question of ignorance, as opposed to uncertainty. Consider the flipping of a coin. If we know that the coin is fair, then a probability of 0.5 for heads is reasonable. If we know that the coin is biased, but we do not know which way, then 0.5 is the only reasonable probability. Obviously, the two cases are different, yet probability seems not to distinguish them. The Dempster— Shafer theory uses interval-valued degrees of belief to represent an agent’s knowledge of the probability of a proposition. Other methods using second-order probabilities are also discussed.

524

Chapter 14. Probabilistic Reasoning

LOCALITY

DETACHMENT

TRUTH FUNCTIONALITY

e Probability makes the same ontological commitment as logic: that events are true or false in the world, even if the agent is uncertain as to which is the case. Researchers in fuzzy logic have proposed an ontology that allows vagueness: that an event can be “sort of” true. Vagueness and uncertainty are in fact orthogonal issues, as we will see.

The next three subsections treat some of these approaches in slightly more depth. We will not provide detailed technical material, but we cite references for further study.

Rule-based methods for uncertain reasoning

Rule-based systems emerged from early work on practical and intuitive systems for logical inference. Logical systems in general, and logical rule-based systems in particular, have three desirable properties:

¢& Locality: In logical systems, whenever we have a rule of the form A = B, we can conclude B, given evidence A, without worrying about any other rules. In probabilistic systems, we need to consider all the evidence in the Markov blanket.

¢& Detachment: Once a logical proof is found for a proposition B, the proposition can be used regardless of how it was derived. That is, it can be detached from its justification. In dealing with probabilities, on the other hand, the source of the evidence for a belief is important for subsequent reasoning.

¢& Truth-functionality: In logic, the truth of complex sentences can be computed from the truth of the components. Probability combination does not work this way, except under strong global independence assumptions.

There have been several attempts to devise uncertain reasoning schemes that retain these advantages. The idea is to attach degrees of belief to propositions and rules and to devise purely local schemes for combining and propagating those degrees of belief. The schemes are also truth-functional; for example, the degree of belief in A VV B is a function of the belief in A and the belief in B.

The bad news for rule-based systems is that the properties of locality, detachment, and truth-functionality are simply not appropriate for uncertain reasoning. Let us look at truthfunctionality first. Let H; be the event that a fair coin flip comes up heads, let 77 be the event that the coin comes up tails on that same flip, and let Il be the event that the coin comes up heads on a second flip. Clearly, all three events have the same probability, 0.5, and so a truth-functional system must assign the same belief to the disjunction of any two of them. But we can see that the probability of the disjunction depends on the events themselves and not just on their probabilities:

[]

It gets worse when we chain evidence together. Truth-functional systems have rules of the form A + B that allow us to compute the belief in B as a function of the belief in the rule

Section 14.7.

Other Approaches to Uncertain Reasoning 525

CERTAINTY FACTORS

DEMPSTER-SHAFER

BELIEF FUNCTION

and the belief in A. Both forwardand backward-chaining systems can be devised. The belief in the rule is assumed to be constant and is usually specified by the knowledge engineer—for example, as A v-qq9 B.

Consider the wet-grass situation from Figure 14.11(a). If we wanted to be able to do both causal and diagnostic reasoning, we would need the two rules

Rain v— WetGrass and WetGrass v— Rain .

These two rules form a feedback loop: evidence for Rain increases the belief in WetGrass, which in turn increases the belief in Rain even more. Clearly, uncertain reasoning systems need to keep track of the paths along which evidence is propagated.

Intercausal reasoning (or explaining away) is also tricky. Consider what happens when we have the two rules

Sprinkler — WetGrass and WetGrass + Rain .

Suppose we see that the sprinkler is on. Chaining forward through our rules, this increases the belief that the grass will be wet, which in turn increases the belief that it is raining. But this is ridiculous: the fact that the sprinkler is on explains away the wet grass and should reduce the belief in rain. A truth-functional system acts as if it also believes Sprinkler + Rain.

Given these difficulties, how is it possible that truth-functional systems were ever considered useful? The answer lies in restricting the task and in carefully engineering the rule base so that undesirable interactions do not occur. The most famous example of a truthfunctional system for uncertain reasoning is the certainty factors model, which was developed for the MYCIN medical diagnosis program and was widely used in expert systems of the late 1970s and 1980s. Almost all uses of certainty factors involved rule sets that were either purely diagnostic (as in MYCIN) or purely causal. Furthermore, evidence was entered only at the “roots” of the rule set, and most rule sets were singly connected. Heckerman (1986) has shown that, under these circumstances, a minor variation on certainty-factor inference was exactly equivalent to Bayesian inference on polytrees. In other circumstances, certainty factors could yield disastrously incorrect degrees of belief through overcounting of evidence. As rule sets became larger, undesirable interactions between rules became more common, and practitioners found that the certainty factors of many other rules had to be “tweaked” when new rules were added. Needless to say, the approach is no longer recommended.

Representing ignorance: Dempster-Shafer theory

The Dempster-Shafer theory is designed to deal with the distinction between uncertainty and ignorance. Rather than computing the probability of a proposition, it computes the probability that the evidence supports the proposition. This measure of belief is called a belief function, written Bel(X).

We return to coin flipping for an example of belief functions. Suppose a shady character comes up to you and offers to bet you $10 that his coin will come up heads on the next flip. Given that the coin might or might not be fair, what belief should you ascribe to the event that it comes up heads? Dempster—Shafer theory says that because you have no evidence either way, you have to say that the belief Bel( Heads) = 0 and also that Bel(— Heads) = 0.

526

Chapter 14. Probabilistic Reasoning

FUZZY SET THEORY

This makes Dempster—Shafer reasoning systems skeptical in a way that has some intuitive appeal. Now suppose you have an expert at your disposal who testifies with 90% certainty that the coin is fair (i.e., he is 90% sure that P(Heads) = 0.5). Then Dempster-Shafer theory gives Bel(Heads) = 0.9 x 0.5 = 0.45 and likewise Bel(— Heads) = 0.45. There is still a 10 percentage point “gap” that is not accounted for by the evidence. “Dempster’s rule” (Dempster, 1968) shows how to combine evidence to give new values for Bel, and Shafer’s work extends this into a complete computational model.

As with default reasoning, there is a problem in connecting beliefs to actions. With probabilities, decision theory says that if P(Heads) = P(—Heads) = 0.5, then (assuming that winning $10 and losing $10 are considered equal magnitude opposites) the reasoner will be indifferent between the action of accepting and declining the bet. A Dempster— Shafer reasoner has Bel(—Heads) = 0 and thus no reason to accept the bet, but then it also has Bel( Heads) = 0 and thus no reason to decline it. Thus, it seems that the Dempster— Shafer reasoner comes to the same conclusion about how to act in this case. Unfortunately, Dempster-Shafer theory allows no definite decision in many other cases where probabilistic inference does yield a specific choice. In fact, the notion of utility in the Dempster—Shafer model is not yet well understood.

One interpretation of Dempster-Shafer theory is that it defines a probability interval: the interval for Heads is [0, 1] before our expert testimony and [0.45, 0.55] after. The width of the interval might be an aid in deciding when we need to acquire more evidence: it can tell you that the expert’s testimony will help you if you do not know whether the coin is fair, but will not help you if you have already learned that the coin is fair. However, there are no clear guidelines for how to do this, because there is no clear meaning for what the width of an interval means. In the Bayesian approach, this kind of reasoning can be done easily by examining how much one’s belief would change if one were to acquire more evidence. For example, knowing whether the coin is fair would have a significant impact on the belief that it will come up heads, and detecting an asymmetric weight would have an impact on the belief that the coin is fair. A complete Bayesian model would include probability estimates for factors such as these, allowing us to express our “ignorance” in terms of how our beliefs would change in the face of future information gathering.

Representing vagueness: Fuzzy sets and fuzzy logic

Fuzzy set theory is a means of specifying how well an object satisfies a vague description. For example, consider the proposition “Nate is tall.” Is this true, if Nate is 5° 10”? Most people would hesitate to answer “true” or “false,” preferring to say, “sort of.” Note that this is not a question of uncertainty about the external world—we are sure of Nate's height. The issue is that the linguistic term “tall” does not refer to a sharp demarcation of objects into two classes—there are degrees of tallness. For this reason, fuzzy set theory is not a method for uncertain reasoning at all. Rather, fuzzy set theory treats Tall as a fuzzy predicate and says that the truth value of Tall(Nate) is a number between 0 and 1, rather than being just true or false. The name “fuzzy set” derives from the interpretation of the predicate as implicitly defining a set of its members—a set that does not have sharp boundaries.

Section 14.7.

Other Approaches to Uncertain Reasoning 527

FUZZY LOGIC

FUZZY CONTROL

RANDOM SETS

Fuzzy logic is a method for reasoning with logical expressions describing membership in fuzzy sets. For example, the complex sentence Tall(Nate) A Heavy( Nate) has a fuzzy truth value that is a function of the truth values of its components. The standard rules for evaluating the fuzzy truth, T', of a complex sentence are

T(AA B) =min(T (A), T(B))

T(AV B) = max(T(A),T(B))

T(-~A)=1-T(A).

Fuzzy logic is therefore a truth-functional system—a fact that causes serious difficulties. For example, suppose that T'( Tall ( Nate)) = 0.6 and T'( Heavy(Nate)) = 0.4. Then we have T(Tall( Nate) A T(Heavy(Nate)) = 0.4, which seems reasonable, but we also get the result T(Tall( Nate) A —Tall( Nate)) = 0.4, which does not. Clearly, the problem arises from the inability of a truth-functional approach to take into account the correlations or anticorrelations among the component propositions.

Fuzzy control is a methodology for constructing control systems in which the mapping between real-valued input and output parameters is represented by fuzzy rules. Fuzzy control has been very successful in commercial products such as automatic transmissions, video cameras, and electric shavers. Critics (see, e.g., Elkan, 1993) argue that these applications are successful because they have small rule bases, no chaining of inferences, and tunable parameters that can be adjusted to improve the system’s performance. The fact that they are implemented with fuzzy operators might be incidental to their success; the key is simply to provide a concise and intuitive way to specify a smoothly interpolated, real-valued function.

There have been attempts to provide an explanation of fuzzy logic in terms of probability theory. One idea is to view assertions such as “Nate is Tall” as discrete observations made concerning a continuous hidden variable, Nate’s actual Height. The probability model specifies P(Observer says Nate is tall | Height), perhaps using a probit distribution as described on page 503. A posterior distribution over Nate’s height can then be calculated in the usual way, for example if the model is part of a hybrid Bayesian network. Such an approach is not truth-functional, of course. For example, the conditional distribution

P(Observer says Nate is tall and heavy | Height, Weight)

allows for interactions between height and weight in the causing of the observation. Thus, someone who is eight feet tall and weighs 190 pounds is very unlikely to be called “tall and heavy,” even though “eight feet” counts as “tall” and “190 pounds” counts as “heavy.”

Fuzzy predicates can also be given a probabilistic interpretation in terms of random sets—that is, random variables whose possible values are sets of objects. For example, Tall is a random set whose possible values are sets of people. The probability P( Tall = S51), where 5S; is some particular set of people, is the probability that exactly that set would be identified as “tall” by an observer. Then the probability that “Nate is tall” is the sum of the probabilities of all the sets of which Nate is a member.

Both the hybrid Bayesian network approach and the random sets approach appear to capture aspects of fuzziness without introducing degrees of truth. Nonetheless, there remain many open issues concerning the proper representation of linguistic observations and continuous quantities—issues that have been neglected by most outside the fuzzy community.

528 Chapter 14. Probabilistic Reasoning

14.8 SUMMARY

This chapter has described Bayesian networks, a well-developed representation for uncertain knowledge. Bayesian networks play a role roughly analogous to that of propositional logic for definite knowledge.

e A Bayesian network is a directed acyclic graph whose nodes correspond to random variables; each node has a conditional distribution for the node, given its parents.

e Bayesian networks provide a concise way to represent conditional independence relationships in the domain.

e A Bayesian network specifies a full joint distribution; each joint entry is defined as the product of the corresponding entries in the local conditional distributions. A Bayesian network is often exponentially smaller than the full joint distribution.

e Many conditional distributions can be represented compactly by canonical families of distributions. Hybrid Bayesian networks, which include both discrete and continuous variables, use a variety of canonical distributions.

e Inference in Bayesian networks means computing the probability distribution of a set of query variables, given a set of evidence variables. Exact inference algorithms, such as variable elimination, evaluate sums of products of conditional probabilities as efficiently as possible.

¢ In polytrees (singly connected networks), exact inference takes time linear in the size of the network. In the general case, the problem is intractable.

e Stochastic approximation techniques such as likelihood weighting and Markov chain Monte Carlo can give reasonable estimates of the true posterior probabilities in a network and can cope with much larger networks than can exact algorithms.

e Probability theory can be combined with representational ideas from first-order logic to produce very powerful systems for reasoning under uncertainty. Relational probability models (RPMs) include representational restrictions that guarantee a well-defined probability distribution that can be expressed as an equivalent Bayesian network.

e Various alternative systems for reasoning under uncertainty have been suggested. Generally speaking, truth-functional systems are not well suited for such reasoning.

BIBLIOGRAPHICAL AND HISTORICAL NOTES

The use of networks to represent probabilistic information began early in the 20th century, with the work of Sewall Wright on the probabilistic analysis of genetic inheritance and animal growth factors (Wright, 1921, 1934). One of his networks appears on the cover of this book. I. J. Good (1961), in collaboration with Alan Turing, developed probabilistic representations and Bayesian inference methods that could be regarded as a forerunner of modern Bayesian

Section 14.8.

Summary 529

NONSERIAL DYNAMIC PROGRAMMING

networks—although the paper is not often cited in this context.!9 The same paper is the original source for the noisy-OR model.

The influence diagram representation for decision problems, which incorporated a DAG representation for random variables, was used in decision analysis in the late 1970s (see Chapter 16), but only enumeration was used for evaluation. Judea Pearl developed the message-passing method for carrying out inference in tree networks (Pearl, 1982a) and polytree networks (Kim and Pearl, 1983) and explained the importance of constructing causal rather than diagnostic probability models, in contrast to the certainty-factor systems then in vogue. The first expert system using Bayesian networks was CONVINCE (Kim, 1983; Kim and Pearl, 1987). More recent systems include the MUNIN system for diagnosing neuromuscular disorders (Andersen et al., 1989) and the PATHFINDER system for pathology (Heckerman, 1991). By far the most widely used Bayesian network systems have been the diagnosis-and-repair modules (e.g., the Printer Wizard) in Microsoft Windows (Breese and Heckerman, 1996) and the Office Assistant in Microsoft Office (Horvitz et al., 1998).

Pearl (1986) developed a clustering algorithm for exact inference in general Bayesian networks, utilizing a conversion to a directed polytree of clusters in which message passing was used to achieve consistency over variables shared between clusters. A similar approach, developed by the statisticians David Spiegelhalter and Steffen Lauritzen (Spiegelhalter, 1986; Lauritzen and Spiegelhalter, 1988), is based on conversion to an undirected (Markov) network. This approach is implemented in the HUGIN system, an efficient and widely used tool for uncertain reasoning (Andersen et al., 1989). Ross Shachter, working in the influence diagram community, developed an exact method based on goal-directed reduction of the network, using posterior-preserving transformations (Shachter, 1986).

The variable elimination method described in the chapter is closest in spirit to Shachter’s method, from which emerged the symbolic probabilistic inference (SPI) algorithm (Shachter et al., 1990). SPI attempts to optimize the evaluation of expression trees such as that shown in Figure 14.8. The algorithm we describe is closest to that developed by Zhang and Poole (1994, 1996). Criteria for pruning irrelevant variables were developed by Geiger ef al. (1990) and by Lauritzen er al. (1990); the criterion we give is a simple special case of these. Rina Dechter (1999) shows how the variable elimination idea is essentially identical to nonserial dynamic programming (Bertele and Brioschi, 1972), an algorithmic approach that can be applied to solve a range of inference problems in Bayesian networks—for example, finding the most probable explanation for a set of observations. This connects Bayesian network algorithms to related methods for solving CSPs and gives a direct measure of the complexity of exact inference in terms of the hypertree width of the network.

The inclusion of continuous random variables in Bayesian networks was considered by Pearl (1988) and Shachter and Kenley (1989); these papers discussed networks containing only continuous variables with linear Gaussian distributions. The inclusion of discrete variables has been investigated by Lauritzen and Wermuth (1989) and implemented in the

10 I. J. Good was chief statistician for Turing’s code-breaking team in World War IL. In 2001: A Space Odyssey (Clarke, 1968a), Good and Minsky are credited with making the breakthrough that led to the development of the HAL 9000 computer.

530

Chapter 14. Probabilistic Reasoning

VARIATIONAL APPROXIMATION

VARIATIONAL PARAMETERS

MEAN FIELD

cHUGIN system (Olesen, 1993). The probit distribution was studied first by Finney (1947), who called it the sigmoid distribution. It has been used widely for modeling discrete choice phenomena and can be extended to handle more than two choices (Daganzo, 1979). Bishop (1995) gives a justification for the use of the logit distribution.

Cooper (1990) showed that the general problem of inference in unconstrained Bayesian networks is NP-hard, and Paul Dagum and Mike Luby (1993) showed the corresponding approximation problem to be NP-hard. Space complexity is also a serious problem in both clustering and variable elimination methods. The method of cutset conditioning, which was developed for CSPs in Chapter 5, avoids the construction of exponentially large tables. In a Bayesian network, a cutset is a set of nodes that, when instantiated, reduces the remaining nodes to a polytree that can be solved in linear time and space. The query is answered by summing over all the instantiations of the cutset, so the overall space requirement is still linear (Pearl, 1988). Darwiche (2001) describes a recursive conditioning algorithm that allows a complete range of space/time tradeoffs.

The development of fast approximation algorithms for Bayesian network inference is a very active area, with contributions from statistics, computer science, and physics. The rejection sampling method is a general technique that is long known to statisticians; it was first applied to Bayesian networks by Max Henrion (1988), who called it logic sampling. Likelihood weighting, which was developed by Fung and Chang (1989) and Shachter and Peot (1989), is an example of the well-known statistical method of importance sampling. A large-scale application of likelihood weighting to medical diagnosis appears in Shwe and Cooper (1991). Cheng and Druzdzel (2000) describe an adaptive version of likelihood weighting that works well even when the evidence has very low prior likelihood.

Markov chain Monte Carlo (MCMC) algorithms began with the Metropolis algorithm, due to Metropolis er al. (1953), which was also the source of the simulated annealing algorithm described in Chapter 4. The Gibbs sampler was devised by Geman and Geman (1984) for inference in undirected Markov networks. The application of MCMC to Bayesian networks is due to Pearl (1987). The papers collected by Gilks ef al. (1996) cover a wide variety of applications of MCMC, several of which were developed in the well-known BUGS package (Gilks et al., 1994).

There are two very important families of approximation methods that we did not cover in the chapter. The first is the family of variational approximation methods, which can be used to simplify complex calculations of all kinds. The basic idea is to propose a reduced version of the original problem that is simple to work with, but that resembles the original problem as closely as possible. The reduced problem is described by some variational parameters \ that are adjusted to minimize a distance function D between the original and the reduced problem, often by solving the system of equations 9D/9X =0. In many cases, strict upper and lower bounds can be obtained. Variational methods have long been used in statistics (Rustagi, 1976). In statistical physics, the mean field method is a particular variational approximation in which the individual variables making up the model are assumed to be completely independent. This idea was applied to solve large undirected Markov networks (Peterson and Anderson, 1987; Parisi, 1988). Saul ef al. (1996) developed the mathematical foundations for applying variational methods to Bayesian networks and obtained

Section 14.8.

Summary 531

BELIEF PROPAGATION

TURBO DECODING

accurate lower-bound approximations for sigmoid networks with the use of mean-field methods. Jaakkola and Jordan (1996) extended the methodology to obtain both lower and upper bounds. Variational approaches are surveyed by Jordan et al. (1999).

A second important family of approximation algorithms is based on Pearl’s polytree message-passing algorithm (1982a). This algorithm can be applied to general networks, as suggested by Pearl (1988). The results might be incorrect, or the algorithm might fail to terminate, but in many cases, the values obtained are close to the true values. Little attention was paid to this so-called belief propagation (or loopy propagation) approach until McEliece et al. (1998) observed that message passing in a multiply-connected Bayesian network was exactly the computation performed by the turbo decoding algorithm (Berrou et al., 1993), which provided a major breakthrough in the design of efficient error-correcting codes. The implication is that loopy propagation is both fast and accurate on the very large and very highly connected networks used for decoding and might therefore be useful more generally. Murphy ef al. (1999) present an empirical study of where it does work. Yedidia et al. (2001) make further connections between loopy propagation and ideas from statistical physics.

The connection between probability and first-order languages was first studied by Carnap (1950). Gaifman (1964) and Scott and Krauss (1966) defined a language in which probabilities could be associated with first-order sentences and for which models were probability measures on possible worlds. Within Al, this idea was developed for propositional logic by Nilsson (1986) and for first-order logic by Halpern (1990). The first extensive investigation of knowledge representation issues in such languages was carried out by Bacchus (1990), and the paper by Wellman et al. (1992) surveys early implementation approaches based on the construction of equivalent propositional Bayesian networks. More recently, researchers have come to understand the importance of complete knowledge bases—that is, knowledge bases that, like Bayesian networks, define a unique joint distribution over all possible worlds. Methods for doing this have been based on probabilistic versions of logic programming (Poole, 1993; Sato and Kameya, 1997) or semantic networks (Koller and Pfeffer, 1998). Relational probability models of the kind described in this chapter are investigated in depth by Pfeffer (2000). Pasula and Russell (2001) examine both issues of relational and identity uncertainty within RPMs and the use of MCMC inference.

As explained in Chapter 13, early probabilistic systems fell out of favor in the early 1970s, leaving a partial vacuum to be filled by alternative methods. Certainty factors were invented for use in the medical expert system MYCIN (Shortliffe, 1976), which was intended both as an engineering solution and as a model of human judgment under uncertainty. The collection Rule-Based Expert Systems (Buchanan and Shortliffe, 1984) provides a complete overview of MYCIN and its descendants (see also Stefik, 1995). David Heckerman (1986) showed that a slightly modified version of certainty factor calculations gives correct probabilistic results in some cases, but results in serious overcounting of evidence in other cases. The PROSPECTOR expert system (Duda ef al., 1979) used a rule-based approach in which the rules were justified by a (seldom tenable) global independence assumption.

Dempster—Shafer theory originates with a paper by Arthur Dempster (1968) proposing a generalization of probability to interval values and a combination rule for using them. Later work by Glenn Shafer (1976) led to the Dempster-Shafer theory’s being viewed as a

532

Chapter 14. Probabilistic Reasoning

POSSIBILITY THEORY

competing approach to probability. Ruspini ef al. (1992) analyze the relationship between the Dempster-Shafer theory and standard probability theory. Shenoy (1989) has proposed a method for decision making with Dempster-Shafer belief functions.

Fuzzy sets were developed by Lotfi Zadeh (1965) in response to the perceived difficulty of providing exact inputs to intelligent systems. The text by Zimmermann (2001) provides a thorough introduction to fuzzy set theory; papers on fuzzy applications are collected in Zimmermann (1999). As we mentioned in the text, fuzzy logic has often been perceived incorrectly as a direct competitor to probability theory, whereas in fact it addresses a different set of issues. Possibility theory (Zadeh, 1978) was introduced to handle uncertainty in fuzzy systems and has much in common with probability. Dubois and Prade (1994) provide a thorough survey of the connections between possibility theory and probability theory.

The resurgence of probability depended mainly on the discovery of Bayesian networks as a method for representing and using conditional independence information. This resurgence did not come without a fight; Peter Cheeseman’s (1985) pugnacious “In Defense of Probability,” and his later article “An Inquiry into Computer Understanding” (Cheeseman, 1988, with commentaries) give something of the flavor of the debate. One of the principal objections of the logicists was that the numerical calculations that probability theory was thought to require were not apparent to introspection and presumed an unrealistic level of precision in our uncertain knowledge. The development of qualitative probabilistic networks (Wellman, 1990a) provided a purely qualitative abstraction of Bayesian networks, using the notion of positive and negative influences between variables. Wellman shows that in many cases such information is sufficient for optimal decision making without the need for the precise specification of probability values. Work by Adnan Darwiche and Matt Ginsberg (1992) extracts the basic properties of conditioning and evidence combination from probability theory and shows that they can also be applied in logical and default reasoning.

The heart disease treatment system described in the chapter is due to Lucas (1996). Other fielded applications of Bayesian networks include the work at Microsoft on inferring computer user goals from their actions (Horvitz ef al., 1998) and on filtering junk email (Sahami et al., 1998), the Electric Power Research Institute’s work on monitoring power generators (Morjaria et al., 1995), and NASA's work on displaying time-critical information at Mission Control in Houston (Horvitz and Barry, 1995).

Some important early papers on uncertain reasoning methods in Al are collected in the anthologies Readings in Uncertain Reasoning (Shafer and Pearl, 1990) and Uncertainty in Artificial Intelligence (Kanal and Lemmer, 1986). The most important single publication in the growth of Bayesian networks was undoubtedly the text Probabilistic Reasoning in Intelligent Systems (Pearl, 1988). Several excellent texts, including Lauritzen (1996), Jensen (2001) and Jordan (2003), contain more recent material. New research on probabilistic reasoning appears both in mainstream Al journals such as Artificial Intelligence and the Journal of Al Research, and in more specialized journals, such as the International Journal of Approximate Reasoning. Many papers on graphical models, which include Bayesian networks, appear in statistical journals. The proceedings of the conferences on Uncertainty in Artificial Intelligence (UAI), Neural Information Processing Systems (NIPS), and Artificial Intelligence and Statistics (AISTATS) are excellent sources for current research.

EXERCISES

Figure 14.18 A Bayesian network describing some features of a car’s electrical system and engine. Each variable is Boolean, and the true value indicates that the corresponding aspect of the vehicle is in working order.

14.1

Consider the network for car diagnosis shown in Figure 14.18.

Extend the network with the Boolean variables Icy Weather and StarterMotor.

eight Boolean nodes, assuming that no conditional independence relations are known to hold among them?

Define this family in general and relate it to the noisy-OR distribution.

Draw a Bayesian network for this domain, given that the gauge is more likely to fail when the core temperature gets too high.

Is your network a polytree?

. Suppose there are just two possible actual and measured temperatures, normal and high; the probability that the gauge gives the correct temperature is x when it is working, but y when it is faulty. Give the conditional probability table associated with G.

534 Chapter 14. Probabilistic Reasoning

d. Suppose the alarm works correctly unless it is faulty, in which case it never sounds. Give the conditional probability table associated with A.

e. Suppose the alarm and gauge are working and the alarm sounds. Calculate an expression for the probability that the temperature of the core is too high, in terms of the various conditional probabilities in the network.

14.3 Two astronomers in different parts of the world make measurements M; and Ms of the number of stars IV in some small region of the sky, using their telescopes. Normally, there is a small possibility e of error by up to one star in each direction. Each telescope can also (with a much smaller probability f) be badly out of focus (events F] and Fb), in which case the scientist will undercount by three or more stars (or, if NV is less than 3, fail to detect any stars at all). Consider the three networks shown in Figure 14.19.

a. Which of these Bayesian networks are correct (but not necessarily efficient) representations of the preceding information?

b. Which is the best network? Explain.

¢. Write out a conditional distribution for P(M;|N), for the case where N € {1,2, 3} and M,€{0,1,2,3,4}. Each entry in the conditional distribution should be expressed as a function of the parameters e and/or f.

d. Suppose M7; =1 and M>=3. What are the possible numbers of stars if we assume no prior constraint on the values of N?

e. What is the most likely number of stars, given these observations? Explain how to compute this, or, if it is not possible to compute, explain what additional information is needed and how it would affect the result.

14.4 Consider the network shown in Figure 14.19(ii), and assume that the two telescopes work identically. N €{1,2,3} and M;, M> € {0,1,2,3,4}, with the symbolic CPTs as described in Exercise 14.3. Using the enumeration algorithm, calculate the probability distribution P(N|M; =2, My =2).

AW

(ii) (iii)

Figure 14.19 Three possible networks for the telescope problem.

Section 14.8. Summary 535

14.5 Consider the family of linear Gaussian networks, as illustrated on page 502.

a. In a two-variable network, let X; be the parent of X5, let Xj have a Gaussian prior, and let P(X5|X) be a linear Gaussian distribution. Show that the joint distribution P(X3, X32) is a multivariate Gaussian, and calculate its covariance matrix.

b. Prove by induction that the joint distribution for a general linear Gaussian network on X1,...,X, is also a multivariate Gaussian.

14.6 The probit distribution defined on page 503 describes the probability distribution for a Boolean child, given a single continuous parent,

a. How might the definition be extended to cover multiple continuous parents?

b. How might it be extended to handle a multivalued child variable? Consider both cases where the child’s values are ordered (as in selecting a gear while driving, depending on speed, slope, desired acceleration, etc.) and cases where they are unordered (as in selecting bus, train, or car to get to work). [Hint: Consider ways to divide the possible values into two sets, to mimic a Boolean variable.]

14.7 This exercise is concerned with the variable elimination algorithm in Figure 14.10, a. Section 14.4 applies variable elimination to the query P(Burglary| John Calls = true, Mary Calls = true) .

Perform the calculations indicated and check that the answer is correct.

b. Count the number of arithmetic operations performed , and compare it with the number performed by the enumeration algorithm.

¢. Suppose a network has the form of a chain: a sequence of Boolean variables X1,..., X, where Parents(X;) ={X;\_1} fori=2,...,n. What is the complexity of computing P(X1|X, = true) using enumeration? Using variable elimination?

d. Prove that the complexity of running variable elimination on a polytree network is linear in the size of the tree for any variable ordering consistent with the network structure.

14.8 Investigate the complexity of exact inference in general Bayesian networks:

a. Prove that any 3-SAT problem can be reduced to exact inference in a Bayesian network constructed to represent the particular problem and hence that exact inference is NPhard. [Hint: Consider a network with one variable for each proposition symbol, one for each clause, and one for the conjunction of clauses.]

b. The problem of counting the number of satisfying assignments for a 3-SAT problem is #P-complete. Show that exact inference is at least as hard as this.

14.9 Consider the problem of generating a random sample from a specified distribution on a single variable. You can assume that a random number generator is available that returns a random number uniformly distributed between 0 and 1.

a. Let X be a discrete variable with P(X =x;) =p; fori € {1,...,k}. The cumulative SHMULITIVE distribution of X gives the probability that X € {z1,...,z;} for each possible j. Ex

536 Chapter 14. Probabilistic Reasoning

plain how to calculate the cumulative distribution in O(k) time and how to generate a single sample of X from it. Can the latter be done in less than O(k) time?

b. Now suppose we want to generate N samples of X, where N >> k. Explain how to do this with an expected runtime per sample that is constant (i.e., independent of k).

¢. Now consider a continuous-valued variable with a parametrized distribution (e.g., Gaussian). How can samples be generated from such a distribution?

d. Suppose you want to query a continuous-valued variable and you are using a sampling algorithm such as LIKELTHOODWEIGHTING to do the inference. How would you have to modify the query-answering process?

14.10 The Markov blanket of a variable is defined on page 499. a. Prove that a variable is independent of all other variables in the network, given its Markov blanket. b. Derive Equation (14.11).

14.11 Consider the query P(Rain| Sprinkler = true, WetGrass = true) in Figure 14.11(a) and how MCMC can answer it. a. How many states does the Markov chain have? Calculate the transition matrix Q containing ¢(y — y’) for all y, y'. What does Q2?, the square of the transition matrix, represent? What about Q" as n — oo?

Explain how to do probabilistic inference in Bayesian networks, assuming that Q" is available. Is this a practical way to do inference?

ce Bo =

14.12 Three soccer teams A, B, and C, play each other once. Each match is between two teams, and can be won, drawn, or lost. Each team has a fixed, unknown degree of quality— an integer ranging from 0 to 3—and the outcome of a match depends probabilistically on the difference in quality between the two teams.

a. Construct a relational probability model to describe this domain, and suggest numerical values for all the necessary probability distributions.

b. Construct the equivalent Bayesian network.

c¢. Suppose that in the first two matches A beats B and draws with C. Using an exact inference algorithm of your choice, compute the posterior distribution for the outcome of the third match.

d. Suppose there are n teams in the league and we have the results for all but the last match. How does the complexity of predicting the last game vary with n?

e. Investigate the application of MCMC to this problem. How quickly does it converge in practice and how well does it scale?

PROBABILISTIC REASONING OVER TIME

In which we try to interpret the present, understand the past, and perhaps predict the future, even when very little is crystal clear.

Agents in uncertain environments must be able to keep track of the current state of the environment, just as logical agents must. The task is made more difficult by partial and noisy percepts and uncertainty about how the environment changes over time. At best, the agent will be able to obtain only a probabilistic assessment of the current situation. This chapter describes the representations and inference algorithms that make that assessment possible, building on the ideas introduced in Chapter 14.

The basic approach is described in Section 15.1: a changing world is modeled using a random variable for each aspect of the world state at each point in time. The relations among these variables describe how the state evolves. Section 15.2 defines the basic inference tasks and describes the general structure of inference algorithms for temporal models. Then we describe three specific kinds of models: hidden Markov models, Kalman filters, and dynamic Bayesian networks (which include hidden Markov models and Kalman filters as special cases). Finally, Section 15.6 explains how temporal probability models form the core of modern speech recognition systems. Learning plays a central role in the construction of all these models, but a detailed investigation of learning algorithms is left until Part VI.

15.1 TIME AND UNCERTAINTY

We have developed our techniques for probabilistic reasoning in the context of static worlds, in which each random variable has a single fixed value. For example, when repairing a car, we assume that whatever is broken remains broken during the process of diagnosis; our job is to infer the state of the car from observed evidence, which also remains fixed.

Now consider a slightly different problem: treating a diabetic patient. As in the case of car repair, we have evidence such as recent insulin doses, food intake, blood sugar measurements, and other physical signs. The task is to assess the current state of the patient, including the actual blood sugar level and insulin level. Given this information, the doctor (or patient) makes a decision about the patient’s food intake and insulin dose. Unlike the case

537

538

Chapter 15. Probabilistic Reasoning over Time

TIME SLICE

of car repair, here the dynamic aspects of the problem are essential. Blood sugar levels and measurements thereof can change rapidly over time, depending on one’s recent food intake and insulin doses, one’s metabolic activity, the time of day, and so on. To assess the current state from the history of evidence and to predict the outcomes of treatment actions, we must model these changes.

The same considerations arise in many other contexts, ranging from tracking the economic activity of a nation, given approximate and partial statistics, to understanding a sequence of spoken words, given noisy and ambiguous acoustic measurements. How can dynamic situations like these be modeled?

States and observations

The basic approach we will adopt is similar to the idea underlying situation calculus, as described in Chapter 10: the process of change can be viewed as a series of snapshots, each of which describes the state of the world at a particular time. Each snapshot, or time slice, contains a set of random variables, some of which are observable and some of which are not. For simplicity, we will assume that the same subset of variables is observable in each slice (although this is not strictly necessary in anything that follows). We will use X; to denote the set of unobservable state variables at time ¢ and E; to denote the set of observable evidence variables. The observation at time ¢ is E; = ¢; for some set of values e;.

Consider the following oversimplified example: Suppose you are the security guard at some secret underground installation. You want to know whether it’s raining today, but your only access to the outside world occurs each morning when you see the director coming in with, or without, an umbrella. For each day ¢, the set E; thus contains a single evidence variable U; (whether the umbrella appears), and the set X; contains a single state variable R; (whether it is raining). Other problems can involve larger sets of variables. In the diabetes example, we might have evidence variables such as MeasuredBloodSugar,; and PulseRatey, and state variables such as BloodSugar; and StomachContents;.!

The interval between time slices also depends on the problem. For diabetes monitoring, a suitable interval might be an hour rather than a day. In this chapter, we will generally assume a fixed, finite interval; this means that times can be labeled by integers. We will assume that the state sequence starts at { = 0; for various uninteresting reasons, we will assume that evidence starts arriving at t = 1 rather than ¢ = 0. Hence, our umbrella world is represented by state variables Ry, Ri, Rs, ... and evidence variables Uy, Us, .... We will use the notation a:b to denote the sequence of integers from a to b (inclusive), and the notation X,., to denote the corresponding set of variables from X, to X;. For example, Uy.3 corresponds to the variables U7, Us, Us.

Stationary processes and the Markov assumption

With the set of state and evidence variables for a given problem decided on, the next step is to specify the dependencies among the variables. We could follow the procedure laid down

I Notice that BloodSugar: and MeasuredBloodSugar: are not the same variable; this is how we deal with noisy measurements of actual quantities.

Section 15.1.

Time and Uncertainty 539

STATIONARY PROCESS

MARKOV ASSUMPTION

MARKOV PROCESSES

FIRST-ORDER MARKOV PROCESS

SENSOR MODEL

in Chapter 14, placing the variables in some order and asking questions about conditional independence of predecessors, given some set of parents. One obvious choice is to order the variables in their natural temporal order, since cause usually precedes effect and we prefer to add the variables in causal order.

We would quickly run into an obstacle, however: the set of variables is unbounded, because it includes the state and evidence variables for every time slice. This actually creates two problems: first, we might have to specify an unbounded number of conditional probability tables, one for each variable in each slice; second, each one might involve an unbounded number of parents.

The first problem is solved by assuming that changes in the world state are caused by a stationary process—that is, a process of change that is governed by laws that do not themselves change over time. (Don’t confuse stationary with static: in a static process, the state itself does not change.) In the umbrella world, then, the conditional probability that the umbrella appears, P(U;|Parents(U,)), is the same for all ¢. Given the assumption of stationarity, therefore, we need specify conditional distributions only for the variables within a “representative” time slice.

The second problem, that of handling the potentially infinite number of parents, is solved by making what is called a Markov assumption—that is, that the current state depends on only a finite history of previous states. Processes satisfying this assumption were first studied in depth by the Russian statistician Andrei Markov and are called Markov processes or Markov chains. They come in various flavors; the simplest is the first-order Markov process, in which the current state depends only on the previous state and not on any earlier states. In other words, a state is the information you need to make the future independent of the past given the state. Using our notation, the corresponding conditional independence assertion states that, for all £,

P(X; Xo:t—1) = P(X¢[X¢—1) . (15.1)

Hence, in a first-order Markov process, the laws describing how the state evolves over time are contained entirely within the conditional distribution P(X;|X;\_1), which we call the transition model for first-order processes.” The transition model for a second-order Markov process is the conditional distribution P(X¢|X;—\_2,X;\_1). Figure 15.1 shows the Bayesian network structures corresponding to first-order and second-order Markov processes.

In addition to restricting the parents of the state variables X,, we must restrict the parents of the evidence variables E;. Typically, we will assume that the evidence variables at time ¢ depend only on the current state:

P(E:|Xo, Eo. 1) = P(E¢[X,) . (15.2)

The conditional distribution P(E;|X;) is called the sensor model (or sometimes the observation model), because it describes how the “sensors”—that is, the evidence variables—are affected by the actual state of the world. Notice the direction of the dependence: the “arrow” goes from state to sensor values because the state of the world causes the sensors to take on

2 The transition model is the probabilistic analog of the Boolean update circuits in Chapter 7 and the successorstate axioms in Chapter 10.

540

Chapter 15. Probabilistic Reasoning over Time

RANDOM WALK

@ —~E ED) OEE) 0) FF FFE)

Figure 15.1 (a) Bayesian network structure corresponding to a first-order Markov process with state defined by the variables X;. (b) A second-order Markov process.

Ri | PR) t 0.7 fr 0.3

Umbrella,

Figure 15.2 Bayesian network structure and conditional distributions describing the umbrella world. The transition model is P(Rain:|Rain;—1) and the sensor model is P(Umbrella,| Rainy).

Umbrella,

particular values. In the umbrella world, for example, the rain causes the umbrella to appear. (The inference process, of course, goes in the other direction; the distinction between the direction of modeled dependencies and the direction of inference is one of the principal advantages of Bayesian networks.)

In addition to the transition model and sensor model, we need to specify a prior probability P(Xg) over the states at time 0. These three distributions, combined with the conditional independence assertions in Equations (15.1) and (15.2), give us a specification of the complete joint distribution over all the variables. For any finite ¢, we have

t

P(X0,X1,..., Xt, Eq, ... Ey) = P(Xo) [] P(X:[Xio1)P(Ei|X:)

i=1 The independence assumptions correspond to a very simple structure for the Bayesian network describing the whole system. Figure 15.2 shows the network structure for the umbrella example, including the conditional distributions for the transition and sensor models.

The structure in the figure assumes a first-order Markov process, because the probability of rain is assumed to depend only on whether it rained the previous day. Whether such an assumption is reasonable depends on the domain itself. The first-order Markov assumption says that the state variables contain all the information needed to characterize the probability distribution for the next time slice. Sometimes the assumption is exactly true—for example, if a particle is executing a random walk along the x-axis, changing its position by £1 at each time step, then using the z-coordinate as the state gives a first-order Markov process.

Section 15.2. Inference in Temporal Models 541

Sometimes the assumption is only approximate, as in the case of predicting rain only on the basis of whether it rained the previous day. There are two possible fixes if the approximation proves too inaccurate;

1. Increasing the order of the Markov process model. For example, we could make a second-order model by adding Rain; \_» as a parent of Rain, which might give slightly more accurate predictions (for example, in Palo Alto it very rarely rains more than two days in a row).

2. Increasing the set of state variables. For example, we could add Season; to allow us to incorporate historical records of rainy seasons, or we could add Temperature, Humidity; and Pressure; to allow us to use a physical model of rainy conditions.

Exercise 15.1 asks you to show that the first solution—increasing the order—can always be reformulated as an increase in the set of state variables, keeping the order fixed. Notice that adding state variables might improve the system’s predictive power but also increases the prediction requirements: we now have to predict the new variables as well. Thus, we are looking for a “self-sufficient” set of variables, which really means that we have to understand the “physics” of the process being modeled. The requirement for accurate modeling of the process is obviously lessened if we can add new sensors (e.g., measurements of temperature and pressure) that provide information directly about the new state variables.

Consider, for example, the problem of tracking a robot wandering randomly on the X-Y plane. One might propose that the position and velocity are a sufficient set of state variables: one can simply use Newton’s laws to calculate the new position, and the velocity may change unpredictably. If the robot is battery-powered, however, then battery exhaustion would tend to have a systematic effect on the change in velocity. Because this in turn depends on how much power was used by all previous maneuvers, the Markov property is violated. We can restore the Markov property by including the charge level Battery; as one of the state variables that make up X;. This helps in predicting the motion of the robot, but in turn requires a model for predicting Battery; from Battery; and the velocity. In some cases, that can be done reliably; accuracy would be improved by adding a new sensor for the battery level

15.2 INFERENCE IN TEMPORAL MODELS

Having set up the structure of a generic temporal model, we can formulate the basic inference tasks that must be solved:

FILTERING $ Filtering or monitoring: This is the task of computing the belief state—the posteMONITORING rior distribution over the current state, given all evidence to date. That is, we wish to BELIEF STATE compute P(X; |eq.;), assuming that evidence arrives in a continuous stream beginning

at t = 1. In the umbrella example, this would mean computing the probability of rain today, given all the observations of the umbrella carrier made so far. Filtering is what a rational agent needs to do in order to keep track of the current state so that rational decisions can be made. (See Chapter 17.) It turns out that an almost identical calculation provides the likelihood of the evidence sequence, P(e;.;).

542 Chapter 15. Probabilistic Reasoning over Time

PREDICTION <> Prediction: This is the task of computing the posterior distribution over the future state, given all evidence to date. That is, we wish to compute P(X; x|e1.;) for some k > 0. In the umbrella example, this might mean computing the probability of rain three days from now, given all the observations of the umbrella-carrier made so far. Prediction is useful for evaluating possible courses of action.

SMOOTHING ¢$ Smoothing, or hindsight: This is the task of computing the posterior distribution over a

HINDSIGHT past state, given all evidence up to the present. That is, we wish to compute P(X; |e.) for some k such that 0 < k < ¢. In the umbrella example, it might mean computing the probability that it rained last Wednesday, given all the observations of the umbrella carrier made up to today. Hindsight provides a better estimate of the state than was available at the time, because it incorporates more evidence.

{> Most likely explanation: Given a sequence of observations, we might wish to find the sequence of states that is most likely to have generated those observations. That is, we wish to compute argmax, P(Xi.t|e1.t). For example, if the umbrella appears on each of the first three days and is absent on the fourth, then the most likely explanation is that it rained on the first three days and did not rain on the fourth. Algorithms for this task are useful in many applications, including speech recognition—where the aim is to find the most likely sequence of words, given a series of sounds—and the reconstruction of bit strings transmitted over a noisy channel.

In addition to these tasks, methods are needed for learning the transition and sensor models from observations. Just as with static Bayesian networks, dynamic Bayes net learning can be done as a by-product of inference. Inference provides an estimate of what transitions actually occurred and of what states generated the sensor readings, and these estimates can be used to update the models. The updated models provide new estimates, and the process iterates to convergence. The overall process is an instance of the expectation-maximization or EM algorithm. (See Section 20.3.) One point to note is that learning requires the full smoothing inference, rather than filtering, because it provides better estimates of the states of the process. Learning with filtering can fail to converge correctly; consider, for example, the problem of learning to solve murders: hindsight is always required to infer what happened at the murder scene from the observable variables.

Algorithms for the four inference tasks listed in the preceding paragraph can be described first at a generic level, independently of the particular kind of model employed. Improvements specific to each model will be described in the corresponding sections.

Filtering and prediction

Let us begin with filtering. We will show that this can be done in a simple online fashion: given the result of filtering up to time ¢, one can easily compute the result for ¢ + 1 from the new evidence €;,1. That is,

P(Xir1le141) = ferr1, P(X¢ler))

RECURS for some function f. This process is often called recursive estimation. We can view the calculation as actually being composed of two parts: first, the current state distribution is

Section 15.2. Inference in Temporal Models 543

projected forward from ¢ to ¢ + 1; then it is updated using the new evidence e.;. This two-part process emerges quite simply: P(Xi+1le1er1) = P(Xer1]€1:, e401) (dividing up the evidence) = aP(e, 1|Xey1,€14)P(Xei1]€1) (using Bayes’ rule) = aPerr1|Xe1)P(Xiia]ers) (by the Markov property of evidence). Here and throughout this chapter, « is a normalizing constant used to make probabilities sum up to 1. The second term, P(X; ,|e1.+) represents a one-step prediction of the next state, and the first term updates this with the new evidence; notice that P(e; 1|X;4+1) is obtainable directly from the sensor model. Now we obtain the one-step prediction for the next state by conditioning on the current state X;: P(Xir1ler1) = aPle1|Xer1) > P(X¢11]Xe, 1:4) P(X¢€1:¢)

Xt = aP(e1|X¢41) > P(Xp1|xt) P(xe[er) (using the Markov property). (15.3) Xt Within the summation, the first factor is simply the transition model and the second is the current state distribution. Hence, we have the desired recursive formulation. We can think of the filtered estimate P(X;|e1.t) as a “message” fi; that is propagated forward along the sequence, modified by each transition and updated by each new observation. The process is

£1.11 = « FORWARD(f1,€;11)

where FORWARD implements the update described in Equation (15.3).

When all the state variables are discrete, the time for each update is constant (i.e., independent of t), and the space required is also constant. (The constants depend, of course, on the size of the state space and the specific type of the temporal model in question.) The time and space requirements for updating must be constant if an agent with limited memory is to keep track of the current state distribution over an unbounded sequence of observations.

Let us illustrate the filtering process for two steps in the basic umbrella example. (See Figure 15.2.) We assume that our security guard has some prior belief about whether it rained on day 0, just before the observation sequence begins. Let's suppose this is P(Rp) = (0.5,0.5). Now we process the two observations as follows:

e On day 1, the umbrella appears, so Uj = true. The prediction from t{=0tot=11is P(R1) = Y P(Ri|ro)P(ro) To

[]

544

Chapter 15. Probabilistic Reasoning over Time

MIXING TIME

and updating it with the evidence for t = 2 gives P(Rz|u1,u2) = aP(ug|R2)P(Ra|u1) = «(0.9,0.2)(0.627,0.373) = «(0.565,0.075) ~ (0.883,0.117) .

Intuitively, the probability of rain increases from day 1 to day 2 because rain persists. Exercise 15.2(a) asks you to investigate this tendency further.

The task of prediction can be seen simply as filtering without the addition of new evidence. In fact, the filtering process already incorporates a one-step prediction, and it is easy to derive the following recursive computation for predicting the state at ¢ + k + 1 from a prediction for ¢ + k:

P(Xpikr1lers) = > P(Xepni1 Kern) P(Xepnlers) (15.4)

Xt+k Naturally, this computation involves only the transition model and not the sensor model.

It is interesting to consider what happens as we try to predict further and further into the future. As Exercise 15.2(b) shows, the predicted distribution for rain converges to a fixed point (0.5,0.5), after which it remains constant for all time. This is the stationary distribution of the Markov process defined by the transition model. (See also page 517.) A great deal is known about the properties of such distributions and about the mixing time— roughly, the time taken to reach the fixed point. In practical terms, this dooms to failure any attempt to predict the actual state for a number of steps that is more than a small fraction of the mixing time. The more uncertainty there is in the transition model, the shorter will be the mixing time and the more the future is obscured.

In addition to filtering and prediction, we can use a forward recursion to compute the likelihood of the evidence sequence, P(e;.;). This is a useful quantity if we want to compare different temporal models that might have produced the same evidence sequence; for example, in Section 15.6, we compare different words that might have produced the same sound sequence. For this recursion, we use a likelihood message £1.; = P(X, eq). It is a simple exercise to show that

L111 = FORWARD(/;.¢, etl) . Having computed £;.;, we obtain the actual likelihood by summing out Xj: Lit = P(e) = > lra(xe) . (15.5) Xt

Smoothing

As we said earlier, smoothing is the process of computing the distribution over past states given evidence up to the present; that is, P(Xy|e1.) for 1 < k < ¢. (See Figure 15.3.) This is done most conveniently in two parts—the evidence up to k and the evidence from k + 1 to ¢, P(Xyler) = P(Xglerx, ext) = aP(Xylerr)P(exi1.4|Xk,€1.x) (using Bayes’ rule) = aP(Xy|er.r)P(er1../Xx) (using conditional independence) = afipbriie, (15.6)

[]

Section 15.2. Inference in Temporal Models 545

E& ®

Figure 15.3 Smoothing computes P(Xy|e;.;), the posterior distribution of the state at some past time k given a complete sequence of observations from 1 to ¢.

where we have defined a “backward” message by 1.. = P(e; 1.¢|Xx), analogous to the forward message f1.,. The forward message f;.; can be computed by filtering forward from 1 to k, as given by Equation (15.3). It turns out that the backward message by, 1; can be computed by a recursive process that runs backwards from t: P(e; 1.4/Xx) = > Peg 1.¢| Xx, Xp) P(xp 1 Xk) (conditioning on X511) XE+1 = > Peg 1:|Xk+1)P(xk1+1/X%) (by conditional independence) XEp+1

= > Pleri1, ep: Xnp1)P(Xpp1|Xx)

Xk+1 = > Plepi1/xp+1) PlentouXni1)P(Xnp1[Xk) (15.7) XE+1 where the last step follows by the conditional independence of ej; and ey 2.¢, given Xj 1. Of the three factors in this summation, the first and third are obtained directly from the model, and the second is the “recursive call.” Using the message notation, we have

b+ 1:6 = BACKWARD(bgy2:¢, €441:t) where BACKWARD implements the update described in Equation (15.7). As with the forward recursion, the time and space needed for each update are constant and thus independent of ¢.

We can now see that the two terms in Equation (15.6) can both be computed by recursions through time, one running forward from 1 to k and using the filtering equation (15.3) and the other running backward from ¢ to k +1 and using Equation (15.7). Note that the backward phase 1s initialized with by 1. = P(e; 1.¢|X¢) = 1, where 1 is a vector of ones. (Because e; 1: 1S an empty sequence, the probability of observing itis 1.)

Let us now apply this algorithm to the umbrella example, computing the smoothed estimate for the probability of rain at £ = 1, given the umbrella observations on days 1 and 2. From Equation (15.6), this is given by

P(R; lug, u2) = aP(R1|ur) P(uz|R1) . (15.8) The first term we already know to be (.818,.182), from the forward filtering process de

scribed earlier. The second term can be computed by applying the backward recursion in Equation (15.7):

P(us|Ry) = > Pug|ra)P(|r2)P(rz|Ry)

= (0.9% 1x (0.7,0.3)) + (0.2 x 1 x (0.3,0.7)) = (0.69,0.41) .

546

Chapter 15. Probabilistic Reasoning over Time

FORWARDBACKWARD ALGORITHM

Plugging this into Equation (15.8), we find that the smoothed estimate for rain on day 1 is P(R;|u1, uz) = « (0.818,0.182) x (0.69,0.41) ~ (0.883,0.117) .

Thus, the smoothed estimate is higher than the filtered estimate (0.818) in this case. This is because the umbrella on day 2 makes it more likely to have rained on day 2; in turn, because rain tends to persist, that makes it more likely to have rained on day 1.

Both the forward and backward recursions take a constant amount of time per step; hence, the time complexity of smoothing with respect to evidence ej; is O(t). This is the complexity for smoothing at a particular time step k. If we want to smooth the whole sequence, one obvious method is simply to run the whole smoothing process once for each time step to be smoothed. This results in a time complexity of O(¢?). A better approach uses a very simple application of dynamic programming to reduce the complexity to O(t). A clue appears in the preceding analysis of the umbrella example, where we were able to reuse the results of the forward filtering phase. The key to the linear-time algorithm is to record the results of forward filtering over the whole sequence. Then we run the backward recursion from ¢ down to 1, computing the smoothed estimate at each step k from the computed backward message by, 1+ and the stored forward message f1.;. The algorithm, aptly called the forward-backward algorithm, is shown in Figure 15.4.

The alert reader will have spotted that the Bayesian network structure shown in Figure 15.3 is a polytree in the terminology of Chapter 14. This means that a straightforward application of the clustering algorithm also yields a linear-time algorithm that computes smoothed estimates for the entire sequence. It is now understood that the forward-backward algorithm is in fact a special case of the polytree propagation algorithm used with clustering methods (although the two were developed independently).

function FORWARD-BACKWARD(ev, prior) returns a vector of probability distributions

inputs: ev, a vector of evidence values for steps 1,...,1¢ prior, the prior distribution on the initial state, P(X) local variables: fv, a vector of forward messages for steps 0, ..., 1 b, a representation of the backward message, initially all 1s sv, a vector of smoothed estimates for steps 1,...,t

tv{0] — prior fori= 1tot do fv[i] — FORWARD (fv[i — 1], ev]i]) for i = ¢ downto 1 do sv[i) «— NORMALIZE(fv[z] x b) b «— BACKWARD(b, ev]i]) return sv

Figure 15.4 The forward-backward algorithm for computing posterior probabilities of a sequence of states given a sequence of observations. The FORWARD and BACKWARD operators are defined by Equations (15.3) and (15.7), respectively.

Section 15.2. Inference in Temporal Models 547

The forward-backward algorithm forms the backbone of the computational methods employed in many applications that deal with sequences of noisy observations, ranging from speech recognition to radar tracking of aircraft. As described, it has two practical drawbacks. The first is that its space complexity can be too high for applications where the state space is large and the sequences are long. It uses O(|f|t) space where |f] is the size of the representation of the forward message. The space requirement can be reduced to O(|f|logt) with a concomitant increase in the time complexity by a factor of log ¢, as shown in Exercise 15.3. In some cases (see Section 15.3), a constant-space algorithm can be used with no time penalty.

The second drawback of the basic algorithm is that it needs to be modified to work in an online setting where smoothed estimates must be computed for earlier time slices as new observations are continuously added to the end of the sequence. The most common

FEDLAG | requirement is for fixed-lag smoothing, which requires computing the smoothed estimate P(X; \_g|e1.) for fixed d. That is, smoothing is done for the time slice d steps behind the current time ¢; as ¢ increases, the smoothing has to keep up. Obviously, we can run the forward-backward algorithm over the d-step “window” as each new observation is added, but this seems inefficient. In Section 15.3, we will see that fixed-lag smoothing can, in some cases, be done in constant time per update, independently of the lag d.

Finding the most likely sequence

Suppose that [true, true, false, true, true] is the umbrella sequence for the security guard’s first five days on the job. What is the weather sequence most likely to explain this? Does the absence of the umbrella on day 3 mean that it wasn’t raining, or did the director forget to bring it? If it didn’t rain on day 3, perhaps (because weather tends to persist) it didn’t rain on day 4 either, but the director brought the umbrella just in case. In all, there are 2° possible weather sequences we could pick. Is there a way to find the most likely one, short of enumerating all of them?

One approach we could try is the following linear-time procedure: use the smoothing algorithm to find the posterior distribution for the weather at each time step; then construct the sequence, using at each step the weather most likely according to the posterior. Such an approach should set off alarm bells in the reader’s head, because the posteriors computed by smoothing are distributions over single time steps, whereas to find the most likely sequence we must consider joint probabilities over all the time steps. The results can in fact be quite different. (See Exercise 15.4.)

There is a linear-time algorithm for finding the most likely sequence, but it requires a little more thought. It relies on the same Markov property that yielded efficient algorithms for filtering and smoothing. The easiest way to think about the problem is to view each sequence as a path through a graph whose nodes are the possible states at each time step. Such a graph is shown for the umbrella world in Figure 15.5(a). Now consider the task of finding the most likely path through this graph, where the likelihood of any path is the product of the transition probabilities along the path and the probabilities of the given observations at each state. Let’s focus in particular on paths that reach the state Rains = true. Because of the Markov property, it follows that the most likely path to the state Ran; = true consists of

548 Chapter 15. Probabilistic Reasoning over Time Rain, Rain, Rain, Rainy Rains true true K | true true true false Jalse false false false Umbrella, true true false true true 8182 S155 0361 0334F1.0210

[]

mg mi, m3 my mi.s

Figure 15.5 (a) Possible state sequences for Rain can be viewed as paths through a graph of the possible states at each time step. (States are shown as rectangles to avoid confusion with nodes in a Bayes net.) (b) Operation of the Viterbi algorithm for the umbrella observation sequence [true, true, false, true, true]. For each {, we have shown the values of the message my .¢, which gives the probability of the best sequence reaching each state at time ¢. Also, for each state, the bold arrow leading into it indicates its best predecessor as measured by the product of the preceding sequence probability and the transition probability. Following the bold arrows back from the most likely state in m;.5 gives the most likely sequence.

the most likely path to some state at time 4 followed by a transition to Rains = true; and the state at time 4 that will become part of the path to Rains = true is whichever maximizes the likelihood of that path. In other words, there is a recursive relationship between most likely paths to each state Xi 1 and most likely paths to each state x;. We can write this relationship as an equation connecting the probabilities of the paths:

max P(x1,...,X¢, Xeq1ler41) = aP(e1|X¢41) max (PXerfx) Jnax P(xq,... pan) . (15.9)

Equation (15.9) is identical to the filtering equation (15.3) except that 1. The forward message f;.. =P(X;|e;.+) is replaced by the message

mp; = max P(X1,...,X1,X¢le1), XX]... Xt—1

that is, the probabilities of the most likely path to each state x;; and

2. the summation over Xx; in Equation (15.3) is replaced by the maximization over X; in Equation (15.9).

Thus, the algorithm for computing the most likely sequence is similar to filtering: it runs forward along the sequence, computing the m message at each time step, using Equation (15.9). The progress of this computation is shown in Figure 15.5(b). At the end, it will have the probability for the most likely sequence reaching each of the final states. One can thus easily select the most likely sequence overall (the state outlined in bold). In order to identify the actual sequence, as opposed to just computing its probability, the algorithm will also need

Section 15.3.

Hidden Markov Models 549

VITERBI ALGORITHM

to keep pointers from each state back to the best state that leads to it (shown in bold); the sequence is identified by following the pointers back from the best final state.

The algorithm we have just described is called the Viterbi algorithm, after its inventor. Like the filtering algorithm, its complexity is linear in ¢, the length of the sequence. Unlike filtering, however, its space requirement is also linear in ¢. This is because the Viterbi

HIDDEN MARKOV MODEL

The preceding section developed algorithms for temporal probabilistic reasoning using a general framework that was independent of the specific form of the transition and sensor models. In this and the next two sections, we discuss more concrete models and applications that illustrate the power of the basic algorithms and in some cases allow further improvements.

We begin with the hidden Markov model, or HMM. An HMM is a temporal probabilistic model in which the state of the process is described by a single discrete random variable. The possible values of the variable are the possible states of the world. The umbrella example described in the preceding section is therefore an HMM, since it has just one state variable: Rasn;. Additional state variables can be added to a temporal model while staying within the HMM framework, but only by combining all the state variables into a single “megavariable” whose values are all possible tuples of values of the individual state variables. We will see that the restricted structure of HMMs allows for a very simple and elegant matrix implementation of all the basic algorithms.> Section 15.6 shows how HMMs are used for speech recognition.

Simplified matrix algorithms

With a single, discrete state variable X;, we can give concrete form to the representations of the transition model, the sensor model, and the forward and backward messages. Let the state variable X; have values denoted by integers 1, ...,.S, where S is the number of possible states. The transition model P(X;|X;\_1) becomes an S x S matrix T, where

T;; = P(X =j|X—1 =1) .

That is, T;; is the probability of a transition from state ¢ to state j. For example, the transition matrix for the umbrella world is

0.7 0.3 T = P(X, X,; 1) = (oF 2) :

We also put the sensor model in matrix form. In this case, because the value of the evidence variable EF is known to be, say, e;, we need use only that part of the model specifying the probability that e; appears. For each time step ¢, we construct a diagonal matrix OQ; whose

3 The reader unfamiliar with basic operations on vectors and matrices might wish to consult Appendix A before proceeding with this section.

550

~ Chapter 15. Probabilistic Reasoning over Time

diagonal entries are given by the values P(e: X;=1) and whose other entries are 0. For example, on day 1 in the umbrella world, Uj = true, so, from Figure 15.2, we have

0.9 0 or( 0 ’)

Now, if we use column vectors to represent the forward and backward messages, the computations become simple matrix—vector operations. The forward equation (15.3) becomes

frp = a 0p T fry (15.10) and the backward equation (15.7) becomes byi1:t = TOR bro: (15.11)

From these equations, we can see that the time complexity of the forward-backward algorithm (Figure 15.4) applied to a sequence of length ¢ is O(S?t), because each step requires multiplying an S-element vector by an S x .S matrix. The space requirement is O(St), because the forward pass stores ¢ vectors of size S.

Besides providing an elegant description of the filtering and smoothing algorithms for HMMs, the matrix formulation reveals opportunities for improved algorithms. The first is a simple variation on the forward-backward algorithm that allows smoothing to be carried out in constant space, independently of the length of the sequence. The idea is that smoothing for any particular time slice k requires the simultaneous presence of both the forward and backward messages, f;.;, and by, 1, according to Equation (15.6). The forward-backward algorithm achieves this by storing the fs computed on the forward pass so that they are available during the backward pass. Another way to achieve this is with a single pass that propagates both f and b in the same direction. For example, the “forward” message f can be propagated backwards if we manipulate Equation (15.10) to work in the other direction:

fre = o/ (TT) 10 frien

The modified smoothing algorithm works by first running the standard forward pass to compute f;; (forgetting all the intermediate results) and then running the backward pass for both b and f together, using them to compute the smoothed estimate at each step. Since only one copy of each message is needed, the storage requirements are constant (i.e. independent of ¢, the length of the sequence). There is one significant restriction on this algorithm: it requires that the transition matrix be invertible and that the sensor model have no zeroes—that is, every observation is possible in every state.

A second area in which the matrix formulation reveals an improvement is in online smoothing with a fixed lag. The fact that smoothing can be done in constant space suggests that there should exist an efficient recursive algorithm for online smoothing—that is, an algorithm whose time complexity is independent of the length of the lag. Let us suppose that the lag is d; that is, we are smoothing at time slice ¢ — d, where the current time is ¢{. By Equation (15.6), we need to compute

afi abide for slice t — d. Then, when a new observation arrives, we need to compute

afi gribi\_gioni

Section 15.4.

Kalman Filters 551

for slice t —d + 1. How can this be done incrementally? First, we can compute f;.;\_ 4,1 from f1.:\_q, using the standard filtering process, Equation (15.3).

Computing the backward message incrementally is more tricky, because there is no simple relationship between the old backward message b;\_g4, 1.; and the new backward message b;\_4.2:¢11. Instead, we will examine the relationship between the old backward message b;\_q.1.: and the backward message at the front of the sequence, by; 1::. To do this, we apply Equation (15.11) d times to get

t bi\_gi1: = II to) bei: =Bigy1:l, (15.12) i=t—-d+1 where the matrix B,\_4,1:; is the product of the sequence of T and O matrices. B can be thought of as a “transformation operator” that transforms a later backward message into an earlier one. A similar equation holds for the new backward messages after the next observation arrives: t+1

b\_ai2:11 = I1 10) byio:er1 =Bi\_gioaial. (15.13) i=t—d+2

Examining the product expressions in Equations (15.12) and (15.13), we see that they have a

simple relationship: to get the second product, “divide” the first product by the first element

TO;\_4, 1, and multiply by the new last element TO, 1. In matrix language, then, there is a

simple relationship between the old and new B matrices:

Bi\_gi241 = 0; 4 1 T "By gy1:4TO 1 (15.14) This equation provides an incremental update for the B matrix, which in turn (through Equa

tion (15.13)) allows us to compute the new backward message b;\_4.2.+,1. The complete algorithm, which requires storing and updating f and B, is shown in Figure 15.6.

KALMAN FILTERING

Imagine watching a small bird flying through dense jungle foliage at dusk: you glimpse brief, intermittent flashes of motion; you try hard to guess where the bird is and where it will appear next so that you don’t lose it. Or imagine that you are a World War II radar operator peering at a faint, wandering blip that appears once every 10 seconds on the screen. Or, going back further still, imagine you are Kepler trying to reconstruct the motions of the planets from a collection of highly inaccurate angular observations taken at irregular and imprecisely measured intervals. In all these cases, you are trying to estimate the state (position and velocity, for example) of a physical system from noisy observations over time. The problem can be formulated as inference in a temporal probability model, where the transition model describes the physics of motion and the sensor model describes the measurement process. This section examines the special representations and inference algorithms that have been developed to solve these sorts of problems; the method we will cover is called Kalman filtering, after its inventor, Rudolf E. Kalman.

552 Chapter 15. Probabilistic Reasoning over Time function FIXED-LAG-SMOOTHING(e;, hmm, d) returns a distribution over X;\_4 inputs: e;, the current evidence for time step ¢ hmm, a hidden Markov model with S x S transition matrix T d, the length of the lag for smoothing static: ¢, the current time, initially 1 f, a probability distribution, the forward message P(X¢|e1.+), initially PRIOR [hmm] B, the d-step backward transformation matrix, initially the identity matrix e+—q:t, double-ended list of evidence from t — d to ¢, initially empty local variables: O;\_,, O;, diagonal matrices containing the sensor model information add e; to the end of e;\_ 4.4 0; — diagonal matrix containing P(e;|X}) if ¢ > d then f — FORWARD(f, e;) remove e;\_,—1 from the beginning of e; 4.4 O;\_; — diagonal matrix containing P(e; 4] X;\_4) B—O0;, T"'BTO, else B — BTO; t—t+1 if t > d then return NORMALIZE(f x Bl) else return null Figure 15.6 An algorithm for smoothing with a fixed time lag of d steps, implemented as an online algorithm that outputs the new smoothed estimate given the observation for a new time step.

Clearly, we will need several continuous variables to specify the state of the system. For example, the bird’s flight might be specified by position (X,Y, Z) and velocity (xX JY. Z ) at each point in time. We will also need suitable conditional densities to represent the transition and sensor models; as in Chapter 14, we will use linear Gaussian distributions. This means that the next state X; 1 must be a linear function of the current state X;, plus some Gaussian noise, a condition that turns out to be quite reasonable in practice. Consider, for example, the X-coordinate of the bird, ignoring the other coordinates for now. Let the interval between

rd observations be A, and let us assume constant velocity; then the position update is given by

Xoa=Xi+ XA.

If we add Gaussian noise then we have a linear Gaussian transition model: P(X =x Al Xe = Tt, Xi = dy) = N(x + Tt A, o) (xen) . The Bayesian network structure for a system with position X; and velocity X, is shown in Figure 15.7. Note that this is a very specific form of linear Gaussian model; the general form will be described later in this section and covers a vast array of applications beyond the simple motion examples of the first paragraph. The reader might wish to consult Appendix A for some of the mathematical properties of Gaussian distributions; for our immediate purposes, MOL TYARIATE the most important is that a multivariate Gaussian distribution for d variables is specified

by a d-element mean pu and a d x d covariance matrix X.

Section 15.4.

Kalman Filters 553

GE)

Figure 15.7 Bayesian network structure for a linear dynamical system with position X;, velocity X,, and position measurement Z;.

Updating Gaussian distributions

In Chapter 14, we alluded to a key property of the linear Gaussian family of distributions: it remains closed under the standard Bayesian network operations. Here, we make this claim precise in the context of filtering in a temporal probability model. The required properties correspond to the two-step filtering calculation in Equation (15.3):

1. If the current distribution P(X; |e;.;) is Gaussian and the transition model P(X; ,1|x;) is linear Gaussian, then the one-step predicted distribution given by

P(X;yiers) = / P(X, 11 x0) P(e[er) dx (15.15) Xt

is also a Gaussian distribution.

2. If the predicted distribution P(X; 1|e;.;) is Gaussian and sensor model P(e; 1|X;,1) is linear Gaussian, then, after conditioning on the new evidence, the updated distribution

P(Xipilerner1) = aPlery1]Xey1)P(Xer1/€1:) (15.16) is also a Gaussian distribution.

Thus, the FORWARD operator for Kalman filtering takes a Gaussian forward message f.;, specified by a mean pt, and covariance matrix 3, and produces a new multivariate Gaussian forward message fy, 1, specified by a mean pu, ; and covariance matrix X;11. So, if we start with a Gaussian prior f1.0 = P(X) = N (pg, Xo), filtering with a linear Gaussian model produces a Gaussian state distribution for all time.

This seems to be a nice, elegant result, but why is it so important? The reason is that, except for a few special cases such as this, filtering with continuous or hybrid (discrete and continuous) networks generates state distributions whose representation grows without bound over time. This statement is not easy to prove in general, but Exercise 15.5 shows what happens for a simple example.

554

Chapter 15. Probabilistic Reasoning over Time

[]

COMPLETING THE SQUARE

A simple one-dimensional example

We have said that the FORWARD operator for the Kalman filter maps a Gaussian into a new Gaussian. This translates into computing a new mean and covariance matrix from the previous mean and covariance matrix. Deriving the update rule in the general (multivariate) case requires rather a lot of linear algebra, so we will stick to a very simple univariate case for now; later we will give the results for the general case. Even for the univariate case, the calculations are somewhat tedious, but we feel that they are worth seeing because the usefulness of the Kalman filter is tied so intimately to the mathematical properties of Gaussian distributions. The temporal model we will consider describes a random walk of a single continuous state variable X; with a noisy observation Z;. An example might be the “consumer confidence” index, which can be modeled as undergoing a random Gaussian-distributed change each month and is measured by a random consumer survey that also introduces Gaussian

sampling noise. The prior distribution is assumed to be Gaussian with variance oZ:

2

I! P(xp) =e d 70

(eos)

(For simplicity, we will use the same symbol « for all normalizing constants in this section.) The transition model simply adds a Gaussian perturbation of constant variance 02 to the current state:

~g (fret) P(x q|zg) = ae \* 7% .

The sensor model then assumes Gaussian noise with variance 2:

~1 (Le =) P(z|ze) = ae 7% .

Now, given the prior P( Xj), we can compute the one-step predicted distribution using Equation (15.15):

=a / iE ) dzxo . —00

This integral looks rather complicated. The key to progress is to notice that the exponent is the sum of two expressions that are quadratic in xg and hence is itself a quadratic in zg. A simple trick known as completing the square allows the rewriting of any quadratic az + bag + ¢ as the sum of a squared term a(xg — 52)? and a residual term ¢ — © that is independent of

xg. The residual term can be taken outside the integral, giving us

[]

—00 Now the integral is just the integral of a Gaussian over its full range, which is simply 1. Thus, we are left with only the residual term from the quadratic.

Section 15.4. Kalman Filters 555

0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05

Pk 12,225)

P(x)

Figure 15.8 Stages in the Kalman filter update cycle for a random walk with a prior given by 19 =0.0 and oy = 1.0, transition noise given by o,, = 2.0, sensor noise given by 0, = 1.0, and a first observation z; = 2.5 (marked on the x-axis). Notice how the prediction P(z;) is flattened out, relative to P(xg), by the transition noise. Notice also that the mean of the posterior P(x |z1) is slightly to the left of the observation z; because the mean is a weighted average of the prediction and the observation,

The second key step is to notice that the residual term has to be a quadratic in x1; in fact, after simplification, we obtain

1 ( tg) P(z1) = ae 07 That is, the one-step predicted distribution is a Gaussian with the same mean fo and a variance equal to the sum of the original variance 02 and the transition variance g2. A momentary exercise of intuition reveals that this is intuitively reasonable.

To complete the update step, we need to condition on the observation at the first time step, namely, z1. From Equation (15.16), this is given by

P(x1]21) = a P(z1)|z1) P(x) if =e? 1 (21s) = «ae i ( ). i ( QT )

Oz

Once again, we combine the exponents and complete the square (Exercise 15.6), obtaining

2 2 2 (eg+oz)zitozpg 2

(zg — o2+o2+o?

N=

(03 +02)02/ (02 +02 +02) P(z1]z) = ae . (15.17) Thus, after one update cycle, we have a new Gaussian distribution for the state variable.

From the Gaussian formula in Equation (15.17), we see that the new mean and standard deviation can be calculated from the old mean and standard deviation as follows:

1 \_ (of+0R)zeitoue t+1 02402402 . (15.18) 2 \_ (of+03)o?

Tt \_ oP+02+02 .

556

Chapter 15. Probabilistic Reasoning over Time

KALMAN GAIN MATRIX

So

Figure 15.8 shows one update cycle for particular values of the transition and sensor models.

The preceding pair of equations plays exactly the same role as the general filtering equation (15.3) or the HMM filtering equation (15.10). Because of the special nature of Gaussian distributions, however, the equations have some interesting additional properties. First, we can interpret the calculation for the new mean p41 as simply a weighted mean of the new observation z,,; and the old mean y;. If the observation is unreliable, then 02 is large and we pay more attention to the old mean; if the old mean is unreliable (0? is large) or the process is highly unpredictable (02 is large), then we pay more attention to the observation. Second, notice that the update for the variance a? 1 18 independent of the observation. We can therefore compute in advance what the sequence of variance values will be. Third, the sequence of variance values converges quickly to a fixed value that depends only on 02 and 02, thereby substantially simplifying the subsequent calculations. (See Exercise 15.7.)

The general case

The preceding derivation illustrates the key property of Gaussian distributions that allows Kalman filtering to work: the fact that the exponent is a quadratic form. This is true not just for the univariate case; the full multivariate Gaussian distribution has the form i Ty 1

Na E)) = ae? (x-w7=" x) Multiplying out the terms in the exponent makes it clear that the exponent is also a quadratic function of the random variables x; in x. As in the univariate case, the filtering update preserves the Gaussian nature of the state distribution.

Let us first define the general temporal model used with Kalman filtering. Both the transition model and the sensor model allow for a linear transformation with additive Gaussian noise. Thus, we have

P(xp1]xt) = N(Fx¢, Xp) (x41) P(z:|x;) = N (Hx, (ze) , where F and X, are matrices describing the linear transition model and transition noise co

variance and H and XZ, are the corresponding matrices for the sensor model. Now the update equations for the mean and covariance, in their full, hairy horribleness, are

(15.19)

Biri = Fp + Keyi(zen — HRpy) Ti = I-Kiy)(FEFT + X,)

where Ky; = (FS, FT + Z,)H (HFXZ FT + Z,)H' + X,)7 is called the Kalman gain matrix. Believe it or not, these equations make some intuitive sense. For example, consider the update for the mean state estimate yp. The term Fp, is the predicted state at t + 1, so HF us, is the predicted observation. Therefore, the term z;; — HF ut, represents the error in the predicted observation. This is multiplied by K;, 1 to correct the predicted state; hence K;,; is a measure of how seriously to take the new observation relative to the prediction. As in Equation (15.18), we also have the property that the variance update is independent of the observations. The sequence of values for 3; and K; can therefore be computed offline, and the actual calculations required during online tracking are quite modest.

(15.20)

Section 15.4. Kalman Filters 557

To illustrate these equations at work, we have applied them to the problem of tracking an object moving on the X-Y plane. The state variables are X = (X,Y X, v7 soF, 2, H, and X, are 4 x 4 matrices. Figure 15.9(a) shows the true trajectory, a series of noisy observations, and the trajectory estimated by Kalman filtering, along with the covariances indicated by the one-standard-deviation contours. The filtering process does a good job of tracking the actual motion, and, as expected, the variance quickly reaches a fixed point.

We can also derive equations for smoothing as well as filtering with linear Gaussian models. The smoothing results are shown in Figure 15.9(b). Notice how the variance in the position estimate is sharply reduced, except at the ends of the trajectory (why?), and that the estimated trajectory is much smoother.

[]

Figure 159 (a) Results of Kalman filtering for an object moving on the X-Y plane, showing the true trajectory (left to right), a series of noisy observations, and the trajectory estimated by Kalman filtering. Variance in the position estimate is indicated by the ovals. (b) The results of Kalman smoothing for the same observation sequence.

Applicability of Kalman filtering

The Kalman filter and its elaborations are used in a vast array of applications. The “classical” application is in radar tracking of aircraft and missiles. Related applications include acoustic tracking of submarines and ground vehicles and visual tracking of vehicles and people. In a slightly more esoteric vein, Kalman filters are used to reconstruct particle trajectories from bubble-chamber photographs and ocean currents from satellite surface measurements. The range of application is much larger than just the tracking of motion: any system characterized by continuous state variables and noisy measurements will do. Such systems include pulp mills, chemical plants, nuclear reactors, plant ecosystems, and national economies. The fact that Kalman filtering can be applied to a system does not mean that the results will be valid or useful. The assumptions made-—a linear Gaussian transition and sensor FEN ER MAN models—are very strong. The extended Kalman filter (EKF) attempts to overcome nonlin

558

/ Chapter 15. Probabilistic Reasoning over Time

SWITCHING KALMAN FILTER

(a) ®)

Figure 15.10 A bird flying toward a tree (top views). (a) A Kalman filter will predict the location of the bird using a single Gaussian centered on the obstacle. (b) A more realistic model allows for the bird’s evasive action, predicting that it will fly to one side or the other.

earities in the system being modeled. A system is nonlinear if the transition model cannot be described as a matrix multiplication of the state vector, as in Equation (15.19). The EKF works by modeling the system as locally linear in x, in the region of x; = p1,, the mean of the current state distribution. This works well for smooth, well-behaved systems and allows the tracker to maintain and update a Gaussian state distribution that is a reasonable approximation to the true posterior.

What does it mean for a system to be “unsmooth” or “poorly behaved”? Technically, it means that there is significant nonlinearity in system response within the region that is “close” (according to the covariance X;) to the current mean p,. To understand this idea in nontechnical terms, consider the example of trying to track a bird as it flies through the jungle. The bird appears to be heading at high speed straight for a tree trunk. The Kalman filter, whether regular or extended, can make only a Gaussian prediction of the location of the bird, and the mean of this Gaussian will be centered on the trunk, as shown in Figure 15.10(a). A reasonable model of the bird, on the other hand, would predict evasive action to one side or the other, as shown in Figure 15.10(b). Such a model is highly nonlinear, because the bird’s decision varies sharply depending on its precise location relative to the trunk.

In order to handle examples like these, we clearly need a more expressive language for representing the behavior of the system being modeled. Within the control theory community, for which problems such as evasive maneuvering by aircraft raise the same kinds of difficulties, the standard solution is the switching Kalman filter. In this approach, multiple Kalman filters run in parallel, each using a different model of the system—for example, one for straight flight, one for sharp left turns, and one for sharp right turns. A weighted sum of predictions is used, where the weight depends on how well each filter fits the current data. We

Section 15.5. Dynamic Bayesian Networks 559

will see in the next section that this is simply a special case of the general dynamic Bayesian network model, obtained by adding a discrete “maneuver” state variable to the network shown in Figure 15.7. Switching Kalman filters are discussed further in Exercise 15.5.

15.5 DYNAMIC BAYESIAN NETWORKS

RIVMICBAYESIAN A dynamic Bayesian network, or DBN, is a Bayesian network that represents a temporal probability model of the kind described in Section 15.1. We have already seen examples of DBNs: the umbrella network in Figure 15.2 and the Kalman filter network in Figure 15.7. In general, each slice of a DBN can have any number of state variables X; and evidence variables E;. For simplicity, we will assume that the variables and their links are exactly replicated from slice to slice and that the DBN represents a first-order Markov process, so that each variable can have parents only in its own slice or the immediately preceding slice.

It should be clear that every hidden Markov model can be represented as a DBN with a single state variable and a single evidence variable. It is also the case that every discretevariable DBN can be represented as an HMM; as explained in Section 15.3, we can combine all the state variables in the DBN into a single state variable whose values are all possible tuples of values of the individual state variables. Now, if every HMM is a DBN and every DBN can be translated into an HMM, what's the difference? The difference is that, by decomposing the state of a complex system into its constituent variables, the DBN is able to take advantage of sparseness in the temporal probability model. Suppose, for example, that a DBN has 20 Boolean state variables, each of which has three parents in the preceding slice. Then the DBN transition model has 20 x 2% = 160 probabilities, whereas the corresponding HMM has 220 states and therefore 2%°, or roughly a trillion, probabilities in the transition matrix. This is bad for at least three reasons: first, the HMM itself requires much more space; second, the huge transition matrix makes HMM inference much more expensive; and third, the problem of learning such a huge number of parameters makes the pure HMM model unsuitable for large problems. The relationship between DBNs and HMMs is roughly analogous to the relationship between ordinary Bayesian networks and full tabulated joint distributions.

We have already explained that every Kalman filter model can be represented in a DBN with continuous variables and linear Gaussian conditional distributions (Figure 15.7). It should be clear from the discussion at the end of the preceding section that not every DBN can be represented by a Kalman filter model. In a Kalman filter, the current state distribution is always a single multivariate Gaussian distribution—that is, a single “bump” in a particular location. DBNs, on the other hand, can model arbitrary distributions. For many real-world applications, this flexibility is essential. Consider, for example, the current location of my keys. They might be in my pocket, on the bedside table, on the kitchen counter, or dangling from the front door. A single Gaussian bump that included all these places would have to allocate significant probability to the keys being in mid-air in the front hall. Aspects of the real world such as purposive agents, obstacles, and pockets introduce “nonlinearities” that require combinations of discrete and continuous variables in order to get reasonable models.

560

Chapter 15. Probabilistic Reasoning over Time

Ry | PR) PRy FT 07 0.7 fl] 03

Ry | PUY)

fr | 09

Yilr| o2

Umbrella,

(a)

Figure 15.11 (a) Specification of the prior, transition model, and sensor model for the umbrella DBN. All subsequent slices are assumed to be copies of slice 1. (b) A simple DBN for robot motion in the X-Y plane.

Constructing DBNs

To construct a DBN, one must specify three kinds of information: the prior distribution over the state variables, P(Xp); the transition model P(X; 1|X¢); and the sensor model P(E;|X;). To specify the transition and sensor models, one must also specify the topology of the connections between successive slices and between the state and evidence variables. Because the transition and sensor models are assumed to be stationary—the same for all t—it is most convenient simply to specify them for the first slice. For example, the complete DBN specification for the umbrella world is given by the three-node network shown in Figure 15.11(a). From this specification, the complete (semi-infinite) DBN can be constructed as needed by copying the first slice.

Let us now consider a more interesting example: monitoring a battery-powered robot moving in the X-Y plane, as introduced in Section 15.1. First, we need state variables, which will include both X; = (X, Y;) for position and X; = (Xs, Y:) for velocity. We will assume some method of measuring position—perhaps a fixed camera or onboard GPS (Global Positioning System)—-yielding measurements Z;. The position at the next time step depends on the current position and velocity, as in the standard Kalman filter model. The velocity at the next step depends on the current velocity and the state of the battery. We add Battery; to represent the actual battery charge level, which has as parents the previous battery level and the velocity, and we add BMeter;, which measures the battery charge level. This gives us the basic model shown in Figure 15.11(b).

It is worth looking in more depth at the nature of the sensor model for BMeter;. Let us suppose, for simplicity, that both Battery; and BMeter; can take on discrete values 0 through 5—rather like the battery meter on a typical laptop computer. If the meter is always accurate, then the CPT P(BMeter;|Battery;) should have probabilities of 1.0 “along the

Section 15.5.

Dynamic Bayesian Networks 561

GAUSSIAN ERROR MODEL

TRANSIENT FAILURE

diagonal” and probabilities of 0.0 elsewhere. In reality, noise always creeps into measurements. For continuous measurements, a Gaussian distribution with a small variance might be used instead.\* For our discrete variables, we can approximate a Gaussian using a distribution in which the probability of error drops off in the appropriate way, so that the probability of a large error is very small. We will use the term Gaussian error model to cover both the continuous and discrete versions.

Anyone with hands-on experience of robotics, computerized process control, or other forms of automatic sensing will readily testify to the fact that small amounts of measurement noise are often the least of one’s problems. Real sensors fail. When a sensor fails, it does not necessarily send a signal saying, “Oh, by the way, the data I’m about to send you is a load of nonsense.” Instead, it simply sends the nonsense. The simplest kind of failure is called a transient failure, where the sensor occasionally decides to send some nonsense. For example, the battery level sensor might have a habit of sending a zero when someone bumps the robot, even if the battery is fully charged.

Let's see what happens when a transient failure occurs with a Gaussian error model that doesn’t accommodate such failures. Suppose, for example, that the robot is sitting quietly and observes 20 consecutive battery readings of 5. Then the battery meter has a temporary seizure and the next reading is BMeters; =0. What will the simple Gaussian error model lead us to believe about Battery21? According to Bayes’ rule, the answer depends on both the sensor model P(BMeters1 = 0| Batterys;) and the prediction P( Batteryai | BMetery.p). If the probability of a large sensor error is significantly less likely than the probability of a transition to Batterys; = 0, even if the latter is very unlikely, then the posterior distribution will assign a high probability to the battery’s being empty. A second reading of zero at ¢ = 22 will make this conclusion almost certain. If the transient failure then disappears and the reading returns to 5 from ¢ = 23 onwards, the estimate for the battery level will quickly return to 5, as if by magic. This course of events is illustrated in the upper curve of Figure 15.12(a), which shows the expected value of Battery; over time using a discrete Gaussian error model.

Despite the recovery, there is a time (¢ = 22) when the robot is convinced that its battery is empty; presumably, then, it should send out a mayday signal and shut down. Alas, its oversimplified sensor model has led it astray. How can this be fixed? Consider a familiar example from everyday human driving: on sharp curves or steep hills, one’s “fuel tank empty” warning light sometimes turns on. Rather than looking for the emergency phone, one simply recalls that the fuel gauge sometimes gives a very large error when the fuel is sloshing around in the tank. The moral of the story is the following: in order for the system to handle sensor failure properly, the sensor model must include the possibility of failure.

The simplest kind of failure model for a sensor allows a certain probability that the sensor will return some completely incorrect value, regardless of the true state of the world. For example, if the battery meter fails by returning 0, we might say that

P(BMeter; =0| Battery; = 5) = 0.03 , which is presumably much larger than the probability assigned by the simple Gaussian error

4 Strictly speaking, a Gaussian distribution is problematic because it assigns nonzero probability to large negative charge levels. The beta distribution is sometimes a better choice for a variable whose range is restricted.

562

Chapter 15. Probabilistic Reasoning over Time

[]

TRANSIENT FAILURE MODEL

PERSISTENT FAILURE MODEL

PERSISTENCE ARC

(a) (b)

Figure 15.12 (a) Upper curve: trajectory of the expected value of Battery; for an observation sequence consisting of all 5s except for Os at t = 21 and ¢ = 22, using a simple Gaussian error model. Lower curve: trajectory when the observation remains at 0 from ¢ = 21 onwards. (b) The same experiment run with the transient failure model. Notice that the transient failure is handled well, but the persistent failure results in excessive pessimism.

|

model. Let's call this the transient failure model. How does it help when we are faced with a reading of 0? Provided that the predicted probability of an empty battery, according to the readings so far, is much less than 0.03, then the best explanation of the observation BMeters; =0 is that the sensor has temporarily failed. Intuitively, we can think of the belief about the battery level as having a certain amount of “inertia” that helps to overcome temporary blips in the meter reading. The upper curve in Figure 15.12(b) shows that the transient failure model can handle transient failures without a catastrophic change in beliefs.

So much for temporary blips. What about a persistent sensor failure? Sadly, failures of this kind are all too common. If the sensor returns 20 readings of 5 followed by 20 readings of 0, then the transient sensor failure model described in the preceding paragraph will result in the robot gradually coming to believe that its battery is empty when in fact it may be that the meter has failed. The lower curve in Figure 15.12(b) shows the belief “trajectory” for this case. By t= 25—five readings of 0—-the robot is convinced that its battery is empty. Obviously, we would prefer the robot to believe that its battery meter is broken—if indeed this is the more likely event.

Unsurprisingly, to handle persistent failure, we will need a persistent failure model that describes how the sensor behaves under normal conditions and after failure. To do this, we need to augment the hidden state of the system with an additional variable, say BM Broken, that describes the status of the battery meter. The persistence of failure must be modeled by an arc linking BM Broken to BMBroken,. This persistence arc has a CPT that gives a small probability of failure in any given time step, say 0.001, but specifies that the sensor stays broken once it breaks. When the sensor is OK, the sensor model for BMeter is identical to the transient failure model; when the sensor is broken, it says BMeter is always 0, regardless of the actual battery charge.

Section 15.5. Dynamic Bayesian Networks 563

[]

(a) (b)

Figure 15.13 (a) A DBN fragment showing the sensor status variable required for modeling persistent failure of the battery sensor. (b) Upper curves: trajectories of the expected value of Battery; for the “transient failure” and “permanent failure” observations sequences. Lower curves: probability trajectories for BM Broken given the two observation sequences.

The persistent failure model for the battery sensor is shown in Figure 15.13(a). Its performance on the two data sequences (temporary blip and persistent failure) is shown in Figure 15.13(b). There are several things to notice about these curves. First, in the case of the temporary blip, the probability that the sensor is broken rises significantly after the second 0 reading, but immediately drops back to zero once a 5 is observed. Second, in the case of persistent failure, the probability that the sensor is broken rises quickly to almost 1 and stays there. Finally, once the sensor is known to be broken, the robot can only assume that its battery discharges at the “normal” rate, as shown by the gradually descending level of E(Batteryy|...).

So far, we have merely scratched the surface of the problem of representing complex processes. The variety of transition models is huge, encompassing topics as disparate as modeling the human endocrine system and modeling multiple vehicles driving on a freeway. Sensor modeling is also a vast subfield in itself, but even subtle phenomena, such as sensor drift, sudden decalibration, and the effects of exogenous conditions (such as weather) on sensor readings, can be handled by explicit representation within dynamic Bayesian networks.

Exact inference in DBNs

Having sketched some ideas for representing complex processes as DBNs, we now turn to the question of inference. In a sense, this question has already been answered: dynamic Bayesian networks are Bayesian networks, and we already have algorithms for inference in Bayesian networks. Given a sequence of observations, one can construct the full Bayesian network representation of a DBN by replicating slices until the network is large enough to accommodate UNROLLING the observations, as in Figure 15.14. This technique is called unrolling. (Technically, the DBN is equivalent to the semi-infinite network obtained by unrolling forever. Slices added

564

Chapter 15. Probabilistic Reasoning over Time

[]

Figure 15.14 Unrolling a dynamic Bayesian network: slices are replicated to accommodate the observation sequence (shaded nodes). Further slices have no effect on inferences within the observation period.

beyond the last observation have no effect on inferences within the observation period and can be omitted.) Once the DBN is unrolled, one can use any of the inference algorithms— variable elimination, join-tree methods, and so on—described in Chapter 14.

Unfortunately, a naive application of unrolling would not be particularly efficient. If we want to perform filtering or smoothing with a long sequence of observations ey.;, the unrolled network would require O(t) space and would thus grow without bound as more observations were added. Moreover, if we simply run the inference algorithm anew each time an observation is added, the inference time per update will also increase as O(t).

Looking back to Section 15.2, we see that constant time and space per filtering update can be achieved if the computation can be done in a recursive fashion. Essentially, the filtering update in Equation (15.3) works by summing out the state variables of the previous time step to get the distribution for the new time step. Summing out variables is exactly what the variable elimination (Figure 14.10) algorithm does, and it turns out that running variable elimination with the variables in temporal order exactly mimics the operation of the recursive filtering update in Equation (15.3). The modified algorithm keeps at most two slices in memory at any one time: starting with slice 0, we add slice 1, then sum out slice 0, then add slice 2, then sum out slice 1, and so on. In this way, we can achieve constant space and time per filtering update. (The same performance can be achieved by making suitable modifications to the join tree algorithm.) Exercise 15.10 asks you to verify this fact for the umbrella network.

So much for the good news; now for the bad news: It turns out that the “constant” for the per-update time and space complexity is, in almost all cases, exponential in the number of state variables. What happens is that, as the variable elimination proceeds, the factors grow to include all the state variables (or, more precisely, all those state variables that have parents in the previous time slice). The maximum factor size is O(d""!) and the update cost is O(d"\*?).

Of course, this is much less than the cost of HMM updating, which is O(d?"), but it is still infeasible for large numbers of variables. This grim fact is somewhat hard to accept. What it means is that even though we can use DBNs to represent very complex temporal processes with many sparsely connected variables, we cannot reason efficiently and exactly about those processes. The DBN model itself, which represents the prior joint distribution over all the variables, is factorable into its constituent CPTs, but the posterior joint distribu

Section 15.5.

Dynamic Bayesian Networks 565

tion conditioned on an observation sequence—that is, the forward message—is generally not factorable. So far, no one has found a way around this problem, despite the fact that many important areas of science and engineering would benefit enormously from its solution. Thus, we must fall back on approximate methods.

Approximate inference in DBNs

Chapter 14 described two approximation algorithms: likelihood weighting (Figure 14.14) and Markov chain Monte Carlo (MCMC, Figure 14.15). Of the two, the former is most easily adapted to the DBN context. We will see, however, that several improvements are required over the standard likelihood weighting algorithm before a practical method emerges.

Recall that likelihood weighting works by sampling the non-evidence nodes of the network in topological order, weighting each sample by the likelihood it accords to the observed evidence variables. As with the exact algorithms, we could apply likelihood weighting directly to an unrolled DBN, but this would suffer from the same problems in terms of increasing time and space requirements per update as the observation sequence grows. The problem is that the standard algorithm runs each sample in turn, all the way through the network. Instead, we can simply run all NV samples together through the DBN, one slice at a time. The modified algorithm fits the general pattern of filtering algorithms, with the set of N samples as the forward message. The first key innovation, then, is to use the samples themselves as an approximate representation of the current state distribution. This meets the requirement of a “constant” time per update, although the constant depends on the number of samples required to maintain a reasonable approximation to the true posterior distribution. There is also no need to unroll the DBN, because we need to have in memory only the current slice and the next slice.

In our discussion of likelihood weighting in Chapter 14, we pointed out that the algorithm’s accuracy suffers if the evidence variables are “downstream” from the variables being sampled, because in that case the samples are generated without any influence from the evidence. Looking at the typical structure of a DBN—say, the umbrella DBN in Figure 15.14—we see that indeed the early state variables will be sampled without the benefit of the later evidence. In fact, looking more carefully, we see that none of the state variables has any evidence variables among its ancestors! Hence, although the weight of each sample will depend on the evidence, the actual set of samples generated will be completely independent of the evidence. For example, even if the boss brings in the umbrella every day, the sampling process could still hallucinate endless days of sunshine. What this means in practice is that the fraction of samples that remain reasonably close to the actual series of events drops exponentially with ¢, the length of the observation sequence; in other words, to maintain a given level of accuracy, we need to increase the number of samples exponentially with ¢. Given that a filtering algorithm that works in real time can only use a fixed number of samples, what happens in practice is that the error blows up after a very small number of update steps.

Clearly, we need a better solution. The second key innovation is to focus the set of samples on the high-probability regions of the state space. This can be done by throwing away samples that have very low weight, according to the observations, while multiplying

566

Chapter 15. Probabilistic Reasoning over Time

PARTICLE FILTERING

function PARTICLE-FILTERING(e, N, dbn) returns a set of samples for the next time step inputs: e, the new incoming evidence N, the number of samples to be maintained dbn, a DBN with prior P(Xj), transition model P(X; X,), and sensor model P(E;|X;) static: S, a vector of samples of size N, initially generated from P(X) local variables: W, a vector of weights of size N

fori=1to N do

S[i] — sample from P(X;|X, = S[i])

Wi] —P(e|X; = S[i]) S «— WEIGHTED-SAMPLE-WITH-REPLACEMENT(N, S, W) return S

Figure 15.15 The particle filtering algorithm implemented as a recursive update operation with state (the set of samples). Each of the sampling steps involves sampling the relevant slice variables in topological order, much as in PRIOR-SAMPLE. The WEIGHTED-SAMPLEWITH-REPLACEMENT operation can be implemented to run in O(V) expected time.

those that have high weight. In that way, the population of samples will stay reasonably close to reality. If we think of samples as a resource for modeling the posterior distribution, then it makes sense to use more samples in regions of the state space where the posterior is higher. A family of algorithms called particle filtering is designed to do just that. Particle filtering works as follows: First, a population of N samples is created by sampling from the prior distribution at time 0, P(Xp). Then the update cycle is repeated for each time step:

e Each sample is propagated forward by sampling the next state value Xx;; given the current value x; for the sample, and using the transition model P(X¢1]x;).

e Each sample is weighted by the likelihood it assigns to the new evidence, P(e; y1|X¢+1).

e The population is resampled to generate a new population of N samples. Each new sample is selected from the current population; the probability that a particular sample is selected is proportional to its weight. The new samples are unweighted.

The algorithm is shown in detail in Figure 15.15, and its operation for the umbrella DBN is illustrated in Figure 15.16.

We can show that this algorithm is consistent—gives the correct probabilities as NV tends to infinity—by considering what happens during one update cycle. We will assume that the sample population starts with a correct representation of the forward message fy.; at time ¢. Writing N (x;|e;.+) for the number of samples occupying state x; after observations e;;; have been processed, we therefore have

N(x¢|er.t)/N = P(x¢|eq.) (15.21)

for large NV. Now we propagate each sample forward by sampling the state variables at ¢ + 1, given the values for the sample at ¢. The number of samples reaching state x; 1 from each

Section 15.5. Dynamic Bayesian Networks 567 Rain, Rain, Rain, Rain, eee eee 660 @ frue eeee eee coe ® ® ®e ee ®00e false |e ®® @e eee (a) Propagate (b) Weight (c) Resample

Figure 15.16 The particle filtering update cycle for the umbrella DBN with N = 10, showing the sample populations of each state. (a) At time ¢, 8 samples indicate Rain and 2 indicate — Rain. Each is propagated forward by sampling the next state via the transition model. At time ¢ + 1, 6 samples indicate Rain and 4 indicate — Rain, (b) = Umbrella is observed at t + 1. Each sample is weighted by its likelihood for the observation, as indicated by the size of the circles. (c) A new set of 10 samples is generated by weighted random selection from the current set, resulting in 2 samples that indicate Rain and 8 that indicate — Rain.

x; is the transition probability times the population of x¢; hence, the total number of samples reaching X; 1 is

N(xt1]en:t) = 2 Xt41]Xt) N (Xe|eq:t)

Now we weight each sample by its likelihood for the evidence at ¢t + 1. A sample in state X; +1 receives weight P(e; 1|X:41). The total weight of the samples in x; 1 after seeing e;; is therefore

W(X¢r1l€1:641) = P€11]Xeq1) N(Xep1/€rt)

Now for the resampling step. Since each sample is replicated with probability proportional to its weight, the number of samples in state x; after resampling is proportional to the total weight in x; 1 before resampling:

[]

Xt = P(x¢tq1lersr1) (by 15.3).

Therefore the sample population after one update cycle correctly represents the forward message at time ¢ + 1.

Particle filtering is consistent, therefore, but is it efficient? In practice, it seems that the answer is yes: particle filtering seems to maintain a good approximation to the true posterior using a constant number of samples. There are, as yet, no theoretical guarantees; particle

568

Chapter 15. Probabilistic Reasoning over Time

filtering is currently an area of intensive study. Many variants and improvements have been proposed, and the set of applications is growing rapidly. Because it is a sampling algorithm, particle filtering can be used easily with hybrid and continuous DBNs, allowing it to be applied to areas such as tracking complex motion patterns in video (Isard and Blake, 1996) and predicting the stock market (de Freitas et al., 2000).

SPEECH RECOGNITION

ACOUSTIC MODEL

HOMOPHONES

PHONOLOGY PHONES

PHONEME

In this section, we look at one of the most important applications of temporal probability models—speech recognition. The task is to identify a sequence of words uttered by a speaker, given the acoustic signal. Speech is the dominant modality for communication between humans, and reliable speech recognition by machines would be immensely useful. Still more useful would be speech understanding—the identification of the meaning of the utterance. For this, we must wait until Chapter 22.

Speech provides our first contact with the raw, unwashed world of real sensor data. These data are noisy, quite literally: there can be background noise as well as artifacts introduced by the digitization process; there is variation in the way that words are pronounced, even by the same speaker; different words can sound the same; and so on. For these reasons, speech recognition has come to be viewed as a problem of probabilistic inference.

At the most general level, we can define the probabilistic inference problem as follows. Let Words be a random variable ranging over all possible sequences of words that might be uttered, and let signal be the observed acoustic signal sequence. Then the most likely interpretation of the utterance is the value of Words that maximizes P(words|signal). As is often the case, applying Bayes’ rule is helpful:

P(words|signal) = a P(signal| words) P(words) .

P(signal|words) is the acoustic model. It describes the sounds of words—that “ceiling” begins with a soft “c” and sounds the same as “sealing.” (Words that sound the same are called homophones.) P(words) is known as the language model. It specifies the prior probability of each utterance—for example, that “high ceiling” is a much more likely word sequence than “high sealing.”

The language models used in speech recognition systems are usually very simple. The bigram model that we describe later in this section gives the probability of each word following each other word. The acoustic model is much more complex. At its heart is an important discovery made in the field of phonology (the study of how language sounds), namely, that all human languages use a limited repertoire of about 40 or 50 sounds, called phones. Roughly speaking, a phone is the sound that corresponds to a single vowel or consonant, but there are some complications: combinations of letters, such as “th” and “ng” produce single phones, and some letters produce different phones in different contexts (e.g., the “a” in rar and rate. Figure 15.17 lists phones that are used English with an example of each. A phoneme is the smallest unit of sound that has a distinct meaning to speakers of a particular language. For example, in English the “t” phone in “stick” is the same phoneme as the “t” phone in “tick,”

Section 15.6.

Speech Recognition 569

PRONUNCIATION

SIGNAL PROCESSING

Vowels Consonants B-N Consonants P-Z Phone Example Phone Example Phone Example [iy] beat [b] bet [pl pet [ih] bit [ch] Chet [r] rat [eh] bet [d] debt [s] set [2] bat (f] fat [sh] shoe [ah] but [g] get [t] ten [ao] bought [hh] hat [th] thick [ow] boat [hv] high [dh] that [uh] book [jh] jet [dx] butter [ey] bait [k] kick [vl vet [er] Bert [1] let [Ww] wet [ay] buy [el] bottle [wh] which [oy] boy [m] met [yl] yet [axr] diner [em] bottom [z] Z00 [aw] down [n] net [zh] measure [ax] about [en] button [ix] roses [ng] sing [aa] cot [eng] washing [-] silence Figure 15.17 The DARPA phonetic alphabet, or ARPAbet, listing all the phones used in American English. There are several alternative notations, including an International Phonetic Alphabet (IPA), which contains the phones in all known languages.

but in Thai they are distinguishable as two separate phonemes.

The existence of phones makes it possible to divide the acoustic model into two parts. The first part deals with pronunciation and specifies, for each word, a probability distribution over possible phone sequences. For example, “ceiling” is pronounced [s iy 1 ih ng], or sometimes [s iy 1 ix ng], or sometimes even [s iy 1 en]. The phones are not directly observable, so, roughly speaking, speech is represented as a hidden Markov model whose state variable X; specifies which phone is being uttered at time ¢.

The second part of the acoustic model deals with the way that phones are realized as acoustic signals: that is, the evidence variable FE; for the hidden Markov model gives the observed features of the acoustic signal at time ¢, and the acoustic model specifies P(Ex|X¢), where X; is the current phone. The model must allow for variations in pitch, speed, and volume, and relies on techniques from signal processing to provide signal descriptions that are reasonably robust against these kinds of variations.

The remainder of the section describes the models and algorithms from the bottom up, beginning with acoustic signals and phones, then individual words, and finally entire sentences. We conclude with a description of how all these models are trained and how well the resulting systems work.

570

Chapter 15. Probabilistic Reasoning over Time

SAMPLING RATE

QUANTIZATION FACTOR

FRAMES

FEATURES

Speech sounds

Sound waves are periodic changes in pressure that propagate through the air. Sound can be measured by a microphone whose diaphragm is displaced by the pressure changes and generates a continuously varying current. An analog-to-digital converter measures the size of the current—which corresponds to the amplitude of the sound wave—at discrete intervals determined by the sampling rate. For speech, a sampling rate between § and 16 kHz (i.e, 8 to 16,000 times per second) is typical. (High-quality music recordings sample at a rate of 44 kHz or more.) The precision of each measurement is determined by the quantization factor; speech recognizers typically keep 8 to 12 bits. That means that a low-end system, sampling at 8 kHz with 8-bit quantization, would require nearly half a megabyte per minute of speech. It would be impractical to construct and manipulate P(signal|phone) distributions with so much signal information; therefore, we need to develop more concise descriptions of the acoustic signal.

First, we observe that although the sound frequencies in speech may be several kHz, the changes in the content of the signal occur much less often, perhaps at no more than 100 Hz. Therefore, speech systems summarize the properties of the signal over extended intervals called frames. A frame length of about 10 msecs (i.e., 80 samples at 8 kHz) is short enough to ensure that few short-duration phenomena will be smudged out by the summarization process. Within each frame, we represent what is happening with a vector of features. For example, we might want to characterize the amount of energy at each of several frequency ranges. Other important features include the overall energy in a frame and the difference from the previous frame. Picking out features from a speech signal is like listening to an orchestra and saying “here the French horns are playing loudly and the violins are playing softly.” Figure 15.18 shows the sequence of transformations from the raw sound to a sequence of frames. Note that the frames overlap; this prevents us from losing information if an important acoustic event just happens to fall on a frame boundary.

In our example, we have shown frames with just three features. Real systems have tens or even hundreds of features. If there are n features and each has, say, 256 possible values, then a frame is described by a point in n-dimensional space and there are 256™ possible

ha Pont

Sampled, quantized | | ul alii, | 1

Analog acoustic signal:

digital signal:

10 15 38 | 22 63 24 , 1012 73 |

I Frames with features: 527 52 20 94 11

Figure 15.18 Translating the acoustic signal into a sequence of frames; each frame is described by the values of three acoustic features.

Section 15.6. Speech Recognition 571

frames. For n > 2 it would be impractical to represent the distribution P(features|phone) as an explicit table, so we need further compression. There are two possible approaches:

SECTOR (TION e The method of vector quantization, or VQ, divides the n-dimensional space into, say, 256 regions labeled C1 through C256. Each frame can then be represented with a single label rather than a vector of n numbers. Thus, the tabulated distribution P(VQ|phone) has 256 probabilities specified for each phone. Vector quantization is no longer popular in large-scale systems.

o Instead of discretizing the feature space, we can use a parameterized continuous distribution to describe P(features|phone). For example, we could use a Gaussian distribution with a different mean and covariance matrix for each phone. This works well if the acoustic realizations of each phone are clustered in a single region of feature space. In practice, the sounds can be spread over several regions, and a mixture of Gaussians must be used. A mixture is a weighted sum of & individual distributions, so P(features|phone) has k weights, k mean vectors of size n, and k covariance matrices of size n2—that is, O(kn?) parameters for each phone.

Of course, some information is lost in going from the full speech signal to a VQ label or a set of mixture parameters. The art of signal processing lies in choosing features and regions (or Gaussians) so that the loss of useful information is minimized. A given speech sound can be pronounced so many ways: loud or soft, fast or slow, high pitched or low, against a background of silence or of noise, and by any of millions of different speakers, each with a different accent and vocal tract. Signal processing hopes to eliminate the variations while keeping the commonalities that define the sound.’

There are two more refinements we need to make to the simple model we have described so far. The first deals with the temporal structure of phones. In normal speech, most phones have a duration of 50-100 milliseconds, or 5-10 frames. The probability model P(features|phone) is the same for all these frames, whereas most phones have a good deal

sopconsonants of internal structure. For example, [t] is one of several stop consonants, in which the flow of air is cut off for a short period before a sharp release. Examining the acoustic signal, we find that [t] has a silent beginning, a small explosion in the middle, and (usually) a hissing at the

mreesarepHoNE end. This internal structure of phones can be captured by the three-state phone model; each phone has Onset, Mid, and End states, and each state has its own distribution over features.

The second refinement deals with the context in which the phone is uttered. The sound of a given phone can be changed by the surrounding phones.® Remember that speech sounds are produced by moving the lips, tongue, and jaw and forcing air through the vocal tract. To coordinate these complex movements at the rate of five or more phones per second, the brain initiates action for a second phone before the first is completed, thereby altering one or both phones. For example, in pronouncing “sweet” the lips are rounded during the production of

COeRTIGULATION [s] in anticipation of the following [w]. These coarticulation effects are partially captured 5 The complementary problem, speaker identification, eliminates the commonalities and keeps the individual differences, and then tries to match the differences to models of individual speakers.

6 In this sense, the “phone model” of speech should be thought of as a useful approximation rather than an immutable law.

572

Chapter 15. Probabilistic Reasoning over Time

TRIPHONE

by the triphone model, in which the acoustic model for each phone is allowed to depend on the preceding and succeeding phones. Thus, the [w] in “sweet is written [w(s,iy)], i.e., [W] with left-context [s] and right-context [iy].

The combined effect of the three-state and triphone models is to increase the number of possible states of the temporal process from n phones in the original phone alphabet (n ~ 50 for the ARPAbet) to 3n3. Experience shows that the improved accuracy more than offsets the extra expense in terms of inference and learning.

Words

We can think of each word as specifying a distinct probability distribution P(X;..|word), where X; specifies the phone state in the ith frame. Typically, we separate this distribution into two parts. The pronunciation model gives a distribution over phone sequences (ignoring metric time and frames), while the phone model describes how a phone maps into a sequence of frames.

Consider the word “tomato.” According to Gershwin (1937), you say [t ow m ey t ow] and I say [t ow m aa t ow]. The top of Figure 15.19 shows a transition model that provides for this variation. There are only two possible paths through the model, one corresponding to the phone sequence [t ow m ey t ow] and the other to [t ow m aa t ow]. The probability of a path is the product of the probabilities on the arcs that make up the path:

P([towmeytow]|“tomato”) = P([towmaatow]|“tomato”) = 0.5.

The second source of phonetic variation is coarticulation. For example, the [t] phone is produced with the tongue at the top of the mouth, whereas the [ow] has the tongue near the bottom. When spoken quickly, the tongue often goes to an intermediate position, and we get [t ah] rather than [t ow]. The bottom half of Figure 15.19 gives a more complicated pronunciation model for “tomato” that takes this coarticulation effect into account. In this model, there are four distinct paths, and we have

P([towmeytow]|“tomato”) = P([towmaatow]|“tomato”) = 0.1 , P([tahmeytow]| tomato”) = P([tahmaatow]|“tomato™) = 0.4 .

Similar models can be constructed for every word we want to be able to recognize.

The model for a three-state phone is shown as a state transition diagram in Figure 15.20. The model is for a particular phone, [m], but all phones will have models with similar topology. For each phone state, we show the associated acoustic model, assuming that the signal is represented by a VQ label. For example, the model asserts that P(E; = C1| Xt = [mlopget) = 0.5. Notice the self-loops in the figure; for instance, the [m]yy;q state persists with probability 0.9, which means that the [m]y;q state has an expected duration of 10 frames. In the model we have, the duration of each phone is independent of the duration of other phones; a more sophisticated model could distinguish between fast and slow speech.

We can construct similar models for each phone, possibly depending on the triphone context. Each word model, when combined with the phone models, gives a complete specification of an HMM. The model specifies the transition probabilities between phone states from frame to frame, as well as the acoustic feature probabilities for each phone state.

Section 15.6. Speech Recognition 573

[]

Figure 15.19 Two pronunciation models of the word “tomato.” Each model is shown as a transition diagram with states as circles and arrows showing allowed transitions with their associated probabilities. (a) A model allowing for dialect differences. The 0.5 numbers are estimates based on the two authors’ preferred pronunciations. (b) A model with a coarticulation effect on the first vowel, allowing either the [ow] or the [ah] phone.

[]

Output probabilities for the phone HMM:

Onset: Mid: End:

Ci: 05 Cy: 02 Cy 0.1 Cy: 0.2 Cy 0.7 Ce: 0.5 C3: 03 Cs: 0.1 C04

Figure 15.20 An HMM for the three-state phone [m]. Each state has several possible outputs, each with its own probability. The VQ labels Cy through Care arbitrary.

574

Chapter 15. Probabilistic Reasoning over Time

ISOLATED WORDS

CONTINUOUS SPEECH

SEGMENTATION

BIGRAM

If we want to recognized isolated words—that is, words spoken without any surrounding context and with clear boundaries—then we need to find the word that maximizes

P(word|e1+) = a P(ey:t| word) P(word) . The prior probability P(word) can be obtained from actual text data. P(e1.|word) is the likelihood of the sequence of acoustic features according to the word model. Section 15.2 covered the computation of such likelihoods; in particular, Equation (15.5) gives a simple recursive computation whose cost is linear in ¢ and in the number of states of the Markov chain. To find the most likely word, we can perform this calculation for each possible word model, multiply by the prior, and select the best word accordingly.

Sentences

To have a conversation with a human, a machine needs to be able to recognize continuous speech rather than just isolated words. One might think that continuous speech is nothing more than a sequence of words, to each of which we can apply the algorithm from the previous section. This approach fails for two reasons. First, we have already seen (page 547) that the sequence of most likely words is not the most likely sequence of words. For example, in the movie Take the Money and Run, a bank teller interprets Woody Allen’s sloppily written holdup note as saying “I have a gub.” A good language model would suggest “I have a gun” as a much more likely sequence, even though the last word looks more like “gub” than “gun.” The second issue we must face with continuous speech is segmentation—the problem of deciding where one word ends and the next begins. Anyone who has tried to learn a foreign language will appreciate this problem: at first all the words seem to run together. Gradually, one learns to pick out words from the jumble of sounds. In this case, first impressions are correct; a spectrographic analysis shows that in fluent speech, the words really do run together with no silence between them. We learn to identify word boundaries despite the lack of silence.

Let us begin with the language model, whose job in speech recognition is to specify the probability of each possible sequence of words. Using the notation wy wy, to denote a string of n words and w; to denote the ith word of the string, we can write an expression for the probability of a string with the use of the chain rule as follows:’

P(wy---wy) = Pw) Plwa|wi) P(ws|lwiws) --Plwp|wy «+ wp—1)

J P(w;|un cee Wi—1) .

Most of these terms are quite complex and difficult to estimate or compute. Fortunately, we can approximate this formula with something simpler and still capture a large part of the language model. One simple, popular, and effective approach is the bigram model. This model approximates P(w;|w; --wj\_1) with P(w;|w; 1). In other words, it makes a firstorder Markov assumption for word sequences.

A big advantage of the bigram model is that it is easy to train the model by counting

the number of times each word pair occurs in a representative corpus of strings and using the

7 Strictly speaking, the probability of a word sequence depends strongly on the context of the utterance; for example, “I have a gun” is much more common on notes passed to a bank teller than it is in, say, the Wall Street Journal. Few speech recognizers handle context, other than by training a special-purpose language model for a particular task.

Section 15.6.

Speech Recognition 575

[]

TRIGRAM

Unigram Previous words Word count of in is on to from model | agent the 33508 3833| 2479 832| 944| 1365 597 28 24 on 2573 1 0 33 2 1 0 0 6 of 15474 0 0 29 1 0 0 88 7 to 11527 0 4 450 21 4 16 9 82 is 10566 3 6 1 4 2 1 47 127 model 752 8 1 0 1 14 0 6 4 agent 2100 10 3 3 2 3 0 0 36 idea 241 0 0 0 0 0 0 0 0 Figure 15.21 A partial table of unigram and bigram counts for the words in this book. “The” is the most common single word with a count of 33,508 (out of 513,893 total words). The bigram “of the” is the most common, at 3,833. Some counts are higher than expected (e.g. 4 for “on is”) because the bigram counts ignore punctuation: one sentence might end with “on” and the next begin with “is.”

counts to estimate the probabilities. For example, if “a” appears 10,000 times in the training corpus and it is followed by “gun” 37 times, then P(gun;|a;\_1) = 37/10, 000, where by P we mean the estimated probability. After such training one would expect “I have” and “a gun” to have high estimated probabilities, while “I has” and “an gun” would have low probabilities. Figure 15.21 shows some bigram counts derived from the words in this book.

It is possible to go to a trigram model that provides values for P(w;|w;\_jw;—2). This is a more powerful language model, capable of judging that “ate a banana” is more likely than “ate a bandanna.” For trigram models and to a lesser extent for bigram and unigram models, there is a problem with counts of zero: We wouldn’t want to say that a combination of words is impossible just because they didn’t happen to appear in the training corpus. The process of smoothing gives a small non-zero probability to such combinations. It is discussed on page 835.

Bigram or trigram models are not as sophisticated as some of the grammar models we will see in Chapters 22 and 23, but they account for local context-sensitive effects better and manage to capture some local syntax. For example, the fact that the word pairs “I has” and “man have” get low scores is reflective of subject—verb agreement. The problem is that these relationships can be detected only locally: “the man have” gets a low score, but “the man with the yellow hat have” is not penalized.

Now we consider how to combine the language model with the word models, so that we can handle word sequences properly. We’ll assume a bigram language model for simplicity. With such a model, we can combine all the word models (which are composed in turn of pronunciation models and phone models) into one large HMM model. A state in a singleword HMM is a frame labeled by the current phone and phone state (for example, [m]opset); a state in a continuous-speech HMM is also labeled with a word, as in [m]%™mate, If each word has an average of p three-state phones in its pronunciation model, and there are W

576

Chapter 15. Probabilistic Reasoning over Time

A\* DECODER

words, then the continuous-speech HMM has 3pW states. Transitions can occur between phone states within a given phone, between phones in a given word, and between the final state of one word and the initial state of another. The transitions between words occur with probabilities specified by the bigram model.

Once we have constructed the combined HMM, we can use it to analyze the continuous speech signal. In particular, the Viterbi algorithm embodied in Equation (15.9) can be used to find the most likely state sequence. From this state sequence, we can then extract a word sequence simply by reading the word labels from the states. Thus, the Viterbi algorithm solves the word segmentation problem by using dynamic programming to consider (in effect) all possible word sequences and word boundaries simultaneously.

Notice that we didn’t say “we can extract the most likely word sequence.” The most likely word sequence is not necessarily the one that contains the most likely state sequence. This is because the probability of a word sequence is the sum of the probabilities over all possible state sequences consistent with that word sequence. Comparing two word sequences, say, “a back” and “aback,” it might be that case that there are ten alternative state sequences for “a back,” each with probability 0.03, but just one state sequence for “aback,” with probability 0.20. Viterbi chooses “aback,” but “a back” is actually more likely.

In practice, this difficulty is not life-threatening, but it is serious enough that other approaches have been tried. The most common is the A\* decoder, which makes ingenious use of A\* search (see Chapter 4) to find the most likely word sequence. The idea is to view each word sequence as a path through a graph whose nodes are labeled with words. The successors of a node are all the words that can come next; thus, the graph for all sentences of length n or less has n layers, each of width at most W, where W is the number of possible words. With a bigram model, the cost g(w1, wg) of an arc between nodes labeled wy to ws is given by —log P(wz|w1); in this way, the total path cost of a sequence is

n n Cost(wy + wy) = > — log P(wj|wi—1) = — log 11 P(w;|wi—1) . i=1 i=1 With this definition of path cost, finding the shortest path is exactly equivalent to finding the most likely word sequence. For the process to be efficient, we also need a good heuristic h(w;) to estimate the cost of completing the word sequence. Obviously, this has something to do with how much of the speech signal is not yet covered by the words on the current path. As yet, no especially interesting heuristics have been devised for this problem.

Building a speech recognizer

The quality of a speech recognition system depends on the quality of all of its components— the language model, the word pronunciation models, the phone models, and the signal processing algorithms used to extract spectral features from the acoustic signal. We have discussed how the language model can be constructed, and we leave the details of signal processing to other textbooks. We are left with the pronunciation and phone models. The structure of the pronunciation models—such as the tomato models in Figure 15.19—is usually developed by hand. Large pronunciation dictionaries are now available for English and other languages, although their accuracy varies greatly. The structure of the three-state phone models is the

Section 15.6.

Speech Recognition 577

same for all phones, as shown in Figure 15.20. That leaves the probabilities themselves. How are these to be obtained, given that the models could require hundreds of thousands or millions of parameters?

The only plausible method is to learn the models from actual speech data, of which there is certainly no shortage. The next question is how to do the learning. We give the answer in full in Chapter 20, but we can present the main ideas here. Consider the bigram language model; we explained how to learn it by looking at frequencies of word pairs in actual text. Can we do the same for, say, phone transition probabilities in the pronunciation model? The answer is yes, but only if someone goes to the trouble of annotating every occurrence of each word with the right phone sequence. This is a difficult and error-prone task, but it has been carried out for some standard data sets containing several hours of speech. If we know the phone sequences, we can estimate transition probabilities for the pronunciation models from frequencies of phone pairs. Similarly, if we are given the phone state for each frame—an even more excruciating manual labeling task—then we can estimate transition probabilities for the phone models. Given the phone state and the acoustic features in each frame, we can also estimate the acoustic model, either directly from frequencies (for VQ models) or by using statistical fitting methods (for mixture-of-Gaussian models; see Chapter 20).

The cost and rarity of hand-labeled data, and the fact that the available hand-labeled data sets might not represent the kinds of speakers and acoustic conditions found in a new recognition context, could doom this approach to failure. Fortunately, the expectation—maximization or EM algorithm learns HMM transition and sensor models without the need for labeled data. Estimates derived from hand-labeled data can be used to initialize the models; after that, EM takes over and trains the models for the task at hand. The idea is simple: given an HMM and an observation sequence, we can use the smoothing algorithms from Sections 15.2 and 15.3 to compute the probability of each state at each time step and, by a simple extension, the probability of each state—state pair at consecutive time steps. These probabilities can be viewed as uncertain labels. From the uncertain labels, we can estimate new transition and sensor probabilities, and the EM procedure repeats. The method is guaranteed to increase the fit between model and data on each iteration, and it generally converges to a much better set of parameter values than those provided by the initial, hand-labeled estimates.

State-of-the-art speech systems use enormous data sets and massive computational resources to train their models. For isolated word recognition under good acoustic conditions (no background noise or reverberation) with a vocabulary of a few thousand words and a single speaker, accuracy can be over 99%. For unrestricted continuous speech with a variety of speakers, 60-80% accuracy is common, even with good acoustic conditions. With background noise and telephone transmission, accuracy degrades further. Although fielded systems have improved continuously for decades, there is still room for many new ideas.

578 Chapter 15. Probabilistic Reasoning over Time

15.7 SUMMARY

This chapter has addressed the general problem of representing and reasoning about probabilistic temporal processes. The main points are as follows:

e The changing state of the world is handled by using a set of random variables to represent the state at each point in time.

e Representations can be designed to satisfy the Markov property, so that the future is independent of the past given the present. Combined with the assumption that the process is stationary—that is, the dynamics do not change over time—this greatly simplifies the representation.

e A temporal probability model can be thought of as containing a transition model describing the evolution and a sensor model describing the observation process.

e The principal inference tasks in temporal models are filtering, prediction, smoothing, and computing the most likely explanation. Each of these can be achieved using simple, recursive algorithms whose runtime is linear in the length of the sequence.

e Three families of temporal models were studied in more depth: hidden Markov models, Kalman filters, and dynamic Bayesian networks (which include the other two as special cases).

e Speech recognition and tracking are two important applications for temporal probability models.

e Unless special assumptions are made, as in Kalman filters, exact inference with many state variables appears to be intractable. In practice, the particle filtering algorithm seems to be an effective approximation algorithm,

BIBLIOGRAPHICAL AND HISTORICAL NOTES

Many of the basic ideas for estimating the state of dynamical systems came from the mathematician C. F. Gauss (1809), who formulated a deterministic least-squares algorithm for the problem of estimating orbits from astronomical observations. The Russian mathematician A. A. Markov (1913) developed what was later called the Markov assumption in his analysis of stochastic processes; he estimated a first-order Markov chain on letters from the text of Eugene Onegin. Significant classified work on filtering was done during World War II by Wiener (1942) for continuous-time processes and by Kolmogorov (1941) for discretetime processes. Although this work led to important technological developments over the next 20 years, its use of a frequency-domain representation made many calculations quite cumbersome. Direct state-space modeling of the stochastic process turned out to be simpler, as shown by Swerling (1959) and Kalman (1960). The latter paper introduced what is now known as the Kalman filter for forward inference in linear systems with Gaussian noise. Important results on smoothing were derived by Rauch et al. (1965), and the impressively named Rauch-Tung—Striebel smoother is still a standard technique today. Many early results

Section 15.7.

Summary 579

DATA ASSOCIATION

are gathered in Gelb (1974). Bar-Shalom and Fortmann (1988) give a more modern treatment with a Bayesian flavor, as well as many references to the vast literature on the subject. Chatfield (1989) covers the “classical” approach to time series analysis.

In many applications of Kalman filtering, one must deal not only with uncertain sensing and dynamics, but also with uncertain identity; that is, if there are multiple objects being monitored, the system must determine which observations were generated by which objects before it can update each of the state estimates. This is the problem of data association (BarShalom and Fortmann, 1988; Bar-Shalom, 1992). With n. observations and n tracks (a fairly benign case), there are n! possible assignments of observations to tracks; a proper probabilistic treatment must take all of them into account, and this can be shown to be NP-hard (Cox, 1993; Cox and Hingorani, 1994). Polynomial-time approximation methods based on MCMC appear to work well in practice (Pasula et al., 1999). It is interesting to note that the data association problem is an instance of probabilistic inference in a first-order language; unlike most probabilistic inference problems, which are purely propositional, data association involves objects as well as the identity relation. It is therefore intimately connected to the first-order probabilistic languages that were mentioned in Chapter 14. Recent work has shown that reasoning about identity in general, and data association in particular, can be carried out within the first-order probabilistic framework (Pasula and Russell, 2001).

The hidden Markov model and associated algorithms for inference and learning, including the forward-backward algorithm, were developed by Baum and Petrie (1966). Similar ideas also appeared independently in the Kalman filtering community (Rauch et al., 1965). The forward-backward algorithm was one of the main precursors of the general formulation of the EM algorithm (Dempster et al., 1977); see also Chapter 20. Constant-space smoothing appears in Binder ef al. (1997b), as does the divide-and-conquer algorithm developed in Exercise 15.3.

Dynamic Bayesian networks (DBNs) can be viewed as a sparse encoding of a Markov process and were first used in Al by Dean and Kanazawa (1989b), Nicholson (1992), and Kjaerulff (1992). The last work includes a generic extension to the HUGIN belief net system to provide the necessary facilities for dynamic Bayesian network generation and compilation. Dynamic Bayesian networks have become popular for modeling a variety of complex motion processes in computer vision (Huang et al., 1994; Intille and Bobick, 1999). The link between HMMs and DBN5s, and between the forward—backward algorithm and Bayesian network propagation, was made explicitly by Smyth et al. (1997). A further unification with Kalman filters (and other statistical models) appears in Roweis and Ghahramani (1999).

The particle filtering algorithm described in Section 15.5 has a particularly interesting history. The first sampling algorithms for filtering were developed in the control theory community by Handschin and Mayne (1969), and the resampling idea that is the core of particle filtering appeared in a Russian control journal (Zaritskii et al., 1975). It was later reinvented in statistics as sequential importance-sampling resampling, or SIR (Rubin, 1988; Liu and Chen, 1998), in control theory as particle filtering (Gordon et al., 1993; Gordon, 1994), in Al as survival of the fittest (Kanazawa et al., 1995), and in computer vision as condensation (Isard and Blake, 1996). The paper by Kanazawa ef al. (1995) includes an improvement called evidence reversal whereby the state at time ¢ + 1 is sampled conditional on both the

580

Chapter 15. Probabilistic Reasoning over Time

state at time ¢ and the evidence at time t + 1. This allows the evidence to influence sample generation directly and was proved by Doucet (1997) to reduce the approximation error.

Other methods for approximate filtering include the decayed MCMC algorithm (Marthi et al., 2002) and the factored approximation method of Boyen et al. (1999). Both of these methods have the important property that the approximation error does not diverge over time. Variational techniques (see Chapter 14) have also been developed for temporal models. Ghahramani and Jordan (1997) discuss an approximation algorithm for the factorial HMM, a DBN in which two or more independently evolving Markov chains are linked by a shared observation stream. Jordan et al. (1998) cover a number of other applications. Properties of mixing times are discussed by Pak (2001) and by (Luby and Vigoda, 1999).

The prehistory of speech recognition began in the 1920s with Radio Rex, a voiceactivated toy dog. Rex jumped in response to sound frequencies near 500 Hz, which corresponds to the [eh] vowel in “Rex!” Somewhat more serious work began after World War II. At AT&T Bell Labs, a system was built for recognizing isolated digits (Davis et al., 1952) by means of simple pattern matching of acoustic features. Phone transition probabilities were first used in a system built at University College, London, by Fry (1959) and Denes (1959). Starting in 1971, the Defense Advanced Research Projects Agency (DARPA) of the United States Department of Defense funded four competing five-year projects to develop high-performance speech recognition systems. The winner, and the only system to meet the goal of 90% accuracy with a 1000-word vocabulary, was the HARPY system at CMU (Lowerre, 1976; Lowerre and Reddy, 1980).% The final version of HARPY was derived from a system called DRAGON built by CMU graduate student James Baker (1975); DRAGON was the first to use HMMs for speech. Almost simultaneously, Jelinek (1976) at IBM had developed another HMM-based system. From that point onwards, probabilistic methods in general, and HMMs in particular, came to dominate speech recognition research and development. Recent years have been characterized by incremental progress, larger data sets and models, and more rigorous competitions on more realistic speech tasks. Some researchers have explored the possibility of using DBNs instead of HMMs for speech, with the aim of using the greater expressive power of DBNs to capture more of the complex hidden state of the speech apparatus (Zweig and Russell, 1998; Richardson et al., 2000).

Several good textbooks on speech recognition are available (Rabiner and Juang, 1993; Jelinek, 1997; Gold and Morgan, 2000; Huang et al., 2001). Waibel and Lee (1990) collect important papers in the area, including some tutorial ones. The presentation in this chapter drew on the survey by Kay, Gawron, and Norvig (1994) and on the textbook by Jurafsky and Martin (2000). Speech recognition research is published in Computer Speech and Language, Speech Communications, and the IEEE Transactions on Acoustics, Speech, and Signal Processing and at the DARPA Workshops on Speech and Natural Language Processing and the Eurospeech, ICSLP, and ASRU conferences.

8 The second-ranked system in the competition, HEARS AY-II (Erman ef al., 1980), had a great deal of influence on other branches of Al research because of its use of the blackboard architecture. It was a rule-based expert system with a number of more or less independent, modular knowledge sources that communicated via a common blackboard from which they could write and read. Blackboard systems are the foundation of modern user interface architectures.

Section 15.7.

Summary 581

EXERCISES

15.1 Show that any second-order Markov process can be rewritten as a first-order Markov process with an augmented set of state variables. Can this always be done parsimoniously; that is, without increasing the number of parameters needed to specify the transition model?

15.2 In this exercise, we examine what happens to the probabilities in the umbrella world in the limit of long time sequences.

a. Suppose we observe an unending sequence of days on which the umbrella appears. Show that, as the days go by, the probability of rain on the current day increases monotonically towards a fixed point. Calculate this fixed point.

b. Now consider forecasting further and further into the future, given just the first two umbrella observations. First, compute the probability P(Ra|U1, Us) fork=1...20 and plot the results. You should see that the probability converges towards a fixed point. Calculate the exact value of this fixed point.

15.3 This exercise develops a space-efficient variant of the forward-backward algorithm described in Figure 15.4. We wish to compute P(Xy|e;.t) for k=1,... t. This will be done with a divide-and-conquer approach.

a. Suppose, for simplicity, that ¢ is odd, and let the halfway point be h = (¢ + 1)/2. Show that P(Xj|e;.+) can be computed for k=1,..., h given just the initial forward message f1.0, the backward message by, 1+, and the evidence e;.p.

b. Show a similar result for the second half of the sequence.

c¢. Given the results of (a) and (b), a recursive divide-and-conquer algorithm can be constructed by first running forward along the sequence and then backwards from the end, storing just the required messages at the middle and the ends. Then the algorithm is called on each half. Write out the algorithm in detail.

d. Compute the time and space complexity of the algorithm as a function of ¢, the length of the sequence. How does this change if we divide the input into more than two pieces?

15.4 On page 547, we outlined a flawed procedure for finding the most likely state sequence, given an observation sequence. The procedure involves finding the most likely state at each time step, using smoothing, and returning the sequence composed of these states. Show that, for some temporal probability models and observation sequences, this procedure returns an impossible state sequence (i.e., the posterior probability of the sequence is zero).

15.5 Often, we wish to monitor a continuous-state system whose behavior switches unpredictably among a set of k distinct “modes.” For example, an aircraft trying to evade a missile can execute a series of distinct maneuvers that the missile may attempt to track. A Bayesian network representation of such a switching Kalman filter model is shown in Figure 15.22.

582

Chapter 15. Probabilistic Reasoning over Time

[]

Figure 15.22 A Bayesian network representation of a switching Kalman filter. The switching variable S; is a discrete state variable whose value determines the transition model for the continuous state variables X;. For any discrete state £, the transition model P(X; 1|X¢, 5; =1) is a linear Gaussian model, just as in a regular Kalman filter. The transition model for the discrete state, P(S¢1+1|S5¢), can be thought of as a matrix, as in a hidden Markov model.

a. Suppose that the discrete state Sy has k possible values and that the prior continuous state estimate P(Xp) is a multivariate Gaussian distribution. Show that the prediction P(X) is a mixture of Gaussians—that is, a weighted sum of Gaussians such that the weights sum to 1.

b. Show that if the current continuous state estimate P(X¢|e;.;) is a mixture of m Gaussians, then in the general case the updated state estimate P(X; 1/e1.¢+1) will be a mixture of km Gaussians.

¢. What aspect of the temporal process do the weights in the Gaussian mixture represent?

Together, the results in (a) and (b) show that the representation of the posterior grows without limit even for switching Kalman filters, which are the simplest hybrid dynamic models.

15.6 Complete the missing step in the derivation of Equation (15.17), the first update step for the one-dimensional Kalman filter.

15.7 Let us examine the behavior of the variance update in Equation (15.18).

a. Plot the value of 7 as a function of ¢, given various values for 02 and 02.

b. Show that the update has a fixed point 02 such that 0? — 02 as t — oo, and calculate the value of 02.

c. Give a qualitative explanation for what happens as 02 — 0 and as 02 — 0.

15.8 Show how to represent an HMM as a recursive relational probabilistic model, as suggested in Section 14.6.

15.9 In this exercise, we analyze in more detail the persistent-failure model for the battery sensor in Figure 15.13(a).

a. Figure 15.13(b) stops at t = 32. Describe qualitatively what should happen as ¢t — oo if the sensor continues to read 0.

Section 15.7.

Summary 583

b. Suppose that the external temperature affects the battery sensor in such a way that transient failures become more likely as temperature increases. Show how to augment the DBN structure in Figure 15.13(a), and explain any required changes to the CPTs.

c. Given the new network structure, can battery readings be used by the robot to infer the current temperature?

15.10 Consider applying the variable elimination algorithm to the umbrella DBN unrolled for three slices, where the query is P(R3|U1, Us, Us). Show that the complexity of the algorithm—the size of the largest factor—is the same, regardless of whether the rain variables are eliminated in forward or backward order.

15.11 The model of “tomato” in Figure 15.19 allows for a coarticulation on the first vowel by giving two possible phones. An alternative approach is to use a triphone model in which the [ow(t,m)] phone automatically includes the change in vowel sound. Draw a complete triphone model for “tomato,” including the dialect variation.

15.12 Calculate the most probable path through the HMM in Figure 15.20 for the output sequence [C1, Cy, C3, Cy, Cy, Cg, C7]. Also give its probability.

] 6 MAKING SIMPLE DECISIONS

In which we see how an agent should make decisions so that it gets what it wants— on average, at least.

In this chapter, we return to the idea of utility theory that was introduced in Chapter 13 and show how it is combined with probability theory to yield a decision-theoretic agent—an agent that can make rational decisions based on what it believes and what it wants. Such an agent can make decisions in contexts where uncertainty and conflicting goals leave a logical agent with no way to decide. In effect, a goal-based agent has a binary distinction between good (goal) and bad (non-goal) states, while a decision-theoretic agent has a continuous measure of state quality.

Section 16.1 introduces the basic principle of decision theory: the maximization of expected utility. Section 16.2 shows that the behavior of any rational agent can be captured by supposing a utility function that is being maximized. Section 16.3 discusses the nature of utility functions in more detail, and in particular their relation to individual quantities such as money. Section 16.4 shows how to handle utility functions that depend on several quantities. In Section 16.5, we describe the implementation of decision-making systems. In particular, we introduce a formalism called decision networks (also known as influence diagrams) that extends Bayesian networks by incorporating actions and utilities. The remainder of the chapter discusses issues that arise in applications of decision theory to expert systems.

16.1 COMBINING BELIEFS AND DESIRES UNDER UNCERTAINTY

In the Port-Royal Logic, written in 1662, the French philosopher Arnauld stated

To judge what one must do to obtain a good or avoid an evil, it is necessary to consider not only the good and the evil in itself, but also the probability that it happens or does not happen; and to view geometrically the proportion that all these things have together.

Modern texts talk of utility rather than good and evil, but the principle is exactly the same. An agent’s preferences between world states are captured by a utility function, which assigns

584

Section 16.1.

Combining Beliefs and Desires under Uncertainty 585

EXPECTED UTILITY

MAXIMUM EXPECTED UTILITY

a single number to express the desirability of a state. Utilities are combined with outcome probabilities of actions to give an expected utility for each action.

We will use the notation U(.S) to denote the utility of state S according to the agent that is making the decisions. For now, we will consider states as complete snapshots of the world, similar to the situations of Chapter 10. This will simplify our initial discussions, but it can become rather cumbersome to specify the utility of each possible state separately. In Section 16.4, we will see how states can be decomposed under some circumstances for the purpose of assigning utilities.

A nondeterministic action A will have possible outcome states Result;(A), where the index ¢ ranges over the different outcomes. Prior to the execution of A, the agent assigns probability P(Result;(A)|Do(A), E) to each outcome, where F summarizes the agent’s available evidence about the world and Do( A) is the proposition that action A is executed in the current state. Then we can calculate the expected utility of the action given the evidence, EU (A|E), using the following formula:

EU(A|E) = >" P(Result;(A)|Do(A), E) U(Result;(A)) . (16.1)

The principle of maximum expected utility (MEU) says that a rational agent should choose an action that maximizes the agent’s expected utility. If we wanted to choose the best sequence of actions using this equation we would have to enumerate all action sequences and choose the best; this is clearly infeasible for long sequences. Therefore, this chapter will focus on simple decisions (usually a single action) and the next chapter will introduce new techniques for efficiently dealing with action sequences.

In a sense, the MEU principle could be seen as defining all of Al. All an intelligent agent has to do is calculate the various quantities, maximize utility over its actions, and away it goes. But this does not mean that the Al problem is solved by the definition!

Although the MEU principle defines the right action to take in any decision problem, the computations involved can be prohibitive, and it is sometimes difficult even to formulate the problem completely. Knowing the initial state of the world requires perception, learning, knowledge representation, and inference. Computing P(Result;(A)|Do(A), E) requires a complete causal model of the world and, as we saw in Chapter 14, NP-hard inference in Bayesian networks. Computing the utility of each state, U(Result;(A)), often requires searching or planning, because an agent does not know how good a state is until it knows where it can get to from that state. So, decision theory is not a panacea that solves the Al problem. On the other hand, it does provide a framework into which we can see where all the components of an Al system fit.

The MEU principle has a clear relation to the idea of performance measures introduced in Chapter 2. The basic idea is very simple. Consider the environments that could lead to an agent having a given percept history, and consider the different agents that we could design. If an agent maximizes a utility function that correctly reflects the performance measure by which its behavior is being judged, then it will achieve the highest possible performance score if we average over the environments in which the agent could be placed. This is the central justification for the MEU principle itself. While the claim may seem tautological, it does in fact embody a very important transition from a global, external criterion of rationality—the

586 Chapter 16. Making Simple Decisions performance measure over environment histories—to a local, internal criterion involving the maximization of a utility function applied to the next state.

ONE.SHOT In this chapter, we will be concerned only with single or one-shot decisions, whereas

DECISIONS p y g

Chapter 2 defined performance measures over environment histories, which usually involve many decisions. In the next chapter, which covers the case of sequential decisions, we will show how these two views can be reconciled.

LOTTERIES

Intuitively, the principle of Maximum Expected Utility (MEU) seems like a reasonable way to make decisions, but it is by no means obvious that it is the only rational way. After all, why should maximizing the average utility be so special? Why not try to maximize the sum of the cubes of the possible utilities, or try to minimize the worst possible loss? Also, couldn’t an agent act rationally just by expressing preferences between states, without giving them numeric values? Finally, why should a utility function with the required properties exist at all? Perhaps a rational agent can have a preference structure that is too complex to be captured by something as simple as a single real number for each state.

Constraints on rational preferences

These questions can be answered by writing down some constraints on the preferences that a rational agent should have and then showing that the MEU principle can be derived from the constraints. We use the following notation to describe an agent’s preferences:

A> B Ais preferred to B. A ~ B the agent is indifferent between A and B. AZ B the agent prefers A to B or is indifferent between them.

Now the obvious question is, what sorts of things are A and B? If the agent’s actions are deterministic, then A and B will typically be the concrete, fully specified outcome states of those actions. In the more general, nondeterministic case, A and B will be lotteries. A lottery is essentially a probability distribution over a set of actual outcomes (the “prizes” of the lottery). A lottery I with possible outcomes C1, .. ., C), that can occur with probabilities Ply... , Pp 1s Written

L = [p1, Ch; pa, Co; cee Pn, Cr) .

(A lottery with only one outcome can be written either as A or as [1, A].) In general, each outcome of a lottery can be either an atomic state or another lottery. The primary issue for utility theory is to understand how preferences between complex lotteries are related to preferences between the underlying states in those lotteries.

To do this, we impose reasonable constraints on the preference relation, much as we imposed rationality constraints on degrees of belief in order to obtain the axioms of probability in Chapter 13. One reasonable constraint is that preference should be transitive: that is, if A > Band B > C, then we would expect that A >= C. We argue for transitivity by showing

Section 16.2. The Basis of Utility Theory 587

that an agent whose preferences do not respect transitivity would behave irrationally. Suppose, for example, that an agent has the nontransitive preferences A = B = C = A, where A, B, and C are goods that can be freely exchanged. If the agent currently has A, then we could offer to trade C for A and some cash. The agent prefers C, and so would be willing to give up some amount of cash to make this trade. We could then offer to trade B for C, extracting more cash, and finally trade A for B. This brings us back where we started from, except that the agent has less money (Figure 16.1(a)). We can keep going around the cycle until the agent has no money at all. Clearly, this case the agent has not acted rationally.

[]

A A B 1¢ 1 ¢ (1-p) (1-¢g) Cc is equivalent to A B C ~~ ’ B l¢ (1-p)(1-g) (@ (b) Figure 16.1 (a) A cycle of exchanges showing that the nontransitive preferences A > B > C > A result in irrational behavior. (b) The decomposability axiom.

The following six constraints are known as the axioms of utility theory. They specify the most obvious semantic constraints on preferences and lotteries.

ORDERABILITY $ Orderability: Given any two states, a rational agent must either prefer one to the other or else rate the two as equally preferable. That is, the agent cannot avoid deciding. As we said on page 474, refusing to bet is like refusing to allow time to pass.

(A-B)V(B>=A)V(A~B). TRANSITIVITY <& Transitivity: Given any three states, if an agent prefers A to B and prefers B to C, then the agent must prefer A to C. (A-BYA(B>C) = (A> CO). CONTINUITY ¢& Continuity: If some state B is between A and C in preference, then there is some

probability p for which the rational agent will be indifferent between getting B for sure and the lottery that yields A with probability p and C with probability 1 — p.

AB» C = 3p pA; 1—p,C]~B.

588 Chapter 16. Making Simple Decisions

SUBSTITUTABILITY {> Substitutability: If an agent is indifferent between two lotteries, A and B, then the agent is indifferent between two more complex lotteries that are the same except that B is substituted for A in one of them. This holds regardless of the probabilities and the other outcome(s) in the lotteries.

A~B = [p,4;1-p,C| ~[p,B;1-p,C].

MONOTONICITY {> Monotonicity: Suppose there are two lotteries that have the same two outcomes, A and B. If an agent prefers A to B, then the agent must prefer the lottery that has a higher probability for A (and vice versa).

AB = (p>q © [pA 1-p, BZ [q,4; 1-¢,B)).

DECOMPOSABILITY {> Decomposability: Compound lotteries can be reduced to simpler ones using the laws of probability. This has been called the “no fun in gambling” rule because it says that two consecutive lotteries can be compressed into a single equivalent lottery, as shown in Figure 16.1(b).!

lp, A; 1 p,q, B; 1—¢,Cll ~ [p, 4; (1 -p)g, B; (1-p)(1—0q),C]. And then there was Utility

Notice that the axioms of utility theory do not say anything about utility: They talk only about preferences. Preference is assumed to be a basic property of rational agents. The existence of a utility function follows from the axioms of utility:

1. Utility principle If an agent’s preferences obey the axioms of utility, then there exists a real-valued function U that operates on states such that U(A) > U(B) if and only if A is preferred to B, and U(A) = U(B) if and only if the agent is indifferent between A and B. UA) >U(B) & A» B; UA) =U(B) & A~B. 2. Maximum Expected Utility principle The utility of a lottery is the sum of the probability of each outcome times the utility of that outcome.

U([p1, S153 Pn, Sl) = >\_piU(S) :

In other words, once the probabilities and utilities of the possible outcome states are specified, the utility of a compound lottery involving those states is completely determined. Because the outcome of a nondeterministic action is a lottery, this gives us the MEU decision rule from Equation (16.1).

It is important to remember that the existence of a utility function that describes an agent’s preference behavior does not necessarily mean that the agent is explicitly maximizing that utility function in its own deliberations. As we showed in Chapter 2, rational behavior can be generated in any number of ways, some of which are more efficient than explicit

1 We can account for the enjoyment of gambling by encoding gambling events into the state description; for example, “Have $10 and gambled” could be preferred to “Have $10 and didn’t gamble.”

Section 16.3. Utility Functions 589

utility maximization. By observing a rational agent’s preferences, however, it is possible to construct the utility function that represents what it is that the agent’s actions are actually trying to achieve.

16.3 UTILITY FUNCTIONS

Utility is a function that maps from states to real numbers. Is that all we can say about utility functions? Strictly speaking, that is it. Beyond the constraints listed earlier, an agent can have any preferences it likes. For example, an agent might prefer to have a prime number of dollars in its bank account; in which case, if it had $16 it would give away $3. It might prefer a dented 1973 Ford Pinto to a shiny new Mercedes. Preferences can also interact: for example, it might only prefer prime numbers of dollars when it owns the Pinto, but when it owns the Mercedes, it might prefer more dollars to less.

If all utility functions were as arbitrary as this, however, then utility theory would not be of much help because we would have to observe the agent’s preferences in every possible combination of circumstances before being able to make any predictions about its behavior. Fortunately, the preferences of real agents are usually more systematic. Conversely, there are systematic ways of designing utility functions that, when installed in an artificial agent, cause it to generate the kinds of behavior we want.

The utility of money

Utility theory has its roots in economics, and economics provides one obvious candidate for a utility measure: money (or more specifically, an agent’s total net assets). The almost universal exchangeability of money for all kinds of goods and services suggests that money plays a significant role in human utility functions. (In fact, most people think of economics as the study of money, when actually the root of the word economy refers to management, and its current emphasis is on the management of choice.)

If we restrict our attention to actions that only affect the amount of money that an agent has, then it will usually be the case that the agent prefers more money to less, all other things

TONG being equal. We say that the agent exhibits a monotonic preference for definite amounts of money. This is not, however, sufficient to guarantee that money behaves as a utility function, because it says nothing about preferences between lotteries involving money.

Suppose you have triumphed over the other competitors in a television game show. The host now offers you a choice: either you can take the $1,000,000 prize or you can gamble it on the flip of a coin. If the coin comes up heads, you end up with nothing, but if it comes up tails, you get $3,000,000. If you're like most people, you would decline the gamble and pocket the million. Are you being irrational?

ED VALUE Assuming you believe that the coin is fair, the expected monetary value (EMV) of the gamble is ($0) + ($3,000,000) = $1,500,000, and the EMV of taking the original prize is of course $1,000,000, which is less. But that does not necessarily mean that accepting the gamble is a better decision. Suppose we use 5, to denote the state of possessing total

590

Chapter 16. Making Simple Decisions

wealth $n, and that your current wealth is $k. Then the expected utilities of the two actions of accepting and declining the gamble are

EU (Accept) = 3U(Sk) + 5U(Sk+3,000,000) » EU(Decline) = U(Sk+1,000,000)

To determine what to do, we need to assign utilities to the outcome states. Utility is not directly proportional to monetary value, because the utility—the positive change in lifestyle— for your first million is very high (or so we are told), whereas the utility for an additional million is much smaller. Suppose you assign a utility of 5 to your current financial status (Sk), a 10 to the state Si 3,000,000, and an 8 to the state Sk+1,000,000Then the rational action would be to decline, because the expected utility of accepting is only 7.5 (less than the 8 for declining). On the other hand, suppose that you happen to have $500,000,000 in the bank already (and appear on game shows just for fun, one assumes). In this case, the gamble is probably acceptable, because the additional benefit of the 503rd million is probably about the same as that of the 501st million.

In a pioneering study of actual utility functions, Grayson (1960) found that the utility of money was almost exactly proportional to the logarithm of the amount. (This idea was first suggested by Bernoulli (1738); see Exercise 16.3.) One particular curve, for a certain Mr. Beard, is shown in Figure 16.2(a). The data obtained for Mr. Beard’s preferences are consistent with a utility function

U(Skin) = —263.31 + 22.09 log(n + 150, 000)

for the range between n = —$150, 000 and n = $800, 000.

We should not assume that this is the definitive utility function for monetary value, but it is likely that most people have a utility function that is concave for positive wealth. Going into debt is usually considered disastrous, but preferences between different levels of debt can display a reversal of the concavity associated with positive wealth. For example,

U U

~~

= 4 T

150,000 800,000

(a) (b

Figure 16.2 The utility of money. (a) Empirical data for Mr. Beard over a limited range. (b) A typical curve for the full range.

Utility Functions 591

RISK-AVERSE

RISK-SEEKING

CERTAINTY EQUIVALENT

INSURANCE PREMIUM

RISK-NEUTRAL

VALUE FUNCTION ORDINAL UTILITY FUNCTION

someone already $10,000,000 in debt might well accept a gamble on a fair coin with a gain of $10,000,000 for heads and a loss of $20,000,000 for tails.> This yields the S-shaped curve shown in Figure 16.2(b).

If we restrict our attention to the positive part of the curves, where the slope is decreasing, then for any lottery L, the utility of being faced with that lottery is less than the utility of being handed the expected monetary value of the lottery as a sure thing:

U(L)

That is, agents with curves of this shape are risk-averse: they prefer a sure thing with a payoff that is less than the expected monetary value of a gamble. On the other hand, in the “desperate” region at large negative wealth in Figure 16.2(b), the behavior is risk-seeking. The value an agent will accept in lieu of a lottery is called the certainty equivalent of the lottery. Studies have shown that most people will accept about $400 in lieu of a gamble that gives $1000 half the time and $0 the other half—that is, the certainty equivalent of the lottery is $400. The difference between the expected monetary value of a lottery and its certainty equivalent is called the insurance premium. Risk aversion is the basis for the insurance industry, because it means that insurance premiums are positive. People would rather pay a small insurance premium than gamble the price of their house against the chance of a fire. From the insurance company’s point of view, the price of the house is very small compared with the firm’s total reserves. This means that the insurer’s utility curve is approximately linear over such a small region, and the gamble costs the company almost nothing.

Notice that for small changes in wealth relative to the current wealth, almost any curve will be approximately linear. An agent that has a linear curve is said to be risk-neutral. For gambles with small sums, therefore, we expect risk neutrality. In a sense, this justifies the simplified procedure that proposed small gambles to assess probabilities and to justify the axioms of probability in Chapter 13.

Utility scales and utility assessment

The axioms of utility do not specify a unique utility function for an agent, given its preference behavior. For example, we can transform a utility function U (S) into

U'(S) =k1 + kU(S),

where kj is a constant and kg is any positive constant. Clearly, this linear transformation leaves the agent’s behavior unchanged.

In deterministic contexts, where there are states but no lotteries, behavior is unchanged by any monotonic transformation. For example, we can take the cube root of all the utilities without affecting the preference ordering on actions. An agent in a deterministic environment is said to have a value function or ordinal utility function; the function really provides just rankings of states rather than meaningful numerical values. We saw this distinction in Chapter 6 for games: evaluation functions in deterministic games such as chess are value functions, whereas evaluation functions in nondeterministic games like backgammon are true utility functions.

2 Such behavior might be called desperate, but it is rational if one is already in a desperate situation.

592 Chapter 16. Making Simple Decisions

HUMAN JUDGMENT AND FALLIBILITY

Decision theory is a normative theory: it describes how a rational agent should act. The application of economic theory would be greatly enhanced if it were also a descriptive theory of actual human decision making. However, there is experimental evidence indicating that people systematically violate the axioms of utility theory. An example is given by the psychologists Tversky and Kahneman (1982), based on an example by the economist Allais (1953). Subjects in this experiment are given a choice between lotteries A and B and then between C' and D:

A: 80% chance of $4000 C : 20% chance of $4000

B : 100% chance of $3000 D : 25% chance of $3000 .

The majority of subjects choose B over A and C over D. But if we assign U ($0) = 0, then the first of these choices implies that 0.80 ($4000) < U($3000), whereas the second choice implies exactly the reverse. In other words, there seems to be no utility function that is consistent with these choices. One possible conclusion is that humans are simply irrational by the standards of our utility axioms. An alternative view is that the analysis does not take into account regret—the feeling that humans know they would experience if they gave up a certain reward (B) for an 80% chance at a higher reward and then lost. In other words, if A is chosen,

there is a 20% chance of getting no money and feeling like a complete idiot. Kahneman and Tversky go on to develop a descriptive theory that explains 22 ae how people are risk-averse with high-probability events, but are willing to take iE more risks with unlikely payoffs. The connection between this finding and Al is DR that the choices our agents can make are only as good as the preferences they are based on. If our human informants insist on contradictory preference judgments,

there is nothing our agent can do to be consistent with them.

Fortunately, preference judgments made by humans are often open to revision in the light of further consideration. In early work at Harvard Business School on assessing the utility of money, Keeney and Raiffa (1976, p. 210) found the following:

A great deal of empirical investigation has shown that there is a serious deficiency in the assessment protocol. Subjects tend to be too risk-averse in the small and therefore ... the fitted utility functions exhibit unacceptably large risk premiums for lotteries with a large spread. . . . Most of the subjects, however, can reconcile their inconsistencies and feel that they have learned an important lesson about how they want to behave. As a consequence, some subjects cancel their automobile collision insurance and take out more term insurance on their lives.

Even today, human (ir)rationality is the subject of intensive investigation.

Section 16.4.

Multiattribute Utility Functions 593

NORMALIZED UTILITIES

STANDARD LOTTERY

MICROMORT QALY

One procedure for assessing utilities is to establish a scale with a “best possible prize” at U(S) = ur and a “worst possible catastrophe” at U(S) = u,. Normalized utilities use a scale with uv; = 0 and ut = 1. Utilities of intermediate outcomes are assessed by asking the agent to indicate a preference between the given outcome state .S and a standard lottery [p, ut; (1 =p), uy]. The probability p is adjusted until the agent is indifferent between .S and the standard lottery. Assuming normalized utilities, the utility of .S is given by p.

In medical, transportation, and environmental decision problems, among others, people’s lives are at stake. In such cases, u is the value assigned to immediate death (or perhaps many deaths). Although nobody feels comfortable with putting a value on human life, it is a fact that tradeoffs are made all the time. Aircraft are given a complete overhaul at intervals determined by trips and miles flown, rather than after every trip. Car bodies are made with relatively thin sheet metal to reduce costs, despite the decrease in accident survival rates. Leaded fuel is still widely used even though it has known health hazards. Paradoxically, a refusal to “put a monetary value on life” means that life is often undervalued. Ross Shachter relates an experience with a government agency that commissioned a study on removing asbestos from schools. The study assumed a particular dollar value for the life of a school-age child, and argued that the rational choice under that assumption was to remove the asbestos. The government agency, morally outraged, rejected the report out of hand. It then decided against asbestos removal.

Some attempts have been made to find out the value that people place on their own lives. Two common “currencies” used in medical and safety analysis are the micromort (a one in a million chance of death) and the QALY, or quality-adjusted life year (equivalent to a year in good health with no infirmities). A number of studies across a wide range of individuals have shown that a micromort is worth about $20 (1980 dollars). We have already seen that utility functions need not be linear, so this does not imply that a decision maker would kill himself for $20 million. Again, the local linearity of any utility curve means that micromort and QALY values are most appropriate for small incremental risks and rewards.

MULTIATTRIBUTE UTILITY THEORY

Decision making in the field of public policy involves both millions of dollars and life and death. For example, in deciding what levels of a carcinogenic substance to allow into the environment, policy makers must weigh the prevention of deaths against the economic hardship that might result from the elimination of certain products and processes. Siting a new airport requires consideration of the disruption caused by construction; the cost of land; the distance from centers of population; the noise of flight operations; safety issues arising from local topography and weather conditions; and so on. Problems like these, in which outcomes are characterized by two or more attributes, are handled by multiattribute utility theory. We will call the attributes X= X71, ..., X,; a complete vector of assignments will be x= {(z1,...,Zn). Each attribute is generally assumed to have discrete or continuous scalar values. For simplicity, we will assume that each attribute is defined in such a way that,

594 Chapter 16. Making Simple Decisions

all other things being equal, higher values of the attribute correspond to higher utilities. For example, if we choose Absence OfNoise as an attribute in the airport problem then the greater its value, the better the solution. In some cases, it may be necessary to subdivide the range of values so that utility varies monotonically within each range.

We begin by examining cases in which decisions can be made without combining the attribute values into a single utility value. Then we look at cases where the utilities of attribute combinations can be specified very concisely.

Dominance

Suppose that airport site S; costs less, generates less noise pollution, and is safer than site So.

staictoomnance One would not hesitate to reject So. We then say that there is strict dominance of S; over So. In general, if an option is of lower value on all attributes than some other option, it need not be considered further. Strict dominance is often very useful in narrowing down the field of choices to the real contenders, although it seldom yields a unique choice. Figure 16.3(a) shows a schematic diagram for the two-attribute case.

[]

(a) (b)

Figure 16.3 Strict dominance. (a) Deterministic: Option A is strictly dominated by B but not by C or D. (b) Uncertain: A is strictly dominated by B but not by C.

That is fine for the deterministic case, in which the attribute values are known for sure. What about the general case, where the action outcomes are uncertain? A direct analog of strict dominance can be constructed, where, despite the uncertainty, all possible concrete outcomes for SS; strictly dominate all possible outcomes for S52. (See Figure 16.3(b).) Of course, this will probably occur even less often than in the deterministic case.

SLoCHASTIC Fortunately, there is a more useful generalization called stochastic dominance, which occurs very frequently in real problems. Stochastic dominance is easiest to understand in the context of a single attribute. Suppose we believe that the cost of siting the airport at Sy is uniformly distributed between $2.8 billion and $4.8 billion and that the cost at S53 is uniformly distributed between $3 billion and $5.2 billion. Figure 16.4(a) shows these distributions, with cost plotted as a negative value. Then, given only the information that utility decreases with cost, we can say that S7 stochastically dominates Ss (i.e., S2 can be discarded). It is important

Section 16.4. Multiattribute Utility Functions 595

0.6 1.2 0.5 1 — 5 04 i 5 08 = = S, 203 s, | s 2 0.6 2 ti 8 s & ! & 1 0.2 04 0.1 ‘ 0.2 0 A———t——— r Ll 0 + Ll — 6 -55 5 45 4 -35 -3 25 -2 6 -55 -5 45 4 35 -3 25 -2 Negative cost Negative cost (a) (b) Figure 16.4 Stochastic dominance. (a) S; stochastically dominates Sy on cost. (b) Cumulative distributions for the negative cost of Sy and Ss.

to note that this does not follow from comparing the expected costs. For example, if we knew the cost of S7 to be exactly $3.8 billion, then we would be unable to make a decision without additional information on the utility of money.’

The exact relationship between the attribute distributions needed to establish stochastic dominance is best seen by examining the cumulative distributions, shown in Figure 16.4(b). The cumulative distribution measures the probability that the cost is less than or equal to any given amount—that is, it integrates the original distribution. If the cumulative distribution for 51 is always to the right of the cumulative distribution for Sa, then, stochastically speaking, S, is cheaper than S5. Formally, if two actions A; and As lead to probability distributions p1(z) and pa(x) on attribute X, then A; stochastically dominates A on X if

Va [ mt) dz’ < | pa) dz’.

The relevance of this definition to the selection of optimal decisions comes from the following property: if Ay stochastically dominates As, then for any monotonically nondecreasing utility function U(x), the expected utility of A; is at least as high as the expected utility of As. Hence, if an action is stochastically dominated by another action on all attributes, then it can be discarded.

The stochastic dominance condition might seem rather technical and perhaps not so easy to evaluate without extensive probability calculations. In fact, it can be decided very easily in many cases. Suppose, for example, that the construction cost depends on the distance to centers of population. The cost itself is uncertain, but the greater the distance, the greater the cost. If Sj is less remote than So, then S; will dominate S5 on cost. Although we will not

3 Tt might seem odd that more information on the cost of S; could make the agent less able to decide. The paradox is resolved by noting that the decision reached in the absence of exact cost information is less likely to have the highest utility payoff. :

596

Chapter 16. Making Simple Decisions

QUALITATIVE PROBABILISTIC NETWORKS

REPRESENTATION THEOREMS

PREFERENCE INDEPENDENCE

MUTUAL PREFERENTIAL INDEPENDENCE

present them here, there exist algorithms for propagating this kind of qualitative information among uncertain variables in qualitative probabilistic networks, enabling a system to make rational decisions based on stochastic dominance, without using any numeric values.

Preference structure and multiattribute utility

Suppose we have n attributes, each of which has d distinct possible values. To specify the complete utility function U(x, ..., Zn), we need d" values in the worst case. Now, the worst case corresponds to a situation in which the agent’s preferences have no regularity at all. Multiattribute utility theory is based on the supposition that the preferences of typical agents have much more structure than that. The basic approach is to identify regularities in the preference behavior we would expect to see and to use what are called representation theorems to show that an agent with a certain kind of preference structure has a utility function

U(zxy,cy Zn) = flfi(z1),RO a ']

where f is, we hope, a simple function such as addition. Notice the similarity to the use of Bayesian networks to decompose the joint probability of several random variables.

Preferences without uncertainty

Let us begin with the deterministic case. Remember that for deterministic environments the agent has a value function V(x1,...,,); the aim is to represent this function concisely. The basic regularity that arises in deterministic preference structures is called preference independence. Two attributes X; and X; are preferentially independent of a third attribute Xj if the preference between outcomes (x1, 2, x3) and (x, x5, x3) does not depend on the particular value x3 for attribute X35.

Going back to the airport example, where we have (among other attributes) Noise, Cost, and Deaths to consider, one may propose that Noise and Cost are preferentially independent of Deaths. For example, if we prefer a state with 20,000 people residing in the flight path and a construction cost of $4 billion to a state with 70,000 people residing in the flight path and a cost of $3.7 billion when the safety level is 0.06 deaths per million passenger miles in both cases, then we would have the same preference when the safety level is 0.13 or 0.01; and the same independence would hold for preferences between any other pair of values for Noise and Cost. It is also apparent that Cost and Deaths are preferentially independent of Noise and that Noise and Deaths are preferentially independent of Cost. We say that the set of attributes { Noise, Cost, Deaths} exhibits mutual preferential independence (MPI). MPI says that, whereas each attribute may be important, it does not affect the way in which one trades off the other attributes against each other.

Mutual preferential independence is something of a mouthful, but thanks to a remarkable theorem due to the economist Debreu (1960), we can derive from it a very simple form for the agent’s value function: If attributes Xi, ..., X, are mutually preferentially independent, then the agent’s preference behavior can be described as maximizing the function

V(zi,-.. Tn) = PIRACY ,

Decision Networks 597

ADDITIVE VALUE FUNCTION

UTILITY INDEPENDENCE

MUTUALLY UTILITYINDEPENDENT

MULTIPLICATIVE UTILITY FUNCTION

where each V; is a value function referring only to the attribute X;. For example, it might well be the case that the airport decision can be made using a value function

V (noise, cost, deaths) = —noise x 10\* — cost — deaths x 10'?

A value function of this type is called an additive value function. Additive functions are an extremely natural way to describe an agent’s value function and are valid in many real-world situations. Even when MPI does not strictly hold, as might be the case at extreme values of the attributes, an additive value function might still provide a good approximation to the agent’s preferences. This is especially true when the violations of MPI occur in portions of the attribute ranges that are unlikely to occur in practice.

Preferences with uncertainty

When uncertainty is present in the domain, we will also need to consider the structure of preferences between lotteries and to understand the resulting properties of utility functions, rather than just value functions. The mathematics of this problem can become quite complicated, so we will present just one of the main results to give a flavor of what can be done. The reader is referred to Keeney and Raiffa (1976) for a thorough survey of the field.

The basic notion of utility independence extends preference independence to cover lotteries: a set of attributes X is utility-independent of a set of attributes Y if preferences between lotteries on the attributes in X are independent of the particular values of the attributes in Y. A set of attributes is mutually utility-independent (MUI) if each of its subsets is utility-independent of the remaining attributes. Again, it seems reasonable to propose that the airport attributes are MUI.

MUI implies that the agent’s behavior can be described using a multiplicative utility function (Keeney, 1974). The general form of a multiplicative utility function is best seen by looking at the case for three attributes. For conciseness, we will use U; to mean U; (z;):

U = kiUy + koUa + ksUs + k1koU Ua + koksUaUs + kk UsUy + k1koksU UUs .

Although this does not look very simple, it contains just three single-attribute utility functions and three constants. In general, an n-attribute problem exhibiting MUI can be modeled using n single-attribute utilities and n constants. Each of the single-attribute utility functions can be developed independently of the other attributes, and this combination will be guaranteed to generate the correct overall preferences. Additional assumptions are required to obtain a purely additive utility function.

INFLUENCE DIAGRAM DECISION NETWORK

In this section, we will look at a general mechanism for making rational decisions. The notation is often called an influence diagram (Howard and Matheson, 1984), but we will use the more descriptive term decision network. Decision networks combine Bayesian networks with additional node types for actions and utilities. We will use airport siting as an example.

598

Chapter 16. Making Simple Decisions

CHANCE NODES

UTILITY NODES

Figure 16.5 A simple decision network for the airport-siting problem.

agent could be uncertain about the construction cost, the level of air traffic and the potential for litigation, and the Deaths, Noise, and total Cost variables, each of which also depends on the site chosen. Each chance node has associated with it a conditional distribution that is indexed by the state of the parent nodes. In decision networks, the parent nodes can include decision nodes as well as chance nodes. Note that each of the current-state chance nodes could be part of a large Bayes net for assessing construction costs, air traffic levels, or litigation potentials.

actions. In this case, the AirportSite action can take on a different value for each site under consideration. The choice influences the cost, safety, and noise that will result. In this chapter, we will assume that we are dealing with a single decision node. Chapter 17 deals with cases where more than one decision must be made.

as parents all variables describing the outcome that directly affect utility. Associated with the utility node is a description of the agent’s utility as a function of the parent attributes. The description could be just a tabulation of the function, or it might be a parameterized additive or multilinear function.

Section 16.5.

Decision Networks 599

ACTION-UTILITY TABLES

A simplified form is also used in many cases. The notation remains identical, but the chance nodes describing the outcome state are omitted. Instead, the utility node is connected directly to the current-state nodes and the decision node. In this case, rather than representing a utility function on states, the utility node represents the expected utility associated with each action, as defined in Equation (16.1). We therefore call such tables action-utility tables. Figure 16.6 shows the action-utility representation of the airport problem.

Airport Site

Figure 16.6 A simplified representation of the airport-siting problem. Chance nodes corresponding to outcome states have been factored out.

Notice that, because the Noise, Deaths, and Cost chance nodes in Figure 16.5 refer to future states, they can never have their values set as evidence variables. Thus, the simplified version that omits these nodes can be used whenever the more general form can be used. Although the simplified form contains fewer nodes, the omission of an explicit description of the outcome of the siting decision means that it is less flexible with respect to changes in circumstances. For example, in Figure 16.5, a change in aircraft noise levels can be reflected by a change in the conditional probability table associated with the Noise node, whereas a change in the weight accorded to noise pollution in the utility function can be reflected by a change in the utility table. In the action-utility diagram, Figure 16.6, on the other hand, all such changes have to be reflected by changes to the action-utility table. Essentially, the action-utility formulation is a compiled version of the original formulation.

Evaluating decision networks

Actions are selected by evaluating the decision network for each possible setting of the decision node. Once the decision node is set, it behaves exactly like a chance node that has been set as an evidence variable. The algorithm for evaluating decision networks is the following: 1. Set the evidence variables for the current state. 2. For each possible value of the decision node;

(a) Set the decision node to that value.

600

Chapter 16. Making Simple Decisions

(b) Calculate the posterior probabilities for the parent nodes of the utility node, using a standard probabilistic inference algorithm.

(c) Calculate the resulting utility for the action.

3. Return the action with the highest utility.

This is a straightforward extension of the Bayes net algorithm and can be incorporated directly into the agent design given in Figure 13.1. We will see in Chapter 17 that the possibility of executing several actions in sequence makes the problem much more interesting.

INFORMATION VALUE THEORY

In the preceding analysis, we have assumed that all relevant information, or at least all available information, is provided to the agent before it makes its decision. In practice, this is hardly ever the case. One of the most important parts of decision making is knowing what questions to ask. For example, a doctor cannot expect to be provided with the results of all possible diagnostic tests and questions at the time a patient first enters the consulting room. Tests are often expensive and sometimes hazardous (both directly and because of associated delays). Their importance depends on two factors: whether the test results would lead to a significantly better treatment plan, and how likely the various test results are.

This section describes information value theory, which enables an agent to choose what information to acquire. The acquisition of information is achieved by sensing actions, as described in Chapter 12. Because the agent’s utility function seldom refers to the contents of the agent’s internal state, whereas the whole purpose of sensing actions is to affect the internal state, we must evaluate sensing actions by their effect on the agent’s subsequent “real” actions. Thus, information value theory involves a form of sequential decision making.

A simple example

Suppose an oil company is hoping to buy one of n indistinguishable blocks of ocean drilling rights. Let us assume further that exactly one of the blocks contains oil worth C' dollars and that the price of each block is C/n dollars. If the company is risk-neutral, then it will be indifferent between buying a block and not buying one.

Now suppose that a seismologist offers the company the results of a survey of block number 3, which indicates definitively whether the block contains oil. How much should the company be willing to pay for the information? The way to answer this question is to examine what the company would do if it had the information:

e With probability 1/n, the survey will indicate oil in block 3. In this case, the company will buy block 3 for C'/n dollars and make a profit of C' — C/n = (n — 1)C/n dollars.

e With probability (n—1)/n, the survey will show that the block contains no oil, in which case the company will buy a different block. Now the probability of finding oil in one

5 In the United States, the only question that is always asked beforehand is whether the patient has insurance.

Section 16.6.

The Value of Information 601

VALUE OF PERFECT INFORMATION

of the other blocks changes from 1/n to 1/(n — 1), so the company makes an expected profitof C/(n — 1) — C/n = C/n(n — 1) dollars.

Now we can calculate the expected profit, given the survey information: 1 (n-1)C n-1 C — x + x n n n n(n —1)

=C/n.

Therefore, the company should be willing to pay the seismologist up to C/n dollars for the information: the information is worth as much as the block itself.

The value of information derives from the fact that with the information, one’s course of action can be changed to suit the actual situation. One can discriminate according to the situation, whereas without the information, one has to do what’s best on average over the possible situations. In general, the value of a given piece of information is defined to be the difference in expected value between best actions before and after information is obtained.

A general formula

It is simple to derive a general mathematical formula for the value of information. Usually, we assume that exact evidence is obtained about the value of some random variable E;, so the phrase value of perfect information (VPI) is used.’ Let the agent’s current knowledge be E. Then the value of the current best action « is defined by

[]

and the value of the new best action (after the new evidence E; is obtained) will be

EU(ag|E, Ej) = max» U(Resulti(A)) P(Result;(A)|Do(A), E, Ej) .

But EF; is a random variable whose value is currently unknown, so we must average over all possible values ej; that we might discover for Ej, using our current beliefs about its value. The value of discovering F;, given current information F, is then defined as

VPIg(E;) = (SPE =eimE ony B=) — BU(a|E) . k

In order to get some intuition for this formula, consider the simple case where there are only two actions, A; and Ag, from which to choose. Their current expected utilities are Uy and Us. The information E; will yield some new expected utilities U{ and Uj for the actions, but before we obtain E;, we will have some probability distributions over the possible values of U7 and Uy, (which we will assume are independent).

Suppose that A; and As represent two different routes through a mountain range in winter. A; is a nice, straight highway through a low pass, and As is a winding dirt road over the top. Just given this information, A; is clearly preferable, because it is quite likely that the second route is blocked by avalanches, whereas it is quite unlikely that the first route is blocked by traffic. Uj is therefore clearly higher than Uj. It is possible to obtain satellite

6 Imperfect information about a variable X can be modeled as perfect information about a variable Y that is probabilistically related to X. See Exercise 16.11 for an example of this.

602

Chapter 16. Making Simple Decisions

P(UIE) P(UIE) P(UIE)

U, U, U,U, U,U, (a) (b) (c)

Figure 16.7 Three generic cases for the value of information. In (a), A; will almost certainly remain superior to Ag, so the information is not needed. In (b), the choice is unclear and the information is crucial. In (c), the choice is unclear but because it makes little difference, the information is less valuable.

reports E; on the actual state of each road that would give new expectations, U] and Us, for the two crossings. The distributions for these expectations are shown in Figure 16.7(a). Obviously, in this case, it is not worth the expense of obtaining satellite reports, because it is unlikely that the information derived from them will change the plan. With no change, information has no value.

Now suppose that we are choosing between two different winding dirt roads of slightly different lengths and we are carrying a seriously injured passenger. Then, even when U; and Us are quite close, the distributions of U] and Uj are very broad. There is a significant possibility that the second route will turn out to be clear while the first is blocked, and in this case the difference in utilities will be very high. The VPI formula indicates that it might be worthwhile getting the satellite reports. Such a situation is shown in Figure 16.7(b).

Now suppose that we are choosing between the two dirt roads in summertime, when blockage by avalanches is unlikely. In this case, satellite reports might show one route to be more scenic than the other because of flowering alpine meadows, or perhaps wetter because of errant streams. It is therefore quite likely that we would change our plan if we had the information. But in this case, the difference in value between the two routes is still likely to be very small, so we will not bother to obtain the reports. This situation is shown in Figure 16.7(c).

In sum, information has value to the extent that it is likely to cause a change of plan and to the extent that the new plan will be significantly better than the old plan.

Properties of the value of information

One might ask whether it is possible for information to be deleterious: can it actually have negative expected value? Intuitively, one should expect this to be impossible. After all, one could in the worst case just ignore the information and pretend that one has never received it. This is confirmed by the following theorem, which applies to any decision-theoretic agent:

Section 16.6.

The Value of Information 603

The value of information is nonnegative: Vj, E VPIE(E;)>0.

The theorem follows directly from the definition of VPI, and we leave the proof as an exercise (Exercise 16.12). It is important to remember that VPI depends on the current state of information, which is why it is subscripted. It can change as more information is acquired. In the extreme case, it will become zero if the variable in question already has a known value. Thus, VPI is not additive. That is,

VPIE(E;, Ey) # VPIE(E;) + VPIg( Ey) (in general) . VPI is, however, order-independent, which should be intuitively obvious. That is, VPIg(Ej, Ex) = VPIg(Ej) + VPIgg, (Ex) = VPIg(Ey) + VPI E, (Ej)

Order independence distinguishes sensing actions from ordinary actions and simplifies the problem of calculating the value of a sequence of sensing actions.

Implementing an information-gathering agent

A sensible agent should ask questions of the user in a reasonable order, should avoid asking questions that are irrelevant, should take into account the importance of each piece of information in relation to its cost, and should stop asking questions when that is appropriate. All of these capabilities can be achieved by using the value of information as a guide.

Figure 16.8 shows the overall design of an agent that can gather information intelligently before acting. For now, we will assume that with each observable evidence variable E;, there is an associated cost, Cost([;), which reflects the cost of obtaining the evidence through tests, consultants, questions, or whatever. The agent requests what appears to be the most valuable piece of information, compared with its cost. We assume that the result of the action Request([;) is that the next percept provides the value of Ij. If no observation is worth its cost, the agent selects a “real” action.

The agent algorithm we have described implements a form of information gathering that is called myopic. This is because it uses the VPI formula shortsightedly, calculating

function INFORMATION-GATHERING-AGENT( percept) returns an action static: D, a decision network

integrate percept into D J «the value that maximizes VPI(E;) — Cost(E;) if VPI(E;) > Cost(E;) then return REQUEST(E;) else return the best action from D

Figure 16.8 Design of a simple information-gathering agent. The agent works by repeatedly selecting the observation with the highest information value, until the cost of the next observation is greater than its expected benefit.

604

Chapter 16. Making Simple Decisions

the value of information as if only a single evidence variable will be acquired. If there is no single evidence variable that will help a lot, a myopic agent might hastily take an action when it would have been better to request two or more variables first and then take action. Myopic control is based on the same heuristic idea as greedy search and often works well in practice. (For example, it has been shown to outperform expert physicians in selecting diagnostic tests.) However, a perfectly rational information-gathering agent should consider all possible sequences of information requests terminating in an external action and all possible outcomes of those requests. Because the value of the second request depends on the outcome of the first request, the agent needs to explore the space of conditional plans, as described in Chapter 12.

DECISION ANALYSIS

DECISION MAKER DECISION ANALYST

The field of decision analysis, which evolved in the 1950s and 1960s, studies the application of decision theory to actual decision problems. It is used to help make rational decisions in important domains where the stakes are high, such as business, government, law, military strategy, medical diagnosis and public health, engineering design, and resource management. The process involves a careful study of the possible actions and outcomes, as well as the preferences placed on each outcome. It is traditional in decision analysis to talk about two roles: the decision maker states preferences between outcomes, and the decision analyst enumerates the possible actions and outcomes and elicits preferences from the decision maker to determine the best course of action. Until the early 1980s, the main purpose of decision analysis was to help humans make decisions that actually reflect their own preferences. In the current day, more and more decision processes are automated, and decision analysis is used to make sure that the automated processes are behaving as desired.

As we discussed in Chapter 14, early expert system research concentrated on answering questions, rather than on making decisions. Those systems that did recommend actions rather than providing opinions on matters of fact generally did so using condition-action rules, rather than with explicit representations of outcomes and preferences. The emergence of Bayesian networks in the late 1980s made it possible to build large-scale systems that generated sound probabilistic inferences from evidence. The addition of decision networks means that expert systems can be developed that recommend optimal decisions, reflecting the preferences of the user as well as the available evidence.

A system that incorporates utilities can avoid one of the most common pitfalls associated with the consultation process: confusing likelihood and importance. A common strategy in early medical expert systems, for example, was to rank possible diagnoses in order of likelihood and report the most likely. Unfortunately, this can be disastrous! For the majority of patients in general practice, the two most likely diagnoses are usually “There’s nothing wrong with you” and “You have a bad cold,” but if the third-most-likely diagnosis for a given patient is lung cancer, that’s a serious matter. Obviously, a testing or treatment plan should depend both on probabilities and utilities.

Section 16.7.

Decision-Theoretic Expert Systems 605

AORTIC COARCTATION

GOLD STANDARD

We will now describe the knowledge engineering process for decision-theoretic expert systems. As an example we will consider the problem of selecting a medical treatment for a kind of congenital heart disease in children (see Lucas, 1996).

About 0.8% of children are born with a heart anomaly, the most common being aortic coarctation (a constriction of the aorta). It can be treated with surgery, angioplasty (expanding the aorta with a balloon placed inside the artery) or medication. The problem is to decide what treatment to use and when to do it: the younger the infant the greater the risks of certain treatments, but one mustn’t wait too long. A decision-theoretic expert system for this problem can be created by a team consisting of at least one domain expert (a pediatric cardiologist) and one knowledge engineer. The process can be broken down into the following steps (which you can compare to the steps in developing a logic-based system in Section 8.4).

Create a causal model. Determine what are the possible symptoms, disorders, treatments, and outcomes. Then draw arcs between them, indicating what disorders cause what symptoms, and what treatments alleviate what disorders. Some of this will be well known to the domain expert, and some will come from the literature. Often the model will match well with the informal graphical descriptions given in medical textbooks.

Simplify to a qualitative decision model. Since we are using the model to make treatment decisions and not for other purposes (such as determining the joint probability of certain symptom/disorder combinations), we can often simplify by removing variables that are not involved in treatment decisions. Sometimes variables will have to be split or joined to match the expert’s intuitions. For example, the original aortic coarctation model had a Treatment variable with values surgery, angioplasty and medication, and a separate variable for Timing of the treatment. But the expert had a hard time thinking of these separately, so they were combined, with Treatment taking on values such as surgery in 1 month. This gives us the model of Figure 16.9.

Assign probabilities. Probabilities can come from patient databases, literature studies, or the expert’s subjective assessments. In cases where the wrong kinds of probabilities are given in the literature, techniques such as Bayes’ rule and marginalization can be used to compute the desired probabilities. It has been found that experts are best able to assess the probability of an effect given a cause (e.g. P(dyspnoea|heartfailure)) rather than the other way around.

Assign utilities. When there are a small number of possible outcomes, they can be enumerated and evaluated individually. We would create a scale from best to worst outcome and give each a numeric value, for example -1000 for death and O for complete recovery. We would then place the other outcomes on this scale. This can be done by the expert, but it is better if the patient (or in the case of infants, the patient’s parents) can be involved, because different people have different preferences. If there are exponentially many outcomes, we need some way to combine them using multiattribute utility functions. For example, we may say that the negative utility of various complications is additive.

Verify and refine the model. To evaluate the system we will need a set of correct (input, output) pairs; a so-called gold standard to compare against. For medical expert systems this usually means assembling the best available doctors, presenting them with a few cases, and asking them for their diagnosis and recommended treatment plan. We then see

606

Chapter 16. Making Simple Decisions

[]

Figure 16.9 Influence diagram for aortic coarctation (courtesy of Peter Lucas).

how well the system matches their recommendations. If it does poorly, we try to isolate the parts that are going wrong and fix them. It can be useful to run the system “backwards.” Instead of presenting the system with symptoms and asking for a diagnosis, we can present it with a diagnosis such as “heart failure,” examine the predicted probability of symptoms such as tachycardia, and compare to the medical literature.

Perform sensitivity analysis. This important step checks whether the best decision is sensitive to small changes in the assigned probabilities and utilities by systematically varying those parameters and running the evaluation again. If small changes lead to significantly different decisions, then it could be worthwhile to spend more resources to collect better data. If all variations lead to the same decision, then the user will have more confidence that it is the right decision. Sensitivity analysis is particularly important, because one of the main criticisms of probabilistic approaches to expert systems is that it is too difficult to assess the

Probability theory describes what an agent should believe on the basis of evidence, utility theory describes what an agent wants, and decision theory puts the two together to describe what an agent should do.

We can use decision theory to build a system that makes decisions by considering all possible actions and choosing the one that leads to the best expected outcome. Such a system is known as a rational agent.

Utility theory shows that an agent whose preferences between lotteries are consistent with a set of simple axioms can be described as possessing a utility function; furthermore, the agent selects actions as if maximizing its expected utility.

Multiattribute utility theory deals with utilities that depend on several distinct attributes of states. Stochastic dominance is a particularly useful technique for making unambiguous decisions, even without precise utility values for attributes.

Decision networks provide a simple formalism for expressing and solving decision problems. They are a natural extension of Bayesian networks, containing decision and utility nodes in addition to chance nodes.

Sometimes, solving a problem involves finding more information before making a decision. The value of information is defined as the expected improvement in utility compared with making a decision without the information.

Expert systems that incorporate utility information have additional capabilities compared with pure inference systems. In addition to being able to make decisions, they can use the value of information to decide whether to acquire it and they can calculate the sensitivity of their decisions to small changes in probability and utility assessments.

608

Chapter 16. Making Simple Decisions

for lotteries, writing “the value of an item must not be based on its price, but rather on the utility that it yields” (italics his). Jeremy Bentham (1823) proposed the hedonic calculus for weighing “pleasures” and “pains,” arguing that all decisions (not just monetary ones) could be reduced to utility comparisons.

The derivation of numerical utilities from preferences was first carried out by Ramsey (1931); the axioms for preference in the present text are closer in form to those rediscovered in Theory of Games and Economic Behavior (von Neumann and Morgenstern, 1944). A good presentation of these axioms, in the course of a discussion on risk preference, is given by Howard (1977). Ramsey had derived subjective probabilities (not just utilities) from an agent’s preferences; Savage (1954) and Jeffrey (1983) carry out more recent constructions of this kind. Von Winterfeldt and Edwards (1986) provide a modern perspective on decision analysis and its relationship to human preference structures. The micromort utility measure is discussed by Howard (1989). A 1994 survey by the Economist set the value of a life at between $750,000 and $2.6 million. However, Richard Thaler (1992) found irrational variation in the price one is willing to pay to avoid a risk of death versus the price one is willing to be paid to accept a risk. For a 1/1000 chance, a respondent wouldn’t pay more than $200 to remove the risk, but wouldn’t accept $50,000 to take on the risk.

QALYs are much more widely used in medical and social policy decision making than are micromorts; see (Russell, 1990) for a typical example of an argument for a major change in public health policy on grounds of increased expected utility measured in QALY’.

The book Decisions with Multiple Objectives: Preferences and Value Tradeoffs (Keeney and Raiffa, 1976) gives a thorough introduction to multiattribute utility theory. It describes early computer implementations of methods for eliciting the necessary parameters for a multiattribute utility function and includes extensive accounts of real applications of the theory. In Al, the principal reference for MAUT is Wellman’s (1985) paper, which includes a system called URP (Utility Reasoning Package) that can use a collection of statements about preference independence and conditional independence to analyze the structure of decision problems. The use of stochastic dominance together with qualitative probability models was investigated extensively by Wellman (1988, 1990a). Wellman and Doyle (1992) provide a preliminary sketch of how a complex ‘set of utility-independence relationships might be used to provide a structured model of a utility function, in much the same way that Bayesian networks provide a structured model of joint probability distributions. Bacchus and Grove (1995, 1996) and La Mura and Shoham (1999) give further results along these lines.

Decision theory has been a standard tool in economics, finance, and management science since the 1950s. Until the 1980s, decision trees were the main tool used for representing simple decision problems. Smith (1988) gives an overview of the methodology of decision analysis. Decision networks or influence diagrams were introduced by Howard and Matheson (1984), based on earlier work by a group (including Howard and Matheson) at SRI (Miller et al., 1976). Howard and Matheson’s method involved the derivation of a decision tree from a decision network, but in general the tree is of exponential size. Shachter (1986) developed a method for making decisions based directly on a decision network, without the creation of an intermediate decision tree. This algorithm was also one of the first to provide complete inference for multiply connected Bayesian networks. Recent work by Nilsson and

Section 16.8.

Summary 609

Lauritzen (2000) links algorithms for decision networks to ongoing developments in clustering algorithms for Bayesian networks. The collection by Oliver and Smith (1990) has a number of useful articles on decision networks, as does the 1990 special issue of the journal Networks. Papers on decision networks and utility modeling also appear regularly in the journal Management Science.

Information value theory was first analyzed by Ron Howard (1966). His paper ends with the remark “If information value theory and associated decision theoretic structures do not in the future occupy a large part of the education of engineers, then the engineering profession will find that its traditional role of managing scientific and economic resources for the benefit of man has been forfeited to another profession.” To date, the implied revolution in managerial methods has not occurred, although this may change as the use of information value theory in Bayesian expert systems becomes more widespread.

Surprisingly few Al researchers adopted decision-theoretic tools after the early applications in medical decision making described in Chapter 13. One of the few exceptions was Jerry Feldman, who applied decision theory to problems in vision (Feldman and Yakimovsky, 1974) and planning (Feldman and Sproull, 1977). After the resurgence of interest in probabilistic methods in Al in the 1980s, decision-theoretic expert systems gained widespread acceptance (Horvitz et al., 1988). In fact, from 1991 onward, the cover design of the journal Artificial Intelligence has depicted a decision network, although some artistic license appears to have been taken with the direction of the arrows.

EXERCISES

16.1 (Adapted from David Heckerman.) This exercise concerns the Almanac Game, which is used by decision analysts to calibrate numeric estimations. For each of the questions that follow, give your best guess of the answer, that is, a number that you think is as likely to be too high as it is to be too low. Also give your guess at a 25th percentile estimate, that is, a number that you think has a 25% chance of being too high, and a 75% chance of being too low. Do the same for the 75th percentile. (Thus, you should give three estimates in all—low, median, and high—for each question.)

a. Number of passengers who flew between New York and Los Angeles in 1989. Population of Warsaw in 1992. Year in which Coronado discovered the Mississippi River. Number of votes received by Jimmy Carter in the 1976 presidential election. Age of the oldest living tree, as of 2002. Height of the Hoover Dam in feet. Number of eggs produced in Oregon in 1985. Number of Buddhists in the world in 1992, Number of deaths due to AIDS in the United States in 1981. Jj. Number of U.S. patents granted in 1901.

FR ome 0 T

o wmie

610

Chapter 16. Making Simple Decisions

The correct answers appear after the last exercise of this chapter. From the point of view of decision analysis, the interesting thing is not how close your median guesses came to the real answers, but rather how often the real answer came within your 25% and 75% bounds. If it was about half the time, then your bounds are accurate. But if you're like most people, you will be more sure of yourself than you should be, and fewer than half the answers will fall within the bounds. With practice, you can calibrate yourself to give realistic bounds, and thus be more useful in supplying information for decision making. Try this second set of questions and see if there is any improvement:

a. Year of birth of Zsa Zsa Gabor.

Maximum distance from Mars to the sun in miles.

Value in dollars of exports of wheat from the United States in 1992. . Tons handled by the port of Honolulu in 1991.

Annual salary in dollars of the governor of California in 1993. Population of San Diego in 1990.

Year in which Roger Williams founded Providence, Rhode Island.

2 I

Height of Mt. Kilimanjaro in feet.

—.

Length of the Brooklyn Bridge in feet. Jj. Number of deaths due to automobile accidents in the United States in 1992.

16.2 Tickets to a lottery cost $1. There are two possible prizes: a $10 payoff with probability 1/50, and a $1,000,000 payoff with probability 1/2,000,000. What is the expected monetary value of a lottery ticket? When (if ever) is it rational to buy a ticket? Be precise—show an equation involving utilities. You may assume current wealth of $& and that U (Sy) = 0. You may also assume that U (Sk, 10) = 10 x U(Sk41)), but you may not make any assumptions about U (Sk+41 000,000). Sociological studies show that people with lower income buy a disproportionate number of lottery tickets. Do you think this is because they are worse decision makers or because they have a different utility function?

16.3 In 1738, J. Bernoulli investigated the St. Petersburg paradox, which works as follows. You have the opportunity to play a game in which a fair coin is tossed repeatedly until it comes up heads. If the first heads appears on the nth toss, you win 2" dollars.

a. Show that the expected monetary value of this game is infinite. b. How much would you, personally, pay to play the game?

¢. Bernoulli resolved the apparent paradox by suggesting that the utility of money is measured on a logarithmic scale (i.e., U(Sy,) = a logy n +b, where S,, is the state of having $n). What is the expected utility of the game under this assumption?

d. What is the maximum amount that it would be rational to pay to play the game, assuming that one’s initial wealth is $& ?

16.4 Assess your own utility for different incremental amounts of money by running a series of preference tests between some definite amount M; and a lottery [p, Ma; (1—p), 0]. Choose

Section 16.8.

Summary 611

different values of M; and Mj, and vary p until you are indifferent between the two choices. Plot the resulting utility function.

16.5 Write a computer program to automate the process in Exercise 16.4. Try your program out on several people of different net worth and political outlook. Comment on the consistency of your results, both for an individual and across individuals.

16.6 How much is a micromort worth to you? Devise a protocol to determine this. Ask questions based both on paying to avoid risk and being paid to accept risk.

16.7 Show that if X; and X are preferentially independent of X3, and X5 and X3 are preferentially independent of X71, then X3 and X; are preferentially independent of Xo.

16.8 This exercise completes the analysis of the airport-siting problem in Figure 16.5.

a. Provide reasonable variable domains, probabilities, and utilities for the network, assuming that there are three possible sites.

b. Solve the decision problem.

¢. What happens if changes in technology mean that each aircraft generates half as much noise?

d. What if noise avoidance becomes three times more important?

e. Calculate the VPI for AirTraffic, Litigation, and Construction in your model.

16.9 Repeat Exercise 16.8, using the action-utility representation shown in Figure 16.6.

16.10 For either of the airport-siting diagrams from Exercises 16.8 and 16.9, to which conditional probability table entry is the utility most sensitive, given the available evidence?

16.11 (Adapted from Pearl (1988).) A used-car buyer can decide to carry out various tests with various costs (e.g., kick the tires, take the car to a qualified mechanic) and then, depending on the outcome of the tests, decide which car to buy. We will assume that the buyer is deciding whether to buy car cj, that there is time to carry out at most one test, and that ¢; is the test of ¢; and costs $50.

A car can be in good shape (quality ¢\*) or bad shape (quality ¢ 7), and the tests might help to indicate what shape the car is in. Car ¢; costs $1,500, and its market value is $2,000 if it is in good shape; if not, $700 in repairs will be needed to make it in good shape. The buyer’s estimate is that ¢; has a 70% chance of being in good shape.

a. Draw the decision network that represents this problem.

b. Calculate the expected net gain from buying cj, given no test.

¢. Tests can be described by the probability that the car will pass or fail the test given that the car is in good or bad shape. We have the following information: P(pass(c1,t1)|gt(c1)) = 0.8 P(pass(e1,t1)|lg (e1)) = 0.35 Use Bayes’ theorem to calculate the probability that the car will pass (or fail) its test and hence the probability that it is in good (or bad) shape given each possible test outcome.

Chapter 16. Making Simple Decisions

d. Calculate the optimal decisions given either a pass or a fail, and their expected utilities.

e. Calculate the value of information of the test, and derive an optimal conditional plan for the buyer.

16.12 Prove that the value of information is nonnegative and order-independent, as stated in Section 16.6. Explain how it is that one can make a worse decision after receiving information than one would have made before receiving it.

16.13 Modify and extend the Bayesian network code in the code repository to provide for creation and evaluation of decision networks and the calculation of information value.

The answers to Exercise 16.1 (where M stands for million): First set: 3M, 1.6M, 1541, 41M, 4768, 221, 649M, 295M, 132, 25,546. Second set: 1917, 155M, 4,500M, 11M, 120,000, 1.1M, 1,636, 19,340, 1,595, 41,710.

] 7 MAKING COMPLEX DECISIONS

In which we examine methods for deciding what to do today, given that we may decide again tomorrow.

In this chapter, we address the computational issues involved in making decisions. Whereas Chapter 16 was concerned with one-shot or episodic decision problems, in which the utility of each action’s outcome was well known, we will be concerned here with sequential

SEQUENTIAL . . . ) iqs .

DECISION decision problems, in which the agent’s utility depends on a sequence of decisions. Sequential decision problems, which include utilities, uncertainty, and sensing, generalize the search and planning problems described in Parts II and IV. Section 17.1 explains how sequential decision problems are defined, and Sections 17.2 and 17.3 explain how they can be solved to produce optimal behavior that balances the risks and rewards of acting in an uncertain environment. Section 17.4 extends these ideas to the case of partially observable environments, and Section 17.5 develops a complete design for decision-theoretic agents in partially observable environments, combining dynamic Bayesian networks from Chapter 15 with decision networks from Chapter 16.

The second part of the chapter covers environments with multiple agents. In such environments, the notion of optimal behavior becomes much more complicated by the interactions among the agents. Section 17.6 introduces the main ideas of game theory, including the idea that rational agents might need to behave randomly. Section 17.7 looks at how multiagent systems can be designed so that multiple agents can achieve a common goal.

17.1 SEQUENTIAL DECISION PROBLEMS

An example

Suppose that an agent is situated in the 4 x 3 environment shown in Figure 17.1(a). Beginning in the start state, it must choose an action at each time step. The interaction with the environment terminates when the agent reaches one of the goal states, marked +1 or 1. In each location, the available actions are called Up, Down, Left, and Right. We will assume for now that the environment is fully observable, so that the agent always knows where it is.

613

614

Chapter 17. Making Complex Decisions

TRANSITION MODEL

REWARD

1 START

(a) b)

Figure 17.1 (a) A simple 4 x 3 environment that presents the agent with a sequential decision problem. (b) Illustration of the transition model of the environment: the “intended” outcome occurs with probability 0.8, but with probability 0.2 the agent moves at right angles to the intended direction. A collision with a wall results in no movement. The two terminal states have reward +1 and —1, respectively, and all other states have a reward of —0.04.

If the environment were deterministic, a solution would be easy: (Up, Up, Right, Right, Right]. Unfortunately, the environment won’t always go along with this solution, because the actions are unreliable. The particular model of stochastic motion that we adopt is illustrated in Figure 17.1(b). Each action achieves the intended effect with probability 0.8, but the rest of the time, the action moves the agent at right angles to the intended direction. Furthermore, if the agent bumps into a wall, it stays in the same square. For example, from the start square (1,1), the action Up moves the agent to (1,2) with probability 0.8, but with probability 0.1, it moves right to (2,1), and with probability 0.1, it moves left, bumps into the wall, and stays in (1,1). In such an environment, the sequence [Up, Up, Right, Right, Right] goes up around the barrier and reaches the goal state at (4,3) with probability 0.8° = 0.32768. There is also a small chance of accidentally reaching the goal by going the other way around with probability 0.1% x 0.8, for a grand total of 0.32776. (See also Exercise 17.1.)

A specification of the outcome probabilities for each action in each possible state is called a transition model (or just “model,” whenever no confusion can arise). We will use T(s,a,s’) to denote the probability of reaching state s’ if action a is done in state s. We will assume that transitions are Markovian in the sense of Chapter 15, that is, the probability of reaching s’ from s depends only on s and not on the history of earlier states. For now, you can think of T'(s, a,s’) as a big three-dimensional table containing probabilities. Later, in Section 17.5, we will see that the transition model can be represented as a dynamic Bayesian network, just as in Chapter 15.

To complete the definition of the task environment, we must specify the utility function for the agent. Because the decision problem is sequential, the utility function will depend on a sequence of states—an environment history—rather than on a single state. Later in this section, we will investigate how such utility functions can be specified in general; for now, we will simply stipulate that in each state s, the agent receives a reward R(s), which may be positive or negative, but must be bounded. For our particular example, the reward is —0.04 in all states except the terminal states (which have rewards +1 and —1). The utility of

Section 17.1.

Sequential Decision Problems 615

MARKOV DECISION PROCESS

POLICY

OPTIMAL POLICY

an environment history is just (for now) the sum of the rewards received. For example, if the agent reaches the +1 state after 10 steps, its total utility will be 0.6. The negative reward of —0.04 gives the agent an incentive to reach (4,3) quickly, so our environment is a stochastic generalization of the search problems of Chapter 3. Another way of saying this is that the agent does not enjoy living in this environment and so wants to get out of the game as soon as possible.

The specification of a sequential decision problem for a fully observable environment with a Markovian transition model and additive rewards is called a Markov decision process, or MDP. An MDP is defined by the following three components:

Initial State: Sp

Transition Model: T'(s, a, s")

Reward Function:! R(s)

The next question is, what does a solution to the problem look like? We have seen that any fixed action sequence won’t solve the problem, because the agent might end up in a state other than the goal. Therefore, a solution must specify what the agent should do for any state that the agent might reach. A solution of this kind is called a policy. We usually denote a policy by 7, and 7(s) is the action recommended by the policy 7 for state s. If the agent has a complete policy, then no matter what the outcome of any action, the agent will always know what to do next.

Each time a given policy is executed starting from the initial state, the stochastic nature of the environment will lead to a different environment history. The quality of a policy is therefore measured by the expected utility of the possible environment histories generated by that policy. An optimal policy is a policy that yields the highest expected utility. We use 7\* to denote an optimal policy. Given 7\*, the agent decides what to do by consulting its current percept, which tells it the current state s, and then executing the action 7\*(s). A policy represents the agent function explicitly and is therefore a description of a simple reflex agent, computed from the information used for a utility-based agent:

An optimal policy for the world of Figure 17.1 is shown in Figure 17.2(a). Notice that, because the cost of taking a step is fairly small compared with the penalty for ending up in (4,2) by accident, the optimal policy for the state (3,1) is conservative. The policy recommends taking the long way round, rather than taking the short cut and thereby risking entering (4,2).

The balance of risk and reward changes depends on the value of R(s) for the nonterminal states. Figure 17.2(b) shows optimal policies for four different ranges of R(s). When R(s) < —1.6284, life is so painful that the agent heads straight for the nearest exit, even if the exit is worth —1. When —0.4278 < R(s) < —0.0850, life is quite unpleasant; the agent takes the shortest route to the +1 state and is willing to risk falling into the —1 state by accident. In particular, the agent takes the shortcut from (3,1). When life is only slightly dreary (—0.0221 < R(s) < 0), the optimal policy takes no risks at all. In (4,1) and (3,2), the agent

1 Some definitions of MDPs allow the reward to depend on the action and outcome too, so the reward function is R(s,a,s'). This simplifies the description of some environments but does not change the problem in any fundamental way.

616

Chapter 17. Making Complex Decisions

[]

FINITE HORIZON INFINITE HORIZON

Figure 17.2 (a) An optimal policy for the stochastic environment with R(s) = — 0.04 in the nonterminal states. (b) Optimal policies for four different ranges of R(s).

heads directly away from the —1 state so that it cannot fall in by accident, even though this means banging its head against the wall quite a few times. Finally, if R(s) > 0, then life is positively enjoyable and the agent avoids both exits. As long as the actions in (4,1), (3,2), and (3,3) are as shown, every policy is optimal, and the agent obtains infinite total reward because it never enters a terminal state. Surprisingly, it turns out that there are six other optimal policies for various ranges of R(s); Exercise 17.7 asks you to find them.

The careful balancing of risk and reward is a characteristic of MDPs that does not arise in deterministic search problems; moreover, it is a characteristic of many real-world decision problems. For this reason, MDPs have been studied in several fields, including Al, operations research, economics, and control theory. Dozens of algorithms have been proposed for calculating optimal policies. In sections 17.2 and 17.3 we will describe two of the most important algorithm families. First, however, we must complete our investigation of utilities and policies for sequential decision problems.

Optimality in sequential decision problems

In the MDP example in Figure 17.1, the performance of the agent was measured by a sum of rewards for the states visited. This choice of performance measure is not arbitrary, but it is not the only possibility. This section investigates the possible choices for the performance measure—that is, choices for the utility function on environment histories, which we will write as Up([so, 51, --, 8n]). The section draws on ideas from Chapter 16 and is somewhat technical; the main points are summarized at the end.

The first question to answer is whether there is a finite horizon or an infinite horizon for decision making. A finite horizon means that there is a fixed time N after which nothing matters—the game is over, so to speak. Thus, Ux ([so, s1,--, Sn+k]) = Un([s0, 81, -,8N]) for all £ > 0. For example, suppose an agent starts at (3,1) in the 4 x 3 world of Figure 17.1,

Sequential Decision Problems 617

NONSTATIONARY POLICY

STATIONARY POLICY

STATIONARY PREFERENCE

ADDITIVE REWARDS

DISCOUNTED REWARDS

DISCOUNT FACTOR

and suppose that N = 3. Then, to have any chance of reaching the +1 state, the agent must head directly for it, and the optimal action is to go Up. On the other hand, if N =100 then there is plenty of time to take the safe route by going Left. So, with a finite horizon, the optimal action in a given state could change over time. We say that the optimal policy for a finite horizon is nonstationary. With no fixed time limit, on the other hand, there is no reason to behave differently in the same state at different times. Hence, the optimal action depends only on the current state, and the optimal policy is stationary. Policies for the infinite-horizon case are therefore simpler than those for the finite-horizon case, and we will deal mainly with the infinite-horizon case in this chapter. Note that “infinite horizon” does not necessarily mean that all state sequences are infinite; it just means that there is no fixed deadline. In particular, there can be finite state sequences in an infinite-horizon MDP containing a terminal state.

The next question we must decide is how to calculate the utility of state sequences. We can view this as a question in multiattribute utility theory (see Section 16.4), where each state s; is viewed as an attribute of the state sequence [sg, 51, $2. . .]. To obtain a simple expression in terms of the attributes, we will need to make some sort of preference independence assumption. The most natural assumption is that the agent’s preferences between state sequences are stationary. Stationarity for preferences means the following: if two state sequences [so, s1, 52, ..] and [s), 8], sh, ...] begin with the same state (i.e., so =s() then the two sequences should be preference-ordered the same way as the sequences [s1, sa, . . | and [s], s5,...]. In English, this means that if you prefer one future to another starting tomorrow, then you should still prefer that future if it were to start today. Stationarity is a fairly innocuous-looking assumption with very strong consequences: it turns out that under stationarity there are just two ways to assign utilities to sequences:

1. Additive rewards: The utility of a state sequence is Un([s0, 81, 82, --]) = R(so) + R(s1) + R(s2) +--. The 4 x 3 world in Figure 17.1 uses additive rewards. Notice that additivity was used implicitly in our use of path cost functions in heuristic search algorithms (Chapter 4). 2. Discounted rewards: The utility of a state sequence is Un([s0, 51,52, --]) = R(s0) + YR(s1) + ¥2R(s2) + --, where the discount factor + is a number between 0 and 1. The discount factor describes the preference of an agent for current rewards over future rewards. When 7 is close to 0, rewards in the distant future are viewed as insignificant. When + is 1, discounted rewards are exactly equivalent to additive rewards, so additive rewards are a special case of discounted rewards. Discounting appears to be a good model of both animal and human preferences over time. A discount factor of + is equivalent to an interest rate of (1/7) —1. For reasons that will shortly become clear, we will assume discounted rewards in the remainder of the chapter, although sometimes we will allow v= 1.

2 This is for completely observable environments. We will see later that for partially observable environments, the infinite-horizon case is not so simple.

618 Chapter 17. Making Complex Decisions Lurking beneath our choice of infinite horizons is a problem: if the environment does not contain a terminal state, or if the agent never reaches one, then all environment histories will be infinitely long, and utilities with additive rewards will generally be infinite. Now, we can agree that +o is better than —oo, but comparing two state sequences, both having +oo utility is more difficult. There are three solutions, two of which we have seen already: 1. With discounted rewards, the utility of an infinite sequence is finite. In fact, if rewards are bounded by Rax and vy < 1, we have oo oo Un([s0, 81, 82,...]) = > ~Y R(st) < > Y Rmax = Rmax/(1 — 7), (17.1) t=0 t=0 using the standard formula for the sum of an infinite geometric series. 2. If the environment contains terminal states and if the agent is guaranteed to get fo one eventually, then we will never need to compare infinite sequences. A policy that is PROPER POLICY guaranteed to reach a terminal state is called a proper policy. With proper policies, we

AVERAGE REWARD

can use v= 1 (i.e., additive rewards). The first three policies shown in Figure 17.2(b) are proper, but the fourth is improper. It gains infinite total reward by staying away from the terminal states when the reward for the nonterminal states is positive. The existence of improper policies can cause the standard algorithms for solving MDPs to fail with additive rewards, and so provides a good reason for using discounted rewards.

3. Another possibility is to compare infinite sequences in terms of the average reward obtained per time step. Suppose that square (1,1) in the 4 x 3 world has a reward of 0.1 while the other nonterminal states have a reward of 0.01. Then a policy that does its best to stay in (1,1) will have higher average reward than one that stays elsewhere. Average reward is a useful criterion for some problems, but the analysis of averagereward algorithms is beyond the scope of this book.

In sum, the use of discounted rewards presents the fewest difficulties in evaluating state sequences. The final step is to show how to choose between policies, bearing in mind that a given policy 7 generates not one state sequence, but a whole range of possible state sequences, each with a specific probability determined by the transition model for the environment. Thus, the value of a policy is the expected sum of discounted rewards obtained, where the expectation is taken over all possible state sequences that could occur, given that the policy is executed. An optimal policy 7\* satisfies

oo

7m = argmax E > Y'R(st) | d . (17.2) T t=0

The next two sections describe algorithms for finding optimal policies.

VALUE ITERATION

In this section, we present an algorithm, called value iteration, for calculating an optimal policy. The basic idea is to calculate the utility of each state and then use the state utilities to select an optimal action in each state.

Section 17.2.

Value Iteration 619

Utilities of states

The utility of states is defined in terms of the utility of state sequences. Roughly speaking, the utility of a state is the expected utility of the state sequences that might follow it. Obviously, the state sequences depend on the policy that is executed, so we begin by defining the utility U™(s) with respect to a specific policy 7. If we let s; be the state the agent is in after executing 7 for t steps (note that s; is a random variable), then we have

U'(s)=F > Y R(s¢) | 7,850 =5] . (17.3) t=0

Given this definition, the true utility of a state, which we write as U(s), is just U™ (s)—that is, the expected sum of discounted rewards if the agent executes an optimal policy. Notice that U(s) and R(s) are quite different quantities; R(s) is the “short-term” reward for being in s, whereas U(s) is the “long-term” total reward from s onwards. Figure 17.3 shows the utilities for the 4 x 3 world. Notice that the utilities are higher for states closer to the +1 exit, because fewer steps are required to reach the exit.

[]

1 2 3 4

Figure 17.3 The utilities of the states in the 4 x 3 world, calculated with y=1 and R(s) = — 0.04 for nonterminal states.

The utility function U(s) allows the agent to select actions by using the Maximum Expected Utility principle from Chapter 16—that is, choose the action that maximizes the expected utility of the subsequent state:

7 (s) = argmax »\_T(s,a,s")U(s) . (17.4)

Now, if the utility of a state is the expected sum of discounted rewards from that point onwards, then there is a direct relationship between the utility of a state and the utility of its neighbors: the utility of a state is the immediate reward for that state plus the expected discounted utility of the next state, assuming that the agent chooses the optimal action. That is, the utility of a state is given by

U(s) = R(s) +7 max > T(s,a,8YU(s). (17.5)

620

Chapter 17. Making Complex Decisions

BELLMAN EQUATION

BELLMAN UPDATE

Equation (17.5) is called the Bellman equation, after Richard Bellman (1957). The utilities of the states—defined by Equation (17.3) as the expected utility of subsequent state sequences—are solutions of the set of Bellman equations. In fact, they are the unique solutions, as we show in the next two sections.

Let us look at one of the Bellman equations for the 4 x 3 world. The equation for the state (1,1) is

U(1,1) = —0.04 + vy max{ 0.8U(1,2) + 0.1U(2,1) + 0.1U(1, 1), (Up) 0.9U(1,1) + 0.1U(1, 2), (Left) 0.9U(1,1) + 0.1U(2,1), (Down) 0.8U(2,1)+0.1U(1,2) + 0.1U(1,1) } (Right)

When we plug in the numbers from Figure 17.3, we find that Up is the best action.

The value iteration algorithm

The Bellman equation is the basis of the value iteration algorithm for solving MDPs. If there are n possible states, then there are n Bellman equations, one for each state. The n equations contain n unknowns—the utilities of the states. So we would like to solve these simultaneous equations to find the utilities. There is one problem: the equations are nonlinear, because the “max” operator is not a linear operator. Whereas systems of linear equations can be solved quickly using linear algebra techniques, systems of nonlinear equations are more problematic. One thing to try is an iterative approach. We start with arbitrary initial values for the utilities, calculate the right-hand side of the equation, and plug it into the left-hand side—thereby updating the utility of each state from the utilities of its neighbors. We repeat this until we reach an equilibrium. Let U; (s) be the utility value for state s at the sth iteration. The iteration step, called a Bellman update, looks like this:

Uir1(s) « R(s) + ymax y\_T(s,a,s)Us(s') . (17.6)

If we apply the Bellman update infinitely often, we are guaranteed to reach an equilibrium (see the next subsection), in which case the final utility values must be solutions to the Bellman equations. In fact, they are also the unique solutions, and the corresponding policy (obtained using Equation (17.4)) is optimal. The algorithm, called VALUE-ITERATION, is shown in Figure 17.4.

We can apply value iteration to the 4 x 3 world in Figure 17.1(a). Starting with initial values of zero, the utilities evolve as shown in Figure 17.5(a). Notice how the states at different distances from (4,3) accumulate negative reward until, at some point, a path is found to (4,3) whereupon the utilities start to increase. We can think of the value iteration algorithm as propagating information through the state space by means of local updates.

Convergence of value iteration

We said that value iteration eventually converges to a unique set of solutions of the Bellman equations. In this section, we explain why this happens. We introduce some useful mathematical ideas along the way, and we obtain some methods for assessing the error in the utility

CONTRACTION

Value Iteration 621

function VALUE-ITERATION(mdp, €) returns a utility function inputs: mdp, an MDP with states .S, transition model 7T', reward function R, discount ~y €, the maximum error allowed in the utility of any state local variables: U, U’, vectors of utilities for states in , initially zero 4, the maximum change in the utility of any state in an iteration

repeat UU; 60 for each state sin 5S do U'ls] — R[s] + max > T(s,a,s) Us]

it |U'[s) — Us] > 6thend—|U'[s] — Ul] until § < €(1—7)/v return U

Figure 17.4 The value iteration algorithm for calculating utilities of states. The termination condition is from Equation (17.8).

le+07 1 1e+06 08 1, % ; B 100000 5 06 {i E £ i g 10000 5 04 qi & 2 i £1000 2 024 & = 2 100 0 -0.2 4 T T T ] +¥———r— r v — 0 5 10 15 20 25 30 0.50.550.60.650.70.750.80.850.90.95 1 Number of iterations Discount factor y (a) (b)

Figure 17.5 (a) Graph showing the evolution of the utilities of selected states using value iteration. (b) The number of value iterations k required to guarantee an error of at most

€ = c+ Rpax, for different values of ¢, as a function of the discount factor ~.

function returned when the algorithm is terminated early; this is useful because it means that we don’t have to run forever. The section is quite technical.

The basic concept used in showing that value iteration converges is the notion of a contraction. Roughly speaking, a contraction is a function of one argument that, when applied to two different inputs in turn, produces two output values that are “closer together,” by at least some constant amount, than the original arguments. For example, the function “divide by two” is a contraction, because, after we divide any two numbers by two, their difference is halved. Notice that the “divide by two” function has a fixed point, namely zero, that is un

622

Chapter 17. Making Complex Decisions

changed by the application of the function. From this example, we can discern two important properties of contractions:

e A contraction has only one fixed point; if there were two fixed points they would not get closer together when the function was applied, so it would not be a contraction.

e When the function is applied to any argument, the value must get closer to the fixed point (because the fixed point does not move), so repeated application of a contraction always reaches the fixed point in the limit.

Now, suppose we view the Bellman update (Equation (17.6)) as an operator B that is applied simultaneously to update the utility of every state. Let U; denote the vector of utilities for all the states at the ith iteration. Then the Bellman update equation can be written as

Ui — BU; .

Next, we need a way to measure distances between utility vectors. We will use the max norm, which measures the length of a vector by the length of its biggest component:

IU] = max [U(s)]

With this definition, the “distance” between two vectors, ||[U — U’|], is the maximum difference between any two corresponding elements. The main result of this section is the following: Let U; and U] be any two utility vectors. Then we have

|BU; — BUJl| <~||Us = Uf] (17.7)

That is, the Bellman update is a contraction by a factor of v on the space of utility vectors. Hence, value iteration always converges to a unique solution of the Bellman equations.

In particular, we can replace U; in Equation (17.7) with the true utilities U, for which BU =U. Then we obtain the inequality

IBU:; Ul <~||U: = UII.

So, if we view ||U; — U]| as the error in the estimate U;, we see that the error is reduced by a factor of at least v on each iteration. This means that value iteration converges exponentially fast. We can calculate the number of iterations required to reach a specified error bound € as follows: First, recall from Equation (17.1) that the utilities of all states are bounded by + Rmax/(1 — 7). This means that the maximum initial error ||Uy — U|| < 2Rpax/(1 — 7). Suppose we run for N iterations to reach an error of at most e. Then, because the error is reduced by at least y each time, we require ¥V 2R 4x /(1 — 7) < e. Taking logs, we find

N= [log(2Rmax/€(1 —v))/log(1/7)] iterations suffice. Figure 17.5(b) shows how NN varies with +, for different values of the ratio €¢/RmaxThe good news is that, because of the exponentially fast convergence, N does not depend much on the ratio € / Ryax. The bad news is that N' grows rapidly as becomes close to 1. We can get fast convergence if we make ~ small, but this effectively gives the agent a short horizon and could miss the long-term effects of the agent’s actions.

The error bound in the preceding paragraph gives some idea of the factors influencing the runtime of the algorithm, but is sometimes overly conservative as a method of deciding when to stop the iteration. For the latter purpose, we can use a bound relating the error

Section 17.2.

Value Iteration 623

POLICY LOSS

to the size of the Bellman update on any given iteration. From the contraction property (Equation (17.7)), it can be shown that if the update is small (i.e., no state’s utility changes by much), then the error, compared with the true utility function, also is small. More precisely,

if Ui+1 — Us|

This is the termination condition used in the VALUE-ITERATION algorithm of Figure 17.4. So far, we have analyzed the error in the utility function returned by the value iteration algorithm. What the agent really cares about, however, is how well it will do if it makes its decisions on the basis of this utility function. Suppose that after 7 iterations of value iteration, the agent has an estimate U; of the true utility U and obtains the MEU policy 7; based on one-step look-ahead using U; (as in Equation (17.4)). Will the resulting behavior be nearly as good as the optimal behavior? This is a crucial question for any real agent, and it turns out that the answer is yes. U™i(s) is the utility obtained if 7; is executed starting in s, and the policy loss |[U™ — U]| is the most the agent can lose by executing 7; instead of the optimal policy 7\*. The policy loss of 7; is connected to the error in U; by the following inequality:

if ||[Ui=U||

In practice, it often occurs that 7; becomes optimal long before U; has converged. Figure 17.6 shows how the maximum error in U; and the policy loss approach zero as the value iteration process proceeds for the 4 x 3 environment with y = 0.9. The policy 7; is optimal when i = 4, even though the maximum error in Uj is still 0.46.

Now we have everything we need to use value iteration in practice. We know that it converges to the correct utilities, we can bound the error in the utility estimates if we stop after a finite number of iterations, and we can bound the policy loss that results from executing the corresponding MEU policy. As a final note, all of the results in this section depend on discounting with v < 1. If y=1 and the environment contains terminal states, then a similar set of convergence results and error bounds can be derived whenever certain technical conditions are satisfied.

—\_

Max error 0.8 Policy loss ======~

Max error/Policy loss

0 2 4 6 8 10 12 14 Number of iterations

Figure 17.6 The maximum error ||U; — U]|| of the utility estimates and the policy loss [|[U™ — U|| compared with the optimal policy, as a function of the number of iterations of value iteration.

624

Chapter 17. Making Complex Decisions

POLICY ITERATION

POLICY EVALUATION

POLICY IMPROVEMENT

In the previous section, we observed that it is possible to get an optimal policy even when the utility function estimate is inaccurate. If one action is clearly better than all others, then the exact magnitude of the utilities on the states involved need not be precise. This insight suggests an alternative way to find optimal policies. The policy iteration algorithm alternates the following two steps, beginning from some initial policy 7g:

e Policy evaluation: given a policy 7;, calculate U; = U™, the utility of each state if 7; were to be executed.

e Policy improvement: Calculate a new MEU policy 7;41, using one-step look-ahead based on Uj; (as in Equation (17.4)).

The algorithm terminates when the policy improvement step yields no change in the utilities. At this point, we know that the utility function Uj is a fixed point of the Bellman update, so it is a solution to the Bellman equations, and 7; must be an optimal policy. Because there are only finitely many policies for a finite state space, and each iteration can be shown to yield a better policy, policy iteration must terminate. The algorithm is shown in Figure 17.7.

The policy improvement step is obviously straightforward, but how do we implement the POLICY-EVALUATION routine? It turns out that doing so is much simpler than solving the standard Bellman equations (which is what value iteration does), because the action in each state is fixed by the policy. At the ith iteration, the policy 7; specifies the action 7;(s) in state s. This means that we have a simplified version of the Bellman equation (17.5) relating

function POLICY-ITERATION(mdp) returns a policy inputs: mdp, an MDP with states S, transition model T local variables: U, U’, vectors of utilities for states in 5, initially zero m, a policy vector indexed by state, initially random

repeat U «— POLICY-EVALUATION(m, U, mdp) unchanged? «— true for each state sin .S do if max, Y T(s,a,s') Uls'] > 3. T(s,n[s],s’) Uls'] then

7[s] «—argmax, > T(s,a,s) Uls']

$s unchanged? « false until unchanged? return mT

Figure 17.7 The policy iteration algorithm for calculating an optimal policy.

Section 17.4. Partially observable MDPs 625

the utility of s (under 7;) to the utilities of its neighbors: Ui(s) = R(s) + > T(s, mi(s),s)Us(s) . (17.10) s For example, suppose 7; is the policy shown in Figure 17.2(a). Then we have 7;(1,1) = Up, m;(1,2) = Up, and so on, and the simplified Bellman equations are

Us(1,1) = —0.04 4 0.8U;(1,2) 4 0.1U;(1,1) + 0.1U3(2, 1), Us(1,2) = —0.04 + 0.8U;(1,3) + 0.2U(1,2)

The important point is that these equations are linear, because the “max” operator has been removed. For n states, we have n linear equations with n unknowns, which can be solved exactly in time O(n?) by standard linear algebra methods.

For small state spaces, policy evaluation using exact solution methods is often the most efficient approach. For large state spaces, O(n?) time might be prohibitive. Fortunately, it is not necessary to do exact policy evaluation. Instead, we can perform some number of simplified value iteration steps (simplified because the policy is fixed) to give a reasonably good approximation of the utilities. The simplified Bellman update for this process is

Uit1(s) «< R(s) + > T(s,mi(s),sUi(s)

¥ and this is repeated k times to produce the next utility estimate. The resulting algorithm is MIDIRESTOLCY called modified policy iteration. Itis often much more efficient than standard policy iteration or value iteration.

The algorithms we have described so far require updating the utility or policy for all states at once. It turns out that this is not strictly necessary. In fact, on each iteration, we can pick any subset of states and apply either kind of updating (policy improvement or simplified value iteration) to that subset. This very general algorithm is called asynchronous policy

Rncrnonous, iteration. Given certain conditions on the initial policy and utility function, asynchronous policy iteration is guaranteed to converge to an optimal policy. The freedom to choose any states to work on means that we can design much more efficient heuristic algorithms—for example, algorithms that concentrate on updating the values of states that are likely to be reached by a good policy. This makes a lot of sense in real life: if one has no intention of throwing oneself off a cliff, one should not spend time worrying about the exact value of the resulting states.

17.4 PARTIALLY OBSERVABLE MDPS

The description of Markov decision processes in Section 17.1 assumed that the environment was fully observable. With this assumption, the agent always knows which state it is in. This, combined with the Markov assumption for the transition model, means that the optimal policy depends only on the current state. When the environment is only partially observable, the situation is, one might say, much less clear. The agent does not necessarily know which

626

Chapter 17. Making Complex Decisions

PARTIALLY OBSERVABLE MDP

OBSERVATION MODEL

state it is in, so it cannot execute the action 7(s) recommended for that state. Furthermore, the utility of a state s and the optimal action in $s depend not just on s, but also on how much the agent knows when it is in s. For these reasons, partially observable MDPs (or POMDPs— pronounced “pom-dee-pees”) are usually viewed as much more difficult than ordinary MDPs. We cannot avoid POMDPs, however, because the real world is one.

As an example, consider again the 4 x 3 world of Figure 17.1, but now let’s suppose that the agent has no sensors whatsoever and has no idea where it is. More precisely, let’s suppose the agent’s initial state is equally likely to be any of the nine nonterminal states (Figure 17.8(a)). Clearly, if the agent knew it was in (3,3), it would move Righr; if it knew it was in (1,1), it would move Up; but since it could be anywhere, what should it do? One possible answer is that the agent should first act so as to reduce its uncertainty, and only then should it try heading for the +1 exit. For example, if the agent moves Left five times, then it is quite likely to be at the left wall (Figure 17.8(b)). Then, if it moves Up five times, it is quite likely to be at the top, probably in the top left corner (Figure 17.8(c)). Finally, if it moves Right five times, it has a good chance—about 77.5%—of reaching the +1 exit (Figure 17.8(d)). Continuing to move right thereafter increases its chances to 81.8%. This policy is therefore surprisingly safe, but under it, the agent is rather slow to reach the exit, and has an expected utility of only about 0.08. The optimal policy, which we will describe shortly, does much better.

0.000 0.300 | 0.010 | 0.008 | [0.000 0.622 | 0.221 | 0.071 0.005 | 0.007 | 0.019 | [0775

0.000 0.221 0.059 0.005 0.003 0.034 a 0.007

0.111 0.371 | 0.012 | 0.008 | 0.000 0.003 | 0.024 | 0.003 | 0.000 0.005 | 0.006 | 0.008 | 0.030 (a) (b) (©) @

Figure 17.8 (a) The initial probability distribution for the agent’s location. (b) After moving Left five times. (c) After moving Up five times. (d) After moving Right five times.

To get a handle on POMDPs, we must first define them properly. A POMDP has the same elements as an MDP—the transition model T'(s, a, s’) and the reward function R(s)— but it also has an observation model O(s, 0) that specifies the probability of perceiving the observation o in state s.> For example, our agent with no sensors has only one possible observation (the empty observation), and this occurs with probability 1 in every state.

In Chapters 3 and 12, we studied nondeterministic and partially observable planning problems and identified the belief state—the set of actual states the agent might be in—as a key concept for describing and calculating solutions. In POMDDPs, the concept is refined somewhat. A belief state b is now a probability distribution over all possible states. For exam

111111111

ple, the initial belief state in Figure 17.8(a) could be written as (3, 5,5: 5:5 3+ 5 5° 5+ 0+ 0)

3 The observation model is essentially identical to the sensor model for temporal processes, as described in Chapter 15. As with the reward function for MDPs, the observation model can also depend on the action and outcome state, but again this change is not fundamental.

Section 17.4.

Partially observable MDPs 627

We will write b(s) for the probability assigned to the actual state s by belief state b. The agent can calculate its current belief state as the conditional probability distribution over the actual states given the sequence of observations and actions so far. This is essentially the filtering task. (See Chapter 15.) The basic recursive filtering equation (15.3 on page 543) shows how to calculate the new belief state from the previous belief state and the new observation. For POMDPs, we also have an action to consider and a slightly different notation, but the result is essentially the same. If b(s) was the previous belief state, and the agent does action a and perceives observation o, then the new belief state is given by

b(s') = a0(s',0)Y T(s,a,s)b(s) (17.11)

where « is a normalizing constant that makes the belief state sum to 1. We can abbreviate this equation as ¥' = FORWARD(b, a, 0).

The fundamental insight required to understand POMDPs is this: the optimal action depends only on the agent’s current belief state. That is, the optimal policy can be described by a mapping 7\*(b) from belief states to actions. It does not depend on the actual state the agent is in. This is a good thing, because the agent does not know its actual state; all it knows is the belief state. Hence, the decision cycle of a POMDP agent is this:

1. Given the current belief state b, execute the action a = 7\*(b). 2. Receive observation o. 3. Set the current belief state to FORWARD(b, a, 0) and repeat.

Now we can think of POMDPs as requiring a search in belief state space, just like the methods for sensorless and contingency problems in Chapter 3. The main difference is that the POMDP belief state space is continuous, because a POMDP belief state is a probability distribution. For example, a belief state for the 4 x 3 world is a point in an 11-dimensional continuous space. An action changes the belief state, not just the physical state, so it is evaluated according to the information the agent acquires as a result. POMDPs therefore include the value of information (Section 16.6) as one component of the decision problem.

Let's look more carefully at the outcome of actions. In particular, let’s calculate the probability that an agent in belief state b reaches belief state b’ after executing action a. Now, if we knew the action and the subsequent observation, then Equation (17.11) would provide a deterministic update to the belief state: b’ = FORWARD(b, a, 0). Of course, the subsequent observation is not yet known, so the agent might arrive in one of several possible belief states b', depending on the observation that occurs. The probability of perceiving o, given that a was performed starting in belief state b, is given by summing over all the actual states s’ that the agent might reach:

[]

628

Chapter 17. Making Complex Decisions

[]

Let us write the probability of reaching ' from b, given action a, as 7(b, a,b’). Then that gives us

where P(b'|o,a,b) is 1 if ¥ = FORWARD(b, a, 0) and 0 otherwise.

Equation (17.12) can be viewed as defining a transition model for the belief state space. We can also define a reward function for belief states (i.e., the expected reward for the actual states the agent might be in):

=D b(s)R(s)

So it seems that 7(b, a,b’) and p(b) together define an observable MDP on the space of belief states. Furthermore, it can be shown that an optimal policy for this MDP, 7\*(b), is also an optimal policy for the original POMDP. In other words, solving a POMDP on a physical state space can be reduced to solving an MDP on the corresponding belief state space. This fact is perhaps less surprising if we remember that the belief state is always observable to the agent, by definition.

Notice that, although we have reduced POMDPs to MDPs, the MDP we obtain has a continuous (and usually high-dimensional) state space. None of the MDP algorithms described in Sections 17.2 and 17.3 applies directly to such MDPs. It turns out that we can develop versions of value and policy iteration that apply to continuous-state MDPs. The basic idea is that a policy 7(b) can be represented as a set of regions of belief state space, each of which is associated with a particular optimal action.\* The value function associates a distinct linear function of b with each region. Each valueor policy-iteration step refines the boundaries of the regions and might introduce new regions.

The details of the algorithms are beyond the scope of this book, but we will report the solution for the sensorless 4 x 3 world. The optimal policy is the following:

[Left, Up, Up, Right, Up, Up, Right, Up, Up, Right, Up, Right, Up, Right, Up, ...] .

The policy is a sequence because this problem is deterministic in belief state space—there are no observations. The “trick” it embodies is to have the agent move Leff once to ensure that it’s not in (4,1), so that it’s then fairly safe to keep moving Up and Right to reach the +1 exit. The agent reaches the +1 exit 86.6% of the time and does so much faster than the policy given earlier in the section, so its expected utility is 0.38 compared with 0.08.

For more complex POMDPs with observations, finding approximately optimal policies is very difficult (PSPACE-hard, in fact—i.e., very hard indeed). Problems with a few dozen states are often infeasible. The next section describes a different, approximate method for solving POMDPs, one based on look-ahead search.

4 For some POMDPs, the optimal policy has infinitely many regions, so the simple list-of-regions approach fails and more ingenious methods are needed to find even an approximation.

Section 17.5.

Decision-Theoretic Agents 629

DYNAMIC DECISION NETWORK

In this section, we outline a comprehensive approach to agent design for partially observable, stochastic environments. The basic elements of the design are already familiar:

o The transition and observation models are represented by a dynamic Bayesian network (as described in Chapter 15).

e The dynamic Bayesian network is extended with decision and utility nodes, as used in decision networks in Chapter 16. The resulting model is called a dynamic decision network or DDN.

eo A filtering algorithm is used to incorporate each new percept and action and to update the belief state representation.

e Decisions are made by projecting forward possible action sequences and choosing the best one.

The primary advantage of using a dynamic Bayesian network to represent the transition and sensor models is that it decomposes the state description into a set of random variables, much as planning algorithms use logical representations to decompose the state space used by search algorithms. The agent design is therefore a practical implementation of the utilitybased agent sketched in Chapter 2.

Because we are using dynamic Bayesian networks, we will revert to the notation of Chapter 15, where X; referred to the set of state variables for time ¢ and E; referred to the evidence variables. Thus, where we have used s; (the state at time ¢) so far in this chapter, we will now use X;. We will use A; to refer to the action at time ¢, so the transition model T(s,a,s') is the same as P(X;1|X, At) and the observation model O(s, 0) is the same as P(E;|X;). We will use R; to refer to the reward received at time ¢ and Uy to refer to the utility of the state at time ¢{. With this notation, a dynamic decision network looks like the one shown in Figure 17.9.

[]

Figure 17.9 The generic structure of a dynamic decision network. Variables with known values are shaded. The current time is ¢ and the agent must decide what to do—that is, choose a value for A;. The network has been unrolled into the future for three steps and represents future rewards, as well as the utility of the state at the look-ahead horizon.

630

Chapter 17. Making Complex Decisions

D;in P(X; Ey.)

Ep.

Dy in PXyq 1 Epp)

KE.»

Dy; in P(Xpy | Eq0) £3

Ez

UX13) A 7S \ 2

Figure 17.10 Part of the look-ahead solution of the DDN in Figure 17.9.

Dynamic decision networks provide a concise representation for large POMDPs, so they can be used as inputs for any POMDP algorithm including those for valueand policyiteration methods. In this section, we will focus on look-ahead methods that project action sequences forward from the current belief state in much the same way as do the game-playing algorithms of Chapter 6. The network in Figure 17.9 has been projected three steps into the future; the current and future decisions and the future observations and rewards are all unknown. Notice that the network includes nodes for the rewards for X¢y1 and Xo, but the utility for X;,3. This is because the agent must maximize the (discounted) sum of all future rewards, and U (X;3) represents the reward for X;3 and all subsequent rewards. As in Chapter 6, we assume that U is available only in some approximate form: if exact utility values were available, there would be no need for look-ahead beyond depth 1.

Figure 17.10 shows part of the search tree corresponding to the three-step look-ahead DDN in Figure 17.9. Each of the triangular nodes is a belief state in which the agent makes a decision Ay; for i=0,1,2,.... The round nodes correspond to choices by the environment, namely, what observation E;; occurs. Notice that there are no chance nodes corresponding to the action outcomes; this is because the belief state update for an action is deterministic regardless of the actual outcome.

The belief state at each triangular node can be computed by applying a filtering algorithm to the sequence of observations and actions leading to it. In this way, the algorithm takes into account the fact that, for decision A;4;, the agent will have available percepts Eit1, ..., Et, even though at time ¢ it does not know what those percepts will be. In this way, a decision-theoretic agent automatically takes into account the value of information and will execute information-gathering actions where appropriate.

A decision can be extracted from the search tree by backing up the utility values from the leaves, taking an average at the chance nodes and taking the maximum at the decision

Section 17.6.

Decisions with Multiple Agents: Game Theory 631

nodes. This is similar to the EXPECTIMINIMAX algorithm for game trees with chance nodes, except that (1) there can also be rewards at non-leaf states and (2) the decision nodes correspond to belief states rather than actual states. The time complexity of an exhaustive search to depth d is O(|D|\*[E|%), where | D| is the number of available actions and E is the number of possible observations. For problems in which the discount factor + is not too close to 1, a shallow search is often good enough to give near-optimal decisions. It is also possible to approximate the averaging step at the chance nodes, by sampling from the set of possible observations instead of summing over all possible observations. There are various other ways of finding good approximate solutions quickly, but we defer them to Chapter 21.

Decision-theoretic agents based on dynamic decision networks have a number of advantages compared with other, simpler agent designs presented in earlier chapters. In particular, they handle partially observable, uncertainty environments and can easily revise their “plans” to handle unexpected observations. With appropriate sensor models, they can handle sensor failure and can plan to gather information. They exhibit “graceful degradation” under time pressure and in complex environments, using various approximation techniques. So what is missing? The most important defect of our DDN-based algorithm is its reliance on forward search, just like the state-space search algorithms of Part II. In Part IV, we explained how the ability to consider partially ordered, abstract plans via goal-directed search provided a massive increase in problem-solving power, particularly when combined with plan libraries. There have been attempts to extend these methods into the probabilistic domain, but so far they have proven to be inefficient. A second, related problem is the basically propositional nature of the DDN language. We would like to be able to extend some of the ideas for firstorder probabilistic languages in Section 14.6 to the problem of decision making. Current research has shown that this extension is possible and has significant benefits, as discussed in the notes at the end of the chapter.

GAME THEORY

This chapter has concentrated on making decisions in uncertain environments. But what if the uncertainty is due to other agents and the decisions they make? And what if the decisions of those agents are in turn influenced by our decisions? We addressed this question once before, when we studied games in Chapter 6. There, however, we were concerned with turn-taking games with perfect information, for which minimax search can be used to find optimal moves. In this section we study the aspects of game theory that can be used to analyze games with simultaneous moves. To simplify matters, we will look first at games that are only one move long. The word “game” and the simplification to single moves might make this seem trivial, but in fact, game theory is used in very serious decision making situations including bankruptcy proceedings, the auctioning of wireless frequency spectrums, product development and pricing decisions, and national defense, situations involving billions of dollars and hundreds of thousands of lives. Game theory can be used in at least two ways:

632

Chapter 17. Making Complex Decisions

PLAYERS

ACTIONS

PAYOFF MATRIX

[]

PURE STRATEGY

MIXED STRATEGY

STRATEGY PROFILE OUTCOME

1. Agent design: Game theory can analyze the agent’s decisions, and compute the expected utility for each decision (under the assumption that other agents are acting optimally according to game theory). For example, in the game two-finger Morra, two players, O and E, simultaneously display one or two fingers. Let the total number of fingers be f. If f is odd, O collects f dollars from F, and if f is even, I collects f dollars from O. Game theory can determine the best strategy against a rational player and the expected return for each player.’

2. Mechanism Design: When an environment is inhabited by many agents, it might be possible to define the rules of the environment (i.e., the game that the agents must play) so that the collective good of all agents is maximized when each agent adopts the game-theoretic solution that maximizes its own utility. For example, game theory can help design the protocols for a collection of Internet traffic routers so that each router has an incentive to act in such a way that global throughput is maximized. Mechanism design can also be used to construct intelligent multiagent systems that solve complex problems in a distributed fashion without the need for each agent to know about the whole problem being solved.

A game in game theory is defined by the following components:

¢ Players or agents who will be making decisions. Two-player games have received the most attention, although n-player games for n > 2 are also common. We will give players capitalized names, like Alice and Bob or O and FE.

¢ Actions that the players can choose. We will give actions lowercase names, like one or testify. The players may or may not have the same set of actions available.

¢ A payoff matrix that gives the utility to each player for each combination of actions by all the players. The payoff matrix for two-finger Morra is as follows:

O: one 0: two E:one|E=20=-2|E=-30=3 E:two|E=-3,0=3|E=4,0=—4 For example, the lower-right corner shows that when O chooses action two and E also chooses two, the payoff is 4 for E and —4 for O.

Each player in a game must adopt and then execute a strategy (which is the name used in game theory for a policy). A pure strategy is a deterministic policy specifying a particular action to take in each situation; for a one-move game, a pure strategy is just a single action. The analysis of games leads to the idea of a mixed strategy, which is a randomized policy that selects particular actions according to a specific probability distribution over actions. The mixed strategy that chooses action a with probability p and action b otherwise is written [p:a; (1 — p):b]. For example, a mixed strategy for two-finger Morra might be [0.5: one; 0.5: two]. A strategy profile is an assignment of a strategy to each player; given the strategy profile, the game’s outcome is a numeric value for each player.

5 Morra is a recreational version of an inspection game. In such games, an inspector chooses a day to inspect a

facility (such as a restaurant or a biological weapons plant), and the facility operator chooses a day to hide all the nasty stuff. The inspector wins if the days are different and the facility operator wins if they are the same.

Decisions with Multiple Agents: Game Theory 633

SOLUTION

PRISONER'S DILEMMA

[]

DOMINANT STRATEGY

STRONGLY DOMINATES

WEAKLY DOMINATES

PARETO OPTIMAL PARETO DOMINATED

DOMINANT STRATEGY EQUILIBRIUM EQUILIBRIUM

A solution to a game is a strategy profile in which each player adopts a rational strategy. We will see that the most important issue in game theory is to define what “rational” means when each agent chooses only part of the strategy profile that determines the outcome. It is important to realize that outcomes are actual results of playing a game, while solutions are theoretical constructs used to analyze a game. We will see that some games only have a solution in mixed strategies. But that does not mean that a player must literally be adopting a mixed strategy to be rational.

Consider the following story: Two alleged burglars, Alice and Bob, are caught redhanded near the scene of a burglary and are interrogated separately by the police. Both know that if they both confess to the crime, they will each serve 5 years in prison for burglary, but if both refuse to confess, they will serve only 1 year each for the lesser charge of possessing stolen property. However, the police separately offer each a deal: if you testify against your partner as the leader of a burglary ring, you'll go free, while the partner will serve 10 years. Now Alice and Bob face the so-called prisoner’s dilemma: should they testify or refuse? Being rational agents, Alice and Bob each want to maximize their own expected utility. Let’s assume that Alice is callously unconcerned about her partner’s fate, so her utility decreases in proportion to the number of years she will spend in prison, regardless of what happens to Bob. Bob feels exactly the same way. To help reach a rational decision, they both construct the following payoff matrix:

Alice analyzes the payoff matrix as follows: Suppose Bob testifies. Then I get 5 years if 1 testify and 10 years if I don’t, so in that case testifying is better. On the other hand, if Bob refuses, then I get 0 years if I testify and 1 year if I refuse, so in that case as well testifying is better. So in either case, it’s better for me to testify, so that’s what I must do.

Alice has discovered that testify is a dominant strategy for the game. We say that a strategy s for player p strongly dominates strategy s’ if the outcome for s is better for p than the outcome for s', for every choice of strategies by the other players. Strategy s weakly dominates s’ if s is better than s’ on at least one strategy profile and no worse on any other. A dominant strategy is a strategy that dominates all others. It is irrational to play a strongly dominated strategy, and irrational not to play a dominant strategy if one exists. Being rational, Alice chooses the dominant strategy. We need just a bit more terminology before we go on: we say that an outcome is Pareto optimal® if there is no other outcome that all players would prefer. An outcome is Pareto dominated by another outcome if all players would prefer the other outcome.

If Alice is clever as well as rational, she will continue to reason as follows: Bob’s dominant strategy is also to testify. Therefore, he will testify and we will both get five years. When each player has a dominant strategy, the combination of those strategies is called a dominant strategy equilibrium. In general, a strategy profile forms an equilibrium if no

6 Pareto optimality is named after the economist Vilfredo Pareto (1848-1923).

634

Chapter 17. Making Complex Decisions

NASH EQUILIBRIUM

[]

player can benefit by switching strategies, given that every other player sticks with the same strategy. An equilibrium is essentially a local optimum in the space of policies; it is the top of a peak that slopes downward along every dimension, where a dimension corresponds to a player’s strategy choices.

The dilemma in the prisoner’s dilemma is that the outcome of the equilibrium point is worse for both players than the outcome they would get if they both refused to testify. In other words, the outcome for the equilibrium solution is Pareto dominated by the (-1, -1) outcome of (refuse, refuse).

Is there any way for Alice and Bob to arrive at the (-1, -1) outcome? It is certainly an allowable option for both of them to refuse to testify, but is is an unlikely option. Either player contemplating playing refuse will realize that he or she would do better by playing testify. That is the attractive power of an equilibrium point.

The mathematician John Nash (1928-) proved that every game has an equilibrium of the type defined here. It is now called a Nash equilibrium in his honor. Clearly, a dominant strategy equilibrium is a Nash equilibrium (Exercise 17.9), but not all games have dominant strategies. Nash's theorem means that there are equilibrium strategies even when there is no dominant strategy.

For the prisoner’s dilemma, only the strategy profile (testify, testify) is a Nash equilibrium. It is hard to see how rational players can avoid this outcome, because in any proposed non-equilibrium solution at least one of the players will be tempted to change strategies. Game theorists agree that being a Nash equilibrium is a necessary condition for being a solution—although they disagree whether it is a sufficient condition.

It is easy enough to avoid the (testify, testify) solution if we change the game (or the players) in some way. For example, we could change to an iterated game in which the players know that they will meet again (but crucially, they must be uncertain about how many times they will meet again). Or if the agents have moral beliefs that encourage cooperation and fairness, we could change the payoff matrix to reflect the utility to each agent of cooperating with the other. We will see later that changing the agents to have limited computational powers, rather than the ability to reason absolutely rationally, can also affect the outcome, as can telling one agent that the other has limited rationality.

Now, let’s look at a game that has no dominant strategy. Acme, a video game hardware manufacturer, has to decide whether its next game machine will use DVDs or CDs. Meanwhile, the video game software producer Best needs to decide whether to produce its next game on DVD or CD. The profits for both will be positive if they agree and negative if they disagree, as shown in the following payoff matrix:

There is no dominant strategy equilibrium for this game, but there are two Nash equilibria: (dvd, dvd) and (cd, cd). We know these are Nash equilibria because, if either player unilaterally moves to a different strategy, that player will be worse off. Now the agents have a problem: there are multiple acceptable solutions, but if each agent chooses a different

Section 17.6.

Decisions with Multiple Agents: Game Theory 635

COORDINATION GAME

ZERO-SUM GAME

MAXIMIN

solution then the resulting strategy profile won't be a solution at all and both agents will suffer. How can they agree on a solution? One answer is that both should choose the Paretooptimal solution (dvd, dvd); that is, we can restrict the definition of “solution” to the unique Pareto-optimal Nash equilibrium provided that one exists. Every game has at least one Paretooptimal solution, but a game might have several, or they might not be equilibrium points. For example, we could set the payoffs for (dvd, dvd) to 5 instead of 9. In that case, there are two equal Pareto-optimal equilibrium points. To choose between them the agents can either guess or communicate, which can be done either by establishing a convention that orders the solutions before the game begins or by negotiating to reach a mutually beneficial solution during the game (which would mean including communicative actions as part of a multimove game). Communication thus arises in game theory for exactly the same reasons that it arose in multiagent planning in Chapter 12. Games in which players need to communicate like this are called coordination games.

We have seen that a game can have more than one Nash equilibrium; how do we know that every game must have at least one? It can be that a game has no pure-strategy Nash equilibria. Consider, for example, any pure strategy profile for two-finger Morra (page 632). If the total number of fingers is even then O will want to switch; if the total is odd then FE will want to switch. Therefore no pure strategy profile can be an equilibrium and we must look to mixed strategies.

But which mixed strategy? In 1928, von Neumann developed a method for finding the optimal mixed strategy for two-player, zero-sum games. A zero-sum game is a game in which the payoffs in each cell of the payoff matrix sum to zero.” Clearly, Morra is such a game. For two-player, zero-sum games, we know that the payoffs are equal and opposite, so we need consider the payoffs of only one player, who will be the maximizer (just as in Chapter 6). For Morra, we pick the even player E to be the maximizer, so we can define the payoff matrix by the values Ug (e, 0)—the payoff to E if E does e and O does o.

Von Neumann’s method is called the the maximin technique, and it works as follows:

e Suppose we change the rules to force FE to reveal his or her strategy first, followed by O. Then we have a turn-taking game to which we can apply the standard minimax algorithm from Chapter 6. Let's suppose this gives an outcome Up o. Clearly, this game favors O, so the true utility U of the game (from E’s point of view) is at least Ugo. For example, if we just look at pure strategies, the minimax game tree has a root value of —3 (see Figure 17.11(a)), so we know that U > —3.

e Now suppose we change the rules to force O to reveal his or her strategy first, followed by E. Then the minimax value of this game is Upp, and because this game favors Ef we know that U is at most Up, g. With pure strategies, the value is +2 (see Figure 17.11(b)), so we know U < +2.

7 More general is the concept of constant-sum games, in which the sum of every cell in the game adds up to a constant, c. An n-person constant-sum game can be turned into a zero-sum game by subtracting ¢/n from every payoff. Thus chess, with traditional payoff of 1 for a win, 1/2 for a draw, and 0 for a loss, is technically a constant-sum game with ¢ = 1, but can easily be transformed into a zero-sum game by subtracting 1/2 from every payoff.

636

Chapter 17. Making Complex Decisions

MAXIMIN EQUILIBRIUM

Combining these two arguments, we see that the true utility I of the solution must satisfy Ugo

To pinpoint the value of IU, we need to turn our analysis to mixed strategies. First, observe the following: once the first player has revealed his or her strategy, the second player cannot lose by playing a pure strategy. The reason is simple: if the second player plays a mixed strategy, [p: one; (1 — p): two], its expected utility is a linear combination (p%ene + (1 — Pp) Uso) Of the utilities of the pure strategies, woe and t4y,. This linear combination can never be better than the best of %,pne and sy, sO the second player might as well play a pure strategy.

With this observation in mind, the minimax trees can be thought of as having infinitely many branches at the root, corresponding to the infinitely many mixed strategies the first player can choose. Each of these leads to a node with two branches corresponding to the pure strategies for the second player. We can depict these infinite trees finitely by having one “parameterized” choice at the root:

e If E moves first, the situation is as shown in Figure 17.11(c). FE plays [p: one; (1 — p): two] at the root, and then O chooses a move given the value of p. If O chooses one, the expected payoff (to E) is 2p — 3(1 — p) = 5p — 3; if O chooses two, the expected payoff is —3p + 4(1 — p) =4 — 7p. We can draw these two payoffs as straight lines on a graph, where p ranges from 0 to 1 on the z-axis, as shown in Figure 17.11(e). O, the minimizer, will always choose the lower of the two lines, as shown by the heavy lines in the figure. Therefore, the best that FE can do at the root is to choose p to be at the intersection point, which is where

bp—3=4-"Tp = p="T/12. The utility for FE at this pointis Ugo = — 1/12.

o If O moves first, the situation is as shown in Figure 17.11(d). O plays [q: one; (1 — q): two) at the root, and then FE chooses a move given the value of ¢. The payoffs are 2g — 3(1 — q)=5q — 3 and —3q + 4(1 — ¢) =4 — 7¢q.8 Again, Figure 17.11(f) shows that the best O can do at the root is to choose the intersection point:

bg—3=4-Tq = q="7/12. The utility for E at this point is Up p= — 1/12.

Now we know that the true utility of the game lies between —1/12 and —1/12, that is, it is exactly —1/12! (The moral is that it is better to be O than E if you are playing this game.) Furthermore, the true utility is attained by the mixed strategy [7/12: one; 5/12: two], which should be played by both players. This strategy is called the maximin equilibrium of the game, and is a Nash equilibrium. Note that each component strategy in an equilibrium mixed strategy has the same expected utility. In this case, both one and two have the same expected utility, —1/12, as the mixed strategy itself.

Our result for two-finger Morra is an example of the general result by von Neumann; every two-player zero-sum game has a maximin equilibrium when you allow mixed strategies.

8 It is a coincidence that these equations are the same as those for p; the coincidence arises because Ug (one, two) = Ug (two, one) = — 3. This also explains why the optimal strategy is the same for both players.

Section 17.6.

637

(a E

© E

[p: one; (1 — p): two]

(b) O

d o

lg: one; (1 — gq): two]

Oo E one wo one wo 2p-31-p) 3p+4l-p) 2g-31-¢q9) 3g+41-q ee) vu ® vu +4 1 ' +4 4 oN 1 i B34 0 ! +3 4 “2 ied Pp 2 wo | \, s | |

0

-1 4

2

| ~S L Le | I \, \ N N \ ~

3 34

Figure 17.11 (a) and (b): Minimax game trees for two-finger Morra if the players take turns playing pure strategies. (c) and (d): Parameterized game trees where the first player plays a mixed strategy. The payoffs depend on the probability parameter (p or ¢) in the mixed strategy. (€) and (f): For any particular value of the probability parameter, the second player will choose the “better” of the two actions, so the value of the first player's mixed strategy is given by the heavy lines. The first player will choose the probability parameter for the mixed strategy at the intersection point.

Furthermore, every Nash equilibrium in a zero-sum game is a maximin for both players. The general algorithm for finding maximin equilibria in zero-sum games is somewhat more involved than Figures 17.11(e) and (f) might suggest. When there are n possible actions, a mixed strategy is a point in n-dimensional space and the lines become hyperplanes. It’s

638

Chapter 17. Making Complex Decisions

REPEATED GAME

also possible for some pure strategies for the second player to be dominated by others, so that they are not optimal against any strategy for the first player. After removing all such strategies (which might have to be done repeatedly), the optimal choice at the root is the highest (or lowest) intersection point of the remaining hyperplanes. Finding this choice is an example of a linear programming problem: maximizing an objective function subject to linear constraints. Such problems can be solved by standard techniques in time polynomial in the number of actions (and in the number of bits used to specify the reward function, if you want to get technical).

The question remains, what should a rational agent actually do in playing a single game of Morra? The rational agent will have derived the fact that [7/12: one; 5/12: two] is the maximin equilibrium strategy, and will assume that this is mutual knowledge with a rational opponent. The agent could use a 12-sided die or a random number generator to pick randomly according to this mixed strategy, in which case the expected payoff would be -1/12 for E. Or the agent could just decide to play one, or two. In either case, the expected payoff remains -1/12 for E. Curiously, unilaterally choosing a particular action does not harm one’s expected payoff, but allowing the other agent to know that one has made such a unilateral decision does affect the expected payoff, because then the opponent can adjust his strategy accordingly.

Finding solutions to non-zero-sum finite games (i.e., Nash equilibria) is somewhat more complicated. The general approach has two steps: (1) Enumerate all possible subsets of actions that might form mixed strategies. For example, first try all strategy profiles where each player uses a single action, then those where each player uses either one or two actions, and so on. This is exponential in the number of actions, and so only applies to relatively small games. (2) For each strategy profile enumerated in (1), check to see if it is an equilibrium. This is done by solving a set of equations and inequalities that are similar to the ones used in the zero-sum case. For two players these equations are linear and can be solved with basic linear programming techniques, but for three or more players they are nonlinear and may be very difficult to solve.

So far we have looked only at games that last a single move. The simplest kind of multiple-move game is the repeated game, in which players face the same choice repeatedly, but each time with knowledge of the history of all players’ previous choices. A strategy profile for a repeated game specifies an action choice for each player at each time step for every possible history of previous choices. As with MDPs, payoffs are additive over time.

Let’s consider the repeated version of the prisoner’s dilemma. Will Alice and Bob work together and refuse to testify, knowing that they will meet again? The answer depends on the details of the engagement. For example, suppose Alice and Bob know that they must play exactly 100 rounds of prisoner’s dilemma. Then they both know that the 100th round will not be a repeated game—that is, its outcome can have no effect on future rounds—and therefore they will both choose the dominant strategy, testify, in that round. But once the 100th round is determined, the 99th round can have no effect on subsequent rounds, so it too will have a dominant strategy equilibrium at (testify, testify). By induction, both players will choose testify on every round, earning a total jail sentence of 500 years each.

We can get different solutions by changing the rules of the interaction. For example, suppose that after each round there is a 99% chance that the players will meet again. Then

Section 17.6.

Decisions with Multiple Agents: Game Theory 639

PERPETUAL PUNISHMENT

TITFOR-TAT

the expected number of rounds is still 100, but neither player knows for sure which round will be the last. Under these conditions, more cooperative behavior is possible. For example, one equilibrium strategy is for each player to refuse unless the other player has ever played testify. This strategy could be called perpetual punishment. Suppose both players have adopted this strategy, and this is mutual knowledge. Then as long as neither player has played testify, then at any point in time the expected future total payoff for each player is

> 0.99 (1) = —100. t=0

A player who chooses testify will gain a score of 0 rather than —1 on the very next move, but his or her total expected future payoff becomes

oo 0+) 0.99 (-10) = —99.

t=1 Therefore, at every step, there is no incentive to deviate from (refuse, refuse). Perpetual punishment is the “mutually assured destruction” strategy of the prisoner’s dilemma: once either player decides to testify, it assures that both players suffer a great deal. But it only works as a deterrent if the other player believes you have adopted this strategy—or at least that you might have adopted it.

There are other strategies that are more forgiving. The most famous, called tit-fortat, calls for starting with refuse and then echoing the other player’s previous move on all subsequent moves. So Alice would refuse as long as Bob refuses and would testify the move after Bob testified, but would go back to refusing if Bob did. Although very simple, this strategy has proven to be highly robust and effective against a wide variety of strategies.

We can also get different solutions by changing the agents, rather than changing the rules of engagement. Suppose the agents are finite-state machines with n states and they are playing a game with m > n total steps. The agents are thus incapable of representing the number of remaining steps, and must treat it as an unknown. Therefore they cannot do the induction, and are free to arrive at the more favorable (refuse, refuse) equilibrium. In this case, ignorance is bliss—or rather, having your opponent believe that you are ignorant is bliss. Your success in these repeated games depends on the other player’s perception of you as a bully or a simpleton, and not on your actual characteristics.

Repeated games in full generality are beyond the scope of this book, but they arise in many settings. For example, we can construct a sequential game by putting two agents in the 4 x 3 world of Figure 17.1. If we specify that no movement occurs when the two agents try to move into the same square simultaneously (a common problem at many traffic intersections), then certain pure strategies can get stuck forever. One solution is for each agent to randomize its choice between moving forward and staying put; the stalemate will be resolved quickly and both agents will be happy. This is exactly what is done to resolve packet collisions in Ethernet networks.

Currently known solution methods for repeated games resemble those for turn-taking games in Chapter 6, in that a game tree can be constructed from the root downwards and solved from the leaves upwards. The main difference is that, instead of simply taking the

640

Chapter 17. Making Complex Decisions

PARTIAL INFORMATION

BAYES-NASH EQUILIBRIUM

maximum or minimum of the child values, the algorithm must solve a game in mixed strategies at each level, assuming that the child nodes have been solved and have well-defined values to work with.

Repeated games in partially observable environments are called games of partial information. Examples include card games such as poker and bridge, wherein each player can see only a subset of the cards, and more serious “games” such as abstractions of nuclear war, where neither side knows the location of all its opponent’s weapons. Games of partial information are solved by considering a tree of belief states, as in POMDPs. (See Section 17.4.) One important difference is that, while one’s own belief state is observable, the opponent’s belief state is not. Only recently have practical algorithms been developed for such games. Some simplified versions of poker have been solved, proving that bluffing is indeed a rational choice, as part of a well balanced mixed strategy. One important insight to emerge from such studies is that mixed strategies are useful not just for making one’s actions unpredictable, but also for minimizing the amount of information that one’s opponent can learn from observing one’s actions. It is interesting that, although designers of programs for playing bridge are well aware of the importance of gathering and hiding information, none has yet proposed the use of randomized strategies.

So far, there have been some barriers that have prevented game theory from being widely used in agent design. First, note that in a Nash equilibrium solution, a player is assuming that the opponents will definitely play the equilibrium strategy. This means that the player is unable to incorporate any beliefs it might have about how the other players are likely to act, and therefore that it might be wasting some of its value defending against threats that will never materialize. The notion of a Bayes—Nash equilibrium partially addresses this point: it is an equilibrium with respect to a player’s prior probability distribution over the other players’ strategies—in other words, it expresses a player’s beliefs about the other players’ likely strategies. Second, there is currently no good way to combine game theoretic and POMDP control strategies. Because of these and other problems, game theory has been used primarily to analyze environments that are at equilibrium, rather than to control agents within an environment. We shall soon see how it can help design environments.

MECHANISM DESIGN

In the previous section, we looked at the question “Given a game, what is a rational strategy?” In this section, we ask “Given that agents are rational, what game should we design?” More specifically, we would like to design a game whose solutions, consisting of each agent pursuing its own rational strategy, result in the maximization of some global utility function. This problem is called mechanism design, or sometimes inverse game theory. Mechanism design is a staple of economics and political science. For collections of agents, it holds the possibility of using game-theoretic mechanisms to construct smart systems out of a collection of more limited systems-—even noncooperative systems—in much the same way that teams of humans can achieve goals far beyond the reach of any individual.

Section 17.7.

Mechanism Design 641

MECHANISM

TRAGEDY OF THE COMMONS

EXTERNALITIES

ENGLISH AUCTION

Examples of mechanism design include auctioning off cheap airline tickets, routing TCP packets between computers, deciding how medical interns will be assigned to hospitals, and deciding how robotic soccer players will cooperate with their teammates. Mechanism design became more than an academic subject in the 1990s when several nations, faced with the problem of auctioning off licenses to broadcast in various frequency bands, lost hundreds of millions of dollars in potential revenue as a result of poor mechanism design. Formally, a mechanism consists of (1) a language for describing the (possibly infinite) set of allowable strategies that agents may adopt and (2) an outcome rule G that determines the payoffs to the agents given a strategy profile of allowable strategies.

At first sight, the mechanism design problem can seem trivial. Suppose that the global utility function U is decomposed into any set of individual agent utility functions U;, such that U = }\_, U;. Then, one might say, if each agent maximizes its own utility, surely that will lead automatically to the maximization of the global utility. (For example, Capitalism 101 says that if everyone tries to get rich, the total wealth of society will increase.) Unfortunately, this doesn’t work. The actions of each agent could affect the well-being of other agents in ways that decrease global utility. One example of this is the tragedy of the commons, a situation in which individual farmers all bring their livestock to graze for free on the town commons, with the result being the destruction of the commons and a negative utility for all the farmers. Each farmer individually acted rationally, reasoning that the use of the commons was free and that, although using the commons could lead to its destruction, refraining from using it would not help (because the others would use it anyway). Similar arguments apply to the use of the atmosphere and the oceans for free dumping of pollutants.

A standard approach in mechanism design for dealing with such problems is to charge each agent for using the commons. More generally, we need to ensure that all externalities— effects on global utility that are not recognized in the individual agents’ transactions—are made explicit. Setting the prices correctly is the difficult part. In the limit, this approach amounts to creating a mechanism in which each agent is effectively required to maximize global utility. This is an impossibly difficult task for the agent, who can neither assess the current state of the world nor observe the effects of its actions on all other agents. Mechanism design therefore concentrates on finding mechanisms for which the decision problem for the individual agents is straightforward.

Let’s consider auctions first. In the most common form, an auction is a mechanism for selling some goods to members of a pool of bidders. The strategies are the bids and the outcome determines who gets the goods and how much they pay. One example of where auctions can come into play within Al is when a collection of agents are deciding whether to cooperate on a joint plan. Hunsberger and Grosz (2000) show that this can be accomplished efficiently with an auction in which the agents bid for roles in the joint plan.

For now, we'll consider auctions wherein (1) there is a single good, (2) each bidder has a utility value v; for the good, and (3) these values are known only to the bidder. Bidders make bids b;, and the highest bid wins the goods, but the mechanism determines how the bids are made and the price paid by the winner (it need not be b;). The best-known type of auction is the English auction, in which the auctioneer increments the price of the goods, checking whether bidders are still interested, until only one bidder is left. This mechanism

642

Chapter 17. Making Complex Decisions

STRATEGY-PROOF

SEALED BID AUCTION

SEALED BID SECOND-PRICE AUCTION

VICKREY AUCTION

has the property that the bidder with the highest value v; gets the goods at a price of b,,, + d, where b,, is the highest bid among all the other players and d is the auctioneer’s increment between bids.” The English auction also has the property that bidders have a simple dominant strategy: keep bidding as long as the current cost is below your personal value. Recall that “dominant” means that the strategy works against all other strategies, which in turn means that a player can adopt it without regard for the other strategies. Therefore, players don’t have to waste time and energy contemplating other players’ possible strategies. A mechanism where players have a dominant strategy that involves revealing their true incentives is called a strategy-proof mechanism.

One negative property of the English auction is its high communication costs, so either the auction takes place in one room or all bidders have to have high-speed, secure communication lines. An alternative mechanism that requires much less communication is the sealed bid auction. Here, each bidder makes a single bid and communicates it to the auctioneer, and the highest bid wins. With this mechanism, the strategy of bidding your true value is no longer dominant. If your value is v; and you believe that the maximum of all the other players’ bids will be b,,, then you should bid the lower of v; and b,,, + €. Two drawbacks of the sealed bid auction are that the player with the highest v; might not get the goods and that players must spend effort contemplating the other players’ strategies.

A small change in the rules for sealed bid auctions produces the sealed bid secondprice auction, also known as a Vickrey auction.!® In such auctions, the winner pays the price of the second highest bid, rather than paying his own bid. This simple modification completely eliminates the complex deliberations required for standard (or first-price) sealed bid auctions, because the dominant strategy is now to bid your actual value. To see that, we note that any player can think of the auction as a two-player game, ignoring all players except himself and the highest bidder among the other players. The utility of player { in terms of his bid b;, his value v;, and the best bid among the other players, b,,, is

[]

0 otherwise. To see that b; = v; is a dominant strategy, note that when (v; — by) is positive, any bid that wins the auction is optimal, and bidding v; in particular wins the auction. On the other hand, when (v; — by,) is negative, any bid that loses the auction is optimal, and bidding v; in particular loses the auction. So bidding v; is optimal for all possible values of b,,, and in fact, v; is the only bid that has this property. Because of its simplicity and the minimal computation requirements for both seller and bidders, the Vickrey auction is widely used in constructing distributed Al systems.

Now let’s consider the Internet routing problem. The players correspond to edges in the graph of network connections. Each player knows the cost ¢; of sending a message along its own edge; the cost of not having a message to send is 0. The goal is to find the cheapest path for a message to travel from origin to destination, where the cost of the whole route

9 There is actually a small chance that the player with highest v; fails to get the goods, in the case where bm < v; < bm + d. The chance of this happening can be made arbitrarily small by decreasing the increment d. 10 Named after William Vickrey (1914-1996), winner of the 1996 Nobel prize in economics.

Section 17.8.

Summary 643

is the sum of the individual edge costs. Chapter 4 gives several algorithms for computing the shortest path, given the edge costs, so all we have to do is get each agent to report its true cost, ¢;. Unfortunately, if we just ask each agent, it will report costs that are high, to encourage us to send traffic elsewhere. We need to develop a strategy-proof mechanism. One such mechanism is to pay each player a payoff p; equal to the length of the shortest path that does not contain the ith edge minus the length of the shortest path (as computed by a search algorithm) where the cost of the ith edge is assumed to be 0:

p; = LENGTH(path with ¢; = co) — LENGTH(path with ¢; = 0) .

We can show that, under this mechanism, the dominant strategy for each player is to report ¢; truthfully and that doing so will result in a cheapest path. Despite this desirable property, the mechanism outlined here is not used in practice, because of the high communication and central computation cost. The mechanism designer must communicate with all n players and then must solve the optimization problem. This might be worth it if the costs could be amortized over many messages, but in a real network the costs ¢; would fluctuate constantly, because of traffic congestion and because some machines will crash and others will come online. No completely satisfactory solution has yet been devised.

This chapter shows how to use knowledge about the world to make decisions even when the outcomes of an action are uncertain and the rewards for acting might not be reaped until many actions have passed. The main points are as follows:

e Sequential decision problems in uncertain environments, also called Markov decision processes, or MDPs, are defined by a transition model specifying the probabilistic outcomes of actions and a reward function specifying the reward in each state.

e The utility of a state sequence is the sum of all the rewards over the sequence, possibly discounted over time. The solution of an MDP is a policy that associates a decision with every state that the agent might reach. An optimal policy maximizes the utility of the state sequences encountered when it is executed.

e The utility of a state is the expected utility of the state sequences encountered when an optimal policy is executed, starting in that state. The value iteration algorithm for solving MDPs works by iteratively solving the equations relating the utilities of each state to that of its neighbors.

¢ Policy iteration alternates between calculating the utilities of states under the current policy and improving the current policy with respect to the current utilities.

e Partially observable MDPs, or POMDPs, are much more difficult to solve than are MDPs. They can be solved by conversion to an MDP in the continuous space of belief states. Optimal behavior in POMDPs includes information gathering to reduce uncertainty and therefore make better decisions in the future.

644

Chapter 17. Making Complex Decisions

¢ A decision-theoretic agent can be constructed for POMDP environments. The agent uses a dynamic decision network to represent the transition and observation models, to update its belief state, and to project forward possible action sequences.

¢ Game theory describes rational behavior for agents in situations where multiple agents interact simultaneously. Solutions of games are Nash equilibria—strategy profiles in which no agent has an incentive to deviate from the specified strategy.

¢ Mechanism design can be used to. set the rules by which agents will interact, in order to maximize some global utility through the operation of individually rational agents. Sometimes, mechanisms exist that achieve this goal without requiring each agent to consider the choices made by other agents.

We shall return to the world of MDPs and POMDP in Chapter 21, when we study reinforcement learning methods that allow an agent to improve its behavior from experience in sequential, uncertain environments.

Richard Bellman (1957) initiated the modern approach to sequential decision problems and proposed the dynamic programming approach in general and the value iteration algorithm in particular. Ron Howard’s Ph.D. thesis (1960) introduced policy iteration and the idea of average reward for solving infinite-horizon problems. Several additional results were introduced by Bellman and Dreyfus (1962). Modified policy iteration is due to van Nunen (1976) and Puterman and Shin (1978). Asynchronous policy iteration was analyzed by Williams and Baird (1993), who also proved the policy loss bound in Equation (17.9). The analysis of discounting in terms of stationary preferences is due to Koopmans (1972). The texts by Bertsekas (1987), Puterman (1994), and Bertsekas and Tsitsiklis (1996) provide a rigorous introduction to sequential decision problems. Papadimitriou and Tsitsiklis (1987) describe results on the computational complexity of MDPs,

Seminal work by Sutton (1988) and Watkins (1989) on reinforcement learning methods for solving MDPs played a significant role in introducing MDPs into the AI community, as did the later survey by Barto er al. (1995). (Earlier work by Werbos (1977) contained many similar ideas, but was not taken up to the same extent.) The connection between MDPs and Al planning problems was made first by Sven Koenig (1991), who showed how probabilistic STRIPS operators provide a compact representation for transition models. (See also Wellman (1990b).) Work by Dean et al. (1993) and Tash and Russell (1994) attempted to overcome the combinatorics of large state spaces by using a limited search horizon and abstract states. Heuristics based on the value of information can be used to select areas of the state space where a local expansion of the horizon will yield a significant improvement in decision quality. Agents using this approach can tailor their effort to handle time pressure and generate some interesting behaviors such as using familiar “beaten paths” to find their way around the state space quickly without having to recompute optimal decisions at each point. Recent work by Boutilier and others (Boutilier er al., 2000, 2001) has focused on dynamic program

Section 17.8.

Summary 645

ming using symbolic representations of both transition models and value functions, based on propositional and first-order formule.

The observation that a partially observable MDP can be transformed into a regular MDP by using the belief states is due to Astrom (1965). The first complete algorithm for the exact solution of partially-observable Markov decision processes (POMDPs) was proposed by Edward Sondik (1971) in his Ph.D. thesis. (A later journal paper by Smallwood and Sondik (1973) contains some errors, but is more accessible.) Lovejoy (1991) surveys the state of the art in POMDPs. The first significant contribution within AI was the Witness algorithm (Cassandra er al., 1994; Kaelbling et al., 1998), an improved version of POMDP value iteration. Other algorithms soon followed, including an approach due to Hansen (1998) that constructs a policy incrementally in the form of a finite-state automaton. In this policy representation, the belief state corresponds directly to a particular state in the automaton. Approximately optimal policies for POMDPs can be constructed by forward search combined with sampling of possible observations and action outcomes (Kearns ef al., 2000; Ng and Jordan, 2000). Additional work on POMDP algorithms is covered in Chapter 21.

The basic ideas for an agent architecture using dynamic decision networks were proposed by Dean and Kanazawa (1989a). The book Planning and Control by Dean and Wellman (1991) goes into much greater depth, making connections between DBN/DDN models and the classical control literature on filtering. Tatman and Shachter (1990) showed how to apply dynamic programming algorithms to DDN models. Russell (1998) explains various ways in which such agents can be scaled up and identifies a number of open research issues.

The early roots of game theory can be traced back to proposals made in the 17th century by Christiaan Huygens and Gottfried Leibniz to study competitive and cooperative human interactions scientifically and mathematically. Throughout the 19th century, several leading economists created simple mathematical examples to analyze particular examples of competitive situations. The first formal results in game theory are due to Zermelo (1913) (who had, the year before, suggested a form of minimax search for games, albeit an incorrect one). Emile Borel (1921) introduced the notion of a mixed strategy. John von Neumann (1928) proved that every two-person, zero-sum game has a maximin equilibrium in mixed strategies and a well-defined value. Von Neumann's collaboration with the economist Oskar Morgenstern led to the publication in 1944 of the Theory of Games and Economic Behavior, the defining book for game theory. Publication of the book was delayed by the wartime paper shortage until a member of the Rockefeller family personally subsidized its publication.

In 1950, at the age of 21, John Nash published his ideas concerning equilibria in general games. His definition of an equilibrium solution, although originating in the work of Cournot (1838), became known as Nash equilibrium. After a long delay due to the schizophrenia he suffered from 1959 onwards, Nash was awarded the Nobel prize in Economics (along with Reinhart Selten and John Harsanyi) in 1994,

The Bayes—Nash equilibrium is described by Harsanyi (1967) and discussed by Kadane and Larkey (1982). Some issues in the use of game theory for agent control are covered by Binmore (1982).

The prisoner’s dilemma was invented as a classroom exercise by Albert W. Tucker in 1950 and is covered extensively by Axelrod (1985). Repeated games were introduced by

646

Chapter 17. Making Complex Decisions

Luce and Raiffa (1957) as were games of partial information by Kuhn (1953). The first practical algorithm for partial information games was developed within Al by Koller et al. (1996); the paper by Koller and Pfeffer (1997) provides a readable introduction to the general

theory and MDPs are combined in the theory of Markov games (Littman, 1994). Shapley (1953) actually described the value iteration algorithm before Bellman, but his results were not widely appreciated, perhaps because they were presented in the context of Markov games. Textbooks on game theory include those by Myerson (1991), Fudenberg and Tirole (1991), and Osborne and Rubinstein (1994).

The tragedy of the commons, a motivating problem for the field of mechanism design, was presented by Hardin (1968). Hurwicz (1973) created a mathematical foundation for mechanism design. Milgrom (1997) writes about the multibillion-dollar spectrum auction mechanism he designed. Auctions can also be used in planning (Hunsberger and Grosz, 2000) and scheduling (Rassenti ef al., 1982). Varian (1995) gives a brief overview with connections to the computer science literature, and Rosenschein and Zlotkin (1994) present a book-length treatment with applications to distributed Al. Related work on distributed Al also goes under other names, including Collective Intelligence (Tumer and Wolpert, 2000) and market-based control (Clearwater, 1996). Papers on computational issues in auctions often appear in the ACM Conferences on Electronic Commerce.

EXERCISES

17.1 For the 4 x 3 world shown in Figure 17.1, calculate which squares can be reached from (1,1) by the action sequence [Up, Up, Right, Right, Right] and with what probabilities. Explain how this computation is related to the task of projecting a hidden Markov model.

17.2 Suppose that we define the utility of a state sequence to be the maximum reward obtained in any state in the sequence. Show that this utility function does not result in stationary preferences between state sequences. Is it still possible to define a utility function on states such that MEU decision making gives optimal behavior?

17.3 Can any finite search problem be translated exactly into a Markov decision problem such that an optimal solution of the latter is also an optimal solution of the former? If so, explain precisely how to translate the problem and how to translate the solution back; if not, explain precisely why not (i.e., give a counterexample).

17.4 Consider an undiscounted MDP having three states, (1, 2, 3), with rewards —1, —2,0 respectively. State 3 is a terminal state. In states 1 and 2 there are two possible actions: a and b. The transition model is as follows:

e In state 1, action a moves the agent to state 2 with probability 0.8 and makes the agent stay put with probability 0.2.

¢ In state 2, action a moves the agent to state 1 with probability 0.8 and makes the agent stay put with probability 0.2.

Section 17.8.

Summary 647

¢ In either state 1 or state 2, action b moves the agent to state 3 with probability 0.1 and makes the agent stay put with probability 0.9.

Answer the following questions:

a. What can be determined qualitatively about the optimal policy in states 1 and 2?

b. Apply policy iteration, showing each step in full, to determine the optimal policy and the values of states 1 and 2. Assume that the initial policy has action b in both states.

¢. What happens to policy iteration if the initial policy has action a in both states? Does discounting help? Does the optimal policy depend on the discount factor?

17.5 Sometimes MDPs are formulated with a reward function R(s,a) that depends on the action taken or a reward function R(s, a, s) that also depends on the outcome state.

a. Write the Bellman equations for these formulations.

b. Show how an MDP with reward function R(s, a, s') can be transformed into a different MDP with reward function R(s, a), such that optimal policies in the new MDP correspond exactly to optimal policies in the original MDP.

c¢. Now do the same to convert MDPs with R(s, a) into MDPs with R(s).

17.6 Consider the 4 x 3 world shown in Figure 17.1.

a. Implement an environment simulator for this environment, such that the specific geography of the environment is easily altered. Some code for doing this is already in the online code repository.

b. Create an agent that uses policy iteration, and measure its performance in the environment simulator from various starting states. Perform several experiments from each starting state, and compare the average total reward received per run with the utility of the state, as determined by your algorithm.

c. Experiment with increasing the size of the environment. How does the runtime for policy iteration vary with the size of the environment?

17.7 For the environment shown in Figure 17.1, find all the threshold values for R(s) such that the optimal policy changes when the threshold is crossed. You will need a way to calculate the optimal policy and its value for fixed R(s). [Hint: Prove that the value of any fixed policy varies linearly with R(s).]

17.8 In this exercise we will consider two-player MDPs that correspond to zero-sum, turntaking games like those in Chapter 6. Let the players be A and B, and let R(s) be the reward for player A in s. (The reward for B is always equal and opposite.)

a. Let U,y(s) be the utility of state s when it is A’s turn to move in s, and let Ug(s) be the utility of state s when itis B’s turn to move in s. All rewards and utilities are calculated from A’s point of view (just as in a minimax game tree). Write down Bellman equations defining U4(s) and Ug(s).

b. Explain how to do two-player value iteration with these equations, and define a suitable stopping criterion.

648

Chapter 17. Making Complex Decisions

¢. Consider the game described in Figure 6.14 on page 190. Draw the state space (rather than the game tree), showing the moves by A as solid lines and moves by B as dashed lines. Mark each state with R(s). You will find it helpful to arrange the states (s4, sp) on a two-dimensional grid, using s 4 and sp as “coordinates.”

d. Now apply two-player value iteration to solve this game, and derive the optimal policy.

17.9 Show that a dominant strategy equilibrium is a Nash equilibrium, but not vice versa.

17.10 In the children’s game of rock-paper—scissors each player reveals at the same time a choice of rock, paper, or scissors. Paper wraps rock, rock blunts scissors, and scissors cut paper. In the extended version rock—paper—scissors—fire—water, fire beats rock, paper and scissors; rock, paper and scissors beat water, and water beats fire. Write out the payoff matrix and find a mixed-strategy solution to this game.

17.11 Solve the game of three-finger Morra.

17.12 Prior to 1999, teams in the National Hockey League received 2 points for a win, 1 for a tie, and O for a loss. Is this a constant-sum game? In 1999, the rules were amended so that a team receives 1 point for a loss in overtime. The winning team still gets 2 points. How does this modification change the answers to the questions above? If it were legal to do so, when would it be rational for the two teams to secretly agree to end regulation play in a tie and then battle it out in overtime? Assume that the utility to each team is the number of points it receives, and that there is a mutually known prior probability p that the first team will win in overtime. For what values of p would both teams agree to this arrangement?

17.13 The following payoff matrix, from Blinder (1983) by way of Bernstein (1996), shows a game between politicians and the Federal Reserve.

[]

Politicians can expand or contract fiscal policy, while the Fed can expand or contract monetary policy. (And of course either side can choose to do nothing.) Each side also has preferences for who should do what—neither side wants to look like the bad guys. The payoffs shown are simply the rank orderings: 9 for first choice through 1 for last choice. Find the Nash equilibrium of the game in pure strategies. Is this a Pareto optimal solution? The reader might wish to analyze the policies of recent administrations in this light.

1 X LEARNING FROM OBSERVATIONS

In which we describe agents that can improve their behavior through diligent study of their own experiences.

The idea behind learning is that percepts should be used not only for acting, but also for improving the agent’s ability to act in the future. Learning takes place as the agent observes its interactions with the world and its own decision-making processes. Learning can range from trivial memorization of experience, as exhibited by the wumpus-world agent in Chapter 10, to the creation of entire scientific theories, as exhibited by Albert Einstein. This chapter describes inductive learning from observations. In particular, we describe how to learn simple theories in propositional logic. We also give a theoretical analysis that explains why inductive learning works.

18.1 FORMS OF LEARNING

In Chapter 2, we saw that a learning agent can be thought of as containing a performance element that decides what actions to take and a learning element that modifies the performance element so that it makes better decisions. (See Figure 2.15.) Machine learning researchers have come up with a large variety of learning elements. To understand them, it will help to see how their design is affected by the context in which they will operate. The design of a learning element is affected by three major issues:

e Which components of the performance element are to be learned.

e What feedback is available to learn these components.

e What representation is used for the components, We now analyze each of these issues in turn. We have seen that there are many ways to build the performance element of an agent. Chapter 2 described several agent designs (Figures 2.9, 2.11, 2.13, and 2.14). The components of these agents include the following:

1. A direct mapping from conditions on the current state to actions.

2. A means to infer relevant properties of the world from the percept sequence.

649

650

Chapter 18. Learning from Observations

SUPERVISED LEARNING

UNSUPERVISED LEARNING

REINFORCEMENT LEARNING

REINFORCEMENT

3. Information about the way the world evolves and about the results of possible actions the agent can take.

4. Utility information indicating the desirability of world states. 5. Action-value information indicating the desirability of actions. 6. Goals that describe classes of states whose achievement maximizes the agent's utility.

Each of these components can be learned from appropriate feedback. Consider, for example, an agent training to become a taxi driver. Every time the instructor shouts “Brake!” the agent can learn a condition—action rule for when to brake (component 1). By seeing many camera images that it is told contain buses, it can learn to recognize them (2). By trying actions and observing the results—for example, braking hard on a wet road—it can learn the effects of its actions (3). Then, when it receives no tip from passengers who have been thoroughly shaken up during the trip, it can learn a useful component of its overall utility function (4).

The type of feedback available for learning is usually the most important factor in determining the nature of the learning problem that the agent faces. The field of machine learning usually distinguishes three cases: supervised, unsupervised, and reinforcement learning.

The problem of supervised learning involves learning a function from examples of its inputs and outputs. Cases (1), (2), and (3) are all instances of supervised learning problems. In (1), the agent learns condition—action rule for braking—this is a function from states to a Boolean output (to brake or not to brake), In (2), the agent learns a function from images to a Boolean output (whether the image contains a bus). In (3), the theory of braking is a function from states and braking actions to, say, stopping distance in feet. Notice that in cases (1) and (2), a teacher provided the correct output value of the examples; in the third, the output value was available directly from the agent’s percepts. For fully observable environments, it will always be the case that an agent can observe the effects of its actions and hence can use supervised learning methods to learn to predict them. For partially observable environments, the problem is more difficult, because the immediate effects might be invisible.

The problem of unsupervised learning involves learning patterns in the input when no specific output values are supplied. For example, a taxi agent might gradually develop a concept of “good traffic days” and “bad traffic days” without ever being given labelled examples of each. A purely unsupervised learning agent cannot learn what to do, because it has no information as to what constitutes a correct action or a desirable state. We will study unsupervised learning primarily in the context of probabilistic reasoning systems (Chapter 20.

The problem of reinforcement learning, which we cover in Chapter 21, is the most general of the three categories. Rather than being told what to do by a teacher, a reinforcement learning agent must learn from reinforcement.! For example, the lack of a tip at the end of the journey (or a hefty bill for rear-ending the car in front) give the agent some indication that its behavior is undesirable. Reinforcement learning typically includes the subproblem of learning how the environment works.

The representation of the learned information also plays a very important role in determining how the learning algorithm must work. Any of the components of an agent can be represented using any of the representation schemes in this book. We have seen sev

1 The term reward as used in Chapter 17 is a synonym for reinforcement.

Section 18.2.

Inductive Learning 651

eral examples: linear weighted polynomials for utility functions in game-playing programs; propositional and first-order logical sentences for all of the components in a logical agent; and probabilistic descriptions such as Bayesian networks for the inferential components of a decision-theoretic agent. Effective learning algorithms have been devised for all of these. This chapter will cover methods for propositional logic, Chapter 19 describes methods for first-order logic, and Chapter 20 covers methods for Bayesian networks and for neural networks (which include linear polynomials as a special case).

The last major factor in the design of learning systems is the availability of prior knowledge. The majority of learning research in Al, computer science, and psychology has studied the case in which the agent begins with no knowledge at all about what it is trying to learn. It has access only to the examples presented by its experience. Although this is an important special case, it is by no means the general case. Most human learning takes place in the context of a good deal of background knowledge. Some psychologists and linguists claim that even newborn babies exhibit knowledge of the world. Whatever the truth of this claim, there is no doubt that prior knowledge can help enormously in learning. A physicist examining a stack of bubble-chamber photographs might be able to induce a theory positing the existence of a new particle of a certain mass and charge; but an art critic examining the same stack might learn nothing more than that the “artist” must be some sort of abstract expressionist. Chapter 19 shows several ways in which learning is helped by the use of existing knowledge; it also shows how knowledge can be compiled in order to speed up decision making. Chapter 20 shows how prior knowledge helps in the learning of probabilistic theories.

EXAMPLE

PURE INDUCTIVE INFERENCE

HYPOTHESIS

GENERALIZATION

PROBLEM OF INDUCTION

HYPOTHESIS SPACE

CONSISTENT

An algorithm for deterministic supervised learning is given as input the correct value of the unknown function for particular inputs and must try to recover the unknown function or something close to it. More formally, we say that an example is a pair (x, f(z)), where x is the input and f(x) is the output of the function applied to x. The task of pure inductive inference (or induction) is this:

Given a collection of examples of f, return a function h that approximates f.

The function h is called a hypothesis. The reason that learning is difficult, from a conceptual point of view, is that it is not easy to tell whether any particular 4 is a good approximation of f. A good hypothesis will generalize well—that is, will predict unseen examples correctly. This is the fundamental problem of induction. The problem has been studied for centuries; Section 18.5 provides a partial solution.

Figure 18.1 shows a familiar example: fitting a function of a single variable to some data points. The examples are (z, f(x)) pairs, where both z and f(z) are real numbers. We choose the hypothesis space H—the set of hypotheses we will consider—to be the set of polynomials of degree at most k, such as 32% + 2, 7 — 423, and so on. Figure 18.1(a) shows some data with an exact fit by a straight line (a polynomial of degree 1). The line is called a consistent hypothesis because it agrees with all the data. Figure 18.1(b) shows a

652

Chapter 18. Learning from Observations

OCKHAM'S RAZOR

REALIZABLE UNREALIZABLE

fo) fx fe) fx)

= = a =

(a) (® © (d)

Figure 18.1 (a) Example (x, f()) pairs and a consistent, linear hypothesis. (b) A consistent, degree-7 polynomial hypothesis for the same data set. (c) A different data set that admits an exact degree-6 polynomial fit or an approximate linear fit. (d) A simple, exact sinusoidal fit to the same data set.

high-degree polynomial that is also consistent with the same data. This illustrates the first issue in inductive learning: how do we choose from among multiple consistent hypotheses? One answer is Ockham’s? razor: prefer the simplest hypothesis consistent with the data. Intuitively, this makes sense, because hypotheses that are no simpler than the data themselves are failing to extract any pattern from the data. Defining simplicity is not easy, but it seems reasonable to say that a degree-1 polynomial is simpler than a degree-12 polynomial.

Figure 18.1(c) shows a second data set. There is no consistent straight line for this data set; in fact, it requires a degree-6 polynomial (with 7 parameters) for an exact fit. There are just 7 data points, so the polynomial has as many parameters as there are data points; thus, it does not seem to be finding any pattern in the data and we do not expect it to generalize well. It might be better to fit a simple straight line that is not exactly consistent but might make reasonable predictions. This amounts to accepting the possibility that the true function is not deterministic (or, roughly equivalently, that the true inputs are not fully observed). For nondeterministic functions, there is an inevitable tradeoff between the complexity of the hypothesis and the degree of fit to the data. Chapter 20 explains how to make this tradeoff using probability theory.

One should keep in mind that the possibility or impossibility of finding a simple, consistent hypothesis depends strongly on the hypothesis space chosen. Figure 18.1(d) shows that the data in (c) can be fit exactly by a simple function of the form ax + b + ¢sinz. This example shows the importance of the choice of hypothesis space. A hypothesis space consisting of polynomials of finite degree cannot represent sinusoidal functions accurately, so a learner using that hypothesis space will not be able to learn from sinusoidal data. We say that a learning problem is realizable if the hypothesis space contains the true function; otherwise, it is unrealizable. Unfortunately, we cannot always tell whether a given learning problem is realizable, because the true function is not known. One way to get around this barrier is to use prior knowledge to derive a hypothesis space in which we know the true function must lie. This topic is covered in Chapter 19.

2 Named after the 14th-century English philosopher, William of Ockham. The name is often misspelled as “Qccam,” perhaps from the French rendering, “Guillaume d’Occam.”

Section 18.3.

Learning Decision Trees 653

Another approach is to use the largest possible hypothesis space. For example, why not let H be the class of all Turing machines? After all, every computable function can be represented by some Turing machine, and that is the best we can do. The problem with this idea is that it does not take into account the computational complexity of learning. There is a tradeoff between the expressiveness of a hypothesis space and the complexity of finding simple, consistent hypotheses within that space. For example, fitting straight lines to data is very easy; fitting high-degree polynomials is harder; and fitting Turing machines is very hard indeed because determining whether a given Turing machine is consistent with the data is not even decidable in general. A second reason to prefer simple hypothesis spaces is that the resulting hypotheses may be simpler to use—that is, it is faster to compute h(z) when h is a linear function than when it is an arbitrary Turing machine program.

For these reasons, most work on learning has focused on relatively simple representations. In this chapter, we concentrate on propositional logic and related languages. Chapter 19 looks at learning theories in first-order logic. We will see that the expressiveness—complexity tradeoff is not as simple as it first seems: it is often the case, as we saw in Chapter 8, that an expressive language makes it possible for a simple theory to fit the data, whereas restricting the expressiveness of the language means that any consistent theory must be very complex. For example, the rules of chess can be written in a page or two of first-order logic, but require thousands of pages when written in propositional logic. In such cases, it should be possible to learn much faster by using the more expressive language.

DECISION TREE ATTRIBUTES

CLASSIFICATION REGRESSION POSITIVE

NEGATIVE

Decision tree induction is one of the simplest, and yet most successful forms of learning algorithm. It serves as a good introduction to the area of inductive learning, and is easy to implement. We first describe the performance element, and then show how to learn it. Along the way, we will introduce ideas that appear in all areas of inductive learning.

Decision trees as performance elements

A decision tree takes as input an object or situation described by a set of attributes and returns a “decision”—the predicted output value for the input. The input attributes can be discrete or continuous. For now, we assume discrete inputs. The output value can also be discrete or continuous; learning a discrete-valued function is called classification learning; learning a continuous function is called regression. We will concentrate on Boolean classification, wherein each example is classified as true (positive) or false (negative).

A decision tree reaches its decision by performing a sequence of tests. Each internal node in the tree corresponds to a test of the value of one of the properties, and the branches from the node are labeled with the possible values of the test. Each leaf node in the tree specifies the value to be returned if that leaf is reached. The decision tree representation seems to be very natural for humans; indeed, many “How To” manuals (e.g., for car repair) are written entirely as a single decision tree stretching over hundreds of pages.

654

Chapter 18. Learning from Observations

GOAL PREDICATE

A somewhat simpler example is provided by the problem of whether to wait for a table at a restaurant. The aim here is to learn a definition for the goal predicate WillWait. In setting this up as a learning problem, we first have to state what attributes are available to describe examples in the domain. In Chapter 19, we will see how to automate this task; for now, let’s suppose we decide on the following list of attributes:

. Alternate: whether there is a suitable alternative restaurant nearby.

. Bar: whether the restaurant has a comfortable bar area to wait in.

. Fri/Sat: true on Fridays and Saturdays.

Hungry: whether we are hungry.

. Patrons: how many people are in the restaurant (values are None, Some, and Full). Price: the restaurant’s price range ($, $$, $$$).

Raining: whether it is raining outside.

Reservation: whether we made a reservation.

© 0 NEw

Type: the kind of restaurant (French, Italian, Thai, or burger). WaitEstimate: the wait estimated by the host (0-10 minutes, 10-30, 30-60, >60).

@

The decision tree usually used by one of us (SR) for this domain is shown in Figure 18.2. Notice that the tree does not use the Price and Type attributes, in effect considering them to be irrelevant. Examples are processed by the tree starting at the root and following the appropriate branch until a leaf is reached. For instance, an example with Patrons = Full and WaitEstimate = 0-10 will be classified as positive (i.e., yes, we will wait for a table).

Patrons? |

None Some Full

Tl [Yes WailEstimate?

>60

Figure 18.2 A decision tree for deciding whether to wait for a table.

Section 18.3.

Learning Decision Trees 655

PARITY FUNCTION

MAJORITY FUNCTION

Expressiveness of decision trees

Logically speaking, any particular decision tree hypothesis for the WillWait goal predicate can be seen as an assertion of the form

Vs WillWait(s) < (Pi(s)V Pa(s) V---V Py(s)),

where each condition P;(s) is a conjunction of tests corresponding to a path from the root of the tree to a leaf with a positive outcome. Although this looks like a first-order sentence, it is, in a sense, propositional, because it contains just one variable and all the predicates are unary. The decision tree is really describing a relationship between WillWait and some logical combination of attribute values. We cannot use decision trees to represent tests that refer to two or more different objects—for example,

dry Nearby(ra,) A Price(r,p) A Price(ra, p2) A Cheaper(pa, p)

(is there a cheaper restaurant nearby?). Obviously, we could add another Boolean attribute with the name CheaperRestaurant Nearby, but it is intractable to add all such attributes. Chapter 19 will delve further into the problem of learning in first-order logic proper.

Decision trees are fully expressive within the class of propositional languages; that is, any Boolean function can be written as a decision tree. This can be done trivially by having each row in the truth table for the function correspond to a path in the tree. This would yield an exponentially large decision tree representation because the truth table has exponentially many rows. Clearly, decision trees can represent many functions with much smaller trees.

For some kinds of functions, however, this is a real problem. For example, if the function is the parity function, which returns 1 if and only if an even number of inputs are 1, then an exponentially large decision tree will be needed. It is also difficult to use a decision tree to represent a majority function, which returns 1 if more than half of its inputs are 1.

In other words, decision trees are good for some kinds of functions and bad for others. Is there any kind of representation that is efficient for all kinds of functions? Unfortunately, the answer is no. We can show this in a very general way. Consider the set of all Boolean functions on n attributes. How many different functions are in this set? This is just the number of different truth tables that we can write down, because the function is defined by its truth table. The truth table has 2" rows, because each input case is described by n attributes. We can consider the “answer” column of the table as a 2™-bit number that defines the function. No matter what representation we use for functions, some of the functions (almost all of them, in fact) are going to require at least that many bits to represent.

If it takes 2" bits to define the function, then there are 22" different functions on n attributes. This is a scary number. For example, with just six Boolean attributes, there are 22° = 18, 446, 744,073,709, 551, 616 different functions to choose from. We will need some ingenious algorithms to find consistent hypotheses in such a large space.

Inducing decision trees from examples

An example for a Boolean decision tree consists of a vector of input attributes, X, and a single Boolean output value y. A set of examples (X1,y1)..., (X12, y12) is shown in Figure 18.3.

[]

656 Chapter 18. Learning from Observations Attributes Goal Example Alt | Bar| Fri | Hun| Pat | Price| Rain| Res | Type Est WillWait Xi Yes | No | No | Yes | Some | $$$ | No | Yes | French | 0-10 Yes Xs Yes| No | No | Yes | Full $ No | No Thai | 30-60 No Xs No | Yes| No | No | Some $ No | No | Burger | 0-10 Yes Xa Yes | No | Yes | Yes | Full $ Yes | No Thai 10-30 Yes Xs Yes | No| Yes| No | Full | $$$ | No | Yes | French| >60 No Xe No | Yes | No | Yes | Some | $$ | Yes | Yes | Italian | 0-10 Yes Xy No | Yes | No | No | None $ Yes | No | Burger | 0-10 No Xs No | No | No | Yes| Some | $$ | Yes | Yes | Thai 0-10 Yes Xq No | Yes| Yes | No | Full $ Yes | No | Burger | >60 No Xi Yes | Yes | Yes | Yes | Full | $$$ | No | Yes | Italian | 10-30 No Xi No | No| No | No | None $ No | No Thai 0-10 No X12 Yes | Yes | Yes | Yes | Full $ No | No | Burger | 30-60 Yes Figure 18.3 Examples for the restaurant domain. The positive examples are the ones in which the goal WillWait is true (X71, X3, .. .); the negative examples are the ones in which it is false (X32, X5,...). The complete set of examples TRAINING SET is called the training set.

The problem of finding a decision tree that agrees with the training set might seem difficult, but in fact there is a trivial solution. We could simply construct a decision tree that has one path to a leaf for each example, where the path tests each attribute in turn and follows the value for the example and the leaf has the classification of the example. When given the same example again,® the decision tree will come up with the right classification. Unfortunately, it will not have much to say about any other cases!

The problem with this trivial tree is that it just memorizes the observations. It does not extract any pattern from the examples, so we cannot expect it to be able to extrapolate to examples it has not seen. Applying Ockham’s razor, we should find instead the smallest decision tree that is consistent with the examples. Unfortunately, for any reasonable definition of “smallest,” finding the smallest tree is an intractable problem. With some simple heuristics, however, we can do a good job of finding a “smallish” one. The basic idea behind the DECISION-TREE-LEARNING algorithm is to test the most important attribute first. By “most important,” we mean the one that makes the most difference to the classification of an example. That way, we hope to get to the correct classification with a small number of tests, meaning that all paths in the tree will be short and the tree as a whole will be small.

Figure 18.4 shows how the algorithm gets started. We are given 12 training examples, which we classify into positive and negative sets. We then decide which attribute to use as the first test in the tree. Figure 18.4(a) shows that Type is a poor attribute, because it leaves us with four possible outcomes, each of which has the same number of positive and negative examples. On the other hand, in Figure 18.4(b) we see that Patrons is a fairly important

3 The same example or an example with the same description—this distinction is very important, and we will return to it in Chapter 19.

Section 18.3. Learning Decision Trees 657

[WE ® EE az 13 4 86 8 12 BE HAE DR BHHEBERE [ Type? [patrons? | French — Traian % \ Thai ~\_Burger None \_— en k Full | [m W WE BE | EEE NE B iE Bm a im BEBE 2 | Yes ] Hungry? No/ \Yes [ TY BO BE (@) (b)

Figure 18.4 Splitting the examples by testing on attributes. (a) Splitting on Type brings us no nearer to distinguishing between positive and negative examples. (b) Splitting on Patrons does a good job of separating positive and negative examples. After splitting on Patrons, Hungry is a fairly good second test.

attribute, because if the value is None or Some, then we are left with example sets for which we can answer definitively (No and Yes, respectively). If the value is Full, we are left with a mixed set of examples. In general, after the first attribute test splits up the examples, each outcome is a new decision tree learning problem in itself, with fewer examples and one fewer attribute. There are four cases to consider for these recursive problems:

1. If there are some positive and some negative examples, then choose the best attribute to split them. Figure 18.4(b) shows Hungry being used to split the remaining examples.

2. If all the remaining examples are positive (or all negative), then we are done: we can answer Yes or No. Figure 18.4(b) shows examples of this in the None and Some cases.

3. If there are no examples left, it means that no such example has been observed, and we return a default value calculated from the majority classification at the node’s parent.

4. If there are no attributes left, but both positive and negative examples, we have a problem. It means that these examples have exactly the same description, but different NOISE classifications. This happens when some of the data are incorrect; we say there is noise in the data. It also happens either when the attributes do not give enough information to describe the situation fully, or when the domain is truly nondeterministic. One simple way out of the problem is to use a majority vote.

The DECISION-TREE-LEARNING algorithm is shown in Figure 18.5. The details of the method for CHOOSE-ATTRIBUTE are given in the next subsection.

The final tree produced by the algorithm applied to the 12-example data set is shown in Figure 18.6. The tree is clearly different from the original tree shown in Figure 18.2, despite the fact that the data were actually generated from an agent using the original tree. One might conclude that the learning algorithm is not doing a very good job of learning the correct

658 Chapter 18. Learning from Observations

function DECISION-TREE-LEARNING (examples, attribs, default) returns a decision tree inputs: examples, set of examples attribs, set of attributes default, default value for the goal predicate

if examples is empty then return defauli else if all examples have the same classification then return the classification else if atiribs is empty then return MAJORITY-VALUE( examples) else best — CHOOSE-ATTRIBUTE(attribs, examples) tree «— a new decision tree with root test best m «— MAJORITY-VALUE(examples) for each value v; of best do examples; <— {elements of examples with best = v;} subtree «+ DECISION-TREE-LEARNING(examples;, attribs — best, m) add a branch to {ree with label v; and subtree subtree return tree

[]

Figure 18.6 The decision tree induced from the 12-example training set.

function. This would be the wrong conclusion to draw, however. The learning algorithm looks at the examples, not at the correct function, and in fact, its hypothesis (see Figure 18.6) not only agrees with all the examples, but is considerably simpler than the original tree. The learning algorithm has no reason to include tests for Raining and Reservation, because it can classify all the examples without them. It has also detected an interesting and previously unsuspected pattern: the first author will wait for Thai food on weekends.

Of course, if we were to gather more examples, we might induce a tree more similar to the original. The tree in Figure 18.6 is bound to make a mistake; for example, it has never seen a case where the wait is 0—10 minutes but the restaurant is full. For a case where

Section 18.3.

Learning Decision Trees 659

INFORMATION

Hungry is false, the tree says not to wait, but I (SR) would certainly wait. This raises an obvious question: if the algorithm induces a consistent, but incorrect, tree from the examples, how incorrect will the tree be? We will show how to analyze this question experimentally, after we explain the details of the attribute selection step.

Choosing attribute tests

The scheme used in decision tree learning for selecting attributes is designed to minimize the depth of the final tree. The idea is to pick the attribute that goes as far as possible toward providing an exact classification of the examples. A perfect attribute divides the examples into sets that are all positive or all negative. The Patrons attribute is not perfect, but it is fairly good. A really useless attribute, such as Tipe, leaves the example sets with roughly the same proportion of positive and negative examples as the original set.

All we need, then, is a formal measure of “fairly good” and “really useless” and we can implement the CHOOSE-ATTRIBUTE function of Figure 18.5. The measure should have its maximum value when the attribute is perfect and its minimum value when the attribute is of no use at all. One suitable measure is the expected amount of information provided by the attribute, where we use the term in the mathematical sense first defined in Shannon and Weaver (1949). To understand the notion of information, think about it as providing the answer to a question—for example, whether a coin will come up heads. The amount of information contained in the answer depends on one’s prior knowledge. The less you know, the more information is provided. Information theory measures information content in bits. One bit of information is enough to answer a yes/no question about which one has no idea, such as the flip of a fair coin. In general, if the possible answers v; have probabilities P(v;), then the information content I of the actual answer is given by

[]

i=1 To check this equation, for the tossing of a fair coin, we get (4,4) =—1logs} — Llogy 4 = 1bit If the coin is loaded to give 99% heads, we get I (1/100,99/100) = 0.08 bits, and as the probability of heads goes to 1, the information of the actual answer goes to 0.

For decision tree learning, the question that needs answering is; for a given example, what is the correct classification? A correct decision tree will answer this question. An estimate of the probabilities of the possible answers before any of the attributes have been tested is given by the proportions of positive and negative examples in the training set. Suppose the

training set contains p positive examples and n negative examples. Then an estimate of the information contained in a correct answer is

Pn y\_\_\_P pn \_n\_ (35 2) pin log, p+n pn log, pn’ The restaurant training set in Figure 18.3 has p = n = 6, so we need 1 bit of information.

Now a test on a single attribute A will not usually tell us this much information, but it will give us some of it. We can measure exactly how much by looking at how much

660

Chapter 18. Learning from Observations

INFORMATION GAIN

TEST SET

LEARNING CURVE

information we still need after the attribute test. Any attribute A divides the training set F into subsets E1,. .., FE, according to their values for A, where A can have v distinct values. Each subset F; has p; positive examples and 7; negative examples, so if we go along that branch, we will need an additional I (p;/(p; + ni), ni/(ps + n4)) bits of information to answer the question. A randomly chosen example from the training set has the ith value for the attribute with probability (p; + n;)/(p + n), so on average, after testing attribute A, we will need

v ; — pitns Di ng Remainder(A) = nd (525 Sai) i=1 bits of information to classify the example. The information gain from the attribute test is the difference between the original information requirement and the new requirement: . \_ p n \_ .

Gain(A) =1 (725 2) Remainder (A) . The heuristic used in the CHOOSE-ATTRIBUTE function is just to choose the attribute with the largest gain. Returning to the attributes considered in Figure 18.4, we have

Gain( Patrons) = 1 — [F1(0,1) + 51(1,0) + ST (3, 4)] ~ 0.541 bits.

. — 2 7(1 1 27(11 47(2 2 47(2 2\] \_

Gain( Type) =1EX (3. 5) + S51 (3. 2) + ol (3 9) + ol (3 3] =0. confirming our intuition that Patrons is a better attribute to split on. In fact, Patrons has the highest gain of any of the attributes and would be chosen by the decision-tree learning algorithm as the root.

Assessing the performance of the learning algorithm

A learning algorithm is good if it produces hypotheses that do a good job of predicting the classifications of unseen examples. In Section 18.5, we will see how prediction quality can be estimated in advance. For now, we will look at a methodology for assessing prediction quality after the fact.

Obviously, a prediction is good if it turns out to be true, so we can assess the quality of a hypothesis by checking its predictions against the correct classification once we know it. We do this on a set of examples known as the test set. If we train on all our available examples, then we will have to go out and get some more to test on, so often it is more convenient to adopt the following methodology:

1. Collect a large set of examples.

2. Divide it into two disjoint sets; the training set and the test set.

3. Apply the learning algorithm to the training set, generating a hypothesis A.

4. Measure the percentage of examples in the test set that are correctly classified by A.

5. Repeat steps 2 to 4 for different sizes of training sets and different randomly selected

training sets of each size.

The result of this procedure is a set of data that can be processed to give the average prediction quality as a function of the size of the training set. This function can be plotted on a graph, giving what is called the learning curve for the algorithm on the particular domain. The

Section 18.3.

Learning Decision Trees 661

PEEKING

0.9

0.8

0.7

0.5

Proportion correct on test set

04 + r v —\_ 0 20 40 60 80 100

Training set size

Figure 18.7 A learning curve for the decision tree algorithm on 100 randomly generated examples in the restaurant domain. The graph summarizes 20 trials.

learning curve for DECISION-TREE-LEARNING with the restaurant examples is shown in Figure 18.7. Notice that, as the training set grows, the prediction quality increases. (For this reason, such curves are also called happy graphs.) This is a good sign that there is indeed some pattern in the data and the learning algorithm is picking it up.

Obviously, the learning algorithm must not be allowed to “see” the test data before the learned hypothesis is tested on them. Unfortunately, it is all too easy to fall into the trap of peeking at the test data. Peeking typically happens as follows: A learning algorithm can have various “knobs” that can be twiddled to tune its behavior—for example, various different criteria for choosing the next attribute in decision tree learning. We generate hypotheses for various different settings of the knobs, measure their performance on the test set, and report the prediction performance of the best hypothesis. Alas, peeking has occurred! The reason is that the hypothesis was selected on the basis of its test set performance, so information about the test set has leaked into the learning algorithm. The moral of this tale is that any process that involves comparing the performance of hypotheses on a test set must use a new test set to measure the performance of the hypothesis that is finally selected. In practice, this is too difficult, so people continue to run experiments on tainted sets of examples.

Noise and overfitting

We saw earlier that if there are two or more examples with the same description (in terms of the attributes) but different classifications, then the DECISION-TREE-LEARNING algorithm must fail to find a decision tree consistent with all the examples. The solution we mentioned before is to have each leaf node report either the majority classification for its set of examples, if a deterministic hypothesis is required, or report the estimated probabilities of each classification using the relative frequencies. Unfortunately, this is far from the whole story. It is quite possible, and in fact likely, that even when vital information is missing, the decision tree learning algorithm will find a decision tree that is consistent with all the examples. This is because the algorithm can use the irrelevant attributes, if any, to make spurious distinctions among the examples.

662

Chapter 18. Learning from Observations

OVERFITTING

DECISION TREE PRUNING

SIGNIFICANCE TEST NULL HYPOTHESIS

Consider the problem of trying to predict the roll of a die. Suppose that experiments are carried out during an extended period of time with various dice and that the attributes describing each training example are as follows:

1. Day: the day on which the die was rolled (Mon, Tue, Wed, Thu). 2. Month: the month in which the die was rolled (Jan or Feb). 3. Color: the color of the die (Red or Blue).

As long as no two examples have identical descriptions, DECISION-TREE-LEARNING will find an exact hypothesis. The more attributes there are, the more likely it is that an exact hypothesis will be found. Any such hypothesis will be totally spurious. What we would like is that DECISION-TREE-LEARNING return a single leaf node with probabilities close to 1/6 for each roll, once it has seen enough examples.

Whenever there is a large set of possible hypotheses, one has to be careful not to use the resulting freedom to find meaningless “regularity” in the data. This problem is called overfitting. A very general phenomenon, overfitting occurs even when the target function is not at all random. It afflicts every kind of learning algorithm, not just decision trees.

A complete mathematical treatment of overfitting is beyond the scope of this book. Here we present a simple technique called decision tree pruning to deal with the problem. Pruning works by preventing recursive splitting on attributes that are not clearly relevant, even when the data at that node in the tree are not uniformly classified. The question is, how do we detect an irrelevant attribute?

Suppose we split a set of examples using an irrelevant attribute. Generally speaking, we would expect the resulting subsets to have roughly the same proportions of each class as the original set. In this case, the information gain will be close to zero.\* Thus, the information gain is a good clue to irrelevance. Now the question is, how large a gain should we require in order to split on a particular attribute?

We can answer this question by using a statistical significance test. Such a test begins by assuming that there is no underlying pattern (the so-called null hypothesis). Then the actual data are analyzed to calculate the extent to which they deviate from a perfect absence of pattern. If the degree of deviation is statistically unlikely (usually taken to mean a 5% probability or less), then that is considered to be good evidence for the presence of a significant pattern in the data. The probabilities are calculated from standard distributions of the amount of deviation one would expect to see in random sampling.

In this case, the null hypothesis is that the attribute is irrelevant and, hence, that the information gain for an infinitely large sample would be zero. We need to calculate the probability that, under the null hypothesis, a sample of size v would exhibit the observed deviation from the expected distribution of positive and negative examples. We can measure the deviation by comparing the actual numbers of positive and negative examples in each subset, p; and n;, with the expected numbers, p; and n;, assuming true irrelevance:

Pi + 1 . Di +1

nN; =n XxX p+n p+n

pi=p

4 In fact, the gain will be positive unless the proportions are all exactly the same. (See Exercise 18.10.)

Section 18.3.

Learning Decision Trees 663

[]

x2 PRUNING

CROSSVALIDATION

GAIN RATIO

A convenient measure of the total deviation is given by

i=1 Under the null hypothesis, the value of D is distributed according to the x? (chi-squared) distribution with v — 1 degrees of freedom. The probability that the attribute is really irrelevant can be calculated with the help of standard x? tables or with statistical software. Exercise 18.11 asks you to make the appropriate changes to DECISION-TREE-LEARNING to implement this form of pruning, which is known as x? pruning.

With pruning, noise can be tolerated: classification errors give a linear increase in prediction error, whereas errors in the descriptions of examples have an asymptotic effect that gets worse as the tree shrinks down to smaller sets. Trees constructed with pruning perform significantly better than trees constructed without pruning when the data contain a large amount of noise. The pruned trees are often much smaller and hence easier to understand.

Cross-validation is another technique that reduces overfitting. It can be applied to any learning algorithm, not just decision tree learning. The basic idea is to estimate how well each hypothesis will predict unseen data. This is done by setting aside some fraction of the known data and using it to test the prediction performance of a hypothesis induced from the remaining data. K-fold cross-validation means that you run k experiments, each time setting aside a different 1/k of the data to test on, and average the results. Popular values for k are 5 and 10. The extreme is k = n, also known as leave-one-out cross-validation. Crossvalidation can be used in conjunction with any tree-construction method (including pruning) in order to select a tree with good prediction performance. To avoid peeking, we must then measure this performance with a new test set.

Broadening the applicability of decision trees

In order to extend decision tree induction to a wider variety of problems, a number of issues must be addressed. We will briefly mention each, suggesting that a full understanding is best obtained by doing the associated exercises:

{> Missing data: In many domains, not all the attribute values will be known for every example. The values might have gone unrecorded, or they might be too expensive to obtain. This gives rise to two problems: First, given a complete decision tree, how should one classify an object that is missing one of the test attributes? Second, how should one modify the information gain formula when some examples have unknown values for the attribute? These questions are addressed in Exercise 18.12.

> Multivalued attributes:When an attribute has many possible values, the information gain measure gives an inappropriate indication of the attribute’s usefulness. In the extreme case, we could use an attribute, such as RestaurantName, that has a different value for every example. Then each subset of examples would be a singleton with a unique classification, so the information gain measure would have its highest value for this attribute. Nonetheless, the attribute could be irrelevant or useless. One solution is to use the gain ratio (Exercise 18.13).

664

Chapter 18. Learning from Observations

SPLIT POINT

REGRESSION TREE

¢$ Continuous and integer-valued input attributes: Continuous or integer-valued attributes such as Height and Weight, have an infinite set of possible values. Rather than generate infinitely many branches, decision-tree learning algorithms typically find the split point that gives the highest information gain. For example, at a given node in the tree, it might be the case that testing on Weight > 160 gives the most information. Efficient dynamic programming methods exist for finding good split points, but it is still by far the most expensive part of real-world decision tree learning applications.

¢& Continuous-valued output attributes: If we are trying to predict a numerical value, such as the price of a work of art, rather than a discrete classification, then we need a regression tree. Such a tree has at each leaf a linear function of some subset of numerical attributes, rather than a single value. For example, the branch for handcolored engravings might end with a linear function of area, age, and number of colors. The learning algorithm must decide when to stop splitting and begin applying linear regression using the remaining attributes (or some subset thereof).

A decision-tree learning system for real-world applications must be able to handle all of these problems. Handling continuous-valued variables is especially important, because both physical and financial processes provide numerical data. Several commercial packages have been built that meet these criteria, and they have been used to develop several hundred fielded systems. In many areas of industry and commerce, decision trees are usually the first method tried when a classification method is to be extracted from a data set. One important property of decision trees is that it is possible for a human to understand the output of the learning algorithm. (Indeed, this is a legal requirement for financial decisions that are subject to antidiscrimination laws.) This is a property not shared by neural networks (see Chapter 20).

ENSEMBLE LEARNING

So far we have looked at learning methods in which a single hypothesis, chosen from a hypothesis space, is used to make predictions. The idea of ensemble learning methods is to select a whole collection, or ensemble, of hypotheses from the hypothesis space and combine their predictions. For example, we might generate a hundred different decision trees from the same training set and have them vote on the best classification for a new example.

The motivation for ensemble learning is simple. Consider an ensemble of M =5 hypotheses and suppose that we combine their predictions using simple majority voting. For the ensemble to misclassify a new example, at least three of the five hypotheses have to misclassify it. The hope is that this is much less likely than a misclassification by a single hypothesis. Suppose we assume that each hypothesis bh; in the ensemble has an error of p—that is, the probability that a randomly chosen example is misclassified by A; is p. Furthermore, suppose we assume that the errors made by each hypothesis are independent. In that case, if p is small, then the probability of a large number of misclassifications occurring is minuscule. For example, a simple calculation (Exercise 18.14) shows that using an ensemble of five hypotheses reduces an error rate of 1 in 10 down to an error rate of less than 1 in 100. Now, obviously

Section 18.4.

Ensemble Learning 665

[]

BOOSTING WEIGHTED TRAINING SET

Figure 18.8 Illustration of the increased expressive power obtained by ensemble learning. We take three linear threshold hypotheses, each of which classifies positively on the nonshaded side, and classify as positive any example classified positively by all three. The resulting triangular region is a hypothesis not expressible in the original hypothesis space.

the assumption of independence is unreasonable, because hypotheses are likely to be misled in the same way by any misleading aspects of the training data. But if the hypotheses are at least a little bit different, thereby reducing the correlation between their errors, then ensemble learning can be very useful.

Another way to think about the ensemble idea is as a generic way of enlarging the hypothesis space. That is, think of the ensemble itself as a hypothesis and the new hypothesis space as the set of all possible ensembles constructible from hypotheses in the original space. Figure 18.8 shows how this can result in a more expressive hypothesis space. If the original hypothesis space allows for a simple and efficient learning algorithm, then the ensemble method provides a way to learn a much more expressive class of hypotheses without incurring much additional computational or algorithmic complexity.

The most widely used ensemble method is called boosting. To understand how it works, we need first to explain the idea of a weighted training set. In such a training set, each example has an associated weight w; > 0. The higher the weight of an example, the higher is the importance attached to it during the learning of a hypothesis. It is straightforward to modify the learning algorithms we have seen so far to operate with weighted training sets.

Boosting starts with w; =1 for all the examples (i.e., a normal training set). From this set, it generates the first hypothesis, h;. This hypothesis will classify some of the training examples correctly and some incorrectly. We would like the next hypothesis to do better on the misclassified examples, so we increase their weights while decreasing the weights of the correctly classified examples. From this new weighted training set, we generate hypothesis ho. The process continues in this way until we have generated M hypotheses, where M is

5 For learning algorithms in which this is not possible, one can instead create a replicated training set where the ith example appears w; times, using randomization to handle fractional weights.

666

Chapter 18. Learning from Observations

WEAK LEARNING

DECISION STUMP

Ly 4 £4

Yigg,

wd

Figure 18.9 How the boosting algorithm works. Each shaded rectangle corresponds to an example; the height of the rectangle corresponds to the weight. The ticks and crosses indicate whether the example was classified correctly by the current hypothesis. The size of the decision tree indicates the weight of that hypothesis in the final ensemble.

an input to the boosting algorithm. The final ensemble hypothesis is a weighted-majority combination of all the M hypotheses, each weighted according to how well it performed on the training set. Figure 18.9 shows how the algorithm works conceptually. There are many variants of the basic boosting idea with different ways of adjusting the weights and combining the hypotheses. One specific algorithm, called ADABOOST, is shown in Figure 18.10. While the details of the weight adjustments are not so important, ADABOOST does have a very important property: if the input learning algorithm L is a weak learning algorithm—which means that I always returns a hypothesis with weighted error on the training set that is slightly better than random guessing (i.e., 50% for Boolean classification)—then ADABOOST will return a hypothesis that classifies the training data perfectly for large enough M. Thus, the algorithm boosts the accuracy of the original learning algorithm on the training data. This result holds no matter how inexpressive the original hypothesis space and no matter how complex the function being learned.

Let us see how well boosting does on the restaurant data. We will choose as our original hypothesis space the class of decision stumps, which are decision trees with just one test at the root. The lower curve in Figure 18.11(a) shows that unboosted decision stumps are not very effective for this data set, reaching a prediction performance of only 81% on 100 training examples. When boosting is applied (with M = 5), the performance is better, reaching 93% after 100 examples.

An interesting thing happens as the ensemble size M increases. Figure 18.11(b) shows the training set performance (on 100 examples) as a function of M. Notice that the error reaches zero (as the boosting theorem tells us) when M is 20; that is, a weighted-majority

Section 18.4. Ensemble Learning 667

function ADABOOST(examples, L, M) returns a weighted-majority hypothesis inputs: examples, set of N labelled examples (z1,v1),..., (ZN, yn) L, alearning algorithm M , the number of hypotheses in the ensemble local variables: w, a vector of N example weights, initially 1/N h, a vector of M hypotheses z, a vector of M hypothesis weights

for m =1to M do h[m] «— L(examples, w) error «0 for j=1to N do ifh{m](z;) # y; then error — error + w[j] forj=1to N do if h{m](z;) = y; then w[j] — w[j] error/(1 — error) w «— NORMALIZE(w) z[m] «log (1 — error) /error return WEIGHTED-MAJORITY (h, z)

Figure 18.10 The ADABOOST variant of the boosting method for ensemble learning. The algorithm generates hypotheses by successively reweighting the trainng examples. The function WEIGHTED-MAJORITY generates a hypothesis that returns the output value with the highest vote from the hypotheses in h, with votes weighted by z.

[]

Figure 18.11 (a) Graph showing the performance of boosted decision stumps with M =5 versus decision stumps on the restaurant data. (b) The proportion correct on the training set and the test set as a function of M, the number of hypotheses in the ensemble. Notice that the test set accuracy improves slightly even after the training accuracy reaches 1, i.e., after the ensemble fits the data exactly.

668

Chapter 18. Learning from Observations

combination of 20 decision stumps suffices to fit the 100 examples exactly. As more stumps are added to the ensemble, the error remains at zero. The graph also shows that the fest set performance continues to increase long after the training set error has reached zero. At M = 20, the test performance is 0.95 (or 0.05 error), and the performance increases to 0.98 as late as M = 137, before gradually dropping to 0.95.

This finding, which is quite robust across data sets and hypothesis spaces, came as quite a surprise when it was first noticed. Ockham’s razor tells us not to make hypotheses more complex than necessary, but the graph tells us that the predictions improve as the ensemble hypothesis gets more complex! Various explanations have been proposed for this. One view is that boosting approximates Bayesian learning (see Chapter 20), which can be shown to be an optimal learning algorithm, and the approximation improves as more hypotheses are added. Another possible explanation is that the addition of further hypotheses enables the ensemble to be more definite in its distinction between positive and negative examples, which helps it when it comes to classifying new examples.

COMPUTATIONAL LEARNING THEORY

[5 Ly

PROBABLY APPROXIMATELY CORRECT

PAC-LEARNING

STATIONARITY

The main unanswered question posed in Section 18.2 was this: how can one be sure that one’s learning algorithm has produced a theory that will correctly predict the future? In formal terms, how do we know that the hypothesis A is close to the target function f if we don’t know what f is? These questions have been pondered for several centuries. Until we find answers, machine learning will, at best, be puzzled by its own success.

The approach taken in this section is based on computational learning theory, a field at the intersection of Al, statistics, and theoretical computer science. The underlying principle is the following: any hypothesis that is seriously wrong will almost certainly be “found out” with high probability after a small number of examples, because it will make an incorrect prediction. Thus, any hypothesis that is consistent with a sufficiently large set of training examples is unlikely to be seriously wrong: that is, it must be probably approximately correct. Any learning algorithm that returns hypotheses that are probably approximately correct is called a PAC-learning algorithm.

There are some subtleties in the preceding argument. The main question is the connection between the training and the test examples; after all, we want the hypothesis to be approximately correct on the test set, not just on the training set. The key assumption is that the training and test sets are drawn randomly and independently from the same population of examples with the same probability distribution. This is called the stationarity assumption. Without the stationarity assumption, the theory can make no claims at all about the future, because there would be no necessary connection between future and past. The stationarity assumption amounts to supposing that the process that selects examples is not malevolent. Obviously, if the training set consists only of weird examples—two-headed dogs, for instance—then the learning algorithm cannot help but make unsuccessful generalizations about how to recognize dogs.

Section 18.5.

Why Learning Works: Computational Learning Theory 669

ERROR

€BALL

How many examples are needed?

In order to put these insights into practice, we will need some notation:

o Let X be the set of all possible examples.

o Let D be the distribution from which examples are drawn. o Let H be the set of possible hypotheses.

e Let N be the number of examples in the training set.

Initially, we will assume that the true function f is a member of H. Now we can define the error of a hypothesis h with respect to the true function f given a distribution DD over the examples as the probability that A is different from f on an example:

error(h) = P(h(z) # f(z)|x drawn from D) .

This is the same quantity being measured experimentally by the learning curves shown earlier.

A hypothesis h is called approximately correct if error(h) < e€, where € is a small constant. The plan of attack is to show that after seeing /V examples, with high probability, all consistent hypotheses will be approximately correct. One can think of an approximately correct hypothesis as being “close” to the true function in hypothesis space: it lies inside what is called the e-ball around the true function f. Figure 18.12 shows the set of all hypotheses H, divided into the e-ball around f and the remainder, which we call Hyg.

H

Hpaq

Figure 18.12 Schematic diagram of hypothesis space, showing the “e-ball” around the true function f.

We can calculate the probability that a “seriously wrong” hypothesis hy, € Hpag is consistent with the first N examples as follows. We know that error(h;) > e. Thus, the probability that it agrees with a given example is at least 1 — e. The bound for N examples is

P(hy agrees with N examples) < (1 — ¢)"

The probability that Hy,q contains at least one consistent hypothesis is bounded by the sum of the individual probabilities:

P(Hyaq contains a consistent hypothesis) < [Hyaq|(1 — €)™ < |H|(1 — €)V,

670

Chapter 18. Learning from Observations

SAMPLE COMPLEXITY

DECISION LIST

where we have used the fact that [Hp,g| < |H|. We would like to reduce the probability of this event below some small number J:

H|(1-e)V <6. Given that 1 — € < ¢™¢, we can achieve this if we allow the algorithm to see 1 1 N> = (123 + nH) (18.1) €

examples. Thus, if a learning algorithm returns a hypothesis that is consistent with this many examples, then with probability at least 1 — §, it has error at most ¢. In other words, it is probably approximately correct. The number of required examples, as a function of € and 4, is called the sample complexity of the hypothesis space.

It appears, then, that the key question is the size of the hypothesis space. As we saw earlier, if H is the set of all Boolean functions on n attributes, then |[H| = 22". Thus, the sample complexity of the space grows as 2". Because the number of possible examples is also 2", this says that any learning algorithm for the space of all Boolean functions will do no better than a lookup table if it merely returns a hypothesis that is consistent with all known examples. Another way to see this is to observe that for any unseen example, the hypothesis space will contain as many consistent hypotheses that predict a positive outcome as it does hypotheses that predict a negative outcome.

The dilemma we face, then, is that unless we restrict the space of functions the algorithm can consider, it will not be able to learn; but if we do restrict the space, we might eliminate the true function altogether. There are two ways to “escape” this dilemma. The first way is to insist that the algorithm return not just any consistent hypothesis, but preferably a simple one (as is done in decision tree learning). The theoretical analysis of such algorithms is beyond the scope of this book, but in cases where finding simple consistent hypotheses is tractable, the sample complexity results are generally better than for analyses based only on consistency. The second escape, which we pursue here, is to focus on learnable subsets of the entire set of Boolean functions. The idea is that in most cases we do not need the full expressive power of Boolean functions, and can get by with more restricted languages. We now examine one such restricted language in more detail.

Learning decision lists

A decision list is a logical expression of a restricted form. It consists of a series of tests, each of which is a conjunction of literals. If a test succeeds when applied to an example description, the decision list specifies the value to be returned. If the test fails, processing continues with the next test in the list.® Decision lists resemble decision trees, but their overall structure is simpler. In contrast, the individual tests are more complex. Figure 18.13 shows a decision list that represents the following hypothesis:

Va WillWait(xz) < Patrons(xz, Some) V (Patrons(z, Full) A Fri/Sat(x)) .

If we allow tests of arbitrary size, then decision lists can represent any Boolean function (Exercise 18.15). On the other hand, if we restrict the size of each test to at most k literals,

6 A decision list is therefore identical in structure to a COND statement in Lisp.

Section 18.5.

Why Learning Works: Computational Learning Theory 671

k-DL k-DT

No

Patrons(x, Some Patrons(x, Full) A Fri/Sat(x)

Figure 18.13 A decision list for the restaurant problem.

then it is possible for the learning algorithm to generalize successfully from a small number of examples. We call this language k-pDL. The example in Figure 18.13 is in 2-DL. It is easy to show (Exercise 18.15) that k-DL includes as a subset the language &-pT, the set of all decision trees of depth at most k. It is important to remember that the particular language referred to by k-pL depends on the attributes used to describe the examples. We will use the notation k-pL(n) to denote a k-DL language using n Boolean attributes.

The first task is to show that k-pL is learnable—that is, that any function in k-DL can be approximated accurately after training on a reasonable number of examples. To do this, we need to calculate the number of hypotheses in the language. Let the language of tests— conjunctions of at most k literals using n attributes—be Conj(n, k). Because a decision list is constructed of tests, and because each test can be attached to either a Yes or a No outcome or can be absent from the decision list, there are at most 3/€°™ (| distinct sets of component tests. Each of these sets of tests can be in any order, so

|k-pL(n)| < 31Co WR Cong (n, KE)! .

The number of conjunctions of & literals from n attributes is given by

. k 2n k comin = =0(n").

; i i=0

Hence, after some work, we obtain

|k-pL(n)| = 20" logz(n®))

We can plug this into Equation (18.1) to show that the number of examples needed for PAClearning a k-pL function is polynomial in n:

N > : (125 + O(nk logs(n)) . Therefore, any algorithm that returns a consistent decision list will PAC-learn a k-pL function in a reasonable number of examples, for small k.

The next task is to find an efficient algorithm that returns a consistent decision list. We will use a greedy algorithm called DECISION-LIST-LEARNING that repeatedly finds a test that agrees exactly with some subset of the training set. Once it finds such a test, it adds it to the decision list under construction and removes the corresponding examples. It then constructs the remainder of the decision list, using just the remaining examples. This is repeated until there are no examples left. The algorithm is shown in Figure 18.14.

This algorithm does not specify the method for selecting the next test to add to the decision list. Although the formal results given earlier do not depend on the selection method,

672

Chapter 18. Learning from Observations

IDENTIFICATION IN THE LIMIT

function DECISION-LIST-LEARNING( examples) returns a decision list, or failure

if examples is empty then return the trivial decision list No t «— a test that matches a nonempty subset examples, of examples such that the members of examples, are all positive or all negative if there is no such ¢ then return failure if the examples in examples, are positive then o — Yes else 0 «— No return a decision list with initial test { and outcome o and remaining tests given by DECISION-LIST-LEARNING (examples — examples.)

Figure 18.14 An algorithm for learning decision lists.

fot ©

2 oo

Proportion correct on test set

0.7 i Decision list -------

0.6 /

0.5

0.4 + , i \_ 0 20 40 60 80 100

Training set size

Figure 18.15 Learning curve for DECISION-LIST-LEARNING algorithm on the restaurant data. The curve for DECISION-TREE-LEARNING is shown for comparison.

it would seem reasonable to prefer small tests that match large sets of uniformly classified examples, so that the overall decision list will be as compact as possible. The simplest strategy is to find the smallest test ¢ that matches any uniformly classified subset, regardless of the size of the subset. Even this approach works quite well, as Figure 18.15 suggests.

Discussion

Computational learning theory has generated a new way of looking at the problem of learning. In the early 1960s, the theory of learning focused on the problem of identification in the limit. According to this notion, an identification algorithm must return a hypothesis that exactly matches the true function. One way to do that is as follows: First, order all the hypotheses in H according to some measure of simplicity. Then, choose the simplest hypothesis consistent with all the examples so far. As new examples arrive, the method will abandon a simpler hypothesis that is invalidated and adopt a more complex one instead. Once it reaches the true function, it will never abandon it. Unfortunately, in many hypothesis spaces, the number of examples and the computation time required to reach the true function are enormous. Thus, computational learning theory does not insist that the learning agent find the “one true

Section 18.6.

Summary 673

law” governing its environment, but instead that it find a hypothesis with a certain degree of predictive accuracy. Computational learning theory also brings sharply into focus the tradeoff between the expressiveness of the hypothesis language and the complexity of learning, and has led directly to an important class of learning algorithms called support vector machines. The PAC-learning results we have shown are worst-case complexity results and do not necessarily reflect the average-case sample complexity as measured by the learning curves we have shown. An average-case analysis must also make assumptions about the distribution of examples and the distribution of true functions that the algorithm will have to learn. As these issues become better understood, computational learning theory continues to provide valuable guidance to machine learning researchers who are interested in predicting or modifying the learning ability of their algorithms. Besides decision lists, results have been obtained for almost all known subclasses of Boolean functions, for sets of first-order logical sentences (see Chapter 19), and for neural networks (see Chapter 20). The results show that the pure inductive learning problem, where the agent begins with no prior knowledge about the target function, is generally very hard. As we show in Chapter 19, the use of prior knowledge to guide inductive learning makes it possible to learn quite large sets of sentences from reasonable numbers of examples, even in a language as expressive as first-order logic.

This chapter has concentrated on inductive learning of deterministic functions from examples. The main points were as follows:

e Learning takes many forms, depending on the nature of the performance element, the component to be improved, and the available feedback.

eo If the available feedback, either from a teacher or from the environment, provides the correct value for the examples, the learning problem is called supervised learning. The task, also called inductive learning, is then to learn a function from examples of its inputs and outputs. Learning a discrete-valued function is called classification; learning a continuous function is called regression.

¢ Inductive learning involves finding a consistent hypothesis that agrees with the examples. Ockham’s razor suggests choosing the simplest consistent hypothesis. The difficulty of this task depends on the chosen representation.

o Decision trees can represent all Boolean functions. The information gain heuristic provides an efficient method for finding a simple, consistent decision tree.

e The performance of a learning algorithm is measured by the learning curve, which shows the prediction accuracy on the test set as a function of the training set size.

e Ensemble methods such as boosting often perform better than individual methods.

¢ Computational learning theory analyzes the sample complexity and computational complexity of inductive learning. There is a tradeoff between the expressiveness of the hypothesis language and the ease of learning.

674

Chapter 18. Learning from Observations

BAGGING

Chapter 1 outlined the history of philosophical investigations into inductive learning. William of Ockham (1280-1349), the most influential philosopher of his century and a major contributer to medieval epistemology, logic, and metaphysics, is credited with a statement called “Ockham’s Razor”—in Latin, Entia non sunt multiplicanda praeter necessitatem, and in English, “Entities are not to be multiplied beyond necessity.” Unfortunately, this laudable piece of advice is nowhere to be found in his writings in precisely these words.

EPAM, the “Elementary Perceiver And Memorizer” (Feigenbaum, 1961), was one of the earliest systems to use decision trees (or discrimination nets). EPAM was intended as a cognitive-simulation model of human concept learning. CLS (Hunt ef al., 1966) used a heuristic look-ahead method to construct decision trees. ID3 (Quinlan, 1979) added the crucial idea of using information content to provide the heuristic function. Information theory itself was developed by Claude Shannon to aid in the study of communication (Shannon and Weaver, 1949). (Shannon also contributed one of the earliest examples of machine learning, a mechanical mouse named Theseus that learned to navigate through a maze by trial and error.) The x? method of tree pruning was described by Quinlan (1986). C4.5, an industrial-strength decision tree package, can be found in Quinlan (1993). An independent tradition of decision tree learning exists in the statistical literature. Classification and Regression Trees (Breiman et al., 1984), known as the “CART book,” is the principal reference.

Many other algorithmic approaches to learning have been tried. The current-besthypothesis approach maintains a single hypothesis, specializing it when it proves too broad and generalizing it when it proves too narrow. This is an old idea in philosophy (Mill, 1843). Early work in cognitive psychology also suggested that it is a natural form of concept learning in humans (Bruner ef al., 1957). In Al, the approach is most closely associated with the work of Patrick Winston, whose Ph.D. thesis (Winston, 1970) addressed the problem of learning descriptions of complex objects. The version space method (Mitchell, 1977, 1982) takes a different approach, maintaining the set of all consistent hypotheses and eliminating those found to be inconsistent with new examples. The approach was used in the Meta-DENDRAL expert system for chemistry (Buchanan and Mitchell, 1978), and later in Mitchell’s (1983) LEX system, which learns to solve calculus problems. A third influential thread was formed by the work of Michalski and colleagues on the AQ series of algorithms, which learned sets of logical rules (Michalski, 1969; Michalski et al., 1986b).

Ensemble learning is an increasingly popular technique for improving the performance of learning algorithms. Bagging (Breiman, 1996), the first effective method, combines hypotheses learned from multiple bootstrap data sets, each generated by subsampling the original data set. The boosting method described in the chapter originated with theoretical work by Schapire (1990). The ADABOOST algorithm was developed by Freund and Schapire (1996) and analyzed theoretically by Schapire (1999). Friedman et al. (2000) explain boosting from a statistician’s viewpoint.

Theoretical analysis of learning algorithms began with the work of Gold (1967) on identification in the limit. This approach was motivated in part by models of scientific

Summary 675

KOLMOGOROV COMPLEXITY

MINIMUM DESCRIPTION LENGTH

UNIFORM CONVERGENCE THEORY

VC DIMENSION

discovery from the philosophy of science (Popper, 1962), but has been applied mainly to the problem of learning grammars from example sentences (Osherson ef al., 1986).

Whereas the identification-in-the-limit approach concentrates on eventual convergence, the study of Kolmogorov complexity or algorithmic complexity, developed independently by Solomonoff (1964) and Kolmogorov (1965), attempts to provide a formal definition for the notion of simplicity used in Ockham’s razor. To escape the problem that simplicity depends on the way in which information is represented, it is proposed that simplicity be measured by the length of the shortest program for a universal Turing machine that correctly reproduces the observed data. Although there are many possible universal Turing machines, and hence many possible “shortest” programs, these programs differ in length by at most a constant that is independent of the amount of data. This beautiful insight, which essentially shows that any initial representation bias will eventually be overcome by the data itself, is marred only by the undecidability of computing the length of the shortest program. Approximate measures such as the minimum description length, or MDL (Rissanen, 1984) can be used instead and have produced excellent results in practice. The text by Li and Vitanyi (1993) is the best source for Kolmogorov complexity.

Computational learning theory—that is, the theory of PAC-learning—was inaugurated by Leslie Valiant (1984). Valiants work stressed the importance of computational and sample complexity. With Michael Kearns (1990), Valiant showed that several concept classes cannot be PAC-learned tractably, even though sufficient information is available in the examples. Some positive results were obtained for classes such as decision lists (Rivest, 1987).

An independent tradition of sample complexity analysis has existed in statistics, beginning with the work on uniform convergence theory (Vapnik and Chervonenkis, 1971). The so-called VC dimension provides a measure roughly analogous to, but more general than, the In [H| measure obtained from PAC analysis. The VC dimension can be applied to continuous function classes, to which standard PAC analysis does not apply. PAC-learning theory and VC theory were first connected by the “four Germans” (none of whom actually is German): Blumer, Ehrenfeucht, Haussler, and Warmuth (1989). Subsequent developments in VC theory led to the invention of the support vector machine or SVM (Boser et al., 1992; Vapnik, 1998), which we describe in Chapter 20.

A large number of important papers on machine learning have been collected in Readings in Machine Learning (Shavlik and Dietterich, 1990). The two volumes Machine Learning 1 (Michalski ef al., 1983) and Machine Learning 2 (Michalski er al., 1986a) also contain many important papers, as well as huge bibliographies. Weiss and Kulikowski (1991) provide a broad introduction to function-learning methods from machine learning, statistics, and neural networks. The STATLOG project (Michie ef al., 1994) is by far the most exhaustive investigation into the comparative performance of learning algorithms. Good current research in machine learning is published in the annual proceedings of the International Conference on Machine Learning and the conference on Neural Information Processing Systems, in Machine Learning and the Journal of Machine Learning Research, and in mainstream Al journals. Work in computational learning theory also appears in the annual ACM Workshop on Computational Learning Theory (COLT), and is described in the texts by Kearns and Vazirani (1994) and Anthony and Bartlett (1999).

676

Chapter 18. Learning from Observations

EXERCISES

CLASS PROBABILITY

18.1 Consider the problem faced by an infant learning to speak and understand a language. Explain how this process fits into the general learning model, identifying each of the components of the model as appropriate.

18.2 Repeat Exercise 18.1 for the case of learning to play tennis (or some other sport with which you are familiar). Is this supervised learning or reinforcement learning?

18.3 Draw a decision tree for the problem of deciding whether to move forward at a road intersection, given that the light has just turned green.

18.4 We never test the same attribute twice along one path in a decision tree. Why not?

18.5 Suppose we generate a training set from a decision tree and then apply decision-tree learning to that training set. Is it the case that the learning algorithm will eventually return the correct tree as the training set size goes to infinity? Why or why not?

18.6 A good “straw man” learning algorithm is as follows: create a table out of all the training examples. Identify which output occurs most often among the training examples; call it d. Then when given an input that is not in the table, just return d. For inputs that are in the table, return the output associated with it (or the most frequent output, if there is more than one). Implement this algorithm and see how well it does on the restaurant domain. This should give you an idea of the baseline for the domain—the minimal performance that any algorithm should be able to obtain.

18.7 Suppose you are running a learning experiment on a new algorithm. You have a data set consisting of 25 examples of each of two classes. You plan to use leave-one-out crossvalidation. As a baseline, you run your experimental setup on a simple majority classifier. (A majority classifier is given a set of training data and then always outputs the class that is in the majority in the training set, regardless of the input.) You expect the majority classifier to score about 50% on leave-one-out cross-validation, but to your surprise, it scores zero. Can you explain why?

18.8 In the recursive construction of decision trees, it sometimes happens that a mixed set of positive and negative examples remains at a leaf node, even after all the attributes have been used. Suppose that we have p positive examples and n negative examples.

a. Show that the solution used by DECISION-TREE-LEARNING, which picks the majority classification, minimizes the absolute error over the set of examples at the leaf.

b. Show that the class probability p/(p + n) minimizes the sum of squared errors.

18.9 Suppose that a learning algorithm is trying to find a consistent hypothesis when the classifications of examples are actually random. There are n Boolean attributes, and examples are drawn uniformly from the set of 2" possible examples. Calculate the number of examples required before the probability of finding a contradiction in the data reaches 0.5.

Section 18.6.

Summary 677

18.10 Suppose that an attribute splits the set of examples FE into subsets E; and that each subset has p; positive examples and n; negative examples. Show that the attribute has strictly positive information gain unless the ratio p; /(p; + n;) is the same for all i.

18.11 Modify DECISION-TREE-LEARNING to include x 2-pruning. You might wish to consult Quinlan (1986) for details.

18.12 The standard DECISION-TREE-LEARNING algorithm described in the chapter does not handle cases in which some examples have missing attribute values.

a. First, we need to find a way to classify such examples, given a decision tree that includes tests on the attributes for which values can be missing. Suppose that an example X has a missing value for attribute A and that the decision tree tests for A at a node that X reaches. One way to handle this case is to pretend that the example has all possible values for the attribute, but to weight each value according to its frequency among all of the examples that reach that node in the decision tree. The classification algorithm should follow all branches at any node for which a value is missing and should multiply the weights along each path. Write a modified classification algorithm for decision trees that has this behavior.

b. Now modify the information gain calculation so that in any given collection of examples C at a given node in the tree during the construction process, the examples with missing values for any of the remaining attributes are given “as-if” values according to the frequencies of those values in the set C'.

18.13 In the chapter, we noted that attributes with many different possible values can cause problems with the gain measure. Such attributes tend to split the examples into numerous small classes or even singleton classes, thereby appearing to be highly relevant according to the gain measure. The gain ratio criterion selects attributes according to the ratio between their gain and their intrinsic information content—that is, the amount of information contained in the answer to the question, “What is the value of this attribute?” The gain ratio criterion therefore tries to measure how efficiently an attribute provides information on the correct classification of an example. Write a mathematical expression for the information content of an attribute, and implement the gain ratio criterion in DECISION-TREE-LEARNING.

18.14 Consider an ensemble learning algorithm that uses simple majority voting among M learned hypotheses. Suppose that each hypothesis has error € and that the errors made by each hypothesis are independent of the others’. Calculate a formula for the error of the ensemble algorithm in terms of M and €, and evaluate it for the cases where A =5, 10, and 20 and e =0.1, 0.2, and 0.4. If the independence assumption is removed, is it possible for the ensemble error to be worse than €?

18.15 This exercise concerns the expressiveness of decision lists (Section 18.5).

a. Show that decision lists can represent any Boolean function, if the size of the tests is not limited.

b. Show that if the tests can contain at most k literals each, then decision lists can represent any function that can be represented by a decision tree of depth k.

1 0 KNOWLEDGE IN LEARNING

PRIOR KNOWLEDGE

In which we examine the problem of learning when you know something already.

In all of the approaches to learning described in the previous three chapters, the idea is to construct a function that has the input/output behavior observed in the data. In each case, the learning methods can be understood as searching a hypothesis space to find a suitable function, starting from only a very basic assumption about the form of the function, such as “second degree polynomial” or “decision tree” and a bias such as “simpler is better.” Doing this amounts to saying that before you can learn something new, you must first forget (almost) everything you know. In this chapter, we study learning methods that can take advantage of prior knowledge about the world. In most cases, the prior knowledge is represented as general first-order logical theories; thus for the first time we bring together the work on knowledge representation and learning.

Chapter 18 defined pure inductive learning as a process of finding a hypothesis that agrees with the observed examples. Here, we specialize this definition to the case where the hypothesis is represented by a set of logical sentences. Example descriptions and classifications will also be logical sentences, and a new example can be classified by inferring a classification sentence from the hypothesis and the example description. This approach allows for incremental construction of hypotheses, one sentence at a time. It also allows for prior knowledge, because sentences that are already known can assist in the classification of new examples. The logical formulation of learning may seem like a lot of extra work at first, but it turns out to clarify many of the issues in learning. It enables us to go well beyond the simple learning methods of Chapter 18 by using the full power of logical inference in the service of learning.

Examples and hypotheses

Recall from Chapter 18 the restaurant learning problem: learning a rule for deciding whether to wait for a table. Examples were described by attributes such as Alternate, Bar, Fri/Sat,

678

Section 19.1.

A Logical Formulation of Learning 679

CANDIDATE DEFINITION

EXTENSION

FALSE NEGATIVE

and so on. In a logical setting, an example is an object that is described by a logical sentence; the attributes become unary predicates. Let us generically call the ith example X;. For instance, the first example from Figure 18.3 is described by the sentences

Alternate(X1) A 2 Bar(X1) A ~Fri/Sat(X1) A Hungry(X1) A...

We will use the notation D; (X;) to refer to the description of X;, where D; can be any logical expression taking a single argument. The classification of the object is given by the sentence

WillWait(X,) .

We will use the generic notation Q(X) if the example is positive, and ~Q(X;) if the example is negative. The complete training set is then just the conjunction of all the description and classification sentences.

The aim of inductive learning in the logical setting is to find an equivalent logical expression for the goal predicate () that we can use to classify examples correctly. Each hypothesis proposes such an expression, which we call a candidate definition of the goal predicate. Using C; to denote the candidate definition, each hypothesis II; is a sentence of the form Vz Q(z) © Ci(x). For example, a decision tree asserts that the goal predicate is true of an object if only if one of the branches leading to true is satisfied. Thus, the Figure 18.6 expresses the following logical definition (which we will call H, for future reference):

Vr WillWait(r) < Patrons(r, Some) V Patrons(r, Full) A Hungry(r) A Type(r, French) V' Patrons(r, Full) A Hungry(r) A Type(r, Thai) (19.1) A Fri/Sat(r) V Patrons(r, Full) A Hungry(r) A Type(r, Burger) .

Each hypothesis predicts that a certain set of examples—namely, those that satisfy its candidate definition—will be examples of the goal predicate. This set is called the extension of the predicate. Two hypotheses with different extensions are therefore logically inconsistent with each other, because they disagree on their predictions for at least one example. If they have the same extension, they are logically equivalent.

The hypothesis space H is the set of all hypotheses { H1, . . . , H,, } that the learning algorithm is designed to entertain. For example, the DECISION-TREE-LEARNING algorithm can entertain any decision tree hypothesis defined in terms of the attributes provided; its hypothesis space therefore consists of all these decision trees. Presumably, the learning algorithm believes that one of the hypotheses is correct; that is, it believes the sentence

HivHy;yVH3Vv...VH,. (19.2)

As the examples arrive, hypotheses that are not consistent with the examples can be ruled out. Let us examine this notion of consistency more carefully. Obviously, if hypothesis H; is consistent with the entire training set, it has to be consistent with each example. What would it mean for it to be inconsistent with an example? This can happen in one of two ways:

¢ An example can be a false negative for the hypothesis, if the hypothesis says it should be negative but in fact it is positive. For instance, the new example X13 described by Patrons( X13, Full) A Wait(Xi3,0-10) A ~Hungry(Xiz) A... A WillWait( X13)

680

Chapter 19. Knowledge in Learning

FALSE POSITIVE

CURRENT-BESTHYPOTHESIS

GENERALIZATION

SPECIALIZATION

would be a false negative for the hypothesis H, given earlier. From H,. and the example description, we can deduce both Will Wait(X;3), which is what the example says, and ~ WillWait(X13), which is what the hypothesis predicts. The hypothesis and the example are therefore logically inconsistent.

e An example can be a false positive for the hypothesis, if the hypothesis says it should be positive but in fact it is negative!

If an example is a false positive or false negative for a hypothesis, then the example and the hypothesis are logically inconsistent with each other. Assuming that the example is a correct observation of fact, then the hypothesis can be ruled out. Logically, this is exactly analogous to the resolution rule of inference (see Chapter 9), where the disjunction of hypotheses corresponds to a clause and the example corresponds to a literal that resolves against one of the literals in the clause. An ordinary logical inference system therefore could, in principle, learn from the example by eliminating one or more hypotheses. Suppose, for example, that the example is denoted by the sentence I, and the hypothesis space is H; V Hy V Hg Vv Hy. Then if I; is inconsistent with Hs and Hj, the logical inference system can deduce the new hypothesis space H1 V Hy.

We therefore can characterize inductive learning in a logical setting as a process of gradually eliminating hypotheses that are inconsistent with the examples, narrowing down the possibilities. Because the hypothesis space is usually vast (or even infinite in the case of first-order logic), we do not recommend trying to build a learning system using resolutionbased theorem proving and a complete enumeration of the hypothesis space. Instead, we will describe two approaches that find logically consistent hypotheses with much less effort.

Current-best-hypothesis search

The idea behind current-best-hypothesis search is to maintain a single hypothesis, and to adjust it as new examples arrive in order to maintain consistency. The basic algorithm was described by John Stuart Mill (1843), and may well have appeared even earlier.

Suppose we have some hypothesis such as H,., of which we have grown quite fond. As long as each new example is consistent, we need do nothing. Then along comes a false negative example, X13. What do we do? Figure 19.1(a) shows H, schematically as a region: everything inside the rectangle is part of the extension of I1,.. The examples that have actually been seen so far are shown as “+” or “~”, and we see that I, correctly categorizes all the examples as positive or negative examples of WillWait. In Figure 19.1(b), a new example (circled) is a false negative: the hypothesis says it should be negative but it is actually positive. The extension of the hypothesis must be increased to include it. This is called generalization; one possible generalization is shown in Figure 19.1(c). Then in Figure 19.1(d), we see a false positive: the hypothesis says the new example (circled) should be positive, but it actually is negative. The extension of the hypothesis must be decreased to exclude the example. This is called specialization; in Figure 19.1(e) we see one possible specialization of the hypothesis.

1 The terms “false positive” and “false negative” are used in medicine to describe erroneous results from lab tests. A result is a false positive if it indicates that the patient has the disease when in fact no disease is present.

Section 19.1.

A Logical Formulation of Learning 681

[]

Figure 19.1 (a) A consistent hypothesis. (b) A false negative. (c) The hypothesis is generalized. (d) A false positive. (e) The hypothesis is specialized.

The “more general than” and “more specific than” relations between hypotheses provide the logical structure on the hypothesis space that makes efficient search possible.

We can now specify the CURRENT-BEST-LEARNING algorithm, shown in Figure 19.2. Notice that each time we consider generalizing or specializing the hypothesis, we must check for consistency with the other examples, because an arbitrary increase/decrease in the extension might include/exclude previously seen negative/positive examples.

function CURRENT-BEST-LEARNING( examples) returns a hypothesis

H «+ any hypothesis consistent with the first example in examples for each remaining example in examples do if e is false positive for H then H « choose a specialization of H consistent with examples else if ¢ is false negative for H then H « choose a generalization of H consistent with examples if no consistent specialization/generalization can be found then fail return H

Figure 19.2 The current-best-hypothesis learning algorithm. It searches for a consistent hypothesis and backtracks when no consistent specialization/generalization can be found.

We have defined generalization and specialization as operations that change the extension of a hypothesis. Now we need to determine exactly how they can be implemented as syntactic operations that change the candidate definition associated with the hypothesis, so that a program can carry them out. This is done by first noting that generalization and specialization are also logical relationships between hypotheses. If hypothesis Hj, with definition C1, is a generalization of hypothesis Hy with definition C5, then we must have

Va C(x) = Ci(z).

Therefore in order to construct a generalization of Hy, we simply need to find a definition C7 that is logically implied by Co. This is easily done. For example, if Cy(z) is

682

Chapter 19. Knowledge in Learning

DROPPING CONDITIONS

Alternate(x) A Patrons(x, Some), then one possible generalization is given by C(x) = Patrons(z, Some). This is called dropping conditions. Intuitively, it generates a weaker definition and therefore allows a larger set of positive examples. There are a number of other generalization operations, depending on the language being operated on. Similarly, we can specialize a hypothesis by adding extra conditions to its candidate definition or by removing disjuncts from a disjunctive definition. Let us see how this works on the restaurant example, using the data in Figure 18.3.

e The first example X is positive. Alternate(X1) is true, so let the initial hypothesis be Hy: Vz WillWait(x) < Alternate(z) .

e The second example Xs is negative. H; predicts it to be positive, so it is a false positive. Therefore, we need to specialize Hy. This can be done by adding an extra condition that will rule out Xs. One possibility is

Hy: Va WillWait(z) < Alternate(z) A Patrons(z, Some) .

o The third example X3 is positive. Il predicts it to be negative, so it is a false negative. Therefore, we need to generalize Hy. We drop the Alternate condition, yielding

Hz: Ya WillWait(x) < Patrons(xz, Some) .

o The fourth example X is positive. Hs predicts it to be negative, so it is a false negative. We therefore need to generalize 3. We cannot drop the Patrons condition, because that would yield an all-inclusive hypothesis that would be inconsistent with X53. One possibility is to add a disjunct:

Hy: Va WillWait(z) < Patrons(z, Some) V (Patrons(x, Full) A Fri/Sat(z)) . Already, the hypothesis is starting to look reasonable. Obviously, there are other possibilities consistent with the first four examples; here are two of them:

Hy: Vz WillWait(z) < —WaitEstimate(z, 30-60) .

HY: Vz WillWait(z) < Patrons(z, Some) V (Patrons(x, Full) A WaitEstimate(x, 10-30)) .

The CURRENT-BEST-LEARNING algorithm is described nondeterministically, because at any point, there may be several possible specializations or generalizations that can be applied. The choices that are made will not necessarily lead to the simplest hypothesis, and may lead to an unrecoverable situation where no simple modification of the hypothesis is consistent with all of the data. In such cases, the program must backtrack to a previous choice point.

The CURRENT-BEST-LEARNING algorithm and its variants have been used in many machine learning systems, starting with Patrick Winston’s (1970) “arch-learning” program. With a large number of instances and a large space, however, some difficulties arise:

1. Checking all the previous instances over again for each modification is very expensive.

2. The search process may involve a great deal of backtracking. As we saw in Chapter 18, hypothesis space can be a doubly exponentially large place.

Section 19.1. A Logical Formulation of Learning 683

Least-commitment search

Backtracking arises because the current-best-hypothesis approach has to choose a particular hypothesis as its best guess even though it does not have enough data yet to be sure of the choice. What we can do instead is to keep around all and only those hypotheses that are consistent with all the data so far. Each new instance will either have no effect or will get rid of some of the hypotheses. Recall that the original hypothesis space can be viewed as a disjunctive sentence

Hv Hy;yvH;...vVH,.

As various hypotheses are found to be inconsistent with the examples, this disjunction shrinks,

retaining only those hypotheses not ruled out. Assuming that the original hypothesis space

does in fact contain the right answer, the reduced disjunction must still contain the right an

swer because only incorrect hypotheses have been removed. The set of hypotheses remaining VERSION SPACE is called the version space, and the learning algorithm (sketched in Figure 19.3) is called the CTE version space learning algorithm (also the candidate elimination algorithm).

function VERSION-SPACE-LEARNING(ezamples) returns a version space local variables: V, the version space: the set of all hypotheses

V «the set of all hypotheses for each example e in examples do

if V is not empty then V «— VERSION-SPACE-UPDATE(V, ¢) return V

function VERSION-SPACE-UPDATE(V, e) returns an updated version space V—{h € V: his consistent with e}

Figure 19.3 The version space learning algorithm. It finds a subset of V that is consistent with the examples.

One important property of this approach is that it is incremental: one never has to go back and reexamine the old examples. All remaining hypotheses are guaranteed to be consistent with them anyway. It is also a least-commitment algorithm because it makes no arbitrary choices (cf. the partial-order planning algorithm in Chapter 11). But there is an obvious problem. We already said that the hypothesis space is enormous, so how can we possibly write down this enormous disjunction?

The following simple analogy is very helpful. How do you represent all the real numbers between 1 and 2? After all, there is an infinite number of them! The answer is to use an interval representation that just specifies the boundaries of the set: [1,2]. It works because we have an ordering on the real numbers.

We also have an ordering on the hypothesis space, namely, generalization/specialization. This is a partial ordering, which means that each boundary will not be a point but rather a

BOUNDARY SET set of hypotheses called a boundary set. The great thing is that we can represent the entire

684

Chapter 19. Knowledge in Learning

G-SET S-8ET

This region all inconsistent

G G, G,

eee G, A— A / More general / A \ 1 | \ J \ ‘ / N . \ \ AN \ / \ “Y ’ N / | ~~ ’ » More specific A Si Ss se.

n

This region all inconsistent

Figure 19.4 The version space contains all hypotheses consistent with the examples.

version space using just two boundary sets: a most general boundary (the G-set) and a most specific boundary (the S-set). Everything in between is guaranteed to be consistent with the examples. Before we prove this, let us recap:

e The current version space is the set of hypotheses consistent with all the examples so far. It is represented by the S-set and G-set, each of which is a set of hypotheses.

eo Every member of the S-set is consistent with all observations so far, and there are no consistent hypotheses that are more specific.

e Every member of the G-set is consistent with all observations so far, and there are no consistent hypotheses that are more general.

We want the initial version space (before any examples have been seen) to represent all possible hypotheses. We do this by setting the G-set to contain True (the hypothesis that contains everything), and the S-set to contain False (the hypothesis whose extension is empty). Figure 19.4 shows the general structure of the boundary set representation of the version space. To show that the representation is sufficient, we need the following two properties:

1. Every consistent hypothesis (other than those in the boundary sets) is more specific than some member of the G-set, and more general than some member of the S-set. (That is, there are no “stragglers” left outside.) This follows directly from the definitions of S and G. If there were a straggler h, then it would have to be no more specific than any member of G, in which case it belongs in G; or no more general than any member of S, in which case it belongs in S.

2. Every hypothesis more specific than some member of the G-set and more general than some member of the S-set is a consistent hypothesis. (That is, there are no “holes” be

tween the boundaries.) Any h between .S and G must reject all the negative examples rejected by each member of GG (because it is more specific), and must accept all the positive examples accepted by any member of .S' (because it is more general). Thus, h must agree with all the examples, and therefore cannot be inconsistent. Figure 19.5 shows the situation: there are no known examples outside S but inside (7, so any hypothesis in the gap must be consistent.

Figure 19.5 The extensions of the members of G and S. No known examples lie in between the two sets of boundaries.

We need to worry about the members 5; and G; of the Sand G-sets. For each one, the

False positive for .S;: This means 5; is too general, but there are no consistent specializations of .S; (by definition), so we throw it out of the S-set.

generalizations, provided they are more specific than some member of G.

specializations, provided they are more general than some member of S. False negative for GG;: This means G; is too specific, but there are no consistent generalizations of G; (by definition) so we throw it out of the G-set.

We have exactly one concept left in the version space, in which case we return it as the unique hypothesis.

no consistent hypotheses for the training set. This is the same case as the failure of the simple version of the decision tree algorithm.

686

Chapter 19. Knowledge in Learning

GENERALIZATION HIERARCHY

3. We run out of examples with several hypotheses remaining in the version space. This means the version space represents a disjunction of hypotheses. For any new example, if all the disjuncts agree, then we can return their classification of the example. If they disagree, one possibility is to take the majority vote.

We leave as an exercise the application of the VERSION-SPACE-LEARNING algorithm to the restaurant data. There are two principal drawbacks to the version-space approach:

o If the domain contains noise or insufficient attributes for exact classification, the version space will always collapse.

o If we allow unlimited disjunction in the hypothesis space, the S-set will always contain a single most-specific hypothesis, namely, the disjunction of the descriptions of the positive examples seen to date. Similarly, the G-set will contain just the negation of the disjunction of the descriptions of the negative examples.

e For some hypothesis spaces, the number of elements in the S-set of G-set may grow exponentially in the number of attributes, even though efficient learning algorithms exist for those hypothesis spaces.

To date, no completely successful solution has been found for the problem of noise. The problem of disjunction can be addressed by allowing limited forms of disjunction or by including a generalization hierarchy of more general predicates. For example, instead of using the disjunction WaitEstimate(x,30-60) Vv WaitEstimate(z,>60), we might use the single literal LongWait(x). The set of generalization and specialization operations can be easily extended to handle this.

The pure version space algorithm was first applied in the Meta-DENDRAL system, which was designed to learn rules for predicting how molecules would break into pieces in a mass spectrometer (Buchanan and Mitchell, 1978). Meta-DENDRAL was able to generate rules that were sufficiently novel to warrant publication in a journal of analytical chemistry— the first real scientific knowledge generated by a computer program. It was also used in the elegant LEX system (Mitchell ef al., 1983), which was able to learn to solve symbolic integration problems by studying its own successes and failures. Although version space methods are probably not practical in most real-world learning problems, mainly because of noise, they provide a good deal of insight into the logical structure of hypothesis space.

The preceding section described the simplest setting for inductive learning. To understand the role of prior knowledge, we need to talk about the logical relationships among hypotheses, example descriptions, and classifications. Let Descriptions denote the conjunction of all the example descriptions in the training set, and let Classifications denote the conjunction of all the example classifications. Then a Hypothesis that “explains the observations” must satisfy

Section 19.2.

Knowledge in Learning 687

the following property (recall that = means “logically entails”): Hypothesis A Descriptions |= Classifications . (19.3)

We call this kind of relationship an entailment constraint, in which Hypothesis is the “unknown.” Pure inductive learning means solving this constraint, where Hypothesis is drawn from some predefined hypothesis space. For example, if we consider a decision tree as a logical formula (see Equation (19.1) on page 679), then a decision tree that is consistent with all the examples will satisfy Equation (19.3). If we place no restrictions on the logical form of the hypothesis, of course, then Hypothesis = Classifications also satisfies the constraint. Ockham’s razor tells us to prefer small, consistent hypotheses, so we try to do better than simply memorizing the examples.

This simple knowledge-free picture of inductive learning persisted until the early 1980s. The modern approach is to design agents that already know something and are trying to learn some more. This may not sound like a terrifically deep insight, but it makes quite a difference to the way we design agents. It might also have some relevance to our theories about how science itself works. The general idea is shown schematically in Figure 19.6.

[]

Figure 19.6 A cumulative learning process uses, and adds to, its stock of background knowledge over time.

If we want to build an autonomous learning agent that uses background knowledge, the agent must have some method for obtaining the background knowledge in the first place, in order for it to be used in the new learning episodes. This method must itself be a learning process. The agent’s life history will therefore be characterized by cumulative, or incremental, development. Presumably, the agent could start out with nothing, performing inductions in vacuo like a good little pure induction program. But once it has eaten from the Tree of Knowledge, it can no longer pursue such naive speculations and should use its background knowledge to learn more and more effectively. The question is then how to actually do this.

Some simple examples

Let us consider some commonsense examples of learning with background knowledge. Many apparently rational cases of inferential behavior in the face of observations clearly do not follow the simple principles of pure induction.

¢ Sometimes one leaps to general conclusions after only one observation. Gary Larson once drew a cartoon in which a bespectacled caveman, Zog, is roasting his lizard on

688

Chapter 19. Knowledge in Learning

the end of a pointed stick. He is watched by an amazed crowd of his less intellectual contemporaries, who have been using their bare hands to hold their victuals over the fire. This enlightening experience is enough to convince the watchers of a general principle of painless cooking.

e Or consider the case of the traveller to Brazil meeting her first Brazilian. On hearing him speak Portuguese, she immediately concludes that Brazilians speak Portuguese, yet on discovering that his name is Fernando, she does not conclude that all Brazilians are called Fernando. Similar examples appear in science. For example, when a freshman physics student measures the density and conductance of a sample of copper at a particular temperature, she is quite confident in generalizing those values to all pieces of copper. Yet when she measures its mass, she does not even consider the hypothesis that all pieces of copper have that mass. On the other hand, it would be quite reasonable to make such a generalization over all pennies.

o Finally, consider the case of a pharmacologically ignorant but diagnostically sophisticated medical student observing a consulting session between a patient and an expert internist. After a series of questions and answers, the expert tells the patient to take a course of a particular antibiotic. The medical student infers the general rule that that particular antibiotic is effective for a particular type of infection.

These are all cases in which the use of background knowledge allows much faster learning than one might expect from a pure induction program.

Some general schemes

In each of the preceding examples, one can appeal to prior knowledge to try to justify the generalizations chosen. We will now look at what kinds of entailment constraints are operating in each case. The constraints will involve the Background knowledge, in addition to the Hypothesis and the observed Descriptions and Classifications.

In the case of lizard toasting, the cavemen generalize by explaining the success of the pointed stick: it supports the lizard while keeping the hand away from the fire. From this explanation, they can infer a general rule: that any long, rigid, sharp object can be used to toast small, soft-bodied edibles. This kind of generalization process has been called explanationbased learning, or EBL. Notice that the general rule follows logically from the background knowledge possessed by the cavemen. Hence, the entailment constraints satisfied by EBL are the following:

Hypothesis A Descriptions |= Classifications Background |= Hypothesis .

Because EBL uses Equation (19.3), it was initially thought to be a better way to learn from examples. But because it requires that the background knowledge be sufficient to explain the Hypothesis, which in turn explains the observations, the agent does not actually learn anything factually new from the instance. The agent could have derived the example from what it already knew, although that might have required an unreasonable amount of computation. EBL is now viewed as a method for converting first-principles theories into useful, special-purpose knowledge. We describe algorithms for EBL in Section 19.3.

Knowledge in Learning 689

RELEVANCE

RELEVANCE-BASED LEARNING

KNOWLEDGE-BASED INDUCTIVE LEARNING

INDUCTIVE LOGIC PROGRAMMING

The situation of our traveler in Brazil is quite different, for she cannot necessarily explain why Fernando speaks the way he does, unless she knows her Papal bulls. Moreover, the same generalization would be forthcoming from a traveler entirely ignorant of colonial history. The relevant prior knowledge in this case is that, within any given country, most people tend to speak the same language; on the other hand, Fernando is not assumed to be the name of all Brazilians because this kind of regularity does not hold for names. Similarly, the freshman physics student also would be hard put to explain the particular values that she discovers for the conductance and density of copper. She does know, however, that the material of which an object is composed and its temperature together determine its conductance. In each case, the prior knowledge Background concerns the relevance of a set of features to the goal predicate. This knowledge, fogether with the observations, allows the agent to infer a new, general rule that explains the observations:

Hypothesis A Descriptions |= Classifications , 19.4) Background A Descriptions A Classifications |= Hypothesis . (19.

We call this kind of generalization relevance-based learning, or RBL (although the name is not standard). Notice that whereas RBL does make use of the content of the observations, it does not produce hypotheses that go beyond the logical content of the background knowledge and the observations. It is a deductive form of learning and cannot by itself account for the creation of new knowledge starting from scratch.

In the case of the medical student watching the expert, we assume that the student’s prior knowledge is sufficient to infer the patient’s disease D from the symptoms. This is not, however, enough to explain the fact that the doctor prescribes a particular medicine M. The student needs to propose another rule, namely, that M generally is effective against D. Given this rule and the student’s prior knowledge, the student can now explain why the expert prescribes M in this particular case. We can generalize this example to come up with the entailment constraint:

Background A Hypothesis A Descriptions |= Classifications . (19.5)

That is, the background knowledge and the new hypothesis combine to explain the examples. As with pure inductive learning, the learning algorithm should propose hypotheses that are as simple as possible, consistent with this constraint. Algorithms that satisfy constraint (19.5) are called knowledge-based inductive learning, or KBIL, algorithms.

KBIL algorithms, which are described in detail in Section 19.5, have been studied mainly in the field of inductive logic programming, or ILP. In ILP systems, prior knowledge plays two key roles in reducing the complexity of learning:

1. Because any hypothesis generated must be consistent with the prior knowledge as well as with the new observations, the effective hypothesis space size is reduced to include only those theories that are consistent with what is already known.

2. For any given set of observations, the size of the hypothesis required to construct an explanation for the observations can be much reduced, because the prior knowledge will be available to help out the new rules in explaining the observations. The smaller the hypothesis, the easier it is to find.

690

Chapter 19. Knowledge in Learning

In addition to allowing the use of prior knowledge in induction, ILP systems can formulate hypotheses in general first-order logic, rather than in the restricted attribute-based language of Chapter 18. This means that they can learn in environments that cannot be understood by simpler systems.

MEMOIZATION

As we explained in the introduction to this chapter, explanation-based learning is a method for extracting general rules from individual observations. As an example, consider the problem of differentiating and simplifying algebraic expressions (Exercise 9.15). If we differentiate an expression such as X? with respect to X, we obtain 2X. (Notice that we use a capital letter for the arithmetic unknown X, to distinguish it from the logical variable x.) In a logical reasoning system, the goal might be expressed as ASK(Derivative(X? X)=d, KB), with solutiond = 2X.

Anyone who knows differential calculus can see this solution “by inspection” as a result of practice in solving such problems. A student encountering such problems for the first time, or a program with no experience, will have a much more difficult job. Application of the standard rules of differentiation eventually yields the expression 1 x (2 x (X(271)), and eventually this simplifies to 2X. In the authors’ logic programming implementation, this takes 136 proof steps, of which 99 are on dead-end branches in the proof. After such an experience, we would like the program to solve the same problem much more quickly the next time it arises..

The technique of memoization has long been used in computer science to speed up programs by saving the results of computation. The basic idea of memo functions is to accumulate a database of input/output pairs; when the function is called, it first checks the database to see whether it can avoid solving the problem from scratch. Explanation-based learning takes this a good deal further, by creating general rules that cover an entire class of cases. In the case of differentiation, memoization would remember that the derivative of X?2 with respect to X is 2X, but would leave the agent to calculate the derivative of Z2 with respect to Z from scratch. We would like to be able to extract the general rule? that for any arithmetic unknown u, the derivative of ©? with respect to u is 2u. In logical terms, this is expressed by the rule

ArithmeticUnknown(u) = Derivative(u®, uv) =2u .

If the knowledge base contains such a rule, then any new case that is an instance of this rule can be solved immediately.

This is, of course, merely a trivial example of a very general phenomenon. Once something is understood, it can be generalized and reused in other circumstances. It becomes an “obvious” step and can then be used as a building block in solving problems still more complex. Alfred North Whitehead (1911), co-author with Bertrand Russell of Principia Mathe

2 Of course, a general rule for u™ can also be produced, but the current example suffices to make the point.

Section 19.3.

Explanation-Based Learning 691

matica, wrote “Civilization advances by extending the number of important operations that we can do without thinking about them,” perhaps himself applying EBL to his understanding of events such as Zog’s discovery. If you have understood the basic idea of the differentiation example, then your brain is already busily trying to extract the general principles of explanation-based learning from it. Notice that you hadn’t already invented EBL before you saw the example. Like the cavemen watching Zog, you (and we) needed an example before we could generate the basic principles. This is because explaining why something is a good idea is much easier than coming up with the idea in the first place.

Extracting general rules from examples

The basic idea behind EBL is first to construct an explanation of the observation using prior knowledge, and then to establish a definition of the class of cases for which the same explanation structure can be used. This definition provides the basis for a rule covering all of the cases in the class. The “explanation” can be a logical proof, but more generally it can be any reasoning or problem-solving process whose steps are well defined. The key is to be able to identify the necessary conditions for those same steps to apply to another case.

We will use for our reasoning system the simple backward-chaining theorem prover described in Chapter 9. The proof tree for Derivative(X?, X)=2X is too large to use as an example, so we will use a simpler problem to illustrate the generalization method. Suppose our problem is to simplify 1 x (0 + X). The knowledge base includes the following rules:

Rewrite(u, v) A Simplify(v,w) = Simplify(u,w) . Primitive(u) = Simplify(u, u) . ArithmeticUnknown(u) = Primitive(u) . Number(u) = Primitive(u) .

Rewrite(1 x u, u)

Rewrite(0 + u, u) .

The proof that the answer is X is shown in the top half of Figure 19.7. The EBL method actually constructs two proof trees simultaneously. The second proof tree uses a variabilized goal in which the constants from the original goal are replaced by variables. As the original proof proceeds, the variabilized proof proceeds in step, using exactly the same rule applications. This could cause some of the variables to become instantiated. For example, in order to use the rule Rewrite(1 x u, u), the variable x in the subgoal Rewrite(x x (y + z), v) must be bound to 1. Similarly, y must be bound to 0 in the subgoal Rewrite(y + z,v’) in order to use the rule Rewrite(0 + u,v). Once we have the generalized proof tree, we take the leaves (with the necessary bindings) and form a general rule for the goal predicate:

Rewrite(1 x (0+ 2),0+ 2) A Rewrite(0 + z, z) A ArithmeticUnknown(z) = Simplify(1 x (0+ 2), 2) . Notice that the first two conditions on the left-hand side are true regardless of the value of z. We can therefore drop them from the rule, yielding

ArithmeticUnknown(z) = Simplify(1 x (0+ 2), 2) .

692

Chapter 19. Knowledge in Learning

[]

Yes, { }

Figure 19.7 Proof trees for the simplification problem. The first tree shows the proof for the original problem instance, from which we can derive

ArithmeticUnknown(z) = Simplify(1 x (0+ 2), 2) .

The second shows the proof for a problem instance with all constants replaced by variables, from which we can derive a variety of other rules.

In general, conditions can be dropped from the final rule if they impose no constraints on the variables on the right-hand side of the rule, because the resulting rule will still be true and will be more efficient. Notice that we cannot drop the condition ArithmeticUnknown(z), because not all possible values of z are arithmetic unknowns. Values other than arithmetic unknowns might require different forms of simplification: for example, if =z were 2 x 3, then the correct simplification of 1 x (0 + (2 x 3)) would be 6 and not 2 x 3.

To recap, the basic EBL process works as follows:

1. Given an example, construct a proof that the goal predicate applies to the example using the available background knowledge.

2. In parallel, construct a generalized proof tree for the variabilized goal using the same inference steps as in the original proof.

3. Construct a new rule whose left-hand side consists of the leaves of the proof tree and whose right-hand side is the variabilized goal (after applying the necessary bindings from the generalized proof).

4. Drop any conditions that are true regardless of the values of the variables in the goal.

Section 19.3.

Explanation-Based Learning 693

OPERATIONALITY

Improving efficiency

The generalized proof tree in Figure 19.7 actually yields more than one generalized rule. For example, if we terminate, or prune, the growth of the right-hand branch in the proof tree when it reaches the Primitive step, we get the rule

Primitive(z) = Simplify(1 x (0+ 2), 2) .

This rule is as valid as, but more general than, the rule using Arithmetic Unknown, because it covers cases where z is a number. We can extract a still more general rule by pruning after the step Simplify(y + z,w), yielding the rule

Simplify(y + z,w) = Simplify(1 x (y + 2), w) .

In general, a rule can be extracted from any partial subtree of the generalized proof tree. Now we have a problem: which of these rules do we choose?

The choice of which rule to generate comes down to the question of efficiency. There are three factors involved in the analysis of efficiency gains from EBL:

1. Adding large numbers of rules can slow down the reasoning process, because the inference mechanism must still check those rules even in cases where they do not yield a solution. In other words, it increases the branching factor in the search space.

2. To compensate for the slowdown in reasoning, the derived rules must offer significant increases in speed for the cases that they do cover. These increases come about mainly because the derived rules avoid dead ends that would otherwise be taken, but also because they shorten the proof itself.

3. Derived rules should be as general as possible, so that they apply to the largest possible set of cases.

A common approach to ensuring that derived rules are efficient is to insist on the operationality of each subgoal in the rule. A subgoal is operational if it is “easy” to solve. For example, the subgoal Primitive(z) is easy to solve, requiring at most two steps, whereas the subgoal Simplify(y + z,w) could lead to an arbitrary amount of inference, depending on the values of y and z. If a test for operationality is carried out at each step in the construction of the generalized proof, then we can prune the rest of a branch as soon as an operational subgoal is found, keeping just the operational subgoal as a conjunct of the new rule.

Unfortunately, there is usually a tradeoff between operation