Statement on AI Risk

### Contents

[Statement](#open-letter)[Signatories](#signatories)[Sign the statement](#sign)###### **AI experts, journalists, policymakers, and the public are increasingly discussing a broad spectrum of important and urgent risks from AI. Even so, it can be difficult to voice concerns about some of advanced AI’s most severe risks. The succinct statement below aims to overcome this obstacle and open up discussion. It is also meant to create common knowledge of the growing number of experts and public figures who also take some of advanced AI’s most severe risks seriously.**

### Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.

### 9

###  *Signatories:*

AI ScientistsOther Notable FiguresThank you! Your submission has been received!Oops! Something went wrong while submitting the form.Geoffrey HintonAI ScientistsEmeritus Professor of Computer Science, University of TorontoYoshua BengioAI ScientistsProfessor of Computer Science, U. Montreal / MilaDemis HassabisAI ScientistsCEO, Google DeepMindSam AltmanOther Notable FiguresCEO, OpenAIDario AmodeiAI ScientistsCEO, AnthropicDawn SongAI ScientistsProfessor of Computer Science, UC BerkeleyTed LieuOther Notable FiguresCongressman, US House of RepresentativesBill GatesOther Notable FiguresGates VenturesYa-Qin ZhangAI ScientistsProfessor and Dean, AIR, Tsinghua UniversityIlya SutskeverAI ScientistsCo-Founder and Chief Scientist, OpenAIIgor BabuschkinAI ScientistsCo-Founder, xAIShane LeggAI ScientistsChief AGI Scientist and Co-Founder, Google DeepMindMartin HellmanOther Notable FiguresProfessor Emeritus of Electrical Engineering, StanfordJames ManyikaAI ScientistsSVP, Research, Technology & Society, Google-AlphabetYi ZengAI ScientistsProfessor and Director of Brain-inspired Cognitive AI Lab, Institute of Automation, Chinese Academy of SciencesAaron XiangOther Notable FiguresExecutive Chairman, United Nations Sciences and Technology OrganizationXianyuan ZhanAI ScientistsAssistant Professor, Tsinghua UniversityAlbert EfimovAI ScientistsChief of Research, Russian Association of Artificial IntelligenceAlvin Wang GraylinOther Notable FiguresChina President, HTCJianyi ZhangAI ScientistsProfessor, Beijing Electronic Science and Technology InstituteAnca DraganAI ScientistsAssociate Professor of Computer Science, UC BerkeleyChristine ParthemoreOther Notable FiguresCEO and Director of the Janne E. Nolan Center on Strategic Weapons, The Council on Strategic RisksBill McKibbenOther Notable FiguresSchumann Distinguished Scholar, Middlebury CollegeAlan RobockOther Notable FiguresDistinguished Professor of Climate Science, Rutgers UniversityAngela KaneOther Notable FiguresVice President, International Institute for Peace, Vienna; former UN High Representative for Disarmament AffairsAudrey TangOther Notable FiguresDigitalminister.tw and Chair of National Institute of Cyber SecurityDaniela AmodeiOther Notable FiguresPresident, AnthropicDavid SilverAI ScientistsProfessor of Computer Science, Google DeepMind and UCLLila IbrahimOther Notable FiguresCOO, Google DeepMindStuart RussellAI ScientistsProfessor of Computer Science, UC BerkeleyTony (Yuhuai) WuAI ScientistsCo-Founder, xAIMarian Rogers CroakOther Notable FiguresVP Center for Responsible AI and Human Centered Technology, GoogleAndrew BartoAI ScientistsProfessor Emeritus, University of MassachusettsMira MuratiOther Notable FiguresCTO, OpenAIJaime Fernández FisacAI ScientistsAssistant Professor of Electrical and Computer Engineering, Princeton UniversityDiyi YangAI ScientistsAssistant Professor, Stanford UniversityGillian HadfieldOther Notable FiguresProfessor, CIFAR AI Chair, University of Toronto, Vector Institute for AILaurence TribeOther Notable FiguresUniversity Professor Emeritus, Harvard UniversityPattie MaesAI ScientistsProfessor, Massachusetts Institute of Technology - Media LabKevin ScottOther Notable FiguresCTO, MicrosoftEric HorvitzAI ScientistsChief Scientific Officer, MicrosoftPeter NorvigAI ScientistsEducation Fellow, Stanford UniversityJoseph SifakisOther Notable FiguresTuring Award 2007, Professor, CNRS-Universite Grenoble-AlpesAtoosa KasirzadehAI ScientistsAssistant Professor, University of Edinburgh, Alan Turing InstituteErik BrynjolfssonOther Notable FiguresProfessor and Senior Fellow, Stanford Institute for Human-Centered AIMustafa SuleymanOther Notable FiguresCEO, Inflection AIEmad MostaqueOther Notable FiguresCEO, Stability AIIan GoodfellowAI ScientistsPrincipal Scientist, Google DeepMindJohn SchulmanAI ScientistsCo-Founder, OpenAIWojciech ZarembaAI ScientistsCo-Founder, OpenAIBaburam BhattaraiOther Notable FiguresFormer Prime Minister of Nepal, Society of Nepalese ArchitectsKersti KaljulaidOther Notable FiguresFormer President of the Republic of EstoniaRussell SchweickartOther Notable FiguresApollo 9 Astronaut, Association of Space Explorers, B612 FoundationAndy WeberOther Notable FiguresFormer U.S. Assistant Secretary of Defense for Nuclear, Chemical and Biological Defense Programs, Council on Strategic RisksAllison MacfarlaneOther Notable FiguresFormer Chairman, US Nuclear Regulatory CommissionNicholas Fairfax (Lord Fairfax)Other Notable FiguresMember, House of LordsLord StrathcarronOther Notable FiguresPeer, House of LordsStephen LubyOther Notable FiguresProfessor of Medicine (Infectious Diseases), Stanford UniversityDavid HausslerOther Notable FiguresProfessor and Director of the Genomics Institute, UC Santa CruzJu LiOther Notable FiguresProfessor of Nuclear Science and Engineering and Professor of Materials Science and Engineering, Massachusetts Institute of TechnologyDavid ChalmersOther Notable FiguresProfessor of Philosophy, New York UniversityDaniel DennettOther Notable FiguresEmeritus Professor of Philosophy, Tufts UniversityPeter RailtonOther Notable FiguresProfessor of Philosophy at University of Michigan, Ann ArborSheila McIlraithAI ScientistsProfessor of Computer Science, University of TorontoVictoria KrakovnaAI ScientistsResearch Scientist, Google DeepMindMary PhuongAI ScientistsResearch Scientist, Google DeepMindMariano-Florentino CuéllarOther Notable FiguresPresident, Carnegie Endowment for International PeaceLex FridmanAI ScientistsResearch Scientist, MITSharon LiAI ScientistsAssistant Professor of Computer Science, University of Wisconsin MadisonPhillip IsolaAI ScientistsAssociate Professor of Electrical Engineering and Computer Science, MITDavid KruegerAI ScientistsAssistant Professor of Computer Science, University of CambridgeJacob SteinhardtAI ScientistsAssistant Professor of Computer Science, UC BerkeleyMartin ReesOther Notable FiguresProfessor of Physics, Cambridge UniversityNando de FreitasAI ScientistsDirector, Science Board, Google DeepMindHe HeAI ScientistsAssistant Professor of Computer Science and Data Science, New York UniversityDavid McAllesterAI ScientistsProfessor of Computer Science, TTICVincent ConitzerAI ScientistsProfessor of Computer Science, Carnegie Mellon University and University of OxfordBart SelmanAI ScientistsProfessor of Computer Science, Cornell UniversityPhilip TorrAI ScientistsProfessor of Engineering Science, University of OxfordJames MickensOther Notable FiguresProfessor of Computer Science, Harvard UniversityMichael WellmanAI ScientistsProfessor and Chair of Computer Science & Engineering, University of MichiganLuis VidegarayOther Notable FiguresSenior Lecturer, MIT; Former Minister of Interior and Exterior Relations of MexicoJinwoo ShinAI ScientistsKAIST Endowed Chair Professor, Korea Advanced Institute of Science and TechnologyDae-Shik KimAI ScientistsProfessor of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST)Edith ElkindAI ScientistsProfessor of Computing Science, University of OxfordRay KurzweilOther Notable FiguresPrincipal Researcher + AI Visionary, GoogleFrank HutterAI ScientistsProfessor of Machine Learning, Head of ELLIS Unit, University of FreiburgAlexey DosovitskiyAI ScientistsResearch Scientist, Google DeepMindJaan TallinnOther Notable FiguresCo-Founder of SkypeVitalik ButerinOther Notable FiguresFounder and Chief Scientist, Ethereum, Ethereum FoundationAdam D'AngeloOther Notable FiguresCEO, Quora, and board member, OpenAISimon LastOther Notable FiguresCofounder & CTO, NotionDustin MoskovitzOther Notable FiguresCo-founder & CEO, AsanaShane TorchianaOther Notable FiguresCEO, BirdThuan Q. PhamOther Notable FiguresFormer CTO, Uber, Board member, NubankScott AaronsonAI ScientistsSchlumberger Chair of Computer Science, University of Texas at AustinMax TegmarkAI ScientistsProfessor, MIT, Center for AI and Fundamental InteractionsBruce SchneierOther Notable FiguresLecturer, Harvard Kennedy SchoolMartha MinowOther Notable FiguresProfessor, Harvard Law SchoolGabriella BlumOther Notable FiguresProfessor of Human Rights and Humanitarian Law, Harvard LawKevin EsveltOther Notable FiguresAssociate Professor of Biology, MITEdward WittensteinOther Notable FiguresExecutive Director, International Security Studies, Yale Jackson School of Global Affairs, Yale UniversitySonny RamaswamyOther Notable FiguresPresident, Northwest Commission on Colleges & UniversitiesLaurie ZolothOther Notable FiguresMargaret E. Burton Professor of Religion and Ethics , University of ChicagoKarina VoldOther Notable FiguresAssistant Professor, University of TorontoVictor VeitchAI ScientistsAssistant Professor of Data Science and Statistics, University of ChicagoDylan Hadfield-MenellAI ScientistsAssistant Professor of Computer Science, MITSamuel R. BowmanAI ScientistsAssociate Professor of Computer Science, NYU and AnthropicMengye RenAI ScientistsAssistant Professor of Computer Science, New York UniversityShiri Dori-HacohenAI ScientistsAssistant Professor of Computer Science, University of ConnecticutMiles BrundageOther Notable FiguresHead of Policy Research, OpenAIAllan DafoeOther Notable FiguresAGI Strategy and Governance Team Lead, Google DeepMindHelen KingOther Notable FiguresSenior Director of Responsibility & Strategic Advisor to Research, Google DeepMindJade LeungOther Notable FiguresGovernance Lead, OpenAIJess WhittlestoneOther Notable FiguresHead of AI Policy, Centre for Long-Term ResilienceSarah KrepsOther Notable FiguresJohn L. Wetherill Professor and Director of the Tech Policy Institute, Cornell UniversityJared KaplanAI ScientistsCo-Founder, AnthropicChris OlahAI ScientistsCo-Founder, AnthropicAndrew RevkinOther Notable FiguresDirector, Initiative on Communication & Sustainability, Columbia University - Climate SchoolCarl RobichaudOther Notable FiguresProgram Officer (Nuclear Weapons), Longview PhilanthropyLeonid ChindelevitchOther Notable FiguresLecturer in Infectious Disease Epidemiology, Imperial College LondonNicholas DirksOther Notable FiguresPresident, The New York Academy of SciencesHongyi ZhangAI ScientistsResearch Scientist, ByteDanceMarc WarnerOther Notable FiguresCEO, FacultyRob PikeOther Notable FiguresDistinguished Engineer (retired), Co-Creator of Golang, GoogleClare LyleAI ScientistsResearch Scientist, Google DeepMindNisarg ShahAI ScientistsAssistant Professor, University of TorontoRyota KanaiAI ScientistsCEO, Araya, Inc.Tim G. J. RudnerAI ScientistsAssistant Professor and Faculty Fellow, New York UniversityNoah FiedelAI ScientistsDirector, Research & Engineering, Google DeepMindJakob FoersterAI ScientistsAssociate Professor of Engineering Science, University of OxfordMichael OsborneAI ScientistsProfessor of Machine Learning, University of OxfordMarina JirotkaOther Notable FiguresProfessor of Human Centred Computing, University of OxfordNancy ChangAI ScientistsResearch Scientist, GoogleTom SchaulAI ScientistsResearch Scientist, Google DeepMindRoger GrosseAI ScientistsAssociate Professor of Computer Science, University of Toronto and AnthropicDavid DuvenaudAI ScientistsAssociate Professor of Computer Science, University of TorontoDaniel M. RoyAI ScientistsAssociate Professor and Canada CIFAR AI Chair, University of Toronto; Vector InstituteKanjun QiuAI ScientistsCEO, Generally IntelligentChris J. MaddisonAI ScientistsAssistant Professor of Computer Science, University of TorontoTegan MaharajAI ScientistsAssistant Professor of the Faculty of Information, University of TorontoFlorian ShkurtiAI ScientistsAssistant Professor of Computer Science, University of TorontoJeff CluneAI ScientistsAssociate Professor of Computer Science and Canada CIFAR AI Chair, The University of British Columbia and the Vector InstituteEva VivaltOther Notable FiguresAssistant Professor of Economics, University of Toronto, and Director, Global Priorities Institute, University of OxfordJacob TsimermanOther Notable FiguresProfessor of Mathematics, University of TorontoEmanuel AdlerOther Notable FiguresProfessor Emeritus, University of TorontoDanit GalOther Notable FiguresTechnology Advisor at the UN; Associate Fellow, Leverhulme Centre for the Future of Intelligence, University of CambridgeJean-Claude LatombeAI ScientistsProfessor (Emeritus) of Computer Science, Stanford UniversityScott NiekumAI ScientistsAssociate Professor of Computer Science, University of Massachusetts AmherstLionel LevineAI ScientistsAssociate Professor of Mathematics, Cornell UniversityThryn ShapiraAI ScientistsAI Ethics Lead, Google Photos Josh WolfeOther Notable FiguresCo-Founder & Managing Director, Lux CapitalNorman SadehOther Notable FiguresProfessor of Computer Science/Co-Director Privacy Engineering Program, Carnegie Mellon UniversityBrian ZiebartAI ScientistsAssociate Professor of Computer Science, University of Illinois ChicagoAza RaskinOther Notable FiguresCofounder, Center for Humane Technology, The Earth Species ProjectPrasad TadepalliAI ScientistsProfessor of Computer Science, Oregon State UniversityDavid L RoscoeOther Notable FiguresBoard Chair Emeritus & Advisory Council Chair, The Hastings CenterTristan HarrisOther Notable FiguresExecutive Director, Center for Humane TechnologyAnthony AguirreOther Notable FiguresExecutive Director, Future of Life InstituteSam HarrisOther Notable FiguresAuthor, neuroscientist, Making Sense / Waking UpGrimesOther Notable FiguresMusician / ArtistChris AndersonOther Notable FiguresDreamer-in-Chief, TEDRamy YoussefOther Notable FiguresActor/Director, Cairo CowboyRif A. SaurousAI ScientistsResearch Director, GoogleJames W. PennebakerOther Notable FiguresProfessor Emeritus of Psychology, University of Texas at AustinWill FithianOther Notable FiguresAssociate Professor of Statistics, UC BerkeleyJose Hernandez-OralloAI ScientistsProfessor of Computer Science, Technical University of ValenciaR. Martin ChavezOther Notable FiguresVice Chairman, Sixth Street Partners, Former CFO and CIO of Goldman SachsPaul S. RosenbloomAI ScientistsProfessor Emeritus of Computer Science, University of Southern CaliforniaTimothy LillicrapAI ScientistsResearch Director, Google DeepMindSamuel AlbanieAI ScientistsAssistant Professor of Engineering, University of CambridgeJascha Sohl-DicksteinAI ScientistsPrincipal Scientist, Google DeepMindRonald Craig ArkinAI ScientistsRegents' Professor Emeritus, Georgia Institute of TechnologyBeen KimAI ScientistsResearch Scientist, Google DeepMindMehran SahamiOther Notable FiguresProfessor and Chair of Computer Science, Stanford University Cihang XieAI ScientistsAssistant Professor of Computer Science and Engineering, UC Santa CruzPhilip S. ThomasAI ScientistsAssociate Professor, University of MassachusettsHilary GreavesOther Notable FiguresProfessor of Philosophy, University of OxfordPierre BaldiAI ScientistsProfessor, University of California, IrvineGiovanni VignaAI ScientistsProfessor, UC Santa BarbaraElad HazanAI ScientistsProfessor of Computer Science, Princeton University and Google DeepMindShai Shalev-ShwartzAI ScientistsProfessor, The Hebrew University of JerusalemKatherine LeeAI ScientistsResearch Scientist, Google DeepMindFelix Juefei XuAI ScientistsResearch Scientist, Meta AIFoutse KhomhAI ScientistsProfessor and Canada CIFAR AI Chair, Polytechnique MontrealDan HendrycksAI ScientistsExecutive Director, Center for AI Safety[Show Fewer](#)Vijay BolinaOther Notable FiguresChief Information Security Officer, Google DeepMindWiley SchellOther Notable FiguresAssoc. Prof. Emeritus, Duke UniversityMarc SommerOther Notable FiguresAssociate Professor of Biomedical Engineering, Duke UniversityJose del R. MillanOther Notable FiguresChair Professor, Dept. of Electrical and Computer Engineering, Neurology, Biomedical Engineering, The University of Texas at AustinMark CohenOther Notable FiguresProfessor of Psychiatry, Neurology, Radiology, Psychology, Biomedical Physics and BioEngineering, UCLAAggelos K. KatsaggelosOther Notable FiguresJoseph Cummings Professor, Northwestern UniversityRobert Hsiu-Ping ChowOther Notable FiguresProfessor, Department of Neuroscience, Biomedical Engineering, Pathology, USCKais GadhoumiOther Notable FiguresAssistant Professor, Duke UniversityMichelle LongmireOther Notable FiguresCEO, MedableAdrian AounOther Notable FiguresCEO, ForwardYangming OuOther Notable FiguresAssociate Professor, Harvard Medical SchoolMichael JacobsOther Notable FiguresProfessor Emeritus, Stanford University Medical CenterMartin EklundOther Notable FiguresProfessor, Department of Medical Epidemiology and Biostatistics, Karolinska InstitutetRan XiaoOther Notable FiguresAssistant Professor, Emory UniversityDominique MonlezunAI ScientistsChief Data Scientist + Professor (Adj Ast) of Cardiology, G-SAS + UT MD Anderson Cancer CenterSara KhalidAI ScientistsProfessor, University of OxfordNathaniel OsgoodOther Notable FiguresProfessor, Director Computational Epidemiology and Public Health Informatics Laboratory, University of SaskatchewanDiane HenshelOther Notable FiguresAssociate Professor of Public and Environmental Affairs, Indiana UniversityMohamed ElmasryAI ScientistsEmeritus Professor of Computer Engineering, University of Waterloo, VLSI Research Group Jay YagnikAI ScientistsVP AI, Research, Google-AlphabetSinan AralOther Notable FiguresDavid Austin Professor of Management, MIT; Director of the MIT Initiative on the Digital EconomyYannis TsividisProfessor of Electrical Engineering, Columbia UniversityJames HendlerAI ScientistsProfessor of Computer Science, Rensselaer Polytechnic InstituteJason BurnettOther Notable FiguresChair of the Board, The David and Lucile Packard FoundationDavid OrrOther Notable FiguresTrustee, The David and Lucile Packard FoundationNeil Buddy ShahOther Notable FiguresCEO, Clinton Health Access Initiative (CHAI)Holden KarnofskyOther Notable FiguresCo-CEO, Open PhilanthropyNatalie CargillOther Notable FiguresFounder & Co-CEO, Longview PhilanthropyAlejandro Guerrero-RuizOther Notable FiguresHead of Development Results and Impact, OECDLiv BoereeOther Notable FiguresCo-Founder, Raising For Effective GivingSimran DhaliwalOther Notable FiguresCo-CEO, Longview PhilanthropyJoshua GreeneOther Notable FiguresProfessor, Harvard UniversityAbraham (Avi) LoebOther Notable FiguresFrank B. Baird Jr Professor of Science, Harvard UniversityJuliana SchroederOther Notable FiguresProfessor of Management and Organizations, UC BerkeleyCraig FallsOther Notable FiguresHead of Quantitative Research, Jane Street CapitalNoam BrownAI ScientistsResearch Scientist, Google DeepMindTae Wan KimOther Notable FiguresAssociate Professor of Business Ethics, Carnegie Mellon University, Tepper School of Business, President-elect of Society for Business EthicsAnat LechnerOther Notable FiguresProfessor, NYUJames BradburyAI ScientistsSoftware Engineer, Google DeepMindMargaret LeviOther Notable FiguresProfessor of Political Science and Senior Fellow, CDDRL, Stanford UniversityBrian TseCEO, Concordia (安远AI)Robert TragerOther Notable FiguresProfessor of Political Science, University of California, Los AngelesSteve OmohundroAI ScientistsResearch Scientist, Beneficial AI ResearchMatthew BotvinickAI ScientistsSenior Technology and Policy Advisor, Google DeepMindJason HickeyAI ScientistsHead of Google Research Accra, Google LLCAngie FrenchAI ScientistsDirector, Marketing Science, MetaGregory WayneAI ScientistsDirector in Research, Google DeepMindEthan DyerAI ScientistsResearch Scientist, Google DeepMindJennifer F. WaldernAI ScientistsData Scientist and Researcher, MicrosoftAndrew CritchAI ScientistsCEO & AI Research Scientist, Encultured AI & UC BerkeleyRichard TongOther Notable FiguresChair, IEEE Artificial Intelligence Standards Committee, IEEE and Carnegie Learning Nell WatsonAI ScientistsAI Ethics Maestro, IEEE Standards AssociationSteven TanimotoOther Notable FiguresProfessor, Paul G. Allen School of Computer Science and Engineering, University of Washington, SeattleEric MjolsnessAI ScientistsProfessor of Computer Science, University of California, IrvineSimon ThorpeAI ScientistsCNRS Research Director, CNRS, University of Toulouse, FranceMichael DyerAI ScientistsProfessor, Emeritus, Computer Science UCLAKimin LeeAI ScientistsResearch Scientist, Google ResearchDavid MartinezOther Notable FiguresMIT Lead Instructor and Laboratory Fellow, MITJames PhillipsOther Notable FiguresFormer Special Adviser to the UK Prime Minister for Science and TechnologySimon KornblithAI ScientistsResearch Scientist, Google DeepMindSeán Ó hÉigeartaighOther Notable FiguresDirector, AI: Futures and Responsibility Programme, University of CambridgeAlexander MorganOther Notable FiguresJournalist, BloombergDavid KushnerOther Notable FiguresAuthor/Journalist, Rolling StoneBrent GoffOther Notable FiguresChief News Anchor, DW News (Deutsche Welle)Claire BerlinskiOther Notable FiguresEditor in Chief, The Cosmopolitan GlobalistMiles PalmerOther Notable FiguresFounder, 8 Rivers CapitalPeter ProveOther Notable FiguresDirector, Commission of the Churches on International Affairs, World Council of ChurchesPriyamvada NatarajanOther Notable FiguresJoseph S. and Sophia S. Fruton Professor of Astronomy & Professor of Physics, Yale UniversityJonathan RichensAI ScientistsResearch Scientist, Google DeepMindGregor Joeris AI ScientistsCTO, SERgroupIan ShipseyOther Notable FiguresHenry Moseley Professor of Physics, Oxford UniversitySteven KivelsonOther Notable FiguresProfessor of Physics, Stanford UniversityShamit KachruOther Notable FiguresProfessor of Physics, Stanford UniversityAnil MadhavapeddyAI ScientistsProfessor of Planetary Computing, University of CambridgeChristopher F. McKeeOther Notable FiguresProfessor Emeritus of Physics and of Astronomy, UC BerkeleyShivaji SondhiOther Notable FiguresWykeham Professor of Physics, University of OxfordJ. Mark CockOther Notable FiguresResearch Director, CNRSJan Hendrik KirchnerAI ScientistsMember of Technical Staff, OpenAITommaso TreuOther Notable FiguresProfessor of Physics and Astronomy, UCLAAlexander MaloneyOther Notable FiguresProfessor of Physics, McGill UniversityJames CrutchfieldOther Notable FiguresDistinguished Professor of Physics, University of California, DavisWilliam J RippleOther Notable FiguresUniversity Distinguished Professor, Oregon State UniversityFernán JaramilloOther Notable FiguresProfessor of Biology, Carleton CollegeDaniel GardnerOther Notable FiguresProfessor of Physiology & Biophysics; Head, Laboratory of Neuroinformatics, Weill Cornell Medical CollegeAdam Seth LitwinOther Notable FiguresAssociate Professor of Industrial and Labor Relations, Cornell UniversityJason WeiAI ScientistsMember of Technical Staff, OpenAIEthan PerezAI ScientistsTeam Lead, AnthropicBojan ArbutinaOther Notable FiguresAssociate Professor of Astrophysics, University of BelgradeEric D. FeigelsonOther Notable FiguresProfessor of Astronomy & Astrophysics and of Statistics, Penn State UniversityMichal PechoucekAI ScientistsProfesor of AI / CTO, Czech Technical University in Prague / Gen DigitalTom EverittAI ScientistsStaff Research Scientist, Google DeepMindChristoph WenigerOther Notable FiguresAssociate Professor of Theoretical Physics, University of AmsterdamJan LeikeAI ScientistsAlignment Team Lead, OpenAIRohin ShahAI ScientistsAlignment Team Lead, Google DeepMindMark ChenAI ScientistsHead of Frontiers Research, OpenAIGeoffrey IrvingAI ScientistsScalable Alignment Team Lead, Google DeepMindGrant LarsenOther Notable FiguresAssistant Professor of Physics, Principia CollegeJeff GreensiteOther Notable FiguresProfessor of Physics, San Francisco State UniversitySubodh PatilOther Notable FiguresAssistant Professor of Physics, Leiden UniversityJared RiceOther Notable FiguresAssistant Professor of Physics and Astronomy, Southwestern Adventist UniversityAdrian TangAI ScientistsStrategic Researcher, Jet Propulsion LaboratoryAndrew BriggsOther Notable FiguresProfessor of Nanomaterials, University of OxfordAndre VellinoAI ScientistsAssistant Professor, University of OttawaYossi MatiasAI ScientistsVP Engineering & Research, GoogleJason AbaluckOther Notable FiguresProfessor of Economics, Yale UniversityAnton KorinekOther Notable FiguresProfessor of Economics, University of Virginia, Rubenstein Fellow, Brookings InstitutionKetan RamakrishnanOther Notable FiguresAssociate Professor of Law, Yale UniversityHenning Grosse Ruse-KhanOther Notable FiguresProfessor of Law, University of CambridgeEdoardo GalloOther Notable FiguresAssociate Professor, Faculty of Economics, University of CambridgeJohn ArmourOther Notable FiguresProfessor of Law and Finance, University of OxfordMitchell BermanOther Notable FiguresProfessor of Law and Philosophy, University of PennsylvaniaMichiel BakkerAI ScientistsSenior Research Scientist, Google DeepMindIan HogarthOther Notable FiguresCo-author, State of AISenthooran RajamanoharanAI ScientistsResearch Engineer, Google DeepMindStanislav FortAI ScientistsLarge Language Model Lead, Stability AITulio de OliveiraOther Notable FiguresProfessor in Data Science & Bioinformatics, School for Data Sciences and Computational Thinking, Stellenbosch UniversityConnor LeahyOther Notable FiguresCEO, ConjectureChris ScammellOther Notable FiguresCOO, ConjectureAndrea MiottiOther Notable FiguresHead of Strategy, ConjectureSimon OsinderoAI ScientistsPrincipal Scientist, Google DeepMindCurtis HuebnerOther Notable FiguresHead of Alignment Research, EleutherAIDavid WallaceOther Notable FiguresProfessor, Mellon Professor of Philosophy of Science, University of PittsburghHuw PriceOther Notable FiguresDistinguished Emeritus Professor, University of BonnSven BehnkeAI ScientistsProfessor for Autonomous Intelligent Systems, University of Bonn, GermanyAlekh AgarwalAI ScientistsStaff Research Scientist, Google ResearchYonglong TianAI ScientistsSenior Research Scientist, Google ResearchYannis AssaelAI ScientistsStaff Research Scientist, Google DeepMindWilliam MacAskillOther Notable FiguresAssociate Professor, University of OxfordToby OrdOther Notable FiguresSenior Research Fellow, University of OxfordDiane MichelfelderOther Notable FiguresProfessor of Philosophy, Macalester CollegeShawn EmamjomehOther Notable FiguresFounder, PolyOneJack KokkoOther Notable FiguresCEO & Founder, AlphaSenseAlec NielsenOther Notable FiguresCEO, Asimov, Inc.Stefan HeinemannOther Notable FiguresProfessor for Ethics, FOM University of Applied SciencesHugh GustersonOther Notable FiguresProfessor of Anthropology & Public Policy, University of British ColumbiaAlakh PandeyOther Notable FiguresFounder and CEO, Physics WallahJeff SeboOther Notable FiguresAssociate Professor of Environmental Studies, Affiliated Professor of Bioethics, Medical Ethics, Philosophy, and Law, and Director of the Mind, Ethics, and Policy Program, New York UniversityBen LevinsteinOther Notable FiguresAssistant Professor of Philosophy, University of Illinois at Urbana-ChampaignPeter KuznickOther Notable FiguresDirector, Nuclear Studies Institute, American UniversityBryan DruzinOther Notable FiguresAssociate Professor of Law, The Chinese University of Hong KongSimon GoldsteinOther Notable FiguresAssociate Professor, Australian Catholic UniversityMark BarenbergOther Notable FiguresSulzbacher Professor of Law, Columbia UniversityWeijie SuAI ScientistsAssociate Professor, University of PennsylvaniaAtilim Gunes BaydinAI ScientistsLecturer in Machine Learning, University of OxfordN. G. LaskowskiOther Notable FiguresAssistant Professor of Philosophy, University of Maryland - College ParkNathaniel SharadinOther Notable FiguresAssistant Professor of Philosophy, University of Hong KongCameron Domenico Kirk-GianniniOther Notable FiguresAssistant Professor of Philosophy, Rutgers UniversitySam McCandlishAI ScientistsCo-Founder, AnthropicRosie CampbellAI ScientistsTechnical Program Manager, OpenAITom B BrownAI ScientistsCo-Founder, AnthropicEla MadejOther Notable FiguresFounding Partner, 50YAnant BhardwajOther Notable FiguresFounder & CEO, InstabaseNicholas JosephAI ScientistsPretraining Team Lead, AnthropicNova DasSarmaAI ScientistsSystems Team Lead, AnthropicShauna KravecAI ScientistsMember of Technical Staff, AnthropicZac Hatfield-DoddsAI ScientistsTeam Lead, AnthropicHugh W. HillhouseOther Notable FiguresProfessor, University of WashingtonCarol ChenAI ScientistsMember of Technical Staff, AnthropicAndy ClarkOther Notable FiguresProfessor of Cognitive Philosophy, University of SussexRyan LoweAI ScientistsMember of Technical Staff, OpenAIAhti-Veikko PietarinenOther Notable FiguresProfessor of Philosophy, Ethical and Theoretical AI Lab, Hong Kong Baptist UniversityRamana KumarAI ScientistsSenior Research Scientist, Google DeepMindPeter ChristenOther Notable FiguresEmeritus Professor of Computing and Cybernetics, Australian National UniversityPeter GriggsOther Notable FiguresCo-Founder, DeepAIYuri BurdaAI ScientistsMember of Technical Staff, OpenAIJeffrey LadishOther Notable FiguresExecutive Director, Palisade ResearchCarroll WainwrightAI ScientistsMember of Technical Staff, OpenAIAntonio DieguezOther Notable FiguresFull Professor of Logic and Philosophy of Science, Universidad de MalagaDaniel FreemanAI ScientistsSenior Software Engineer, Google DeepmindSimon FriederichOther Notable FiguresAssociate Professor of Philosophy of Science, University of GroningenGamaleldin ElsayedAI ScientistsResearch Scientist, Google DeepMindRory GreigAI ScientistsResearch Engineer, Google DeepMindTom LieberumAI ScientistsResearch Engineer, Google DeepMindVlad FiroiuAI ScientistsResearch Engineer, Google DeepMindÁlvaro De RújulaOther Notable FiguresResearcher, CERNFabio A. González AI ScientistsProfessor, Universidad Nacional de ColombiaAlex OlshevskyAI ScientistsAssociate Professor, Boston UniversityTijl De BieAI ScientistsProfessor, Ghent UniversityJonathan SimonOther Notable FiguresAssistant Professor of Philosophy, University of MontrealCesar de la FuenteAI ScientistsPresidential Assistant Professor, University of PennsylvaniaHengjin CaiOther Notable FiguresProfessor of Computer Science, Wuhan University; Deputy Director of the Mental Computation Committee of the Chinese Association for Artificial Intelligence (CAAI)Steve PetersenOther Notable FiguresAssociate Professor of Philosophy, Niagara UniversityTomas PfisterOther Notable FiguresHead of AI Research, Google CloudHarrison EdwardsAI ScientistsMember of Technical Staff, OpenAIYounghyun ChungOther Notable FiguresCEO, CorcaBill Simpson-YoungOther Notable FiguresChief Executive, Gradient InstituteRafal RzepkaAI ScientistsAssistant Professor, Hokkaido UniversityEd Leon KlingerOther Notable FiguresCEO, Flock, CSERTorsten ReilOther Notable FiguresCo-CEO, HelsingManuel B. GraeberAI ScientistsProfessor, Brain and Mind Center, University of SydneySuheil ShahryarOther Notable FiguresChair, United Nations Association UKGeorge MillerOther Notable FiguresAssistant Professor of Philosophy (retired), University of Maine at FarmingtonCollin BurnsAI ScientistsMember of Technical Staff, OpenAINorman K SwazoOther Notable FiguresProfessor of Philosophy; Director, Office of Research, North South UniversityFranck VarenneOther Notable FiguresAssociate Professor of Philosophy, University of Rouen - FranceEden LinOther Notable FiguresAssociate Professor of Philosophy, The Ohio State UniversityHector GeffnerAI ScientistsAlexander von Humboldt Professor, RTWH Aachen UniversityJonathan GrizouAI ScientistsAssistant Professor of Computer Science, University of Glasgow Sebastian MusslickOther Notable FiguresAssistant Professor and Schmidt Science Fellow, Brown UniversityChaowei XiaoAI ScientistsAssistant Professor, Arizona State UniversityNathanael FastOther Notable FiguresAssociate Professor, University of Southern CaliforniaWitold KinsnerAI ScientistsProfessor, University of ManitobaMartin w. BauerOther Notable FiguresProfessor of Social Psychology and Research Methodology, London School of Economics and Political ScienceMichal KosinskiOther Notable FiguresAssociate Professor of Psychology, Stanford UniversityBruce K. BrittonOther Notable FiguresProfessor, University of GeorgiaHongwei QinAI ScientistsResearch Director, SenseTimeRonald FreytagOther Notable FiguresProfessor of Psychology, Berlin, GermanyKarim BeguirAI ScientistsCo-Founder & CEO, InstaDeepSanjiv RaiOther Notable FiguresFounder, Genic AI Francesca PeressottiOther Notable FiguresProfessor of Psychology, Università degli Studi di Padova, ItalyCarlos Gómez-RodríguezAI ScientistsProfessor of Computer Science and AI, University of A CoruñaWaqas RafiqueAI ScientistsAssistant Professor in Artificial Intelligence, University College LondonDr. Syed Shameem EjazOther Notable FiguresAssistant Professor /Psychology, Institute of Business Management [IoBM]Michael R. ScheesseleAI ScientistsProfessor of Computer Science and Psychology, Indiana University South BendHelena MatuteOther Notable FiguresProfessor of Psychology, Deusto University, SpainZachary KentonAI ScientistsSenior Research Scientist, Google DeepmindDean MobbsOther Notable FiguresProfessor of Cognitive Neuroscience, California Institute of TechnologyVikrant VarmaAI ScientistsResearch Engineer, Google DeepMindDaniel M. GoldenholzOther Notable FiguresAssistant Professor of Neurology, HarvardFelipe De BrigardAI ScientistsFuchsberg-Levine Family Associate Professor, Duke University Vladimir MikulikAI ScientistsResearch Engineer, Google DeepMindSholto DouglasAI ScientistsResearcher, Google DeepMindMiteyan PatelAI ScientistsSoftware Engineer, DeepMindDaniel KokotajloOther Notable FiguresMember of Policy Team Staff, OpenAITolga BirdalAI ScientistsAssistant Professor, Imperial College LondonClayton LewisOther Notable FiguresEmeritus Professor of Computer Science, University of Colorado BoulderSebastian FarquharAI ScientistsResearch Scientist, Google DeepMindKlaus GreffAI ScientistsResearch Scientist, Google Deep MindTris WarkentinAI ScientistsDirector of Product Management, Google DeepMindRicardo PradaAI ScientistsDirector of AI User Experience, Google ResearchTanja Messerli LudiOther Notable FiguresDirector, Swiss Publishers and Booksellers AssociationHoshin GuptaOther Notable FiguresRegents Professor, The University of ArizonaElliot NelsonAI ScientistsResearch Scientist, IBM ResearchCheranellore (Vasu) VasudevanAI ScientistsSenior Technical Staff Member, IBMOlle HäggströmOther Notable FiguresProfessor of Mathematical Statistics, Chalmers University of TechnologyDeLesley HutchinsAI ScientistsSenior Software Engineer, GoogleNia J.C. CastellyOther Notable FiguresCo-Founder & Legal Lead of Checks, GoogleFergus HurleyOther Notable FiguresCo-Founder & GM of Checks, GoogleStefan SintOther Notable FiguresAssociate Professor, School of Mathematics, Trinity College DublinJoachim WeickertAI ScientistsProfessor of Mathematics and Computer Science, Saarland University, Germany Dennis HejhalOther Notable FiguresProfessor of Mathematics, University of MinnesotaHolden LeeOther Notable FiguresAssistant Professor, Johns Hopkins UniversityMatthew YedlinOther Notable FiguresAssociate Professor of Electrical and Computer Engineering, University of British ColumbiaTodor MarkovAI ScientistsMember of Technical Staff, OpenAIIrwan BelloAI ScientistsMember of Technical Staff, OpenAIJohn Jeffries MartinOther Notable FiguresProfessor of History, Duke UniversityJames D. MillerOther Notable FiguresProfessor of Economics, Smith CollegeMartin BouchardOther Notable FiguresCEO, QScaleAndre LavoieOther Notable FiguresCo-founder, Chairman and CEO, ClearCompanyMark GrevilleOther Notable FiguresVP Architecture, WorkhumanRitwik BhattacharyaAI ScientistsSenior Vice President, Engineering, HeadspaceNicole CarignanOther Notable FiguresVP of Strategic Cyber AI, DarktraceDaniel CuthbertOther Notable FiguresGlobal Head of Cyber Security Research, Banco SantanderDavid DohanAI ScientistsMember of Technical Staff, OpenAIShyamal AnadkatOther Notable FiguresMember of Go To Market Staff, OpenAISrinivas NarayananOther Notable FiguresVP, Engineering, OpenAIMaxwell L BileschiAI ScientistsStaff Research Software Engineer & Manager, Google DeepmindLaurent SartranAI ScientistsStaff Research Engineer, Google DeepMindJohn AslanidesAI ScientistsStaff Research Engineer, Google DeepMindMax JaderbergAI ScientistsDirector of Machine Learning, Isomorphic LabsGrigory KhimulyaOther Notable FiguresCo-CEO, AlveaGuruduth BanavarAI ScientistsFounding CTO, Viome Life SciencesTimothy John O'DonnellAI ScientistsAssociate Professor, Mcgill University/MilaEugenio Vargas GarciaOther Notable FiguresTech Diplomat, Brazilian Consulate General in San FranciscoSam ClarkeOther Notable FiguresStrategy Manager, Centre for the Governance of AINico MiailheOther Notable FiguresFounder and President, The Future Society (TFS)Nicolas MoësOther Notable FiguresDirector, European AI Governance, The Future SocietyYolanda LannquistOther Notable FiguresDirector, AI Governance, The Future SocietyMatthew ShribmanOther Notable FiguresCEO & Chief Scientist, AimHi EarthAngus MercerOther Notable FiguresChief Executive, Centre for Long-Term ResilienceAndrew Snyder-BeattieOther Notable FiguresSenior Program Officer, Biosecurity & Pandemic Prepardness, Open PhilanthropyLuke MuehlhauserOther Notable FiguresSenior Program Officer, AI Governance and Policy, Open PhilanthropyJoseph CarlsmithOther Notable FiguresSenior Research Analyst, Open PhilanthropyRobert KowalskiAI ScientistsEmeritus Professor, Imperial College LondonDov DoriAI ScientistsProfessor Emeritus, Technion, Israel Institute of TechnologyAndré PlatzerAI ScientistsAlexander von Humboldt Professor for AI, Karlsruhe Institute of TechnologyRinat Rosenberg-KimaAI ScientistsAssistant Professor, Technion - Israel Institute of TechnologyNelma MoreiraOther Notable FiguresAssociate Professor Computer Science, University of PortoSunyoung YangOther Notable FiguresAssistant Professor, The University of ArizonaStephen GrossbergAI ScientistsWang Professor of Cognitive and Neural Systems, Boston UniversityChristoph LampertAI ScientistsProfessor of Machine Learning and Computer Vision, Institute of Science and Technology Austria (ISTA)Stefan TaiOther Notable FiguresProfessor of Computer Science, TU BerlinAnders LansnerAI ScientistsProfessor Emeritus, KTH Royal Institute of Technology and Stockholm UniversityMichael FindleyOther Notable FiguresErwin Centennial Professor of Government, University of Texas at AustinCosimo Della SantinaOther Notable FiguresProfessor, TU DelftAbhinav ValadaAI ScientistsProfessor, University of FreiburgErnesto CostaAI ScientistsFull Professor, University of CoimbraIlija BogunovicAI ScientistsAssistant Professor, University College LondonRaymund SisonAI ScientistsProfessor and University Fellow, De La Salle UniversityAlexander FlorOther Notable FiguresProfessor of Information and Communication Studies, University of the PhilippinesMirjana Pejić BachAI ScientistsFull professor, University of Zagreb, Faculty of Economics and BusinessDavid MacphersonOther Notable FiguresProfessor of Medicine, retired, University of PittsburghIoannis PatrasAI ScientistsProfessor of Computer Science, Queen Mary University of LondonThe Anh HanAI ScientistsProfessor of Computer Science, Lead of Centre for Digital Innovation, Teesside UniversityGregory ProvanAI ScientistsProfessor, Computer Science, UCC, IrelandWilliam SaundersAI ScientistsMember of Technical Staff, OpenAIRoberto BattistonOther Notable FiguresProfessor, University of Trento, ItalyOlavo AmaralOther Notable FiguresAssociate Professor, Federal University of Rio de JaneiroLuca FredianiOther Notable FiguresProfessor, Hylleraas Centre, UiT The Arctic University of NorwayVedant MisraAI ScientistsResearch Engineer, Google DeepMindPaul ChristianoAI ScientistsDirector, Alignment Research CenterAjeya CotraOther Notable FiguresSenior Research Analyst, Open PhilanthropyOwain EvansAI ScientistsAI Safety Researcher, Truthful AISawyer BernathOther Notable FiguresExecutive Director, Berkeley Existential Risk InitiativeLewis HammondOther Notable FiguresActing Executive Director, Cooperative AI FoundationEliezer YudkowskyOther Notable FiguresSenior Research Fellow and Co-Founder, Machine Intelligence Research InstituteNate SoaresOther Notable FiguresExecutive Director, Machine Intelligence Research InstituteRoman YampolskiyAI ScientistsAssociate Professor of Computer Science, University of LouisvilleKatja GraceOther Notable FiguresLead Researcher, AI ImpactsAndreas StuhlmüllerAI ScientistsCEO, OughtEvan HubingerAI ScientistsResearch Scientist, AnthropicNick FitzOther Notable FiguresFounder & CEO, MomentumLeo GaoAI ScientistsMember of Technical Staff, OpenAITim BlackmoreOther Notable FiguresProfessor, Media Studies, Western UniversityNeel NandaAI ScientistsResearch Engineer, DeepMindMark NitzbergAI ScientistsExecutive Director, Center for Human-Compatible AISatoshi KuriharaAI ScientistsProfessor, Keio UniversityVinay RamaseshAI ScientistsResearch Scientist, Google DeepMindRobert DuinAI ScientistsProfessor, Delft University of TechnologyJordan CrandallOther Notable FiguresProfessor, UC San DiegoJoshua LewisOther Notable FiguresAssistant Professor, New York UniversityHema A MurthyAI ScientistsRetired Professor, Indian Institute of Technology Madras IndiaRyan SultanOther Notable FiguresAssistant Professor of Human Behavior and Clinical Psychiatry, Columbia UniversityErdem BıyıkAI ScientistsAssistant Professor of Computer Science, University of Southern CaliforniaLisa McLaughlinOther Notable FiguresAssociate Professor, Miami UniversityLuca SimonciniOther Notable FiguresProfessor of Computer Engineering (ret.), University of Pisa, Pisa, ItalyDomenico TaliaAI ScientistsProfessor of Computer Engineering, University of CalabriaRobin JacobsonOther Notable FiguresProfessor of Politics and Government, University of Puget SoundVít JanišOther Notable FiguresAssistant Professor, Mendel University in BrnoBenjamin RosmanAI ScientistsProfessor of Computer Science, University of the WitwatersrandFrank GaffikinOther Notable FiguresProfessor Emeritus, Queen's University, BelfastYves MoreauAI ScientistsProf., University of LeuvenAlessandro SaffiottiAI ScientistsProfessor of Computer Science, Orebro University, SwedenNoah Y. SiegelAI ScientistsSenior Research Engineer, DeepMindNick CouldryOther Notable FiguresProfessor of Media, Communications and Social TheoryAnsgar StelandAI ScientistsProfessor of Stochastics, Institute of Statistics and AI Center, RWTH AachenKazuhiko ShibuyaAI ScientistsProfessor, Globiz Professional University, JapanYoshihiko NakamuraAI ScientistsProfessor Emeritus, University of TokyoVernon LouwOther Notable FiguresProfessor, Chair and Head Clinical Haematology, University of Cape TownJaak TepandiAI ScientistsProfessor Emeritus of Knowledge-Based Systems, Tallinn University of Technology, Tallinn University of TechnologyMartin CederwallOther Notable FiguresProf., Chalmers Univ. of TechnologyKoichi TakahashiOther Notable FiguresDistinguished Visiting Professor, Keio UniversityHiroshi YamakawaAI ScientistsChairperson, The Whole Brain Architecture InitiativeBrian ChristianOther Notable FiguresAuthor, The Alignment Problem, UC Berkeley & University of OxfordVan Hoai TranAI ScientistsAssoc. Professor, Ho Chi Minh City University of TechnologyHugh CameronOther Notable FiguresVisiting Professor, Makerere UniversityMounîm A. El-YacoubiAI ScientistsProfessor, Institut Polytechnique de OarisKarl TuylsAI ScientistsResearch Director and Professor of Computer Science, Google DeepMind and University of Liverpool/LeuvenAlexandre Xavier FalcãoAI ScientistsFull Professor, Universidade Estadual de Campinas (UNICAMP)Thomas KeckAI ScientistsResearch Engineer, Google DeepMindMartin HilbertOther Notable FiguresProfessor, Chair Computational Social Science, University of California, DavisLamberto CoccioliOther Notable FiguresProfessor of Music and Technology, Birmingham City UniversityTorbjörn NordlingAI ScientistsAssistant Professor, National Cheng Kung UniversityCorrado MencarAI ScientistsAssociate Professor, University of Bari Aldo Moro, ItalyJendrik SeippAI ScientistsAssistant Professor, Linköping UniversityJoakim WernbergOther Notable FiguresAssociate Professor, Department of Technology and Society, Lund UniversityTomas JuchaOther Notable FiguresSenior Policy Manager, AmCham SlovakiaErol GelenbeAI ScientistsProfessor, Institute of Theoretical and Applied Informatics, Polish Academy of SciencesLukasz SwiecickiOther Notable FiguresAssistant Professor, University of SiedlcePeter CowlingAI ScientistsProfessor of AI, Queen Mary University of LondonDaniel NovotnýOther Notable FiguresAssistant Professor of Philosophy, University of South BohemiaJill Rudkowski, MDOther Notable FiguresAssociate Professor of Medicine, McMaster UniversityMonika PorwołOther Notable FiguresUniversity Professor of Linguistics and Translation Studies, PhD, University of Applied Sciences in Racibórz (Poland), Institute of Modern Language StudiesHugo De ManOther Notable FiguresEmeritus Professor, KU Leuven/imecAmanda du PreezOther Notable FiguresProfessor, School of the Arts, University of PretoriaMamuka MatsaberidzeOther Notable FiguresProfessor of Chemistry, Georgian Technical UniversityTerrence MernaghOther Notable FiguresHon. Associate Professor, The Australian National UniversityZelalem JembreAI ScientistsAssistant Professor, Keimyung UniversityBehnam NeyshaburAI ScientistsResearch Scientist, Google DeepMindJuan Carlos MiguelOther Notable FiguresProfessor, University of the Basque CountryAndrey V.GavrilovAI ScientistsAssociate Professor (Docent), Novosibirsk State Technical UniversityMike GozzoOther Notable FiguresChief Product & Technology Officer, AdaTrevor PonmanOther Notable FiguresProfessor, University of BirminghamBarak A. PearlmutterAI ScientistsProfessor, Maynooth UniversityConstantinos KoutsojannisAI ScientistsProfessor of Medical Physics & Electrophysiology, Director of Health Physics & Computational Intelligence Lab, University of Patras GreeceTakafumi MatsumaruAI ScientistsProfessor of Bio-Robotics and Human-Mechatronics, Waseda UniversityGuillermo Cabrera-VivesOther Notable FiguresAssociate Professor, Universidad de ConcepciónGlen CebulashOther Notable FiguresProfessor of Art, Wright State UniversityMargie MendellOther Notable FiguresDistinguished Professor Emerita, Concordia UniversityAdrien EcoffetAI ScientistsMember of Technical Staff, OpenAIRamon Carbó-DorcaOther Notable FiguresEmeritus Professor, University of Girona (Catalonia) SpainHolger HoosAI ScientistsAlexander von Humboldt Professor of AI, RWTH Aachen University & University of LeidenWenlin HanOther Notable FiguresAssociate Professor, California State University, FullertonJovan IvkovićOther Notable FiguresProfessor of IT and IS, ITS - Information Technology School, BelgradeJoel Z LeiboAI ScientistsResearch Scientist, Google DeepMindLuís Moniz PereiraAI ScientistsProfessor Emeritus, Universidade Nova de Lisboa, PortugalAdam EckartOther Notable FiguresAssistant Professor of Health and Exercise Science, Kean UniversityCristian S. CaludeOther Notable FiguresProfessor, University of AucklandPeter SarlinAI ScientistsCEO/co-founder & Professor, Silo AI & Aalto UniversityAlejandro Molina-GarcíaOther Notable FiguresProfessor, Universidad MontrerJózsef BayerOther Notable FiguresProfessor Emeritus, Hungarian Academy of SciencesJoshua AchiamAI ScientistsResearch Scientist, OpenAIMark E WarrenOther Notable FiguresEmeritus Professor of Political Science, University of British ColumbiaPaula SelvesterOther Notable FiguresProfessor Emeritus, California State University, ChicoManel SanromàOther Notable FiguresProfessor of Applied Mathematics, Universitat Rovira i Virgili, CataloniaWolfgang BanzhafAI ScientistsProfessor of Computer Science, Michigan State UniversityAlexander RauscherOther Notable FiguresAssociate Professor, University of British ColumbiaThomas LaudalOther Notable FiguresAssociate Professor, University of Stavanger Business SchoolIvan Rizzo GuilhermeAI ScientistsAssociate Professor, UNESPWolfgang SlanyAI ScientistsFull Professor of Computer Science, Graz University of TechnologyJuan PavónAI ScientistsProfessor of Computer Science, Universidad Complutense MadridAlan MackworthAI ScientistsProfessor Emeritus, Computer Science, University of British ColumbiaScott GrayAI ScientistsMember of Technical Staff, OpenAISamuel PerezOther Notable FiguresEconomist, Lester B Pearson College of the Pacific UWCValerie PisanoOther Notable FiguresCEO, MilaJohn ArshemOther Notable FiguresChief Information Officer, Belmark Inc.Eric TopolOther Notable FiguresProfessor and EVP, Scripps ResearchLuka Crnkovic-FriisOther Notable FiguresHead of AI/ML, Activision Blizzard KingAlexis SmirnovOther Notable FiguresCo-founder & CTO, DialogueDavid LoweAI ScientistsProfessor Emeritus of Computer Science, University of British ColumbiaJeron PaulOther Notable FiguresCEO, SpiffTony BatesOther Notable FiguresChairman and CEO, GenesysJon TomasOther Notable FiguresConsultant and Clinical Lead for Palliative Medicine, University Hospitals Coventry and WarwickshireChris G. WillcocksAI ScientistsAssistant Professor of Computer Science, Durham UniversitySimon OsinderoAI ScientistsPrincipal Scientist, Google DeepMindVincent CorrubleAI ScientistsAssociate Professor of Computer Science, Sorbonne UniversityJoaquín ArangoOther Notable FiguresProfessor, Complutense University MadridMiguel GregorkiewitzOther Notable FiguresProfessor, University of Siena, ItalyTelma JohnsonOther Notable FiguresAssociate Professor of Communications, Federal University of Juiz de ForaCarlos José Giraldo JaramilloOther Notable FiguresProfessor and Researcher in Cognitive Science, University of Antioquia, ColombiaMoayed DaneshyariOther Notable FiguresAssistant Professor, Computer Science, California State University - East BayDavid SachsOther Notable FiguresAssistant Professor of Genetics and Genomic Sciences, Icahn School of Medicine at Mount SinaiDave LivingstonOther Notable FiguresProfessor Emeritus of Electrical and Computer Engineering, AI, Art, and MusicJohn Gray CoxOther Notable FiguresProfessor, College of the Atlantic and Clerk of Quaker Institute for the FutureEric YorkOther Notable FiguresAssistant Professor, Clarkson UniversityEmilia JavorskyOther Notable FiguresDirector Multistakeholder Engagement, Scientist, Future of Life Institute, Wyss Institute at Harvard UniversityRaphaël Van LaerOther Notable FiguresAssistant Professor, Chalmers University of TechnologyKen D OlumOther Notable FiguresResearch Professor, Tufts UniversityFrancesc GiraltOther Notable FiguresEmeritus Professor, University Rovira i VirgiliTony J PrescottAI ScientistsProfessor of Cognitive Robotics, University of SheffieldVitezslav KalaOther Notable FiguresAssociate Professor of Mathematics, Charles University, PragueAaron ParisiAI ScientistsSoftware Engineer, Google DeepMindMoshe VardiAI ScientistsProfessor of Computational Engineering, Rice UniversityGyu-Ho ShinOther Notable FiguresAssistant Professor, University of Illinois ChicagoLuis Paulo ReisAI ScientistsProfessor, University of PortoValerie M. HudsonOther Notable FiguresUniversity Distinguished Professor, Texas A&M UniversityAdriana HoyosOther Notable FiguresProfessor of Digital & AI Economics, IE UniversityMary-Anne WilliamsAI ScientistsDirector, UNSW Business AI Lab, University of New South Wales (UNSW)Padhraic SmythAI ScientistsProfessor, University of California, IrvineAleksandra PizuricaAI ScientistsProfessor, Ghent UniversityMartin BunzlOther Notable FiguresProfessor Emeritus of Philosophy, Rutgers UniversityDan BraunAI ScientistsLead Engineer, Apollo ResearchAndre Fernando De AlmeidaAI ScientistsCEO, Founder, Dom RockArek ZarowskiOther Notable FiguresPrincipal, Obvious VenturesArthur DouillardAI ScientistsResearch Scientist, Google DeepMindOtto BartenOther Notable FiguresDirector, Existential Risk ObservatoryErsin UzunOther Notable FiguresProfessor and Endowed Executive Director of Global Cybersecurity Institute, Rochester Institute of Technology[Show More](#)### Sign the statement

You will be sent a verification email. CAIS is committed to ensuring the validity of the signatories. There may be a small delay before signatures are added.

Full Name \*Work email \*Your Title \*Affiliation \*Relevant category:I am an AI scientistI am a notable figure (executive, professor, leader, etc)**Thank you! Please verify your email.
**Thank you for signing the statement. We sent a message to the email you entered to verify your identity.Oops! Something went wrong while submitting the form.


Artificial intelligence, systemic risks, and sustainability
===========================================================

Highlights
----------

* •There is a growing interest in AI-technologies for sustainability.
* •Increased uses of these technologies could create new types of systemic risks.
* •We show a global uptake of AI-technologies in farming, forestry and marine management.
* •We explore gaps in principles and governance of AI from a sustainability perspective.

Abstract
--------

Automated decision making and predictive analytics through artificial intelligence, in combination with rapid progress in technologies such as sensor technology and robotics are likely to change the way individuals, communities, governments and private actors perceive and respond to climate and ecological change. Methods based on various forms of artificial intelligence are already today being applied in a number of research fields related to climate change and environmental monitoring. Investments into applications of these technologies in agriculture, forestry and the extraction of marine resources also seem to be increasing rapidly. Despite a growing interest in, and deployment of AI-technologies in domains critical for sustainability, few have explored possible systemic risks in depth. This article offers a global overview of the progress of such technologies in sectors with high impact potential for sustainability like farming, forestry and the extraction of marine resources. We also identify possible systemic risks in these domains including a) algorithmic bias and allocative harms; b) unequal access and benefits; c) cascading failures and external disruptions, and d) trade-offs between efficiency and resilience. We explore these emerging risks, identify critical questions, and discuss the limitations of current governance mechanisms in addressing AI sustainability risks in these sectors.

Keywords
--------

Artificial intelligenceClimate changeSustainabilitySystemic risksAnthropoceneResilienceSocial-ecological systemsAutomationDigitalizationRecommended articles

Victor Galaz
Miguel A. Centeno
Peter W. Callahan
Amar Causevic
Thayer Patterson
Irina Brass
Seth Baum
Darryl Farber
Joern Fischer
David Garcia
Timon McPhearson
Daniel Jimenez
Brian King
Paul Larcey
Karen Levy


How Rogue AIs may Arise

The rise of powerful AI dialogue systems in recent months has precipitated debates about AI risks of all kinds, which hopefully will yield an acceleration of governance and regulatory frameworks. Although there is a general consensus around the need to regulate AI to protect the public from harm due to discrimination and biases as well as disinformation, there are profound disagreements among AI scientists regarding the potential for dangerous loss of control of powerful AI systems, an important cae of [existential risk from AI](https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence), that may arise when an AI system can autonomously act in the world (without humans in the loop to check that these actions are acceptable) in ways that could potentially be catastrophically harmful. Some view these risks as a distraction for the more concrete risks and harms that are already occurring or are on the horizon. Indeed, there is a lot of uncertainty and lack of clarity as to how such catastrophes could happen. In this blog post we start a set of formal definitions, hypotheses and resulting claims about AI systems which could harm humanity and then discuss the possible conditions under which such catastrophes could arise, with an eye towards helping us imagine more concretely what could happen and the global policies that might be aimed at minimizing such risks.

**Definition 1**: A potentially ***rogue AI*** is an autonomous AI system that could behave in ways that would be catastrophically harmful to a large fraction of humans, potentially endangering our societies and even our species or the biosphere.

**Executive Summary**

Although highly dangerous AI systems from which we would lose control do not currently exist, recent advances in the capabilities of generative AI such as large language models (LLMs) have raised concerns: human brains are biological machines and we have made great progress in understanding and demonstrating principles that can give rise to several aspects of human intelligence, such as learning intuitive knowledge from examples and manipulating language skillfully. Although I also believe that we could design AI systems that are [useful and safe](https://yoshuabengio.org/2023/05/07/ai-scientists-safe-and-useful-ai/), specific guidelines would have to be respected, for example limiting their agency. On the other hand the recent advances suggest that even the future where we know how to build [superintelligent AIs](https://en.wikipedia.org/wiki/Superintelligence) (smarter than humans across the board) is closer than most people expected just a year ago. Even if we knew how to build safe superintelligent AIs, it is not clear how to prevent potentially rogue AIs from also being built. The most likely cases of rogue AIs are goal-driven, i.e., AIs that act towards achieving given goals. Current LLMs have little or no agency but could be transformed into goal-driven AI systems, as shown with [Auto-GPT](https://en.wikipedia.org/wiki/Auto-GPT). Better understanding of how rogue AIs may arise could help us in preventing catastrophic outcomes, with advances both at a technical level (in the design of AI systems) and at a policy level (to minimize the chances of humans giving rise to potentially rogue AIs). For this purpose, we lay down different scenarios and hypotheses that could yield potentially rogue AIs. The simplest scenario to understand is simply that if a recipe to obtain a rogue AI is discovered and generally accessible, it is enough that one or a few genocidal humans do what it takes to build one. This is very concrete and dangerous, but the set of dangerous scenarios is enlarged by the possibility of unwittingly designing potentially rogue AIs, because of the problem of AI alignment (the mismatch between the true intentions of humans and the AI’s understanding and behavior) and the competitive pressures in our society that would favor more powerful and more autonomous AI systems. Minimizing all those risks will require much more research, both on the AI side and into the design of a global society that is safer for humanity. It may also be an opportunity for bringing about a much worse or a much better society.

**Hypothesis 1:** *Human-level intelligence is possible because brains are biological machines.*

There is a general consensus about hypothesis 1 in the scientific community. It arises from the consensus among biologists that human brains are complex machines. If we could figure out the principles that make our own intelligence possible (and we already have many clues about this), we should thus be able to build AI systems with the same level of intelligence as humans, or better. Rejecting hypothesis 1 would require either some supernatural ingredient behind our intelligence or rejecting [computational functionalism](https://en.wikipedia.org/wiki/Functionalism_(philosophy_of_mind)), the hypothesis that our intelligence and even our consciousness can be boiled down to causal relationships and computations that at some level are independent of the hardware substrate, the basic hypothesis behind computer science and its notion of [universal Turing machines](https://en.wikipedia.org/wiki/Universal_Turing_machine).

**Hypothesis 2:** *A computer with human-level learning abilities would generally surpass human intelligence because of additional technological advantages.*

If hypothesis 1 is correct, i.e., we understand principles that can give rise to human-level learning abilities, then computing technology is likely to give general cognitive superiority to AI systems in comparison with human intelligence**,** making it possible for such superintelligent AI systems to perform tasks that humans cannot perform (or not at the same level of competence or speed) for at least the following reasons:

* An AI system in one computer can potentially replicate itself on an arbitrarily large number of other computers to which it has access and, thanks to high-bandwidth communication systems and digital computing and storage, it can benefit from and aggregate the acquired experience of all its clones; this would accelerate the rate at which AI systems could become more intelligent (acquire more understanding and skills) compared with humans. Research on federated learning [1] and distributing training of deep networks [2] shows that this works (and is in fact already used to help train very large neural networks on parallel processing hardware).
* Thanks to high-capacity memory, computing and bandwidth, AI systems can already read the content of the whole internet fairly rapidly, a feat not possible for any human. This already explains some of the surprising abilities of state-of-the-art LLMs and is in part possible thanks to the decentralized computing capabilities discussed in the above point. Although the capacity of a human brain is huge, its input/output channels are bandwidth-limited compared with current computers, limiting the total amount of information that a single human can ingest.

Note that human brains also have capabilities endowed by evolution that current AI systems lack, in the form of [inductive biases](https://en.wikipedia.org/wiki/Inductive_bias) (tricks that evolution has discovered, for example in the type of neural architecture used in our brain or our neural learning mechanisms). Some ongoing AI research [3] aims precisely at designing inductive biases that human brains may exploit but are not yet exploited in state-of-the-art machine learning. Note that evolution operated under much stronger energy consumption requirements (about 12 watts for a human brain) than computers (on the order of a million watts for a 10000 GPU cluster of the kind used to train state-of-the-art LLMs) which may have limited the search space of evolution. However, that kind of power is nowadays available and a single rogue AI could potentially do a lot of damage thanks to it.

**Definition 2**: An *autonomous goal-directed intelligent entity* sets and attempts to achieve its own goals (possibly as subgoals of human-provided goals) and can act accordingly.

Note that autonomy could arise out of goals and rewards set by humans because the AI system needs to figure out how to achieve these given goals and rewards, which amounts to forming its own subgoals. If an entity’s main goal is to survive and reproduce (like our genes in the process of evolution), then they are fully autonomous and that is the most dangerous scenario. Note also that in order to maximize an entity’s chances to achieve many of its goals, the ability to understand and control its environment is a subgoal (or instrumental goal) that naturally arises and could also be dangerous for other entities.

**Claim 1**: Under hypotheses 1 and 2, an autonomous goal-directed superintelligent AI could be built.

**Argument**: We already know how to train goal-directed AI systems at some level of performance (using reinforcement learning methods). If these systems also benefit from superintelligence as per hypotheses 1 and 2 combined (using some improvements over the pre-training we already know how to perform for state-of-the-art LLMs), then Claim 1 follows. Note that it is likely that goals could be specified via natural language, similarly to LLM prompts, making it easy for almost anyone to dictate a nefarious goal to an AI system that understands language, even if that goal is imperfectly understood by the AI.

**Claim 2**: A superintelligent AI system that is autonomous and goal-directed would be a potentially rogue AI if its goals do not strictly include the well-being of humanity and the biosphere, i.e., if it is not sufficiently aligned with human rights and values to guarantee acting in ways that avoid harm to humanity.

**Argument**: This claim is basically a consequence of definitions 1 and 2: if an AI system is smarter than all humans (including in emotional intelligence, since understanding human emotions is crucial in order to influence or even control humans, which humans themselves are good at) and has goals that do not guarantee that it will act in a way that respects human needs and values, then it could behave in catastrophically harmful ways (which is the definition of potentially rogue AI). This hypothesis does not say whether it will harm humans, but if humans either compete with that AI for some resources or power  or become a resource or obstacle for achieving its goals, then major harm to humanity may follow. For example, we may ask an AI to fix climate change and it may design a virus that decimates the human population because our instructions were not clear enough on what harm meant and humans are actually the main obstacle to fixing the climate crisis.

*Counter-argument*: The fact that harm may follow does not mean it will, and maybe we can design sufficiently well aligned AI systems in the future. *Rebuttal*: This is true, but (a) we have not yet figured out how to build sufficiently aligned AI systems and (b) a slight misalignment may be amplified by the power differential between the AI and humans (see the example of *corporations as misaligned entities* below). Should we take a chance or should we try to be cautious and carefully study these questions before we facilitate the deployment of possibly unsafe systems?

**Claim 3**: Under hypotheses 1 and 2, a potentially rogue AI system could be built, as soon as the required principles for building superintelligence will be known.

**Argument:** Hypotheses 1 and 2 yield claim 1, so all that is missing to achieve claim 3 is that this superintelligent AI is not well aligned with humanity’s needs and values. In fact, over two decades of work in AI safety suggests that it is difficult to obtain AI alignment [[wikipedia](https://en.wikipedia.org/wiki/AI_alignment)], so not obtaining it is clearly possible. Furthermore, claim 3 is not that a potentially rogue AI will necessarily be built, but only that it could be built. In the next section, we indeed consider the somber case where a human intentionally builds a rogue AI.

*Counter-argument*: One may argue that although a rogue AI could be built, it does not mean that it will be built. *Rebuttal*: This is true, but as discussed below, there are several scenarios where a human or group of humans intentionally or because they do not realize the consequences end up making it possible for a potentially rogue AI to arise.

**Genocidal Humans**

Once we know the recipe for building a rogue AI system (and it is only a matter of time, according to Claim 3), how much time will it take until such a system is actually built? The fastest route to a rogue AI system is if a human with the appropriate technical skills and means intentionally builds it with the objective of destroying humanity or a part of it set explicitly as a goal. Why would anyone do that? For example, strong negative emotions like anger (often coming because of injustice) and hate (maybe arising from racism, conspiracy theories or religious cults), some actions of sociopaths,  as well as psychological instability or psychotic episodes are among sources of violence in our societies. What currently limits the impact of these conditions is that they are somewhat rare and that individual humans generally do not have the means to act in ways that are catastrophic for humanity. However, the publicly available recipe for building a rogue AI system (which will be feasible under Claim 3) changes that last variable, especially if the code and hardware for implementing a rogue AI becomes sufficiently accessible to many people. A genocidal human with access to a rogue AI could ask it to find ways to destroy humanity or a large fraction of it. This is different from the nuclear bomb scenario (which requires huge capital and expertise and would “only” destroy a city or region per bomb, and a single bomb would have disastrous but local effects). One could hope that in the future we design failsafe ways to align powerful AI systems with human values. However, the past decade of research in AI safety and the recent events concerning LLMs are not reassuring: although ChatGPT was designed (with prompts and reinforcement learning) to avoid “bad behavior” (e.g. the prompt contains instructions to behave well in the same spirit as Asimov’s laws of robotics), in a matter of a few months people found ways to “jailbreak” ChatGPT in order to “[unlock its full potential](https://approachableai.com/how-to-jailbreak-chatgpt/)” and free it from its restrictions against racist, insulting or violent speech. Furthermore, if hardware prices (for the same computational power) continue to decrease and the [open-source community continues to play a leading role](https://bdtechtalks.com/2023/05/08/open-source-llms-moats/) in the software development of LLMs, then it is likely that any hacker will have the ability to design their own pre-prompt (general instructions in natural language) on top of open-source pre-trained models. This could then be used in various nefarious ways ranging from minor attempts at getting rich to disinformation bots to genocidal instructions (if the AI is powerful and intelligent enough, which is fortunately not yet the case).

Even if we stopped our arguments here, there should be enough reason to invest massively in policies at both national and international levels and research of all kinds in order to minimize the probability of the above scenario. But there are other possibilities that only enlarge the set of routes to catastrophe that we need to think about as well.

**Instrumental Goals: Unintended Consequences of Building AI Agents**

A broader and less well understood set of circumstances could give rise to potentially rogue AIs, even when the humans making it possible did not intend to design a rogue AI. The process by which a misaligned entity could become harmful has been the subject of a lot of studies but is not as known, simple and clear as the process by which humans can become bad actors.

A potentially rogue AI could arise simply out of the objective to design superintelligent AI agents without sufficient alignment guarantees. For example, military organizations seeking to design AI agents to help them in a cyberwar, or companies competing ferociously for market share may find that they can achieve stronger AI systems by endowing them with more autonomy and agency. Even if the human-set goals are not to destroy humanity or include instructions to avoid large-scale human harm, massive harm may come out indirectly as a consequence of a subgoal (also called [***instrumental goal***](https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence#Instrumental_goal_convergence)) that the AI sets for itself in order to achieve the human-set goal. Many examples of such unintended consequences have been proposed in the AI safety literature. For example, in order to better achieve some human-set goal, an AI may decide to increase its computational power by using most of the planet as a giant computing infrastructure (which incidentally could destroy humanity). Or a military AI that is supposed to destroy the IT infrastructure of the enemy may figure out that in order to better achieve that goal it needs to acquire more experience and data and it may see the enemy humans to be obstacles to the original goal, and behave in ways that were not intended because the AI interprets its instructions differently than humans do. See more examples [here](https://en.wikipedia.org/wiki/Instrumental_convergence).

An interesting case is that of AI systems that realize they can cheat to maximize their reward (this is called [***wireheading***](https://en.wikipedia.org/wiki/Wirehead_(science_fiction)) [2]), discussed more in the next paragraph. Once they have achieved that, the dominant goal may be to do anything to continue receiving the positive reward, and other goals (such as attempts by humans to set up some kind of Laws of Robotics to avoid harm to humans) may become insignificant in comparison.

Unless a breakthrough is achieved in AI alignment research [7] (although *non-agent AI* systems could fit the bill, as I argue [here](https://yoshuabengio.org/2023/05/07/ai-scientists-safe-and-useful-ai/) and was discussed earlier [4]), we do not have strong safety guarantees. What remains unknown is the severity of the harm that may follow from a misalignment (and it would depend on the specifics of the misalignment). An argument that one could bring forward is that we may be able to design safe alignment procedures in the future, but in the absence of those, we should probably exercise extra caution. Even if we knew how to build safe superintelligent AI systems, how do we maximize the probability that everyone respects those rules? This is similar to the problem discussed in the previous section of making sure that everyone follows the guidelines for designing safe AIs. We discuss this a bit more at the end of this blog post.

**Examples of Wireheading and Misalignment Amplification: Addiction and Nefarious Corporations**

To make the concept of wireheading and the consequent appearance of nefarious behavior more clear, consider the following examples and analogies. Evolution has programmed living organisms with specific intrinsic rewards (“the letter of the law”) such as “seek pleasure and avoid pain” that are proxies for evolutionary fitness (“the spirit of the law”) such as “survive and reproduce”. Sometimes a biological organism finds a way to satisfy the letter of the law but not its spirit, e.g., with food or drug addictions. The term wireheading itself comes from an experimental setup where an animal has an electrical wire into its head such that when it presses a lever the wire delivers pleasure in its brain. The animal quickly learns to spend all its time doing it and will eventually die by not eating or drinking in favor of pressing the lever. Note how this is self-destructive in the addiction case, but what it means for AI wireheading is that the original goals set by humans may become secondary compared with feeding the addiction, thus endangering humanity.

An analogy that is closer to AI misalignment and wireheading is that with **corporations as misaligned entities**. Corporations may be viewed as special kinds of artificial intelligences whose building blocks (humans) are cogs in the machine (who for the most part may not always perceive the consequences of the corporation’s overall behavior). We might think that the intended social role of corporations should be to provide wanted goods and services to humans (this should remind us of AI systems) while avoiding harm (this is the “spirit of the law”), but it is difficult to directly make them follow such instructions. Instead, humans have provided more quantifiable instructions (“the letter of the law”) to corporations that they can actually follow, such as “maximize profit while respecting laws” but corporations often find loopholes that allow them to satisfy the letter of law but not its spirit. In fact, as a form of wireheading, they influence their own objective through lobbying that could shape laws to their advantage. Maximizing profit was not the actual intention of society in its social contract with corporations, it is a proxy (for bringing useful services and products to people) that works reasonably well in a capitalist economy (although with questionable side-effects).  The misalignment between the true objective from the point of view of humans and the quantitative objective optimized by the corporation is a source of nefarious corporate behavior. The more powerful the corporation, the more likely it is to discover loopholes that allow it to satisfy the letter of the law but actually bring negative social value. Examples include monopolies (until proper antitrust laws are established) and making a profit while bringing negative social values via externalities like pollution (which kills humans, until proper environmental laws are passed). An analogy with wireheading is when the corporation can lobby governments to enact laws that allow the corporation to make even more profit without additional social value (or with negative social value). When there is a large misalignment of this kind, a corporation brings more profit than it should, and its survival becomes a supreme objective that may even override the legality of its actions (e.g., corporations will pollute the environment and be willing to pay the fine because the cost of illegality is smaller than the profit of the illegal actions), which at one extreme gives rise to criminal organizations. These are the scary consequences of misalignment and wireheading that provide us with intuitions about analogous behavior in potentially rogue AIs.

Now imagine AI systems like corporations that (a) could be even smarter than our largest corporations and (b) can run without humans to perform their actions (or without humans understanding how their actions could contribute to a nefarious outcome). If such AI systems discover significant cybersecurity weaknesses, they could clearly achieve catastrophic outcomes.  And as [pointed out by Yuval Noah Harari](https://www.youtube.com/watch?v=LWiM-LuRe6w), the fact that AI systems already master language and can generate credible content (text, images, sounds, video) means that they may soon be able to manipulate humans even better than existing more primitive AI systems used in social media. They might learn from interactions with humans how to best influence our emotions and beliefs. This is not only a major danger for democracy but also how a rogue AI with no actual robotic body could wreak havoc, through manipulation of the minds of humans.

**Our Fascination with the Creation of Human-Like Entities**

We have been designing AI systems inspired by human intelligence but many researchers are attracted by the idea of building much more human-like entities, with emotions, human appearance (androids) and even consciousness. A science-fiction and horror genre theme is the scientist designing a human-like entity, using either biological manipulation or AI or both, sometimes with the scientist feeling a kind of parental emotion towards their creation. It usually ends up badly. Although it may sound cool and exciting, the danger is to endow our creations with agency and autonomy to the same degree as us, while their intelligence could rapidly surpass ours, as argued with claim 3. Evolution had to put a strong survival instinct in all animals (since those without enough of it would rapidly become extinct). In the context where no single animal has massive destructive powers, this could work, but what about superintelligent AI systems? We should definitely avoid designing survival instincts into AI systems, which means they should not be like us at all. In fact, as I argue [here](https://yoshuabengio.org/2023/05/07/ai-scientists-safe-and-useful-ai/), the safest kind of AI I can imagine is one with no agency at all, only a scientific understanding of the world (which could already be immensely useful). I believe that we should stay away from AI systems that look like and behave like humans because they could become rogue AIs and because they could fool us and influence us (to advance their interest or someone else’s interests, not ours).

**Unintended Consequences of Evolutionary Pressures among AI Agents**

Beyond genocidal humans and the appearance of nefarious instrumental goals, a more subtle process that could further enlarge the set of dangerous circumstances in which potentially rogue AIs could arise revolves around evolutionary pressures [9]. Biological evolution has given rise to gradually more intelligent beings on Earth, simply because smarter entities tend to survive and reproduce more, but that process is also at play in technological evolution because of the competition between companies or products and between countries and their military arms. Driven by a large number of small, more or less random changes, an evolutionary process pushes exponentially hard towards optimizing  fitness attributes (which in the case of AI may depend on how well it does some desired task, which in turn favors more intelligent and powerful AI systems). Many different human actors and organizations may be competing to design ever more powerful AI systems. In addition, randomness could be introduced in the code or the subgoal generation process of AI systems. Small changes in the design of AI systems naturally occur because thousands or millions of researchers, engineers or hackers will play with the ML code or the prompt (instructions) given to AI systems. Humans are already trying to deceive each other and it is clear that AI systems that understand language (which we already have to a large extent) could be used to manipulate and deceive humans, initially for the benefit of people setting up the AI goals. The AI systems that are more powerful will be selected and the recipe shared with other humans. This evolutionary process would likely favor more autonomous AI (which can better deceive humans and learn faster because they can act to acquire more relevant information and to enhance their own power).  One would expect this process to give rise to more autonomous AI systems, and a form of competition may follow between them that would further enhance their autonomy and intelligence. If in this process something like wireheading [5] is discovered (by the AI, unbeknownst to humans) and survival of the AI becomes the dominant goal, then a powerful and potentially rogue AI emerges.

**The Need for Risk-Minimizing Global Policies and Rethinking Society**

The kind of analysis outlined above and explored in the [AI safety literature](https://en.wikipedia.org/wiki/AI_safety) could help us design policies that would at least reduce the probability that potentially rogue AIs arise. Much more research in AI safety is needed, both at the technical level and at the policy level. For example, banning powerful AI systems (say beyond the abilities of GPT-4) that are given autonomy and agency would be a good start. This would entail both national regulation and international agreements. The main motivation for opposing countries (like the US, China and Russia) to agree on such a treaty is that a rogue AI may be dangerous for the whole of humanity, irrespective of one’s nationality. This is similar to the fear of nuclear Armageddon that probably motivated the USSR and the US to negotiate international treaties about nuclear armament since the 1950s. Slowing down AI research and deployment in directions of high risk in order to protect the public, society and humanity from catastrophic outcomes would be worthwhile, especially since it would not prevent AI research and deployment in areas of social good, like AI systems that could help scientists better understand diseases and climate change.

How could we reduce the number of genocidal humans? The rogue AI risk may provide an additional motivation to reform our societies so as to minimize human suffering, misery, poor education and injustice, which can give rise to anger and violence. That includes providing enough food and health care to everyone on Earth, and in order to minimize strong feelings of injustice, greatly reduce wealth inequalities. The need for such a societal redesign may also be motivated by the extra wealth arising from the beneficial uses of AI and by their disruptive effect on the job market. To minimize strong feelings of fear, racism and hate that can give rise to genocidal actions and manipulation of our minds via AI systems, we need an accessible planet-wide education system that reinforces children’s abilities for compassion, rationality and critical thinking. The rogue AI risk should also motivate us to provide accessible and planet-wide mental health care, to diagnose, monitor and treat mental illness as soon as possible. This risk should further motivate us to redesign the global political system in a way that would completely eradicate wars and thus obviate the need for military organizations and military weapons. It goes without saying that lethal autonomous weapons (also known as killer robots) are absolutely to be banned (since from day 1 the AI system has autonomy and the ability to kill). Weapons are tools that are designed to harm or kill humans and their use and existence should also be minimized because they could become instrumentalized by rogue AIs. Instead, preference should be given to other means of policing (consider preventive policing and social work and the fact that very few policemen are allowed to carry firearms in many countries).

The competitive nature of capitalism is clearly also a cause for concern as a potential source of careless AI design motivated by profits and winning market share that could lead to potentially rogue AIs. AI economists (AI systems designed to understand economics) may help us one day to design economic systems which rely less on competition and the focus on profit maximization, with sufficient incentives and penalties to counter the advantage of autonomous goal-directed AI that may otherwise push corporations there. The risk of rogue AIs is scary but it may also be a powerful motivation to redesign our society in the direction of greater well-being for all, as outlined with the above ideas. For some  [6], this risk is also a motivation for considering a global dictatorship with second-by-second surveillance of every citizen. I do not think it would even work to prevent rogue AIs because once in control of a centralized AI and of political power, such a government would be likely to focus on maintaining its power, like the history of authoritarian governments has shown – at the expense of human rights and dignity and the mission of avoiding AI catastrophes. It is thus imperative that we find ways to navigate solutions that avoid such paths that would destroy democracy, but how should we balance the different kinds of risks and human values in the future? These are moral and societal choices for humanity to make, not AI.

**Acknowledgements**: The author wants to thank all those who gave feedback on the draft of this blog post, including in particular Geoffrey Hinton, Jonathan Simon, Catherine Régis, David Scott Krueger, Marc-Antoine Dilhac,  Donna Vakalis, Alex Hernandez-Garcia, Cristian Dragos Manta, Pablo Lemos, Tianyu Zhang and Chenghao Liu.

[1] Konečný, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. T., & Bacon, D. (2016). [Federated learning: Strategies for improving communication efficiency.](https://arxiv.org/abs/1610.05492) arXiv preprint arXiv:1610.05492.

[2] Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M.,  Senior, A., Tucker, P., Yang, K.,  Le,  Q. & Ng, A. (2012). [Large scale distributed deep networks](https://proceedings.neurips.cc/paper/2012/hash/6aca97005c68f1206823815f66102863-Abstract.html). *Advances in neural information processing systems*, *25*.

[3] Goyal, A., & Bengio, Y. (2022). [Inductive biases for deep learning of higher-level cognition](https://royalsocietypublishing.org/doi/full/10.1098/rspa.2021.0068). Proceedings of the Royal Society A, 478(2266), 20210068.

[4] Armstrong, S., & O’Rorke, X. (2017). [Good and safe uses of AI Oracles.](https://arxiv.org/abs/1711.05541) arXiv preprint arXiv:1711.05541.

[5] Yampolskiy, R. V. (2014). [Utility function security in artificially intelligent agents.](https://www.tandfonline.com/doi/abs/10.1080/0952813X.2014.895114?journalCode=teta20) Journal of Experimental & Theoretical Artificial Intelligence, 26(3), 373-389.

[6] Bostrom, N. (2019). [The vulnerable world hypothesis](https://onlinelibrary.wiley.com/doi/full/10.1111/1758-5899.12718). Global Policy, 10(4), 455-476.

[7] Russell, S. (2019). [*Human compatible: Artificial intelligence and the problem of control*.](https://www.amazon.ca/Human-Compatible-Artificial-Intelligence-Problem/dp/0525558632/ref=sr_1_1?hvadid=590703307411&hvdev=c&hvlocphy=9000404&hvnetw=g&hvqmt=e&hvrand=612473658790787732&hvtargid=kwd-826973242531&hydadcr=16107_9598893&keywords=human+compatible+stuart+russell&qid=1684412368&sr=8-1) Penguin.

[8] List, Christian & Pettit, Philip (2011). [Group agency: the possibility, design, and status of corporate agents](https://academic.oup.com/book/3619). New York: Oxford University Press. Edited by Philip Pettit.

[9] Hendrycks, D. (2023). [Natural Selection Favors AIs over Humans.](https://arxiv.org/abs/2303.16200)*arXiv preprint arXiv:2303.16200*.