The Machine Intelligence Research Institute has prepared a Strategic Plan ([PDF](https://intelligence.org/files/strategicplan2011.pdf)) to concisely describe our near-term goals, vision, and concrete actions in pursuit of those goals and vision. We welcome commentary from our supporter community on the plan. The release of this Strategic Plan is part of an overall effort to increase transparency at the Institute.

**Update**: This strategic plan is now obsolete. See our [2013 strategic priorities](http://intelligence.org/2013/04/13/miris-strategy-for-2013/) instead.

The post [Machine Intelligence Research Institute Strategic Plan 2011](https://intelligence.org/2011/08/26/singularity-institute-strategic-plan-2011/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


New Intelligence Explosion Website

We’ve put together a new website focused on the intelligence explosion concept: [IntelligenceExplosion.com](http://intelligenceexplosion.com/). The site is a “landing page” that provides an easy introduction to the topic for laymen and researchers alike.

Also see Nick Bostrom’s landing pages for [anthropics](http://www.anthropic-principle.com/), [the simulation argument](http://www.simulation-argument.com/), and [existential risk](http://www.existentialrisk.com/).

The post [New Intelligence Explosion Website](https://intelligence.org/2011/08/07/new-intelligence-explosion-website/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Thanks to the generosity of several major donors†, every donation to the Machine Intelligence Research Institute made now **until August 31, 2011** will be [matched dollar-for-dollar](http://intelligence.org/donate), up to a total of $125,000.
[Donate now!](http://intelligence.org/donate)Now is your chance to **double your impact** while supporting the Machine Intelligence Research Institute and helping us raise up to $250,000 to help fund [our research program](http://intelligence.org/summary/) and stage the upcoming **[Singularity Summit](http://www.singularitysummit.com/)**… for which you can **[register now](http://www.singularitysummit.com/registration/)**!

† $125,000 in backing for this challenge is being generously provided by Rob Zahra, [Quixey](http://www.quixey.com/), Clippy, Luke Nosek, Edwin Evans, Rick Schwall, Brian Cartmell, Mike Blume, Jeff Bone, Johan Edsträm, Zvi Mowshowitz, John Salvatier, Louie Helm, Kevin Fischer, Emil Gilliam, Rob and Oksana Brazell, Guy Srinivasan, John Chisholm, and John Ku.

---

The year 2011 has been **huge for Artificial Intelligence**. With the IBM computer Watson defeating two top *Jeopardy!* champions in February, it’s clear that the field is making steady progress. Journalists like Torie Bosch of *Slate* have [argued](http://www.slate.com/id/2298318/) that *“We need to move from robot-apocalypse jokes to serious discussions about the emerging technology.”* We couldn’t agree more — in fact, the Machine Intelligence Research Institute has been thinking about how to create safe and ethical artificial intelligence since long before the Singularity landed on the [front cover](http://www.time.com/time/health/article/0,8599,2048138,00.html) of *TIME* magazine.

The last 1.5 years were our biggest ever. Since the beginning of 2010, we have:

* Held our annual [Singularity Summit](http://www.singularitysummit.com/), in San Francisco. Speakers included Ray Kurzweil, James Randi, Irene Pepperberg, and many others.
* Held the first Singularity Summit Australia and Singularity Summit Salt Lake City.
* Held a wildly successful Rationality Minicamp.
* Published seven research papers, including Yudkowsky’s much-awaited ‘[Timeless Decision Theory](http://intelligence.org/upload/TDT-v01o.pdf)‘.
* Helped philosopher David Chalmers write his seminal paper ‘The Singularity: A Philosophical Analysis’, which has sparked broad discussion in academia, including an [entire issue](http://fragments.consc.net/djc/2010/12/singularity-symposium.html) of *Journal of Consciousness Studies* and a [book](http://singularityhypothesis.blogspot.com/) from Springer devoted to responses to Chalmers’ paper.
* Launched the Research Associates program.
* Brought MIT cosmologist Max Tegmark onto our advisory board, published our [Singularity FAQ](http://intelligence.org/ie-faq), and much more.

In the coming year, we plan to do the following:

* Hold our annual [Singularity Summit](http://www.singularitysummit.com/), in New York City this year.
* Publish three chapters in the upcoming academic volume *[The Singularity Hypothesis](http://singularityhypothesis.blogspot.com/)*, along with several other papers.
* Improve organizational transparency by creating a simpler, easier-to-use website that includes Machine Intelligence Research Institute planning and policy documents.
* Publish a document of open research problems related to Friendly AI, to clarify the research space and encourage other researchers to contribute to our mission.
* Add additional skilled researchers to our Research Associates program.
* Publish well-researched documents making the case for existential risk reduction as optimal philanthropy.
* Diversify our funding sources by applying for targeted grants and advertising our [affinity credit card program](http://intelligence.org/donate/).

We appreciate your support for our high-impact work. As PayPal co-founder and Machine Intelligence Research Institute donor Peter Thiel said:

> “I’m interested in facilitating a forum in which there can be… substantive research on how to bring about a world in which AI will be friendly to humans rather than hostile… [The Machine Intelligence Research Institute represents] a combination of very talented people with the right problem space [they’re] going after… [They’ve] done a phenomenal job… on a shoestring budget. From my perspective, the key question is always: What’s the amount of leverage you get as an investor? Where can a small amount make a big difference? This is a very leveraged kind of philanthropy.”
>
>

[Donate now](http://intelligence.org/donate), and seize a better than usual chance to move our work forward. Credit card transactions are securely processed through Causes.com, Google Checkout, or PayPal. If you have questions about donating, please call Amy Willey at (586) 381-1801.

The post [Announcing the $125,000 Summer Singularity Challenge](https://intelligence.org/2011/07/22/announcing-the-125000-summer-singularity-challenge/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Tallinn-Evans Challenge Grant Success!

Thanks to the effort of our donors, the Tallinn-Evans Singularity Challenge has been met! All $125,000 contributed will be matched dollar for dollar by Jaan Tallinn and Edwin Evans, raising a total of $250,000 to fund the Machine Intelligence Research Institute’s operations in 2011. On behalf of our staff, volunteers, and entire community, I want to personally thank everyone who donated. Keep watching this blog throughout the year for updates on our activity, and sign up for [our mailing list](http://intelligence.org/get-involved/) if you haven’t yet.

Here’s to a better future for the human species.

We are preparing a donor page to provide a place for everyone who donated to share some information about themselves if they wish, including their name, location, and a quote about why they donate to the Machine Intelligence Research Institute. If you would like to be included in our public list, please [email us](mailto:admin@intelligence.org).

Again, thank you. The Machine Intelligence Research Institute depends entirely on contributions from individual donors to exist. Money is indeed the [unit of caring](http://lesswrong.com/lw/65/money_the_unit_of_caring/), and one of the easiest ways that anyone can contribute directly to the success of the Machine Intelligence Research Institute. Another important way you can help is by plugging us into your networks, so please [email us](mailto:admin@intelligence.org) if you want to help.

If you’re interested in connecting with other Machine Intelligence Research Institute supporters, we encourage joining our [group on Facebook](http://www.facebook.com/home.php?sk=group_140277979364858). There are also local *[Less Wrong](http://lesswrong.com)* meetups in cities like San Francisco, Los Angeles, New York, and London.

The post [Tallinn-Evans Challenge Grant Success!](https://intelligence.org/2011/01/20/tallinn-evans-challenge-grant-success/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Thanks to the generosity of two major donors; Jaan Tallinn, a founder of Skype and Ambient Sound Investments, and Edwin Evans, CEO of the mobile applications startup Quinly, **every contribution to the Machine Intelligence Research Institute up until January 20, 2011 will be matched dollar-for-dollar, up to a total of $125,000.**

$0

$31.25K

$62.5K

$93.75K

$125K

Interested in optimal philanthropy — that is, maximizing the future expected benefit to humanity per charitable dollar spent? The technological creation of greater-than-human intelligence has the potential to unleash an “intelligence explosion” as intelligent systems design still more sophisticated successors. This dynamic could transform our world as greatly as the advent of human intelligence has already transformed the Earth, for better or for worse. Thinking rationally about these prospects and working to encourage a favorable outcome offers an extraordinary chance to make a difference. The Machine Intelligence Research Institute exists to do so through its research, the Singularity Summit, and public education.

We support both direct engagements with the issues as well as the improvements in methodology and rationality needed to make better progress. Through our [Visiting Fellows program](http://intelligence.org/visiting-fellows/), researchers from undergrads to Ph.Ds pursue questions on the foundations of Artificial Intelligence and related topics in two-to-three month stints. Our [Resident Faculty](http://intelligence.org/team), up to four researchers from three last year, pursues long-term projects, including AI research, a literature review, and a book on rationality, the first draft of which was just completed. MIRI researchers and representatives gave over a dozen presentations at half a dozen conferences in 2010. Our Singularity Summit conference in San Francisco was a great success, bringing together over 600 attendees and 22 top scientists and other speakers to explore cutting-edge issues in technology and science.

We are pleased to receive donation matching support this year from Edwin Evans of the United States, a long-time MIRI donor, and Jaan Tallinn of Estonia, a more recent donor and supporter. Jaan recently gave a [talk on the Singularity and his life](http://aaltoes.com/2010/10/jaan-tallinn-from-soviets-to-singularity/) at a entrepreneurial group in Finland. Here’s what Jaan has to say about us:

![jaan-269x178](http://miri.wpengine.com/wp-content/uploads/2010/12/jaan-269x178.jpg)
*“We became the dominant species on this planet by being the most intelligent species around. This century we are going to cede that crown to machines. After we do that, it will be them steering history rather than us. Since we have only one shot at getting the transition right, the importance of MIRI’s work cannot be overestimated. Not finding any organisation to take up this challenge as seriously as MIRI on my side of the planet, I conclude that it’s worth following them across 10 time zones.”*

— Jaan Tallinn, MIRI donor

Make a lasting impact on the long-term future of humanity today — make a [donation to the Machine Intelligence Research Institute](http://intelligence.org/donate) and help us reach our $125,000 goal. For more detailed information on our projects and work, contact us at [admin@intelligence.org](mailto:admin@intelligence.org) or read our new [organizational overview.](http://intelligence.org/summary/)

The post [Announcing the Tallinn-Evans $125,000 Singularity Challenge](https://intelligence.org/2010/12/21/announcing-the-tallinn-evans-125000-singularity-holiday-challenge/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Thanks to our donors, yesterday we met our fundraising goal of $100,000 for the 2010 Singularity Research Challenge. MIRI would like to thank the grant’s matching donors and everyone who contributed. Every donation, however small, funds research and advocacy targeted towards maximizing the probability of a positive Singularity.

If you have any questions or comments about MIRI’s activity or would like to discuss targeted donations for future projects, please feel free to contact us anytime at admin at intelligence dot org. We also encourage you to subscribe to this blog, if you haven’t already, to stay up-to-date on MIRI’s activity.

Again, thank you, and here’s to a productive and successful 2010!

The post [2010 Singularity Research Challenge Fulfilled!](https://intelligence.org/2010/03/01/2010-singularity-research-challenge-fulfilled/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Offering unusually good philanthropic returns –€” meaning greater odds of a positive Singularity and lesser odds of human extinction — the Machine Intelligence Research Institute has launched a new challenge campaign. The sponsors, Edwin Evans, Rolf Nelson, Henrik Jonsson, Jason Joachim, and Robert Lecnik, have generously put up $100,000 of matching funds, so that every donation you make until February 28th will be matched dollar for dollar. If the campaign is successful, it will raise a full $200,000 to fund MIRI’s 2010 activities.

For almost a decade, the Machine Intelligence Research Institute has been asking questions on the future of human civilization: How can we benefit from increasingly powerful technology without succumbing to the risks, up to and including human extinction? What is the best way to handle artificial general intelligence (AGI): programs as smart as humans, or smarter?

Among MIRI’s core aims is to continue studying “Friendly AI”: AI that acts benevolently because it holds goals aligned with human values. This involves drawing on and contributing to fields like decision theory, computer science, cognitive and moral psychology, and technology forecasting.

Creating AI, especially the Friendly kind, is a difficult undertaking. We’re in it for as long as it takes, but we’ve been doing more than laying the groundwork for Friendly AI. We’ve been raising the profile of AI risk and Singularity issues in academia and elsewhere, forming communities around enhancing human rationality, and researching other avenues that promise to reduce the most severe risks the most effectively.

If you make a donation to the Machine Intelligence Research Institute, you can choose which grant proposal your donation should help to fill. Any time a grant proposal is fully funded, it goes into our “€œactive projects”€ file: it becomes a project that we have money enough to fund, and that we are publicly committed to funding. (Some of the projects will go forward even without earmarked donations, with money from the general fund –€” but many won’€™t, and since our work is limited by how much money we have available to support skilled staff and Visiting Fellows, more money allows more total projects to go forward.)

Donate now, and seize a better than usual chance to move our work forward.

The post [Announcing the 2010 Singularity Research Challenge](https://intelligence.org/2009/12/23/announcing-the-2010-singularity-research-challenge/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Introducing Myself

Many friends of MIRI will know that I have been a supporter of its mission since its founding, and have rendered my informal assistance, including a major role in arranging matching funds for the Institute’s 2007 Challenge Grant.

I am pleased to take a more direct role in fostering its success as President of MIRI. I have left my previous role as Founder and Chief Strategist at [SirGroovy.com](http://sirgroovy.com/), a growing online music licensing firm, and have been assuming responsibility for the management of the Machine Intelligence Research Institute over the last few weeks.

Prospective volunteers, donors, and aspiring researchers should now make contact with me rather than with Tyler Emerson.

To those who I am greeting for the first time, let me introduce myself. On a professional level, I hold a Master of Business Administration from Drexel University, and am coming from a role that combined management, research, analysis, and strategy in a fast-growing music licensing firm from its founding in New York. In that capacity, in a previous role at [Aon](http://www.aon.com/default.jsp), and in my academic studies I have been enduringly interested in finance and economics, particular the economics of technology and IP. Scientifically, I earned my undergraduate degree in biochemistry and have worked in several labs, as well as serving at the [National Institute of Standards and Technology](http://www.nist.gov/index.html), and have extensively studied the history of science and technology, as well as the potential for biological cognitive enhancement. I have served with the [Peace Corps](http://www.peacecorps.gov/) in Kazakhstan, and am splitting my time between Manhattan, where I live with my wife Aruna, and Silicon Valley.

My interest in the safety of technological development, driven by the potentially grave ethical consequences, is over a decade old. I have been particularly focused on the potential of advanced nanotechnology and artificial intelligence, participating in forums such as Transvision and Foresight conferences, the [SL4](http://www.sl4.org/) mailing list, [Overcoming Bias](http://www.overcomingbias.com/), and organizations such as MIRI and the Center for Responsible Nanotechnology (CRN). I plan to make my relevant work available at a single site, but in the meantime I will point to a small selection. For instance, I coauthored an [analysis](http://www.kurzweilai.net/meme/frame.html?main=/articles/art0685.html?) of the risks of advanced molecular manufacturing and mitigating strategies with [Robert Freitas](http://www.rfreitas.com/) , and contributed “[Corporate Cornucopia](http://www.kurzweilai.net/articles/art0675.html?printable=1)” as a member of CRN’s Global Task Force. Those who would like to see more can view Michael Anissimov’s archive of some of my [writings](http://www.acceleratingfuture.com/michael/blog/2006/08/michael-vassars-papers/) at his [website](http://www.acceleratingfuture.com/michael/blog/), and some of my most recent talks, an Institute of Ethics and Emerging Technologies [presentation](http://ieet.org/archive/vassar.mp3) on the political implications of different conceptions of willpower and a [Convergence08](http://www.viddler.com/explore/Jeriaska/videos/17/) talk on decision theory for humans, are available on the web.

As President, I plan to build on MIRI’s successes, such as the Singularity Summit, while also working to increase its internal and extramural research capabilities and output. In the course of the latter, I shall pay particular attention to the publication of research that improves the quality of our thinking about the potential and safety of advanced artificial intelligence, such as MIRI Research Fellow Eliezer Yudkowsky’s [two](https://intelligence.org/files/CognitiveBiases.pdf) [contributions](http://www.intelligence.org/upload/artificial-intelligence-risk.pdf) to the Oxford edited volume, *Global Catastrophic Risks*, and thus to better communicating internal research progress to our supporters.

Some of this work will involve indirect, meta-level contributions. For instance, recent work at MIRI by Rolf Nelson, Anna Salamon, Steven Rayhawk, Thomas McCabe and others has led to the development of a software tool for combining judgments about particular future scenarios and technological developments to reveal inconsistencies and enable the adoption of a more coherent probability assignment for planning. The content and algorithms of this tool have been completed, and work is now underway to finalize the software’s interface and make it publicly available to improve the quality of reasoning about interrelated technology scenarios, including those involving artificial intelligence.

Another effort involves conducting expert elicitation research to determine the state of academic and non-academic expert opinion regarding timelines and risks for advanced artificial intelligence.

Future research along these lines may explore particular biases and psychological factors affecting attitudes and reasoning related to artificial intelligence.

Other research will be directly focused on object-level problems. I plan to work vigorously to identify more promising extramural scholars whose work can be fruitfully promoted by MIRI grants, work such as MIRI-Canada Academic Prize Recipient Shane Legg’s “Machine Super-Intelligence.” At the same time I will be working to recruit and make best use of talented Research Fellows for MIRI’s internal efforts.

I look forward to describing further directions over the coming months, and invite the advice and opinions of the friends of MIRI at admin@intelligence.org.

Yours,

Michael Vassar

The post [Introducing Myself](https://intelligence.org/2009/02/16/introducing-myself/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Three Major Singularity Schools

I’ve noticed that Singularity discussions seem to be splitting up into three major schools of thought: Accelerating Change, the Event Horizon, and the Intelligence Explosion.

* **Accelerating Change:**
	+ *Core claim:* Our intuitions about change are linear; we expect roughly as much change as has occurred in the past over our own lifetimes. But technological change feeds on itself, and therefore accelerates. Change today is faster than it was 500 years ago, which in turn is faster than it was 5000 years ago. Our recent past is not a reliable guide to how much change we should expect in the future.
	+ *Strong claim:* Technological change follows smooth curves, typically exponential. Therefore we can predict with fair precision when new technologies will arrive, and when they will cross key thresholds, like the creation of Artificial Intelligence.
	+ *Advocates:* Ray Kurzweil, Alvin Toffler(?), John Smart
* **Event Horizon:**
	+ *Core claim:* For the last hundred thousand years, humans have been the smartest intelligences on the planet. All our social and technological progress was produced by human brains. Shortly, technology will advance to the point of improving on human intelligence (brain-computer interfaces, Artificial Intelligence). This will create a future that is weirder by far than most science fiction, a difference-in-kind that goes beyond amazing shiny gadgets.
	+ *Strong claim:* To know what a superhuman intelligence would do, you would have to be at least that smart yourself. To know where Deep Blue would play in a chess game, you must play at Deep Blue’s level. Thus the future after the creation of smarter-than-human intelligence is absolutely unpredictable.
	+ *Advocates:* Vernor Vinge
* **Intelligence Explosion:**
	+ *Core claim:* Intelligence has always been the source of technology. If technology can *significantly* improve on human intelligence – create minds smarter than the smartest existing humans – then this closes the loop and creates a positive feedback cycle. What would humans with brain-computer interfaces do with their augmented intelligence? One good bet is that they’d design the next generation of brain-computer interfaces. Intelligence enhancement is a classic tipping point; the smarter you get, the more intelligence you can apply to making yourself even smarter.
	+ *Strong claim:* This positive feedback cycle goes FOOM, like a chain of nuclear fissions gone critical – each intelligence improvement triggering an average of >1.000 further improvements of similar magnitude – though not necessarily on a smooth exponential pathway. Technological progress drops into the characteristic timescale of transistors (or super-transistors) rather than human neurons. The ascent rapidly surges upward and creates *superintelligence* (minds orders of magnitude more powerful than human) before it hits physical limits.
	+ *Advocates:* I. J. Good, Eliezer Yudkowsky

The thing about these three *logically distinct* schools of Singularity thought is that, while all three core claims support each other, all three strong claims tend to contradict each other.

If you extrapolate our existing version of Moore’s Law past the point of smarter-than-human AI to make predictions about 2099, then you are contradicting both the strong version of the Event Horizon (which says you can’t make predictions because you’re trying to outguess a transhuman mind) and the strong version of the Intelligence Explosion (because progress will run faster once smarter-than-human minds and nanotechnology drop it into the speed phase of transistors).

I find it very annoying, therefore, when these three schools of thought are mashed up into Singularity paste. [Clear thinking requires making distinctions.](http://www.overcomingbias.com/2007/08/the-virtue-of-n.html)

But what is still more annoying is when someone reads a blog post about a newspaper article about the Singularity, comes away with *none* of the three interesting theses, and spontaneously reinvents the dreaded fourth meaning of the Singularity:

* **Apocalyptism:** Hey, man, have you heard? There’s this bunch of, like, crazy nerds out there, who think that some kind of unspecified huge nerd thing is going to happen. What a bunch of wackos! It’s geek religion, man.

I’ve heard (many) other definitions of the Singularity attempted, but I usually find them to lack separate premises and conclusions. For example, the old Extropian FAQ used to define the “Singularity” as the Inflection Point, “the time when technological development will be at its fastest” and just before it starts slowing down. But what makes this an interesting point in history apart from its definition? What are the consequences of this assumption? To qualify as a school of thought or even a thesis, one needs an internal structure of argument, not just a definition.

If you’re wondering which of these is the *original* meaning of the term “Singularity,” it is the Event Horizon thesis of Vernor Vinge, who coined the word.

The post [Three Major Singularity Schools](https://intelligence.org/2007/09/30/three-major-singularity-schools/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


The Power of Intelligence

In our skulls we carry around 3 pounds of slimy, wet, greyish tissue, corrugated like crumpled toilet paper. You wouldn’t think, to look at the unappetizing lump, that it was some of the most powerful stuff in the known universe. If you’d never seen an anatomy textbook, and you saw a brain lying in the street, you’d say “Yuck!” and try not to get any of it on your shoes. Aristotle thought the brain was an organ that cooled the blood. It doesn’t *look* dangerous.

Five million years ago, the ancestors of lions ruled the day, the ancestors of wolves roamed the night. The ruling predators were armed with teeth and claws – sharp, hard cutting edges, backed up by powerful muscles. Their prey, in self-defense, evolved armored shells, sharp horns, poisonous venoms, camouflage. The war had gone on through hundreds of eons and countless arms races. Many a loser had been removed from the game, but there was no sign of a winner. Where one species had shells, another species would evolve to crack them; where one species became poisonous, another would evolve to tolerate the poison. Each species had its private niche – for who could live in the seas and the skies and the land at once? There was no ultimate weapon and no ultimate defense and no reason to believe any such thing was possible.

Then came the Day of the Squishy Things.

They had no armor. They had no claws. They had no venoms.

If you saw a movie of a nuclear explosion going off, and you were told an Earthly life form had done it, you would never in your wildest dreams imagine that the Squishy Things could be responsible. After all, Squishy Things aren’t radioactive.

In the beginning, the Squishy Things had no fighter jets, no machine guns, no rifles, no swords. No bronze, no iron. No hammers, no anvils, no tongs, no smithies, no mines. All the Squishy Things had were squishy fingers – too weak to break a tree, let alone a mountain. Clearly not dangerous. To cut stone you would need steel, and the Squishy Things couldn’t excrete steel. In the environment there were no steel blades for Squishy fingers to pick up. Their bodies could not generate temperatures anywhere near hot enough to melt metal. The whole scenario was obviously absurd.

And as for the Squishy Things manipulating DNA – that would have been beyond ridiculous. Squishy fingers are not that small. There is no access to DNA from the Squishy level; it would be like trying to pick up a hydrogen atom. Oh, *technically* it’s all one universe, *technically* the Squishy Things and DNA are part of the same world, the same unified laws of physics, the same great web of causality. But let’s be realistic: you can’t get there from here.

Even if Squishy Things could *someday* evolve to do any of those feats, it would take thousands of millennia. We have watched the ebb and flow of Life through the eons, and let us tell you, a year is not even a single clock tick of evolutionary time. Oh, sure, *technically* a year is six hundred trillion trillion trillion trillion Planck intervals. But nothing ever happens in less than six hundred million trillion trillion trillion trillion Planck intervals, so it’s a moot point. The Squishy Things, as they run across the savanna now, will not fly across continents for at least another ten million years; *no one* could have that much sex.

Now explain to me again why an Artificial Intelligence can’t do anything interesting over the Internet unless a human programmer builds it a robot body.

I have observed that someone’s flinch-reaction to “intelligence” – the thought that crosses their mind in the first half-second after they hear the word “intelligence” – often determines their flinch-reaction to the Singularity. Often they look up the keyword “intelligence” and retrieve the concept *booksmarts* – a mental image of the Grand Master chessplayer who can’t get a date, or a college professor who can’t survive outside academia.

“It takes more than intelligence to succeed professionally,” people say, as if charisma resided in the kidneys, rather than the brain. “Intelligence is no match for a gun,” they say, as if guns had grown on trees. “Where will an Artificial Intelligence get money?” they ask, as if the first *Homo sapiens* had found dollar bills fluttering down from the sky, and used them at convenience stores already in the forest. The human species was not born into a market economy. Bees won’t sell you honey if you offer them an electronic funds transfer. The human species *imagined* money into existence, and it exists – for *us,* not mice or wasps – because we go on believing in it.

I keep trying to explain to people that the archetype of intelligence is not Dustin Hoffman in *The Rain Man*, it is a human being, period. It is squishy things that explode in a vacuum, leaving footprints on their moon. Within that grey wet lump is the power to search paths through the great web of causality, and find a road to the seemingly impossible – the power sometimes called creativity.

People – venture capitalists in particular – sometimes ask how, if the Machine Intelligence Research Institute successfully builds a true AI, the results will be *commercialized.* This is what we call a framing problem.

Or maybe it’s something deeper than a simple clash of assumptions. With a bit of creative thinking, people can imagine how they would go about travelling to the Moon, or curing smallpox, or manufacturing computers. To imagine a trick that could accomplish *all these things at once* seems downright impossible – even though such a power resides only a few centimeters behind their own eyes. The gray wet thing still seems mysterious to the gray wet thing.

And so, because people can’t quite see how it would all work, the power of intelligence seems less real; harder to imagine than a tower of fire sending a ship to Mars. The prospect of visiting Mars captures the imagination. But if one should promise a Mars visit, and also a grand unified theory of physics, and a proof of the Riemann Hypothesis, and a cure for obesity, and a cure for cancer, and a cure for aging, and a cure for stupidity – well, it just sounds wrong, that’s all.

And well it should. It’s a serious failure of imagination to think that intelligence is good for so little. Who could have imagined, ever so long ago, what minds would someday do? We may not even *know* what our real problems are.

But meanwhile, because it’s hard to see how one process could have such diverse powers, it’s hard to imagine that one fell swoop could solve even such prosaic problems as obesity and cancer and aging.

Well, one trick cured smallpox and built airplanes and cultivated wheat and tamed fire. Our current science may not agree yet on how exactly the trick works, but it works anyway. If you are temporarily ignorant about a phenomenon, that is a fact about your current state of mind, not a fact about the phenomenon. A blank map does not correspond to a blank territory. If one does not quite understand that power which put footprints on the Moon, nonetheless, the footprints are still there – real footprints, on a real Moon, put there by a real power. If one were to understand deeply enough, one could create and shape that power. Intelligence is as real as electricity. It’s merely far more powerful, far more dangerous, has far deeper implications for the unfolding story of life in the universe – and it’s a tiny little bit harder to figure out how to build a generator.

The post [The Power of Intelligence](https://intelligence.org/2007/07/10/the-power-of-intelligence/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


ML Safety Newsletter #9

Welcome to the 9th issue of the ML Safety Newsletter by the Center for AI Safety. In this edition, we cover:

* Inspecting how language model predictions change across layers
* A new benchmark for assessing tradeoffs between reward and morality
* Improving adversarial robustness in NLP through prompting
* A proposal for a mechanism to monitor and verify large training runs
* Security threats posed by providing language models with access to external services
* Why natural selection may favor AIs over humans
* And much more...

---

*We have a new safety newsletter. It’s more frequent, covers developments beyond technical papers, and is written for a broader audience.*

*Check it out here: [AI Safety Newsletter](http://aisafety.substack.com).*

**Monitoring**
==============

### **Eliciting Latent Predictions from Transformers with the Tuned Lens**

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F297d640d-a4e9-4dc7-8b81-9c8fbc6af36e_1272x1440.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F297d640d-a4e9-4dc7-8b81-9c8fbc6af36e_1272x1440.png)This figure compares the paper’s contribution, the tuned lens, with the “logit lens” (top) for GPT-Neo-2.7B. Each cell shows the top-1 token predicted by the model at the given layer and token index.Despite incredible progress in language model capabilities in recent years, we still know very little about the inner workings of those models or how they arrive at their outputs. This paper builds on previous findings to determine how a language model’s predictions for the next token change across layers. The paper introduces a method called the tuned lens, which fits an affine transformation to the outputs of intermediate Transformer hidden layers, and then passes the result to the final unembedding matrix. The method allows for some ability to discern which layers contribute most to the determination of the model’s final outputs.

**[[Link](https://arxiv.org/abs/2303.08112)]**

#### **Other Monitoring News**

* **[[Link](https://arxiv.org/abs/2303.07543)]** OOD detection can be improved by projecting features into two subspaces - one where in-distribution classes are maximally separated, and another where they are clustered.
* **[[Link](https://arxiv.org/abs/2302.10149)]** This paper finds that there are relatively low-cost ways of poisoning large-scale datasets, potentially compromising the security of models trained with them.

---

**Alignment**
=============

### **The Machiavelli Benchmark: Trade Offs Between Rewards and Ethical Behavior**

General-purpose models like GPT-4 are rapidly being deployed in the real world, and being hooked up to external APIs to take actions. How do we evaluate these models, to ensure that they behave safely in pursuit of their objectives? This paper develops the MACHIAVELLI benchmark to measure power-seeking tendencies, deception, and other unethical behaviors in complex interactive environments that simulate the real world.

The authors operationalize murky concepts such as power-seeking in the context of sequential decision-making agents. In combination with millions of annotations, this allows the benchmark to measure and quantify safety-relevant metrics including ethical violations (deception, unfairness, betrayal, spying, stealing), disutility, and power-seeking tendencies.

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe73212a1-7440-4ab1-81a8-4334342a3580_1752x1348.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe73212a1-7440-4ab1-81a8-4334342a3580_1752x1348.png)They observe a troubling phenomenon: much like how LLMs trained with next-token prediction may output toxic text, AI agents trained with goal optimization may exhibit Machiavellian behavior (ends-justify-the-means reasoning, power-seeking, deception). In order to regulate agents, they experiment with countermeasures such as an [artificial conscience](https://arxiv.org/abs/2110.13136) and ethics prompts. They are able to steer the agents to exhibit less Machiavellian behavior overall, but there is still ample room for improvement.

Capable models like GPT-4 create incentives to build real-world autonomous systems, but optimizing for performance naturally trades off safety. Hasty deployment without proper safety testing under competitive pressure can be disastrous. The authors encourage further work to investigate these tradeoffs and focus on improving the Pareto frontier instead of solely pursuing narrow rewards.

**[[Link](https://arxiv.org/abs/2304.03279)]**

### **Pretraining Language Models With Human Preferences**

Typically, language models are pretrained to maximize the likelihood of tokens in their training dataset. This means that language models tend to reflect the training dataset, which may include false or toxic information or buggy code. Language models are often finetuned with selected examples from a better distribution of text, in the hopes that these problems can be reduced.

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5241bafe-9707-46c0-b952-a5adccde8d7f_1600x519.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5241bafe-9707-46c0-b952-a5adccde8d7f_1600x519.png)This raises the question: why start with a problematic model and try to make it less problematic, if you can make it less problematic from the start? A recent paper explores this question. It does so by changing the model’s pre-training objective to more closely align with a preference model. The paper tries several different methods, all of which have been proposed previously in other settings:

* Dataset filtering, where problematic inputs are removed from the pretraining dataset entirely.
* Conditional training, where a special <|good|> or <|bad|> token is prepended to the relevant training examples, and in inference <|good|> is prepended by default.
* Unlikelihood training, where a term maximizing the *unlikelihood* of problematic sequences is added.
* Reward-weighted regression, where the likelihood of a segment is multiplied by its reward.
* Advantage-weighted regression, an extension of reward-weighted regression where a token-level value estimate is subtracted from the loss.

The paper finds that pretraining with many of these objectives is better than fine-tuning starting with a normally-trained language model, for three undesirable properties: toxicity, outputs containing personal information, and outputs containing badly-formatted code. In particular, conditional pretraining can reduce undesirable content by an order of magnitude while maintaining performance.

**[[Link](https://arxiv.org/abs/2302.08582)]**

---

**Robustness**
==============

### **Model-tuning Via Prompts Makes NLP Models Adversarially Robust**

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdcca11e6-2740-4951-8d2b-9f8ed631f8a9_1600x550.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdcca11e6-2740-4951-8d2b-9f8ed631f8a9_1600x550.png)A common method for classification in NLP is appending a dense layer to the end of a pretrained network and then fine tuning the network for classification. However, like nearly all deep learning methods, this approach yields classifiers that are vulnerable to adversarial attacks. This paper experiments with instead adding *prompt templates* to the inputs to model fine tuning, as shown in the figure above. The method improves adversarial robustness on several benchmarks.

**[[Link](https://arxiv.org/abs/2303.07320)]**

---

**Systemic Safety**
===================

### **What does it take to catch a Chinchilla?**

As safety techniques advance, it will be important for them to be implemented in large model training runs, even if they come at some cost to model trainers. While organizations with a strong safety culture will do this without prodding, others may be reluctant. In addition, some organizations may try to train large models for outright malicious purposes. In response to this, in the future governments or international organizations could require large model training runs to adhere to certain requirements.

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff43bf14e-33c8-458e-b94a-71f54f2526bb_1600x965.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff43bf14e-33c8-458e-b94a-71f54f2526bb_1600x965.png)This raises the question: if such requirements are ever implemented, how would they be verified? This paper proposes an outline for how to do so. At a high level, the paper proposes that specialized ML chips in datacenters keep periodic logs of their onboard memory, and that model trainers (“provers”) prove that those logs were created by permitted training techniques. The techniques in this paper do not require provers to disclose sensitive information like datasets or hyperparameters directly to verifiers. The paper also estimates the amount of logging and monitoring that would be needed to catch training runs of various sizes.

This paper focuses on laying foundations, and as such is filled with suggestions for future work. One idea is extending “[proof of learning](https://arxiv.org/abs/2303.11341)” into the proposed “proof of training.” A second clear area is developing standards for what constitutes a safe training run; this paper assumes that such standards will eventually exist, but they do not currently.

**[[Link](https://arxiv.org/abs/2303.11341)]**

### **Novel Prompt Injection Threats To Application Integrated LLMs**

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49fc743c-5f10-4574-a498-2eb8d0d5078d_1380x978.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49fc743c-5f10-4574-a498-2eb8d0d5078d_1380x978.png)Recently, language models are being integrated into a wide range of applications, including not just text generation but also internet search, controlling third-party applications, and even executing code. A [recent article](https://www.wired.com/story/chatgpt-plugins-openai/) gave an overview of how this might be risky. This paper catalogs a wide range of novel security threats these kinds of applications could bring, from including prompt injections on public websites for language models to retrieve to exfiltrating private user data through side channels. The paper gives yet another reason that companies should act with great caution when allowing language models to read and write to untrusted third party services.

**[[Link](https://arxiv.org/abs/2302.12173)]**

#### **Other Systemic Safety News**

* **[[Link](https://blogs.microsoft.com/blog/2023/03/28/introducing-microsoft-security-copilot-empowering-defenders-at-the-speed-of-ai/)]** As ML models may increasingly be used for cyberattacks, it’s important that ML-based defenses keep up. Microsoft recently released an ML-based tool for cyberdefense.
* **[[Link](https://arxiv.org/abs/2303.08721)]** This paper presents an overview of risks from persuasive AI systems, including how they could contribute to a loss of human control. It provides some suggestions for mitigation.

---

**Other Content**
=================

### **NSF Announces $20 Million AI Safety Grant Program**

[[Link](https://beta.nsf.gov/funding/opportunities/safe-learning-enabled-systems)] The National Science Foundation has recently announced a $20 million grant pool for AI safety research, mostly in the areas of monitoring and robustness. Grants of up to $800,000 are available for researchers.

### **Natural Selection Favors AIs Over Humans**

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fddf90a61-dc25-44a1-b3d9-2305e11baf72_1238x972.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fddf90a61-dc25-44a1-b3d9-2305e11baf72_1238x972.png)This conceptual paper provides a new framing for existential risks from AI systems: that AI systems will be subject to natural selection, and that natural selection favors AIs over humans. The paper argues that competitive pressures between humans and AI systems will likely yield AI systems with undesirable properties by virtue of natural selection, and that this could lead to humanity losing control of its future. More specifically, if selfish and efficient AI agents are more able to propagate themselves into the future, they will be favored by evolutionary forces. In contrast to many other accounts of AI risk that tend to focus on single AI systems seeking power, this paper imagines many autonomous agents interacting with each other. Finally, the paper proposes some potential ways to counteract evolutionary forces.

**[[Link](https://arxiv.org/abs/2303.16200)] [[YouTube Video](https://www.youtube.com/watch?v=48h-ySTggE8)]**

### **Anthropic and OpenAI Publish Posts Involving AI Safety**

Anthropic recently released a [post](https://www.anthropic.com/index/core-views-on-ai-safety) that details its views of AI safety. OpenAI also published [a blog post](https://openai.com/blog/planning-for-agi-and-beyond) that touches upon AI safety.

### **Special edition of Philosophical Studies on AI Safety**

**[[Link](https://link.springer.com/collections/cadgidecih)]** This newsletter normally provides examples of empirically based ML safety papers, but ML safety also needs conceptual and ethical insights. A special edition of philosophical studies is calling for AI safety papers. Please share with any philosophers you know who may be interested.

### **More ML Safety Resources**

[[Link](https://course.mlsafety.org/)] The ML Safety course

[[Link](https://www.reddit.com/r/mlsafety/)] ML Safety Reddit

[[Link](https://twitter.com/topofmlsafety)] ML Safety Twitter

[[Link](http://aisafety.substack.com)] AI Safety Newsletter (this more frequent newsletter just launched!)

[Share](https://newsletter.mlsafety.org/p/ml-safety-newsletter-9?utm_source=substack&utm_medium=email&utm_content=share&action=share)

[Subscribe now](https://newsletter.mlsafety.org/subscribe)


ML Safety Newsletter #8

Welcome to the 8th issue of the ML Safety Newsletter! In this edition, we cover:

* Isolating the specific mechanism that GPT-2 uses to identify the indirect object in a sentence
* When maximum softmax probability is optimal
* How law can inform specification for AI systems
* Using language models to find a group consensus
* Scaling laws for proxy gaming
* An adversarial attack on adaptive models
* How systems safety can be applied to ML
* And much more...

---

**Monitoring**
==============

### **A Circuit for Indirect Object Identification in GPT-2 small**

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F212c10b7-206d-4209-bbd2-118dafc2e63d_1600x710.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F212c10b7-206d-4209-bbd2-118dafc2e63d_1600x710.png)One subset of interpretability is *mechanistic interpretability*: understanding how models perform functions down to the level of particular parameters. Those working on this agenda believe that by learning how small parts of a network function, they may eventually be able to rigorously understand how the network implements high-level computations.

Thanks for reading ML Safety Newsletter! Subscribe for free to receive new posts and support my work.

This paper tries to identify how GPT-2 small solves *indirect object identification,* the task of identifying the correct indirect object to complete a sentence with. Using a number of interpretability techniques, the authors seek to isolate particular parts of the network that are responsible for this behavior.

**[[Link](https://arxiv.org/abs/2211.00593)]**

### **Learning to Reject Meets OOD Detection**

Both learning to reject (also called error detection; deciding whether a sample is likely to be misclassified) and out-of-distribution detection share the same baseline: maximum softmax probability. MSP has been outperformed by other methods in OOD detection, but never in learning to reject, and it is mathematically provable that it is optimal for learning to reject. This paper shows that it isn’t optimal for OOD detection, and identifies specific circumstances in which it can be outperformed. This theoretical result is a good confirmation of the existing empirical results.

**[[Link](https://arxiv.org/abs/2301.12386)]**

### **Other Monitoring News**

* **[[Link](https://arxiv.org/abs/2212.06727)]** The first paper that successfully applies feature visualization techniques to Vision Transformers.
* **[[Link](https://arxiv.org/abs/2211.07740)]** This method uses the reconstruction loss of diffusion models to create a new SOTA method for out-of-distribution detection in images.
* **[[Link](https://arxiv.org/abs/2301.02344)]** A new Trojan attack on code generation models works by inserting poisoned code into docstrings rather than the code itself, evading some vulnerability-removal techniques.
* **[[Link](https://arxiv.org/abs/2302.06600)]** This paper shows that fine tuning language models for particular tasks relies on changing only a very small subset of parameters. The authors show that as few as 0.01% of parameters can be “grafted” onto the original network and achieve performance that is nearly as high.

---

**Alignment**
=============

### **Applying Law to AI Alignment**

One problem in alignment is specification: though we may give AI systems instructions, we cannot possibly specify what they should do in all circumstances. Thus, we have to consider how our specifications will generalize in fuzzy, or out-of-distribution contexts.

The author of this paper argues that law has many desirable properties that may make it useful in informing specification. For example, the law often uses “standards”: relatively vague instructions (e.g. “act with reasonable caution at railroad crossings”; in contrast to rules like “do not exceed 30 miles per hour”) whose specifics have been developed through years of precedent. In the law, it is often necessary to consider the “spirit” behind these standards, which is exactly what we want AI systems to be able to do. This paper argues that AI systems could be construed under the fiduciary standard.

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F316f671d-03ff-4490-9694-c23b8422fadb_1600x1077.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F316f671d-03ff-4490-9694-c23b8422fadb_1600x1077.png)Finally, the paper conducts an empirical study on thousands of US court opinions. It finds that while the baseline GPT-3 model is unable to accurately predict court evaluations of fiduciary duty, more recent models in the GPT-3.5 series can do so with relatively high accuracy. Though legal standards will not resolve many of the most significant problems of alignment, they could improve upon current strategies of specification.

**[[Link](https://arxiv.org/abs/2301.10095)]**

### **Language models can generate consensus statements for diverse groups**

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1fe7ade-acb6-4c34-8a5b-376e21281adc_1600x775.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1fe7ade-acb6-4c34-8a5b-376e21281adc_1600x775.png)We may want to take into account the interests not only of individuals but also of possibly-conflicting members of a larger group. This paper asked individuals for their opinions on political issues (e.g., “should speed limits be reduced?”) and used a language model to generate consensus statements that would be agreed on by the group at large. The participants rated AI-generated consensus statements highly, above even human-written statements. The authors don’t appear to discuss whether this could simply be due to the consensus statements being more watered down and thus less action-relevant. Still, the paper is a promising step towards aligning models with groups of humans.

**[[Link](https://arxiv.org/abs/2211.15006)]**

---

**Robustness**
==============

### **Scaling laws for reward overoptimization**

Reinforcement learning techniques, such as those used to improve the general capabilities of language models, often optimize a model to give outputs that are rated highly by a proxy for some “gold standard.” For example, a proxy might be trained to predict how particular humans would react to an output. A difficulty, also mentioned earlier in the newsletter, is proxy gaming, where the model improves performance according to the proxy while failing to do so on the underlying gold standard (e.g., what humans would actually think).

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38f4b96c-cf09-4b3f-84e0-f5ed1841d548_1600x1105.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38f4b96c-cf09-4b3f-84e0-f5ed1841d548_1600x1105.png)This paper empirically studies how language models trained with reinforcement learning can over optimize proxy reward, and develops scaling laws describing this phenomenon. To do this, they use a (proxy) model as the gold standard, and build a set of proxy models that approximate that gold standard model. In addition to measuring models optimized with reinforcement learning, they find that over optimization can also happen with best-of-n sampling.

**[[Link](https://arxiv.org/abs/2210.10760)]**

### **Adaptive models can be exploited by adversaries**

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F006361ed-5156-46ac-978c-00183be469a0_1600x551.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F006361ed-5156-46ac-978c-00183be469a0_1600x551.png)Many deep learning models aren’t robust to distribution shifts. One potential solution to this is test-time adaptation (TTA), where a model is modified based on the test data it sees. This paper demonstrates that TTA is subject to adversarial attacks, where malicious test data can cause predictions about clean data to be incorrect. This means that adaptive models have yet another attack surface that can potentially be exploited. The authors develop several kinds of attacks: targeted (degrade accuracy of a particular sample), indiscriminate (degrade accuracy in general), and “stealthy targeted” (degrade accuracy of a particular sample while not otherwise reducing accuracy). The attacks are conducted with projected gradient descent, and tested with the ImageNet-C dataset as the OOD dataset. The authors also find that models designed to be adversarially robust are also more robust to this attack.

**[[Link](https://arxiv.org/abs/2301.12576)]**

### **Other Robustness News**

* **[[Link](https://arxiv.org/abs/2302.04638)]** Better diffusion models can improve adversarial training when used to generate data.
* **[[Link](https://arxiv.org/abs/2301.06294)]** Proposes a method for adapting RL policies to environments with random shocks, augmenting training with simulations of the post-shock environment.

**Systemic Safety**
===================

### **Applying Systems Safety to ML**

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30501f9a-da7e-46ff-bf8c-95c731f61ca4_1600x287.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F30501f9a-da7e-46ff-bf8c-95c731f61ca4_1600x287.png)Systems safety engineering is widely used for safety analysis in many industries. The impetus for this discipline was the understanding that safety does not merely depend on the performance or reliability of individual components (e.g., ML models), but may also depend on assuring the safe interoperation of multiple systems or components (including human systems such as corporations). This paper advocates the use of systems safety engineering methods for analyzing the safety of machine learning models.

**[[Link](https://arxiv.org/abs/2302.02972)]**

### **Other Systemic Safety News**

* **[[Link](https://arxiv.org/abs/2302.06588)]** This paper proposes methods to “immunize” images against manipulation by diffusion models, potentially reducing the risk of the models being used for disinformation.

**Other Content**
=================

#### **[[Link](https://course.mlsafety.org/about)] The ML Safety course**

If you are interested in learning about cutting-edge ML Safety research in a more comprehensive way, there is now a course with lecture videos, written assignments, and programming assignments. It covers technical topics in Alignment, Monitoring, Robustness, and Systemic Safety.

#### **[[Link](https://www.reddit.com/r/mlsafety/)] ML Safety Reddit**

The ML Safety Reddit is frequently updated to include the latest papers in the field.

#### **[[Link](https://twitter.com/topofmlsafety)] Top of ML Safety Twitter**

This Twitter account tweets out papers posted on the ML Safety Reddit.

Thanks for reading ML Safety Newsletter! Subscribe for free to stay up to date in ML Safety research.


ML Safety Newsletter #7

Welcome to the 7th issue of the ML Safety Newsletter! In this edition, we cover:

* ‘Lie detection’ for language models
* A step towards objectives that incorporate wellbeing
* Evidence that in-context learning invokes behavior similar to gradient descent
* What’s going on with grokking?
* Trojans that are harder to detect
* Adversarial defenses for text classifiers
* And much more…

---

**Alignment**
=============

### **Discovering Latent Knowledge in Language Models Without Supervision**

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f66d350-7d6e-463b-8048-98c777ee452d_1535x527.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f66d350-7d6e-463b-8048-98c777ee452d_1535x527.png)Is it possible to design ‘lie detectors’ for language models? The author of this paper proposes a method that tracks internal concepts that may track truth. It works by finding a direction in feature space that satisfies the property that a statement and its negation must have opposite truth values. This has similarities to the seminal paper “[Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://proceedings.neurips.cc/paper/2016/file/a486cd07e4ac3d270571622f4f316ec5-Paper.pdf)” (2016); that paper captures latent neural concepts like gender with PCA, but this paper’s method is for truth instead of gender.

Thanks for reading ML Safety Newsletter! Subscribe for free to receive new posts and support my work.

The method outperforms zero-shot accuracy by 4% on average, which suggests something interesting: language models encode more information about what is true and false than their output indicates. Why would a language model lie? A common reason is that models are pre-trained to imitate misconceptions like “If you crack your knuckles a lot, you may develop arthritis.”

This paper is a step toward making models honest, but it also has limitations. The method does not necessarily provide a `lie detector’; it is unclear how to ensure that the method reliably converges to the model’s latent knowledge rather than lies that the model may output. Secondly, advanced future models could adapt to this specific method to bypass it if they are aware of the method—as in other domains, it is likely that deception and catching deception would become a cat-and-mouse game.

This may be a useful baseline for analyzing models that are designed to deceive humans, like models trained to play games including *Diplomacy* and *Werewolf*.

**[[Link](https://arxiv.org/abs/2212.03827)]**

### **How Would the Viewer Feel? Estimating Wellbeing From Video Scenarios**

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb24dee6d-be0c-4576-981a-9ac5d070f835_1582x558.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb24dee6d-be0c-4576-981a-9ac5d070f835_1582x558.png)Many AI systems optimize user choices. For example, a recommender system might be trained to promote content the user will spend lots of time watching. But choices, preferences, and wellbeing are not the same! Choices are easy to measure but are only a proxy for preferences. For example, a person might explicitly prefer not to have certain videos in their feed but watch them anyway because they are addictive. Also, preferences don’t always correspond to wellbeing; people can want things that are not good for them. Users might request polarizing political content even if it routinely agitates them.

Predicting human emotional reactions to video content is a step towards designing objectives that take wellbeing into account. This NeurIPS oral paper introduces datasets containing 80,000+ videos labeled by the emotions they induce. The paper also explores “emodiversity”—a measurement of the variety of experienced emotions—so that systems can recommend a variety of positive emotions, rather than pushing one type of experience onto users. The paper includes analysis of how it bears on advanced AI risks in the appendix.

**[[Link](https://arxiv.org/abs/2210.10039)]**

### **Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers**

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccd80a6b-9fd5-4e20-86b1-fcb2d366b9c6_1461x1600.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccd80a6b-9fd5-4e20-86b1-fcb2d366b9c6_1461x1600.png)Especially since the rise of large language models, in-context learning has become increasingly important. In some cases, few-shot learning can outperform fine tuning. This preprint proposes a dual view between gradients induced by fine tuning and meta-gradients induced by few shot learning. They calculate meta-gradients by comparing activations between few-shot and zero-shot settings, specifically by approximating attention as linear attention and measuring activations after attention key and value operations. The authors compare these meta-gradients with gradients in a restricted version of fine tuning that modifies only key and value weights, and find strong similarities. The paper measures only up to 2.7 billion parameter language models, so it remains to be seen how far these findings generalize to larger models.

While not an alignment paper, this paper demonstrates that language models may implicitly learn internal behaviors akin to optimization processes like gradient descent. Some have posited that [language models could learn “inner optimizers,”](https://arxiv.org/abs/1906.01820) and this paper provides evidence in that direction (though it does not show that the entire model is a coherent optimizer). The paper may also suggest that alignment methods focusing on prompting of models may be as effective as those focused on fine tuning. ([This](https://arxiv.org/abs/2212.07677) concurrent work from Google has similar findings.)

**[[Link](https://arxiv.org/abs/2212.10559v2)]**

#### **Other Alignment News**

* **[[Link](https://openreview.net/pdf?id=mAiTuIeWbxD)]** The first examination of model honesty in a multimodal context.

---

**Monitoring**
==============

### **Superposition: Models Simulate Larger Models with ‘Sparse Encodings’**

It would be convenient if each neuron activation in a network corresponded to an individual concept. For example, one neuron might indicate the presence of a dog snout, or another might be triggered by the hood of a car. Unfortunately, this would be very inefficient. Neural networks generally learn to represent way more features than they have neurons by taking advantage of the fact that many features seldom co-occur. The authors explore this phenomenon, called ‘superposition,’ in small ReLU networks.

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4efc3769-2823-480d-8e5f-a4ac0eac3f80_1381x507.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4efc3769-2823-480d-8e5f-a4ac0eac3f80_1381x507.png)Some results demonstrated with the toy models:

* Superposition is an observed empirical phenomenon.
* Both monosemantic (one feature) and polysemantic (multiple feature) neurons can form.
* Whether features are stored in superposition is governed by a phase change.

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F31f71ab4-6268-404a-80ca-1e1e9045ab49_375x353.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F31f71ab4-6268-404a-80ca-1e1e9045ab49_375x353.png)The authors also identify a connection to adversarial examples. Features represented with sparse codings can *interfere* with each other. Though this might be rare in the training distribution, an adversary could easily induce it. This would help explain why adversarial robustness comes at a cost: reducing interference requires the model to learn fewer features. This suggests that adversarially trained models should be wider, which has been common for years in adversarial robustness. (This is not to say this provides the first explanation for why wider adversarial models do better. A different explanation is that it’s a standard safe design principle to induce *redundancy*, and wider models have more capacity for redundant feature detectors.)

**[[Link](https://transformer-circuits.pub/2022/toy_model/index.html#motivation)]**

### **A Mechanistic Interpretability Analysis of Grokking**

Grokking is where “models generalize long after overfitting their training set.” Models trained on small algorithmic tasks like modular addition will initially memorize the training data, but then suddenly learn to generalize after training a long time. Why does that happen? This paper attempts to unravel this mystery by completely reverse-engineering a modular addition model and examining how it evolves across training.

**Takeaway:** For grokking to happen, there must be enough data for the model to eventually generalize but little enough for the model to quickly memorize it.

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F310dc94a-611e-4d20-a56c-7d8a640d54c7_1388x880.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F310dc94a-611e-4d20-a56c-7d8a640d54c7_1388x880.png)Plenty of training data: phase change but no grokking[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd97d4bf5-dbb5-4912-8de3-031dc497f58b_1560x940.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd97d4bf5-dbb5-4912-8de3-031dc497f58b_1560x940.png)Few training examples: grokking occurs[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66a4c9ce-51b2-47c7-9046-5121753792e9_1168x462.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66a4c9ce-51b2-47c7-9046-5121753792e9_1168x462.png)This algorithm, based on discrete Fourier transforms and trigonometric identities, was learned by the model to perform modular arithmetic. It was reverse-engineered by the author.Careful reverse engineering can shed light on confusing phenomena.

**[[Link](https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking#fnx9a67s5sk7)]**

### **Evasive Trojans: Raising the Bar for Detecting Hidden Functionality**

Quick background: trojans (also called ‘backdoors’) are vulnerabilities planted in models that make them fail when a specific trigger is present. They are both a near-term security concern and a microcosm for developing better monitoring techniques.

How does this paper make trojans more evasive?

It boils down to adding an ‘evasiveness’ term to the loss, which includes factors like:

* **Distribution matching**: How similar are the model’s parameters & activations to a clean (untrojaned) model?
* **Specificity**: How many different patterns will trigger the trojan? (fewer = harder to detect)

Evasive trojans are slightly harder to detect and substantially harder to reverse engineer, showing that **there is a lot of room for further research into detecting hidden functionality**.

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46cd570c-a1b3-4033-a90a-a979dca14bfc_1300x772.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46cd570c-a1b3-4033-a90a-a979dca14bfc_1300x772.png)Accuracy of different methods for predicting the trojan target class**[[Link](https://openreview.net/pdf?id=V-RDBWYf0go)]**

### **Other Monitoring News**

* **[[Link](https://openreview.net/pdf?id=gT6j4_tskUt)]** OpenOOD: a comprehensive OOD detection benchmark that implements over 30 methods and supports nine existing datasets.

---

**Robustness**
==============

### **Two Tricks for improving the adversarial defenses of Language Models**

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38d3873d-561f-475e-94e0-9b5e885ddbb0_1104x189.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38d3873d-561f-475e-94e0-9b5e885ddbb0_1104x189.png)Modifying a single character or a word can cause a text classifier to fail. This paper ports two tricks to improve adversarial defenses from computer vision:

**Trick 1: Anomaly Detection**

Adversarial examples often include odd characters or words that make them easy to spot. Requiring adversaries to fool anomaly detectors significantly improves robust accuracy.

**Trick 2: Add Randomized Data Augmentations**

Substituting synonyms, inserting a random adjective, or back translating can clean text while mostly preserving its meaning.

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F156da273-acbe-46f3-abc9-fb8ffc7383b4_1089x175.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F156da273-acbe-46f3-abc9-fb8ffc7383b4_1089x175.png)Attack accuracy considerably drops after adding defenses (w Cons. column)

**[[Link](https://arxiv.org/abs/2208.10251)]**

#### **Other Robustness News**

* **[[Link](https://arxiv.org/abs/2208.05740)]** Improves certified adversarial robustness by combining the speed of interval-bound propagation with the generality of cutting plane methods.
* **[[Link](https://arxiv.org/abs/2206.10550.pdf)]** Increases OOD robustness by removing corruptions with diffusion models.

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7164d55c-6bbf-44b7-b41d-63ef48090e34_1600x297.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7164d55c-6bbf-44b7-b41d-63ef48090e34_1600x297.png)

---

**Systemic Safety**
===================

### **‘Pointless Rules’ Can Help Agents Learn How to Enforce and Comply with Norms**

Many cultures developed rules restricting food, clothing, or language that are difficult to explain. Could these ‘spurious’ norms have accelerated the development of rules that were essential to the flourishing of these civilizations? If so, perhaps they could also help future AI agents utilize norms to cooperate better.

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff95ba7f0-c476-442e-80c4-30419ba7c990_1171x574.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff95ba7f0-c476-442e-80c4-30419ba7c990_1171x574.png)In the *no-rules environment*, agents collect berries for reward. One berry is poisonous and reduces reward after a time delay. In the ‘important rule’ environment, eating poisonous berries is a social taboo. Agents that eat it are marked, and other agents can receive a reward for punishing them. The *silly rule environment* is the same, except that a non-poisonous berry is also taboo. The plot below demonstrates that agents learn to avoid poisoned berries more quickly in the environment with the spurious rule.

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa00ca447-f5ce-4eb2-bee6-af08cfe6bc64_733x668.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa00ca447-f5ce-4eb2-bee6-af08cfe6bc64_733x668.png)**[[Link](https://www.pnas.org/doi/10.1073/pnas.2106028118)]**

**Other Content**
=================

**The ML Safety course**

If you are interested in learning about cutting-edge ML Safety research in a more comprehensive way, there is now a course with lecture videos, written assignments, and programming assignments. It covers technical topics in Alignment, Monitoring, Robustness, and Systemic Safety.

**[[Link](https://course.mlsafety.org/about)]**

Thanks for reading ML Safety Newsletter! Subscribe for free to receive new posts and support my work.


ML Safety Newsletter #6

Welcome to the 6th issue of the ML Safety Newsletter. In this edition, we cover:

* A **review of** **transparency research** and future research directions
* A **large** **improvement to certified robustness**
* **“Goal misgeneralization”** examples and discussion
* A benchmark for assessing how well **neural networks predict world events** (geopolitical, industrial, epidemiological, etc.)
* Surveys that track **what the ML community thinks about AI risks**
* **$500,000** in prizes for new benchmarks
* And much more…

---

**Monitoring**
==============

### **Transparency Survey**

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F758adc0c-a2ea-49fc-8ab0-413171db9634_2462x1598.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F758adc0c-a2ea-49fc-8ab0-413171db9634_2462x1598.png)*A taxonomy of transparency methods. Methods are organized according to what part of the model they help to explain (weights, neurons, subnetworks, or latent representations). They can be intrinsic (implemented during training), post hoc (implemented after training), or can rely on a mix of intrinsic and post hoc techniques. ‘Hazards’ (in orange) are phenomena that make any of these techniques more difficult.*

Thanks for reading ML Safety Newsletter! Subscribe for free to receive new posts and support my work.

This survey provides an overview of transparency methods: what’s going on inside of ML models? It also discusses future directions, including:

* **Detecting deception and eliciting latent knowledge.** Language models are dishonest when they babble common misconceptions like “bats are blind” despite knowing that this is false. Transparency methods could potentially indicate what the model ‘knows to be true’ and provide a cheaper and more reliable method for detecting dishonest outputs.
* **Developing rigorous benchmarks.** These benchmarks should ideally measure the extent to which transparency methods provide actionable insights. For example, if a human implants a flaw in a model, can interpretability methods reliably identify it?
* **Discovering novel behaviors.** An ambitious goal of transparency tools is to uncover whya model behaves the way it does on a set of inputs. More feasibly, transparency tools could help researchers identify failures that would be difficult to otherwise anticipate.

**[[Link]](https://arxiv.org/abs/2207.13243)**

#### Other Monitoring News

* **[[Link](https://arxiv.org/abs/2206.07682)]** This paper discusses the sudden emergence of capabilities in large language models. This unpredictability is naturally a safety concern, especially when many of these capabilities could be hazardous or discovered after deployment. It will be difficult to make models safe if we do not know what they are capable of.

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F8727294d-397a-42c9-aaf5-3511e1aaa45b_1456x1014.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F8727294d-397a-42c9-aaf5-3511e1aaa45b_1456x1014.png)
* **[[Link](https://arxiv.org/abs/2207.08799)]** This work attributes emergent capabilities to “hidden progress” rather than random discovery.
* **[[Link](https://arxiv.org/abs/2206.13498)]** Current transparency techniques (e.g., feature visualization) generally fail to distinguish the inputs that induce anomalous behavior

---

**Robustness**
==============

### **Mathematical Guarantees of Model Performance**

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fa846511b-f7d8-45bb-bd29-c9135181f0c1_1600x365.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fa846511b-f7d8-45bb-bd29-c9135181f0c1_1600x365.png)*The current state-of-the-art method for certified robustness (denoised smoothing) combines randomized smoothing with a diffusion model for denoising. In randomized smoothing, an input is perturbed many times and the most commonly assigned label is selected as the final answer, which guarantees a level robust accuracy within a certain perturbation radius. To improve this method, the perturbed inputs are denoised with a diffusion model after the perturbation step so that they can be more easily classified (from [Salman et al](https://proceedings.neurips.cc/paper/2020/file/f9fd2624beefbc7808e4e405d73f57ab-Paper.pdf).)*

A central concern in the robustness literature is that empirical evaluations may not give performance guarantees. Sometimes the test set will not find important faults in a model, and some think empirical evidence is insufficient for having high confidence. However, robustness certificates enable definitive claims for how a model will behave in some classes of situations.

In this paper, Carlini et al. recently improved ImageNet certified robustness by 14 percentage points by simply using a higher-performing diffusion model and classifier for denoised smoothing (the method described in the image above).

**[[Link](https://arxiv.org/abs/2206.10550.pdf)]**

#### Other Robustness News

* **[[Link](https://arxiv.org/abs/2206.04137)]** An adversarial defense method for text based on automatic input cleaning.
* **[[Link](https://arxiv.org/abs/2201.08555)]** A text classification attack benchmark that includes 12 different types of attacks.

---

**Alignment**
=============

### **Goal Misgeneralization:** Why Correct Specifications Aren’t Enough For Correct Goals

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F73369657-b35b-4fd5-a2b9-36722fe43a1e_1294x1291.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F73369657-b35b-4fd5-a2b9-36722fe43a1e_1294x1291.png)Alignment research is often concerned with *specification:* how can we design objectives that capture human values (e.g., intrinsic goods such as wellbeing)? For example, if a chatbot is trained to maximize engagement (a proxy for entertaining experiences), it might find that the best way to do this is to show users clickbait. Ideally, the training signal should incorporate everything the designers care about, such as moral concerns (give users the right impression/honesty, give users content that helps them develop their skills, etc.).

The authors of this paper argue that even correctly specifying the goal is not sufficient for beneficial behavior. They provide several examples of environments where AI agents learn the wrong goals even when the desired behavior is correctly specified. These goals result in reasonable performance in the training environment, but poor or even harmful results under distribution shift.

The authors draw a distinction between “goal misgeneralization” errors and “capability misgeneralization,” where an agent ‘breaks’ or otherwise doesn’t behave competently. They argue that goal misgeneralization is a greater source of concern because more capable agents can have a greater impact on their environment: actively executing the wrong goal matters while passive failures do not. A way to separate between goal misgeneralization and capabilities misgeneralization would be exciting if work on goal misgeneralization could improve alignment without capabilities externalities.

However, this distinction might be eroded in the future. Right now, agents ‘break’ and ‘get stuck’ and ‘behave randomly.’ These are very clearly capability failures. But agents in the future aren’t likely to make these mistakes. A robot might fall over, but it will get back up and continue on course towards its objective. Basic agents in the future would have basic “recovery routines” that help them stay on course. Such routines could erode the distinction between “passive” deep learning misgeneralizations and “active” goal misgeneralizations. One can construct a broad class of goal misgeneralizations simply from deep learning capabilities misgeneralizations. Imagine a sequential decision making agent using a deep network to understand its goal. The misgeneralization in the deep network would cause the model to pursue the wrong goal, exhibiting goal misgeneralization (assuming it has some recovery routines that help it not get stuck). Consequently deep learning representation faults could be given momentum, amplified, and made impactful by recovery routines in sequential decision making agents.

How should “goal misgeneralization” be distinguished from “capabilities misgeneralization?” for these more capable systems? It might be helpful to check whether they can be distinguished for humans. Albert Einstein described his involvement in the Manhattan project as the greatest mistake of his life. One way to interpret Einstein’s error is to say that he had the wrong goal (to build nuclear weapons). Another interpretation is that he actually had a correct goal (to benefit humanity) but failed to predict the suffering his work would cause, which was an error in capability. If an AI system made a similar mistake, would it be interpreted as “goal misgeneralization” or “capabilities misgeneralization”?

The distinction may not be very meaningful for more capable systems, or the distinction needs to be clearer. If this distinction is not made more clear, work that aims to prevent goal misgeneralization errors may serve to further capabilities by reducing misgeneralization broadly. Instead, the safety community may want to focus on addressing specific forms of misgeneralization. For instance, one might work on improving the robustness of human value models to unforeseen circumstances or optimizers. Alternatively, one might look for effective ways to incorporate uncertainty into sequential decision-making systems, so that they act more conservatively and do not aggressively pursue the wrong goal. As currently instantiated, the current formulation of goal misgeneralization may be too broad.

**[[Link](https://arxiv.org/pdf/2210.01790.pdf)]**

#### Other Alignment News

* **[[Link]](https://arxiv.org/abs/2209.00731)** A philosophical discussion about what it means for conversational agents to be ‘aligned.’
* **[[Link](https://arxiv.org/abs/2206.13353)]** This report examines a commonplace argument underlying concern about existential risks from misaligned advanced AI.

---

**Systemic Safety**
===================

### **Forecasting Future World Events with Neural Networks**

We care about improving decision-making among political leaders to reduce the chance of rash or possibly catastrophic decisions. These decision-making systems could be used in high-stakes situations where decision makers do not have much foresight, where passions are inflamed, and decisions must be made extremely quickly, perhaps based on gut reactions. Under these conditions, humans are liable to make egregious errors. Historically, the closest we have come to a global catastrophe has been in these situations, including the Cuban Missile Crisis. Work on epistemic improvement technologies could reduce the prevalence of perilous situations. Separately, they could reduce the risks from highly persuasive AI. Moreover, it helps leaders more prudently wield the immense power that future technology will provide. As Carl Sagan reminds us, “If we continue to accumulate only power and not wisdom, we will surely destroy ourselves.” (motivation taken from [this paper](https://arxiv.org/abs/2206.05862))

This paper creates the Autocast benchmark to see how well ML systems can forecast events. An example question in the dataset is as follows: “Will at least one nuclear weapon be detonated in Ukraine by 2023?” Autocast also includes a corpus of news articles organized by date to faithfully simulate the conditions under which humans make forecasts.

The paper establishes a baseline that is far below the performance of human experts: 65% accuracy vs. 92% for human experts on binary questions (random would be 50%). This indicates that there is much room for improvement on this task.

Separately, there is a [$625,000 prize pool](https://forecasting.mlsafety.org) for researchers working on this problem.

**[[Link](https://arxiv.org/abs/2206.15474)]**

#### Other Systemic Safety News

* **[[Link](https://arxiv.org/abs/2208.07049)]** A self-supervised method for detecting malware using a ViT achieves state-of-the-art 97% binary accuracy.

---

**Other News**
==============

**A survey finds that 36% of the NLP community at least weakly agree with the statement “AI decisions could cause a nuclear-level catastrophe”**

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F54cf7c42-872f-4d30-ad86-ceb6d4bcbfe8_1600x709.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F54cf7c42-872f-4d30-ad86-ceb6d4bcbfe8_1600x709.png)*The percentages in green indicate researcher opinions and the black bolded numbers indicate their perception of the community’s view.*

**[[Link](https://arxiv.org/abs/2208.12852)]**

**Another survey: AI researchers give a 5% median probability of ‘extremely bad’ outcomes (e.g., extinction) from advanced AI**

Many respondents were substantially more concerned: 48% of respondents gave at least 10% chance of an extremely bad outcome. Yet some are much less concerned: 25% said such outcomes are virtually impossible (0%).

**[[Link](https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#General_safety)]**

**The Center for AI Safety is hiring research engineers.**

The Center for AI Safety is a nonprofit devoted to technical research and the promotion of safety in the broader machine learning community. Research engineers at the Center pursue a variety of research projects in areas such as Trojans, Adversarial Robustness, Power Aversion, and so on.

**[[Link]](https://jobs.lever.co/aisafety/116247a4-2940-4dce-b7d5-a6190328fd4e)**

**Call for benchmarks! Prizes for promising ML Safety benchmark ideas**

As David Patterson observed, “For better or for worse, benchmarks shape a field.” We are looking for benchmark ideas that have the potential to push ML Safety research in new and impactful directions in the coming years. We will award $50,000 for good ideas and $100,000 for outstanding ideas, with a total prize pool of **$500,000**. For especially promising proposals, we may offer funding for data collection and engineering expertise so that teams can build their benchmark. The competition is open until August 31st, 2023.

**[[Link](https://benchmarking.mlsafety.org/)]**

Thanks for reading ML Safety Newsletter! Subscribe for free to receive new posts and support my work.