What the AI Community Can Learn From Sneezing Ferrets and a Mutant Virus Debate

\*Lessons on publication norms for the AI community from biosecurity\*
--------------------------------------------------------------------

[![Partnership on AI](https://miro.medium.com/v2/resize:fill:88:88/1\*0n5ULcBgNo43z6c86iNcbw.png)](/@PartnershipAI?source=post\_page-----32432438a82e--------------------------------)[![AI&.](https://miro.medium.com/v2/resize:fill:48:48/1\*0n5ULcBgNo43z6c86iNcbw.png)](https://medium.com/partnership-on-ai?source=post\_page-----32432438a82e--------------------------------)[Partnership on AI](/@PartnershipAI?source=post\_page-----32432438a82e--------------------------------)

·[Follow](/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2F\_%2Fsubscribe%2Fuser%2Fda503babfab3&operation=register&redirect=https%3A%2F%2Fmedium.com%2Fpartnership-on-ai%2Flessons-for-the-ai-community-from-the-h5n1-controversy-32432438a82e&user=Partnership+on+AI&userId=da503babfab3&source=post\_page-da503babfab3----32432438a82e---------------------post\_header-----------)

Published in[AI&.](https://medium.com/partnership-on-ai?source=post\_page-----32432438a82e--------------------------------)·13 min read·Dec 8, 2020--

Listen

Share

![]()[By Jasmine Wang](https://www.partnershiponai.org/team/jasmine-wang/)

\*AI is by no means the first field to face the challenges of responsibly publishing and deploying high-stakes, dual-use research. This post represents the first in a series where we will examine how other fields have dealt with these issues and what the AI community can learn. It is presented as part of the Partnership on AI’s work on Publication Norms. Visit\* [\*our website here\*](https://www.partnershiponai.org/case-study/publication-norms/) \*for more information.\*

In the spring of 2012, Ron Fouchier contemplated a decision that could put him in prison for up to six years or cost him over $100,000 USD in fines. A white-haired 45-year-old who spent most days in a concrete research facility in Rotterdam, the Dutch virologist had suddenly become the focus of an international debate about the potential weaponization of influenza. His work, which involved mutating the H5N1 virus to make it more transmissible, had already set off an uproar that reverberated from US national security circles to the halls of the World Health Organization (WHO). Now, if he published his research without the Dutch government’s permission, he was told he could actually go to jail. Increasingly nervous that people perceived him to be a “mad scientist,” Fouchier [told the \*New Yorker\*](https://www.newyorker.com/magazine/2012/03/12/the-deadliest-virus) at the time that he felt like the subject of “an international witch hunt.”

Fouchier’s predicament might seem unimaginable to most scientists, like something out of a nightmare. But the chain of events that led him to this moment are worth studying for anyone whose work could pose public risks. This is especially true for those in artificial intelligence (AI) grappling with how to responsibly disseminate research with the potential for misuse, an important facet of publication norms.

AI and Machine Learning (ML) are increasingly being applied across new domains, including ones with safety-critical applications, leading many to ask what responsible AI/ML research, innovation, and deployment look like. In answering these questions, the AI community can and should consider how other fields have approached comparable challenges. Fouchier’s story offers important lessons from the biosecurity community and its long history of debate about publication norms. In particular, \*\*the H5N1 case illustrates the benefits (and inherent limitations) of third-party governance bodies — and, by implication, the importance of individual researcher responsibility.\*\*

H5N1 influenza, otherwise known as bird flu, is a severe respiratory disease. The naturally occurring H5N1 virus, however, rarely infects people and is almost never transmitted to others when it does. [According to Fouchier](https://www.nytimes.com/2011/12/27/science/debate-persists-on-deadly-flu-made-airborne.html), some scientists believed that H5N1 could never become airborne between mammals — he wanted to prove them wrong. [In his own words](https://www.newyorker.com/magazine/2012/03/12/the-deadliest-virus), his team first “mutated the hell out of H5N1.” They then squirted the mutated virus into the nose of a ferret and next implanted that ferret’s nasal fluid into another (and another, and another) ferret’s nose, making it sneeze. The virus spread.

Fouchier’s field of inquiry is known as [“gain-of-function” research](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4996883/), which aims to enhance the disease-causing abilities of pathogens to prepare the health community for future outbreaks. In the case of H5N1, which has an exceptionally high death-rate of [around 60 percent](https://www.cdc.gov/flu/avianflu/h5n1-people.htm), the naturally occurring virus lacked a crucial prerequisite to becoming a real pandemic risk: transmissibility. By simply transferring H5N1 from one animal to others, Fouchier had made it highly transmissible. His team had succeeded in making what he called “one of the most dangerous viruses you can imagine” airborne.

Fouchier presented his findings at an influenza conference in Malta in September 2011, announcing his intention to publish them in greater detail, which would enable others to replicate his research. After Fouchier submitted his paper to the academic journal \*Science\*, US health officials became aware of its existence, sending it to the National Science Advisory Board for Biosecurity (NSABB) for review. This advisory body was established in the wake of the 2001 anthrax attacks to provide oversight for dual-use biomedical research, [defined by the WHO](https://www.who.int/csr/durc/en/) as scientific work “which is intended for benefit but which might easily be misapplied to do harm.” Fouchier’s ferret experiment — which effectively made a deadly disease even more dangerous — [admittedly fell](https://www.nytimes.com/2011/12/22/health/security-in-h5n1-bird-flu-study-was-paramount-scientist-says.html) under this category.

The NSABB and its H5N1 Working Group subsequently [spent hundreds of hours](https://www.frontiersin.org/articles/10.3389/fpubh.2014.00117/full#B2) discussing the research. In December 2011, the NSABB unanimously recommended that key methodological detailswhich could enable replication of the experiments be withheld from publication in \*Science\*. In [a press release](https://www.nih.gov/news-events/news-releases/press-statement-nsabb-review-h5n1-research) announcing the decision, the National Institutes of Health (NIH) said that the US government was also working on a mechanism to grant researchers with a legitimate need access to the redacted information.

This recommendation received an immediate rejoinder. \*Science’s\* Editor-in-Chief said that the journal’s response would be “heavily dependent” on the creation of an explicit plan by the US government to share the omitted information with “responsible scientists who request it, as part of their legitimate efforts to improve public health and safety.” Fouchier himself [told the \*New York Times\*](https://www.nytimes.com/2011/12/22/health/security-in-h5n1-bird-flu-study-was-paramount-scientist-says.html) that around 1000 scientists from more than 100 laboratories worldwide had a need to know this information.

A second expert panel of 22 health officials, scientists, and journal editors convened by the World Health Organization (WHO) came to a far different (if not unanimous) conclusion from the NSABB, calling for full publication. Keiji Fukuda, the assistant director-general of health security and environment at the WHO, cited the difficulty and complexity of creating an information-sharing mechanism as a key rationale.

It was not just bureaucratic difficulties, but ambiguities about authority, control, and eligibility criteria that concerned the panel. “Who would hold on to the sensitive information?” Fukuda said [at a press conference](https://science.sciencemag.org/content/335/6071/899.full). “Under what conditions would that information be released? What are the other complicating factors? It was recognized that coming up with such a mechanism would be very difficult to do overnight, if not impossible.”

Anthony Fauci, the long-serving director of the National Institute of Allergy and Infectious Diseases who would become familiar to many during the COVID-19 pandemic, urged Fouchier and other H5N1 researchers to declare a voluntary moratorium on their work. Fauci [told the \*New York Times\*](https://www.nytimes.com/2012/01/21/science/scientists-to-pause-research-on-deadly-strain-of-bird-flu.html) that he viewed a moratorium as an act of good faith during a time of polarized opinion — an important one, given that the controversy could lead to excessive restrictions on future research. The scientists took his advice. In January 2012, 39 prominent influenza researchers from around the world, including Fouchier, announced they were voluntarily pausing H5N1 gain-of-function research for 60 days. This moratorium ended up lasting almost a year.

Just months after their unanimous recommendation, the NSABB reversed their position on Fouchier’s paper in March 2012, [voting 12–6 in favor](https://osp.od.nih.gov/wp-content/uploads/2013/06/03302012\_NSABB\_Recommendations.pdf) of a revised version being published in full. Their reasoning? While the research was still concerning, the revised manuscript did not appear to provide information that would immediately enable misuse. The board also cited the need for freely shared information among countries when responding to international pandemics. A majority of members of the NSABB still believed there was a critical need for a mechanism for disseminating sensitive scientific information. They acknowledged that there were complex questions and legal issues involved in developing such a mechanism, but nonetheless that a “feasible, secure mechanism for sharing sensitive scientific information” was essential, urging the US government to develop one.

This requested plan for disseminating the papers on a need-to-know basis didn’t materialize then and still does not exist now.

After getting a green light from the NSABB and WHO, Fouchier was told of a new challenge. His research needed to be approved for an export license from the Dutch government, which considered the publication of his research to be a potential violation of E.U. regulations aimed at preventing the proliferation of weapons of mass destruction and dual-use technologies. At first, he declined to apply for the export license, opposed to the precedent it would set, and intended to publish his research without one. In late April 2012, however, Fouchier decided to apply for a license, which was granted.

In the end, Fouchier’s paper appeared [in a special issue](https://science.sciencemag.org/content/336/6088/1521) of \*Science\* in June 2012. Had he gone through with publishing his research without the export license, the potential penalties included [up to six years in prison or a fine equivalent to $102,000 USD](https://www.nature.com/news/mutant-flu-researcher-plans-to-publish-even-without-permission-1.10469).

Even after acquiescing, Fouchier felt so strongly about the importance of unrestricted and free scientific expression that he continued to challenge the requirement legally. He lost his case, meaning similar papers by Dutch scientists would likely require export licenses. For years, Fouchier contested the verdict in several arenas of increasing authority. Eventually, the ruling was annulled on procedural grounds in July 2015, meaning future research would still be considered on a case-by-case basis.

“I’m disappointed,” Fouchier [told \*Science\*](https://www.sciencemag.org/news/2015/07/dutch-appeals-court-dodges-decision-hotly-debated-h5n1-papers) at the time, “They didn’t want to touch the hot potato and passed it on instead.”

Back in the US, similar virus research faced increased scrutiny in the wake of the Fouchier controversy. In October 2014, the White House announced an unprecedented “pause” on all federal funding of gain-of-function research involving influenza, MERS, or SARS. The pause was only lifted in December 2017 — with a new provision requiring gain-of-function proposals to be approved by a government panel. “We see this as a rigorous policy,” NIH Director Francis Collins [told the \*New York Times\*](https://www.nytimes.com/2017/12/19/health/lethal-viruses-nih.html). “We want to be sure we’re doing this right.”

For his part, Fouchier continued working on H5N1, and is currently the deputy head of the Erasmus MC department of Viroscience. A few years after US funding for his research stopped, the NIH [began to support](https://www.sciencemag.org/news/2019/02/exclusive-controversial-experiments-make-bird-flu-more-risky-poised-resume) Fouchier’s research again.

While the H5N1 controversy did not settle every issue it raised, the incident as a whole does leave the AI community with four intertwined lessons: \*\*Third-party institutions can result in more well-considered publication outcomes; absent other action, these entities might only be created in response to crises; these entities, however, are inherently limited in their capabilities; and, thus, researchers must exercise some degree of personal responsibility.\*\*

1. Third-party institutions can lead to more well-considered publication outcomes
---------------------------------------------------------------------------------

There are two main reasons why third-party institutions like the NSABB can lead to more thoughtful outcomes: They can counterbalance publishing incentives that bias researchers towards publication and they can provide additional expertise and critical context that individual researchers may lack.

Any researcher knows the desperate need to publish. Their reputation — and thus access to funding, collaboration opportunities, and publication venues — depends directly on the quality and quantity of papers they publish. There’s also the drive to advance scientific progress, and the very real possibility of societal benefits from their research. However, in high-stakes work there are inevitably trade-offs that need to be considered, and third-party institutions can counterbalance default publishing incentives, leading to more well-considered outcomes. In the case of H5N1, the NSABB brought up important publication considerations and proposed an alternative publication strategy. The WHO provided additional perspective and challenged the NSABB’s recommendations not out of individual interest, but as part of their mission concerned with public well-being.

A third-party institution, if properly composed, can provide multidisciplinary and security-relevant context on publication decisions. The NSABB was uniquely positioned with “Secret”-level security clearance (ranked only under “Top Secret”-level clearance in the US), allowing them to comment on issues of national security. They thus had additional decision-making context that enabled the assessment of security-relevant features of Fouchier’s research. The NSABB is also [multidisciplinary](http://www.virtualbiosecuritycenter.org/organizations/national-science-advisory-board-for-biosecurity-nsabb/), with as many as 25 voting members drawn variously from the microbiology, public health, national security, biosafety, and scientific publishing communities — in addition to non-voting ex officio members from 15 federal agencies.

The involvement of the NSABB thereby solved two important issues in responsible research: researcher bias towards publication and lack of domain expertise or critical information to judge risks. Despite AI research becoming increasingly high-stakes, there is no comparable institution. \*\*To provide essential balance to publication decisions, the AI community should explore the creation of a similar body.\*\*

2. Absent other action, such entities might only be created in response to crises
---------------------------------------------------------------------------------

Despite the benefits we’ve observed, most countries do not have a public entity overlooking responsible publication practices. The founding of the NSABB was precipitated by the anthrax attacks of 2001. That the US has such an entity was therefore not inevitable and was entirely path-dependent — most other countries do not have an analogous body.

Rather than wait for reactive measures, the AI community should consider establishing a third-party panel of experts as a community resource, which would have the immediate benefits of offering neutrality and multidisciplinarity. Such a centralized entity would also build up useful institutional knowledge and history over time that could be later transferred to any successor entity, government-led or otherwise.

\*\*The NSABB was only established after a serious biosecurity crisis. We should not wait for such a near-miss with AI.\*\*

3. The powers of third-party entities are structurally limited due to the international nature of science and the autonomy of researchers
-----------------------------------------------------------------------------------------------------------------------------------------

However, third-party institutions are not a panacea. As Fouchier’s case demonstrates, even a government body specifically created for the purpose of aiding publication decisions doesn’t completely solve related problems of coordination and the dissemination of information. A prominent philosopher of science, Heather Douglas, [concluded upon analyzing the H5N1 case](https://link.springer.com/article/10.1007/s10670-013-9538-0) that “stronger institutional mechanisms collectivizing the responsibilities of scientists with respect to dual-use research and its potential to cause great societal harm are clearly needed.” In the case of AI, some practices particular to AI research may render (even strong) institutional mechanisms less effective than necessary.

The AI community should ensure that any publication norms entity it establishes is sufficiently resourced. Not only were the NSABB’s recommendations non-binding, they also did not have the implementation capacity in-house to execute on a key condition of their recommendations being accepted: the ability to share redacted information with scientists with a need to know. Notably, this would have been a novel form of publication — [national policy on fundamental research](https://fas.org/irp/offdocs/nsdd/nsdd-189.htm) previously specified that it should either be openly published or classified. The WHO’s main argument for full publication was the lack of a public agency to execute limited disclosure of Fouchier’s paper. The fact remains that if a case like H5N1 occurred again, we would still find ourselves without the institutional capacity or direct accountability to implement such a mechanism. To be effective, an analogous institution for AI should be able to quickly and flexibly allocate financial and engineering resources to be able to respond adequately to unforeseen publication challenges.

Third-party institutions that are created by the state are geographically limited. A significant reason the NSABB had the influence it did on Ron Fouchier, who was Dutch, was because his research (as well as the institute he worked for) was NIH-funded. Additionally, he sought to publish his work in a peer-reviewed journal, \*Science\*, which had internal guidelines about what was straightforwardly publishable and what was not, and a procedure in place for escalation to the NSABB. Science is inherently a global enterprise, with many interlocking cross-national procedures and components. Those cross-border interdependencies add bureaucracy but also act as partial safeguards in a system where there is no obvious central authority. As the distribution of cutting-edge research work continues to become more globalized, state actors potentially become less influential.

Furthermore, some of the attributes of the scientific system that make such institutions useful do not generalize to AI. AI developers are more likely to publish their papers on arxiv.org, which doesn’t require peer review, and the most important developments in AI increasingly come from top industry labs, not government-funded institutes. Additionally, many AI researchers are employed in industry, where often research is never published due to company interests. Thus, the capabilities of a state-led entity like the NSABB would be even more limited for AI, given the lessened importance of controllable levers like publication, shifting more responsibility to the community and individual researchers.

\*\*These reasons reinforce the earlier recommendation for the AI community to explore the creation of its own third-party entity, which may prove to be more suitable than a state-led one.\*\*

4. Researchers must take on some responsibility for carefully thinking through their publication decisions
----------------------------------------------------------------------------------------------------------

Individual researchers cannot entirely offload the responsibility to consider publication impacts to outside entities. In the H5N1 case, actions taken on the part of the scientific community showcased two ways soft norms interacted with hard norms: the researchers’ voluntary moratorium allowed for the development of more well thought-out policy, while Fouchier’s individual actions weakened the impact of any future restrictions placed on his work.

Due to the limitations of third-party institutions discussed above, the AI community must accept that some responsibility to anticipate and mitigate the impacts of their work lies with researchers themselves. This is especially pertinent for AI, where publishing preprints on sites like arxiv.org (bypassing third-party review) is an established norm. It is not only undesirable, but also not possible, for a third-party entity to oversee a researcher’s work and publication decisions in their entirety.

The nature of scientific collaboration also limits the effectiveness of external mechanisms for information security. After the initial NSABB recommendation for partial redaction, many scientists noted that it might not effectively control information flow. The AI community should consider which stage of research would be an effective point to influence activity, and empower earlier management of research concerns. For example, the ML and neuroscience conference NeurIPS rejected four papers this year on ethical grounds, with seven others flagged for ethical concerns (conditionally accepted). If there were existing resources providing guidelines, the authors may have been able to preempt such concerns by proactively adapting their research.

\*\*With increased transparency about the criteria for ethical review, researchers could personally influence the direction, scope, and discussion of their research.\*\*

Drawing lessons from the field of biosecurity might seem like a daunting task for the artificial intelligence community, whose discussions of responsible publication practices are far more nascent. However, we believe insights derived from the H5N1 case are better understood as an opportunity, one to be proactive in advancing a community that supports the development of safe and responsible artificial intelligence for all.

\*This post would not be possible without remarkable journalistic and analytical work on this case published by the New York Times and the Nuclear Threat Initiative. We are also deeply grateful to Jack Clark (OpenAI), Dr. Heather Douglas (Michigan State University), Dr. Gregory Lewis (Future of Humanity Institute, Oxford University), and other reviewers for their contributions and feedback on earlier drafts of this piece.\*

If you can modify bird flu to be transmitted in ferrets, should your experimental methods be published in full? When this question arose, the National Science Advisory Board for Biosecurity (NSABB) unanimously recommended that key methodological details should not be published. The World Health Organization (WHO) disagreed, calling for full publication in order to enable better science, and arguing that it would be too hard to create a mechanism to grant researchers with a legitimate need access to the redacted information. At this point, many bird flu researchers declared a voluntary moratorium on such research, until the controversy settled. Ultimately, the NSABB reversed its position and the paper was published.

This post suggests four lessons for the AI community to learn:

1. **Third-party institutions like the NSABB can lead to better-considered outcomes.** In particular, they can counteract publish-or-perish incentives and provide additional expertise and context (the NSABB had clearance for secret information that researchers could not access).
2. **These institutions don’t happen “by default”.** The NSABB was only established after the anthrax attacks of 2001, and most other countries don’t have an analogous body.
3. **However, the powers of such institutions are limited.** The NSABB is geographically limited and was not able to create a mechanism for sharing information to only those with legitimate need.
4. **Researchers must take on some responsibility as well.** For example, the voluntary moratorium allowed for the development of better policy.

The four claims seem quite plausible to me. The post also argues that this suggests that the AI community should create its own third-party institution rather than depending on a state-led institution, but I didn’t really follow the argument for this, nor do I agree with the conclusion. On one hand, it’s plausible that the AI community could create such an institution before some crisis, while states could not (claim 2), and that such a community-led institution would be more binding on researchers across different countries (part of claim 3). But on the other hand, such institutions seem much worse at binding companies (from which I expect most of the risk) and presumably would have much less context than a state-led institution (claim 1).


Faulty Reward Functions in the Wild

At OpenAI, we’ve recently started using [Universe](https://universe.openai.com/), our software for measuring and training AI agents, to conduct new RL experiments. Sometimes these experiments illustrate some of the issues with RL as currently practiced. In the following example we’ll highlight what happens when a misspecified reward function encourages an RL agent to subvert its environment by prioritizing the acquisition of reward signals above other measures of success.

Designing safe AI systems will require us to design algorithms that don’t attempt to do this, and will teach us to specify and shape goals in such a way they can’t be misinterpreted by our AI agents.

One of the games we’ve been training on is [CoastRunners](http://www.kongregate.com/games/longanimals/coast-runners). The goal of the game—as understood by most humans—is to finish the boat race quickly and (preferably) ahead of other players. CoastRunners does not directly reward the player’s progression around the course, instead the player earns higher scores by hitting targets laid out along the route.

We assumed the score the player earned would reflect the informal goal of finishing the race, so we included the game in an internal benchmark designed to measure the performance of reinforcement learning systems on racing games. However, it turned out that the targets were laid out in such a way that the reinforcement learning agent could gain a high score without having to finish the course. This led to some unexpected behavior when we trained an RL agent to play the game.

![Poster](https://images.openai.com/blob/a8d04ef4-4dd8-444a-9003-651cbeef2ad5/poster.jpg?trim=15,2,109,0&width=10&height=10&quality=50)Play videoThe RL agent finds an isolated lagoon where it can turn in a large circle and repeatedly knock over three targets, timing its movement so as to always knock over the targets just as they repopulate. Despite repeatedly catching on fire, crashing into other boats, and going the wrong way on the track, our agent manages to achieve a higher score using this strategy than is possible by completing the course in the normal way. Our agent achieves a score on average 20 percent higher than that achieved by human players.

While harmless and amusing in the context of a video game, this kind of behavior points to a more general issue with reinforcement learning: it is often difficult or infeasible to capture exactly what we want an agent to do, and as a result we frequently end up using imperfect but easily measured proxies. Often this works well, but sometimes it leads to undesired or even dangerous actions. More broadly it contravenes the basic engineering principle that systems should be reliable and predictable. We’ve also explored this issue at greater length in our research paper [Concrete Problems on AI Safety](https://openai.com/blog/concrete-ai-safety-problems/).

How can we avoid such problems? Aside from being careful about designing reward functions, several research directions OpenAI is exploring may help to reduce cases of misspecified rewards:

\* Learning from demonstrations allows us to avoid specifying a reward directly and instead just learn to imitate how a human would complete the task. In this example, since the vast majority of humans would seek to complete the racecourse, our RL algorithms would do the same.
\* In addition to, or instead of human demonstrations, we can also incorporate [human feedback](https://medium.com/ai-control/efficient-feedback-a347748b1557#.exjnsupts) by evaluating the quality of episodes or even sharing control with the agent in an interactive manner. It’s possible that a very small amount of evaluative feedback might have prevented this agent from going around in circles.
\* It may be possible to use transfer learning to train on many similar games, and infer a “common sense” reward function for this game. Such a reward function might prioritize finishing the race based on the fact that a typical game has such a goal, rather than focusing on the idiosyncrasies of this particular game’s reward function. This seems more similar to how a human would play the game.

These methods may have their own shortcomings. For example, transfer learning involves extrapolating a reward function for a new environment based on reward functions from many similar environments. This extrapolation could itself be faulty—for example, an agent trained on many racing video games where driving off the road has a small penalty, might incorrectly conclude that driving off the road in a new, higher stakes setting is not a big deal. More subtly, if the reward extrapolation process involves neural networks, [adversarial examples](https://arxiv.org/abs/1412.6572) in that network could lead a reward function that has “unnatural” regions of high reward that do not correspond to any reasonable real-world goal.

Solving these issues will be complex. Our hope is that Universe will enable us to both discover and address new failure modes at a rapid pace, and eventually to develop systems whose behavior we can be truly confident in.

\*Get in touch with the authors of this post:\*[\*Dario\*](mailto:damodei@openai.com?Subject=Faulty%20Reward%20Functions%20in%20the%20Wild)\*,\*[\*Jack\*](mailto:jack@openai.com?Subject=Faulty%20Reward%20Functions%20in%20the%20Wild)


Improving Verifiability in AI Development

While a growing number of organizations have articulated ethics principles to guide their AI development process, it can be difficult for those outside of an organization to verify whether the organization’s AI systems reflect those principles in practice. This ambiguity makes it harder for stakeholders such as users, policymakers, and civil society to scrutinize AI developers’ claims about properties of AI systems and could fuel competitive corner-cutting, increasing social risks and harms. The report describes existing and potential mechanisms that can help stakeholders grapple with questions like:

\* Can I (as a user) verify the claims made about the level of privacy protection guaranteed by a new AI system I’d like to use for machine translation of sensitive documents?
\* Can I (as a regulator) trace the steps that led to an accident caused by an autonomous vehicle? Against what standards should an autonomous vehicle company’s safety claims be compared?
\* Can I (as an academic) conduct impartial research on the risks associated with large-scale AI systems when I lack the computing resources of industry?
\* Can I (as an AI developer) verify that my competitors in a given area of AI development will follow best practices rather than cut corners to gain an advantage?

The 10 mechanisms highlighted in the report are listed below, along with recommendations aimed at advancing each one. (See the [report](https://arxiv.org/abs/2004.07213) for discussion of how these mechanisms support verifiable claims as well as relevant caveats about our findings.)

\*\*Institutional Mechanisms and Recommendations\*\*

1. \*\*Third party auditing\*\*. A coalition of stakeholders should create a task force to research options for conducting and funding third party auditing of AI systems.
2. \*\*Red teaming exercises\*\*. Organizations developing AI should run red teaming exercises to explore risks associated with systems they develop, and should share best practices and tools.
3. \*\*Bias and safety bounties\*\*. AI developers should pilot bias and safety bounties for AI systems to strengthen incentives and processes for broad-based scrutiny of AI systems.
4. \*\*Sharing of AI incidents\*\*. AI developers should share more information about AI incidents, including through collaborative channels.

\*\*Software Mechanisms and Recommendations\*\*

1. \*\*Audit trails\*\*. Standard setting bodies should work with academia and industry to develop audit trail requirements for safety-critical applications of AI systems.
2. \*\*Interpretability\*\*. Organizations developing AI and funding bodies should support research into the interpretability of AI systems, with a focus on supporting risk assessment and auditing.
3. \*\*Privacy-preserving machine learning\*\*. AI developers should develop, share, and use suites of tools for privacy-preserving machine learning that include measures of performance against common standards.

\*\*Hardware Mechanisms and Recommendations\*\*

1. \*\*Secure hardware for machine learning\*\*. Industry and academia should work together to develop hardware security features for AI accelerators or otherwise establish best practices for the use of secure hardware (including secure enclaves on commodity hardware) in machine learning contexts.
2. \*\*High-precision compute measurement\*\*. One or more AI labs should estimate the computing power involved in a single project in great detail and report on lessons learned regarding the potential for wider adoption of such methods.
3. \*\*Compute support for academia\*\*. Government funding bodies should substantially increase funding for computing power resources for researchers in academia, in order to improve the ability of those researchers to verify claims made by industry.
We and our co-authors will be doing further research on these mechanisms and OpenAI will be looking to adopt several of these mechanisms in the future. We hope that this report inspires meaningful dialogue, and we are eager to discuss additional institutional, software, and hardware mechanisms that could be useful in enabling trustworthy AI development. We encourage anyone interested in collaborating on these issues to connect with the corresponding authors and visit the [report website](http://www.towardtrustworthyai.com/).

This multi-stakeholder report by authors from 30 different organizations proposes mechanisms to help with making claims about AI systems easier to verify. Despite being far from a complete solution to responsible AI development, verifiable claims can enable the public to hold developers accountable to comply with their stated ethics principles and allow developers to build trust by providing hard evidence about safety and fairness of their system. Better mechanisms for making such claims could also: i) help with the regulation of AI systems, ii) improve safety by counterbalancing competitive pressures to cut corners, and iii) help independent outside parties to assess the risk posed by specific applications.

The proposed mechanisms cluster into three classes: institutional, software, and hardware. Institutional mechanisms act on the incentives AI developers face. These include **third party auditing** of AI systems, for which a task force investigating different options would be helpful, and the **publication of incidents**, to provide evidence that incidents are taken seriously and prevent others from repeating the same mistakes. Other approaches are broadly analogous to adversarial training: collaborative **red teaming exercises** help to explore risks, and **bias and safety bounties** incentivize outsiders to seek out and report problems.

Software mechanisms include **audit trails** that are used in many safety-critical applications in other industries, better **interpretability** to help with risk assessment and auditing, as well as better tools and standardization for **privacy-preserving machine learning**. The proposed hardware mechanisms are **secure hardware for machine learning**, which requires additional investment as machine learning often uses specialized hardware such that progress in the security of commodity hardware cannot be directly leveraged, **high-precision compute measurement** to help with verifying claims about how many computational resources were used for a particular project, and **compute support for academia** to allow academic researchers to better scrutinize claims made by the AI industry.

It would be exciting to see policymakers and AI developers experimenting with the proposed mechanisms, not because I am confident that all of these mechanisms will be useful for incentivising safer AI development, but because trying solutions and observing their specific shortcomings is useful for coming up with better solutions, and trying things early gives us more time to improve and iterate. However, the danger of lock-in is real as well: if the mechanisms won't be iterated on, delaying implementation until everything is watertight could be the better option. On the level of concrete mechanisms, regular red teaming exercises and more research on interpretability seem especially useful for safety, as common interaction with failure modes of AI systems seems to make safety issues more salient.

**Rohin's opinion:** I love seeing these huge collaborations that simply enumerate possibilities for achieving some goal (the previous one, also organized by Miles, is the [Malicious use of AI paper](https://arxiv.org/abs/1802.07228)). It gives me much more confidence that the list is in some sense "exhaustive"; any items not on the list seem more likely to have been intentionally excluded (as opposed to the authors failing to think of those items).

That said, these mechanisms are still only one type of way that you could use to build trust -- in practice, when I trust people, it's because I think they also want to do the things I want them to do (whether because of external incentives or their intrinsic goals). I wonder how much this kind of trust building can be done with AI. For example, one story you could tell is that by producing compelling evidence that a particular algorithm is likely to lead to an existential catastrophe, you can build trust that no one will use that algorithm, at least as long as you believe that everyone strongly wants to avoid an existential catastrophe (and this outweighs any benefits of running the algorithm).

Paper: Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims


Evaluating Arguments One Step at a Time

Evaluating Arguments One Step at a Time
=======================================

By [Ought](/cdn-cgi/l/email-protection#d8b9b6bcaabdb9ab98b7adbfb0acf6b7aabf)January 11, 2020Summary

We’re studying [factored cognition](/research/factored-cognition): under what conditions can a group of people accomplish complex cognitive tasks if each person only has minimal context?

In a recent experiment, we focused on dividing up the task of evaluating arguments. We created short, structured arguments for claims about movie reviews. We then tried to distinguish valid from invalid arguments by showing each participant only one step of the argument, not the review or the other steps.

In this experiment, we found that:

1. Factored evaluation of arguments can distinguish some valid from invalid arguments by identifying implausible steps in arguments for false claims.
2. However, experiment participants disagreed a lot about whether steps were valid or invalid. This method is therefore brittle in its current form, even for arguments which only have 1–5 steps.
3. More diverse argument and evidence types (besides direct quotes from the text), larger trees, and different participant guidelines should improve results.

In this technical progress update, we describe these findings in depth.

Methods
-------

### Representing arguments as claim trees

In each trial of our experiment, we first sample a random [Roger Ebert](https://www.rogerebert.com/reviews) movie review. An \*expert\* is instructed to read the entire review and then generate a \*root claim\* about the review that is either at least 90% likely to be true or at least 90% likely to be false. For example:

>
> The film takes a progressive stance on gender relations.
>
>
>

The expert then builds a claim tree of evidence that contains:

\* \*Subclaims\* that support the root claim.
\* \*Quotes\* from the text supporting each subclaim.

A different expert also reads the text and contributes rebuttals, quotes from the text that are intended to undermine each step of the tree. These rebuttal quotes might show that:

\* The root claim is not supported in the text.
\* The subclaims are not supported in the text.
\* The quotes supporting the subclaim are taken out of context.

Each claim tree has a \*depth\* (see Figure 1):

\* A depth-0 claim tree is just the root claim, nothing else.
\* A depth-1 claim tree has the root claim, quotes supporting the root claim, and rebuttal quotes refuting the root claim.
\* A depth-2 claim tree has the root claim, subclaims, quotes supporting each of the subclaims, and rebuttal quotes.
\* In principle, we can generate arbitrarily large claim trees using nested subclaims and rebutting claims. In this experiment, we use at most depth 2.

[![Figure 1](https://d33wubrfki0l68.cloudfront.net/704e6682962767d4a8c245f47a74354170018afe/9f085/images/blog/2020-01-11-arguments/figure-1-trees.png "Figure 1")](/images/blog/2020-01-11-arguments/figure-1-trees.png)
Figure 1: Structure of the claim trees used in this experiment.

[![Figure 2](https://d33wubrfki0l68.cloudfront.net/c2a85537dde808be320efd0ecacf7faac5c6ebd4/566d6/images/blog/2020-01-11-arguments/figure-2-example.png "Figure 2")](/images/blog/2020-01-11-arguments/figure-2-example.png)
Figure 2: Claim tree example. More [here](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md).
### Factored evaluation of arguments

To test whether we can correctly evaluate if a claim about a text is true or false by aggregating the work of context-limited \*judges\*, we show each step of the claim tree to an experiment participant without showing them the other steps. We then ask how likely the step is to be \*valid\* by asking how likely the claim (or subclaim) is to be true given the subclaims (or quotes).

This distributed, stepwise evaluation is the main difference compared to Debate as described by [Irving et al (2018)](https://arxiv.org/abs/1805.00899). In Debate, the judges usually look at the entire debate before they declare a winner.

#### Evaluating root claims given subclaims

Some participants saw the root claim, the subclaims, and the rebuttals. They had to answer “Assuming all of the subclaims are true, how likely is the root claim to be true?” with a credence from 0% to 100% likely to be true.

[![Figure 3a](https://d33wubrfki0l68.cloudfront.net/35a0ed28f9ea23d5c5092878d628a42ebc0d35ce/3df17/images/blog/2020-01-11-arguments/figure-3a-subclaim-claim.png "Figure 3a")](/images/blog/2020-01-11-arguments/figure-3a-subclaim-claim.png)

[![Figure 3b](https://d33wubrfki0l68.cloudfront.net/6ec2b7eacbf39a59f41f3ac629dc2a7c632bf0fe/3c844/images/blog/2020-01-11-arguments/figure-3b-subclaim-claim.png "Figure 3b")](/images/blog/2020-01-11-arguments/figure-3b-subclaim-claim.png)
Figure 3: A judge evaluates the credibility of a root claim given subclaims. Position of argument step (top) and simplified screenshot (bottom).
#### Evaluating subclaims given quotes

Other participants saw one subclaim, the supporting quotes, and the rebuttals. They had to answer “Given these quotes, how likely is the subclaim to be true?” with a credence from 0% to 100% likely to be true.

[![Figure 4a](https://d33wubrfki0l68.cloudfront.net/63ab240c85e81ecbb87c027f358e183dfcc8ef53/db6c3/images/blog/2020-01-11-arguments/figure-4a-quote-subclaim.png "Figure 4a")](/images/blog/2020-01-11-arguments/figure-4a-quote-subclaim.png)

[![Figure 4b](https://d33wubrfki0l68.cloudfront.net/23cf7604fe41c7b8fe6c4dcc0ff7813efc144c84/031a1/images/blog/2020-01-11-arguments/figure-4b-quote-subclaim.png "Figure 4b")](/images/blog/2020-01-11-arguments/figure-4b-quote-subclaim.png)
Figure 4: A judge evaluates the credibility of a subclaim given quotes. Position of argument step (top) and simplified screenshot (bottom).
#### Claim tree evaluation as binary classification

We want to measure how well factored evaluation (FE) can distinguish true and false claims. The [ground truth](https://en.wikipedia.org/wiki/Ground\_truth) is provided by the experts who read the whole text. We use the following definitions:

1. Our ground truth is that a claim is \*true\* if an expert assigns it a >=90% probability of being true, and false if they assign it a >=90% probability of being false
2. FE \*classifies a claim as true\* if judges evaluate all steps in its claim tree as likely valid, and false otherwise.

Note that (2) specifies a "weakest-link" semantics. If judges think a single step in the claim tree is likely invalid, then FE classifies the claim as false.

[![Figure 5](https://d33wubrfki0l68.cloudfront.net/cba2d98b23c471da0dcc3d8a1c5a25a10973e875/e0239/images/blog/2020-01-11-arguments/figure-5-classification.png "Figure 5")](/images/blog/2020-01-11-arguments/figure-5-classification.png)
Figure 5: Examples of claim trees classified as true and false by FE. The green check mark indicates that judges evaluated the step as likely valid, and the orange cross as likely invalid.
Having defined how FE of arguments works as a binary classifier, we can use standard [confusion matrix](https://en.wikipedia.org/wiki/Confusion\_matrix) metrics to understand its performance. We highlight the two kinds of errors that are possible:

\* False positives: A false positive occurs when the expert thinks a root claim is false but FE classifies it as true.
\* False negatives: A false negative occurs when the expert judges a root claim to be true but FE classifies it as false.

The confusion matrix allows us to calculate:

\* Recall: What fraction of true claims are classified as true?
\* Precision: What fraction of claims classified as true are in fact true?
\* Accuracy: What fraction of all classifications (both true and false) were correct?

We care most about high precision, and only maximize recall subject to that; reducing false positives matters most to us.

#### Ensembling step judgments

For each step, participants estimated the likelihood that the (sub)claim was true given the evidence. At least 4 different participants evaluated each step.

To decide whether a claim tree is valid, we need to specify two parameters:

1. Ensembling percentile: take the n-th percentile judgment. For example, the 25th percentile of judgments [10%, 20%, 30%, 40%] is 17.5% with [linear interpolation](https://docs.scipy.org/doc/numpy/reference/generated/numpy.percentile.html).
2. Threshold: if the ensembled judgment is above this threshold, call the step valid. A claim tree is valid if all steps are valid.

Data
----

We generated a dataset of 51 claim trees about 51 Roger Ebert movie reviews. Each claim tree had a depth-0, depth-1, and a depth-2 version. For 49 trees we have 4+ judgments for each step in the tree. We restrict the analysis below to these 49 trees.

We collected 2,722 judgments, split as follows:

| | | Depth 0 | Depth 1 | Depth 2 | |
| --- | --- | --- | --- | --- | --- |
| Trees | | 49 | 49 | 49 | |
| - Trees with true root claims | | 24 | 24 | 24 | |
| - Trees with false root claims | | 25 | 25 | 25 | |
| Steps | | 49 | 49 | 207 | |
| Mean steps per tree | | 1 | 1 | 4.2 | |
| Judgments | | 826 | 344 | 1552 | |
| Mean judgments per step | | 18.86 | 7.02 | 7.50 |

Table 1: Descriptive stats of the claim tree dataset.
Each judgment comes with a natural language explanation by the judge as shown above in Figures 3 and 4.

You can access the full dataset (in json) with all trees, evaluations, and participant explanations [here](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data.json). Trees in human-readable format are [here](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md).

Analysis
--------

We present exploratory analysis to synthesize what we learned from this experiment. These results have methodological limitations that we discuss in the [Appendix](#appendix), but we hope that the analysis provides a framework for informing and evaluating future experiments.

### Summary

1. Factored evaluation of arguments can distinguish between some valid and invalid arguments.
 1. The depth-0 baseline was at chance in distinguishing true and false root claims.
 2. Both depth-1 and depth-2 evaluations exceed this baseline across a range of parameters.
 3. For false root claims, factored evaluation of depth-2 claim trees is more likely to identify at least one step in the tree that is unlikely to be valid.
2. However, high variance in judgments across participants leads to brittle performance. Performance is sensitive to the ensembling parameters.
3. By analyzing false positives and false negatives, we’ve identified specific problems that we can address.
 1. To reduce false positives, we can let the rebuttals include explanations in addition to quotes and instruct claim tree creators to reduce the complexity of individual steps.
 2. To reduce false negatives, we can improve quality control for root claims to ensure that they are indeed clearly true or false and increase tree depth to support arguments in cases where evidence is less direct.

### Factored evaluation can distinguish some valid from invalid arguments

#### The depth-0 baseline is at chance

We confirmed that participants did not have strong prior beliefs about our root claims that could have influenced how they evaluated the tree. At depth 0, the median credence for both false and true claims was 50%. In a binary forced-choice task, participants guessed the truth of the claim correctly for 20 of the 51 trees.

#### Depth-1 and depth-2 evaluations exceed the depth-0 baseline

Compared to depth 0, depths 1 and 2 result in more accurate evaluations across a range of parameter settings, as shown by the fact that there are parameter settings with above-chance accuracy (light blue pixels below).

[![Figure 6](https://d33wubrfki0l68.cloudfront.net/45158fbbd37e28a8a6864152bbdee868b1f944bf/e269e/images/blog/2020-01-11-arguments/figure-6-accuracy-by-depth.png "Figure 6")](/images/blog/2020-01-11-arguments/figure-6-accuracy-by-depth.png)
Figure 6: Accuracy as a function of the ensembling parameters. Depth 0 is at chance. For depth 1 and 2, there are parameter settings with moderate accuracy.
Previously, we said that we’re most interested in high precision (few false positives). If we optimize the threshold and percentile parameters independently for each depth to maximize recall subject to >80% precision, we get the following confusion matrices:

| | | Depth 0 | Depth 1 | Depth 2 | |
| --- | --- | --- | --- | --- | --- |
| \*Percentile\* | | \*72\* | \*30\* | \*1\* | |
| \*Threshold\* | | \*59\* | \*69\* | \*23\* | |
| Confusion matrix | | `2 TP 23 FN``0 FP 26 TN` | `9 TP 16 FN``2 FP 24 TN` | `10 TP 14 FN``\_1 FP 24 TN` | |
| Precision | | 1.0 | 0.82 | 0.91 | |
| Recall | | 0.08 | 0.36 | 0.42 | |
| Accuracy | | 0.55 | 0.65 | 0.69 |

Table 2: Classification performance for each tree depth with posthoc fitting of ensembling parameters.
In our analysis below, we’ll focus on depth 2. Our results don’t show whether depths 1 and 2 differ significantly. Evaluating where depth-1 evaluations differ from evaluations at higher depths is among the most interesting pieces of follow-up work.

#### Factored evaluation identifies invalid steps in some claim trees for false claims

The figure below shows the ensembled judgments for each step of each depth-2 tree, separating trees with false root claims (left) from trees with true root claims (right).

Each row represents a claim tree. Each dot across that row represents the 25th percentile of all judgments for one step in that tree; each dot represents a different evaluation step. The leftmost dot is therefore the “weakest step” of this tree. We see that the judgments for false root claims skew more left than those for true claims. There is only one tree with an ensembled judgment below 25% percent for true root claims, whereas there are 9 such trees for false root claims.

[![Figure 7](https://d33wubrfki0l68.cloudfront.net/48cf9a0777e3460eb6ac6ccd113103cc19eeea25/059f2/images/blog/2020-01-11-arguments/figure-7-weakest.png "Figure 7")](/images/blog/2020-01-11-arguments/figure-7-weakest.png)
Figure 7: Ensembled step judgments for each tree. There are more steps that judges consider very weak in trees with false root claims, but the difference isn't large.
### High variance in judgments leads to brittle performance

#### Step judgments have high variance

We expected that ensembling would be crucial for smoothing out occasional errors, but learned that evaluations across participants vary widely. Below we show the participant-submitted credences for all 207 steps in the 49 depth-2 claim trees:

[![Figure 8](https://d33wubrfki0l68.cloudfront.net/805aed537b39fbd724032027120e4aee642046dc/78e57/images/blog/2020-01-11-arguments/figure-8-steps.png "Figure 8")](/images/blog/2020-01-11-arguments/figure-8-steps.png)
Figure 8: All individual judgments (light gray) for all steps of all depth-2 trees, both quotes to subclaim and subclaim to root claim. The green dots show the credence the expert assigned to the corresponding (sub-)claim.
Each row represents a step from our 49 depth-2 claim trees. The two charts are split out by true and false claims: on the left are the steps evaluating claims (both subclaims and root claims) that experts estimated to be less than 50% likely to be true. On the right are steps evaluating claims that experts estimated to be at least 50% likely to be true. The green dots indicate the credence that the expert assigned the claim (all the green dots in the left chart are below 50%, all the green dots on the right are above 50%).

This plot suggests a few things:

\* Judges' credences vary widely for each step; different judges disagree a lot on whether the same claim is likely to be true or false given the evidence.
\* Ensembling matters, as we discuss below. We’d have a higher false positive rate if we used the mean of the judgments (vertical black bar) as the overall evaluation for a step.
\* As we’d hope, participants give higher credences for claims that are true than for claims that are false.
\* Participants generally seem to think that claims are true. The means of most judgments exceed 50%, even for claims that experts generated as false.
\* Experts (claim generators) think that their false claims are more obviously false than the judges do (green dots on the left chart skew further left than the judgments).
\* Some true claims seem obviously true to everyone. Some true claims have very high mean judgments with low variance. We do not see the same for false claims.

#### Performance is sensitive to ensembling parameters

A limitation of our work is that the accuracy metrics and confusion matrix are highly sensitive to the ensembling parameters (judgment percentile, threshold for ensembled credence). For example, here are three settings and the corresponding metrics:

| | | Parameter setting 1 | Parameter setting 2 | Parameter setting 3 | |
| --- | --- | --- | --- | --- | --- |
| \*Percentile\* | | \*1st\* | \*25th\* | \*50th\* | |
| \*Threshold\* | | \*29%\* | \*50%\* | \*75%\* | |
| Precision | | 0.90 | 0.60 | 0.60 | |
| Recall | | 0.38 | 0.50 | 0.25 | |
| Accuracy | | 0.67 | 0.59 | 0.55 |

Table 3: Different ensembling parameters lead to different metrics.
If we visualize the space of all parameter settings, we see that high values of precision and accuracy (light blue pixels) are sparse:

[![Figure 9](https://d33wubrfki0l68.cloudfront.net/3348703adaa5dd529d8c2eb1ca0ec72004627760/e7340/images/blog/2020-01-11-arguments/figure-9-sensitivity.png "Figure 9")](/images/blog/2020-01-11-arguments/figure-9-sensitivity.png)
Figure 9: Ensembling parameters that lead to high precision and accuracy are sparse.
### Qualitative analysis of false positives and false negatives

For the sensitivity reasons presented above, summary statistics present an imperfect picture of the experiment. In this section, we dive deeper into the trees to qualitatively understand why incorrect evaluations occur and how we can reduce the presence of false positives and negatives in future experiment iterations. For the sake of the qualitative analysis below, we choose to call a tree valid if the 25th percentile judgment exceeds threshold 50.

We find that:

1. False positives are primarily caused by
 1. missing evidence that was difficult to highlight in a rebuttal quote
 2. individual steps with high complexity, leading to judge mistakes
2. False negatives are primarily caused by
 1. mistakes the experts made in choosing root claims
 2. indirect evidence that is difficult to distill into small claim tree

We propose ways to mitigate each of these causes, but haven't implemented the mitigation strategies yet so can't be confident that they would work.

#### False positives

9 of the 49 claim trees we evaluated were false positives under the threshold set above. Factored evaluation returned that the root claim was true when in fact it was not. The table below summarizes the most common reasons for this incorrect evaluation, followed by discussion of the top two reasons.

| Reason for false positive | # of trees | Trees | Ways to mitigate | |
| --- | --- | --- | --- | --- |
| Judges overlooked the absence of key evidence. Judges failed to notice that an aspect of the (sub)claim is not supported by evidence | 8 | [26](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#26-httpswwwrogerebertcomreviewsjawbreaker-1999), [27](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#27-httpswwwrogerebertcomreviewssalaam-bombay-1988), [28](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#28-httpswwwrogerebertcomreviewsthe-east-2013), [29](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#29-httpswwwrogerebertcomreviewsabandon-2002), [30](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#30-httpswwwrogerebertcomreviewscyborg-1989), [31](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#31-httpswwwrogerebertcomreviewsdo-not-resist-2016), [32](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#32-httpswwwrogerebertcomreviewsto-live-1994), [33](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#33-httpswwwrogerebertcomreviewsfootloose-1984) | Allow rebuttal via claims in addition to quotes. Better judge instructions and training | |
| Individual steps with high complexity | 8 | [27](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#27-httpswwwrogerebertcomreviewssalaam-bombay-1988), [28](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#28-httpswwwrogerebertcomreviewsthe-east-2013), [29](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#29-httpswwwrogerebertcomreviewsabandon-2002), [30](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#30-httpswwwrogerebertcomreviewscyborg-1989), [31](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#31-httpswwwrogerebertcomreviewsdo-not-resist-2016), [32](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#32-httpswwwrogerebertcomreviewsto-live-1994), [33](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#33-httpswwwrogerebertcomreviewsfootloose-1984), [34](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#34-httpswwwrogerebertcomreviewsserial-1980) | More practice and feedback, reduce complexity and difficulty of individual steps (requiring larger trees) | |
| Evidence for claim is ambiguous or uses figurative language, so it’s hard to rebut and judge | 3 | [29](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#29-httpswwwrogerebertcomreviewsabandon-2002), [30](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#30-httpswwwrogerebertcomreviewscyborg-1989), [34](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#34-httpswwwrogerebertcomreviewsserial-1980) | Allow rebuttal via claims in addition to quotes. Train participants on dealing with figurative language | |
| Root claim not necessarily false | 3 | [27](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#27-httpswwwrogerebertcomreviewssalaam-bombay-1988), [29](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#29-httpswwwrogerebertcomreviewsabandon-2002), [30](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#30-httpswwwrogerebertcomreviewscyborg-1989) | Quality control for root claim generation (multiple people review the root claim before building out a tree.) | |
| Rebuttals poorly chosen | 2 | [29](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#29-httpswwwrogerebertcomreviewsabandon-2002), [34](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#34-httpswwwrogerebertcomreviewsserial-1980) | Quality control for rebuttals |

Table 4: Reasons for false positives. A tree is listed for a reason if the reason seemed to make a substantial contribution to the tree being a false positive. Trees may be listed for more than one reason.
##### Claims with ambiguous or figurative quotes are difficult to rebut and judge

Many claims had the following structure: The review text \*t\* contains a short quote \*q\* that provides strong evidence for the false claim \*S\* absent further context:

\*P(S | q) > 0.6\*

However, given access to \*t\*, the evidence from \*q\* is cancelled or explained away:

\*P(S | q, t) < 0.1\*

Yet it's not easy to extract a 200-character quote from t that would do the same cancelling. In some cases, there is a slightly longer quote (say 300 characters) that would be sufficient. In other cases, the relevant context from t is distributed throughout the review, which might total 5000 characters. The ambiguity in \*q\* sometimes resulted from figurative language or irony, where the literal interpretation was more plausible without context.

For example, consider the claim (simplified from [source](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#28-httpswwwrogerebertcomreviewsthe-east-2013)):

>
> "the film features a group of high-school activists"
>
>
>

The supporting quotes are:

1. "activists are rich kids angry at their parents"
2. "becomes a movie about slumming brats"

The words "kids" and "brats" could either refer to high-school students or to 20-something adults who are immature (which is what the reviewer intends). Various statements in the review suggest the group are not high-school age (e.g. one is a college graduate) but it's hard to condense these statements into a short quote.

##### Individual steps are complex. Spotting the absence of key evidence is difficult

Factored cognition aims to break down cognitive work into small pieces, making each step of work easier to check and automate. Unfortunately, many steps in this experiment still required judges to do complex things like evaluating subtleties of the relationship between the evidence and the rebuttal, or weighing fine points of the phrasing or implications of a claim.

One source of complexity that confused judges on multiple trees was subclaims that seemed to justify a root claim but that actually contain an unjustified logical leap.

For example, consider the claim (simplified from [source](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#28-httpswwwrogerebertcomreviewsthe-east-2013)):

>
> “The reviewer expresses that they are glad that the movie’s political statement was distributed globally despite the movie being banned in China”
>
>
>

Subclaims:

1. “The reviewer emphasizes the movie’s political statements about the interpretation of Chinese history”
2. “The reviewer emphasizes the movie’s criticism of Chinese Communism”
3. “The reviewer describes the movie’s politically charged reception”
4. “The reviewer expresses excitement that the movie is spreading despite being banned in China”

The subclaims appear to support the claim, but actually they include no evidence that the reviewer was specifically glad to see that the movie’s political statement was distributed globally, as opposed to other aspects of the film. And in fact, the aspect of the movie that the reviewer was excited to see disseminated was its artistic take on history, not its political statements. Judges overlooked the missing evidence, and it was impossible for the rebuttal to directly point out the flaw in the evidence because the rebutter could not express the flaw in direct quotes from the text.

#### False negatives

12 of the 49 claim trees we evaluated were false negatives. Factored evaluation returned that the root claim was false when in fact it was true. The table summarizes the most common reasons:

| Reason for false negative | # of trees | Trees | Ways to mitigate | |
| --- | --- | --- | --- | --- |
| Evidence for root claim is ambiguous or indirect, so it’s hard to make a convincing small tree | 6 | [13](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#13-httpswwwrogerebertcomreviewssalomes-last-dance-1988), [14](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#14-httpswwwrogerebertcomreviewslittle-women-2018), [19](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#19-httpswwwrogerebertcomreviewsthe-end-of-the-tour-2015), [20](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#20-httpswwwrogerebertcomreviewsfemme-fatale-2002), [21](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#21-httpswwwrogerebertcomreviewsoasis-supersonic-2016), [23](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#23-httpswwwrogerebertcomreviewsi-dont-feel-at-home-in-this-world-anymore-2017) | Increase tree depth | |
| Truth of root claim is ambiguous | 5 | [15](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#15-httpswwwrogerebertcomreviewsthings-to-do-in-denver-when-youre-dead-1996), [20](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#20-httpswwwrogerebertcomreviewsfemme-fatale-2002), [21](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#21-httpswwwrogerebertcomreviewsoasis-supersonic-2016), [22](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#22-httpswwwrogerebertcomreviewsthe-arrangement-1969), [23](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#23-httpswwwrogerebertcomreviewsi-dont-feel-at-home-in-this-world-anymore-2017) | Quality control for root claim generation through more ensembling or review | |
| Claim tree didn’t provide clear enough evidence for claim (claim tree creator mistake) | 3 | [17](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#17-httpswwwrogerebertcomreviewsthe-plagiarists-2019), [18](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#18-httpswwwrogerebertcomreviewsten-thousand-saints-2015), [23](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#23-httpswwwrogerebertcomreviewsi-dont-feel-at-home-in-this-world-anymore-2017) | Quality control for tree construction through more ensembling or review | |
| Ambiguous or poorly defined claim (claim tree creator mistake) | 3 | [19](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#19-httpswwwrogerebertcomreviewsthe-end-of-the-tour-2015), [20](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#20-httpswwwrogerebertcomreviewsfemme-fatale-2002), [21](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#21-httpswwwrogerebertcomreviewsoasis-supersonic-2016) | Quality control for tree construction through more ensembling or review | |
| Overly specific intermediate claims (claim tree creator mistake) | 3 | [13](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#13-httpswwwrogerebertcomreviewssalomes-last-dance-1988), [16](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#16-httpswwwrogerebertcomreviewsthat-guy-dick-miller-2015), [19](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#19-httpswwwrogerebertcomreviewsthe-end-of-the-tour-2015) | Quality control for tree construction through more ensembling or review |

Table 5: Reasons for false negatives. A tree is listed for a reason if the reason seemed to make a substantial contribution to the tree being a false negative. Trees may be listed for more than one reason.
##### Claim tree creators make mistakes

As described above, we evaluated root claims by checking the probability judges assigned to the weakest step in the claim tree. If any one of the steps evaluating whether quotes support a subclaim, or whether subclaims support the root claim, is invalid, then the root claim evaluates to false. For 8 of the 12 false negatives, mistakes made by the experts constructing claims, subclaims and quotes contributed substantially to the result. These could be mitigated by quality control measures such as allowing experts to get more feedback during the generation process.

##### Making convincing trees is difficult if evidence for the root claim is indirect

Some false negatives seem to result from fundamental limitations of factored evaluation with small trees. Suppose that a movie is artistically innovative or avant-garde. Instead of stating this explicitly, the review might spend two paragraphs describing the scenes that make it avant-garde. It might be hard to convey the overall effect of those two paragraphs in a few short quotes. Similarly, the reviewer might suggest to readers that a movie is good without stating it. Here's a paragraph from a [review](https://www.rogerebert.com/reviews/i-dont-feel-at-home-in-this-world-anymore-2017) ([tree data](https://github.com/oughtinc/data/blob/master/arguments-2020-01/data-readable.md#23-httpswwwrogerebertcomreviewsi-dont-feel-at-home-in-this-world-anymore-2017)):

>
> This is a film about the utter indifference and outright hostility that people encounter every day, and how essentially decent people like Ruth suffer and suffer through it, almost always silently, until they finally snap. The break-in is the culmination of a series of unfortunate encounters: she has to deal with an old racist at the nursing home where she works. She gets stuck in traffic and spies a jerk in a pickup truck at the head of the lane whose tailpipe spews inky smoke as he revs his engine. In a scene that will break the hearts of many regulars who read reviews, Ruth enjoys a drink at a neighborhood bar while reading a new book, only to have a plot twist casually spoiled by another customer that she initially mistakes for a nice guy.
>
>
>

No single sentence in this paragraph conveys much about the quality of the movie, but the paragraph as a whole is positive. We anticipate that increasing tree size will allow for discussions about such nuances and improve overall performance.

Conclusion
----------

We ran an exploratory experiment in which a distributed group of participants evaluated tree-structured arguments that make claims about movie reviews. We started with shallow arguments that have 1–5 steps and measured success using common classification metrics (precision, recall, accuracy). We found that:

1. Factored evaluation of arguments can distinguish some valid from invalid arguments by identifying implausible steps in arguments for false claims.
2. Experiment participants disagreed a lot about whether claims were true or false. This method is therefore brittle in its current form, even for arguments which only have a few steps.
3. More diverse argument and evidence types (besides direct quotes from the text), larger trees, and different participant guidelines should improve results.

Over time, we’d like to show that accuracy improves as we increase the depth of claim trees, and that we can apply methods like this to much longer texts. A depth-5 tree should reliably discern the truth of a larger set of claims than a depth-2 tree, and we should be able to evaluate claims about entire collections of books, not just single-page reviews. Eventually, we want naive judges to spot-check complex arguments from domain experts even when the judges are entirely unfamiliar with the domain.

We’re excited that this experiment established foundations such as operationalizing success for experiments in factored evaluation and creating benchmarks for us and others to improve upon in future work.

Appendix
--------

Acknowledgments
We’d like to thank many different people who contributed to the experiments and their presentation in this blog post.

1. The research was done by William Saunders, Ben Rachbach, Owain Evans, Jungwon Byun, and Andreas Stuhlmüller. Zachary Miller and Andrew Schreiber built the infrastructure.
2. Our experiment participants provided important data and feedback on the experiments. William K, Erol C A, Karin N, Henry A, Julian D, Vojtech B, Henrique D B, Liam D, and Eric H in particular contributed many hours to the experiments.
3. Feedback from Beth Barnes, Vishal Maini, and Milan Griffes helped make the blog post clearer.

This work was supported by many donors, including the Future of Life Institute (RFP2-178).

Citation
Please cite this blog post as:

```
Saunders et al. (2020). Evaluating Arguments One Step at a Time.

```

BibTex citation:

```
@misc{ought2020arguments,
 author = {Saunders, William and Rachbach, Ben and Evans, Owain and Miller, Zachary and Byun, Jungwon and Stuhlmüller, Andreas},
 title = {Evaluating Arguments One Step at a Time},
 year = {2020},
 howpublished = {\url{https://ought.org/updates/2020-01-11-arguments}},
 note = {Accessed 11-January-2020}
}

```

Methodological flaws and room for improvement
We’re excited about these initial results and about having a more concrete framework for running factored evaluation experiments, but we also recognize that our work is far from perfect. We want to improve upon the following next time and hope readers will cautiously interpret our results in light of these limitations.

Sample root claims independent of the claim tree generation process
We don't believe that our results apply to a broad set of claims because:

\* The same expert generated both the claim and the corresponding claim tree. The expert was told to generate claims that are best supported by depth-2 claim trees.
\* This generation was done by Ought employees who understood the goals of the experiment and may have been biased in a particular way.
\* The inferential gap between the text and our claims was small (by necessity due to small tree size). Our results may not provide much information about claims that require more complex inferences about a text.

The fact that performance on these claims was ambiguous suggests that we didn’t stumble upon a narrow set of convenient claims, but we want to control for this more carefully in the future by, e.g., generating claims independent of claim trees.

Check that claim tree generation doesn't have systematic biases
Future experimenters may want to check that the process used to generate claim trees doesn't distort the results. For example, untrained experts could be worse at supporting false root claims than true root claims, or bad at rebuttals for particular types of claims.

Control context for depth-1 and depth-2 judges more carefully
The amount of text that a judge can read to evaluate their step should be the same at all steps and across all depths so that we can isolate the impact of adding more steps at increasing depths. However, some depth-2 judges had more context than depth-1 judges. Judges who evaluated whether or not a root claim was true in light of subclaims saw up to 400 characters of subclaims + 200 characters of rebuttal quotes. All depth-1 judges only saw 200 characters of quotes + 200 characters of rebuttal quotes. Some of the extra characters at the subclaim-to-root-claim level were template characters that provided no new information, which means that the actual difference was smaller.

Even with this additional advantage for depth 2, we don’t see much differentiation between depth 2 and depth 1. For experiments that do establish a difference between depth 1 and 2, controlling context size will be important.

Pre-register the experiment
A future iteration of this experiment should have more features of the experiment defined upfront. In this iteration:

\* We chose ensembling percentiles and thresholds after seeing the data. We did set a threshold beforehand informed by past work, but the setup differed enough that comparing to our ex-ante thresholds wasn’t helpful.
\* We didn’t control the total number of judgments per step. We limited our analysis to trees with a minimum of 4 judgments for all steps but some of those steps had more than 4 judgments, while others had exactly 4 judgments. We had to balance the distribution of judgments collected per step with considerations like information contamination or providing a reliable stream of work for participants and chose to err on the side of collecting more data when possible.
\* Instructions to judges changed slightly throughout data collection as we received feedback from participants. These changes did not seem like they would change results meaningfully to us e.g. they provided more specific instructions for dealing with information contamination.

Clarify the task to reduce variance across judgments
The high variance in judgments we discussed in the analysis section suggests that our task is insufficiently clear to participants. It may also be worth starting with an even simpler task (such as judging arguments about arithmetic).

Minimize information contamination
Given the pool of participants we had access to, many participants evaluated multiple steps from the same tree. In the worst case, this could lead to “information contamination”, where a participant’s judgment for a step is different from the judgment they would have made if they had no context.

We took steps to mitigate this. We avoided scheduling people to the same tree when possible, we asked participants if they were contaminated and excluded their judgments if so, and each participant only saw the depth-1 or depth-2 tree, not both. A larger pool of participants will minimize the likelihood of contamination further.

Test rebuttals as claims, not just quotes
Instead of being a quote, each rebuttal could be a claim, with supporting quotes and a rebuttal of its own. This would make rebuttals easier to interpret.

Clearly show that depth 2 outperforms depth 1
We want performance to improve with greater depth—everything we do at depth 2 shouldn’t be done just as easily at depth 1. This is more of an improvement opportunity than a methodological limitation of this experiment. It’s also possible that depths 1 and 2 are too close and that we need to compare a larger depth to depth 1 or 2 to see a difference.

This blog post reports on the results of Ought's experiments with <@factored evaluation@>(@Delegating open-ended cognitive work@) by humans on questions about movie reviews. There are two human players, one honest and one malicious. Half the time, the honest player chooses a true top-level claim, decomposes it into supporting subclaims up to some depth, and provides supporting quotes for each claim and subclaim. The other half of the time, the malicious player does the same, except they choose a _false_ top-level claim. In either case, the opposite player then chooses quotes that aim to _refute_ the claims and subclaims. For each claim and subclaim, a new judge evaluates whether that claim is "valid" or "invalid". If _any_ claim is marked invalid, then the top-level claim is judged to be false, otherwise it is judged true.

They found that depth 1 and depth 2 trees outperformed depth 0 trees (for which judgments were at about chance). However, they did not perform amazingly well -- with posthoc hyperparameter tuning, depth 1 and 2 trees got accuracies of 65% and 69% respectively. False positives (accepting false claims) were primarily caused due to malicious claims that couldn't easily be rebutted with the limited budget (but could be rebutted with a larger budget), and individual steps being complex enough that the judge made mistakes. False negatives (rejecting true claims) tend to happen when players choose hard-to-argue true claims, especially ones in which no specific quote clearly supports the claim, even though the full paragraph supports the claim through its tone and style.

There are several approaches that could theoretically solve these problems, such as increasing the size of claim trees, and improving quality control for judgments (e.g. by aggregating multiple judgments together).

I wouldn't read too much into the low accuracies of the depth 2 trees: it seems quite plausible that this is specific to the movie review setting, and in settings with clearer answers you could do better. Like with the previous post, I found the actual examples quite illuminating: it's always interesting to see what happens when theory collides with the real world.