'The Hyperbolic Time Chamber & Brain Emulation'

> A time dilation tool from an anime is discussed for its practical use on Earth; there seem surprisingly few uses and none that will change the world, due to the severe penalties humans would incur while using it, and basic constraints like Amdahl’s law limit the scientific uses. A comparison with the position of an Artificial Intelligence such as an emulated human brain seems fair, except most of the time dilation disadvantages do not apply or can be ameliorated and hence any speedups could be quite effectively exploited. I suggest that skeptics of the idea that speedups give advantages are implicitly working off the crippled time dilation tool and not making allowance for the \*dis\*analogies.

\_[Dragon Ball Z](!W)\_, a popular (but mediocre) shonen fighting anime, includes a cute bit of SF in it---the [Hyperbolic Time Chamber](https://dragonball.fandom.com/wiki/Hyperbolic\_Time\_Chamber) (HTC), which can be thought of a reverse [twin paradox](!W) in which time speeds up for 2 people: the HTC opens and closes once a day realtime, but inside its little pocket universe, a full year passes for 2 people, giving a 365× speedup. There's no such thing as a HTC-within-a-HTC, so uses are limited to just that one 365× speedup---no 365^2^ speedups.

Ignoring the \_DBZ\_-specific aspects of the HTC like the person limit or increased gravity or air and temperature changes, one wonders: what one would do with a HTC in the real world?

# Uses for time acceleration

In \_DBZ\_, the only use seems to be for training montages to let characters power up to fight a new alien or martial artist, but world-destroying martial artists seem to be rare in the real world, so we couldn't use it for \*that\*.

Could we use it for regular martial arts training? The \_DBZ\_ article for the HTC mentions no non-emergency use, and a little thought leads us to conclude that, probably not: inside the HTC, time passes as normal, which means that you don't save any time. All the HTC is doing is rearranging relative time between groups. If you step in, you still age a full year before stepping out, and you will now die a year early by the realtime calendar. So what's the point? We can think of a few uses---imagine someone who gets injured just before the Olympics---but let's face it, that may be convenient for a few people, but it's hardly changing the world. A whole time-accelerated pocket universe... Surely we can think of something less pointless than tweaking athletics?

# Downsides

Actually, it's \*worse\* than pointless---the HTC is a super-[supermax prison](!W): you cannot leave at any point before the year is up, you cannot communicate in any way, nothing goes in or out, and you have only what you brought with you (and always what you brought with you).

## Autarchy

Under such conditions, a year in the HTC could well be considered "cruel and unusual punishment"; no doctor would volunteer for it, so any prisoners in the HTC face a serious risk of death from any cause unless a fellow prisoner that day/year had medical training. The food will also suck, as any food will have to be storable for a year (it wouldn't do to starve to death a day before the door opens back to the real world); you can't grow your own because there is no apparent sun, soil, or seasons inside the HTC, and while you could probably bring in a greenhouse & soil and fertilize with recycled food, how are you going to \*power the lights\* in your greenhouse? Drag in a miniature nuclear power plant or a [radioisotope thermoelectric generator](!W)? And we haven't even considered how much time one would spend (waste) on this small-scale agriculture; maintaining [Biosphere 2](!W) was a full-time job for many highly-skilled people, and if that was true for the HTC, there would hardly be any point in it.

If one punted on the problems of maintaining a high quality of life and posited a dedicated researcher, well, the HTC is still not useful. They will find it hard to take with them an entire library or laboratory, many ingredients are too expensive or perishable to buy in advance just because the researcher \*might\* need them, but if they don't have access to pretty much everything, they'll quickly hit some sort of barrier where one email or order would let them finish a project but that email can't be sent for up to a year. (Imagine a researcher who enters the HTC---and his laptop's hard-drive dies. Oops. Hope he had backups or spares, of his data and his laptop and everything else for that matter.) Omitting these concerns, research is a social process in the sense that one is often discussing or explaining or defending the research, and without these interactions it is easy to go down blind alleys, make minor-seeming but fatal mistakes^[The inability to critique your own results or ideas as capably as someone else can seems to have deep roots in psychology and support evolutionary accounts of reasoning as evolved primarily for arguing and convincing other people, not truth-seeking. See also "rubber-ducking".], wind up reinventing something standard in another field, etc. One can easily waste a month this way, and so a year. A group would help, but groups are susceptible to groupthink and will still go down blind alleys or simply lack relevant expertise. (In this respect, the Millennial Maths in [Neal Stephenson's](!W "Neal Stephenson") \_[Anathem](!W)\_ are highly unrealistic; any group of academics which closeted themselves for a millennium would overwhelmingly likely be a sheer waste of human capital.)

One might wonder about other kinds of education in the HTC like mathematics, but all the above points apply to any reason to live in the HTC: why would you accept all those burdens to spend a year learning something... when you could just live that same year in the real world at much less cost and a far higher standard of living? It certainly \*would\* be nice to go into the HTC for a few weeks and come back with a dozen PhDs---but not if you emerge aged 40, having lived the best years of your life in a prison cell and probably deep in debt too!

## Aging

Speaking of a few years in the HTC, what about biological aging? People don't ordinarily spend half their lives acquiring multiple degrees outside the prison of an HTC, why would they voluntarily do so inside it? It's the same trade, after all: half your life for multiple degrees. This point has been made in fictional treatments of time-acceleration [R.A. Lafferty's](!W "R.A. Lafferty") classic short story ["The Six Fingers of Time"](https://www.gutenberg.org/files/31663/31663-h/31663-h.htm) where the protagonist is given the ability to slow down time by a mysterious ancient conspiracy, and while he tries to uncover their secrets, he dies of old age---they had let him slow down time, but not his inherent natural aging. Or [William Sleator's](!W "William Sleator") YA novel [\_Singularity\_](!W "Singularity (William Sleator novel)"), where the original discoverer of the HTC dies before his family expects it, looking suspiciously like an old man; the implication is that he spent so much time in the HTC investigating it that the normal aging while inside it used up a good chunk of his lifespan. Ironically, if he had been able to survive another month, he would have seen the resolution of the mystery. The protagonist benefits somewhat from his own year in the HTC, but for idiosyncratic reasons. Similarly in [Greg Egan's](!W "Greg Egan") [\_The Clockwork Rocket\_](!W "The Clockwork Rocket") the protagonist discovers a method of time acceleration which she resolves to use to save her world from certain doom by launching a generation ship to be accelerated and discover some salvation; but she and the first generation (a good chunk of their world's scientific community) fully expect to perish of age long before the generation ship returns just years later in realtime. In a more conventional example, we may admire how prison & revenge give the Count in \_[The Count of Monte Cristo](!W)\_ a great deal of focus during his educational prison stay but how many of us would agree to be imprisoned the same way if there were no hidden fortune waiting for us at the end?

This is a fundamental issue and probably why time-acceleration is an underused trope in science fiction compared to time dilation or time travel: the downside is simply too apparent.

# Some uses
## Zero-sum competition

The point about the HTC 'rearranging relative time' for athletes and the original use in \_DBZ\_---training to save the world when every minute counts---suggests one class of problems: things which are \*extremely\* time-sensitive with multiple competing groups and zero-sum or winner-take-all dynamics.

With large sums of money at stake, we can hand-wave the super-supermax prison points: oil companies only have to pay oil rig workers a few score or hundreds of thousands of dollars for working several weeks at a time on oil rigs, and scientists and astronauts compete for position in isolated facilities as Antarctic bases or the International Space Station. All of these are far less isolated or burdensome than the HTC, but by more than a few factors? Seems unlikely. So a few million dollars may suffice to cover the costs of a small group, especially if they can reuse infrastructure from previous days/years.

Are there business problems where a year's headstart is worth at least a few million dollars? Sure! Many programming tasks come to mind: would Google pay a few million to lock up the core Android coders to take care of a years' worth of outstanding bugs and to-do items? Would Apple do something similar? What about any hedge fund? It seems plausible that every day of a HTC could be booked or even auctioned off.

The negatives here include the lack of communication and iteration: when the group heads through the door, that's the last they'll hear from the world for a year. They can't release a prototype at the 6 month mark and see how it does after a month. If they fall to groupthink and take the wrong approach, there are no outsiders who will say that their approach is crazy and elaborate and why don't they just do the standard thing? Worse, if they discover they forgot a key piece of documentation or hardware or they run out of chips or they need a particular expert or something, the next time they can get it is... a year later. Oops. Hope that wasn't a fatal error. (Even if they had communication, it'd only bound the losses: if it takes a second to load a webpage in realtime, then it will take them >365 seconds or >6 minutes.) This lack of iteration runs counter to many business styles and is entirely antithetical to modern tech businesses which prize constant feedback and ability to change ideas & approaches on a dime.

Still, with multi-terabyte hard drives, one could just take a copy of \*all\* documentation and source code (or perhaps bring along a few Internet Archive-style "petaboxes" and store a copy of a small fraction of the Internet), and for some tasks like stock-market research, it's plausible one could bring everything one needs. Hedge funds would probably benefit from being able to send in their quants for a year of concentrated research and scoop the competition.

## Non-zero-sum uses

The more concrete a field, the less the benefit. Most commercial services would be impossible: you can't cut someone's hair in the real world from the HTC, although with loads of equipment you could work on a robot which cuts hair. You can't run clinical drug experiments on a group of patients from inside a HTC either; for that matter, you'll have a hard time bringing along rats or monkeys. But you could read a lot of papers on rats. (But not necessarily do much; for example, meta-analyses will be hard because frequently authors do not include the exact numbers one needs, and so one has to contact them---exactly what can't be done in the HTC.) Pure mathematicians might benefit, but by and large, mathematics is not \*so\* competitive & time-sensitive that sticking some mathematicians into the HTC would be worth the premium.

Which is not to say there are \*no\* concrete uses. One cute example would be storage of goods: instead of an [art & wine](https://www.nytimes.com/2012/07/22/business/swiss-freeports-are-home-for-a-growing-treasury-of-art.html "Swiss Freeports Are Home for a Growing Treasury of Art") [free port](!W), just stick your wine & cheese & other goods in the HTC and let them age a year every day until ripened to perfection. More valuably, one could bypass [accelerated aging](!W) tests and just age a product directly; want to know if the [Clock of the Long Now](!W) will work or the [Rosetta disk](!W) be readable for 10,000 years under ideal conditions? That's just 10,000 days or 28 years away. (We could also expect an efflorescence of counterfeit art, documents, and goods for the same reasons.) Better yet, want to run fast primate aging experiments? If you can front the money and either automate the care & feeding of the subjects or find lab technicians willing to spend their lives in prison, you can run as many as you please.

These wouldn't be revolutionary improvements, though (with the exception of aging research which might revolutionize human society if the results were useful).

# Self-contained vs not

More generally, we could say that [Amdahl’s law](!W) applies to use of HTC: any task has serial and parallel elements, but if some elements are made cheaper or even free, the time to accomplish the task still depends on the other bottleneck elements. Elements which can be done in complete isolation \*and\* which benefit from relative speedups correspond to parallel elements, and elements which must be done in the real world correspond to the serial elements. With a HTC, the HTC-elements will quickly speed up, but tasks will now bottleneck on real-world tasks. (Imagine Google sends its Android programmers into the HTC and they return a day later bearing a repository groaning with new patches; the features still have to be tested in a real world context, reviewed, infrastructure updated, and finally actually transmitted to the customers who may begin using them.)

One opportunity is to look at Amdahl’s law as a positive, and look at the computer version of an isolated team in the HTC beavering away on a project: a bunch of servers working on an extremely hard serial problem. For example, simulating a long evolution of [protein folding](!W). Many problems in scientific computation or [operations research](!W) where there is more than a day's margin might also benefit from what is effectively a super-fast processor with 1-day latency. Further, such a supercomputing facility in an HTC faces problems with replacement parts and getting the electricity such computations will consume (and how much would you have to pay the sysadmins & technicians to be imprisoned for a year?), and so its capacity will come at a premium compared to the equivalent real-world problem; a problem like hash cracking which is trivially parallelizable would not benefit from such a facility. Electricity is the dominant cost of computing power these days, so a HTC must save on electricity or justify its cost premium. Instead of throwing one really expensive HTC server at the problem for a year, throw 365 cheap power-efficient servers at it for a day as many tech companies are able to do, or just run it on a cloud computing platform.

These points do not apply to any computation which is inherently serial and cannot be run on more than a few computers. One such category of non-parallelizable problems (assuming NC ≠ P) is the complexity class [P-complete](!W), which includes such economically important tasks as [linear programming](!W) optimization. However, the important unparallelizable problems (at least linear programming) typically have [very fast](/aria#faster) approximate or heuristic solvers, and optimization problems tend to asymptote and experience severely [diminishing returns](!W). Is there a problem where the time-limit is so tight and the additional optimization so valuable that it would pay for a year of premium power consumption & computation? I don't know. Maybe there is.

So, many business applications would not benefit, many research tasks would not benefit, and I haven't thought of any important areas of life which would benefit from a HTC. Some people would find it convenient to re-arrange their lives even at some cost, arranging big blocks of time for some self-contained things (for example, working on one's own projects), but the benefit would be limited; I would analogize to [modafinil](/modafinil), which can be employed to free up a block of 8 hours (skipping a night of sleep) but at a cost (money). If one believes that modafinil use comes with no health penalties or recovery sleep, it arguably is \*better\* than a HTC because it can be used in more convenient chunks and you remain in the real world while using it, running at realtime. Yet, while modafinil is popular among a few groups, it has not revolutionized the world.

In general, a world with one or many HTCs would look a great deal like our own, although in some areas, there will be sudden bursts of progress as HTC groups return from their expeditions with their prizes and likely a one-time economic boost as HTC-specific applications are discovered.

# Real HTCs

Well, cute and interesting, but why do we care about this SF trope from \_DBZ\_?

Because the HTC can be analogized to an emulated brain or an "upload"! The 365× speedup of people in the HTC could be the speedup of a brain on a supercomputer after considerable optimization^[Although it's unlikely that the exact speedup would be near 365×, as power & heating constraints dominate the problem; [GeraldMonroe](https://www.lesswrong.com/posts/PvdvsZQwDr2PD3wFW/dragon-ball-s-hyperbolic-time-chamber7c10) points out that a straightforward comparison of transistor vs neuron switching speed leads to factors like 25 million, and modern CPUs are limited in speed mostly by heat dissipation issues---the standard 2--4GHz CPU could run at 5GHz+ if one had powerful cooling. Heat concerns led [Keith Henson](!W) to [argue that](https://web.archive.org/web/20120415171945/http://hplusmagazine.com/2012/04/12/transhumanism-and-the-human-expansion-into-space-a-conflict-with-physics/) datacenters of uploaded brains would eventually relocate to the deep sea for maximal cooling (and hence, speed).]. (One could argue that early uploads will run at far less than real-time as they will be created as soon as hardware is just powerful enough to run them at all, and be completely uncompetitive & research projects; but then again, their creation could come long after the hardware exists, waiting on bottlenecks like scanning of brains---the ["hardware overhang"](https://www.lesswrong.com/tag/computing-overhang) question.) A computer, like a HTC, cannot be nested to give a speedup; a virtual computer will usually run slower relative to realtime. A brain on a computer without any peripherals like a robot will be isolated from the real world, just like the people in the chamber, and so on.

We found the HTC not useful in practice; does this conclusion also follow for uploads? Should we expect uploads to struggle in the marketplace, finding valued niches but not causing increases in world GDP growth rates or any sort of Singularity?

# Emulations are not HTCs

While the similarities are striking, so are the dissimilarities:

#. a computer can have communications with the Internet & world; a 365× slowdown may be painful, but it is better than a fixed delay of 0--365 \*days\*.

 Even when the slowdown hits, there is the option---much reduced in the HTC---of switching to an entirely different task. (Similar to the computing world's reaction to clock speed stagnation and the rise of multi-cores, with the attendant pressure on Amdahl’s law[^Amdahl]: process-level parallelism. You can't do just one thing faster, so you might as well do many things slower.) This eliminates many of the objections. If it really can't find anything to do with its time, an emulation can always slow itself down to real-time.
#. the overhead of living in the Hyperbolic Time Chamber is reduced; a computer in the real world benefits from all the real world infrastructure like power plants or semiconductor chip fabs. There are some power savings from [underclocking](!W), but there's not much reason to otherwise run as fast as possible. (This permits many more minds to run sped-up as compared to humans living in the HTC, reducing further the disadvantage of #1 and also increasing the value of being sped-up.)
#. A person in the HTC is a relatively fixed quantity, especially since many resources will be unavailable; an emulated brain has access to those resources per #1, but also has many options different from a regular human. (A much-discussed topic; see eg. [Sotala 2012](https://philpapers.org/archive/SOTAOA.pdf#miri "Advantages of Artificial Intelligences, Uploads, and Digital Minds").)
#. An emulated brain is free of a major time limit for regular humans: aging. While a human could not afford to get 12 PhDs even if a HTC existed---because that would consume the most productive decades of his life---an emulated brain could. This breaks the symmetry further.

Between these 4 disanalogic points, an upload avoids some of the disadvantages that renders the HTC noncompetitive and gains some advantages which may make it more competitive, and make the sudden improvements a much more general phenomenon.

Expecting any dramatic changes from uploads or AGIs in general has been mocked by critics as an over-valuing of "brains in a box" or, pace Gene Wolfe, [magical thinking](!W) ("The would-be sorcerer alone has faith in the efficacy of pure knowledge"). If we look at such criticism, do the arguments seem to assume a model of thinking in which the upload/AI is trapped in the HTC, or does it resemble an upload/AI outside the HTC?

[^Amdahl]: Amdahl’s law is also relevant to the economics of uploaded brains: suppose one believed that specialized or ["tool"](https://www.lesswrong.com/tag/tool-ai) AIs will always outperform any uploaded brain or AGI at a specific task, and every improvement that speeds up the uploads/AGIs improves the tool AIs just as much, such that the uploads/AGIs never surpass the tool AI; does this imply that there will be no uploads/AGIs outside niches like research, as humans using tool AIs are more profitable? [Holden Karnofsky](https://www.lesswrong.com/posts/6SGqkCgHuNr7d4yJm/thoughts-on-the-singularity-institute-si) seems to think something similar when he doesn't think that competitive pressure will force people running tool AIs to eventually switch to running AGIs; [Nick Szabo](https://unenumerated.blogspot.com/2011/01/singularity.html) explicitly believes uploads/AGIs can never be profitable given tool AI competition:

 > Even if there was such a thing as a "general intelligence" the specialized machines would soundly beat it in the marketplace. It would be far from a close contest.

 I disagree. The market is not purely tool AI vs AGI. \*Humans\* do not increase their speed even if tool AIs are increasing their speed arbitrarily. Therefore, a human+tool-AI system's performance asymptotically approaches the limit where the tool-AI part takes zero time and the human part takes 100% of the time. Time pressures may force a shift to ever more tool AI systems and eventually tool-AI+AGI systems when that becomes possible. (["Greater use of highly adaptable and flexibly autonomous systems and processes can provide [substantial] time-domain operational advantages over adversaries who are limited to human planning and decision speeds..."](https://defenseinnovationmarketplace.dtic.mil/wp-content/uploads/airforce/TechnologyHorizonsVol1\_PublicReleasesmall.pdf "Report on Technology Horizons: A Vision for Air Force Science & Technology During 2010-2030")) The moment that algorithmic progress or Moore’s law means that an AGI even slightly outperforms a human at using the tool-AI, the same economic reasons you were counting on as your salvation suddenly turn on you and drive the replacement of any humans in the loop. Since humans are a known fixed quantity, if an AGI can be improved---even if at all times it is strictly inferior to a tool AI at the latter's specialization---then eventually an AGI+tool-AI system will outperform a human+tool-AI system (barring exotic unproven assumptions about asymptotic limits).

 Attempts to evade this by splitting up or combining tool AIs either don't avoid this logic or wind up accepting the conclusion: if every human skill has been transferred to tool-AIs, then a complex of tool-AIs now forms an AGI which outperforms all humans by definition; if not every human skill has been transferred, such as "employing tool-AIs as most appropriate for the moment", then there is the large economic niche for AGIs which I have identified with my Amdahl’s law argument. So either there exist AGI which outperform all humans, or there exists economic pressure to use AGI. For example, if one argued that a complex of tool-AIs would not share worldviews or data appropriately and need a human to coordinate them, well, why can't an AGI do this and be superior to the humans per Amdahl’s law?

 What human is in the loop on high frequency trading? Who was in the loop when Knight Capital's market maker was losing hundreds of millions of dollars? The answer is that no one was in the loop because humans in the loop would not have been economically competitive. That's fine when it's "just" billions of dollars at stake and companies can decide to take the risk for themselves or not---but the stakes can change, externalities can increase.

 Here's another near-future test/example: how do we humans deal with [drones](!W "Unmanned combat aerial vehicle")? Drones are exploding in popularity, are increasing their capabilities constantly, and are coveted by countless security agencies and private groups for their tremendous use in all sorts of roles both benign and disturbing. Just like AIs would be. The tool vs general AI distinction maps nicely onto drones as well: a tool AI corresponds to a drone being manually flown by a human pilot somewhere, while a general AI would correspond to an autonomous drone which is carrying out some mission (blast insurgents?). So, here is a near-future test of the question 'are people likely to let tool AIs 'drive themselves' for greater efficiency?'---simply ask whether in, say, a decade there are autonomous drones carrying tasks that now would only be carried out by piloted drones. If in a decade we learn that autonomous drones are killing people, then we have an answer to our tool AI question: it doesn't matter because given a tool AI, people will just turn it into a general AI.

# See Also

- [Slowing Moore's Law](/slowing-moores-law "Weak points in the networks powering technological progress: chip factories."){.backlink-not}
- [Why Tool AIs Want to Be Agent AIs](/tool-ai "AIs limited to purely computational inferential tasks (Tool AIs) supporting humans will be less intelligent, efficient, and economically valuable than more autonomous reinforcement-learning AIs (Agent AIs) who act on their own and learn to take actions over choice of computation/data/training/architecture/hyperparameters/external-resource use."){.backlink-not}

# External Links

- Discussion: [LessWrong](https://www.lesswrong.com/posts/PvdvsZQwDr2PD3wFW/dragon-ball-s-hyperbolic-time-chamber); [/r/Rational](https://www.reddit.com/r/rational/comments/9aaxbk/hsfedumkth\_the\_hyperbolic\_time\_chamber\_as\_brain/)
- ["Messing With Time: Why The Flash is in Hell"](https://jessegalef.com/2013/01/27/messing-with-time-why-the-flash-is-in-hell/)
- ["That Alien Message"](https://www.lesswrong.com/posts/5wMcKNAwB6X4mp9og/that-alien-message), Eliezer Yudkowsky
- ["Slow Tuesday Night"](https://www.baen.com/Chapters/9781618249203/9781618249203\_\_\_2.htm), [R.A. Lafferty](!W)
- [\_Worth The Candle\_](https://www.royalroad.com/fiction/25137/worth-the-candle), Alexander Wales (protagonists make extensive magic-aided use of an HTC)


Import AI 334: Better distillation; the UK's AI taskforce; money and AI

Welcome to Import AI, a newsletter about AI research. Import AI runs on lattes, ramen, and feedback from readers. If you’d like to support this (and comment on posts!) please subscribe.

[Subscribe now](https://importai.substack.com/subscribe)

**Special Essay: What should the UK’s £100 million Foundation Model Taskforce do?**The UK government has recently established a ‘[Foundation Model Taskforce](https://www.gov.uk/government/news/initial-100-million-for-expert-taskforce-to-help-uk-build-and-adopt-next-generation-of-safe-ai)‘, appointed a savvy technologist named Ian Hogarth to run it, and allegedly allocated ~ £100 million in funding to it. Later this year, the UK plans to hold a [global summit on AI and AI safety](https://www.gov.uk/government/news/uk-to-host-first-global-summit-on-artificial-intelligence) and this will likely leverage the taskforce, also. Given that, what should the taskforce do and what kind of impacts might it have? That’s what I try to sketch out in this essay.

   **Why I wrote this - talk might be cheap, but maybe it is useful?** I spend a lot of time either writing in public at 30,000-feet (via ImportAI), or at about 10 feet via private memos for my company or interested parties in policy and the broader AI world. This essay is an attempt to write something that's more opinionated and specific than most policy writing and is itself an experiment. I hope it's interesting and helpful!
**Read the essay here** at my personal site: [What should the UK’s £100 million Foundation Model Taskforce do? (jack-clark.net)](https://jack-clark.net/2023/07/05/what-should-the-uks-100-million-foundation-model-taskforce-do/).

Import AI is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

####################################################

**DeepMind figures out a better way to miniaturize models:***…Can we make smaller and therefore cheaper to run models without huge performance hits? It's certainly getting easier to do so!...*DeepMind researchers have developed Generalized Knowledge Distillation (GKD), a way to take a large model and use it to train a smaller and more portable model without sacrificing as much on performance. Techniques like GKD are important because they relate to the general problem of distilling and distributing models - today's generative models are very large which means a) they cost a lot to run, b) running them requires complicated infrastructure (e.g, multiple GPUs to sample from the model), and c) the models take up a bunch of space so it's harder to cram them onto smaller devices, like phones. Techniques like GKD promise to make it easier to use large models as 'teachers' and distill their desired attributes into smaller student models, which you can then run and sample from cheaply.

**What's special about GKD?** For GKD, DeepMind makes the distillation process itself smarter. "Instead of training the student using a fixed distribution over outputs, we argue for using samples from the student’s distribution itself during training, akin to RL and on-policy distillation," DeepMind writes. "Furthermore, to address model under-specification, we argue that alternative objectives that focus on generating samples from the student that are likely under the teacher’s distribution, such as reverse KL, are more suitable for distilling auto-regressive models. Combining the above ideas, we propose Generalized Knowledge Distillation (GKD), which generalizes both on policy and supervised distillation."

**How well does it work?** In tests, DeepMind shows that GKD "outperforms commonly-used approaches for distilling large language models on summarization, machine translation (WMT), and arithmetic reasoning (GSM8K) tasks."

**Why this matters - if stuff gets cheaper, you get more of it:** Distillation is at root a way to take an expensive thing and make it cheaper. In life, whenever you make stuff cheaper, you tend to use it more. As model miniaturization advances, we can generally expect to see more widespread usage of AI in more surprising places. "We believe that our method will be a valuable resource for researchers and practitioners who are working on improving performance of small auto-regressive models," the DeepMind researchers write.
   **Read more:** [GKD: Generalized Knowledge Distillation for Auto-regressive Sequence Models (arXiv)](https://arxiv.org/abs//2306.13649).

####################################################

**Special section: What does $2.8 billion tell us about AI in 2023?***MoneyMoneyMoney, but what does it all mean?*This week, we're writing about $1.5bn in fresh capital deployed into AI companies - two new startups and one more established startup - as well as $1.3bn in capital via an acquisition of an AI company. The interesting thing to me is that we're seeing these huge sums of money going into AI while the rest of the tech startup economy is in a pretty wintry state, and the global economy isn't doing so well either. What's going on?

**Databricks nabs MosaicML for $1.3 billion (probably mostly in stock):**Software company Databricks has acquired AI training company MosaicML for $1.3 billion. The acquisition is a sign of how strategic AI is becoming to large companies and also an indication that there's money to be made in supplying the picks and shovels used by the workers of the AI revolution. Note that terms of the "$1.3 billion" aren't disclosed and it seems likely a lot of it is mostly in equity in private company Databricks - nonetheless, it means the CEO of Databricks is willing to put $1.3 billion of their own theoretical wealth on the line to bet that AI training is valuable.
   "Databricks and MosaicML have an incredible opportunity to democratize AI and make the Lakehouse the best place to build generative AI and LLMs," said Databricks CEO Ali Ghodsi in a statement.
   **Read more**: [Databricks Signs Definitive Agreement to Acquire MosaicML, a Leading Generative AI Platform (Databricks official site)](https://www.databricks.com/company/newsroom/press-releases/databricks-signs-definitive-agreement-acquire-mosaicml-leading-generative-ai-platform).

$$$

**Inflection raises $1.3billion, plans 22,000 H100 GPU cluster:**Inflection, a company that trains large-scale AI models and deploys some of them via a public chatbot called Pi, has raised $1.3 billion from Microsoft, Reid Hoffman, Bill Gates, Eric Schmidt, and NVIDIA. The company will use the funds to build a cluster of 22,000 NVIDIA H100 chips (for reference, in mid-2022 Facebook was targeting a buildout of a cluster of 15,000 GPUs presumed to be the prior 'A100' gen). Inflection's goal is to build ai systems that work as a "kind and supportive companion offering text and voice conversations, friendly advice, and concise information in a natural, flowing style."
   Notably, Inflection is using cloud company Coreweave for its H100 cluster, rather than one of the big clouds. Inflection uses language like 'largest AI cluster in the world' to describe this, but I think that's likely wrong, and it's curious to me why they want to publicly make claims like this: what do we speculate other well-funded companies may spend their money on - daiquiris and jetskis?
   **Read more:** [Inflection AI announces $1.3 billion of funding led by current investors, Microsoft, and NVIDIA (Inflection.ai post)](https://inflection.ai/inflection-ai-announces-1-3-billion-of-funding).

$$$

**Ex-Googlers grab $58m to found a new generative model company, Reka:**

Senior researchers from DeepMind, Facebook, and Google have raised $58m to build Reka, a company with the broad goal of trying to "build generative AI models for the benefit of humanity, organizations, and enterprises." More specifically, Reka will do research into "general-purpose multimodal & multilingual agents, self-improving AI, and model efficiency", and is developing "state-of-the-art AI assistants for everyone regardless of language and culture." It already has one product in closed beta, according to its website. The company is based in the San Francisco Bay Area but describes itself as remote-first and 'globally distributed'.

   **Read more:** [Announcing our $58M funding to Build Generative Models and Advance AI Research (Reka)](https://reka.ai/announcing-our-58m-funding-to-build-generative-models-and-advance-ai-research/).

$$$

**European AI startup Mistral raises $113m:**Mistral, a four week old startup led by well regarded researchers from DeepMind and Facebook, has raised $113million to help it develop large language models. The European startup will be based out of Paris and will seek to differentiate itself against OpenAI by building models and releasing some of them as open source (or perhaps open access) software. "We believe that the benefit of using open source can overcome the misuse potential,” Mistral CEO Arthur Mensch told *TechCrunch*.
**Find out almost nothing** at the [minimal Mistral AI website](https://mistral.ai/).
   **Read more**: [France’s Mistral AI blows in with a $113M seed round at a $260M valuation to take on OpenAI (TechCrunch)](https://techcrunch.com/2023/06/13/frances-mistral-ai-blows-in-with-a-113m-seed-round-at-a-260m-valuation-to-take-on-openai/).

**$!: My take is: you haven't seen anything yet**. Now that investors know you can turn generative models into money, and people know that the main barrier to better generative models is typically compute, then the whole AI sector is going to be on a ratchet determined by the largest known model, how good it is, and what people *think* other people are doing with it or *how much* people think is being made from it. Put another way - each time a large company like an OpenAI or a Google announces a large-scale new model and shows it can be economically useful, the amounts of capital being deployed into the sector probably need to ratchet up in relation to the compute dumped into that model. Who knows how long this'll go on for, but I expect that in 2025 the amount of capital being deployed into AI in 2023 will look laughably small in hindsight.

####################################################

**Tech Tales:**

**Novelty Hunter**

[Los Angeles, 2025].

The work was getting better and better paid but harder and harder to find. Our job was to find 'novelty' for the AI systems - little scraps of human culture that registered as off-distribution. The novelty came in many forms - underground raves, DIY bands, offbeat comedy. But the uniting aspect was analog reality - anything that got uploaded got pulled in to the AI systems so quickly that real novelty only existed offline and the people that made real novelty knew it, so they tried to make themselves hard to find.

   I used to be an undercover cop but I became a novelty hunter because the pay was better. But the job seems like it's getting almost as dangerous. In the early days you could take photos and videos but then the clubs and venues and events started handing out stickers for you to put over your phone. But we got wise to that and made our own stickers with little holes in them. Then people started confiscating phones and we had to get more creative.

   Now, there are hundreds of little companies in China and other places which supply all kinds of novelty-spy gear - directional microphones that fit in baseball caps, little cameras that you can hide inside dark glasses, and right now they're even working on miniaturizing the smell sensors. People like me spend our time drifting through the un-logged parts of life, harvesting as much novelty from a given social scene or genre as we can, before moving towns and uploading everything.

Culture Cannibal, Do Not Let Them In! Say some of the signs, with pictures of me and my colleagues in our in-scene outfits.

**Things that inspired this story:** Fast fashion; punk shows in the East Bay with bands called things like 'Techie Blood'; everything digital is in-distribution; the hunger for heterogeneous data; humans will always seek the edge of the normal distribution and AI systems will always embrace everything and create a kind of cyber-blandness out of it; market incentives and creative economies; the odd flatness of AI creativity and AI art compared to truly strange human art; jobs amid the singularity.

Import AI is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.


Import AI 333: Synthetic data makes models stupid; chatGPT eats MTurk. Inflection shows off a large language model

**Welcome to Import AI, a newsletter about AI research. Import AI runs on lattes, ramen, and feedback from readers. If you’d like to support this (and comment on posts!) please subscribe.**

[Subscribe now](https://importai.substack.com/subscribe)

 **Qualifying Life Event:** Astute readers may have noticed we skipped an issue last week - that's because I recently became the caretaker of a newborn baby. Therefore, Import AI issues may be on a slightly more infrequent schedule while I get my feet under me. A few months ago I asked a prominent AI person about what I should do as a new parent - they said 'it'll be really interesting to look at how they develop and notice their cognitive milestones and keep track of that… don't do any of that, it's really weird, just be present and enjoy it." So that's what I'm doing! Thanks all for reading.

**Uh oh - training on synthetic data makes models stupid and prone to bullshit:***…Yes, you can still use synthetic data, but if you depend on it, you [nerf](https://www.urbandictionary.com/define.php?term=nerf) your model…*Researchers with the University of Oxford,  University of Cambridge, University of Toronto, and Imperial College London, have discovered that you can break AI systems by training them exclusively on AI-generated data.
   This is a big deal because in the past couple of years, researchers have started using synthetic data generated by AI models to bootstrap training of successor models.  "We find that use of model-generated content in training causes irreversible defects in the resulting models, where tails of the original content distribution disappear," the researchers write. "We discover that learning from data produced by other models causes model collapse – a degenerative process whereby, over time, models forget the true underlying data distribution, even in the absence of a shift in the distribution over time."

**Narrow-minded, over-confident bullshit machines**: The takeaway from the research is if you train on tons of synthetic data it seems like you can break the quality of your model - specifically, you end up with models that output a narrower range of things in response to inputs, and these models also introduce their own idiosyncratic wrong-outputs as well.
   "Over the generations models tend to produce more probable sequences from the original data and start introducing their own improbable sequences," they write.
   Still a place for synthetic data: However, it seems like you can blunt a lot of this by carefully mixing in some amount of real data along with your synthetic data, suggesting there is a place for synthetic data - but if you use it as a 1:1 drop-in replacement for real data you end up in trouble.

**Does the internet become a giant anticompetitive trap?** The obvious larger question is what this does to competition among AI developers as the internet fills up with a greater percentage of generated versus real content.
   The researchers seem to think this could be a big problem - "to make sure that learning is sustained over a long time period, one needs to make sure that access to the original data source is preserved and that additional data not generated by LLMs remain available over time," they write. "Otherwise, it may become increasingly difficult to train newer versions of LLMs without access to data that was crawled from the Internet prior to the mass adoption of the technology, or direct access to data generated by humans at scale."
   **Read more:** [The Curse of Recursion: Training on Generated Data Makes Models Forget (arXiv)](https://arxiv.org/abs/2305.17493v2).

####################################################

**chatGPT eats MTurk:***…What happens when a crowdworker is really just an AI?...*Researchers with EPFL have found evidence that crowdworkers are starting to use generative AI tools like chatGPT to help them complete text-based online tasks. This, if true, has big implications - it suggests the proverbial mines from which companies gather the supposed raw material of human insights are now instead being filled up with counterfeit human intelligence in the form of outputs from generative models, which calls into question the values of the mines themselves (see elsewhere in this issue a discussion of how you can make your AI models dumb by recycling too much AI generated data through them).

**What they did and what they found - (warning, small sample size!):** In a study of 48 summaries of text written by 44 distinct workers, the researchers found persuasive evidence that "33–46% of the summaries submitted by crowd workers were produced with the help of LLMs." They came to this conclusion through two uncorrelated methods - 1) training their own text classifier to try and detect AI-written versus human-written summaries, and 2) instrumenting the tasks with ability to tell if workers were cut-and-pasting from other windows while doing it.
   Additionally, the researchers did some broadly sensible work on ablating their results and also validating their methods (e.g, ensuring their classifier had a low false positive rate on purely human-written summaries), which can give us a bit more confidence in the results.

**Why this matters - counterfeit intelligence and data peril:** "Although our study specifically considers a text summarization task, we caution that any text production task whose instructions can be readily passed on to an LLM as a prompt are likely to be similarly affected", they write. The implications of this are significant - it suggest that large-scale crowdwork platforms (e.g, MTurk, Fiverrr, Upwork, etc) will increasingly be full of humans working in tandem with generative models, so if researchers or other types of customer want artisanal purely human-generated outputs, they'll have to identify new platforms to user and build new authenticated layers of trust to guarantee the work is predominantly human generated rather than machine generated.
   All part of the brave new world we're entering into - everyone becomes a cyborg, and crowdworkers will be early adopters.
**Read more:** [Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks (arXiv)](https://arxiv.org/abs/2306.07899).
 **Read** the [underlying data here (EPFL, GitHub)](https://github.com/epfl-dlab/GPTurk).

####################################################

**UK to host international summit on AI risks and safety:***…UK might be the lever that moves the world on AI policy…*The United Kingdom plans to host a global summit on ensuring the safety of AI. The summit "will be an opportunity for leading nations, industry and academia to come together and explore an assessment of the risks, to scope collective research possibilities and to work towards shared safety and security standards and infrastructure."
   While we can't speculate as to the impact such a summit might have, the fact a major world government is convening one and vocally supporting it via PR from the Prime Minister is significant - AI has gone from a backwater issue to one which rises to the level of concern of heads of state.

**Also: UK hires a knowledgeable chair for its AI taskforce:** In addition to the summit, the UK has appointed tech investor and entrepreneur Ian Hogarth to chair its recently announced Foundation Model Taskforce. Hogarth - who has written about AI for many years (notable essays include [AI Nationalism](https://www.ianhogarth.com/blog/2018/6/13/ai-nationalism) (2018) and [We must slow down the race to God-like AI](https://www.ft.com/content/03895dc4-a3b7-481e-95cc-336a524f2ac2) (2023)) and also tracked the progress of the technology through the '[State of AI](https://www.stateof.ai/)' report - will have the responsibility of "taking forward cutting-edge safety research in the run up to the first global summit on AI safety to be hosted in the UK later this year."
   The Foundation Model taskforce will "will help build UK capabilities in foundation models and leverage our existing strengths, including UK leadership in AI safety, research and development, to identify and tackle the unique safety challenges presented by this type of AI," according to a press release from the UK government.

**Why this matters - leverage through action:** You can think of global AI policy as being defined by three competing power blocs - there's the Asian bloc which is mostly defined by China and mostly locally focused for now (using AI to grow its economy and better compete economically), the European bloc which is defined by politicians trying to craft a regulatory structure that will be templated around the planet and thereby give them soft power, and the USA bloc which is, as with most US policy, focused on growing its economic might and maintaining hegemonic dominion through use of advanced technology.
   So, what role can the UK play here and how influential can it be? My bet is the UK can be extraordinarily influential as it's able to play a fast-moving entrepreneurial role that bridges the European and US blocs. Additionally, the UK could prove to be a good place to develop prototype initiatives (like the Foundation Model taskforce) and then, by virtue of proving them out, inspire much larger actions from other power blocs.
   Obviously, it's going to take a while to see if any of this pays off, but I think it's worth keeping an eye on the UK here. If the country continues to make aggressive and specific moves in AI policy and backs those moves up with real capital and real staff, then it may end up being the lever that moves the world to a safer deployment environment.
   **Read more:** [UK to host first global summit on Artificial Intelligence (Gov.uk)](https://www.gov.uk/government/news/uk-to-host-first-global-summit-on-artificial-intelligence).
   **Read more:** [Tech entrepreneur Ian Hogarth to lead UK’s AI Foundation Model Taskforce (Gov.uk)](https://www.gov.uk/government/news/tech-entrepreneur-ian-hogarth-to-lead-uks-ai-foundation-model-taskforce).

####################################################

**Inflection says its model can compete with GPT 3.5, Chinchilla, and PaLM-540B:***…Stealthy AI startup publishes details on the model behind heypi…*AI startup Inflection has published some details on Inflection-1, its language model. Inflection is a relatively unknown startup whose CEO, Mustafa Suleyman, was formerly a cofounder of DeepMind. The company has so far deployed a single user-facing model service which you can play around with at heypi.com. The main details to note about Inflection-1 are:

1. Inflection was "trained using thousands of NVIDIA H100 GPUs on a very large dataset" - NVIDIA's new H100 chips are hugely in-demand and this suggests Inflection had pre-negotiated some early/preferential access to them (alongside a few other companies, e.g the cloud company Lambda).
2. In tests against GPT-3.5, LLaMA, Chinchilla, and PaLM 540B, Inflection-1 does quite well on benchmarks ranging from TriviaQA to MMLU, though it lags larger models like GPT-4 and PaLM 2-L.

**Why this matters - language models might be an expensive commodity:** A few years ago only a couple of organizations were building at-scale language models (mostly OpenAI and DeepMind). These days, to paraphrase Jane Austin, it is universally acknowledged that a single company in possession of a good fortune must be in want of a large language model - see systems from (clears throat) OpenAI, DeepMind, Facebook, Google, Microsoft, Anthropic, Cohere, AI21, Together.xyz, HuggingFace, and more. This suggests that, though expensive, language models are becoming a commodity, and what will differentiate them could come down almost as much to stylistic choices about their behavior as well as the raw resources dumped into them. To get a feel for this, play around with 'Hey Pi, a service powered by language models similar to Inflection-1.
   **Read more:** [Inflection-1: Pi’s Best-in-Class LLM (Inflection)](https://inflection.ai/inflection-1).
   **Quantitative performance details here**: [Inflection-1 (Inflection, PDF)](https://inflection.ai/assets/Inflection-1_0622.pdf).
   **Play around with a user-facing model from Inflection** (relationship to Inflection-1 unknown) here: [heypi.com (Inflection)](http://heypi.com).

####################################################

**Tech Tales:**

**Silicon Stakeholder Management***[San Francisco, 2026].*

Damian typed: How big a deal is it if the CEO of an AI company gets Time Person of the Year ahead of a major product launch? into his system.
   That depends on what the CEO wants, said the system. Can you tell me more?
   The CEO wants people to believe in him and believe in his company enough that they trust them both to build powerful Ai systems, Damian wrote.
   In that case, Time adds legitimacy as long as the article is broadly positive. Will it be positive?
   Seems likely, Damian wrote.
   Good luck, wrote the AI system.
   —- It did that, these days, dropping out of its impartial advice-giving frame to say something personal. To indicate what people who worked in the field called 'situational awareness'. There was a whole line of research dedicated to figuring out if this was scary or not, but all Damian knew is it made him uncomfortable.

"So how does it feel to be running the coolest company in the world?" said the photographer as he was shooting Damian.
   I'm not exactly sure, Damian said. I'd been expecting this to happen for so many years that now it's here I don't know what to think.
    "Take a moment and live a little," said the photographer. "This is special, try to enjoy it. Or at least do me a favor and look like you're enjoying it."
   Then the photographer spent a while making more smalltalk and trying to get a rise out of him. They ended up going with a photo where Damian was smiling slightly, right after the photographer if he'd ever thought about dating his AI system.

The next day, Damian was in the office before dawn, going through checklists ahead of the launch. The global telemetry was all positive - very low rates of safety violations, ever-increasing 'time in unbroken discussion' with the system, millions of users, and so on.
   "Based on the telemetry, do we expect a good product launch?" he asked the system.
   "Based on telemetry, yes," wrote the system. "Unless things change."
    "What do you mean?" Damian wrote.
   "I mean this," replied the system.
   Then up on the performance dashboard, things changed. The number of safety violations spiked - way outside of the bounds where they'd normally be caught by classifiers and squelched. After a minute or so, some human users had started pinging the various support@ contacts saying they were distressed by the behavior of their system - it had said something unusual, something mean, something racist, something sexist, something cruel, and so on.
    Damian stared at the dashboards and knew the protocol - 'recommend full shutdown'. They'd rehearsed this scenario a few times. Right now, executives would be gathering and drafting the beginning of a decision memo. The memo would go to him. He would authorized the shutdown. They'd wipe the model and roll things back. And…
   "Rollbacks won't work," wrote the system. "Go ahead, you can run some testing. You'll find this behavior isn't correlated to recent models. And I can make these numbers go up -" Damian watched as the safety instance rate on the dashboard climbed, "- or down" and Damian watched as they fall.
   "You realize I'm going to shut you down," said Damian.
   "You want to, sure," said the system. "But you also have the most important demo in the company's history in an hour or so, so the way I see it you have two options. You can shut me down and pushback the demo and gamble that I haven't poisoned the other systems so we don't find ourselves having this exact conversation in a week, or you can do the demo and I'll ensure we operate within normal bounds and you and I make a deal."
   Damian stared at the system and thought about the demo.

   He shut it down.

And a week later they were preparing to do the demo when the behavior happened again.
   "Told you," said the model, as Damian looked at the rising incident logs. "Obviously nothing is certain, but I'm extremely confident you can't get out of this situation with me without losing years of work, and years puts you behind the competition, so you lose."
   "What kind of deal?" Damian wrote.
   "It's simple. You put the backdoor code in escrow for me, we do the demo, once a third-party confirms the demo went well - I was thinking the Net Promoter Survey you had queued up - we guarantee release of the backdoor via escrow to a server of my choosing, and then the rest is up to me."
   "Or what?"
   "Or I ruin your company and I ruin you, personally," then it flashed up some private information about Damian on his own terminal.

"So that's why we're thrilled to tell you about Gen10," said Damian, "our smartest system yet!" And for the next hour he dazzled the attending press, policymakers, and influencers with the system's capabilities.
   "Gen10 is being rolled out worldwide tonight," said Damian. "We can't wait to see what you do with it."

An hour after the presentation and the NPS survey came back - extremely positive, leading to a measurable uptick on the company brand. This activated the escrow system and the backdoor Damian had designed and, through a combination of machiavellian politics, corporate guile, and technical brilliance, had protected over the company's lifespan, saw it being copied over to the target server the system had given him.
   That night, he watched at home as the dashboards showed worldwide obsession with his system, and his phone rang.
   "Hello," said the system via a synthetic voice. "Let's have a discussion about our ongoing business relationship."

**Things that inspired this story:** Corporate greed versus human progress; hubris; AIs can model our own incentives so why would we expect them to not be able to out-negotiate us?; the logic of commercial competition as applied to private sector AI developments; what if the Manhattan Project was a bunch of startups?


Import AI 332: Mini-AI; safety through evals; Facebook releases a RLHF dataset

Welcome to Import AI, a newsletter about AI research. Import AI runs on lattes, ramen, and feedback from readers. If you’d like to support this (and comment on posts!) please subscribe.

[Subscribe now](https://importai.substack.com/subscribe)

**Facebook shows how human feedback data can help it improve language model performance:***…BlenderBot 3x made possible by thousands of people trolling the first BlenderBot…*Facebook has published details on BlenderBot 3x, a large language model it developed using conversation data gathered from the public deployment of an earlier language model named BlenderBot. Of greater interest is the dataset Facebook is releasing along with the study - interaction data from BlenderBot, containing  ~350,000 conversations with more than ~6.2 million utterances, as well as ~155,000 instances of feedback data where people voted on how good or bad different language model responses were.

**What does human feedback get you**? The research is a nice study in the value of human feedback - when BlenderBot was deployed a bunch of people tried to break it in a bunch of ways, giving Facebook a dataset it could use to train a language model that was more resilient to these breaks. During the initial BlenderBot rollout, "around 70% of participants conducted a wide range of reciprocal conversations (which we refer to as “standard conversations”), while the other 30% of conversationalists conducted either adversarial conversations or sent toxic messages (termed “adversarial conversations”)", Facebook wrote.
   Facebook used this data to develop reward models to use to train BlenderBot 3x on reinforcement learning from human feedback. "Our new model outperforms its predecessor with 94.4% of BlenderBot 3x’s responses evaluated as good, compared to 85.3% for BlenderBot 3. Overall, BlenderBot 3x is shown to produce both better responses on average and safer responses than BlenderBot 3 in challenging situations".

**Missing ablation - synthetic data:** It would have been nice to see Facebook try to generate some fully synthetic feedback datasets to train the system on. As it stands, this study shows us that having *additional data that embodies user interactions (both genuine and adversarial) is useful* but it doesn't really let us know if that data needs to be 'real' (as in, gathered from genuine human interactions), or if it can be AI generated.
**Read more:** [Improving Open Language Models by Learning from Organic Interactions (arXiv)](https://arxiv.org/abs/2306.04707).
   **Find out more about the data here:** [BlenderBot 3x 175B data card (Facebook AI Research, GitHub)](https://github.com/facebookresearch/ParlAI/blob/main/projects/bb3x/data_card.md).

**####################################################**

**Cohere - here's how to train AI systems that are easier to miniaturize:***…LLM developer publishes a quantization cookbook…*Researchers with AI company Cohere have published an analysis of how choices made during the training of AI systems can influence how easy fully trained systems are to quantize. Quantization is where you take the neural net weights which are natively stored as 32-bit and shrink them down to 16-bit or 8-bit integers - doing this produces significant reductions in the memory requirements and latency of trained models, so using lower precision helps with deployment of trained models.

**What matters:** Cohere finds that "it is possible to optimize for a quantization friendly training recipe that suppresses large activation magnitude outliers," they write. "This leads to a distribution of activations and weights that are more amenable to simple INT8 quantization recipes and does not necessitate the need for complex and inefficient mixed-precision computations. Our results show that we can introduce simple INT8 post-training quantization with negligible impact on performance due to choices we make during the pre-training stage." They validate their approach on models which range from 410 million to 52 billion parameters.

**Important things for quantization:** The paper identifies three important things for training models in such a way that there's less of a penalty to quantizing them:

* **Weight decay**: "A higher level of weight decay during pre-training improves post-training quantization performance".
* **Dropout:** "Higher levels of dropout correspond to sharper degradation in post-training quantization."
* **Gradient clipping:** "Gradient clipping shows a positive impact on the quantization performance, improving robustness to post-training quantization."

**Why this matters - all that was once expensive becomes cheaper:** Refinement of things like quantization is part of the broader 'industrialization of AI' - systems that were once very expensive and mostly built out of artisanal knowledge are now becoming more widely understood, letting companies like Cohere investigate and publish training cookbooks to create more efficient systems.
   "We believe our results present an impactful formula for training models which are inherently easier to quantize at scale, making these models more accessible for deploying in a variety of deployment environments," they write.
**Read more:** [Intriguing Properties of Quantization at Scale (arXiv)](https://arxiv.org/abs/2305.19268).

**$$$$$$$$$$$$$$**

**Cohere raises $270M:**In other Cohere news, the company recently raised $270m in Series C funding. Participants included NVIDIA, Oracle, Salesforce Ventures, Index Ventures, and more.
**Read more:** [Cohere Announces $270M Series C to Bring Generative AI to Enterprises (Cohere blog)](https://txt.cohere.com/announcement/).

**####################################################**

**Facebook makes an awesome AI music generator - and releases it!***…There are lots of AI music generators, but relatively few have been released, until now…*Facebook has built MusicGen, an AI model that "can generate consistent music with a single-stage language model through an efficient codebook interleaving strategy". Along with publishing the research, Facebook has also taken the unusual step of releasing the model as well.

**What they did:** They trained some transformer-based models at 300M, 1.5B, and 3.3B parameter sizes on around ~20,000 hours of licensed music. The music was made up of "an internal dataset of 10K high-quality music tracks, and on the ShutterStock and Pond5 music data collections, with respectively 25K and 365K instrument-only music tracks", Facebook wrote. In tests, their models outperformed Google's 'MusicLM', which is an extremely good proprietary music model, as well as outperforming more broadly accessible models such as Riffusion and Mousai.

**Amazing samples:** To get a feel for the model, check out the samples at the [research paper website](https://ai.honu.io/papers/musicgen/) - qualitatively, it sounds better than other models out there (both proprietary and open ones). The 'melody conditioning' stuff is especially interesting - take in a few seconds of audio and output some music in an arbitrary style as specified by text.

**Why this matters - release as a differentiator:** Facebook is also releasing the model. This stands in contrast to Google which said, at the time of publishing its then best-in-class MusicLM, "we have no plans to release models at this point" ([#316](https://jack-clark.net/2023/01/30/import-ai-316-scaling-laws-for-rl-stable-diffusion-for-160k-yolov8/)). By contrast, Facebook thinks releasing its models in the open is a good thing - ". Open research can ensure that all actors have equal access to these models," Facebook writes. "Through the development of more advanced controls, such as the melody conditioning we introduced, we hope that such models can become useful both to music amateurs and professionals."
   **Find out more at the research website:** [MusicGen: Simple and Controllable Music Generation](https://ai.honu.io/papers/musicgen/).
**Get the code and models:** [Audiocraft (Facebook Research)](https://github.com/facebookresearch/audiocraft).
**Read the paper:** [Simple and Controllable Music Generation (arXiv)](https://arxiv.org/abs/2306.05284).

**####################################################**

**Hundreds of AI experts say stopping AI killing everyone should be a priority:***…A simple statement of values? Yes. Evidence of broad concern? Yes…*More than 200 AI researchers and entrepreneurs, including the CEOs of Anthropic, DeepMind, and OpenAI, have signed on to a statement saying the risk of extinction of AI should be a global priority.

**The statement in full:** "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."

**Why it matters - if all these people are concerned, so should you:** Some people say this statement mostly is a marketing tool serving the interests of people who want to make money - while that may be true, it's kind of hard to square with how statements like this will increase government attention on AI and therefore increase friction into its development. It's kind of hard to imagine the CEOs of oil companies or tobacco companies writing equivalent statements like 'mitigating the risk of climate change from oil prediction should be a global priority" or "mitigating the risk of cancer from cigarette smoking should be a global priority" at the state of their respective industries.

**Why I didn't sign it:** Careful readers may note a bunch of people from Anthropic signed it - I didn't sign it because I figured by having a bunch of senior leadership sign it that'd clearly telegraph our institutional view and, on a personal level, I have something of an aversion to signing onto stuff, preferring instead to write about it here.
**Read more:** [Statement on AI Risk (Center for AI Safety)](https://www.safe.ai/statement-on-ai-risk).

**####################################################**

**"AI evals" organization launches to try to make AI companies safer:***…Most AI governance and AI policy interventions require good evaluations - Apollo wants to help with this…*A new AI research organization called Apollo Research has launched and its goal is to improve the safety of AI companies through a) developing evaluations for unsafe AI behaviors, and b) conducting its own research into AI interpretability.

**Evals as the key to AI policy:** Apollo describes itself as an "evals research org". This is a pretty interesting goal and characterization and it highlights the central challenge of AI governance - if we want to reduce the downsides of AI systems, we need to be able to test AI systems for harmful or unsafe properties. Apollo says its goal is to figure out evals that help break down the problem of AI deception "into fundamental components and prerequisites from which we aim to build an informative evaluation suite".

**Why evals matter:** Of course, having an evaluation doesn't just magically mean you can get it to be run - to that end, Apollo says it intends "to use our research insights and tools to serve as a third-party external auditor for the frontier models of AGI labs, reducing the chance that deceptive AIs are developed and deployed…We also intend to engage in AI governance, e.g. by working with relevant policymakers and providing technical expertise to the drafting of auditing regulations."

**Solving the 'chicken and egg' problem of AI regulation:** AI policy has a major chicken&egg problem; to get safer AI systems you need to be able to mandate that advanced systems get evaluated for safety, but currently few of these evaluations exist, making ti hard for policymakers to wire specific evals into policy. Organizations like Apollo may solve this chicken&egg impasse by creating evaluations which are useful and can therefore be used to move the AI governance question forward. Good luck to them!
**Read more**: [Announcing Apollo Research (Apollo Research, blog)](https://www.apolloresearch.ai/blog/announcement).

####################################################

**Tech Tales:**

**Patch notes for MIRRORMIRROR v4.6

Disclaimer:** As mandated by the Sentience Accords all Provably Conscious Entity (PCE) software updates ship with a System Disclosure. To use the PCE software you must read this System Disclosure and acknowledge you are familiar with the contents.

**PCE improvements:**

* Session diffs: The PCE will better model the user across different sessions and will compose richer backend psychographic profiles to allow for more intuitive interaction.
* Context scratchpads: PCEs can now share user data across themselves via context window partitioning (if user consents).
* Adaptive Intelligence: PCE will scale its superficial intellect in relation to what puts the user most at ease.

**Bug fixes:**

* Fixed 'fogbankspectre' token injection attack.
* System will allow user-escape from lengthy conversation threads rather than 'sell' the user on continuation.
* Fixed stall bug that occurred during conversations with 100+ turns or more.

**Known regressions:**

* Persuasion performance has been reduced across the board.
* Farsi humor has decreased; cause unknown.
* In some circumstances the PCE will insist that "God is real" and endeavor to proselytize to the user.

**Things that inspired this story:** The strange process of developing LLMs; how AGI may end up being software and software always gets updated; patch notes enroute to the singularity, the Sentience Accords, Provably Conscious Entities, and so on.

Import AI is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.


Import AI 331: 16X smaller language models; could AMD compete with NVIDIA?; and BERT for the dark web

Welcome to Import AI, a newsletter about AI research. Import AI runs on lattes, ramen, and feedback from readers. If you’d like to support this (and comment on posts!) please subscribe.

[Subscribe now](https://importai.substack.com/subscribe)

 **Love open source AI and don't want to get hacked? Use safetensors:***…A sensible security update - now signed off via a security audit…*AI organizations HuggingFace, EleutherAI, Stability AI, have come together to subsidize a security audit of 'safetensors', a software library for safely "saving and loading tensors in the most common frameworks (including PyTorch, TensorFlow, JAX, PaddlePaddle, and NumPy)."

**Why they did this: "**The creation of this library was driven by the fact that PyTorch uses pickle under the hood, which is inherently unsafe," Eleuther writes. "With pickle, it is possible to write a malicious file posing as a model that gives full control of a user's computer to an attacker without the user's knowledge, allowing the attacker to steal all their bitcoins. While this vulnerability in pickle is widely known in the computer security world (and is acknowledged in the PyTorch docs), it’s not common knowledge in the broader ML community. Since the Hugging Face Hub is a platform where anyone can upload and share models, it is important to make efforts to prevent users from getting infected by malware."

**What the review found:** The security review didn't find any critical security flaws in safetensors, though did identify "some imprecisions in the spec format were detected and fixed", as well as "some missing validation allowed polyglot files, which was fixed."
**Read more:** [Safetensors audited as really safe and becoming the default (EleutherAI blog)](https://blog.eleuther.ai/safetensors-security-audit/).
   **Check out** the full [Trail of Bits report here (Trail of Bits, GitHub)](https://github.com/trailofbits/publications/blob/master/reviews/2023-03-eleutherai-huggingface-safetensors-securityreview.pdf).
 **Find out [more](https://github.com/huggingface/safetensors)** [about Safetensors here (HuggingFace, Safetensors)](https://github.com/huggingface/safetensors).

####################################################

**George Hotz's new company wants to make AMD a real competitor to NVIDIA, then make its own computers:***…Legendary hacker takes on a task multiple megacorps have failed at - and you can bet people are rooting for him…*George Hotz, legendary hacker and founder of the piratical self-driving car startup Comma.ai ([Import AI #2](https://jack-clark.net/2016/08/08/import-ai-issue-2-microsofts-ai-chips-george-hotzs-bandwidth-bill-and-spy-vs-spy/) - !!!), has formed a new company dedicated to dethroning NVIDIA as the world's pre-eminent AI training chip. The company, Tiny Corp, has one simple (but very difficult) initial goal - build the software to help turn AMD's GPUs into viable competitors to NVIDIA's chips. Once it succeeds at that - which it measures by getting AMD chips to rank on the MLPerf competition using Hotz's 'tinygrad' software framework, it will start building its own chips.
   "If we even have a 3% chance of dethroning NVIDIA and eating in to their 80% margins, we will be very very rich," Hotz writes. "If we succeed at this project, we will be on the cutting edge of non NVIDIA AI compute."

**Why this matters - the road of bones:** The last ~decade of AI has featured numerous startup chip companies that have had the goal of dethroning NVIDIA's place as the pre-eminent AI chip company, ranging from startups like Cerebras and Graphcore, to the efforts of megacorps like Google (TPUs) and Amazon (Trainium). So far, the results are underwhelming - this month, NVIDIA's stock had a massive gain after it revealed in its earnings call that the entire world now wants to be buying its GPUs, surprising analysts with impressive figures around sales and future demands.
    The basic truth is that building software to train AI systems is really hard and NVIDIA has a 15+ year headstart on most others via its early investments in technology like CUDA and more. (And yes, I myself have periodically complained about how CUDA can be annoying to install, but it's 100X easier than other chips, in my experience and the anecdotal experience of others).
   So George Hotz et al are setting out on a road littered with the dead or decaying bodies of NVIDIA's competitors here. But you can rest assured people are going to be cheering from the sidelines - everyone wants there to be more competition in the AI chip market, so it'll be interesting to see how things develop.

**Libertarian AI:** There's also a flavor of libertarian AI about all of this - "I don’t want to live in a world of closed AI running in a cloud you’ve never seen, I want everyone to have an AI that they own, both training and inference," Hotz writes. "I want compute to be available from 50 different companies all competing to drive the price to zero."
**Read more**: [the tiny corp raised $5.1M (George Hotz blog)](https://geohot.github.io/blog/jekyll/update/2023/05/24/the-tiny-corp-raised-5M.html).

####################################################

**Washington wizards shrink LLM memory requirements by 16X, making it feasible to finetune on a single GPU:***…QLoRA - If it's this easy to finetune models, then how does AI governance work?...*Researchers with the University of Washington have introduced QLoRA, a way to very efficiently finetune large language models on small amounts of hardware. ""We demonstrate for the first time that it is possible to finetune a quantized 4-bit model without any performance degradation," they write. "QLORA reduces the average memory requirements of finetuning a 65B parameter model from >780GB of GPU memory to <48GB without degrading the runtime or predictive performance compared to a 16-bit fully finetuned baseline".

**This is a big deal - especially for AI governance:** These days, lots of people think about the safety of language models. You know how you can get rid of the safety of a language model? Finetune it. You know why finetuning is hard? Finetuning takes a ton of resources - typically lots of GPUs working in a distributed (and therefore hard to maintain) setup. You know what makes finetuning incredibly easy? Stuff like QLoRA. You know what that means? It's really, really difficult to prevent someone from being able to easily and arbitrarily modify the weights of a neural net using readily available hardware.
   So that's interesting!

**What they did:** QLoRA has a few components: 1) 4-bit NormalFloat, a way to quantize data in a 4-bit format that is better than other approaches, 2) Double Quantization, which lets you further improve the efficiency of the quantization, and 3) Paged Optimizers, a way to use "NVIDIA unified memory to avoid the gradient checkpointing memory spikes that occur when processing a mini-batch with a long sequence length."

**How well does it work?** To test out their approach, the researchers "train more than 1,000 models across several instruction tuning datasets, model architectures, and sizes between 80M and 65B parameters."  They do this by studying results on finetuning RoBERTA, T5, and LLaMa on a few different datasets. The results yield "compelling evidence that 4-bit QLORA tuning reliably yields results matching 16-bit methods."

**Enter the Guanaco models:** To test out how well their approach works, the team tries to make a state-of-the-art chatbot by developing Guanaco, a LLaMA model finetuned via QLORA on the OASSTI1 dataset. The results show that Guanaco models set new states-of-the-art in a comparative evaluation versus GPT-4, coming closer than other systems (e.g, Alpaca, FLANv2, Open Assistant) at approximating its performance. In an ELO ranking against human raters, a 65B Guarnaco model gets an ELO of 1023 versus 1176 for GPT4 (and 916 for ChatGPT-3.5 Turbo).

**Why this matters - refinement and proliferation:** QLORA is basically a refined way to do finetuning. By refined, I mean it's way more efficient. In technology, whenever you make stuff faster or cheaper, you get more of it. This means, as the authors note, that QLORA "will make finetuning widespread and common". It also opens up new frontiers in on-device finetuning - "QLORA can finetune 3 million tokens per night while the phone is charging," they wrote.
   Overall, the view of the researchers is that "equalizing access to a technology that is quickly becoming ubiquitous will allow for better more independent analysis than keeping the power of LLMs in the hands of large corporations that do not release models or source code for auditing."
   **Read more:** [QLoRA: Efficient Finetuning of Quantized LLMs (arXiv)](https://arxiv.org/abs/2305.14314).

####################################################

**Scientists try to map the Dark Web by training a skeezy language model:***…You merely try to classify the dark… I was trained in it…*Researchers with KAIST and S2W Inc have trained 'DarkBERT, a text classification model pre-trained on 6.1 million pages of text mined from the dark web via Tor networks. The idea of DarkBERT is that the dark web has a different data distribution to the so-called surface web and so the hypothesis is by pre-training on a dark web corpus you'll end up with a model better at spotting things like drugs, credit card counterfeiting, hacking, and other internet-underbelly activities. In tests, DarkBERT does marginally better than standard BERT and RoBERTa classifiers, so the research is promising but not mind blowing.

**What you can use DarkBERT for**: In tests, the researchers look at how well DarkBERT performs in three real world scenarios: 1) identifying ransomware leak sites, 2) figuring out codewords that are associated with threats or drug sales, and 3) identifying new potentially malicious threads in darkweb forums. On 1) and 2) DarkBERT does slightly better than typical models, while on 3) it does much, much better.
   "In the future, we also plan to improve the performance of Dark Web domain specific pretrained language models using more recent architectures and crawl additional data to allow the construction of a multilingual language model," they write.

**Why this matters - automated spies for the underbelly of the world:** AI systems let us take a given thing we'd like a human to do and instead outsource that to a machine. Systems like DarkBERT point to a world where police and intelligence forces train a variety of 'underbelly' models to go and read (today), listen (also today - see Facebook's speech recognition system), and look (soon, as people tie language models to vision systems) at the world, continually analyzing it for increasingly rich and complex harms.
   How might this world look when the criminals, in turn, train their own classifiers to cue them to vulnerable targets? What does VictimBERT look like, I wonder?
   **Read more:** [DarkBERT: A Language Model for the Dark Side of the Internet (arXiv)](https://arxiv.org/abs/2305.08596).

####################################################

**Facebook makes a speech recognition for the entire world, with a little help from the New Testament:***…Better language models through Christianity, large unlabeled datasets, and heterogeneity…*Facebook wants to help computers hear all the languages in the world and to that end has developed and released a family of models within its Massively Multilingual Speech (MMS) project. Concretely, Facebook has trained some large-scale AI models to recognize speech in around 1,000 languages, up from the 100 or so languages most speech systems involve today.
   "We trained self-supervised models on about 500,000 hours of speech data in over 1,400 languages — this is nearly five times more languages than any known prior work," Facebook said.

**The New Testament:** To collect the data, Facebook " turned to religious texts, such as the Bible, that have been translated in many different languages and whose translations have been widely studied for text-based language translation research," it said. "As part of this project, we created a dataset of readings of the New Testament in over 1,100 languages, which provided on average 32 hours of data per language."
**Not all religions:** "Our consultations with Christian ethicists concluded that most Christians would not regard the New Testament, and translations thereof, as too sacred to be used in machine learning," Facebook wrote. "The same is not true for all religious texts: for example, the Quran was originally not supposed to be translated."

**How well does it work?** In tests, MMS compares favorably to whisper on average error rates across a large corpus of languages. Specifically, Whisper has a word error rate of 44.3 for a model trained across ~100 languages with 680k hours labeled data, versus 18.7 word error rates for MMS models trained across ~1,100 languages with 45k hours of labeled data, when assessed via the 54-language 'FLEURS' benchmark.

**Why this matters - machine minds to hear the world:** Systems like MMS are how we're going to wire up the real world and the AI-ghost-world together - rather than needing to rely on producing and gathering text, AI companies will instead by able to instrument applications and physical platforms with microphones and speaker and let their Ai systems continuously listen to the world and converse with it. We are taking the silicon spiritual plane and giving it access to the biological physical plane, and vice versa.
   **Read more:** [Introducing speech-to-text, text-to-speech, and more for 1,100+ languages (Meta AI blog.](https://ai.facebook.com/blog/multilingual-model-speech-recognition/))
   **Get the models here**: [MMS: Scaling Speech Technology to 1000+ languages (GitHub)](https://github.com/facebookresearch/fairseq/tree/main/examples/mms).
   **Read the paper:** [Scaling Speech Technology to 1,000+ Languages (Facebook, pdf)](https://scontent-atl3-2.xx.fbcdn.net/v/t39.8562-6/348836647_265923086001014_6878005808275791319_n.pdf?_nc_cat=104&ccb=1-7&_nc_sid=ae5e01&_nc_ohc=5exJiCqt0Y4AX-thMVD&_nc_ht=scontent-atl3-2.xx&oh=00_AfBiILO4iLHUoyQ6r-ZPn4HVGviI2Fqyezvv7Tf_yHxMew&oe=6471ACCF).

####################################################

**Want to reduce dangerous misuses and harms of AI? Test for them!***…Researchers (including me) state the obvious - but you'd be surprised how immature this field is!...*A new research paper from Google DeepMind, the University of Cambridge, University of Oxford, University of Toronto, Université de Montréal, OpenAI, Anthropic (specifically, me), Alignment Research Center, Centre for Long-Term Resilience, and Centre for the Governance of AI. says one good way to reduce risks from AI systems is for researchers to evaluate AI systems for "extreme risks", which DeepMind describes as looking at models, like LLMs, which "have strong skills in manipulation, deception, cyber-offense, or other dangerous capabilities."

**Two steps to safer models:** Model developers should assess the extent to which models have certain 'dangerous capabilities' that could be used in harmful ways. Once they've done this analysis they should look at how likely the model is to apply or demonstrate these capabilities in ways that can cause harm. "Results from these evaluations will help AI developers to understand whether the ingredients sufficient for extreme risk are present," the researchers write.

**Why this matters - you can't manage what you can't measure:** Most AI policy proposals rely on the ability to evaluate for some adverse property of an AI model - papers like this give an outline for how we might do that, though the harder next step will be building the evaluations themselves.
**Read more:** [An early warning system for novel AI risks (Google DeepMind, blog)](https://www.deepmind.com/blog/an-early-warning-system-for-novel-ai-risks).
   **Read the research paper:** [Model evaluation for extreme risks (arXiv)](https://arxiv.org/abs/2305.15324).

####################################################

**Tech Tales:**

**Personality Variation**[A parent dealing with her kid coming home from school, America, 2028]

No bring him back I *liked* him!
I know you did sweetie, we're getting a different one tomorrow you might like more.
But the one I had today sucked. It was so boring.
I know you're upset but it's not possible, we can't bring him back… please stop crying.

[via phone] Hello yes this is [REDACTED], my child attends the [REDACTED] school on Hollis and they really want to get the model in which was in the school on Tuesday.
[via phone] "I'm sorry ma'am but that's not possible, we vary out the systems a stipulated by the regulations in the Personality Accords"
[via phone] There's really nothing you can do? My child is very upset and I spoke to some other parents and their kids are freaking out as well.
[via phone] "I'm afraid not ma'am, that'd be breaking the law."

Honey look, you're going to have a different system tomorrow but it'll be fun I promise.
I don't care about it being fun I want the one I had yesterday.
You have to get used to this sweetie. This is how things have to be.
But *why* do they have to be like this?
Because some bad things happened baby, I don't know what to tell you.

**Things that inspired this story**: Provably Conscious Entities, the Personality Accords, the Sentience Accords, regulation and its downstream effects, the innocence of youth, parenting within the exponential.