Linear Control Theory: Part 0

The purpose of this post is to introduce you to some of the basics of control theory and to introduce the [Linear-Quadratic Regulator](http://en.wikipedia.org/wiki/Linear-quadratic_regulator), an extremely good hammer for solving stabilization problems.

To start with, what do we mean by a control problem? We mean that we have some system with dynamics described by an equation of the form

![\dot{x} = Ax,](https://s0.wp.com/latex.php?latex=%5Cdot%7Bx%7D+%3D+Ax%2C&bg=f0f0f0&fg=555555&s=0&c=20201002)

where ![x](https://s0.wp.com/latex.php?latex=x&bg=f0f0f0&fg=555555&s=0&c=20201002) is the state of the system and ![A](https://s0.wp.com/latex.php?latex=A&bg=f0f0f0&fg=555555&s=0&c=20201002) is some matrix (which itself is allowed to depend on ![x](https://s0.wp.com/latex.php?latex=x&bg=f0f0f0&fg=555555&s=0&c=20201002)). For example, we could have an object that is constrained to move in a line along a frictionless surface. In this case, the system dynamics would be

![\left[ \begin{array}{c} \dot{q} \\ \ddot{q} \end{array} \right] = \left[ \begin{array}{cc} 0 & 1 \\ 0 & 0 \end{array} \right]\left[ \begin{array}{c} q \\ \dot{q} \end{array} \right]. ](https://s0.wp.com/latex.php?latex=%5Cleft%5B+%5Cbegin%7Barray%7D%7Bc%7D+%5Cdot%7Bq%7D+%5C%5C+%5Cddot%7Bq%7D+%5Cend%7Barray%7D+%5Cright%5D+%3D+%5Cleft%5B+%5Cbegin%7Barray%7D%7Bcc%7D+0+%26+1+%5C%5C+0+%26+0+%5Cend%7Barray%7D+%5Cright%5D%5Cleft%5B+%5Cbegin%7Barray%7D%7Bc%7D+q+%5C%5C+%5Cdot%7Bq%7D+%5Cend%7Barray%7D+%5Cright%5D.+&bg=f0f0f0&fg=555555&s=0&c=20201002)

Here ![q](https://s0.wp.com/latex.php?latex=q&bg=f0f0f0&fg=555555&s=0&c=20201002) represents the position of the object, and ![\dot{q}](https://s0.wp.com/latex.php?latex=%5Cdot%7Bq%7D&bg=f0f0f0&fg=555555&s=0&c=20201002) represents the velocity (which is a relevant component of the state, since we need it to fully determine the future behaviour of the system). If there was drag, then we could instead have the following equation of motion:

![\left[ \begin{array}{c} \dot{q} \\  \ddot{q} \end{array} \right] = \left[ \begin{array}{cc} 0 & 1 \\ 0  & -b \end{array} \right]\left[ \begin{array}{c} q \\ \dot{q}  \end{array} \right], ](https://s0.wp.com/latex.php?latex=%5Cleft%5B+%5Cbegin%7Barray%7D%7Bc%7D+%5Cdot%7Bq%7D+%5C%5C++%5Cddot%7Bq%7D+%5Cend%7Barray%7D+%5Cright%5D+%3D+%5Cleft%5B+%5Cbegin%7Barray%7D%7Bcc%7D+0+%26+1+%5C%5C+0++%26+-b+%5Cend%7Barray%7D+%5Cright%5D%5Cleft%5B+%5Cbegin%7Barray%7D%7Bc%7D+q+%5C%5C+%5Cdot%7Bq%7D++%5Cend%7Barray%7D+%5Cright%5D%2C+&bg=f0f0f0&fg=555555&s=0&c=20201002)

where ![b](https://s0.wp.com/latex.php?latex=b&bg=f0f0f0&fg=555555&s=0&c=20201002) is the coefficient of drag.

If you think a bit about the form of these equations, you will realize that it is both redundant and not fully general. The form is redundant because ![A](https://s0.wp.com/latex.php?latex=A&bg=f0f0f0&fg=555555&s=0&c=20201002) can be an arbitrary function of ![x](https://s0.wp.com/latex.php?latex=x&bg=f0f0f0&fg=555555&s=0&c=20201002), yet it also acts on ![x](https://s0.wp.com/latex.php?latex=x&bg=f0f0f0&fg=555555&s=0&c=20201002) as an argument, so the equation ![\ddot{q} = q\dot{q}](https://s0.wp.com/latex.php?latex=%5Cddot%7Bq%7D+%3D+q%5Cdot%7Bq%7D&bg=f0f0f0&fg=555555&s=0&c=20201002), for example, could be written as

![\left[ \begin{array}{c} \dot{q} \\ \ddot{q} \end{array} \right] = \left[ \begin{array}{cc} 0 & 1 \\ \alpha \dot{q} & (1-\alpha) q \end{array} \right] \left[ \begin{array}{c} q \\ \dot{q} \end{array} \right]](https://s0.wp.com/latex.php?latex=%5Cleft%5B+%5Cbegin%7Barray%7D%7Bc%7D+%5Cdot%7Bq%7D+%5C%5C+%5Cddot%7Bq%7D+%5Cend%7Barray%7D+%5Cright%5D+%3D+%5Cleft%5B+%5Cbegin%7Barray%7D%7Bcc%7D+0+%26+1+%5C%5C+%5Calpha+%5Cdot%7Bq%7D+%26+%281-%5Calpha%29+q+%5Cend%7Barray%7D+%5Cright%5D+%5Cleft%5B+%5Cbegin%7Barray%7D%7Bc%7D+q+%5C%5C+%5Cdot%7Bq%7D+%5Cend%7Barray%7D+%5Cright%5D&bg=f0f0f0&fg=555555&s=0&c=20201002)

for any choice of ![\alpha](https://s0.wp.com/latex.php?latex=%5Calpha&bg=f0f0f0&fg=555555&s=0&c=20201002). On the other hand, this form is also not fully general, since ![x = 0](https://s0.wp.com/latex.php?latex=x+%3D+0&bg=f0f0f0&fg=555555&s=0&c=20201002) will always be a fixed point of the system. (We could in principle fix this by making ![\dot{x}](https://s0.wp.com/latex.php?latex=%5Cdot%7Bx%7D&bg=f0f0f0&fg=555555&s=0&c=20201002) affine, rather than linear, in ![x](https://s0.wp.com/latex.php?latex=x&bg=f0f0f0&fg=555555&s=0&c=20201002), but for now we’ll use the form given here.)

So, if this representation doesn’t uniquely describe most systems, and can’t describe other systems, why do we use it? The answer is that, for most systems arising in classical mechanics, the equations naturally take on this form (I think there is a deeper reason for this coming from Lagrangian mechanics, but I don’t yet understand it).

Another thing you might notice is that in both of the examples above, ![x](https://s0.wp.com/latex.php?latex=x&bg=f0f0f0&fg=555555&s=0&c=20201002) was of the form ![\left[ \begin{array}{c} q \\ \dot{q} \end{array} \right]](https://s0.wp.com/latex.php?latex=%5Cleft%5B+%5Cbegin%7Barray%7D%7Bc%7D+q+%5C%5C+%5Cdot%7Bq%7D+%5Cend%7Barray%7D+%5Cright%5D&bg=f0f0f0&fg=555555&s=0&c=20201002). This is another common phenomenon (although ![q](https://s0.wp.com/latex.php?latex=q&bg=f0f0f0&fg=555555&s=0&c=20201002) and ![\dot{q}](https://s0.wp.com/latex.php?latex=%5Cdot%7Bq%7D&bg=f0f0f0&fg=555555&s=0&c=20201002) may be vectors instead of scalars in general), owing to the fact that Newtonian mechanics produces second-order systems, and so we care about both the position and velocity of the system.

So, now we have a mathematical formulation, as well as some notation, for what we mean by the equations of motion of a system. We still haven’t gotten to what we mean by control. What we mean is that we assume that, in addition to the system state ![x](https://s0.wp.com/latex.php?latex=x&bg=f0f0f0&fg=555555&s=0&c=20201002), we have a control input ![u](https://s0.wp.com/latex.php?latex=u&bg=f0f0f0&fg=555555&s=0&c=20201002) (usually we can choose ![u](https://s0.wp.com/latex.php?latex=u&bg=f0f0f0&fg=555555&s=0&c=20201002) independently from ![x](https://s0.wp.com/latex.php?latex=x&bg=f0f0f0&fg=555555&s=0&c=20201002)),  such that the actual equations of motion satisfy

![\dot{x} = Ax+Bu,](https://s0.wp.com/latex.php?latex=%5Cdot%7Bx%7D+%3D+Ax%2BBu%2C&bg=f0f0f0&fg=555555&s=0&c=20201002)

where again, ![A](https://s0.wp.com/latex.php?latex=A&bg=f0f0f0&fg=555555&s=0&c=20201002) and ![B](https://s0.wp.com/latex.php?latex=B&bg=f0f0f0&fg=555555&s=0&c=20201002) can both depend on ![x](https://s0.wp.com/latex.php?latex=x&bg=f0f0f0&fg=555555&s=0&c=20201002). What this really means physically is that, for any configuration of the system, we can choose a control input ![u](https://s0.wp.com/latex.php?latex=u&bg=f0f0f0&fg=555555&s=0&c=20201002), and ![u](https://s0.wp.com/latex.php?latex=u&bg=f0f0f0&fg=555555&s=0&c=20201002) will affect the instantaneous change in state in a linear manner. We normally call each of the entries of ![u](https://s0.wp.com/latex.php?latex=u&bg=f0f0f0&fg=555555&s=0&c=20201002) a torque.

The assumption of linearity might seem strong, but it is again true for most systems, in the sense that a linear increase in a given torque will induce a linear response in the kinematics of the system. But note that this is only true once we talk about mechanical torques. If we think of a control input as an electrical signal, then the system will usually respond non-linearly with respect to the signal. This is simply because the [actuator](http://en.wikipedia.org/wiki/Actuator) itself provides a force that is non-linear with its electrical input.

We can deal with this either by saying that we only care about a local model, and the actuator response is locally linear to its input; or, we can say that the problem of controlling the actuator itself is a disjoint problem that we will let someone worry about. In either case, I will shamelessly use the assumption that the system response is linear in the control input.

So, now we have a general form for equations of motion with a control input. The general goal of a control problem is to pick a function ![f(x,t)](https://s0.wp.com/latex.php?latex=f%28x%2Ct%29&bg=f0f0f0&fg=555555&s=0&c=20201002) such that if we let ![u = f(x,t)](https://s0.wp.com/latex.php?latex=u+%3D+f%28x%2Ct%29&bg=f0f0f0&fg=555555&s=0&c=20201002) then the trajectory $X(t)$ induced by the equation ![\dot{x} = Ax+Bf(x,t)](https://s0.wp.com/latex.php?latex=%5Cdot%7Bx%7D+%3D+Ax%2BBf%28x%2Ct%29&bg=f0f0f0&fg=555555&s=0&c=20201002) minimizes some objective function ![J(X,f)](https://s0.wp.com/latex.php?latex=J%28X%2Cf%29&bg=f0f0f0&fg=555555&s=0&c=20201002). Sometimes our goals are more modest and we really just want to get to some final state, in which case we can make ![J](https://s0.wp.com/latex.php?latex=J&bg=f0f0f0&fg=555555&s=0&c=20201002) just be a function of the final state that assigns a score based on how close we end up to the target state. We might also have hard constraints on ![u](https://s0.wp.com/latex.php?latex=u&bg=f0f0f0&fg=555555&s=0&c=20201002) (because our actuators can only produce a finite amount of torque), in which case we can make ![J](https://s0.wp.com/latex.php?latex=J&bg=f0f0f0&fg=555555&s=0&c=20201002) assign an infinite penalty to any ![f](https://s0.wp.com/latex.php?latex=f&bg=f0f0f0&fg=555555&s=0&c=20201002) that violates these constraints.

As an examples, let’s return to our first example of an object moving in a straight line. This time we will say that ![\left[ \begin{array}{c} \dot{q} \\ \ddot{q} \end{array} \right] = \left[ \begin{array}{cc} 0 & 1 \\ 0 & 0 \end{array} \right] \left[ \begin{array}{c} q \\ \dot{q} \end{array} \right]+\left[ \begin{array}{c} 0 \\ 1 \end{array} \right]u](https://s0.wp.com/latex.php?latex=%5Cleft%5B+%5Cbegin%7Barray%7D%7Bc%7D+%5Cdot%7Bq%7D+%5C%5C+%5Cddot%7Bq%7D+%5Cend%7Barray%7D+%5Cright%5D+%3D+%5Cleft%5B+%5Cbegin%7Barray%7D%7Bcc%7D+0+%26+1+%5C%5C+0+%26+0+%5Cend%7Barray%7D+%5Cright%5D+%5Cleft%5B+%5Cbegin%7Barray%7D%7Bc%7D+q+%5C%5C+%5Cdot%7Bq%7D+%5Cend%7Barray%7D+%5Cright%5D%2B%5Cleft%5B+%5Cbegin%7Barray%7D%7Bc%7D+0+%5C%5C+1+%5Cend%7Barray%7D+%5Cright%5Du&bg=f0f0f0&fg=555555&s=0&c=20201002), with the constraint that ![|u| \leq A](https://s0.wp.com/latex.php?latex=%7Cu%7C+%5Cleq+A&bg=f0f0f0&fg=555555&s=0&c=20201002). We want to get to ![x = \left[ \begin{array}{c} 0 \\ 0 \end{array} \right]](https://s0.wp.com/latex.php?latex=x+%3D+%5Cleft%5B+%5Cbegin%7Barray%7D%7Bc%7D+0+%5C%5C+0+%5Cend%7Barray%7D+%5Cright%5D&bg=f0f0f0&fg=555555&s=0&c=20201002) as quickly as possible, meaning we want to get to ![q = 0](https://s0.wp.com/latex.php?latex=q+%3D+0&bg=f0f0f0&fg=555555&s=0&c=20201002) and then stay there. We could have ![J(X,f)](https://s0.wp.com/latex.php?latex=J%28X%2Cf%29&bg=f0f0f0&fg=555555&s=0&c=20201002) just be the amount of time it takes to  get to the desired endpoint, with a cost of infinity on any ![f](https://s0.wp.com/latex.php?latex=f&bg=f0f0f0&fg=555555&s=0&c=20201002) that violates the torque limits. However, this is a bad idea, for two reasons.

The first reason is that, numerically, you will never really end up at exactly ![\left[ \begin{array}{c} 0 \\ 0 \end{array} \right]](https://s0.wp.com/latex.php?latex=%5Cleft%5B+%5Cbegin%7Barray%7D%7Bc%7D+0+%5C%5C+0+%5Cend%7Barray%7D+%5Cright%5D&bg=f0f0f0&fg=555555&s=0&c=20201002), just very close to it. So if we try to use this function on a computer, unless we are particularly clever we will assign a cost of ![\infty](https://s0.wp.com/latex.php?latex=%5Cinfty&bg=f0f0f0&fg=555555&s=0&c=20201002) to every single control policy.

However, we could instead have ![J(X,f)](https://s0.wp.com/latex.php?latex=J%28X%2Cf%29&bg=f0f0f0&fg=555555&s=0&c=20201002) be the amount of time it takes to get close to the desired endpoint. I personally still think this is a bad idea, and this brings me to my second reason. Once you come up with an objective function, you need to somehow come up with a controller (that is, a choice of ![f](https://s0.wp.com/latex.php?latex=f&bg=f0f0f0&fg=555555&s=0&c=20201002)) that minimizes that objective function, or at the very least performs reasonably well as measured by the objective function. You could do this by being clever and constructing such a controller by hand, but in many cases you would much rather have a computer find the [optimal controller](http://en.wikipedia.org/wiki/Optimal_control). If you are going to have a computer search for a good controller, you want to make the search problem as easy as possible, or at least reasonable. This means that, if we think of ![J](https://s0.wp.com/latex.php?latex=J&bg=f0f0f0&fg=555555&s=0&c=20201002) as a function on the space of control policies, we would like to make the problem of optimizing ![J](https://s0.wp.com/latex.php?latex=J&bg=f0f0f0&fg=555555&s=0&c=20201002) tractable. I don’t know how to make this precise, but there are a few properties we would like ![J](https://s0.wp.com/latex.php?latex=J&bg=f0f0f0&fg=555555&s=0&c=20201002) to satisfy — there aren’t too many local minima, and the minima aren’t approached too steeply (meaning that there is a reasonable large neighbourhood of small values around each local minimum). If we choose an objective function that assigns a value of ![\infty](https://s0.wp.com/latex.php?latex=%5Cinfty&bg=f0f0f0&fg=555555&s=0&c=20201002) to almost everything, then we will end up spending most of our time wading through a sea of infinities without any direction (because all directions will just yield more values of ![\infty](https://s0.wp.com/latex.php?latex=%5Cinfty&bg=f0f0f0&fg=555555&s=0&c=20201002)). So a very strict objective function will be very hard to optimize. Ideally, we would like a different choice of ![J](https://s0.wp.com/latex.php?latex=J&bg=f0f0f0&fg=555555&s=0&c=20201002) that has its minimum at the same location but that decreases gradually to that minimum, so that we can solve the problem using [gradient descent](http://en.wikipedia.org/wiki/Gradient_descent) or some similar method.

In practice, we might have to settle for an objective function that only is trying to minimize the same thing qualitatively, rather than in any precise manner. For example, instead of the choice of ![J](https://s0.wp.com/latex.php?latex=J&bg=f0f0f0&fg=555555&s=0&c=20201002) discussed above for the object moving in a straight line, we could choose

![J(X,f) = \int_{0}^{T} |q(t)|^2 dt,](https://s0.wp.com/latex.php?latex=J%28X%2Cf%29+%3D+%5Cint_%7B0%7D%5E%7BT%7D+%7Cq%28t%29%7C%5E2+dt%2C&bg=f0f0f0&fg=555555&s=0&c=20201002)

where ![T](https://s0.wp.com/latex.php?latex=T&bg=f0f0f0&fg=555555&s=0&c=20201002) is some arbitrary final time. In this form, we are trying to minimize the time-integral of some function of the deviation of ![q](https://s0.wp.com/latex.php?latex=q&bg=f0f0f0&fg=555555&s=0&c=20201002) from ![0](https://s0.wp.com/latex.php?latex=0&bg=f0f0f0&fg=555555&s=0&c=20201002). With a little bit of work, we can deduce that, for large enough ![T](https://s0.wp.com/latex.php?latex=T&bg=f0f0f0&fg=555555&s=0&c=20201002), the optimal controller is a [bang-bang controller](http://en.wikipedia.org/wiki/Bang%E2%80%93bang_control) that accelerates towards $0$ at the greatest rate possible, until accelerating any more would cause the object to overshoot ![q = 0](https://s0.wp.com/latex.php?latex=q+%3D+0&bg=f0f0f0&fg=555555&s=0&c=20201002), at which point the controller should decelerate at the greatest rate possible (there are some additional cases for when the object will overshoot the origin no matter what, but this is the basic idea).

This brings us to my original intention in making this post, which is LQR ([linear-quadratic regulator](http://en.wikipedia.org/wiki/Linear-quadratic_regulator)) control. In this case, we assume that ![A](https://s0.wp.com/latex.php?latex=A&bg=f0f0f0&fg=555555&s=0&c=20201002) and ![B](https://s0.wp.com/latex.php?latex=B&bg=f0f0f0&fg=555555&s=0&c=20201002) are both constant and that our cost function is of the form

![J(X,f) = \int_{0}^{\infty} X(t)^{T}QX(t) + f(X(t),t)^{T}Rf(X(t),t) dt,](https://s0.wp.com/latex.php?latex=J%28X%2Cf%29+%3D+%5Cint_%7B0%7D%5E%7B%5Cinfty%7D+X%28t%29%5E%7BT%7DQX%28t%29+%2B+f%28X%28t%29%2Ct%29%5E%7BT%7DRf%28X%28t%29%2Ct%29+dt%2C&bg=f0f0f0&fg=555555&s=0&c=20201002)

where the ![T](https://s0.wp.com/latex.php?latex=T&bg=f0f0f0&fg=555555&s=0&c=20201002) means transpose and ![Q](https://s0.wp.com/latex.php?latex=Q&bg=f0f0f0&fg=555555&s=0&c=20201002) and ![R](https://s0.wp.com/latex.php?latex=R&bg=f0f0f0&fg=555555&s=0&c=20201002) are both positive definite matrices. In other words, we assume that our goal is to get to ![x = 0](https://s0.wp.com/latex.php?latex=x+%3D+0&bg=f0f0f0&fg=555555&s=0&c=20201002), and we penalize both our distance from ![x = 0](https://s0.wp.com/latex.php?latex=x+%3D+0&bg=f0f0f0&fg=555555&s=0&c=20201002) and the amount of torque we apply at each point in time. If we have a cost function of this form, then we can actually solve analytically for the optimal control policy ![f](https://s0.wp.com/latex.php?latex=f&bg=f0f0f0&fg=555555&s=0&c=20201002). The solution involves solving the [Hamilton-Bellman-Jacobi equations](http://en.wikipedia.org/wiki/Hamilton-Jacobi-Bellman_equation), and I won’t go into the details, but when the smoke clears we end up with a linear feedback policy ![u = -Kx](https://s0.wp.com/latex.php?latex=u+%3D+-Kx&bg=f0f0f0&fg=555555&s=0&c=20201002), where ![K = R^{-1}B^{T}P](https://s0.wp.com/latex.php?latex=K+%3D+R%5E%7B-1%7DB%5E%7BT%7DP&bg=f0f0f0&fg=555555&s=0&c=20201002), and ![P](https://s0.wp.com/latex.php?latex=P&bg=f0f0f0&fg=555555&s=0&c=20201002) is given by the solution to the [algebraic Riccati equation](http://en.wikipedia.org/wiki/Algebraic_Riccati_equation)

![A^TP+PA-PBR^{-1}B^TP+Q=0.](https://s0.wp.com/latex.php?latex=A%5ETP%2BPA-PBR%5E%7B-1%7DB%5ETP%2BQ%3D0.&bg=f0f0f0&fg=555555&s=0&c=20201002)

What’s even better is that MATLAB has a built-in function called lqr that will set up and solve the Riccati equation automatically.

You might have noticed that we had to make the assumption that both ![A](https://s0.wp.com/latex.php?latex=A&bg=f0f0f0&fg=555555&s=0&c=20201002) and ![B](https://s0.wp.com/latex.php?latex=B&bg=f0f0f0&fg=555555&s=0&c=20201002) were constant, which is a fairly strong assumption, as it implies that we have a LTI ([linear time-invariant](http://en.wikipedia.org/wiki/LTI_system_theory)) system. So what is LQR control actually good for? The answer is stabilization. If we want to design a controller that will stabilize a system about a point, we can shift coordinates so that the point is at the origin, then take a linear approximation about the origin. As long as we have a moderately accurate linear model for the system about that point, the LQR controller will successfully stabilize the system to that point within some basin of attraction. More technically, the LQR controller will make the system [locally asymptotically stable](http://en.wikipedia.org/wiki/Stability_theory), and the cost function ![J](https://s0.wp.com/latex.php?latex=J&bg=f0f0f0&fg=555555&s=0&c=20201002) for the linear system will be a valid [local Lyapunov function](http://en.wikipedia.org/wiki/Lyapunov_function#Locally_asymptotically_stable_equilibrium).

Really, the best reason to make use of LQR controllers is that they are a solution to stabilization problems that work out of the box. Many controllers that work in theory will actually require a ton of tuning in practice; this isn’t the case for an LQR controller. As long as you can identify a linear system about the desired stabilization point, even if your identification isn’t perfect, you will end up with a pretty good controller.

I was thinking of also going into techniques for linear system identification, but I think I’ll save that for a future post. The short answer is that you find a least-squares fit of the data you collect. I’ll also go over how this all applies to the underwater cart-pole in a future post.


Robotics

This summer I am working in the Robotics Locomotion group at CSAIL (MIT’s Computer Science and Artificial Intelligence Laboratory). I’ve decided to start a blog to exposit on the ideas involved. This ranges from big theoretical ideas (like general system identification techniques) to problem-specific ideas (specific learning strategies for the system we’re interested in) to useful information on using computational tools (how to make MATLAB’s ode45 do what you want it to).

To start with, I’m going to describe the problem that I’m working on, together with John (a grad student in mechanical engineering).

Last spring, I took 6.832 (Underactuated Robotics) at MIT. In that class, we learned multiple incredibly powerful techniques for nonlinear control. After taking it, I was more or less convinced that we could solve, at least off-line, pretty much any control problem once it was posed properly. After coming to the Locomotion group, I realized that this wasn’t quite right. What is actually true is that we can solve any control problem where we have a good model and a reasonable objective function (we can also run into problems in high dimensions, but even there you can make progress if the objective function is nice enough).

So, we can (almost) solve any problem once we have a good model. That means the clear next thing to do is to come up with really good modelling techniques. Again, this is sort of true. There are actually three steps to constructing a good controller: experimental design, system identification, and a control policy.

System identification is the process of building a good model for your system given physical data from it. But to build a good model, you need good data. That’s where experimental design comes in. Many quick and dirty ways of collecting data (like measuring the response of a sinusoidal input) introduce flaws into the model (which cannot be fixed except by collecting more data). I will explain these issues in more detail in a later post. For simple systems, you can get away with still-quick but slightly-less-dirty methods (such as a [chirp](http://en.wikipedia.org/wiki/Chirp) input), but for more general systems you need better techniques. Ian (a research scientist in our lab) has a nice [paper](http://people.csail.mit.edu/ian/M_CDC_09.pdf) on this topic that involves semidefinite optimization.

Designing a control policy we have already discussed. It is the process of, once we have collected data and built a model, designing an algorithm that will guide our system to its desired end state.

So, we have three tasks — experimental design, system identification, and control policy. If we can do the first two well, then we already know how to do the third. So one solution is to do a really good job on the experimental design and system identification so that we can use our sophisticated control techniques. This is what Michael and Zack are working on with a walking robot. Zack has spent months building the robot in such a way that it will obey all of our idealized models and behave nicely enough that we can run [LQR-trees](https://jsteinhardt.wordpress.com/feed/groups.csail.mit.edu/robotics-center/public_papers/Tedrake09a.pdf) on it.

Another solution is to give up on having a perfect model and have a system identification algorithm that, instead of returning a single model, returns an entire space of models (for example, by giving an uncertainty on each parameter). Then, as long as we can build a controller that works for *every system in this space*, we’ll be good to go. This can be done by using techniques from robust control.

A final idea is to give up on models entirely and try to build a controller that relies only on the qualitative behaviour of the system in question (for example, by using model-free learning techniques). This is what I am working on. More specifically, I’m working on control in fluids with reasonably high [Reynold’s number](http://en.wikipedia.org/wiki/Reynolds_number). Unless you can solve the [Navier-Stokes equations](http://en.wikipedia.org/wiki/Navier-stokes), you can’t hope to get a model for this system, so you’ll have to do something more clever.

The first system we’re working with is the underwater [cart-pole](http://en.wikipedia.org/wiki/Inverted_pendulum) system. This involves a pendulum attached to a cart. The pendulum itself is not actuated (meaning there is no motor to swing it up). Instead, the only way to control the pendulum is indirectly, by moving the cart around (the cart is constrained to move in a line). The fluid dynamics enter when we put the pendulum in a water tunnel and replace the arm of the pendulum with a water foil.

When the pendulum is in a constant-velocity stream, the system becomes a cart and pendulum with non-linear damping. However, when we add objects to the stream, the objects [shed vortices](http://en.wikipedia.org/wiki/Vortex_shedding) and the dynamics become too complicated to model with an ordinary differential equation. Instead, we need to simulate the solution to a partial differential equation, which is significantly more difficult computationally.

Our first goal is to design a controller that will stabilize the pendulum at the top in the case of a constant stream (we have already done this — more about that in a later post). Our next goal is to design a controller to swing the pendulum up to the top, again in a constant stream (this is what I hope to finish tomorrow — again, more details in a later post). Once we have these finished, the more interesting work begins — to accomplish the same tasks in the presence of vortices. If we were to use existing ideas, we would design a robust version of the controller for a constant stream, and treat the vortices as disturbances. And this will probably be the first thing we do, so that we have a standard of comparison. But there are many examples in nature of animals using vortices to their advantage. So our ultimate goal is to do the same in this simple system. Since vortices represent lots of extra energy in the water, our hope is to actually pull energy out of the vortex to aid in the swing-up task, thus actually using *less* energy than would be needed without vortices (if this sounds crazy, consider that [dead trout can swim upstream](http://www.oeb.harvard.edu/lauder/reprints_unzipped/Liao.2004.pdf) using vortices).

Hopefully this gives you a good overview of my project. This is my first attempt at maintaining a research blog, so if you have any comments to help improve the exposition, please give them to me. Also, if you’d like me to elaborate further on anything, let me know. I’ll hopefully have a post or two going into more specific details this weekend.


The basic reasons I expect AGI ruin

I’ve been citing [AGI Ruin: A List of Lethalities](https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities) to explain why the situation with AI looks lethally dangerous to me. But that post is relatively long, and emphasizes specific open technical problems over “the basics”.

Here are 10 things I’d focus on if I were giving “the basics” on why I’m so worried:[[1]](https://intelligence.org/feed/#fn1)

---

1. **General intelligence is very powerful**, and once we can build it at all, **STEM-capable artificial general intelligence (AGI) is likely to vastly outperform human intelligence immediately (or very quickly)**.

When I say “general intelligence”, I’m usually thinking about “whatever it is that lets human brains do astrophysics, category theory, etc. even though our brains evolved under literally zero selection pressure to solve astrophysics or category theory problems”.

It’s possible that we should already be thinking of GPT-4 as “AGI” on some definitions, so to be clear about the threshold of generality I have in mind, I’ll specifically talk about “STEM-level AGI”, though I expect such systems to be good at non-STEM tasks too.

Human brains aren’t perfectly general, and not all narrow AI systems or animals are equally narrow. (E.g., AlphaZero is more general than AlphaGo.) But it sure is interesting that humans evolved cognitive abilities that unlock *all of these sciences at once*, with zero evolutionary fine-tuning of the brain aimed at equipping us for any of those sciences. Evolution just stumbled into a solution to *other* problems, that happened to generalize to millions of wildly novel tasks.

More concretely:

* AlphaGo is a very impressive reasoner, but its hypothesis space is limited to sequences of Go board states rather than sequences of states of the physical universe. Efficiently reasoning about the physical universe requires solving at least some problems that are different in kind from what AlphaGo solves.
	+ These problems might be solved by the STEM AGI’s programmer, and/or solved by the algorithm that finds the AGI in program-space; and some such problems may be solved by the AGI itself in the course of refining its thinking.[[2]](https://intelligence.org/feed/#fn2)
* Some examples of abilities I expect humans to only automate once we’ve built STEM-level AGI (if ever):
	+ The ability to perform open-heart surgery with a high success rate, in a messy non-standardized ordinary surgical environment.
	+ The ability to match smart human performance in a specific hard science field, across all the scientific work humans do in that field.
* In principle, I suspect you could build a narrow system that is good at those tasks while lacking the basic mental machinery required to do par-human reasoning about *all* the hard sciences. In practice, I very strongly expect humans to find ways to build general reasoners to perform those tasks, before we figure out how to build narrow reasoners that can do them. (For the same basic reason evolution stumbled on general intelligence so early in the history of human tech development.)[[3]](https://intelligence.org/feed/#fn3)

When I say “general intelligence is very powerful”, a lot of what I mean is that *science* is very powerful, and that having all of the sciences at once is a lot more powerful than the sum of each science’s impact.[[4]](https://intelligence.org/feed/#fn4)

Another large piece of what I mean is that (STEM-level) general intelligence *is a very high-impact sort of thing to automate* because STEM-level AGI is likely to blow human intelligence out of the water immediately, or very soon after its invention.

80,000 Hours gives the (non-representative) example of how AlphaGo and its successors compared to humanity:

>
> In the span of a year, AI had advanced from being too weak to win a single [Go] match against the worst human professionals, to being impossible for even the best players in the world to defeat.
>
>
>
>

I expect general-purpose science AI to blow human science ability out of the water in a similar fashion.

Reasons for this include:

* Empirically, humans aren’t near a cognitive ceiling, and even narrow AI often suddenly blows past the human reasoning ability range on the task it’s designed for. It would be weird if scientific reasoning [were an exception](https://intelligence.org/2017/10/20/alphago/).
* Empirically, human brains are full of cognitive biases and inefficiencies. It’s doubly weird if scientific reasoning is an exception even though it’s *visibly* a mess with tons of blind spots, inefficiencies, and motivated cognitive processes, and innumerable historical examples of scientists and mathematicians taking decades to make technically simple advances.
* Empirically, human brains are extremely bad at some of the most basic cognitive processes underlying STEM.
	+ E.g., consider the stark limits on human working memory and ability to do basic mental math. We can barely multiply smallish multi-digit numbers together in our head, when in principle a reasoner could hold thousands of complex mathematical structures in its working memory simultaneously and perform complex operations on them. Consider the sorts of technologies and scientific insights that might only ever occur to a reasoner if it can directly see (within its own head, in real time) the connections between hundreds or thousands of different formal structures.
* Human brains underwent no direct optimization for STEM ability in our ancestral environment, beyond traits like “I can distinguish four objects in my visual field from five objects”.[[5]](https://intelligence.org/feed/#fn5)
* In contrast, human engineers can deliberately optimize AGI systems’ brains for math, engineering, etc. capabilities; and human engineers have an enormous variety of tools available to build general intelligence that evolution lacked.[[6]](https://intelligence.org/feed/#fn6)
* Software (unlike human intelligence) scales with more compute.
* Current ML uses far more compute to *find* reasoners than to *run* reasoners. This is very likely to hold true for AGI as well.
* We probably have more than enough compute already, if we knew how to train AGI systems in a remotely efficient way.

And on a meta level: the hypothesis that STEM AGI can quickly outperform humans has a disjunctive character. There are [many different advantages](https://aiimpacts.org/sources-of-advantage-for-artificial-intelligence/) that individually suffice for this, even if STEM AGI doesn’t start off with any other advantages. (E.g., speed, math ability, scalability with hardware, skill at optimizing hardware…)

In contrast, the claim that STEM AGI will [hit the narrow target](https://twitter.com/robbensinger/status/1623835453110775810) of “par-human scientific ability”, *and* stay at around that level for long enough to let humanity adapt and adjust, has a conjunctive character.[[7]](https://intelligence.org/feed/#fn7)

2. A common misconception is that STEM-level AGI is dangerous because of something murky about “agents” or about self-awareness. Instead, I’d say that **the danger is inherent to the nature of action sequences** that push the world toward some sufficiently-hard-to-reach state.[[8]](https://intelligence.org/feed/#fn8)

Call such sequences “plans”.

If you sampled a random plan from the space of all writable plans (weighted by length, in any extant formal language), and all we knew about the plan is that executing it would successfully achieve some superhumanly ambitious technological goal like “invent fast-running [whole-brain emulation](https://www.lesswrong.com/tag/whole-brain-emulation)“, then hitting a button to execute the plan would kill all humans, with very high probability. This is because:

* “Invent fast WBE” is a hard enough task that succeeding in it usually requires gaining a lot of knowledge and cognitive and technological capabilities, enough to do lots of other dangerous things.
* “Invent fast WBE” is likelier to succeed if the plan also includes steps that gather and control as many resources as possible, eliminate potential threats, etc. These are “[convergent instrumental strategies](https://arbital.com/p/instrumental_convergence/)“—strategies that are useful for pushing the world in a particular direction, almost regardless of which direction you’re pushing.
* Human bodies and the food, water, air, sunlight, etc. we need to live are resources (“you are made of atoms the AI can use for something else”); and we’re also potential threats (e.g., we could build a rival superintelligent AI that executes a totally different plan).

The danger is in the cognitive work, not in some complicated or emergent feature of the “agent”; it’s in the task itself.

It isn’t that the abstract space of plans was built by evil human-hating minds; it’s that the [instrumental convergence](https://nickbostrom.com/superintelligentwill.pdf) thesis holds for the plans themselves. In full generality, plans that succeed in goals like “build WBE” tend to be dangerous.

This isn’t true of all plans that successfully push our world into a specific (sufficiently-hard-to-reach) physical state, but it’s true of the vast majority of them.

This is counter-intuitive because most of the impressive “plans” we encounter today are generated by humans, and it’s tempting to view strong plans through a human lens. But humans have hugely overlapping values, thinking styles, and capabilities; AI is drawn from new distributions.

3. Current ML work is on track to produce things that are, in the ways that matter, **more like “randomly sampled plans” than like “the sorts of plans a civilization of human von Neumanns would produce”**. (Before we’re anywhere near being able to produce the latter sorts of things.)[[9]](https://intelligence.org/feed/#fn9)

We’re building “AI” in the sense of building powerful general search processes (and search processes for search processes), not building “AI” in the sense of building friendly ~humans but in silicon.

(Note that “we’re going to build systems that are more like A Randomly Sampled Plan than like A Civilization of Human Von Neumanns” doesn’t imply that the plan we’ll get is the one we wanted! There are two separate problems: that current ML finds things-that-act-like-they’re-optimizing-the-task-you-wanted rather than things-that-actually-internally-optimize-the-task-you-wanted, and also that internally ~maximizing most superficially desirable ends will kill humanity.)

Note that the same problem holds for systems trained to imitate humans, if those systems scale to being able to do things like “build whole-brain emulation”. “We’re training on something related to humans” doesn’t give us “we’re training things that are best thought of as humans plus noise”.

It’s not obvious to me that GPT-like systems can scale to capabilities like “build WBE”. But if they do, we face the [problem](https://twitter.com/ESYudkowsky/status/1628835920328929281) that most ways of successfully imitating humans don’t look like “build a human (that’s somehow superhumanly good at imitating the Internet)”. They look like “build a relatively complex and alien optimization process that is good at imitation tasks (and potentially at many other tasks)”.

You don’t need to be a human in order to model humans, any more than you need to be a cloud in order to model clouds well. The only reason this is more confusing in the case of “predict humans” than in the case of “predict weather patterns” is that humans and AI systems are both intelligences, so it’s easier to slide between “the AI models humans” and “the AI is basically a human”.

4. The key differences between humans and “things that are more easily approximated as random search processes than as humans-plus-a-bit-of-noise” **lies in lots of complicated machinery in the human brain**.

(Cf. [Detached Lever Fallacy](https://www.lesswrong.com/posts/zY4pic7cwQpa9dnyk/detached-lever-fallacy), [Niceness Is Unnatural](https://www.lesswrong.com/posts/krHDNc7cDvfEL8z9a/niceness-is-unnatural), and [Superintelligent AI Is Necessary For An Amazing Future, But Far From Sufficient](https://www.lesswrong.com/posts/HoQ5Rp7Gs6rebusNP/superintelligent-ai-is-necessary-for-an-amazing-future-but-1).)

Humans are not blank slates in the relevant ways, such that just raising an AI like a human solves the problem.

This doesn’t mean the problem is unsolvable; but it means that you either need to reproduce that internal machinery, in a lot of detail, in AI, or you need to build some new kind of machinery that’s safe for reasons other than the specific reasons humans are safe.

(You need cognitive machinery that somehow samples from a much narrower space of plans that are still powerful enough to succeed in at least one task that saves the world, but are constrained in ways that make them far less dangerous than the larger space of plans. And you need a thing that *actually* implements internal machinery like that, as opposed to just being optimized to superficially behave as though it does in the narrow and unrepresentative environments it was in before starting to work on WBE. “Novel science work” means that pretty much everything you want from the AI is out-of-distribution.)

5. **STEM-level AGI timelines don’t look that long** (e.g., probably not 50 or 150 years; could well be 5 years or 15).

I won’t try to argue for this proposition, beyond pointing at the field’s recent [progress](https://twitter.com/leopoldasch/status/1638848850516672513) and echoing [Nate Soares’ comments](https://www.lesswrong.com/posts/cCMihiwtZx7kdcKgt/comments-on-carlsmith-s-is-power-seeking-ai-an-existential#Timelines) from early 2021:

>
> […] I observe that, 15 years ago, everyone was saying AGI is far off because of what it couldn’t do — basic image recognition, go, starcraft, winograd schemas, simple programming tasks. But basically all that has fallen. The gap between us and AGI is made mostly of intangibles. (Computer programming that is Actually Good? Theorem proving? Sure, but on my model, “good” versions of those are a hair’s breadth away from full AGI already. And the fact that I need to clarify that “bad” versions don’t count, speaks to my point that the only barriers people can name right now are intangibles.) That’s a very uncomfortable place to be!
>
>
>
> […] I suspect that I’m in more-or-less the “penultimate epistemic state” on AGI timelines: I don’t know of a project that seems like they’re right on the brink; that would put me in the “final epistemic state” of thinking AGI is imminent. But I’m in the second-to-last epistemic state, where I wouldn’t feel all that shocked to learn that some group has reached the brink. Maybe I won’t get that call for 10 years! Or 20! But it could also be 2, and I wouldn’t get to be indignant with reality. I wouldn’t get to say “but all the following things should have happened first, before I made that observation!”. Those things have happened. I have made those observations. […]
>
>
>
>

I think timing tech is [very difficult](https://intelligence.org/2017/10/13/fire-alarm/) (and plausibly ~impossible when the tech isn’t pretty imminent), and I think reasonable people can disagree a lot about timelines.

I also think converging on timelines is not very crucial, since if AGI is 50 years away I would say it’s still the largest single risk we face, and the bare minimum alignment work required for surviving that transition could easily take longer than that.

Also, “STEM AGI when?” is the kind of argument that requires hashing out people’s predictions about how we get to STEM AGI, which is a bad thing to debate publicly insofar as improving people’s models of pathways can further shorten timelines.

I mention timelines anyway because they are in fact a major reason I’m pessimistic about our prospects; if I learned tomorrow that AGI were 200 years away, I’d be outright optimistic about things going well.

6. **We don’t currently know how to do alignment**, we don’t seem to have a much better idea now than we did 10 years ago, and there are many large novel visible difficulties. (See [AGI Ruin](https://www.lesswrong.com/posts/GNhMPAWcfBCASy8e6/a-central-ai-alignment-problem-capabilities-generalization) and the [Capabilities Generalization, and the Sharp Left Turn](https://www.lesswrong.com/posts/GNhMPAWcfBCASy8e6/a-central-ai-alignment-problem-capabilities-generalization).)

On a more basic level, quoting [Nate Soares](https://intelligence.org/ensuring): “Why do I think that AI alignment looks fairly difficult? The main reason is just that this has been my experience from actually working on these problems.”

7. We should be starting with a pessimistic prior about **achieving reliably good behavior in any complex safety-critical software, particularly if the software is novel**. Even more so if the thing we need to make robust is structured like undocumented spaghetti code, and more so still if the field is highly competitive and you need to achieve some robustness property *while moving faster* than a large pool of less-safety-conscious people who are racing toward the precipice.

The [default](https://www.lesswrong.com/posts/Ke2ogqSEhL2KCJCNx/security-mindset-lessons-from-20-years-of-software-security) assumption is that complex software goes wrong in dozens of different ways you didn’t expect. Reality ends up being thorny and inconvenient in many of the places where your models were absent or fuzzy. Surprises are abundant, and some surprises can be good, but this is empirically a lot rarer than unpleasant surprises in software development hell.

The future is hard to predict, but plans systematically take longer and run into more snags than humans naively expect, as opposed to plans systematically going surprisingly smoothly and deadlines being systematically hit ahead of schedule.

The history of computer security and of safety-critical software systems is almost invariably one of robust software lagging far, far behind non-robust versions of the same software. Achieving any robustness property in complex software that will be deployed in the real world, with all its messiness and adversarial optimization, is very difficult and usually fails.

In many ways I think the foundational discussion of AGI risk is [Security Mindset and Ordinary Paranoia](https://intelligence.org/2017/11/25/security-mindset-ordinary-paranoia/) and [Security Mindset and the Logistic Success Curve](https://intelligence.org/2017/11/26/security-mindset-and-the-logistic-success-curve/), and the main body of the text doesn’t even mention AGI. Adding in the specifics of AGI and smarter-than-human AI takes the risk from “dire” to “seemingly overwhelming”, but adding in those specifics is not required to be massively concerned if you think getting this software right matters for our future.

8. **Neither ML nor the larger world is currently taking this seriously**, as of April 2023.

This is obviously something we can change. But until it’s changed, things will continue to look very bad.

Additionally, most of the people who are taking AI risk somewhat seriously are, to an important extent, not willing to worry about things until after they’ve been experimentally proven to be dangerous. Which is a lethal sort of methodology to adopt when you’re working with smarter-than-human AI.

My basic picture of *why* the world currently isn’t responding appropriately is the one in [Four mindset disagreements behind existential risk disagreements in ML](https://www.lesswrong.com/posts/84BJopKvQ8qYGHY6b/four-mindset-disagreements-behind-existential-risk), [The inordinately slow spread of good AGI conversations in ML](https://www.lesswrong.com/posts/Rkxj7TFxhbm59AKJh/the-inordinately-slow-spread-of-good-agi-conversations-in-ml), and [*Inadequate Equilibria*](https://equilibriabook.com/toc).[[10]](https://intelligence.org/feed/#fn10)

9. As noted above, **current ML is very opaque**, and **it mostly lets you intervene on behavioral proxies** for what we want, rather than letting us directly design desirable features.

ML as it exists today also requires that data is readily available and safe to provide. E.g., we can’t robustly train the AGI on “don’t kill people” because we can’t provide real examples of it killing people to train against the behavior we don’t want; we can only give flawed proxies and work via indirection.

10. **There are lots of specific abilities** which seem like they ought to be possible for the kind of civilization that can safely deploy smarter-than-human optimization, that are far out of reach, with no obvious path forward for achieving them with opaque deep nets even if we had unlimited time to work on some relatively concrete set of research directions.

(Unlimited time suffices if we can set a more abstract/indirect research direction, like “just think about the problem for a long time until you find some solution”. There are presumably paths forward; we just don’t know what they are today, which puts us in a worse situation.)

E.g., we don’t know how to go about inspecting a nanotech-developing AI system’s brain to verify that it’s only thinking about a specific room, that it’s internally representing the intended goal, that it’s directing its optimization at that representation, that it internally has a particular planning horizon and a variety of capability bounds, that it’s unable to think about optimizers (or specifically about humans), or that it otherwise has the right topics internally whitelisted or blacklisted.

Individually, it seems to me that each of these difficulties can be addressed. In combination, they seem to me to put us in a very dark situation.

---

One common response I hear to points like the above is:

>
> The future is generically hard to predict, so it’s just not possible to be rationally confident that things will go well or poorly. Even if you look at dozens of different arguments and framings and the ones that hold up to scrutiny nearly all seem to point in the same direction, it’s always possible that you’re making some invisible error of reasoning that causes correlated failures in many places at once.
>
>
>
>

I’m sympathetic to this because I agree that the future is hard to predict.

I’m not totally confident things will go poorly; if I were, I wouldn’t be trying to solve the problem! I think things are looking extremely dire, but not hopeless.

That said, some people think that even “extremely dire” is an impossible belief state to be in, in advance of an AI apocalypse actually occurring. I disagree here, for two basic reasons:

a. There are many details we can get into, but on a core level **I don’t think the risk is particularly complicated or hard to reason about**. The core concern fits into a [tweet](https://twitter.com/robbensinger/status/1644006731792646145):

>
> STEM AI is likely to vastly exceed human STEM abilities, conferring a decisive advantage. We aren’t on track to knowing how to aim STEM AI at intended goals, and STEM AIs pursuing unintended goals tend to have instrumental subgoals like “control all resources”.
>
>
>
>

Zvi Mowshowitz [puts the core concern](https://thezvi.substack.com/p/response-to-tyler-cowens-existential) in even more basic terms:

>
> I also notice a kind of presumption that things in most scenarios will work out and that doom is dependent on particular ‘distant possibilities,’ that often have many logical dependencies or require a lot of things to individually go as predicted. Whereas I would say that those possibilities are not so distant or unlikely, but more importantly that the result is robust, that once the intelligence and optimization pressure that matters is no longer human that most of the outcomes are existentially bad by my values and that one can reject or ignore many or most of the detail assumptions and still see this.
>
>
>
>

The details do matter for evaluating the exact risk level, but this isn’t the sort of topic where it seems fundamentally impossible for any human to reach a good understanding of the core difficulties and whether we’re handling them.

b. Relatedly, as Nate Soares has argued, **AI disaster scenarios are** [**disjunctive**](https://www.lesswrong.com/posts/ervaGwJ2ZcwqfCcLx/agi-ruin-scenarios-are-likely-and-disjunctive). There are many bad outcomes for every good outcome, and many paths leading to disaster for every path leading to utopia.

Quoting [Eliezer Yudkowsky](https://twitter.com/ESYudkowsky/status/1644045728983969816):

>
> You don’t get to adopt a prior where you have a 50-50 chance of winning the lottery “because either you win or you don’t”; the question is not whether we’re uncertain, but whether someone’s allowed to milk their uncertainty to expect good outcomes.
>
>
>
>

Quoting [Jack Rabuck](https://twitter.com/JackRabuck/status/1644762977504227329):

>
> I listened to the whole 4 hour Lunar Society interview with @ESYudkowsky
> (hosted by @dwarkesh\_sp) that was mostly about AI alignment and I think I identified a point of confusion/disagreement that is pretty common in the area and is rarely fleshed out:
>
>
>
> Dwarkesh repeatedly referred to the conclusion that AI is likely to kill humanity as “wild.”
>
>
>
> Wild seems to me to pack two concepts together, ‘bad’ and ‘complex.’ And when I say complex, I mean in the sense of the Fermi equation where you have an end point (dead humanity) that relies on a series of links in a chain and if you break any of those links, the end state doesn’t occur.
>
>
>
> It seems to me that Eliezer believes this end state is not wild (at least not in the complex sense), but very simple. He thinks many (most) paths converge to this end state.
>
>
>
> That leads to a misunderstanding of sorts. Dwarkesh pushes Eliezer to give some predictions based on the line of reasoning that he uses to predict that end point, but since the end point is very simple and is a convergence, Eliezer correctly says that being able to reason to that end point does not give any predictive power about the particular path that will be taken in this universe to reach that end point.
>
>
>
> Dwarkesh is thinking about the end of humanity as a causal chain with many links and if any of them are broken it means humans will continue on, while Eliezer thinks of the continuity of humanity (in the face of AGI) as a causal chain with many links and if any of them are broken it means humanity ends. Or perhaps more discretely, Eliezer thinks there are a few very hard things which humanity could do to continue in the face of AI, and absent one of those occurring, the end is a matter of when, not if, and the when is much closer than most other people think.
>
>
>
> Anyway, I think each of Dwarkesh and Eliezer believe the other one falls on the side of extraordinary claims require extraordinary evidence – Dwarkesh thinking the end of humanity is “wild” and Eliezer believing humanity’s viability in the face of AGI is “wild” (though not in the negative sense).
>
>
>
>

I don’t consider “AGI ruin is disjunctive” a knock-down argument for high p(doom) on its own. NASA has a high success rate for rocket launches even though success requires many things to go right simultaneously. Humanity is capable of achieving conjunctive outcomes, to some degree; but I think this framing makes it clearer why it’s *possible* to rationally arrive at a high p(doom), at all, when enough evidence points in that direction.[[11]](https://intelligence.org/feed/#fn11)

---

1. Eliezer Yudkowsky’s [So Far: Unfriendly AI Edition](https://www.econlib.org/archives/2016/03/so_far_unfriend.html) and Nate Soares’ [Ensuring Smarter-Than-Human Intelligence Has a Positive Outcome](https://intelligence.org/ensuring) are two other good (though old) introductions to what I’d consider “the basics”.

To state the obvious: this post consists of various claims that increase my probability on AI causing an existential catastrophe, but not all the claims have to be true in order for AI to have a high probability of causing such a catastrophe.

Also, I wrote this post to summarize my own top reasons for being worried, not to try to make a maximally compelling or digestible case for others. I don’t expect others to be similarly confident based on such a quick overview, unless perhaps you’ve read other sources on AI risk in the past. (Including more optimistic ones, since it’s harder to be confident when you’ve only heard from one side of a disagreement. I’ve [written in the past](https://www.lesswrong.com/posts/vT4tsttHgYJBoKi4n/some-abstract-non-technical-reasons-to-be-non-maximally) about some of the things that give *me* small glimmers of hope, but people who are overall far more hopeful will have very different reasons for hope, based on very different heuristics and background models.) [![↩](https://s.w.org/images/core/emoji/14.0.0/72x72/21a9.png)](https://intelligence.org/feed/#fnref1)
2. E.g., the physical world is too complex to simulate in full detail, unlike a Go board state. An effective general intelligence needs to be able to model the world at many different levels of granularity, and strategically choose which levels are relevant to think about, as well as which specific pieces/aspects/properties of the world *at* those levels are relevant to think about.

More generally, being a general intelligence requires an enormous amount of laserlike focus and strategicness when it comes to which thoughts you do or don’t think. A large portion of your compute needs to be relentlessly funneled into *exactly* the tiny subset of questions about the physical world that bear on the question you’re trying to answer or the problem you’re trying to solve. If you fail to be relentlessly targeted and efficient in “aiming” your cognition at the most useful-to-you things, you can easily spend a lifetime getting sidetracked by minutiae, directing your attention at the wrong considerations, etc.

And given the variety of *kinds* of problems you need to solve in order to navigate the physical world well, do science, etc., the *heuristics you use to funnel your compute to the exact right things* need to themselves be very general, rather than all being case-specific.

(Whereas we can more readily imagine that many of the heuristics AlphaGo uses to avoid thinking about the wrong aspects of the game state (or getting otherwise sidetracked) *are Go-specific heuristics*.) [![↩](https://s.w.org/images/core/emoji/14.0.0/72x72/21a9.png)](https://intelligence.org/feed/#fnref2)
3. Of course, if your brain has all the basic mental machinery required to do other sciences, that doesn’t mean that you have the *knowledge* required to actually do well in those sciences. An STEM-level artificial general intelligence could lack physics ability for the same reason many smart humans can’t solve physics problems. [![↩](https://s.w.org/images/core/emoji/14.0.0/72x72/21a9.png)](https://intelligence.org/feed/#fnref3)
4. E.g., because different sciences can synergize, and because you can invent new scientific fields and subfields, and more generally chain one novel insight into dozens of other new insights that critically depended on the first insight. [![↩](https://s.w.org/images/core/emoji/14.0.0/72x72/21a9.png)](https://intelligence.org/feed/#fnref4)
5. More generally, the sciences (and many other aspects of human life, like written language) are a very recent development on evolutionary timescales. So evolution has had very little time to refine and improve on our reasoning ability in many of the ways that matter. [![↩](https://s.w.org/images/core/emoji/14.0.0/72x72/21a9.png)](https://intelligence.org/feed/#fnref5)
6. “Human engineers have an enormous variety of tools available that evolution lacked” is often noted as a reason to think that we may be able to align AGI to our goals, even though evolution failed to align humans to its “goal”. It’s additionally a reason to expect AGI to have greater cognitive ability, if engineers try to achieve great cognitive ability. [![↩](https://s.w.org/images/core/emoji/14.0.0/72x72/21a9.png)](https://intelligence.org/feed/#fnref6)
7. And my understanding is that, e.g., Paul Christiano’s soft-takeoff scenarios don’t involve there being much time between par-human scientific ability and superintelligence. Rather, he’s betting that we have a bunch of decades between GPT-4 and par-human STEM AGI. [![↩](https://s.w.org/images/core/emoji/14.0.0/72x72/21a9.png)](https://intelligence.org/feed/#fnref7)
8. I’ll classify thoughts and text outputs as “actions” too, not just physical movements. [![↩](https://s.w.org/images/core/emoji/14.0.0/72x72/21a9.png)](https://intelligence.org/feed/#fnref8)
9. Obviously, *neither* is a particularly good approximation for ML systems. The point is that our optimism about plans in real life generally comes from the fact that they’re weak, and/or it comes from the fact that the plan generators are human brains with the full suite of human psychological universals. ML systems don’t possess those human universals, and won’t stay weak indefinitely. [![↩](https://s.w.org/images/core/emoji/14.0.0/72x72/21a9.png)](https://intelligence.org/feed/#fnref9)
10. Quoting [Four mindset disagreements behind existential risk disagreements in ML](https://www.lesswrong.com/posts/84BJopKvQ8qYGHY6b/four-mindset-disagreements-behind-existential-risk):

>
>
> 	* People are taking the risks unseriously because they feel weird and abstract.
> 	* When they do think about the risks, they anchor to what’s familiar and known, dismissing other considerations because they feel “unconservative” from a forecasting perspective.
> 	* Meanwhile, social mimesis and the bystander effect make the field sluggish at pivoting in response to new arguments and smoke under the door.
>

Quoting [The inordinately slow spread of good AGI conversations in ML](https://www.lesswrong.com/posts/Rkxj7TFxhbm59AKJh/the-inordinately-slow-spread-of-good-agi-conversations-in-ml):

>
> Info about AGI propagates too slowly through the field, because when one ML person updates, they usually don’t loudly share their update with all their peers. This is because:
>
>
>
> 1. AGI sounds weird, and they don’t want to sound like a weird outsider.
>
>
>
> 2. Their *peers* and the *community as a whole* might perceive this information as an attack on the field, an attempt to lower its status, etc.
>
>
>
> 3. Tech forecasting, differential technological development, long-term steering, [exploratory](https://en.wikipedia.org/wiki/Exploratory_engineering) [engineering](https://intelligence.org/files/ExploratoryEngineeringAI.pdf), ‘not doing certain research because of its long-term social impact’, prosocial research closure, etc. are very novel and foreign to most scientists.
>
>
>
> EAs exert effort to try to dig up precedents like [Asilomar](https://intelligence.org/files/TheAsilomarConference.pdf) partly *because* Asilomar is so unusual compared to the norms and practices of the vast majority of science. Scientists generally don’t think in these terms at all, especially in *advance* of any major disasters their field causes.
>
>
>
> And the scientists who do find any of this intuitive often feel vaguely nervous, alone, and adrift when they talk about it. On a gut level, they see that they have no institutional home and no super-widely-shared ‘this is a virtuous and respectable way to do science’ narrative.
>
>
>
> Normal [science](https://www.lesswrong.com/s/fxynfGCSHpY4FmBZy) is not Bayesian, is not agentic, is not ‘a place where you’re supposed to do arbitrary things just because you heard an argument that makes sense’. Normal science is a specific collection of scripts, customs, and established protocols.
>
>
>
> In trying to move the field toward ‘doing the thing that just makes sense’, even though it’s about a weird topic (AGI), and even though the prescribed response is also weird (closure, differential tech development, etc.), and even though the arguments in support are weird (where’s the experimental data??), we’re inherently fighting our way upstream, against the current.
>
>
>
> Success is possible, but way, way more [dakka](https://thezvi.wordpress.com/2017/12/02/more-dakka/) is needed, and IMO it’s easy to understand why we haven’t succeeded more.
>
>
>
> This is also part of why I’ve increasingly updated toward a strategy of “let’s all be way too blunt and candid about our AGI-related thoughts”.
>
>
>
> The core problem we face isn’t ‘people informedly disagree’, ‘there’s a values conflict’, ‘we haven’t written up the arguments’, ‘nobody has seen the arguments’, or even ‘self-deception’ or ‘self-serving bias’.
>
>
>
> The core problem we face is ‘not enough information is transmitting fast enough, because people feel nervous about whether their private thoughts are in the Overton window’.
>
>
>
>

On the more basic level, [*Inadequate Equilibria*](https://equilibriabook.com/toc) paints a picture of the world’s baseline civilizational competence that I think makes it less mysterious why we could screw up this badly on a novel problem that our scientific and political institutions weren’t designed to address. *Inadequate Equilibria* also talks about the nuts and bolts of Modest Epistemology, which I think is a key part of the failure story. [![↩](https://s.w.org/images/core/emoji/14.0.0/72x72/21a9.png)](https://intelligence.org/feed/#fnref10)
11. Quoting a recent conversation between Aryeh Englander and Eliezer Yudkowsky:

>
> **Aryeh:** […] Yet I still have a very hard time understanding the arguments that would lead to such a high-confidence prediction. Like, I think I understand the main arguments for AI existential risk, but I just don’t understand why some people seem so *sure* of the risks. […]
>
>
>
> **Eliezer:** I think the core thing is the sense that you cannot in this case milk uncertainty for a chance of good outcomes; to get to a good outcome you’d have to actually know where you’re steering, like trying to buy a winning lottery ticket or launching a Moon rocket. Once you realize that uncertainty doesn’t move estimates back toward “50-50, either we live happily ever after or not”, you realize that “people in the EA forums cannot tell whether Eliezer or Paul is right” is not a factor that moves us toward 1:1 good:bad but rather another sign of doom; surviving worlds don’t look confused like that and are able to make faster progress.
>
>
>
> Not as a fully valid argument from which one cannot update further, but as an intuition pump: the more all arguments about the future seem fallible, the more you should expect the future Solar System to have a randomized configuration from your own perspective. Almost zero of those have humans in them. It takes confidence about some argument constraining the future to get to more than that.
>
>
>
> **Aryeh:** when you talk about uncertainty here do you mean uncertain factors *within* your basic world model, or are you also counting model uncertainty? I can see how within your world model extra sources of uncertainty don’t point to lower risk estimates. But my general question I think is more about model uncertainty: how sure can you really be that your world model and reference classes and framework for thinking about this is the right one vs e.g., Robin or Paul or Rohin or lots of others? And in terms of model uncertainty it looks like most of these other approaches imply much lower risk estimates, so adding in that kind of model uncertainty should presumably (I think) point to overall lower risk estimates.
>
>
>
> **Eliezer:** Aryeh, if you’ve got a specific theory that says your rocket design is going to explode, and then you’re also very unsure of how rockets work really, what probability should you assess of your rocket landing safely on target?
>
>
>
> **Aryeh:** how about if you have a specific theory that says you should be comparing what you’re doing to a rocket aiming for the moon but it’ll explode, and then a bunch of other theories saying it won’t explode, plus a bunch of theories saying you shouldn’t be comparing what you’re doing to a rocket in the first place? My understanding of many alignment proposals is that they think we do understand “rockets” sufficiently so that we can aim them, but they disagree on various specifics that lead you to have such high confidence in an explosion. And then there are others like Robin Hanson who use mostly outside-type arguments to argue that you’re framing the issues incorrectly, and we shouldn’t be comparing this to “rockets” at all because that’s the wrong reference class to use. So yes, accounting for some types of model uncertainty won’t reduce our risk assessments and may even raise them further, but other types of model uncertainty – including many of the actual alternative models / framings at least as I understand them – should presumably decrease our risk assessment.
>
>
>
> **Eliezer:** What if people are trying to build a flying machine for the first time, and there’s a whole host of them with wildly different theories about why it ought to fly easily, and you think there’s basic obstacles to stable flight that they’re not getting? Could you force the machine to fly despite all obstacles by recruiting more and more optimists to have different theories, each of whom would have some chance of being right?
>
>
>
> **Aryeh:** right, my point is that in order to have near certainty of not flying you need to be very very sure that your model is right and theirs isn’t. Or in other words, you need to have very low model uncertainty. But once you add in model uncertainty where you consider that maybe those other optimists’ models could be right, then your risk estimates will go down. Of course you can’t arbitrarily add in random optimistic models from random people – it needs to be weighted in some way. My confusion here is that you seem to be very, very certain that your model is the right one, complete with all its pieces and sub-arguments and the particular reference classes you use, and I just don’t quite understand why.
>
>
>
> **Eliezer:** There’s a big difference between “sure your model is the right one” and the whole thing with people wandering over with their own models and somebody else going, “I can’t tell the difference between you and them, how can you possibly be so sure they’re not right?”
>
>
>
> The intuition I’m trying to gesture at here is that you can’t milk success out of uncertainty, even by having a bunch of other people wander over with optimistic models. It *shouldn’t* be able to work in real life. If your epistemology says that you can generate free success probability that way, you must be doing something wrong.
>
>
>
> Or maybe another way to put it: When you run into a very difficult problem that you can see is very difficult, but inevitably a bunch of people with less clear sight wander over and are optimistic about it because they don’t see the problems, for you to update on the optimists would be to update on something that happens inevitably. So to adopt this policy is just to make it impossible for yourself to ever perceive when things have gotten really bad.
>
>
>
> **Aryeh:** not sure I fully understand what you’re saying. It looks to me like to some degree what you’re saying boils down to your views on modest epistemology – i.e., basically just go with your own views and don’t defer to anybody else. It sounds like you’re saying not only don’t defer, but don’t even really incorporate any significant model uncertainty based on other people’s views. Am I understanding this at all correctly or am I totally off here?
>
>
>
> **Eliezer:** My epistemology is such that it’s possible in principle for me to notice that I’m doomed, in worlds which look very doomed, despite the fact that all such possible worlds no matter how doomed they actually are, always contain a chorus of people claiming we’re not doomed.
>
>
>
>

(See [*Inadequate Equilibria*](https://equilibriabook.com/toc) for a detailed discussion of Modest Epistemology, deference, and “[outside views](https://www.lesswrong.com/posts/BcYfsi7vmhDvzQGiF/taboo-outside-view)“, and [Strong Evidence Is Common](https://www.lesswrong.com/posts/JD7fwtRQ27yc8NoqS/strong-evidence-is-common) for the basic first-order case that people can often reach confident conclusions about things.) [![↩](https://s.w.org/images/core/emoji/14.0.0/72x72/21a9.png)](https://intelligence.org/feed/#fnref11)

The post [The basic reasons I expect AGI ruin](https://intelligence.org/2023/04/21/the-basic-reasons-i-expect-agi-ruin/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Misgeneralization as a misnomer

Here’s two different ways an AI can turn out [unfriendly](https://www.lesswrong.com/posts/BSee6LXg4adtrndwy/what-does-it-mean-for-an-agi-to-be-safe):

1. You somehow build an AI that cares about “making people happy”. In training, it tells people jokes and buys people flowers and offers people an ear when they need one. In deployment (and once it’s more capable), it forcibly puts each human in a separate individual heavily-defended cell, and pumps them full of opiates.
2. You build an AI that’s good at making people happy. In training, it tells people jokes and buys people flowers and offers people an ear when they need one. In deployment (and once it’s more capable), it turns out that whatever was causing that “happiness”-promoting behavior was a balance of a variety of other goals (such as basic desires for energy and memory), and it spends most of the universe on some combination of that other stuff that doesn’t involve much happiness.

(To state the obvious: please don’t try to get your AIs to pursue “happiness”; you want something more like [CEV](https://arbital.com/p/cev/) in the long run, and in the short run I strongly recommend [aiming lower, at a pivotal act](https://arbital.com/p/pivotal/) .)

 In both cases, the AI behaves (during training) in a way that looks a lot like trying to make people happy. Then the AI described in (1) is unfriendly because it was optimizing the wrong concept of “happiness”, one that lined up with yours when the AI was weak, but that diverges in various  [edge-cases](https://arbital.com/p/edge_instantiation/)  that matter when the AI is strong. By contrast, the AI described in (2) was never even really trying to pursue happiness; it had a mixture of goals that merely correlated with the training objective, and that balanced out right around where you wanted them to balance out in training, but deployment (and the corresponding capabilities-increases) threw the balance off.

Note that this list of “ways things can go wrong when the AI looked like it was optimizing happiness during training” is not exhaustive! (For instance, consider an AI that cares about something else entirely, and knows you’ll shut it down if it doesn’t look like it’s optimizing for happiness. Or an AI whose goals change heavily as it reflects and self-modifies.)

(This list isn’t even really disjoint! You could get both at once, resulting in, e.g., an AI that spends most of the universe’s resources on acquiring memory and energy for unrelated tasks, and a small fraction of the universe on doped-up human-esque shells.)

The solutions to these two problems are pretty different. To resolve the problem sketched in (1), you have to figure out how to get an instance of the AI’s concept (“happiness”) to match the concept you hoped to transmit, even in the edge-cases and extremes that it will have access to in deployment (when it needs to be powerful enough to pull off some pivotal act that you yourself cannot pull off, and thus capable enough to access extreme edge-case states that you yourself cannot).

To resolve the problem sketched in (2), you have to figure out how to get the AI to care about one concept in particular, rather than a complicated mess that happens to balance precariously on your target (“happiness”) in training.

I note this distinction because it seems to me that various people around these parts are either unduly lumping these issues together, or are failing to notice one of them. For example, they seem to me to be mixed together in “[The Alignment Problem from a Deep Learning Perspective](https://arxiv.org/pdf/2209.00626.pdf) ” under the heading of “goal misgeneralization”.

(I think “misgeneralization” is a misleading term in both cases, but it’s an even worse fit for (2) than (1). A primate isn’t “misgeneralizing” its concept of “inclusive genetic fitness” when it gets smarter and invents condoms; it didn’t even *really have* that concept to misgeneralize, and what shreds of the concept it did have weren’t what the primate was mentally optimizing for.)

(In other words: it’s not that primates were optimizing for fitness in the environment, and then “misgeneralized” after they found themselves in a broader environment full of junk food and condoms. The “aligned” behavior “in training” broke in the broader context of “deployment”, but not because the primates found some weird way to extend an existing “inclusive genetic fitness” concept to a wider domain. Their optimization just wasn’t connected to an internal representation of “inclusive genetic fitness” in the first place.)

---

In mixing these issues together, I worry that it becomes much easier to erroneously dismiss the set. For instance, I have many times encountered people who think that the issue from (1) is a “skill issue”: surely, if the AI were only smarter, it would know what we mean by “make people happy”. (Doubly so if the first transformative AGIs are based on language models! Why, GPT-4 today could explain to you why pumping isolated humans full of opioids shouldn’t count as producing “happiness”.)

And: yep, an AI that’s capable enough to be transformative is pretty likely to be capable enough to figure out what the humans mean by “happiness”, and that doping literally everybody probably doesn’t count. But the issue is, [as](https://www.lesswrong.com/s/SXurf2mWFw8LX2mkG) [always](https://www.lesswrong.com/s/SXurf2mWFw8LX2mkG/p/CcBe9aCKDgT5FSoty), making the AI *care*. The trouble isn’t in making it have *some* understanding of what the humans mean by “happiness” somewhere inside it;[[1]](https://intelligence.org/feed/#fn1) the trouble is making the *stuff the AI pursues* be *that concept*.

Like, it’s possible in principle to reward the AI when it makes people happy, and to separately teach something to observe the world and figure out what humans mean by “happiness”, and to have the trained-in optimization-target concept end up wildly different (in the edge-cases) from the AI’s explicit understanding of what humans meant by “happiness”.

Yes, this is possible even though you used the word “happy” in both cases.

(And this is assuming away the issues described in (2), that the AI probably doesn’t by-default even end up with one clean alt-happy concept that it’s pursuing in place of “happiness”, as opposed to [a thousand shards of desire](https://www.lesswrong.com/posts/cSXZpvqpa9vbGGLtG/thou-art-godshatter) or whatever.)

And I do worry a bit that if we’re not clear about the distinction between all these issues, people will look at the whole cluster and say “eh, it’s a skill issue; surely as the AI gets better at understanding our human concepts, this will become less of a problem”, or whatever.

(As seems to me to be [already](https://twitter.com/MilitantHobo/status/1633040360275341312) happening as people correctly realize that LLMs will probably have a decent grasp on various human concepts.)

---

1. Or whatever you’re optimizing. Which, again, should not be “happiness”; I’m just using that as an example here.

Also, note that the thing you actually want an AI optimizing for in the long term—something like “CEV”—is legitimately harder to get the AI to have any representation of at all. There’s legitimately significantly less writing about object-level descriptions of a [eutopian](https://www.lesswrong.com/posts/K4aGvLnHvYgX9pZHS/the-fun-theory-sequence) universe, than of happy people, and this is related to the eutopia being significantly harder to visualize.

But, again, don’t shoot for the eutopia on your first try! End the acute risk period and then buy time for some reflection instead.[![↩](https://s.w.org/images/core/emoji/14.0.0/72x72/21a9.png)](https://intelligence.org/feed/#fnref1)

The post [Misgeneralization as a misnomer](https://intelligence.org/2023/04/10/misgeneralization-as-a-misnomer/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).