预计阅读本页时间：-
HOW TO PREVENT AN INTELLIGENCE EXPLOSION
Distinguished Professor of Computer Science, Director of Intelligent Systems, Oregon State University
广告：个人专属 VPN，独立 IP，无限流量，多机房切换，还可以屏蔽广告和恶意软件，每月最低仅 5 美元
Much of the rhetoric about the existential risks of artificial intelligence (and superintelligence, more generally) employs the metaphor of the “intelligence explosion.” By analogy with nuclear chain reactions, this rhetoric suggests that AI researchers are somehow working with a kind of smartonium and that if enough of this stuff is concentrated in one place we’ll have a runaway intelligence explosion—an AI chain reaction—with unpredictable results. This is not an accurate depiction of the risks of AI. The mere interconnection of AI algorithms won’t spontaneously take over the universe. Creating an intelligence explosion won’t happen by accident; it will require the construction of a specific kind of AI system that can discover simplifying structures in the world, design computing devices that exploit those structures, and then grant autonomy and resources to those new devices, recursively.
Creating an intelligence explosion requires the recursive execution of four steps. First, a system must be able to conduct experiments on the world. Otherwise it cannot grow its knowledge beyond existing human knowledge. (Most recent advances in AI have been obtained by applying machine learning to reproduce human knowledge, not extend it.) In most philosophical discussions of AI, there’s a natural tendency to focus on pure reasoning, as if this were sufficient to expand knowledge. It’s possible in some special cases (e.g., mathematics and some parts of physics) to advance knowledge through pure reasoning. But across the spectrum of scientific activity, scientific knowledge advances almost exclusively by the collection of empirical evidence for and against hypotheses. This is why we built the Large Hadron Collider, and it’s why all engineering efforts involve building and testing prototypes. This step is clearly feasible, and indeed there already exist some “automated scientists.”
Second, these experiments must discover new simplifying structures that can be exploited to sidestep the computational intractability of reasoning. Virtually all interesting inference problems (such as finding optimal strategies in games, optimizing against sets of complex constraints, proving mathematical theorems, inferring the structures of molecules) are NP-hard. In our current understanding of computational complexity, this means that the cost of solving a problem instance grows exponentially with the size of that instance. Progress in algorithm design generally requires identifying some simplifying structure that can be exploited to defeat this exponential. An intelligence explosion won’t occur unless such structures can be repeatedly discovered (or unless our current understanding of computational complexity is incorrect).
Third, a system must be able to design and implement new computing mechanisms and new algorithms. These mechanisms and algorithms will exploit the scientific discoveries produced in the second step. Indeed, one could argue that this is essentially the same as steps 1 and 2 but focused on computation. Autonomous design and implementation of computing hardware is clearly feasible with silicon-based technologies, and new technologies for synthetic biology, combinatorial chemistry, and 3-D printing will make this even more feasible in the near future. Automated algorithm design has been demonstrated many times, so it’s also feasible.
Fourth, a system must be able to grant autonomy and resources to these new computing mechanisms so they can recursively perform experiments, discover new structures, develop new computing methods, and produce even more powerful “offspring.” I know of no system that has done this.
The first three steps pose no danger of an intelligence chain reaction. It’s the fourth step—reproduction with autonomy—that is dangerous. Of course, virtually all offspring in step 4 will fail, just as virtually all new devices and new software don’t work the first time. But with sufficient iteration—or, equivalently, sufficient reproduction with variation—we can’t rule out the possibility of an intelligence explosion.
How can we prevent an intelligence explosion? We might hope that step 2 fails—that we’ve already found all structural shortcuts to efficient algorithms or that the remaining shortcuts won’t have a big impact. But few electrical engineers or computer scientists would claim that their research has reached its limits.
Step 3 provides a possible control point. Virtually no existing AI systems are applied to design new computational devices and algorithms. Instead, they’re applied to problems such as logistics, planning, robot control, medical diagnosis, face recognition, and so on. These pose no chain-reaction risk. We might consider carefully regulating step-3 research. Similar regulations have been proposed for synthetic biology. But no regulations have been adopted, and they’d be difficult to enforce.
I think we must focus on step 4. We must limit the resources that an automated design-and-implementation system can give to the devices it designs. Some have argued that this is hard, because a “devious” system could persuade people to give it more resources. But whereas such scenarios make for great science fiction, in practice it’s easy to limit the resources a new system is permitted to use. Engineers do this every day when they test new devices and new algorithms.
Steps 1, 2, and 3 have the potential to greatly advance scientific knowledge and computational-reasoning capability, with tremendous benefits for humanity. But it’s essential that we humans understand this knowledge and these capabilities before we devote large amounts of resources to their use. We mustn’t grant autonomy to systems we don’t understand and cannot control.
