Let an ultraintelligent machine be defined as a machine that can
far surpass all the intellectual activities of any any man however
clever. Since the design of machines is one of these intellectual
activities, an ultraintelligent machine could design even better
machines; there would then unquestionably be an "intelligence
explosion," and the intelligence of man would be left far behind.
Thus the first ultraintelligent machine is the _last_ invention that
man need ever make, provided that the machine is docile enough to tell
us how to keep it under control. ... It is more probable than not
that, within the twentieth century, an ultraintelligent machine will
be built and that it will be the last invention that man need make.
Good has captured the essence of the runaway, but does not pursue its
most disturbing consequences. Any intelligent machine of the sort he
describes would not be humankind's "tool" -- any more than humans are
the tools of rabbits or robins or chimpanzees.