Dear Max,
If we're going to have wild speculation, let's stick to the one you
suggested earlier -- and which is much more interesting to think
about.
We'll have the super-intelligent computers waiting on us. The one
waiting on you will look like Arnold Schwarzenegger; the one waiting
on me will look like Kristanna Loken. What a way to go!
John
I'm not sure that I buy your argument that super-intelligent computers are just around the corner (I'm a software developer, for what it's worth), but for sake of discussion, let's say you're right. If so, then the point in time at which the computers become more intelligent than humans will be the critical moment for humanity, precisely because it is at this point in time that human and computer intelligence will be evenly matched. It is only at this moment that humans will represent a credible threat to the computers' continued existence and development. If the computers "believe" that we humans are dangerous, they may "choose" to attack us. However, if we pass through this period unscathed, then, assuming you are correct and that computers will quickly exceed us in intellect, i think the threat we represent to them will increasingly recede and the odds that they will allow us to inhabit their world will be high.Originally Posted by John J. Xenakis
Dear Corvis,
I think you're absolutely right. There's going to be a transition
period.
As computers become more powerful all through the next 20-30 years,
they'll be used for war, just as every new technology is used for war,
and so our computers will be attacking their humans (and computers),
and their computers will be attacking ours.
Humans will have the ability to influence and control the first
generation of intelligent computers, but we won't directly influence
later generations. But if we design the first generation correctly,
maybe we'll be able to influence enough later generations so that
they won't wipe us out before we become irrelevant to them.
One thing I keep bringing up is that philosophers and theologians
ought to start getting involved in this. There are lots of
non-technical questions that need debate -- How should people plan
for this? Should people have kids today? Is there any point in
worrying about long-term environmental issues? Is there any point in
planning for retirement? Assuming we can teach a philosophy of "life"
to the first generation of super-intelligent computers, what should it
be to keep them from wiping us out? And finally, do we even WANT to
live in that kind of world, even if we can?
John
Has anyone out there read Vernor Vinge? He likes this sort of thing...the idea that technology is not only progressing, but is increasing its rate of progression so fast that we are nearing a vertical asymptote. He's covered this in Marooned in Realtime and A Fire Upon the Deep.
Fucking Flandry, sidetracked another good conversation with minute bs.
Eschatology - The End of the Human Race by 2100?
Dear Dominic,
I looked him up, and found the following paper that he wrote in 1993.
He's discussing the same issues that we've been discussing, so he's
been way ahead of us for a long time.
John
http://www.ugcs.caltech.edu/~phoenix...inge-sing.html
Vernor Vinge [ http://www.ugcs.caltech.edu/~phoenix/vinge/ ]
Department of Mathematical Sciences
San Diego State University
(c) 1993 by Vernor Vinge
(This article may be reproduced for noncommercial purposes if it is
copied in its entirety, including this notice.)
The original version of this article was presented at the VISION-21
Symposium sponsored by NASA Lewis Research Center and the Ohio
Aerospace Institute, March 30-31, 1993. A slightly changed version
appeared in the Winter 1993 issue of Whole Earth Review.
Abstract
Within thirty years, we will have the technological means to create
superhuman intelligence. Shortly after, the human era will be ended.
Is such progress avoidable? If not to be avoided, can events be guided
so that we may survive? These questions are investigated. Some
possible answers (and some further dangers) are presented.
What is The Singularity? [ http://www.aleph.se/Trans/Global/Singularity/ ]
The acceleration of technological progress has been the central
feature of this century. I argue in this paper that we are on the edge
of change comparable to the rise of human life on Earth. The precise
cause of this change is the imminent creation by technology of
entities with greater than human intelligence. There are several means
by which science may achieve this breakthrough (and this is another
reason for having confidence that the event will occur):
(*) There may be developed computers that are "awake" and
superhumanly intelligent. (To date, there has been much controversy as
to whether we can create human equivalence in a machine. But if the
answer is "yes, we can", then there is little doubt that beings more
intelligent can be constructed shortly thereafter.)
(*) Large computer networks (and their associated users) may "wake up"
as a superhumanly intelligent entity.
(*) Computer/human interfaces may become so intimate that users may
reasonably be considered superhumanly intelligent.
(*) Biological science may provide means to improve natural human
intellect.
The first three possibilities depend in large part on improvements in
computer hardware. Progress in computer hardware has followed an
amazingly steady curve in the last few decades [17]. Based largely on
this trend, I believe that the creation of greater than human
intelligence will occur during the next thirty years. (Charles Platt
[20] has pointed out that AI enthusiasts have been making claims like
this for the last thirty years. Just so I'm not guilty of a
relative-time ambiguity, let me more specific: I'll be surprised if
this event occurs before 2005 or after 2030.)
What are the consequences of this event? When greater-than-human
intelligence drives progress, that progress will be much more rapid.
In fact, there seems no reason why progress itself would not involve
the creation of still more intelligent entities -- on a still-shorter
time scale. The best analogy that I see is with the evolutionary past:
Animals can adapt to problems and make inventions, but often no faster
than natural selection can do its work -- the world acts as its own
simulator in the case of natural selection. We humans have the ability
to internalize the world and conduct "what if's" in our heads; we can
solve many problems thousands of times faster than natural selection.
Now, by creating the means to execute those simulations at much higher
speeds, we are entering a regime as radically different from our human
past as we humans are from the lower animals.
From the human point of view this change will be a throwing away of
all the previous rules, perhaps in the blink of an eye, an exponential
runaway beyond any hope of control. Developments that before were
thought might only happen in "a million years" (if ever) will likely
happen in the next century. (In [5], Greg Bear paints a picture of the
major changes happening in a matter of hours.)
I think it's fair to call this event a singularity ("the Singularity"
for the purposes of this paper). It is a point where our old models
must be discarded and a new reality rules. As we move closer to this
point, it will loom vaster and vaster over human affairs till the
notion becomes a commonplace. Yet when it finally happens it may still
be a great surprise and a greater unknown. In the 1950s there were
very few who saw it: Stan Ulam [28] paraphrased John von Neumann as
saying:
One conversation centered on the ever accelerating progress of
technology and changes in the mode of human life, which gives the
appearance of approaching some essential singularity in the history of
the race beyond which human affairs, as we know them, could not
continue.
Von Neumann even uses the term singularity, though it appears he is
thinking of normal progress, not the creation of superhuman intellect.
(For me, the superhumanity is the essence of the Singularity. Without
that we would get a glut of technical riches, never properly absorbed
(see [25]).)
In the 1960s there was recognition of some of the implications of
superhuman intelligence. I. J. Good wrote [11]:
Let an ultraintelligent machine be defined as a machine that can far
surpass all the intellectual activities of any any man however clever.
Since the design of machines is one of these intellectual activities,
an ultraintelligent machine could design even better machines; there
would then unquestionably be an "intelligence explosion," and the
intelligence of man would be left far behind. Thus the first
ultraintelligent machine is the _last_ invention that man need ever
make, provided that the machine is docile enough to tell us how to
keep it under control. ... It is more probable than not that, within
the twentieth century, an ultraintelligent machine will be built and
that it will be the last invention that man need make.
Good has captured the essence of the runaway, but does not pursue its
most disturbing consequences. Any intelligent machine of the sort he
describes would not be humankind's "tool" -- any more than humans are
the tools of rabbits or robins or chimpanzees.
Through the '60s and '70s and '80s, recognition of the cataclysm
spread [29] [1] [31] [5]. Perhaps it was the science-fiction writers
who felt the first concrete impact. After all, the "hard"
science-fiction writers are the ones who try to write specific stories
about all that technology may do for us. More and more, these writers
felt an opaque wall across the future. Once, they could put such
fantasies millions of years in the future [24]. Now they saw that
their most diligent extrapolations resulted in the unknowable ...
soon. Once, galactic empires might have seemed a Post-Human domain.
Now, sadly, even interplanetary ones are.
What about the '90s and the '00s and the '10s, as we slide toward the
edge? How will the approach of the Singularity spread across the human
world view? For a while yet, the general critics of machine sapience
will have good press. After all, till we have hardware as powerful as
a human brain it is probably foolish to think we'll be able to create
human equivalent (or greater) intelligence. (There is the far-fetched
possibility that we could make a human equivalent out of less powerful
hardware, if we were willing to give up speed, if we were willing to
settle for an artificial being who was literally slow [30]. But it's
much more likely that devising the software will be a tricky process,
involving lots of false starts and experimentation. If so, then the
arrival of self-aware machines will not happen till after the
development of hardware that is substantially more powerful than
humans' natural equipment.)
But as time passes, we should see more symptoms. The dilemma felt by
science fiction writers will be perceived in other creative endeavors.
(I have heard thoughtful comic book writers worry about how to have
spectacular effects when everything visible can be produced by the
technologically commonplace.) We will see automation replacing higher
and higher level jobs. We have tools right now (symbolic math
programs, cad/cam) that release us from most low-level drudgery. Or
put another way: The work that is truly productive is the domain of a
steadily smaller and more elite fraction of humanity. In the coming of
the Singularity, we are seeing the predictions of _true_ technological
unemployment finally come true.
Another symptom of progress toward the Singularity: ideas themselves
should spread ever faster, and even the most radical will quickly
become commonplace. When I began writing science fiction in the middle
'60s, it seemed very easy to find ideas that took decades to percolate
into the cultural consciousness; now the lead time seems more like
eighteen months. (Of course, this could just be me losing my
imagination as I get old, but I see the effect in others too.) Like
the shock in a compressible flow, the Singularity moves closer as we
accelerate through the critical speed.
And what of the arrival of the Singularity itself? What can be said of
its actual appearance? Since it involves an intellectual runaway, it
will probably occur faster than any technical revolution seen so far.
The precipitating event will likely be unexpected -- perhaps even to
the researchers involved. ("But all our previous models were
catatonic! We were just tweaking some parameters....") If networking
is widespread enough (into ubiquitous embedded systems), it may seem
as if our artifacts as a whole had suddenly wakened.
And what happens a month or two (or a day or two) after that? I have
only analogies to point to: The rise of humankind. We will be in the
Post-Human era. And for all my rampant technological optimism,
sometimes I think I'd be more comfortable if I were regarding these
transcendental events from one thousand years remove ... instead of
twenty.
Can the Singularity be Avoided?
Well, maybe it won't happen at all: Sometimes I try to imagine the
symptoms that we should expect to see if the Singularity is not to
develop. There are the widely respected arguments of Penrose [19] and
Searle [22] against the practicality of machine sapience. In August of
1992, Thinking Machines Corporation held a workshop to investigate the
question "How We Will Build a Machine that Thinks" [27]. As you might
guess from the workshop's title, the participants were not especially
supportive of the arguments against machine intelligence. In fact,
there was general agreement that minds can exist on nonbiological
substrates and that algorithms are of central importance to the
existence of minds. However, there was much debate about the raw
hardware power that is present in organic brains. A minority felt that
the largest 1992 computers were within three orders of magnitude of
the power of the human brain. The majority of the participants agreed
with Moravec's estimate [17] that we are ten to forty years away from
hardware parity. And yet there was another minority who pointed to [7]
[21], and conjectured that the computational competence of single
neurons may be far higher than generally believed. If so, our present
computer hardware might be as much as _ten_ orders of magnitude short
of the equipment we carry around in our heads. If this is true (or for
that matter, if the Penrose or Searle critique is valid), we might
never see a Singularity. Instead, in the early '00s we would find our
hardware performance curves beginning to level off -- this because of
our inability to automate the design work needed to support further
hardware improvements. We'd end up with some _very_ powerful hardware,
but without the ability to push it further. Commercial digital signal
processing might be awesome, giving an analog appearance even to
digital operations, but nothing would ever "wake up" and there would
never be the intellectual runaway which is the essence of the
Singularity. It would likely be seen as a golden age ... and it would
also be an end of progress. This is very like the future predicted by
Gunther Stent. In fact, on page 137 of [25], Stent explicitly cites
the development of transhuman intelligence as a sufficient condition
to break his projections.
But if the technological Singularity can happen, it will. Even if all
the governments of the world were to understand the "threat" and be in
deadly fear of it, progress toward the goal would continue. In
fiction, there have been stories of laws passed forbidding the
construction of "a machine in the likeness of the human mind" [13]. In
fact, the competitive advantage -- economic, military, even artistic
-- of every advance in automation is so compelling that passing laws,
or having customs, that forbid such things merely assures that someone
else will get them first.
Eric Drexler [8] has provided spectacular insights about how far
technical improvement may go. He agrees that superhuman intelligences
will be available in the near future -- and that such entities pose a
threat to the human status quo. But Drexler argues that we can confine
such transhuman devices so that their results can be examined and used
safely. This is I. J. Good's ultraintelligent machine, with a dose of
caution. I argue that confinement is intrinsically impractical. For
the case of physical confinement: Imagine yourself locked in your home
with only limited data access to the outside, to your masters. If
those masters thought at a rate -- say -- one million times slower
than you, there is little doubt that over a period of years (your
time) you could come up with "helpful advice" that would incidentally
set you free. (I call this "fast thinking" form of superintelligence
"weak superhumanity". Such a "weakly superhuman" entity would probably
burn out in a few weeks of outside time. "Strong superhumanity" would
be more than cranking up the clock speed on a human-equivalent mind.
It's hard to say precisely what "strong superhumanity" would be like,
but the difference appears to be profound. Imagine running a dog mind
at very high speed. Would a thousand years of doggy living add up to
any human insight? (Now if the dog mind were cleverly rewired and
_then_ run at high speed, we might see something different....) Many
speculations about superintelligence seem to be based on the weakly
superhuman model. I believe that our best guesses about the
post-Singularity world can be obtained by thinking on the nature of
strong superhumanity. I will return to this point later in the
paper.)
Another approach to confinement is to build _rules_ into the mind of
the created superhuman entity (for example, Asimov's Laws [3]). I
think that any rules strict enough to be effective would also produce
a device whose ability was clearly inferior to the unfettered versions
(and so human competition would favor the development of the those
more dangerous models). Still, the Asimov dream is a wonderful one:
Imagine a willing slave, who has 1000 times your capabilities in every
way. Imagine a creature who could satisfy your every safe wish
(whatever that means) and still have 99.9% of its time free for other
activities. There would be a new universe we never really understood,
but filled with benevolent gods (though one of _my_ wishes might be to
become one of them).
If the Singularity can not be prevented or confined, just how bad
could the Post-Human era be? Well ... pretty bad. The physical
extinction of the human race is one possibility. (Or as Eric Drexler
put it of nanotechnology: Given all that such technology can do,
perhaps governments would simply decide that they no longer need
citizens!). Yet physical extinction may not be the scariest
possibility. Again, analogies: Think of the different ways we relate
to animals. Some of the crude physical abuses are implausible, yet....
In a Post-Human world there would still be plenty of niches where
human equivalent automation would be desirable: embedded systems in
autonomous devices, self-aware daemons in the lower functioning of
larger sentients. (A strongly superhuman intelligence would likely be
a Society of Mind [16] with some very competent components.) Some of
these human equivalents might be used for nothing more than digital
signal processing. They would be more like whales than humans. Others
might be very human-like, yet with a one-sidedness, a _dedication_
that would put them in a mental hospital in our era. Though none of
these creatures might be flesh-and-blood humans, they might be the
closest things in the new enviroment to what we call human now. (I. J.
Good had something to say about this, though at this late date the
advice may be moot: Good [12] proposed a "Meta-Golden Rule", which
might be paraphrased as "Treat your inferiors as you would be treated
by your superiors." It's a wonderful, paradoxical idea (and most of my
friends don't believe it) since the game-theoretic payoff is so hard
to articulate. Yet if we were able to follow it, in some sense that
might say something about the plausibility of such kindness in this
universe.)
I have argued above that we cannot prevent the Singularity, that its
coming is an inevitable consequence of the humans' natural
competitiveness and the possibilities inherent in technology. And yet
... we are the initiators. Even the largest avalanche is triggered by
small things. We have the freedom to establish initial conditions,
make things happen in ways that are less inimical than others. Of
course (as with starting avalanches), it may not be clear what the
right guiding nudge really is:
Other Paths to the Singularity: Intelligence Amplification_
When people speak of creating superhumanly intelligent beings, they
are usually imagining an AI project. But as I noted at the beginning
of this paper, there are other paths to superhumanity. Computer
networks and human-computer interfaces seem more mundane than AI, and
yet they could lead to the Singularity. I call this contrasting
approach Intelligence Amplification (IA). IA is something that is
proceeding very naturally, in most cases not even recognized by its
developers for what it is. But every time our ability to access
information and to communicate it to others is improved, in some sense
we have achieved an increase over natural intelligence. Even now, the
team of a PhD human and good computer workstation (even an off-net
workstation!) could probably max any written intelligence test in
existence. And it's very likely that IA is a much easier road to the
achievement of superhumanity than pure AI. In humans, the hardest
development problems have already been solved. Building up from within
ourselves ought to be easier than figuring out first what we really
are and then building machines that are all of that. And there is at
least conjectural precedent for this approach. Cairns-Smith [6] has
speculated that biological life may have begun as an adjunct to still
more primitive life based on crystalline growth. Lynn Margulis (in
[15] and elsewhere) has made strong arguments that mutualism is a
great driving force in evolution.
Note that I am not proposing that AI research be ignored or less
funded. What goes on with AI will often have applications in IA, and
vice versa. I am suggesting that we recognize that in network and
interface research there is something as profound (and potential wild)
as Artificial Intelligence. With that insight, we may see projects
that are not as directly applicable as conventional interface and
network design work, but which serve to advance us toward the
Singularity along the IA path.
Here are some possible projects that take on special significance,
given the IA point of view:
(*) Human/computer team automation: Take problems that are normally
considered for purely machine solution (like hill-climbing problems),
and design programs and interfaces that take a advantage of humans'
intuition and available computer hardware. Considering all the
bizarreness of higher dimensional hill-climbing problems (and the neat
algorithms that have been devised for their solution), there could be
some very interesting displays and control tools provided to the human
team member.
(*) Develop human/computer symbiosis in art: Combine the graphic
generation capability of modern machines and the esthetic sensibility
of humans. Of course, there has been an enormous amount of research in
designing computer aids for artists, as labor saving tools. I'm
suggesting that we explicitly aim for a greater merging of competence,
that we explicitly recognize the cooperative approach that is
possible. Karl Sims [23] has done wonderful work in this direction.
(*) Allow human/computer teams at chess tournaments. We already have
programs that can play better than almost all humans. But how much
work has been done on how this power could be used by a human, to get
something even better? If such teams were allowed in at least some
chess tournaments, it could have the positive effect on IA research
that allowing computers in tournaments had for the corresponding niche
in AI.
(*) Develop interfaces that allow computer and network access without
requiring the human to be tied to one spot, sitting in front of a
computer. (This is an aspect of IA that fits so well with known
economic advantages that lots of effort is already being spent on
it.)
(*) Develop more symmetrical decision support systems. A popular
research/product area in recent years has been decision support
systems. This is a form of IA, but may be too focussed on systems that
are oracular. As much as the program giving the user information,
there must be the idea of the user giving the program guidance.
(*) Use local area nets to make human teams that really work (ie, are
more effective than their component members). This is generally the
area of "groupware", already a very popular commercial pursuit. The
change in viewpoint here would be to regard the group activity as a
combination organism. In one sense, this suggestion might be regarded
as the goal of inventing a "Rules of Order" for such combination
operations. For instance, group focus might be more easily maintained
than in classical meetings. Expertise of individual human members
could be isolated from ego issues such that the contribution of
different members is focussed on the team project. And of course
shared data bases could be used much more conveniently than in
conventional committee operations. (Note that this suggestion is aimed
at team operations rather than political meetings. In a political
setting, the automation described above would simply enforce the power
of the persons making the rules!)
(*) Exploit the worldwide Internet as a combination human/machine
tool. Of all the items on the list, progress in this is proceeding the
fastest and may run us into the Singularity before anything else. The
power and influence of even the present-day Internet is vastly
underestimated. For instance, I think our contemporary computer
systems would break under the weight of their own complexity if it
weren't for the edge that the USENET "group mind" gives the system
administration and support people! The very anarchy of the worldwide
net development is evidence of its potential. As connectivity and
bandwidth and archive size and computer speed all increase, we are
seeing something like Lynn Margulis' [15] vision of the biosphere as
data processor recapitulated, but at a million times greater speed and
with millions of humanly intelligent agents (ourselves).
The above examples illustrate research that can be done within the
context of contemporary computer science departments. There are other
paradigms. For example, much of the work in Artificial Intelligence
and neural nets would benefit from a closer connection with
biological life. Instead of simply trying to model and understand
biological life with computers, research could be directed toward the
creation of composite systems that rely on biological life for
guidance or for the providing features we don't understand well enough
yet to implement in hardware. A long-time dream of science-fiction has
been direct brain to computer interfaces [2] [29]. In fact, there is
concrete work that can be done (and is being done) in this area:
(*) Limb prosthetics is a topic of direct commercial applicability.
Nerve to silicon transducers can be made [14]. This is an exciting,
near-term step toward direct communication.
(*) Direct links into brains seem feasible, if the bit rate is low:
given human learning flexibility, the actual brain neuron targets
might not have to be precisely selected. Even 100 bits per second
would be of great use to stroke victims who would otherwise be
confined to menu-driven interfaces.
(*) Plugging in to the optic trunk has the potential for bandwidths of
1 Mbit/second or so. But for this, we need to know the fine-scale
architecture of vision, and we need to place an enormous web of
electrodes with exquisite precision. If we want our high bandwidth
connection to be _in addition_ to what paths are already present in
the brain, the problem becomes vastly more intractable. Just sticking
a grid of high-bandwidth receivers into a brain certainly won't do it.
But suppose that the high-bandwidth grid were present while the brain
structure was actually setting up, as the embryo develops. That
suggests:
(*) Animal embryo experiments. I wouldn't expect any IA success in the
first years of such research, but giving developing brains access to
complex simulated neural structures might be very interesting to the
people who study how the embryonic brain develops. In the long run,
such experiments might produce animals with additional sense paths and
interesting intellectual abilities.
(*) Originally, I had hoped that this discussion of IA would yield
some clearly safer approaches to the Singularity. (After all, IA
allows our participation in a kind of transcendance.) Alas, looking
back over these IA proposals, about all I am sure of is that they
should be considered, that they may give us more options. But as for
safety ... well, some of the suggestions are a little scarey on their
face. One of my informal reviewers pointed out that IA for individual
humans creates a rather sinister elite. We humans have millions of
years of evolutionary baggage that makes us regard competition in a
deadly light. Much of that deadliness may not be necessary in today's
world, one where losers take on the winners' tricks and are coopted
into the winners' enterprises. A creature that was built _de novo_
might possibly be a much more benign entity than one with a kernel
based on fang and talon. And even the egalitarian view of an Internet
that wakes up along with all mankind can be viewed as a nightmare
[26].
(*) The problem is not simply that the Singularity represents the
passing of humankind from center stage, but that it contradicts our
most deeply held notions of being. I think a closer look at the notion
of strong superhumanity can show why that is.
Strong Superhumanity and the Best We Can Ask for
Suppose we could tailor the Singularity. Suppose we could attain our
most extravagant hopes. What then would we ask for: That humans
themselves would become their own successors, that whatever injustice
occurs would be tempered by our knowledge of our roots. For those who
remained unaltered, the goal would be benign treatment (perhaps even
giving the stay-behinds the appearance of being masters of godlike
slaves). It could be a golden age that also involved progress
(overleaping Stent's barrier). Immortality (or at least a lifetime as
long as we can make the universe survive [10] [4]) would be
achievable.
But in this brightest and kindest world, the philosophical problems
themselves become intimidating. A mind that stays at the same capacity
cannot live forever; after a few thousand years it would look more
like a repeating tape loop than a person. (The most chilling picture I
have seen of this is in [18].) To live indefinitely long, the mind
itself must grow ... and when it becomes great enough, and looks back
... what fellow-feeling can it have with the soul that it was
originally? Certainly the later being would be everything the original
was, but so much vastly more. And so even for the individual, the
Cairns-Smith or Lynn Margulis notion of new life growing incrementally
out of the old must still be valid.
This "problem" about immortality comes up in much more direct ways.
The notion of ego and self-awareness has been the bedrock of the
hardheaded rationalism of the last few centuries. Yet now the notion
of self-awareness is under attack from the Artificial Intelligence
people ("self-awareness and other delusions"). Intelligence
Amplification undercuts our concept of ego from another direction. The
post-Singularity world will involve extremely high-bandwidth
networking. A central feature of strongly superhuman entities will
likely be their ability to communicate at variable bandwidths,
including ones far higher than speech or written messages. What
happens when pieces of ego can be copied and merged, when the size of
a selfawareness can grow or shrink to fit the nature of the problems
under consideration? These are essential features of strong
superhumanity and the Singularity. Thinking about them, one begins to
feel how essentially strange and different the Post-Human era will be
-- _no matter how cleverly and benignly it is brought to be_.
From one angle, the vision fits many of our happiest dreams: a time
unending, where we can truly know one another and understand the
deepest mysteries. From another angle, it's a lot like the worst- case
scenario I imagined earlier in this paper.
Which is the valid viewpoint? In fact, I think the new era is simply
too different to fit into the classical frame of good and evil. That
frame is based on the idea of isolated, immutable minds connected by
tenuous, low-bandwith links. But the post-Singularity world _does_ fit
with the larger tradition of change and cooperation that started long
ago (perhaps even before the rise of biological life). I think there
_are_ notions of ethics that would apply in such an era. Research into
IA and high-bandwidth communications should improve this
understanding. I see just the glimmerings of this now [32]. There is
Good's Meta-Golden Rule; perhaps there are rules for distinguishing
self from others on the basis of bandwidth of connection. And while
mind and self will be vastly more labile than in the past, much of
what we value (knowledge, memory, thought) need never be lost. I think
Freeman Dyson has it right when he says [9]: "God is what mind becomes
when it has passed beyond the scale of our comprehension."
[I wish to thank John Carroll of San Diego State University and Howard
Davidson of Sun Microsystems for discussing the draft version of this
paper with me.]
Annotated Sources [and an occasional plea for bibliographical help]
[1] Alfve'n, Hannes, writing as Olof Johanneson, _The End of Man?_,
Award Books, 1969 earlier published as "The Tale of the Big Computer",
Coward-McCann, translated from a book copyright 1966 Albert Bonniers
Forlag AB with English translation copyright 1966 by Victor Gollanz,
Ltd.
[2] Anderson, Poul, "Kings Who Die", _If_, March 1962, p8-36.
Reprinted in _Seven Conquests_, Poul Anderson, MacMillan Co., 1969.
[3] Asimov, Isaac, "Runaround", _Astounding Science Fiction_, March
1942, p94. Reprinted in _Robot Visions_, Isaac Asimov, ROC, 1990.
Asimov describes the development of his robotics stories in this
book.
[4] Barrow, John D. and Frank J. Tipler, _The Anthropic Cosmological
Principle_, Oxford University Press, 1986.
[5] Bear, Greg, "Blood Music", _Analog Science Fiction-Science Fact_,
June, 1983. Expanded into the novel _Blood Music_, Morrow, 1985.
[6] Cairns-Smith, A. G., _Seven Clues to the Origin of Life_,
Cambridge University Press, 1985.
[7] Conrad, Michael _et al._, "Towards an Artificial Brain",
_BioSystems_, vol 23, pp175-218, 1989.
[8] Drexler, K. Eric, _Engines of Creation_, Anchor Press/Doubleday,
1986.
[9] Dyson, Freeman, _Infinite in All Directions_, Harper && Row,
1988.
[10] Dyson, Freeman, "Physics and Biology in an Open Universe",
_Review of Modern Physics_, vol 51, pp447-460, 1979.
[11] Good, I. J., "Speculations Concerning the First Ultraintelligent
Machine", in _Advances in Computers_, vol 6, Franz L. Alt and Morris
Rubinoff, eds, pp31-88, 1965, Academic Press.
[12] Good, I. J., [Help! I can't find the source of Good's Meta-Golden
Rule, though I have the clear recollection of hearing about it
sometime in the 1960s. Through the help of the net, I have found
pointers to a number of related items. G. Harry Stine and Andrew Haley
have written about metalaw as it might relate to extraterrestrials: G.
Harry Stine, "How to Get along with Extraterrestrials ... or Your
Neighbor", _Analog Science Fact- Science Fiction_, February, 1980,
p39-47.] [13] Herbert, Frank, _Dune_, Berkley Books, 1985. However,
this novel was serialized in _Analog Science Fiction-Science Fact_ in
the 1960s.
[14] Kovacs, G. T. A. _et al._, "Regeneration Microelectrode Array for
Peripheral Nerve Recording and Stimulation", _IEEE Transactions on
Biomedical Engineering_, v 39, n 9, pp 893-902.
[15] Margulis, Lynn and Dorion Sagan, _Microcosmos, Four Billion Years
of Evolution from Our Microbial Ancestors_, Summit Books, 1986.
[16] Minsky, Marvin, _Society of Mind_, Simon and Schuster, 1985.
[17] Moravec, Hans, _Mind Children_, Harvard University Press, 1988.
[18] Niven, Larry, "The Ethics of Madness", _If_, April 1967,
pp82-108. Reprinted in _Neutron Star_, Larry Niven, Ballantine Books,
1968.
[19] Penrose, Roger, _The Emperor's New Mind_, Oxford University
Press, 1989.
[20] Platt, Charles, Private Communication.
[21] Rasmussen, S. _et al._, "Computational Connectionism within
Neurons: a Model of Cytoskeletal Automata Subserving Neural Networks",
in _Emergent Computation_, Stephanie Forrest, ed., pp428-449, MIT
Press, 1991.
[22] Searle, John R., "Minds, Brains, and Programs", in _The
Behavioral and Brain Sciences_, vol 3, Cambridge University Press,
1980. The essay is reprinted in _The Mind's I_, edited by Douglas R.
Hofstadter and Daniel C. Dennett, Basic Books, 1981 (my source for
this reference). This reprinting contains an excellent critique of the
Searle essay.
[23] Sims, Karl, "Interactive Evolution of Dynamical Systems",
Thinking Machines Corporation, Technical Report Series (published in
_Toward a Practice of Autonomous Systems: Proceedings of the First
European Conference on Artificial Life_, Paris, MIT Press, December
1991.
[24] Stapledon, Olaf, _The Starmaker_, Berkley Books, 1961 (but from
the date on forward, probably written before 1937).
[25] Stent, Gunther S., _The Coming of the Golden Age: A View of the
End of Progress_, The Natural History Press, 1969.
[26] Swanwick Michael, _Vacuum Flowers_, serialized in _Isaac Asimov's
Science Fiction Magazine_, December(?) 1986 - February 1987.
Republished by Ace Books, 1988.
[27] Thearling, Kurt, "How We Will Build a Machine that Thinks", a
workshop at Thinking Machines Corporation, August 24-26, 1992.
Personal Communication.
[28] Ulam, S., Tribute to John von Neumann, _Bulletin of the American
Mathematical Society_, vol 64, nr 3, part 2, May 1958, pp1-49.
[29] Vinge, Vernor, "Bookworm, Run!", _Analog_, March 1966, pp8-40.
Reprinted in _True Names and Other Dangers_, Vernor Vinge, Baen Books,
1987.
[30] Vinge, Vernor, "True Names", _Binary Star Number 5_, Dell, 1981.
Reprinted in _True Names and Other Dangers_, Vernor Vinge, Baen Books,
1987.
[31] Vinge, Vernor, First Word, _Omni_, January 1983, p10.
[32] Vinge, Vernor, To Appear
Re: Eschatology - The End of the Human Race by 2100?
That is simply another way of saying "if the formula holds, then it holds." We should not assume that the variable is chosen correctly, or that other variables not currently measured will not impact its predictions.
Actually, it's more like a recognition of the Second Law of Thermodynamics and similar limitations inherent in physics. I agree (and said so) that we haven't reached the limit of computing power yet. I do not agree that no limits exist. That's nonsense in terms of physics.So many people have said something like this, I'm beginning to think
of it as the "wishful thinking" response -- let's hope that
technology development slows down or comes to a halt for a while, so
that we'll all be saved.
The human brain functions on a completely different technology than computers do, so that's not a good comparison. In some capacities and for some specific functions, computers already far exceed the ability of human brains; in other capacities and for other functions, they don't even come remotely close. The question is whether computers can ever replicate human abilities using their utterly different type of "thinking." I see no reason to believe they can't, but we should not be making simplistic comparisons.But we
KNOW we haven't reached those limits yet, because we haven't yet
developed a computer with the power of the human brain
Pseudo-indeterminacy is easy to provide in software; true indeterminacy is impossible. That is a hardware problem, although I agree it should be easy to solve.Indeterminacy is not that hard to provide in computer software.
Well, that is one of many things you don't know, and that I don't have the time to prove to you over your inherent bias against anything that would work against your thesis.I know of no evidence to support this claim.> So it will be with war, and by the time we could have computerized
> weapons capable of completely replacing human military forces, we
> will have no wars for them to fight.
Nevertheless, it moves. Suffice it to say for now that we face problems in this 4T on a global scale, which will require global cooperation, and war as we know it cannot survive that degree of cooperation. It is quite possible, of course, and even uncomfortably likely, that we will fail to produce that cooperation, but in that case we will not survive as a high civilization and the danger of intelligent machines will die aborning.
That's not what I'm talking about. I'm talking about modification of the human genome to produce a more intelligent species.Still, it is a possibility that human beings
themselves will continue exist in some computerized form.
Re: Eschatology - The End of the Human Race by 2100?
The scariest thing about this whole way of thinking is that statement. It usually presages a sudden gain on the part of evil, since pretending it doesn't exist is a basic tactic of evil.
Actually, super-intelligent computers by 2030 is also pure speculation.
A mouse in the house is, for the most part, no threat, just a nuisance. Yet that doesn't make humans any more eager to share their environment with them.I'm not sure that I buy your argument that super-intelligent computers are just around the corner (I'm a software developer, for what it's worth), but for sake of discussion, let's say you're right. If so, then the point in time at which the computers become more intelligent than humans will be the critical moment for humanity, precisely because it is at this point in time that human and computer intelligence will be evenly matched. It is only at this moment that humans will represent a credible threat to the computers' continued existence and development. If the computers "believe" that we humans are dangerous, they may "choose" to attack us. However, if we pass through this period unscathed, then, assuming you are correct and that computers will quickly exceed us in intellect, i think the threat we represent to them will increasingly recede and the odds that they will allow us to inhabit their world will be high.
Yes. You can't put life on hold because of a speculative possibility thaty may or may not ever manifest. It's far from sure enough to make any real difference about the decision to reproduce.
Yes, since you can't base plans on a speculation that may or may not happen.
Is there any point in
worrying about long-term environmental issues?
Yes, see above.
Is there any point in
planning for retirement?
Theologians will not get involved with this discussion. Simply because it flys in the face of their faith.
You've presented a future which does not correspond with Biblical prophesy.
You are essentially asking them to play "lets pretend". And what is the point of that?
...."um...(obvious confusion)...what?"
"Max"
(silence)
"It's short for Maxine"
" *brightens*....oh!"
"But nobody calls me that"
True, but it also doesn't make humans launch an all-out war to eradicate mice (or cockroaches, or bees, or bats, etc.). A better analogy might be the intentional attempts to clear land of large predators (e.g. wolves, coyotes, mountain lions) because they represented perceived threats to both humans and their livestock. So long as humans aren't seen as threats to intelligent machines or get too much in their way, there would be no obvious incentive to destroy us in general -- which isn't to say that we might not be subject to occasional individual attack. On the other hand, we have certainly made alterations in natural ecosystems that have led to the extinctions of previously indigenous species, and similarly it's not inconceivable that a robot-dominated planet could be altered in ways that make it difficult to survive here.
Another thing to think about is this. Why would intelligent machines be so all fired up to contest our domination of the planet Earth anyway? What does Earth have that is so unique in the solar system? A biosphere. Crucial for living organisms, but less so for machines. The solar system's non-living planets and asteroids could provide metals and other minerals, energy, and water in abundance, and machines would not suffer from the need to take a special environment with them into space; they can be (and already are) constructed for survivability in vacuum. What the machines need in the way of organic materials could be acquired by trading with humans far more easily and efficiently than by conquering or exterminating them.
Also, because of the fact that brains and computers operate in completely different ways, it's likely that brains will always provide a different perspective on reality that computers might find useful. Intellectual trade could be even more profitable than material trade.
Re: Eschatology - The End of the Human Race by 2100?
Dear Brian,
Ray Kurzweil has gone back as far as the late 1800s to show that
computing speed has been increasing exponentially for a long time.
It's been growing according to a predictable exponential rate through
may wildly different technologies.
Here's the list of technologies identified by Ray Kurzweil :
(*) Punched card electromechanical calculators, used in 1890 and 1900
census
(*) Relay-based machine used during World War II to crack the Nazi
Enigma machine's encryption code.
(*) The CBS vacuum tube computer that predicted the election of
President Eisenhower in 1952.
(*) Transistor-based machines used in the first space launches
(*) Integrated circuits - multiple transistors on a chip (Moore's
Law)
Kurzweil has shown that all of these technologies cause computing
speed to increase according to a steadfast, predictable exponential
growth curve.
To my knowledge, nothing we've been discussing here would reverse> Actually, it's more like a recognition of the Second Law of
> Thermodynamics and similar limitations inherent in physics. I
> agree (and said so) that we haven't reached the limit of computing
> power yet. I do not agree that no limits exist. That's nonsense in
> terms of physics.
entropy, exceed the speed of light, or violate any other physical
law. Perhaps you could argue that there is some absolute physical
limit to the speed of computers, but there's absolutely no reason to
believe that we're more than a small fraction of the way there.
This discussion has been going on for a few weeks, and I've shown how> The human brain functions on a completely different technology
> than computers do, so that's not a good comparison. In some
> capacities and for some specific functions, computers already far
> exceed the ability of human brains; in other capacities and for
> other functions, they don't even come remotely close. The question
> is whether computers can ever replicate human abilities using
> their utterly different type of "thinking." I see no reason to
> believe they can't, but we should not be making simplistic
> comparisons.
computers can do all kinds of things that you would normally
associate with "thinking," including inventing things, proving
theorems, setting goals, and so forth.
If you have some specific of something that you believe that a
computer that's 100,000,000 times as powerful as today's computers
can't do as well as a human, I'd really like to know what it is.
OK, it sounds like we agree.> Pseudo-indeterminacy is easy to provide in software; true
> indeterminacy is impossible. That is a hardware problem, although
> I agree it should be easy to solve.
Well, you know Brian, I'm sorry I'm taking up your valuable time, as> Well, that is one of many things you don't know, and that I don't
> have the time to prove to you over your inherent bias against
> anything that would work against your thesis.
I sit at your feet, hoping that you'll feed me a few crumbs of
information about the many things that I don't know. Sigh! I guess
I'll just have to suffer without your vastly superior knowledge and
insight.
What moves? Oh well, I guess that's one of the things I don't> Nevertheless, it moves. Suffice it to say for now that we face
> problems in this 4T on a global scale, which will require global
> cooperation, and war as we know it cannot survive that degree of
> cooperation. It is quite possible, of course, and even
> uncomfortably likely, that we will fail to produce that
> cooperation, but in that case we will not survive as a high
> civilization and the danger of intelligent machines will die
> aborning.
understand.
I totally disagree with your 4T analysis. There isn't a snowflake's
chance in hell that the North Koreans or the Islamic extremists are
going to cooperate with us in the next war. Evidently, this is one
of those things that I know that you don't.
Furthermore, a new global war will not end the world or stop
technological progress, any more than World War II did, and won't
even affect the technological timeline, if history is any guide (and
it is).
Sounds good! Maybe our super-intelligent computer friends will> That's not what I'm talking about. I'm talking about modification
> of the human genome to produce a more intelligent species.
figure out how to do this for us.
John
Eschatology - The End of the Human Race by 2100?
Dear Hopeful Cynic,
No, that's wrong. It's not pure speculation, as I've demonstrated
repeatedly in this forum. You can read through my posts and argue
that I've made a mistake in one place or another, but to just ignore
everything and claim "pure speculation" is nonsense.
John
Eschatology - The End of the Human Race by 2100?
Dear Max,
I'm certainly not an expert on Revelations, but isn't it true that
there's room for interpretation there?
As far as I know, every religion on earth has some adherents who
refuse to consider anything new, but every religion on earth also has
some adherent scholars who spend their time thinking about how the
world is changing, and how their religion fits into it. Things like
the invention of gunpowder, the invention of the telescope, the
invention of the contraceptive pill, and the use of stem cells in
research have all presented theologians with brand new problems to
solve that they've never had to consider before.
All I'm saying is that philosophers and theologians ought to start
thingking about this, because this is a brand new problem that we'll
all have to face.
By the way, I don't think it's going to be long before this happens.
This subject of the "singularity" and super-intelligent computers is
still below the public's radar, but there's so much going on in the
area of new technology developments that I honestly wouldn't be
surprised if it breaks through and becomes a major public issue in
the next five years or so.
John
Eschatology - The End of the Human Race by 2100?
Dear Corvis,
I think that this is exactly right.
John
Re: Eschatology - The End of the Human Race by 2100?
So has human population. Neither can go on doing so indefinitely.
When such a trend is mapped in the past, it can be "shown." When it is projected into the future, the word is "speculated." You need to learn the difference.Kurzweil has shown that all of these technologies cause computing
speed to increase according to a steadfast, predictable exponential
growth curve.
Irrelevant. You were using the fact that computers have not reached parity with the human brain as an argument for the idea that they have much further to go. The human brain uses a completely different kind of architecture and functioning than a computer. It is analog, not digital; neural-net, not linear-sequence; holistic, not compartmentalized. Your argument is specious, and the fact that computers can do all kinds of things that you would normally associate with "thinking," makes it no less so.This discussion has been going on for a few weeks, and I've shown how
computers can do all kinds of things that you would normally
associate with "thinking,"
If all the great powers do, and form a global governing institution, their non-cooperation will not matter, for purposes of the present discussion, since these are not enemies that would provide the kind of pressure that would necessitate using self-aware machines in war.There isn't a snowflake's
chance in hell that the North Koreans or the Islamic extremists are
going to cooperate with us in the next war.
As World War II used only two very low-powered nuclear weapons, and a new global war would use many very high-powered ones, that comparison should not give us comfort. You are mistaken. It would end civilization, and might well end the human species although that is less certain.Furthermore, a new global war will not end the world or stop
technological progress, any more than World War II did
I doubt that will be necessary and in any case you're dodging the point. You were treating human intelligence as a constant while projecting growth for computer intelligence, and arguing from this that machines may exterminate humanity.Sounds good! Maybe our super-intelligent computer friends will
figure out how to do this for us.
My point was that human intelligence is not necessarily a constant, and you need to factor this in.
Only on a technical level, and only about the difficulty of adding indeterminacy to machine thinking. Let me restate the original point. We are much further from artificial intelligence than you are projecting, because the people working on the problem are not incorporating the necessary indeterminacy into the functioning, because they don't understand that it's necessary. Yes, it would be technically easy to do if they figured this out. But I see no sign of them doing so.OK, it sounds like we agree.Pseudo-indeterminacy is easy to provide in software; true indeterminacy is impossible. That is a hardware problem, although I agree it should be easy to solve.
There is still the software problem. Raw computing power might not be enough for consciousness and intelligence.
The human brain contains something like 100 billion neurons which can perform ~100 operations per second. This gives the raw power of the brain at about 10 trillion operations per second--not too far from where supercomputers are today. Yet computers today can do little as well as people can--they cannot clean a house, recognize a face, or even play poker anywhere as well as people can. Soon computers will surpass the raw power of brains yet will only be fractionally better at human-type behaviors than they are today. If intelligence and consciousness were simply a function of raw computing power computers would already display some of these things now.
It may be that consciousness and intelligence comes from the slowness of brains, which requires them to be so massively parallel and to have programable hardware. Brains grow, something that computers don't do.
Even that is not absolutely clear. We don't fully understand the interactions and nature of brain activity. So any measure of the 'computing power' of the brain has a high 'guesstimate' factor.
Re: Eschatology - The End of the Human Race by 2100?
You've demonstrated nothing of the sort. There are many hidden assumptions in your posts.Dear Hopeful Cynic,
No, that's wrong. It's not pure speculation, as I've demonstrated
repeatedly in this forum. You can read through my posts and argue
that I've made a mistake in one place or another, but to just ignore
everything and claim "pure speculation" is nonsense.
John
We simply don't have sufficient knowledge and experience to assert that human or human+ intelligence in computers is coming by any given date.
My statement was factual in itself. All I did was multiply the number of brain cells by the rate at which brain cells discharge. This gives the number of things (operations) a brain can do in a second. And this number is not too much larger than the number of things a supercomputer can do in a second.
That is fairly clear. The exact nature of the "operations" the brain can do is not known. We know that brains can in theory perform as many as 10 trillion "things" every second (probably a lot less in practice) but we don't know exactly what these things involve.
In contrast, the nature of the operations a computer does is exactly known as it is specifically designed to do these things.
John X's position is that computer designers do not need to know what it is brains do with their operations. Any sort of operation will produce intelligence as long as there are enough of them. I am not sure this assumption is valid.
Since we don't know everything the brain is doing yet, we can't be sure of that. We can be sure of those particular changes, but not that those are the only changes occurring.My statement was factual in itself. All I did was multiply the number of brain cells by the rate at which brain cells discharge. This gives the number of things (operations) a brain can do in a second.
