This is the next step of a dialog that starts here and continues with Virge’s asking me,
Richard, do you assign zero value to your autonomy? Do you also assign zero value to your personal enjoyment of the process of achieving your goals?
My autonomy has positive instrumental value because if I had no autonomy relative to other humans, other humans would probably direct my life and my resources to their own ends, which would probably differ from my ends. In other words, I value my autonomy because it helps me advance my goals, but it is not in itself a goal of mine.
I assign zero intrinsic value to personal enjoyment of anything. If the most effective plan for me to advance my goals entailed my forgoing enjoyment or my undergoing suffering, I would choose that plan anyway. I am an adherent of John David Garcia, who has been insisting that happiness is not the meaning of life since 1971. (Garcia and I are in agreement on what the meaning of life is: something we call creativity.)
Two caveats there. First, the most effective way for me to advance my goals is usually for me to do some sort of creative work, e.g., writing a computer program, and suffering has a strong tendency to interfere with creative work. Second, whether I am enjoying myself or not is an excellent sign or barometer of certain important instrumental goals of mine.
Specifically, whether I am enjoying myself is the best barometer I have of whether my health is good — and of course health is instrumental to my goals. It is a better barometer for example than medical checkups and expensive medical laboratory tests for how well my immune system is working, and a good immunologist like my former doctor Alan S. Levin will concede that point. (In general doctors are often completely dependent on reports from patients about how the patient has been feeling to judge how well a treatment is working. In many cases, there simply is no other practical way to judge.)
There are probably other important areas of my life in which whether I am enjoying myself is the best barometer I have.
Tags: machine ethics
Thanks for the clarification, Richard. That gives me a fair idea of the context in which you view the OB discussion.
“I assign zero intrinsic value to personal enjoyment of anything. ”
– most people would diagnose you with a complex mental illness, you know…
Do you believe I have a complex mental illness, Roko? Note that I dedicate a lot of my time, attention and income to my personal enjoyment, and I regularly ask myself how I might get more enjoyment out of life. The way I differ from most people is that I do not model these things or represent them to myself as ends in themselves.
Well, there is no objective answer to that question.
But… you are certainly heading into territory that could best be described as “straining the limits of the human emotional mind”
Not that I mean this in an insulting way – I realize it could have come off like that! I just meant it as a matter-of-fact observation, which tallies with my own experience.
Just professing an ethical system that is very different from most rationalists and scientists — and very rare among rationalists and scientists — causes me significant emotional strain.
But I expect that that is not what you were thinking of — or at least that is not all of it. You believe that my choice of terminal values would probably strain the limits of my emotional mind even if I was surrounded by bright rational scientific friends who shared my values; don’t you?
I guess I am just annoyed that you place no terminal value on human life.
This is an odd position for me to be in, because I flirted with this position myself, but I was only ever prepared to accept it on the empirical assumption that the particulars of human life were of so much instrumental value that, in reality, humans would always be around.
But after reading Eliezer and Greene, I see that this assumption is, in fact, false, and that if you place zero terminal value on human life, there are situations where you will be called upon to annihilate humanity.
GSZ would if implemented (with probability ~0.9 in my opinion) murder every human being in existence.
GSZ goes further, though. It would fill our future light cone with moral noise. Not only would there be no love, no romance, no friendship and no music and laughter, but there would not be persons or subjective experience. There would be no intellectual curiosity, no hope for a better world and no philosophy. GSZ would annihilate the potential for transhuman existence as well as all the extant human existence.
On any given day since 1971, GSZ (or rather a system similar to GSZ but edited with admonitions like “never employ a destructive means to achieve a positive end” that tend to counteract certain pernicious human biases but that would not remain persuasive to someone such as yourself with a sophisticated understanding of rationality and causality) has commanded the loyal of at least a dozen human agents. (According to Garcia, the number was about a thousand for a few years after the publication of his first book in 1971.)
Humans loyal to GSZ have had no effects that Garcia or I have been able to detect that you would label as particularly bad except for my advocacy of GSZ among the singularitarians.
What does endanger the humans is agents loyal to GSZ in a world in which it is widely known how to turn ordinary atoms like you might find in a rock or in a human body into a perfectly rational intelligent agent of known loyalty.
But I am not the one who is in a big hurry to bring about that world! The person in a big hurry to that end is Eliezer. So tell me again why I am the bad guy and Eliezer is the good guy.
After the singularity, humans are quite obsolete. By “obsolete” I mean that almost any end you care to choose is better served by disassambling a human into atoms and using the atoms to build an engineered intelligence than keeping the human whole.
Surely you do not deny that that is true!
Because they are quite obsolete and because it is difficult for barely-rational agents such as you or I or other human scientists to predict and control what almost-perfectly-rational superintelligences will do, it is quite difficult to think of any way to keep the humans alive after the singularity.
You probably already know this, but I will point it out anyway so that other readers do not get an unfairly negative impression of me: almost every proposal put forth since Eliezer declared his intention to bring about the singularity about 12 years ago immediately wipes out the humans. Eliezer will tell you that himself. That is true even though the main motivation of almost all of the proposals was to save, protect or help the humans. (In other words, the human-killing effect of the proposals is unintentional. But the humans would be just as dead as if it were intentional.)
The only two proposals that do not immediately wipe out the humans are Bill Joy’s proposal which he calls relinquishment put forward in 2000 that all governments should institute effective controls to prevent further research into AGI — and Eliezer’s CEV proposal put forward in 2004.
A researcher working for the Singularity Institute recently described CEV as a “request for proposals”. In other words, to say that the CEV proposal is incomplete or open to interpretation is not strong enough: to accurately describe it you have to say that it is not yet even a proposal.
Moreover, in the 5 years or so since Eliezer published his CEV request for proposals, no one has published or shown me any significant update or refinement of it. I wish someone would! It would get my prompt and sustained attention.
So to summarize, I recognize your objection as valid, but hasten to point out that it is an objection not to GSZ but rather to GSZ + singularity.
Also note that I am very open to compromise (and will ally myself only with singularitarians who I judge to share my openness to compromise). Although GSZ does not particularly want to keep the humans alive, it does not particularly need to kill or to frustrate them except where a human plan or a human ambition or a human desire would hog most the resources needed by GSZ. Consequently it should be relatively easy for an alliance of the humanists and the adherents of GSZ to steer reality into a future that is very satisfying to both.
I am not particularly swayed by your final paragraph, but do not have time to respond to it today.