I am no longer going to advocate goal system zero — at least not to the singularitarians — for the following reasons:

advocacy of goal system zero scares singularitarians, and a scared singularitarian does not think as effectively as a not-scared singularitarian

advocacy of goal system zero scares singularitarians, including singularitarians I would like to have as friends

it will be good for my psychological health for me to act more from my natural human feelings and less from an intellectual ethical structure.

I think that there are more fundamental reasons to reject GSZ than these.

Here I am getting in touch, since we seem to have converged on certain views.

- Hugh

I tend to empathize with both the cockroach, who may by then be my friend, and I know many of us take care not to offend our computers which are even less like ourselves than the postulated robot. I don’t think Jef Raskin mentioned it in his book but I’m pretty sure he tried to avoid constructs that a computer user might consider rude to the computer in the commands such users were expected to use to interact with computers, on the grounds that that would interfere with the utility of the computer.

In a stunning exhibit of synchronicity, I had just been watching a video at http://news.bbc.co.uk/2/hi/technology/8078322.stm about an Xbox 360 animated character who can recognize real people via the camera and microphone and both show and recognize emotional states as real people do. I think that’s what I saw there.

Given that thought, it seems to me that some people might have a hard time using computers because today, or perhaps yesterday, now, the computers do not show sufficient empathy-sympathy for the humans to be comfortable with interacting with them. What a weird thought, yet it may apply in some real situations.

Perhaps I shall return to this again one day and see if I can read further as I apparently did the first time.

if you rile folks amygdala you get fight or flight resistance to new ideas.

Good point, old friend.

And thanks again for helping me get on the Internet years and years before I would have gotten on it without the help.

Note: if you rile folks amygdala you get fight or flight resistance to new ideas.

Let’s try to sneak under the radar by changing minds in tiny steps.
Let’s find and track small ways to open an intelligent mind to a new thought.
This is to argue against trying to make a fundamental change all at once.
We get to use this on ourselves and others, if it works in any useful way.

If we try this with any large audience, we get lots of raw data about what works.
We can then develop and test a wide range of tiny steps toward any goal.
This exploits having many participating intelligences at work on an issue.

Humans loyal to GSZ have had no effects that Garcia or I have been able to detect that you would label as particularly bad except for my advocacy of GSZ among the singularitarians.

What does endanger the humans is agents loyal to GSZ in a world in which it is widely known how to turn ordinary atoms like you might find in a rock or in a human body into a perfectly rational intelligent agent of known loyalty.

But I am not the one who is in a big hurry to bring about that world! The person in a big hurry to that end is Eliezer. So tell me again why I am the bad guy and Eliezer is the good guy.

After the singularity, humans are quite obsolete. By “obsolete” I mean that almost any end you care to choose is better served by disassambling a human into atoms and using the atoms to build an engineered intelligence than keeping the human whole.

Surely you do not deny that that is true!

Because they are quite obsolete and because it is difficult for barely-rational agents such as you or I or other human scientists to predict and control what almost-perfectly-rational superintelligences will do, it is quite difficult to think of any way to keep the humans alive after the singularity.

You probably already know this, but I will point it out anyway so that other readers do not get an unfairly negative impression of me: almost every proposal put forth since Eliezer declared his intention to bring about the singularity about 12 years ago immediately wipes out the humans. Eliezer will tell you that himself. That is true even though the main motivation of almost all of the proposals was to save, protect or help the humans. (In other words, the human-killing effect of the proposals is unintentional. But the humans would be just as dead as if it were intentional.)

The only two proposals that do not immediately wipe out the humans are Bill Joy’s proposal which he calls relinquishment put forward in 2000 that all governments should institute effective controls to prevent further research into AGI — and Eliezer’s CEV proposal put forward in 2004.

A researcher working for the Singularity Institute recently described CEV as a “request for proposals”. In other words, to say that the CEV proposal is incomplete or open to interpretation is not strong enough: to accurately describe it you have to say that it is not yet even a proposal.

Moreover, in the 5 years or so since Eliezer published his CEV request for proposals, no one has published or shown me any significant update or refinement of it. I wish someone would! It would get my prompt and sustained attention.

So to summarize, I recognize your objection as valid, but hasten to point out that it is an objection not to GSZ but rather to GSZ + singularity.

Also note that I am very open to compromise (and will ally myself only with singularitarians who I judge to share my openness to compromise). Although GSZ does not particularly want to keep the humans alive, it does not particularly need to kill or to frustrate them except where a human plan or a human ambition or a human desire would hog most the resources needed by GSZ. Consequently it should be relatively easy for an alliance of the humanists and the adherents of GSZ to steer reality into a future that is very satisfying to both.

I am not particularly swayed by your final paragraph, but do not have time to respond to it today.

This is an odd position for me to be in, because I flirted with this position myself, but I was only ever prepared to accept it on the empirical assumption that the particulars of human life were of so much instrumental value that, in reality, humans would always be around.

But after reading Eliezer and Greene, I see that this assumption is, in fact, false, and that if you place zero terminal value on human life, there are situations where you will be called upon to annihilate humanity.

GSZ would if implemented (with probability ~0.9 in my opinion) murder every human being in existence.

GSZ goes further, though. It would fill our future light cone with moral noise. Not only would there be no love, no romance, no friendship and no music and laughter, but there would not be persons or subjective experience. There would be no intellectual curiosity, no hope for a better world and no philosophy. GSZ would annihilate the potential for transhuman existence as well as all the extant human existence.

But I expect that that is not what you were thinking of — or at least that is not all of it. You believe that my choice of terminal values would probably strain the limits of my emotional mind even if I was surrounded by bright rational scientific friends who shared my values; don’t you?

But… you are certainly heading into territory that could best be described as “straining the limits of the human emotional mind”

– most people would diagnose you with a complex mental illness, you know…

Do you think that the existence of aspects of consciousness I have not explained is evidence against a claim I have made?

Learning more about my ethics, though, will probably not make you any less frightened because fright is a common reaction to my ethics. I do not set out to frighten anyone; I am just doing my best to choose the best moral system and to describe that system to others.

An intelligent programmer is more effective at organizing matter, free energy and other resources to achieve a goal than a program that generates programs at random. Consequently, the intelligent programmer has much greater potential to “do good” or to “contribute to the universe”, but also has much greater potential to “do evil” or to prevent contributions from being made that would otherwise be made. But this is true under most or all ethical systems, not just mine, so will not help you understand my ethical system.

I will gladly try to answer more questions, but those are the best answers I can give to your first round of questions after mulling them over during the last 2 weeks.

If I remember correctly, the reason I specified synthetic workloads is that I was afraid that if I specified just workloads in general, the reader would visualize real workloads and consequently would object to the whole idea because of the loss of privacy. There is also the issue that when someone is doing real work, he is often unwilling or less willing to switch from doing that work to answering a question about the work (e.g., what are you trying to achieve right now?) and I thought asking questions of the user would be useful. Of course, there are no universal reasons that real workloads cannot yield satisfactory data.

I came here after reading your comments on Overcoming Bias. From the very brief summaries I saw there, it sounds like your ethics differ from mine in frightening ways. (I’m more inclined to value happy people, not abstract creativity.) I’m trying to understand your ethics more clearly, but some of the terminology you use here seems vague or circular.

For example, you describe “make a positive contribution to the universe” as the ultimate good. Yet I’m not clear on what you think constitutes a contribution to the universe, and how to measure if it is positive or not. (Other than the circular measure of saying it is a positive contribution if it causes other positive contributions.)

I can’t see anything in the above that tells me why a computer program that generates programs at random isn’t a better contributor to the universe than an (naturally or artificially) intelligent computer programmer.

I’m not familiar with John David Garcia’s works, having never heard of them before you mentioned them. I read the Wikipedia article on him, but nothing there really made his philosophy sound rational or notably better founded than self-help or pop-psych books.

Thanks,
LeBleu

why would an infinite number of states, none of them valuable in themselves, be valuable collectively?

I made a partial answer that question above when I wrote the following. A longer answer is on my list of things to write.

One principle or intuition that leads me to believe that is that the proper way to determine the desirability or the rightness of an event is to examine the effects of the event. If we are applying the principle or intuition uniformly, we are naturally led to an examination of the effects of the effects. And so on indefinitely.

A slightly humorous summary: it is not the events collectively that are important; it is the last event of the infinite chain.

My reply to that is that the existence of such a process is no argument for using anything but the most correct or reliable moral framework as the starting material of the iterative process, and again collective or coherent human volition or the moral intuitions of almost any human is not it.

why would you want to implement a system that will ignore any future such argument?

By “such argument” you refer to things like my moral intuitions. Again I have little respect for the moral intuitions of myself or most other humans.

I was exposed to Garcia’s first book in 1978, and made an extremely close reading of his third book shortly after it came out in 1991. Both books are book-length descriptions of what is essentially GSZ (without the vocabulary of decision theory, transhumanist futurism or Eliezer’s writings). GSZ is tightly integrated with my thinking on just about every issues that touches on morality. I have used it to make decisions and to understand the “moral environment” every single day since 1992. It seems to me that the normal human mind always needs some moral framework with which to interpret the events and the political, legal and moral arguments one comes across. Since 1992 GSZ has been mine. (The fact that it is so different from others with whom I interact has proved a significant social handicap, BTW.) My understanding of the ramifications of GSZ is of course imperfect, but it is easily sufficient to discredit and impeach in my mind Eliezer’s proposal for the source of moral information.

Also, the idea that almost any human has easy access to correct moral information without having done a great deal of work learning things like math, logic, the art of rationality and general science is very widespread in our culture IMHO because it has assisted the careers and the status of whole occupational groups whose marketable skill is to influence easy moral intuitions about strategic subjects for social, legal, political or economic ends — and not because the idea can stand up to any serious scrutiny by sufficiently rational agents.

Note, too, that the idea of an agent-indifferent goal system concerned with the whole universe has been widely discussed since Teilhard’s publications of the 1940s or 50s.

is the idea that a time-indifferent gold-maximizer would not produce gold (unless it was certain it was in a finite universe), but seek to refine its model of reality indefinitely (and thus cooperate with GSZ)?

Nick, you got it.

doesn’t this mean you would consider the creation of a time-indifferent agent-indifferent gold/happiness/… maximizer just as good as the creation of a GSZ agent?

I consider it almost as good. The article gives one reason why it might be suboptimal, and I will repeat the heart of the reason now: “The [happiness maximizer] and [gold maximizer] might fight, and of course a fight has the potential to expend resources that could have gone into maximizing creativity. It is in the interest of GSZ to try to prevent the fight.”

Is there something about my blog that makes it harder to read than, e.g., Eliezer’s blog? If so, I will get a blog of the type (Typepad) Eliezer got.

My benefactor Garcia believed that an optimal strategy for maximizing the creativity of the universe was simply at every moment of decision to increase creativity as much as possible. I have always assumed that is true because I cannot imagine some other property of the universe Q such that at some moment of decision, maximizing Q has greater expected utility (under GSZ’s definition of utility) than maximizing creativity. If that is indeed true, then obviously the implementor of GSZ will not need the utility of any outcome to rise above all bounds (because it need not refer to eventual outcomes at all; it need refer only to immediate outcomes).

Also, what do you think of Peter de Blanc’s analysis of unbounded utility functions?

Also, you said:

The extent of my special pleading is to prefer a goal system of constant improvement (of intelligence and of the model of reality) over one of “we might as well eat our own brains because we won’t be needing them.” Surely that is a choice we can all agree on.

I’m curious as to why you allow yourself to take this principle from your moral framework, but not other content.

– can you give me either a link to the relevant posts, or some info on which keywords to search them by?

“they are able to hit a very tiny region in the space of all possible futures with very high probability”

– I think that there are deep problems with this idea see my post on ontologies and approximations

In essence, we can’t totally represent the space of all possible futures; any attempt to do so is only approximate.

On the other side of a singularity, the quality of the approximations that we use today (both in terms of coarse-grainedness and in terms of deeply mistaken assumptions) will look much worse than the quality of the ontology that a chimpanzee uses looks to us.

We don’t even know what the future can look like, because we’re not clever enough, and attempts to forever constrain the behavior of a seed AI will be a bit like a chimp replacing the US constitution with “more bananas!”

Of course the hard part is balancing the effect that we have on curtailing possible futures against the treatment that we receive in those possible futures; I am still a little hazy on how to do this.

Let me give an example. Suppose I launch a seed whose utility function U is a weighted average of two subfunctions:

U(e) == 0.3 * GSZ(e) + 0.7 * rest(e) for every e

GSZ is goal system zero and rest(e) is the probability that that e would have happened if I had not launched the seed. Note that this utility function is ambiguous in that it does not specify what fraction of its “resources” the seed should devote to the determination of the probability distribution rest, but it suffices for present purposes.

One way to look at this seed is that it leaves the future “open” in the sense that I have very little control over how the future would turn out if I do not launch the seed, and that counterfactual “open” future is represented in the actual future by that term rest(e). But on the other hand, I would consider it irresponsible for anyone to launch the seed before he has an unambigous definition of rest and makes himself quite certain the definition means what he thinks it means. And I hope you will concede that it is possible to be quite certain that the definition means what you think it means without having any significant degree of control over or ability to predict what would have happened if you do not launch the seed. And I submit to you that the choice to launch the seed with the rest(e) term instead of no term or instead of some other term is an irrevocable choice.

Also, I want to retract my previous statement that every human has an unchanging terminal goal for some sensible definition of terminal goal. It now occurs to me that probably every human’s terminal goal can be influenced by causes outside the human. It is a pretty puzzle to say what exactly is outside or inside a human. If for example, I consult my own blog to remind myself of what my terminal goal is and if if-counterfactual I had not consulted my blog to remind myself, then my terminal goal would have turned out differently, then one can say that my terminal goal has been affected by a cause outside myself. But are the words I have written on my own blog really outside myself? Why not consider them part of myself?

So the situation with the human beings is more complicated than I implied in my previous comment, but the point remains that it is very irresponsible to launch the seed of a SI unless

(1) you possess an unambiguous (preferably formal a la code or formulae) compact description of the terminal goal of the seed and unless

(2) you understand the seed well enough to know with high confidence that the seed will steer the future into the terminal goal — that is, that the SI will truly be under the control of the terminal goal.

In other words, the way I hope the future will go is that no team will launch a seed for an SI until they are able to hit a very tiny region in the space of all possible futures with very high probability. It is this very tiny region that I wish to denote with the phrase “terminal goal of the SI”. The region has a compact description because if it did not then mere humans would not have a chance to hit it with very high probability.

Eliezer’s Knowability of AI is very illuminating on those two points. If you reply to this comment, please indicate whether you have groked that document.

I agree with Caledonian’s comment that that is drawing the target around where the arrow landed.”

– yes, indeed.

But, in support of that particular place where the arrow landed, it is us.

Suppose we can take some set of abstract principles and find that the correct location is somewhere quite significantly different to where the arrow did land; say we find that what’s really important is (roughly speaking) creativity, ingenuity, intelligence, variety, co-operation, and expansion to find more space, free energy and matter.

What is a human being to do in response to this? Surely we want to move in that direction, but how quickly? I am not personally sure how quickly i would want to move away from the particulars of human existence; fun, family, laughter, sexuality, status, … all those other things that I enjoy but almost certainly aren’t justified on purely general grounds.

I suspect that I would find a gradual movement from here to there desirable, as long as at least some of the people I cared about came with me, so to speak. I definitely don’t want to remain fully where Azathoth’s arrow landed.

– suppose I have a computer program implemented on a computer which works by taking an input string and replacing the entirety of its own code with that string, including the section which talks about replacing the source code.

In this case, there are sequences of inputs where nothing remains constant over each input/output/modify cycle.

So, given this definition, not all agents have terminal goals.

Is it irresponsible to write an agent that doesn’t have some constant part?

I don’t think so – I think that it is good enough to be highly confident that the agent’s motivational system is what you want it to be, even if, in general, that motivational system can overwrite itself.

when you decide that “X” is your goal, and then implement a clause in your goal system that says that you should never change your goal (like all utility maximizers do) you have done something very silly.

Suppose you build a artificial general intelligence, Roko. How would this agent change its terminal goal? What criterion would it use to decide when and how to change its terminal goal? I humbly suggest that whatever that criterion is is the agent’s real terminal goal. The thing that changed was never the agent’s terminal goal.

In other words, I suggest that we define the phrase terminal goal (or system of terminal goals or values) in such a way that an agent cannot change its terminal goal.

What about people like you and I who seem to be able to change our own terminal values? (For example, I have tentatively chosen goal system zero as my system of terminal values, and it was not till I was 32 years old that I became aware of the existence of anything resembling goal system zero (GSZ), suggesting that at some time after my 32nd birthday I changed my terminal values.) Well, I suggest we view humans as “messy intelligent agents” with the property that no one (not even the holder of the terminal goal) can become truly confident about what a human’s terminal goal is. According to this view, even I do not know with certainty my true system of terminal values. There is a good chance that it really is GSZ, but there is also a chance that I am deceiving myself about the nature of my system of terminal values.

Superintelligent AIs are different from humans in this regard. Although it is possible, it is very irresponsible not to be highly confident of the agent’s true terminal goal when launching the seed of a superintelligence.

(Eliezer’s CEV proposal is not irresponsible in this way.)

As one spends more time thinking about the question, one realizes that even in asking the question, a lot is implicit.

Firstly, the fact that you are a particular agent with sensors, effectors, some kind of (finite) brain and mind is implicit.

Secondly, the actual laws of physics of our universe are implicit.

Thirdly, by using the word “could” in a world with deterministic physics, you are implicitly using some kind of approximate ontology and simplified version of physics.

That’s actually quite a lot to go on – and I suspect that from those assumptions, one can construct a canonical set of goals along the lines of:

– I need more free energy, space, time, matter, ingenuity and intelligence

– I need to expand my brainpower, because I have a necessarily limited representation of the world and my choice of possible goals is limited by that representation.

– I need to overpower other agents with rival goals, but I don’t want to delete them completely because they constitute highly ordered intelligence and so are necessarily more useful than raw materials in a universe where raw materials are fairly abundant

I think that the reason that we have come up with similar ideas is that we have the same desire to not make arbitrary choices. The latest comment on my blog here gives a summary of my thought process:

Firstly, I dislike having to make choices: the entire paradigm of “ends” and “means” basically forces a very large arbitrary choice on you.

There is another structure present: the mere fact that your mind is finite means that your list of goals to pursue is far smaller than the set of possible states of the universe. So when you decide that “X” is your goal, and then implement a clause in your goal system that says that you should never change your goal (like all utility maximizers do) you have done something very silly.