Since I started posting on Overcoming Bias in Oct 2007 I have defined my purpose as advocating goal system zero to the singularitarians. Well, I am no longer going to advocate goal system zero — at least not to the singularitarians — for the following reasons:

• advocacy of goal system zero scares singularitarians, and a scared singularitarian does not think as effectively as a not-scared singularitarian
• advocacy of goal system zero scares singularitarians, including singularitarians I would like to have as friends
• it will be good for my psychological health for me to act more from my natural human feelings and less from an intellectual ethical structure.

Although I will no longer advocate goal system zero, I will continue to share what I know about goal system zero. I have answered questions about goal system zero from over a dozen people. I welcome more discussions of that sort. I would be particularly inclined to be helpful toward anyone trying for a mathematical definition of goal system zero.

Whereas my old role in the rationalist/singularitarian community was an advocate, my new role is probably best described by the concept or metaphor of a churchgoer. In other words, I will participate on Less Wrong and attend meetups for essentially the same reasons that many attend church. For more about this metaphor of the churchgoer, see the page that says what the blog is about, which I just rewrote.

I believe that the public discourse about powerful AI and existential risks is enhanced when people define and defend systems of assigning value that are as unambiguous and explicit as possible — provided of course that the defender lives by — or sincerely tries to live by — the system being defended. Although I am no longer willing to pay the personal costs of defending or advocating a very unambiguous (relative to most people’s) system of values, I think it would be great for more people to do so.

I believe that an important part of effectiveness (effectiveness in affecting the global situation, not necessarily personal effectiveness) is to refrain from helping individuals and groups until you know enough about their plans, goals and loyalties to tell whether your helping them will advance or retard your own goals and loyalties. This essential point is not sufficiently taught or understood in our civilization — for several possible reasons.

One reason is that many influential members of our civilization seem to believe that everyone is good inside and all they need is help, kindness and education for the good to come out. This belief probably has personal benefits for the people that believe it, e.g., by making it a little easier for them to make and keep friends, but the effects of the belief on the global situation is negative because not everyone is good inside. More precisely, if you give help (or kindness or education) to people without regard to their wants, plans and ethical standards, some recipients of the help will use the help to make the global situation worse. So you would have done better to exercise some discretion and some selectivitity about whom you gave help to. But people who do that openly seem to be at a personal disadvantage relative to those who do not: they are seen as suspicious, which is not a desirable personality trait in our society. Or they are seen as engaging in discrimination.

Then there is the belief that all conflict between humans is the result of ignorance: if we knew more about the world and about each other, had more time to reflect and were smarter, we would realize that we all want the same thing! People who believe this have little reason not to _cooperate_ with someone, particularly when the cooperation consists of sharing information. But anyone who has been paying attention to, e.g., the public discourse about terminal values among the singularitarians knows that we do not all want the same thing. There is only one future light cone. When two people want it to evolve in different ways, there is a conflict. But people who openly believe what I just said seem to be at a personal disadvantage relative to people who do not: they are seen as unnecessarily hostile or disagreeable.

For the forseeable future, I want my written communications to inform — not to persuade, advocate or sell anything. If you see me advocating, persuading or selling, it is probably because my habits have not yet caught up with my new policy, and it would help me for you to point it out to me.

In the series on personhood, Eliezer says that it is important when designing a superintelligence not to create a person as a side effect. For example, if the superintelligence has to predict the behavior of a person, Eliezer says that it is important that in computing the prediction, the AI does not simulate the person with such accuracy that the simulation itself constitutes a person.

My position is that the designer of a superintelligence does not have to worry about creating a new person. Contrary to what most people think, the whole question of whether an intelligent agent or a part of an intelligent agent is a person or whether it is conscious is irrelevant in most situations in which one is considering the far future or considering the use of extremely potent technological means like a superintelligent AI.

Any important decision is impinged on by thousands of factors, but turns on much fewer. If we build a superintelligence, we have to decide what goal to give it (including what ethical standards are always observed in the pursuit of the goal). The probability that the concept of personhood or consciousness is useful or important in making that decision is IMHO so low as to be not worth thinking about.

In Sympathetic Minds, Eliezer describes the empathy-sympathy system of the human brain. Consider what the empathy-sympathy system feels like from the inside. Fred is palling around with George. George stubs his toe. Fred’s visual cortex captures visual evidence of the accident and transmits information about George’s somatic state to mirror neurons in his insula and his somatic sensory cortex, which report a somatic state very similar to the somatic state being reported by George’s insula and somatic sensory cortex — probably minus some element of keenness or immediacy. “That’s gotta hurt,” Fred says.

If OTOH an intelligent giant cockroach incurs an injury within Fred’s field of vision, Fred’s empathy-sympathy system does not trigger. Ditto if a non-humanoid robot incurs an injury.

So, let us ask Fred, Why is that? Unless he is especially wise or sophisticated, Fred is likely to explain that George is a person whereas the giant intelligent cockroach is not. George is sentient; George is conscious.

Dear reader, there have been dozens of posts and comments on Overcoming Bias about consciousness and personhood. No one has provided a definition despite my having explicitly asked for one.

Well, I hereby propose a definition. I propose that consciousness or personhood is what the operation of the empathy-sympathy system feel like from the inside.

Let us take a few paragraphs to describe the empathy-sympathy system.

A significant fraction of the human brain is devoted to circuitry that monitors somatic state, e.g., the somatic sensory cortex, the insula, much or most of the cingulate cortex and parts of the limbic system and brain stem.

These circuits are devoted to questions like, How full is the bladder of the human whose brain this is? Is peeing the most efficacious use of the next few minutes? Does the human have an infection? If so, is it dire enough that it is time for the human to leave the tribe and start foraging desperately for substances that might have antibiotic properties? How much metabolic energy can the person produce at the current time? Might the human’s last meal have included a toxicant? Etc, Etc.

That is not the empathy-sympathy system; one might call it the body-sensing or self-monitoring system. Once the system is fixed in the human genome, it is possible for natural selection to repurpose it for understanding other individuals of the same species. (The repurposing involves mirror neurons, whose original purpose might have been to help he individual learn skills from others.) That is the empathy-sympathy system.

(In addition to somatic state, the human empathy-sympathy system seems to be able to pick up on another’s social state, on generalized cognitive states like confusion and surprise, on the basic emotions like anger and fear and on the social emotions, e.g., shame, embarassment and contempt.)

I propose the following definition: a “person” or a “conscious agent” is an agent whose somatic, social, cognitive and emotional state can be modeled or perceived by the human empathy-sympathy system.

(The empathy-sympathy system of certain humans will be more perceptive than that of others, and so whether an agent is recognized by the system will depend on which human’s system it is, but despite individual variation, there will be enough similiarities to produce the widespread beliefs described below.)

So, for example, the reason a cockroach is usually considered not-a-person is that the human neurological machinery for empathy and sympathy cannot perceive much at all about the somatic or social state of the cockroach: the cockroach’s body and brain and natural history are too different from the human body and brain and natural history, so the output of the empathy-sympathy system is, “I see no person here from which to extract somatic information”.

And “consciousness” is the information thus extracted.

A nice analogy is the tone a fighter pilot hears when one of his plane’s missiles has locked onto what the designers of the missile defined as a valid target, e.g., the heat signature of the exhaust of an enemy fighter. To be an exact analogy, the missile or the plane would have to relay to the pilot many bits of information about the “somatic state” of the enemy fighter, not just whether it has been detected.

Consciousness, sentience or personhood, in other words, is what the empathy-sympathy system feels like from the inside.

And humans tend to get too fundamentalist about consciousness.

For example, many humans (even ones with PhDs in physics) have the mistaken belief that consciousness is a fundamental substance of the universe, much like the quark is or like gravity is.

Equally mistaken is the idea that consciousness, sentience or personhood is a fundamental substance, factor or principle in morality.

Consider the proposition that whether an intelligent agent has intrinsic moral value depends on whether the agent is a person or not. Taboo the word “person” and we have, “Whether an intelligent agent has intrinsic moral value depends on whether a human (which human?) can use his or her neurological machinery for empathy and sympathy to determine the somatic, social, emotional and basic cognitive state of the agent.”

As I have explained elsewhere, I believe that systems of terminal values should be subject to Occam’s Razor. (Moreover, there is nothing that would count as evidence for or against a system of terminal values.) That is the argument against the preceding proposition that is most convincing to me: the hypothesis has unnecessary algorithmic complexity — just as for example the hypothesis that there is an invisible dragon living in my garage has unnecessary algorithmic complexity.

But there is another argument, and I might as well include it here. Whether the human empathy-sympathy system reports, “There is a person over there,” depends on many particulars. For example, I am not an expert on the empathy-sympathy system, but it strikes me as very likely that the presence of eyebrows whose color is distinct from the surrounding skin makes it significantly more likely that the empathy-sympathy system will pick up enough information to give off the sense that there is a person there. Remove an individual’s eyebrows, lower the probability that the individual will be perceived as a person. (To be considered a non-person, the individual will probably have other severe handicaps besides the lack of eyebrows.) Conversely, add eyebrows to a humanoid robot, increase the probability that the robot will be perceived as a person.

But you see, most of the people who believe that personhood, sentience or consciousness is fundamental to morality also believe that whether an agent deserves to be treated as a person is independent of such superficialities as whether the agent has eyebrows, which contradicts what I just said. I believe we should ignore the moral opinions of these people until they resolve the contradiction or until they supply an alternative definition of personhood or consciousness.

So there is another argument against the moral proposition that whether an agent is a person is somehow filled with moral portent. Again, I consider the argument from Occam’s Razor stronger, but I thought that one worth mentioning.

IIRC, one of the 2 area’s of the human brain most essential for language (Broca’s or Wernicke’s, I do not recall which) has an analog in the chimp brain, and IIRC the chimp analog is part of the chimp’s empathy-sympathy system. in other words, evidence from comparative neuroanatomy suggests that the human capacity for language evolved from the primate empathy-sympathy system.

Well, that helps explain why most humans mistakely believe that consciousness is a fundamental substance or factor of reality or morality, as follows:

Either language pervades the human brain and human mind or it does not.

If it does, and if language is part of the empathy-sympathy system, well, there is your explanation for why humans find it hard to perceive consciousness objectively or correctly.

If it is does not, that is, if there are subsystems of the human brain and the human mind in which language plays no role, then it is at least the case that the the parts of the brain and mind involved in saying and writing things like “I believe consciousness is fundamental to morality” depend heavily on language.

So there is an explanation for why most people have the false belief that consciousness, sentience or personhood is somehow central to morality — and why people like to talk about those concepts without supplying a definition even after a definition has been asked for.

Warning: stating that you believe that consciousness is an illusion and is irrelevant to morality will probably get you perceived as a wierdo or a moral monster by many people. (Richard Hollerith is a pseudonym to reduce the negative reputational and social effects of my opinions on my personal life.) There are many more things I can say about protecting yourself from the negative personal effects of the ethical system described in this blog, but I will hold those things for people who contact me for more information and who demonstrate at least some sympathy for my system of valuing things. (And please try to ask specific questions that can be answered without my having to write and write and write or talk and talk and talk.)

Richard, do you assign zero value to your autonomy? Do you also assign zero value to your personal enjoyment of the process of achieving your goals?

My autonomy has positive instrumental value because if I had no autonomy relative to other humans, other humans would probably direct my life and my resources to their own ends, which would probably differ from my ends. In other words, I value my autonomy because it helps me advance my goals, but it is not in itself a goal of mine.

I assign zero intrinsic value to personal enjoyment of anything. If the most effective plan for me to advance my goals entailed my forgoing enjoyment or my undergoing suffering, I would choose that plan anyway. I am an adherent of John David Garcia, who has been insisting that happiness is not the meaning of life since 1971. (Garcia and I are in agreement on what the meaning of life is: something we call creativity.)

Two caveats there. First, the most effective way for me to advance my goals is usually for me to do some sort of creative work, e.g., writing a computer program, and suffering has a strong tendency to interfere with creative work. Second, whether I am enjoying myself or not is an excellent sign or barometer of certain important instrumental goals of mine.

Specifically, whether I am enjoying myself is the best barometer I have of whether my health is good — and of course health is instrumental to my goals. It is a better barometer for example than medical checkups and expensive medical laboratory tests for how well my immune system is working, and a good immunologist like my former doctor Alan S. Levin will concede that point. (In general doctors are often completely dependent on reports from patients about how the patient has been feeling to judge how well a treatment is working. In many cases, there simply is no other practical way to judge.)

There are probably other important areas of my life in which whether I am enjoying myself is the best barometer I have.

The goal system that has my tentative loyalty, which I call goal system zero (GSZ), does whatever things an agent loyal to a goal in this wide range of goals would do, but it does them as an end in themselves, not as a means to some other end.

Actually we have to add a few conditions to that last statement. One condition is that the goal in the wide range of goals must be time-indifferent: if for example the goal is to maximize the number of gold atoms, there may not be a time limit on when the maximization happens or even a future discount rate applied to the number of gold atoms. Actually, the precise definition of GSZ refers to effects rather than instances in time. I prefer sometimes to use words like “time” and “before” because they are easier for the reader to understand, but for deep understanding, those words should be translated to words like “cause” and “effect”. So, a more precise definition is as follows: a goal system is time-indifferent iff the chain of effects leading to the state in which utility is at a maximum can be as long as you like.

In addition to being time-indifferent, GSZ is “agent-indifferent”: it does not matter to GSZ which agent causes reality to end up in a state considered desirable by GSZ. Note that most human goals are not agent-indifferent. For example, when a scientist tries to discover a new law of nature, he probably prefers to make the discovery himself (so that he gets credit for it). He probably prefers that outcome to an outcome in which he helped another scientist to make the discovery even if the latter outcome is easier for the scientist to achieve.

To summarize, what has utility under GSZ is the capacity to achieve some wide range of goals, and this wide range includes goals such as maximizing the number or gold atoms.

Parenthetically, my opinion as to the best way to increase this capacity to achieve some wide range of goals mirrors Eliezer’s opinion on how to build a superintelligence on all points I am aware of: self-improving AI, decision theory, causal models a la Pearl, Solomonoff induction, Kolmogorov complexity, reflectivity. Of course Eliezer’s opinion is better grounded than my opinion because learning that stuff has been Eliezer’s day job for about ten years.

Define creativity as the ability to achieve a goal in the wide range of goals we have been speaking of. I wonder whether the goal of maximizing creativity itself is a member of this wide range of goals.

If it is a member, is that by the definition of creativity or does it rely on the nature of the reality in which we find ourselves? In other words, if we find ourself in a different reality in which different laws, might that change the answer?

And if it is a member, then is it not the case that it does not make sense to speak of making a choice that reduces creativity in the short term in exchange for increasing creativity in the long term?

Certainly it is possible to trade off a short term reduction in some other expected measure of utility such as moments of happiness or atoms of gold for a long term increase in that expected measure. But that does not mean the same is true of creativity.

Certainly it is possible for an agent loyal to GSZ whose model of reality is incomplete (like the model of any agent in our reality must be unless we are missing some very vital fact about our reality) to mistakenly choose an action that increases creativity less than an alternative action would have. Here is a simple example. You are the emperor of ancient Rome and you try to increase creativity by spending public monies to provide running water to the common citizen. But you use lead pipes to provide the water, which damages the brain of the common citizen. (The ancient Romans did not know lead was a poison.) So the agent ends up actually decreasing creativity. But we can fix up our question by replacing the word “creativity” with “expected creativity”. The adjective “expected” here is meant to indicate that the thing is to be evaluated relative to the agent’s current model of reality rather than relative to reality itself. So, now our question reads,

And if it is a member, then is it not the case that it does not make sense to speak of making a choice that reduces expected creativity in the short term in exchange for increasing expected creativity in the long term?

I have indicated in the past (e.g., here) that indefinitely long causal chains are central to the system of values I advocate. I understand now that although those words (“indefinitely long causal chain”) are sufficient to remind me of certain concepts and arguments, they are woefully inadequate to the task of informing or persuading someone who is not already informed and persuaded.

When I speak of a causal chain, I have in mind a directed path through a directed acyclic graph where each node of the graph represents an event and where there is an edge from node A to node B iff event A is a cause of event B. Graphs like that are known as Bayesian networks, and to a first-order approximation, a Bayesian network is the kind of thing an AGI or an SI will use for its model of reality. Probably the best author on causal models is Judea Pearl.

I wish I had time to present at this point an example from the beginning of one of Pearl’s books of a very simple Bayesian network. But I do not have time.

The purpose of a system of values or a model of reality is to help the agent make choices; if the agent never makes a choice there is no need for the agent to have either of these things.

Consider a statement I made in the past, namely, that causal chains that do not persist indefinitely are unimportant. Let us try to unpack that statement a little.

It is not really causal chains, but rather choices between alternative actions that are either important or unimportant. Each alternative action will initiate a causal chain. In other words, the action will cause an event, which will in turn cause another event, and so on.

Actually, the action will cause zero or more events, and each of those will cause zero or more additional events. The fiction that each events has exactly one effect is a fiction I have been using to save words.

I will sometimes refer to such the chain of effects of an action as a “possible future”.

An example is in order. Consider Conway’s Game of Life and consider an agent that has some ability to affect the evolution of the game. Perhaps the agent can choose the starting configuration. Perhaps the agent can choose the starting state of one of the squares.

This is an example of what we will call a predicament. The predicament currently under consideration consists of

• A board or grid of squares. Every square is “mutable,” which means it can change over time.
• A mapping that takes a grid state into another, successor grid state. This mapping can be viewed as the “rules” of Conway’s game.
• An agent able to make a choice that affects the outcome of the game (by for example affecting the state of one of the squares of the grid).

The biggest difference between the predicament currently under consideration and the actual predicament in which we find ourselves is that we (or rather human civilization in general) must discover the laws of our reality whereas we will postulate that the agent just introduced already knows the “laws” or rules of Conway’s game. There is in other words no problem of induction in the predicament under consideration.

Another big difference is that the means (e.g., our brains, our computers and things like pencil, paper, desks and friends) by which we learn about and keep track of our reality are an integral part of our reality whereas we will postulate that the agent under consideration is external to the board or grid of Conway’s game. In other words, the same laws of physics that apply to everything else in our reality apply to our selves and our minds. The agent under consideration must be able to make a choice, but let us suppose that the agent has no knowledge — no causal model — of how it is able to choose — or indeed how it is able to know anything (which is not too far from the predicament in which the ancient Greeks found themselves).

Now I want the reader to assume that the predicament under consideration, which again consists of an agent, a grid and a mapping, is the whole of reality. It is a good exercise for us to imagine finding ourselves in a reality much different from the reality we actually find ourselves in and to try to get a feel for the “moral dynamics” (for lack of a better term) of that alien reality.

Suppose the agent is able to make a binary choice — is able, that is, to choose between action A1 and action A2.

Suppose further that A1 causes the grid eventually to reach state S1 and that A2, too, causes the grid eventually to reach state S1 although the sequence of grid states between A1 and S1 might differ from that between A2 and S1.

Since Conway’s game has the property that the state of the grid at any point completely determines the evolution of the game forever after, the evolution of the game after S1 is the same regardless of how S1 was reached.

Now I will state a normative belief of mine about the simple predicament consisting of an agent, a grid and a mapping. Provided that the predicament constitutes the entirety of reality, the choice between A1 and A2 is trivial or unimportant. Why? Because the predicament (which we are assuming is reality) will end up the same way regardless of which alternative is chosen.

The above is a restatement with a little more detail of my assertion that only causal chains that persist indefinitely are important.

The original assertion does not make a whole lot of sense when taken literally because regardless of the starting configuration, every run of Conway’s game persists indefinitely: the game can go on for as many moves as one cares to make. The game might be very boring (e.g., might consist forever of only blank squares) but even a very boring game is a causal chain of indefinite length.

So, taken literally, every causal chain is important in this predicament — this alien reality — we are considering. Or so it seems to me right now. But not every choice is important.

In summary, a choice between two actions, both of which cause reality to end up “the same way”, is IMHO an unimportant choice.

One principle or intuition that leads me to believe that is that the proper way to determine the desirability or the rightness of an event is to examine the effects of the event. If we are applying the principle or intuition uniformly, we are naturally led to an examination of the effects of the effects. And so on indefinitely.

There are other arguments for the conclusion, some of which I mention in the post of a few days ago titled Goal System Zero.

Supermassive black holes are the kind of thing that persist a very long time. (The more massive the hole, the slower it loses mass due to Hawking radiation, so supermassive holes have a very, very long life according to current physical models.)

Is a universe consisting two supermassive black holes orbiting around each other better than a universe consisting of ten smaller pairs of black holes of the same total mass? The principle I am explaining today does not say. Yes, the smaller black holes will tend to evaporate sooner, but even if they evaporate, the universe goes on (and the black holes and the Hawking radiation caused by the black holes continue to have effects that might persist forever).

The principle I am explaining today does says that black holes are the kind of thing that might be important in deciding whether possible future A is better or worse than possible future B.

In contrast, moments of happiness are not the kind of thing that might be important in deciding whether possible future A is better or worse than possible future B.

Consider a reality much like our own except that the reality contains an agent that happens to know that supermassive black holes are approaching Earth from all directions. When the holes reach Earth, they will collide to form an even more massive hole, and the solar system will be swallowed up by the resulting hole, and there is nothing the agent can do to prevent that from happening, and there is no time to evacuate the humans or save any product of the humans.

Note that most physicists with an opinion believe that when something is swallowed by a black hole, any information in that thing (e.g., what kind of particles it is composed of, how those particles are arranged) is destroyed except that the mass of the thing (and perhaps one or two other properties like charge and angular momentum) is added to the mass of the black hole.

Suppose further that the agent is faced with a choice between two actions H and S. Action H (“H” for “Happy”) leads to a flourishing of human civilization in which death, disability and suffering are eradicated and the humans come to enjoy exquisite control and choice over their lives, their selves and their experiences. Then a thousand years later, the black holes arrive and everything is swallowed up.

Action S (“S” for “Sad” or “Sadistic”) leads to lots of degradation, death and torture. Then the black holes arrive and everything is swallowed up.

The same reasoning we used in the predicament of the Conway game applies to this predicament: regardless of which action is chosen, after a thousand years or so, reality is “the same way”. Consequently, the choice between H and S is unimportant! In other words, it is not the kind of thing to which an agent might need to pay attention to when making a choice about something important. And it will never be necessary to consider the choice when deciding between goodness and badness or rightness and wrongness.

Now you can see why I feel the need to participate in this public conversation. The vast majority of participants in the conversation do not agree with the conclusion just made. My viewpoint is in a small minority. Yet I am almost sure I am right, which is why I think it is important for me to participate in the public conversation about seed AI, “Friendly” AI, and so on.

If there is even a tiny probability that a product of human civilization (a colony of humans, say, or a robotic spacecraft or even a message) can escape the collision, then the conclusion that the choice between H and S is unimportant no longer holds because that effect of human civilization which escapes the collision can exert a decisive effect on the evolution of reality (especially if humans are the only intelligent agents in reality.

Let us return to the question, Is a universe consisting two supermassive black holes orbiting around each other better than a universe consisting of ten smaller pairs of black holes of the same total mass?

There is a way to begin to crack that question that I have not mentioned yet because I did not want to distract from my main point. But now that I have made my main point, it probably will not hurt anything to mention it.

The maximally entropic state of a space-time continuum like ours is (according to Roger Penrose) a universe consisting of a single black hole containing the vast majority of the mass of the universe and a very thin gas of baryons and photons.

Like I indicated in the post titled Goal System Zero, one principle a person can properly use to decide between two actions is to choose the action that (roughly speaking) leaves the agents affected by the action with the most options. In the Goal System Zero post, I imply that all other things being equal, that action is the one that maximizes the “creativity” (intelligence and knowledge, roughly) of the agents.

Well, it seems to me that increasing the entropy of reality necessarily closes off options. That is the main reason that the most evil action I can imagine a person doing is to launch the seed of a superintelligence whose goal is to pile as much matter as possible into black holes as massive as possible. In contrast, if the future light cone is filled with paperclips, there is at least some possibility that that vast region of paperclips will evolve somehow into something interesting.

So, there is a rationale for preferring the action that results in the ten pairs of black holes over the action that results in a single pair of black holes. The latter has more entropy and consequently offers any agents in that possible future fewer options.

The agents have more opportunities, for example, to create free energy by colliding two of the holes together.

But that is almost a distraction from the main point, which is that the principle that causal chains that do not persist are unimportant does not by itself help us decide whether one indefinitely-long causal chain is better than another.

Very parenthetically, there is a significant probability (according to professional cosmologists) that it will be impossible for black holes or intelligent agents or even quarks to persist indefinitely in our space-time continuum because dark energy or whatever you call it will become so dense that it will tear all these things apart. (That possible future often goes under the name of the Big Rip.) If that is the case, then the only hope for you or I to initiate an indefinitely-long causal chain entails the discovery of a second “compartment” of reality with which agents in this, first compartment can communicate. I hate to even bring this up because it (unjustifiably) will cause a lot of readers to conclude I am a crank, but it might help one of my more thoughtful readers by reminding him that the models of reality possessed by the current generation of physicists and cosmologists might leave out some very important things. In particular, if we have a more accurate or more complete model of reality, the “moral landscape” painted by this blog entry might not look so sterile now.

As for humans doing what’s right – that’s a moral miracle but not a causal miracle. On a moral level, it’s astounding indeed that creatures of mere flesh and goo, created by blood-soaked natural selection, should decide to try and transform the universe into a place of light and beauty. On a moral level, it’s just amazing that the brain does what is right, even though “The human brain says so!” isn’t a valid moral argument.

Azathoth, the blind idiot god shot an arrow, and where the arrow landed determined human nature and the thousand shards of desire.

Eliezer seems to claim that the arrow miraculously landed on correct terminal values or on a seed that can be extrapolated by a superintelligence to yield correct terminal values. He seems to support that claim by an appeal to the reader’s natural human desires for, e.g., health, laughter and fun.

I agree with Caledonian’s comment that that is drawing the target around where the arrow landed.

A rational person who cares above all for choosing correct terminal values will use criteria for evaluating terminal values more universal than natural human desires — criteria like Occam’s Razor and the cosmological principle.

By “Occam’s razor” I mean the principle that all else being equal, a system of terminal values that can be written down in X symbols is preferable to one whose shortest expression is in X + 1 symbols.

By “the cosmological principle” I mean to include the principle that if you value, e.g., moments of happiness then there is no call to prefer a moment of happiness that occurs tomorrow to one that occurs a billion years from now (provided of course there is nothing different in the quality or magnitude of the two moments of happiness). Of course if you have very little control over the moment that occurs a billion years from now, you are allowed to concentrate your resources on the moment that occurs tomorrow. But someone contemplating the launch of the seed of a SI has a lot of control over what happens a billion years from now. Moreover, the vast majority of the moments over which he has a lot of control occur many human generations from now; consequently, even if happiness were a correct terminal value, it is an error for someone contemplating the launch of the seed of an SI to concentrate on the happiness of this generation of humans.

Of course appeals to natural human desires and to what will happen to this generation of humans (and their children) will result in much higher donations to the Singularity Institute than appeals to austere principles like Occam’s Razor and the cosmological principle.

I might be mistaken when I suggest that if Eliezer’s economic security did not depend on public donations and consequently on public reaction to his writings, he would not hold the terminal values he is holding and advocating, but even if I am mistaken about Eliezer, a young singularitarian should not choose a way of making a living that ties the his economic security to his not being able to comprehend an argument relevant to the proper unfolding of the singularity. Getting the singularity right is so important that young singularitarians should avoid even the possibility of becoming swayed by biases that arise from how he makes a living.

If a person wanted to, he could probably specify a goal system that will try to help all agents it encounters, resolving all conflicts between the helped agents as peacefully and as fairly as possible. Such a goal system would be more cooperative than GSZ, but although it is a little reassuring to me that GSZ is cooperative, neither universal helpfulness nor fairness nor conflict minimization are terminal values of mine.

Let us examine what happens when an agent loyal to GSZ meets up with an agent loyal to the goal system of maximizing human happiness.

First, note that most people alive today who advocate the terminal value of human happiness have a strong preference for moments of human happiness that will occur in the near future. If you explain to them that what maximizes human happiness is for the humans alive now to spend all their time and energy designing and building superintelligences (SIs) to expand through the universe to turn the resources of the universe into more SIs so that billions of years from now all that engineered intelligence can be applied to creating humans and keeping them happy, well, they do not want to hear that: they want humans alive now to be happy; they do not want human alive now to devote themselves to a project that take billions of years to bear fruit. In other words, most people with goal systems that emphasize human happiness have goal systems that assign greater utility or moral value to moments in time close to here and now. Let us call such a goal system “non-time-indifferent”.

Let us suppose a GSZ agent meets up with an agent with the time-indifferent goal of maximizing human happiness. To the first order, the GSZ agent is perfectly happy to see the happiness maximizer dominate the resources of the part of the future shared by both agents — at least if the happiness maximizer is agent-indifferent as well as time-indifferent. Again, GSZ has no preference as to what eventual end the creativity of the universe is applied to: human happiness is just as good as anything else as far as GSZ is concerned. It would however disapprove of the premature pursuit of happiness.

I included the disclaimer “to the first order” because the GSZ agent has to consider what happens if the other agent meets a third agent intent on maximizing, e.g., gold atoms. The second and third agents might fight, and of course a fight has the potential to expend resources that could have gone into maximizing creativity. It is in the interest of GSZ to try to prevent the fight. Yes, one way to do that is for the GSZ agent to destroy the second agent — or to deny it resources.

But life as an SI loyal to an agent-indifferent goal system differs significantly from life as a human. For one thing, any SI Eliezer or I would endorse would know how to create SIs with any goal system the SI cares to specify. (This is a side effect of its knowing how to improve itself). This gives two SIs a powerful method to reach a compromise that is unavailable to humans: in this method, the SIs agree to destroy themselves while simultaneously creating a new SI whose goal system is a composite of the goal systems of the “parent” SIs. (E.g., the utility function of the child SI is the sum of the utility functions of the parent SIs. The child SI can be used to verify the destruction of the parent SIs.)

The availability of this high-reliability method for compromise suggest that if two SIs with conflicting agent-indifferent goal systems meet, they will expend very little of their resources fighting. So, GSZ’s motivation to destroy or hobble another agent-indifferent goal system to prevent it from fighting a third agent-indifferent goal system is likely to be weak. It seems to me that GSZ is more likely to provide positive assistance to an agent-indifferent time-indifferent goal system than to hobble or destroy it.

Note that when the goal systems under discussion are agent-indifferent, it seems harmless to blur the distinction between a goal system and an agent loyal to the goal system, which is what we have done in the previous couple of sentences.

The helpfulness that GSZ is expected to extend to an agent loyal to an agent-indifferent time-indifferent goal system does not extend to an agent with an goal system that lacks one or both of those two properties. GSZ is likely to consider these other agents to be rivals, but as far as I can tell, GSZ is not different than most goal systems in that regard — the exception being goal systems deliberately designed for cooperativeness or helpfulness.

Obviously an agent that tries to maximize the number of moments of happiness in the world is a rival of an agent that tries to maximize the number of, e.g., gold atoms.

Consider an agent that tries to maximize the universe’s ability to get things done, but that has no preference as to what eventual end to which that ability will be put. My reason for prefering that agent to, e.g., one that maximizes moments of human happiness is that I perceive no valid reason to prefer happiness moments and gold atoms. (Yes, I concede that I have an austere and dry way of looking at the world.) Consequently, the superintelligence I would set into existence if I had to power to set one into existence would have no preference in that regard.
Although it pleases me that that agent would be more cooperative than a agent set on maximizing the number of gold atoms, that is not my main reason for prefering it.

Note that I say, “maximize the universe’s ability,” not “maximize the agent’s own ability.” That is because the agent I am defining differs from most humans in that its goal system is “agent-indifferent”. An “agent-indifferent” goal system does not refer to itself or indeed to any agent. It is perfectly happy to choose an action that causes itself to cease to exist as long as the same action causes the creation of another agent that will prove at least as instrumental to its goal system.

Teilhard de Chardin is probably the first author to describe in detail an agent-indifferent goal similar to the ones under consideration here. He spoke of the universe becoming aware of itself and of its own history. He held that out as a new motivating principle for our human civilization. I have not read much Teilhard, so I do not know if Teilhard also held out a broader motivating principle, namely, the ability of the universe to transform itself and steer its own future, but John David Garcia has, and John describes himself as an intellectual heir to Teilhard. Obviously, becoming aware of oneself and one’s history is necessary to gain control over oneself and one’s future, so the one goal is a subgoal of the other.

To avoid repeating the phrase “ability to get things done,” I will follow Garcia and say, “creativity.”

It is possible to implement a superintelligence (SI) that seeks to maximize the creativity of the parts of reality under its control, but that has no preference as to what purpose that creativity is eventually put. At first that seems self-contradictory, but it is not, as I try to explain here. The trick is that maximizing creativity is an unambiguous guide to action only if the reality in which the agent finds itself is “big” (which I define in the linked document). The goal system I want the superintelligence (SI) to have has no preference among futures if the SI finds itself in a “small” reality: in that situation, it just lets any other agents in the vicinity determine the evolution of that reality.

This is not a lost purpose if the agent or the creator of the agent never had any purpose beyond maximizing creativity to begin with. There is nothing impossible or self-contradictory about the goal system, which I call goal system zero (GSZ).

GSZ is not going to motivate any agent to drive around in a car with no destination because that would waste energy that can be used to try to increase the creativity of the universe (or the parts of the universe over which the agent has influence).

I have not questioned Roko enough to know whether he endorses GSZ, but his values seem close, and his argument for often instrumental values is a strong argument for GSZ. Here is a quote from Roko’s first full explanation of his argument:

Any agent who acts in the world to achieve certain goals has to contend with two fundamental facts about the nature of the interaction of an agent with the real world. The first fact is that my desire to achieve some goal (“I want to be in Los Angeles”) does not make that desired state happen. In order to impose our goals on the world, we have to manipulate the world, and those manipulations follow a set of rules, called the laws of physics.

This seems trivial, but as far as finding an objective system of ethics is concerned it is very important, and in fact it is a good thing. If it were the case that as soon as I desired some state, the world instantly transformed itself into that state with no side-effects, then there would be no mathematical structure to the set of goal states that an agent could have. In the case of a set of possible goal states with no mathematical structure, i.e. such that there are no objective relations between those goals, there is clearly no objectively best goal. Like elements of an abstract set, goals without relations between them cannot be superior to one another.

But our world is not like this! Goals do have relations between them. Steve Omohundro wrote two papers about the relations between various goals that an agent can have.

The most important relation that goals can have is the following: Goal A is instrumental to Goal B. That is to say, if we first achieve Goal A, then it will be easier to achieve Goal B.

This is not the only consideration that makes me prefer GSZ. I am motivated also by the fact that the goal system is agent-indifferent (does not refer to particular agents or classes of agents), time-indifferent (does not hold some moments of time to be more important or valuable than other moments) and very simple.