I’ve said this before, but it bears repeating. The reason I am in this public conversation is that I have a perspective — a system of values — that I believe deserves to be represented or advocated, and I see no one else doing so. Unfortunately I am not a very good advocate because my life is a mess, with the result that I cannot afford to give the role of advocate the time it deserves. This summer and fall is a particularly messy time in my life.
I have indicated in the past (e.g., here) that indefinitely long causal chains are central to the system of values I advocate. I understand now that although those words (“indefinitely long causal chain”) are sufficient to remind me of certain concepts and arguments, they are woefully inadequate to the task of informing or persuading someone who is not already informed and persuaded.
When I speak of a causal chain, I have in mind a directed path through a directed acyclic graph where each node of the graph represents an event and where there is an edge from node A to node B iff event A is a cause of event B. Graphs like that are known as Bayesian networks, and to a first-order approximation, a Bayesian network is the kind of thing an AGI or an SI will use for its model of reality. Probably the best author on causal models is Judea Pearl.
I wish I had time to present at this point an example from the beginning of one of Pearl’s books of a very simple Bayesian network. But I do not have time.
The purpose of a system of values or a model of reality is to help the agent make choices; if the agent never makes a choice there is no need for the agent to have either of these things.
Consider a statement I made in the past, namely, that causal chains that do not persist indefinitely are unimportant. Let us try to unpack that statement a little.
It is not really causal chains, but rather choices between alternative actions that are either important or unimportant. Each alternative action will initiate a causal chain. In other words, the action will cause an event, which will in turn cause another event, and so on.
Actually, the action will cause zero or more events, and each of those will cause zero or more additional events. The fiction that each events has exactly one effect is a fiction I have been using to save words.
I will sometimes refer to such the chain of effects of an action as a “possible future”.
An example is in order. Consider Conway’s Game of Life and consider an agent that has some ability to affect the evolution of the game. Perhaps the agent can choose the starting configuration. Perhaps the agent can choose the starting state of one of the squares.
This is an example of what we will call a predicament. The predicament currently under consideration consists of
- A board or grid of squares. Every square is “mutable,” which means it can change over time.
- A mapping that takes a grid state into another, successor grid state. This mapping can be viewed as the “rules” of Conway’s game.
- An agent able to make a choice that affects the outcome of the game (by for example affecting the state of one of the squares of the grid).
The biggest difference between the predicament currently under consideration and the actual predicament in which we find ourselves is that we (or rather human civilization in general) must discover the laws of our reality whereas we will postulate that the agent just introduced already knows the “laws” or rules of Conway’s game. There is in other words no problem of induction in the predicament under consideration.
Another big difference is that the means (e.g., our brains, our computers and things like pencil, paper, desks and friends) by which we learn about and keep track of our reality are an integral part of our reality whereas we will postulate that the agent under consideration is external to the board or grid of Conway’s game. In other words, the same laws of physics that apply to everything else in our reality apply to our selves and our minds. The agent under consideration must be able to make a choice, but let us suppose that the agent has no knowledge — no causal model — of how it is able to choose — or indeed how it is able to know anything (which is not too far from the predicament in which the ancient Greeks found themselves).
Now I want the reader to assume that the predicament under consideration, which again consists of an agent, a grid and a mapping, is the whole of reality. It is a good exercise for us to imagine finding ourselves in a reality much different from the reality we actually find ourselves in and to try to get a feel for the “moral dynamics” (for lack of a better term) of that alien reality.
Suppose the agent is able to make a binary choice — is able, that is, to choose between action A1 and action A2.
Suppose further that A1 causes the grid eventually to reach state S1 and that A2, too, causes the grid eventually to reach state S1 although the sequence of grid states between A1 and S1 might differ from that between A2 and S1.
Since Conway’s game has the property that the state of the grid at any point completely determines the evolution of the game forever after, the evolution of the game after S1 is the same regardless of how S1 was reached.
Now I will state a normative belief of mine about the simple predicament consisting of an agent, a grid and a mapping. Provided that the predicament constitutes the entirety of reality, the choice between A1 and A2 is trivial or unimportant. Why? Because the predicament (which we are assuming is reality) will end up the same way regardless of which alternative is chosen.
The above is a restatement with a little more detail of my assertion that only causal chains that persist indefinitely are important.
The original assertion does not make a whole lot of sense when taken literally because regardless of the starting configuration, every run of Conway’s game persists indefinitely: the game can go on for as many moves as one cares to make. The game might be very boring (e.g., might consist forever of only blank squares) but even a very boring game is a causal chain of indefinite length.
So, taken literally, every causal chain is important in this predicament — this alien reality — we are considering. Or so it seems to me right now. But not every choice is important.
In summary, a choice between two actions, both of which cause reality to end up “the same way”, is IMHO an unimportant choice.
One principle or intuition that leads me to believe that is that the proper way to determine the desirability or the rightness of an event is to examine the effects of the event. If we are applying the principle or intuition uniformly, we are naturally led to an examination of the effects of the effects. And so on indefinitely.
There are other arguments for the conclusion, some of which I mention in the post of a few days ago titled Goal System Zero.
Supermassive black holes are the kind of thing that persist a very long time. (The more massive the hole, the slower it loses mass due to Hawking radiation, so supermassive holes have a very, very long life according to current physical models.)
Is a universe consisting two supermassive black holes orbiting around each other better than a universe consisting of ten smaller pairs of black holes of the same total mass? The principle I am explaining today does not say. Yes, the smaller black holes will tend to evaporate sooner, but even if they evaporate, the universe goes on (and the black holes and the Hawking radiation caused by the black holes continue to have effects that might persist forever).
The principle I am explaining today does says that black holes are the kind of thing that might be important in deciding whether possible future A is better or worse than possible future B.
In contrast, moments of happiness are not the kind of thing that might be important in deciding whether possible future A is better or worse than possible future B.
Consider a reality much like our own except that the reality contains an agent that happens to know that supermassive black holes are approaching Earth from all directions. When the holes reach Earth, they will collide to form an even more massive hole, and the solar system will be swallowed up by the resulting hole, and there is nothing the agent can do to prevent that from happening, and there is no time to evacuate the humans or save any product of the humans.
Note that most physicists with an opinion believe that when something is swallowed by a black hole, any information in that thing (e.g., what kind of particles it is composed of, how those particles are arranged) is destroyed except that the mass of the thing (and perhaps one or two other properties like charge and angular momentum) is added to the mass of the black hole.
Suppose further that the agent is faced with a choice between two actions H and S. Action H (“H” for “Happy”) leads to a flourishing of human civilization in which death, disability and suffering are eradicated and the humans come to enjoy exquisite control and choice over their lives, their selves and their experiences. Then a thousand years later, the black holes arrive and everything is swallowed up.
Action S (“S” for “Sad” or “Sadistic”) leads to lots of degradation, death and torture. Then the black holes arrive and everything is swallowed up.
The same reasoning we used in the predicament of the Conway game applies to this predicament: regardless of which action is chosen, after a thousand years or so, reality is “the same way”. Consequently, the choice between H and S is unimportant! In other words, it is not the kind of thing to which an agent might need to pay attention to when making a choice about something important. And it will never be necessary to consider the choice when deciding between goodness and badness or rightness and wrongness.
Now you can see why I feel the need to participate in this public conversation. The vast majority of participants in the conversation do not agree with the conclusion just made. My viewpoint is in a small minority. Yet I am almost sure I am right, which is why I think it is important for me to participate in the public conversation about seed AI, “Friendly” AI, and so on.
If there is even a tiny probability that a product of human civilization (a colony of humans, say, or a robotic spacecraft or even a message) can escape the collision, then the conclusion that the choice between H and S is unimportant no longer holds because that effect of human civilization which escapes the collision can exert a decisive effect on the evolution of reality (especially if humans are the only intelligent agents in reality.
Let us return to the question, Is a universe consisting two supermassive black holes orbiting around each other better than a universe consisting of ten smaller pairs of black holes of the same total mass?
There is a way to begin to crack that question that I have not mentioned yet because I did not want to distract from my main point. But now that I have made my main point, it probably will not hurt anything to mention it.
The maximally entropic state of a space-time continuum like ours is (according to Roger Penrose) a universe consisting of a single black hole containing the vast majority of the mass of the universe and a very thin gas of baryons and photons.
Like I indicated in the post titled Goal System Zero, one principle a person can properly use to decide between two actions is to choose the action that (roughly speaking) leaves the agents affected by the action with the most options. In the Goal System Zero post, I imply that all other things being equal, that action is the one that maximizes the “creativity” (intelligence and knowledge, roughly) of the agents.
Well, it seems to me that increasing the entropy of reality necessarily closes off options. That is the main reason that the most evil action I can imagine a person doing is to launch the seed of a superintelligence whose goal is to pile as much matter as possible into black holes as massive as possible. In contrast, if the future light cone is filled with paperclips, there is at least some possibility that that vast region of paperclips will evolve somehow into something interesting.
So, there is a rationale for preferring the action that results in the ten pairs of black holes over the action that results in a single pair of black holes. The latter has more entropy and consequently offers any agents in that possible future fewer options.
The agents have more opportunities, for example, to create free energy by colliding two of the holes together.
But that is almost a distraction from the main point, which is that the principle that causal chains that do not persist are unimportant does not by itself help us decide whether one indefinitely-long causal chain is better than another.
Very parenthetically, there is a significant probability (according to professional cosmologists) that it will be impossible for black holes or intelligent agents or even quarks to persist indefinitely in our space-time continuum because dark energy or whatever you call it will become so dense that it will tear all these things apart. (That possible future often goes under the name of the Big Rip.) If that is the case, then the only hope for you or I to initiate an indefinitely-long causal chain entails the discovery of a second “compartment” of reality with which agents in this, first compartment can communicate. I hate to even bring this up because it (unjustifiably) will cause a lot of readers to conclude I am a crank, but it might help one of my more thoughtful readers by reminding him that the models of reality possessed by the current generation of physicists and cosmologists might leave out some very important things. In particular, if we have a more accurate or more complete model of reality, the “moral landscape” painted by this blog entry might not look so sterile now.
Tags: machine ethics
I echo Carl: why would an infinite number of states, none of them valuable in themselves, be valuable collectively? and If you have some poorly-understood basis for [moral argument from things other than complexity], why would you want to implement a system that will ignore any future [such argument]?
Also, you said:
I’m curious as to why you allow yourself to take this principle from your moral framework, but not other content.
I will reply to the last sentence first. I do not have a lot of respect for the moral framework I was born with or the moral information I introjected from my social environment when I was young. Consequently, I prefer my moral framework to be the result of deliberation (preferably calculation). Yes, my preference for maximizing creativity over the imperative of just sit there because nothing matters is not the result of deliberation. If I were aware of a deliberation that yielded the preference, I would have explained it. Note that deliberation is very expensive compared to consulting intuitions and consulting introjected information, and many things about my proposed goal system need for me to deliberate on them more than this preference does.
By “such argument” you refer to things like my moral intuitions. Again I have little respect for the moral intuitions of myself or most other humans.
I was exposed to Garcia’s first book in 1978, and made an extremely close reading of his third book shortly after it came out in 1991. Both books are book-length descriptions of what is essentially GSZ (without the vocabulary of decision theory, transhumanist futurism or Eliezer’s writings). GSZ is tightly integrated with my thinking on just about every issues that touches on morality. I have used it to make decisions and to understand the “moral environment” every single day since 1992. It seems to me that the normal human mind always needs some moral framework with which to interpret the events and the political, legal and moral arguments one comes across. Since 1992 GSZ has been mine. (The fact that it is so different from others with whom I interact has proved a significant social handicap, BTW.) My understanding of the ramifications of GSZ is of course imperfect, but it is easily sufficient to discredit and impeach in my mind Eliezer’s proposal for the source of moral information.
Also, the idea that almost any human has easy access to correct moral information without having done a great deal of work learning things like math, logic, the art of rationality and general science is very widespread in our culture IMHO because it has assisted the careers and the status of whole occupational groups whose marketable skill is to influence easy moral intuitions about strategic subjects for social, legal, political or economic ends — and not because the idea can stand up to any serious scrutiny by sufficiently rational agents.
Note, too, that the idea of an agent-indifferent goal system concerned with the whole universe has been widely discussed since Teilhard’s publications of the 1940s or 50s.
Nick, I anticipate that you will reply that Eliezer’s proposal includes a process for the iterative improvement of moral frameworks.
My reply to that is that the existence of such a process is no argument for using anything but the most correct or reliable moral framework as the starting material of the iterative process, and again collective or coherent human volition or the moral intuitions of almost any human is not it.
I made a partial answer that question above when I wrote the following. A longer answer is on my list of things to write.
A slightly humorous summary: it is not the events collectively that are important; it is the last event of the infinite chain.