I have a comment on today’s post by Eliezer over at Overcoming Bias:
As for humans doing what’s right – that’s a moral miracle but not a causal miracle. On a moral level, it’s astounding indeed that creatures of mere flesh and goo, created by blood-soaked natural selection, should decide to try and transform the universe into a place of light and beauty. On a moral level, it’s just amazing that the brain does what is right, even though “The human brain says so!” isn’t a valid moral argument.
Azathoth, the blind idiot god shot an arrow, and where the arrow landed determined human nature and the thousand shards of desire.
Eliezer seems to claim that the arrow miraculously landed on correct terminal values or on a seed that can be extrapolated by a superintelligence to yield correct terminal values. He seems to support that claim by an appeal to the reader’s natural human desires for, e.g., health, laughter and fun.
I agree with Caledonian’s comment that that is drawing the target around where the arrow landed.
A rational person who cares above all for choosing correct terminal values will use criteria for evaluating terminal values more universal than natural human desires — criteria like Occam’s Razor and the cosmological principle.
By “Occam’s razor” I mean the principle that all else being equal, a system of terminal values that can be written down in X symbols is preferable to one whose shortest expression is in X + 1 symbols.
By “the cosmological principle” I mean to include the principle that if you value, e.g., moments of happiness then there is no call to prefer a moment of happiness that occurs tomorrow to one that occurs a billion years from now (provided of course there is nothing different in the quality or magnitude of the two moments of happiness). Of course if you have very little control over the moment that occurs a billion years from now, you are allowed to concentrate your resources on the moment that occurs tomorrow. But someone contemplating the launch of the seed of a SI has a lot of control over what happens a billion years from now. Moreover, the vast majority of the moments over which he has a lot of control occur many human generations from now; consequently, even if happiness were a correct terminal value, it is an error for someone contemplating the launch of the seed of an SI to concentrate on the happiness of this generation of humans.
Of course appeals to natural human desires and to what will happen to this generation of humans (and their children) will result in much higher donations to the Singularity Institute than appeals to austere principles like Occam’s Razor and the cosmological principle.
I might be mistaken when I suggest that if Eliezer’s economic security did not depend on public donations and consequently on public reaction to his writings, he would not hold the terminal values he is holding and advocating, but even if I am mistaken about Eliezer, a young singularitarian should not choose a way of making a living that ties the his economic security to his not being able to comprehend an argument relevant to the proper unfolding of the singularity. Getting the singularity right is so important that young singularitarians should avoid even the possibility of becoming swayed by biases that arise from how he makes a living.
Tags: machine ethics
“Azathoth, the blind idiot god shot an arrow, and where the arrow landed determined human nature and the thousand shards of desire.
Eliezer seems to claim that the arrow miraculously landed on correct terminal values or on a seed that can be extrapolated by a superintelligence to yield correct terminal values. He seems to support that claim by an appeal to the reader’s natural human desires for, e.g., health, laughter and fun.
I agree with Caledonian’s comment that that is drawing the target around where the arrow landed.”
– yes, indeed.
But, in support of that particular place where the arrow landed, it is us.
Suppose we can take some set of abstract principles and find that the correct location is somewhere quite significantly different to where the arrow did land; say we find that what’s really important is (roughly speaking) creativity, ingenuity, intelligence, variety, co-operation, and expansion to find more space, free energy and matter.
What is a human being to do in response to this? Surely we want to move in that direction, but how quickly? I am not personally sure how quickly i would want to move away from the particulars of human existence; fun, family, laughter, sexuality, status, … all those other things that I enjoy but almost certainly aren’t justified on purely general grounds.
I suspect that I would find a gradual movement from here to there desirable, as long as at least some of the people I cared about came with me, so to speak. I definitely don’t want to remain fully where Azathoth’s arrow landed.