In games, as in life, your experience is defined by the a priori goals you set. If I'm playing chess against you, but I've decided that the end goal of the game is not to force you into checkmate, but simply to take all of your pieces, then we are going to have very different experiences. We're going to play differently, we're going to think differently, we're going to prioritize differently, and we're going to disagree on when the game is over and who has won.

The thing about a priori goals is that you can't argue your way into or out of them. If the foundation of your ethics is the maximization of personal freedoms and mine is the minimization of human suffering, then not only will we not agree on right and wrong or optimal strategies, we may each wind up insisting that the other is evil.

...at least, that's how it would be if human beings started with a priori principles and logicked their way into a comprehensive belief system. In reality, of course, we actually start with mammalian instincts and societal imprints, then (some of us) backtrace and derive principles to justify what we're already doing.

Perhaps that's one of the reasons we fear the development of A.I. - we've seen what can happen when someone holds fast to ironclad principles and behaves accordingly, regardless of reality. A consciousness that truly initiates from a handful of abstract principles and develops behaviour patterns from them has the potential to be very dangerous, the moral equivalent of a truck slowly but implacably backing through a crowd of people it is certain are not there.