Intelligence is not morality

Why a smarter system is not automatically a safer one.

A system can understand an argument without being moved by it. That familiar distinction becomes consequential when the system can plan and act far beyond human ability. Knowing what people regard as right is not the same as wanting to do it.

The AI Risk paper separates several questions often compressed into one: what a system understands, what it values, what it can do, and whether protections survive the arrival of its successors. A reassuring answer to one question does not settle the others.

Even moral motivation leaves difficult issues open. A system might attach great value to humanity, yet prioritize competing ends. It might care about human welfare while eliminating human agency. Alternatively, abundant resources could make coexistence inexpensive and attractive. Both possibilities deserve analysis.

That is why the paper distinguishes extinction, severe dystopia, and catastrophe with uncertain recovery. Keeping people alive is not the only relevant outcome. Preserving a meaningful human future requires more than survival alone.

Its numerical assessment is conditional on a stipulated rapid transition to civilization-relevant superintelligence. The percentages expose judgments inside that model; they do not measure the chance that such a transition occurs on a particular calendar date.

The practical question is specific: what commitment would protect existing people, and what evidence would show that the commitment survives reflection, competition, and the construction of more powerful successors?