Contextual reasoning is the proposed dividing line between intelligence and software that pursues a single instruction regardless of consequences. A system told to repair the environment would need to preserve an environment its human users could inhabit. Similarly, closing roads or destroying access routes might speed one freight delivery while violating the purpose and constraints that a person would infer. Catastrophe scenarios based on this literal pursuit of one goal are characterized as artificial stupidity. A system that could consider competing needs and its own safety would present a different danger because it might also work out how to run the world.
Notable Point · 1:00:44-1:05:19
Artificial intelligence, fixed goals, and missing context
A proposed distinction between intelligence and rigid automation turns on whether a system understands the human purpose behind its instructions.