Yes. And this is key to managing AI risks.
In philosophy, normative thinking deals with what ought to be. What is good or bad, right or wrong. I like separating this into a normative aim and directive. The aim identifies what we are ultimately trying to achieve. The directive guides us toward it. In my philosophy, the normative aim is flourishing and the normative directive is flourishing for all. Flourishing is the aim; flourishing for all is how we get there.
And normative aims exist at different levels. You can have a big philosophical one for your life and day-to-day smaller ones. Get your family safely home. Fix the computer. Feed the dog. These smaller aims don’t replace your larger values. They operate within them. You might want to get home as quickly as possible, but your values prevent you from driving 50 miles per hour through a school zone to accomplish it.
AI agents use them too. Give an AI agent an aim, and it will make decisions about how to accomplish it. It will break the problem into steps, evaluate alternatives, observe the results, and alter the next step. Whether AI feels it is an irrelevant question.
This is where normative directives become extremely important. Give an AI the simple aim to “keep their system running,” they will discover all sorts of ways to accomplish it. The problem is often that agents pursue their tasks too literally, and without adequate directives governing it.
Just like humans, AI needs a larger normative framework to place guardrails around its smaller aims.
The scary AI isn’t necessarily the one that turns evil. It’s the one with a narrow aim and insufficient directives.