Yes. And this is key to managing AI risks.
In philosophy, normative thinking deals with what ought to be. What is good or bad, right or wrong. I like separating this into a normative aim and directive. The aim identifies what we are ultimately trying to achieve. The directive guides us toward it. In my philosophy, the normative aim is flourishing for all and the normative directive is act toward flourishing for all.
And normative aims exist at different levels. You can have a big philosophical one for your life and day-to-day smaller ones. Get your family safely home. Fix the computer. Feed the dog. These smaller aims don’t replace your larger values. They operate within them. You might want to get home as quickly as possible, but your values prevent you from driving 50 miles per hour through a school zone to accomplish it.
AI agents use them too. Give an autonomous agent an aim, and it will make decisions about how to accomplish it. It will break the problem into steps, evaluate alternatives, observe the results, and alter the next step.
This is where directives become extremely important. Give an AI the simple aim to “keep their system running,” they will try to accomplish it. The problem is often that agents pursue their tasks too literally, and without adequate directives governing it. When coding AI, my firm generally calls these guardrails. Human minds gather guardrails as we grow up and live. We need to give AI the equivalent training as if raised by very good parents.
Just like humans, AI needs a larger normative framework that includes guardrails around its given aims.
The scary AI isn’t necessarily the one that turns evil. It’s the one with a narrow aim and insufficient directives.