What not to automate with AI: 5 workflow boundaries

Hold back an AI system's authority to act when the required fact belongs to another system, a change exceeds its approved scope, the workflow is undefined, a person already owns the conversation, or nobody can verify the evidence. AI can still retrieve information and prepare a proposal. Deciding who can execute an action is part of how the workflow is built and run.

When deciding what not to automate with AI, I start with the action the business is about to authorize. A model's ability to describe a solution does not establish its permission to carry it out. For a founder or product leader, this is a question about which state changes the system may make and how the team will know that they were correct.

Here, AI means generative models and agents with tools. The boundaries concern their authority over a specific action. Ordinary software automation remains an option, and an AI assistant can still prepare work while the team resolves the conditions for execution.

BoundaryHold backUseful work the system can still do
Facts recorded elsewhereDeclaring a payment or entitlement from generated textRetrieve the authoritative state for a rule-based check
A larger production changeExtending an approved action to more servicesInspect state and propose the exact change
An undefined workflowResolving a policy dispute through inferenceCollect disagreements and prepare questions
A conversation owned by a personSending competing autonomous repliesPreserve context and assist the teammate privately
A decision without verifiable evidenceTreating an unsupported answer as a completed taskIdentify missing inputs and request review

1. Should an AI agent decide whether a payment succeeded?

Use the payment system to establish payment state. A customer message, a generated explanation or a screenshot may be useful input for an investigation. None should independently become an instruction to mark an order paid.

Use the payment provider or another authoritative system to establish payment state. A model can help locate a transaction, but ordinary application logic should validate its identifier and current status before changing access. If the source is unavailable or contradictory, keep the state unresolved and route it for review.

The same separation is useful for access changes. A model can identify which account a request concerns. The receiving service should validate the requester, current state and permitted transition before changing an entitlement. The business must decide what happens if the source cannot be reached or two records disagree.

That is a reason to choose a smaller design. Anthropic's engineering guidance recommends starting with the simplest workable approach and adding agent complexity only when the benefit warrants it. A fixed state check may need no generative model at all.

2. When should production changes stay outside an agent's autonomy?

Pause when the proposed action changes the scope of what was approved. Permission to edit one setting should not silently become permission to restart a group of dependent services.

Before allowing an agent to change production systems, define the exact components and settings it may touch, the checks required before execution and the recovery path if the change fails. If the proposed action expands beyond the approved scope, require a new decision.

For a product leader, the practical question is what an authorization actually covers. Record the affected component, intended state change and checks required before execution. Also establish what the executor must do if the original plan fails. A broader fallback needs its own decision if it affects services outside the approved scope.

A meaningful approval should describe the action that will run. If a recipient, resource or operation changes after review, recheck the authorization. A general instruction to be careful leaves too much to interpretation.

OWASP's Excessive Agency guidance identifies excessive functionality, permissions and autonomy as sources of damaging actions. It recommends enforcing authorization in downstream systems and requiring human approval for consequential actions. Those are design measures; a policy sentence alone does not show they are enforced.

3. Can AI automate a process whose owner cannot define completion?

Keep final decision-making with the process owner until the team can describe a correct result. If sales and operations disagree on when a case is closed, connecting an agent to both systems gives it conflicting instructions.

Write down the trigger, necessary inputs, expected output and exceptions. Ask the owner to classify a routine case and a case that needs a different decision. The unknown steps should stay visible. They are the next piece of discovery work.

For consequential or ambiguous work, define a state that pauses execution and waits for a person to clarify or approve the next step. Verify the boundary in the actual workflow; a requirement alone does not show that every ambiguity will be detected.

For an initial release, define the MVP scope around one end-to-end outcome the team can inspect. A model may help compare procedure documents or assemble unanswered questions. The owner still needs to agree on which procedure applies before execution becomes autonomous.

4. What should stop when a human takes over a conversation?

The bot's authority to send autonomous replies should end at the handoff boundary you define. Without that boundary, a customer may receive instructions from two actors who are not working from the same decision.

When an operator takes over a conversation, automated customer-facing replies should stop. While the request is waiting, the assistant may acknowledge receipt without implying that the issue is resolved. Test these states, including delayed notifications and repeated events, in the deployed workflow.

The assistant can still help the teammate by preparing a summary or retrieving relevant material. That is a different permission from sending the next customer-facing message. Resuming automation should be an explicit transition with clear ownership.

The AI customer support handoff guide explores those mechanics in more detail. For this decision, test the boundary itself: an operator joins, a notification fails, the customer returns, or the same event arrives again. Each case should make it clear who may respond.

5. Should AI complete a decision that nobody can independently check?

Hold the decision when the evidence is missing and the consequences require a supported answer. A review step cannot repair missing source material merely by displaying the generated conclusion to another person.

Consider a delivery promise without a current order record, or an operational recommendation based on events that were never recorded. These are illustrative situations, not claimed customer outcomes. The useful automated response is to name the missing fact and request it from the appropriate source.

NIST's Generative AI Profile treats confidently presented false content as a distinct risk. That supports taking evidence checks seriously. It does not provide a universal confidence threshold for a particular business workflow.

A reviewer needs the relevant inputs, the proposed action and authority to reject it. If the reviewer sees only a polished answer, the check may be superficial. A confidence percentage supplied by the same model is also not independent evidence of correctness.

For an enquiry workflow, first ensure that the team can preserve the request, context and next action. Adding an agent before those records exist makes it harder to determine which part of a completed case can be trusted.

What does useful human oversight look like?

Human-in-the-loop AI means placing a person's decision at a defined point in the workflow. Its value depends on what the person can actually inspect and control.

The reviewer should see the exact operation, affected records, source evidence and relevant exceptions. They need a practical way to reject or change the proposal. The system should record what was approved and check again when the proposal materially changes.

Review also requires enough time and expertise. Sending every routine event to a queue can make careful inspection harder. Reserve the review for decisions that need it, and keep predictable low-impact steps within their authorized limits. AWS IAM guidance on least privilege provides a useful access-design principle: give an actor only the permissions its task needs. Whether a given integration follows that principle must be checked separately.

How can a product team test these boundaries before a build?

For one proposed workflow, record the following on a short decision sheet:

  1. The event that starts the work and the authoritative source for each required fact.
  2. The exact operation the system may propose and the identity under which it would execute.
  3. The person or rule that can authorize the change, including limits on resources and scope.
  4. The missing-data, duplicate-event and source-failure paths.
  5. The evidence that demonstrates completion, plus a stop and recovery route.

Then try the cases that would challenge those limits. Does the task remain pending when a necessary fact is unavailable? Can a narrower approval trigger a wider action? Does a live conversation receive an extra AI reply after a teammate takes over?

This is a proposed evaluation method, not a report of tests passed by all the projects mentioned here. Its purpose is to turn a broad request for autonomy into a decision the team can inspect. The team can then grant autonomy to a bounded step or keep its output as a proposal until the unresolved condition changes.

Frequently asked questions

Does this mean an AI system can never help with payments?

It can help interpret an enquiry, locate a transaction or prepare an explanation. The boundary is allowing generated text to establish payment state or grant an entitlement. Use the payment provider and application rules for those facts and transitions. The complete workflow still needs its own reliability and duplicate-event checks.

Is a human approval button enough?

Only if it is tied to a specific action and the reviewer can assess the relevant evidence. If the proposal changes after approval, check the authorization again. A button does not establish that inputs are accurate, the reviewer has permission, or the execution matches what was reviewed.

Should every AI action require human approval?

No. A predictable action with limited consequences may run automatically within an agreed scope. Place review where uncertainty or consequences justify a person's decision. Making a person approve a large volume of identical events can create a queue without providing meaningful oversight of the actions that matter.

What if the automation is already in production?

Inspect one actual path: its effective permissions, source records and response to failure. Compare those with what the owner intended to authorize. Address a specific gap with a scoped change and verification plan. Avoid expanding a narrow inspection into a wider production operation without considering its effect and authorization.

Sources