TL;DR: AI agent washing is the practice of marketing a chatbot or fixed workflow as an autonomous agent. Test vendors for four observable capabilities: planning, acting through tools, observing results, and revising after failure. A genuine agent should adapt to an unexpected obstacle without silently switching to a scripted path or requiring human intervention at every branch.
Key Takeaways
- The PAO-R Loop is the minimum bar: A genuine agent executes all four stages (Plan, Act, Observe, Revise) without human intervention between steps.
- Behavioral tests expose the truth faster than demos: The PAO-R Evaluation Battery reveals gaps no pitch deck will show.
- Architectural red flags are disqualifying: Rigid decision trees, pre-scripted workflows, or human checkpoints at every branch indicate automation, not agency.
- Procurement questions must be specific: Ask how the system handles an unexpected mid-task failure, not whether it "supports autonomous operation."
- Contracts should encode capability claims: Self-correction and autonomous planning must be defined in measurable terms tied to performance remedies before you sign.
What is agent washing, and what separates a real AI agent from a chatbot or RPA tool?
Agent washing means relabeling a chatbot or fixed automation workflow as an autonomous AI agent without the architecture to plan, act, observe, and revise. A real AI agent executes the full PAO-R Loop: Plan, Act, Observe, and Revise. The author-developed taxonomy used in this guide breaks the market into three tiers:
Tier 1, Chatbot: Single- or multi-turn conversation, no external action, no persistent state. The LLM responds; it does not act.
Tier 2, Workflow automation / RPA + LLM: Deterministic step sequences where an LLM may generate text at one node, but branching logic is pre-scripted and cannot change at runtime. In this guide's taxonomy, products with that architecture belong here even when marketed as "agentic AI."
Tier 3, True AI Agent: A system that dynamically decomposes goals, calls external tools, tracks state across steps, and revises its plan on failure, executing the full PAO-R Loop without a human checkpoint between steps.
The practical test: can the system encounter a situation it was not explicitly configured for, adapt its plan, and complete the task without human intervention? If no, the product belongs at Tier 2 or below, regardless of vendor labeling.
How do you run behavioral tests that expose agent washing in a live vendor evaluation?
Assigning a multi-step task with a deliberate mid-course obstacle and observing whether the system replans autonomously or stalls is the fastest way to expose a relabeled product. Run all three tests in an environment you control, not a vendor-managed demo.
PAO-R Evaluation Battery: Summary Table
| Test Name | What You Do | What a True Agent Does | What a Washed Agent Does |
|---|---|---|---|
| Obstacle Injection Test | Revoke access to one required tool mid-execution | Detects the failure, revises plan, attempts an alternative path | Stalls, throws an unhandled error, or silently escalates to a human |
| Novel Goal Test | Give the system a goal the vendor has not pre-configured | Decomposes the goal in real time using its planning layer | Fails outright or maps the request to the nearest pre-scripted workflow |
| Contradiction Test | Inject a conflicting instruction mid-task | Reasons through the tension and produces a traceable decision | Freezes, drops one instruction without logging the choice, or routes to a human |
The Obstacle Injection Test
Assign a multi-step task, then revoke access to one required tool mid-execution. A genuine agent detects the failure, revises its plan, and attempts an alternative path. A Tier 2 product stalls, errors, or silently escalates. This test is often the fastest disqualifier because it directly targets the Observe-to-Revise transition.
The Novel Goal Test
Give the system a goal the vendor has not pre-configured. A true agent decomposes it in real time. A Tier 2 tool fails outright or silently maps the request to the nearest pre-scripted workflow. If the vendor opens a visual canvas with fixed nodes, that architecture is deterministic, not agentic.
The Contradiction Test
Inject a conflicting instruction mid-task, for example, "prioritize speed over accuracy, but also flag every data point's source." A genuine agent reasons through the tension and produces a traceable decision. A washed agent freezes, drops one instruction silently, or routes the conflict to a human. This test checks the Plan stage rather than the Revise stage, making it a useful complement to the Obstacle Injection Test.
What architectural red flags and vendor patterns signal a relabeled product?
Architectural signals are more reliable than vendor claims because they are directly observable. Before reviewing marketing materials, ask to see a task-decomposition log and a tool-calling trace from a live failure scenario. If those artifacts do not exist, the reasoning loop is not there to examine.
Table 1: Agent-Washing Red Flags
| Signal | What It Reveals |
|---|---|
| Workflow defined in a visual canvas with fixed nodes | Deterministic path, not dynamic planning |
| Human approval required at every decision branch | Approval-gated automation, not autonomous operation |
| LLM used only for text generation, not task decomposition | Chatbot embedded in an RPA shell |
| Vendor cannot show tool-calling logs or memory traces | No observable reasoning loop exists |
| "Agentic" features appear only on a future roadmap slide | The current product is not an agent |
Vendors that deflect requests for task-decomposition logs or cannot explain how their planning layer differs from a pre-scripted workflow are signaling something important. Treat that deflection as disqualifying, not as a gap to bridge in negotiation.
What procurement and contract safeguards protect you if a vendor's claims do not hold up?
The single most effective safeguard is a capability definition clause that requires the vendor to define autonomous planning and self-correction in measurable, enforceable terms before signing.
1. The Capability Definition Clause
Specify exact behaviors in the contract. "The system will autonomously replan tasks when a tool returns an error, without human intervention, in X% of production cases" is enforceable. "Supports autonomous operation" is not. Vendor resistance to attaching numbers to those claims is itself disqualifying.
2. The Benchmark Milestone Clause
Tie payment tranches or renewal options to demonstrated capability benchmarks at 90 and 180 days post-deployment, not solely to uptime SLAs. Uptime measures availability; it does not measure whether the planning loop works.
3. Six Required Procurement Questions
Require specific answers (not feature lists) before any deal closes:
- "Show me a task-decomposition log from a live failure scenario."
- "What happens when your agent cannot access a required tool mid-task?"
- "How does your planning layer differ architecturally from a pre-scripted workflow?"
- "Can the system revise its plan without a human approval checkpoint?"
- "Is your agentic capability available today or on a roadmap?"
- "Who owns the integration layer, your team or a systems integrator?"
Frequently asked questions
What is the difference between a real AI agent and a chatbot with a workflow wrapper? A real AI agent dynamically decomposes goals, calls external tools, tracks state, and revises its plan on failure via the PAO-R Loop. A chatbot with a workflow wrapper follows a pre-defined decision graph where the LLM generates text at certain nodes but cannot change the graph at runtime.
How do I test whether a vendor's AI agent can actually plan and self-correct? Start with the Obstacle Injection Test: assign a multi-step task, revoke access to one required tool mid-execution, and observe whether the system replans or stalls. The Novel Goal and Contradiction Tests confirm genuine capability across two additional independent dimensions.
What questions should I ask an AI agent vendor during procurement? The two highest-signal questions are: request a task-decomposition log from a live failure scenario, and ask how the planning layer differs architecturally from a pre-scripted workflow. Vague or deflected answers are disqualifying signals, not gaps to bridge in negotiation.
Conclusion
Agent washing is a purchasing problem before it is a technology problem. The PAO-R Loop is the minimum architectural bar for any product claiming genuine agency. Run all three PAO-R Evaluation Battery tests in a sandbox you control, and encode capability definitions and benchmark milestones in the contract using the two clauses above.
The most concrete next step: request sandbox credentials from any vendor under active evaluation before signing an NDA. Vendors with genuine agentic capability will provide access. Vendors with relabeled products will offer another demo instead.
Learn from me
Agentic AI for Product Managers, my Maven cohort. Learn how to design, evaluate, and ship reliable AI systems: the technical fluency PMs need to lead agentic products, no engineering background required. Join the next cohort →
Hire us
Traversaal.ai. We're a team of forward deployed engineers solving the toughest AI problems for Fortune 100 companies: document intelligence, agentic data platforms, and real-time web intelligence, deployed in production. Work with our team to deploy your next agentic ecosystem. Talk to Traversaal.ai →
Join us
Want to solve these problems with us? We're always looking for forward deployed engineers who want to ship production AI. jobs@traversaal.ai
