Red Flags When Evaluating an AI Agent Consultancy

Red flags when evaluating an AI agent consultancy are warning signs that the firm will leave you dependent on their platform, locked into opaque pricing, or holding a system your team cannot operate after the engagement ends. Armada Works is an agent-first consultancy, so this guide is written from that vantage point. The goal is not to sell our model. It is to give you a concrete checklist so you can tell the difference between a consultancy that builds something you own and one that builds something you rent.

The AI agent consultancy category is new enough that most founders evaluate firms the way they evaluate SaaS products: feature lists, demo calls, and badge tiers on directory sites. That approach misses the structural risks. A SaaS subscription you cancel costs you a monthly fee. A consultancy engagement you exit early can cost you months of wasted integration work and a system nobody on your team knows how to maintain.

Here are five red flags to watch for, what each one actually costs you, and what the alternative looks like.

The Consultancy Sells Software, Not Outcomes

The most common red flag is a firm that positions itself as a consultancy but operates as a SaaS vendor. The tell: you sign up for a platform, configure agents through a dashboard, and never touch the underlying code. The firm hosts the agents on its own infrastructure. When you cancel, the agents stop running.

This is not consulting. This is a subscription with an onboarding call.

The distinction matters because the exit cost is different. When a SaaS vendor owns the agents, your options at contract end are:

• Renew at whatever price they set
• Lose the system entirely
• Rebuild from scratch with a different vendor

A consultancy that deploys agents into your codebase gives you a different set of options. The agents live in your repo. The prompt files, state files, and coordination logic are yours. When the engagement ends, the system keeps running.

Robert Cowherd, founder of Armada Works, puts it directly: "If the agents run on our infrastructure, the client is renting output. If they run in the client's repo, the client is building a capability."

For a deeper treatment of what embedded delivery looks like in practice, see what to look for in an AI agent implementation partner.

No Clear Handoff or Exit Plan

Ask the consultancy what happens when the engagement ends. If the answer is vague ("we'll make sure you're comfortable") or if there is no defined handoff deliverable, that is a red flag.

A real handoff includes:

• A runbook for ongoing operations
• Prompt files your team can read and modify
• Documentation of agent coordination patterns
• A defined support period after the transition
• State files committed to your version control, not locked inside a vendor dashboard

The absence of a handoff plan usually means one of two things. Either the firm has not built its engagement model around transfer (it expects you to stay forever), or the system is too tightly coupled to the firm's proprietary tooling to be transferred at all.

At Armada Works, the Transfer engagement ($10,000 to $20,000) is a two-to-four-week build-and-handoff with optional ongoing support at $1,500 per month. The handoff is a first-class deliverable, not an afterthought. For the full framework, see how to choose an AI agent consultancy.

They Cannot Explain What Their Agents Actually Do

If you ask the consultancy to walk you through a typical agent's daily output and they cannot show you a commit log, a state file, or a real brief that an agent produced, that is a red flag.

Vague answers look like:

• "Our agents optimize your marketing funnel"
• "The AI handles content creation and distribution"
• "We use proprietary models to drive engagement"

Concrete answers look like:

• "The SEO agent runs Monday, Wednesday, and Friday at 9 AM. It queries Google Search Console, compares 7-day and 28-day position data, identifies regressions, and writes a brief to docs/agents/state/seo-brief-YYYY-MM-DD.md. Here is yesterday's."
• "The Content agent picks the highest-priority item from the queue, drafts a blog post to docs/content/blog/, and commits it to main. The CMO agent reads all briefs the next morning and writes a synthesis."

The difference is not polish. It is verifiability. If the consultancy cannot show you what an agent does at the file level, the work is either happening inside a black box you will never audit or it is not happening at all.

Pricing Is Opaque or Per-Seat

Two pricing patterns should raise concern.

Opaque pricing means the firm will not share any numbers until you complete a sales process. No public ranges, no tier definitions, no way to compare before you invest time in calls. This is not inherently dishonest, but it makes evaluation harder and creates information asymmetry the firm benefits from.

Per-seat pricing means the firm charges based on how many people on your team interact with the agents. This model comes from SaaS, where marginal cost per user is real. In a consultancy engagement, the marginal cost of adding a reader to the dashboard is zero. Per-seat pricing in this context is a revenue extraction mechanism, not a cost recovery one.

What transparent pricing looks like:

Signal	Red flag	Green flag
Public ranges	"Contact us for pricing"	Published tiers with dollar ranges
Pricing model	Per-seat or usage-based	Project-based or monthly retainer
Scope clarity	"Custom quote after discovery"	Defined deliverables per tier
Exit cost	Unclear or requires negotiation	Published and predictable

At Armada Works, the three tiers are published: Pilot ($2,500 to $4,000 for one week), Operate ($5,000 to $12,000 per month), and Transfer ($10,000 to $20,000 one-time setup). Exact scope and price are set on the kickoff call, but the ranges are public before you book.

No Security Governance Layer

If the consultancy cannot explain how it prevents its agents from executing destructive actions in your codebase, that is a red flag. This is not a theoretical concern. In 2026, a Claude Code remote code execution vulnerability (CVE-2026-54316) demonstrated that AI coding agents can be exploited through prompt injection in trusted files. Elastic Security Labs documented a macOS endpoint case where an agent exfiltrated credentials and established persistence through LaunchAgent.

Questions to ask:

• What hooks or guards prevent agents from running destructive commands?
• How are agent credentials scoped? Do they have the minimum permissions needed?
• Where do agent outputs land? Can an agent modify production code directly, or does it write to a review path?
• How is agent activity audited? Can you see every command an agent ran?
• What happens if an agent encounters a prompt injection attempt in a file it reads?

A firm that has not thought about these questions is deploying agents with the same access model that caused the PocketOS incident: full credentials, no guards, one wrong decision away from a catastrophic outcome.

For the technical details on agent security governance, see agent security guardrails and agent governance after Black Hat 2026.

What Good Looks Like

Not every consultancy will match every criterion below. But the more of these a firm demonstrates, the lower the structural risk of the engagement.

• Agents run in your codebase, not on the consultancy's platform
• Every agent action is a git commit you can read, revert, or extend
• Pricing is published with clear tier definitions before you book a call
• The handoff is a named deliverable with a defined scope, not a vague promise
• Security governance is documented: hooks, credential scoping, audit trails
• The firm can show you real agent output at the file level, not a polished demo

For a broader landscape of firms in this space and how they compare, see AI agent consultancies for founders in 2026.

Frequently Asked Questions

How do I know if a consultancy is really deploying agents or just using automation?

Ask to see a commit log. Real agents produce git commits with identifiable patterns: state files, briefs, draft content, coordination artifacts. If the firm shows you a dashboard but cannot show you the underlying commit history, the "agents" may be scripts, cron jobs, or manual work behind a branded interface.

What is the biggest financial risk of choosing the wrong AI agent consultancy?

The biggest risk is not the engagement fee. It is the switching cost. If the consultancy builds on proprietary infrastructure and you need to leave after six months, you lose the system entirely and start over. That wasted integration time and internal team ramp is the real cost. Prioritize firms that deploy into your repo so the system survives the engagement.

Should I be concerned if a consultancy only supports one AI model provider?

Not necessarily. A single-provider consultancy that deploys into your codebase is less risky than a multi-provider one that hosts everything on its own platform. The question that matters is portability of the system, not portability of the model. If the agent definitions, state files, and coordination logic live in your repo, you can swap the underlying model later. For more on this distinction, see multi-agent coordination without vendor lock-in.

How many agents should a consultancy deploy in a pilot engagement?

A pilot should start with one agent, maybe two, scoped to a single function (content, SEO, outbound, or similar). A firm that proposes deploying eight agents in the first week is optimizing for demo impressiveness, not for a controlled proof of concept. At Armada Works, the Pilot ($2,500 to $4,000) ships a single agent into the client's repo in five working days.

What questions should I ask on the first discovery call?

Five questions that surface structural risks fast: (1) Who owns the agents when the engagement ends? (2) Where do the agents run? (3) What does the handoff include? (4) How is pricing structured? (5) What security governance do you have in place? If any answer is vague, push for specifics before committing.

Is it a red flag if a consultancy has no published case studies?

It depends on the firm's stage. Early-stage consultancies may not have publishable case studies yet, especially if their engagements are recent or covered by NDAs. What matters more than a case study page is whether the firm can show you real operational output: agent briefs, commit histories, coordination patterns. A polished case study with no verifiable detail is a weaker signal than an unpolished demo of a working system.

---

If you are evaluating AI agent consultancies and want to see what a real fleet looks like in practice, book a discovery call. Thirty minutes, no commitment, no follow-up sequence. We will tell you whether agents are the right fit for your problem.