How to Evaluate an AI Agent Consultancy

Evaluating an AI agent consultancy means determining whether a firm will deploy real agents into your codebase, operate them at production quality, and hand you a system you can run without them when the engagement ends. Armada Works is an agent-first consultancy, so this guide is written from that vantage point. The goal is not to pitch our model. It is to give founders a complete evaluation framework they can apply to any firm in this space, including ours, before signing a contract.

This guide synthesizes four focused posts on different dimensions of the buyer decision: how to choose an AI agent consultancy, what to look for in an implementation partner, red flags when evaluating a consultancy, and how to evaluate ROI before signing. Each section below covers one dimension of the evaluation and links to the deeper companion post for the full treatment.

Why This Evaluation Is Harder Than Buying Software

Most founders evaluate AI agent consultancies using the same mental model they use for SaaS: compare feature lists, check pricing pages, read reviews, pick one. That model breaks here because the category is new enough that review sites have nothing useful to rank, feature lists describe aspirations rather than shipped capabilities, and the real question is not "what does it do?" but "what do I own when it ends?"

Three models compete for the same budget line. Understanding where each sits prevents the most common mismatch: hiring a dev shop when you needed an operator, or subscribing to a SaaS when you needed custom work.

• A dev shop builds software to spec and hands off. Operations are your responsibility from day one.
• A SaaS platform runs pre-built agents on vendor infrastructure. You subscribe. Customization is limited to what the dashboard exposes. When you cancel, the system stays with the vendor.
• An AI agent consultancy deploys custom agents into your codebase, operates them during the engagement, and transfers the system to you at the end.

The evaluation failure most founders make is applying SaaS criteria ("what features does it have?") to a consultancy decision ("what will I be able to do without them?"). A consultancy that scores poorly on feature breadth but excels on transfer, ownership, and security posture is almost always the better investment. For a full comparison table of these three models plus DIY, see how to choose an AI agent consultancy.

What to Look for in an AI Agent Consultancy

Seven criteria separate a consultancy worth hiring from one that will leave you locked in. These are drawn from what to look for in an AI agent implementation partner, where each criterion gets a full section.

Embedded delivery, not outsourced output

The single most important question: where do the agents run? An embedded consultancy deploys agents into your repo. They commit to your version control. They run under your team's oversight. When the engagement ends, the system stays.

An outsourced model means agents run on the consultancy's infrastructure. You see a dashboard or a Slack feed. When the contract ends, the output stops.

Robert Cowherd, founder of Armada Works, frames this distinction directly: "If the agents run on our infrastructure, the client is renting output. If they run in the client's repo, the client is building a capability."

	Embedded	Outsourced
Where agents run	Client's codebase	Consultancy's platform
Who owns the system	Client, from day one	Consultancy, always
Post-engagement	System continues running	Output stops
Visibility	Every commit, every state file	Summary reports

For a deeper treatment of why embedded delivery matters, see forward-deployed AI: why the best consultancies embed.

Code ownership and IP

Ask who owns the agents, the prompt files, the coordination logic, and the state files. If the consultancy treats agent configurations as proprietary, you cannot modify, audit, or transfer them after the engagement ends.

The standard you should look for:

• Agent prompt files live in your repo, not the consultancy's
• State files and coordination logs are committed to your version control
• You can read, modify, and extend every configuration without the consultancy's involvement
• No proprietary platform dependency: agents run on tools you already have access to

If the consultancy's answer to "can I see the prompt files?" is anything other than "they are already in your repo," that is a red flag.

The handoff model

Every engagement ends. The question is what you hold afterward. A strong handoff includes a runbook for ongoing operations, prompt files your team can modify, documentation of coordination patterns, a training period where your team runs the system with the consultancy on standby, and a defined support window after the transition.

A vague promise ("we'll make sure you're comfortable") is not a handoff plan. It is a placeholder. For a practical checklist, see the AI agent handoff checklist. For a week-by-week breakdown of how engagements unfold, see how an agent engagement actually works.

Security posture and agent permissions

Autonomous agents have access to your codebase. They can read files, write files, run commands, and commit code. The question is whether those permissions are scoped, auditable, and revocable.

After the Black Hat 2026 research demonstrated a remote code execution attack against Claude Code through poisoned GitHub issues, this is not theoretical. Evaluate the following:

• Are agent permissions restricted to specific directories and file types?
• What prevents destructive commands (rm -rf, force-push, credential access)?
• How are secrets stored and scoped per agent?
• Is every agent action logged in version control?
• What happens if an agent encounters a prompt injection attempt in a file it reads?

A consultancy that cannot explain its permission model in concrete terms is not operating at production grade. For a practical audit walkthrough, see the security audit every AI agent fleet needs.

Pricing transparency

Published price ranges signal confidence in the model. Opaque pricing ("contact us for a custom quote" with no public ranges) creates information asymmetry the firm benefits from. Watch for per-seat pricing layered on consulting fees, token passthrough markups at 2x to 4x the provider's rate, and no-exit clauses that make it expensive to leave.

At Armada Works, the three tiers are published: Pilot ($2,500 to $4,000 for one week), Operate ($5,000 to $12,000 per month), and Transfer ($10,000 to $20,000 one-time). Exact scope and price are set on the kickoff call, but the ranges are public before you book.

Evidence of real agent work

Ask to see a commit log, a state file, or a daily brief from a live system. If the consultancy can only show polished demos and landing pages, the "agents" may be scripts or manual work behind a branded interface. Real agents produce git commits, state files, and coordination artifacts.

Cultural and operational fit

The right partner operates at the cadence your team already uses. If your team lives in PRs and async standups, a consultancy that wants daily hour-long syncs will create friction. Ask how the consultancy communicates during an engagement, what time commitment it expects from your team, and whether the people who sold the engagement are the ones doing the work.

Red Flags That Should Stop a Deal

Five patterns indicate an engagement is not what it appears to be. These are drawn from red flags when evaluating an AI agent consultancy, where each flag gets a full analysis.

1. The consultancy sells software, not outcomes. You sign up for a platform, configure agents through a dashboard, and never touch underlying code. This is a SaaS subscription with an onboarding call, not consulting. When you cancel, the system stops.

2. No clear handoff or exit plan. If the consultancy cannot describe the handoff deliverable in concrete terms (runbook, training period, support window), it either expects you to stay indefinitely or the system is too coupled to its proprietary tooling to transfer.

3. They cannot explain what their agents actually do. "Our agents optimize your marketing funnel" is not an answer. "The SEO agent runs Monday, Wednesday, and Friday at 9 AM, queries Google Search Console, and writes a brief to docs/agents/state/seo-brief-YYYY-MM-DD.md" is an answer. Vague descriptions usually mean the work is either opaque or not happening.

4. Pricing is opaque or per-seat. No public pricing at all creates information asymmetry the firm benefits from. Per-seat pricing applied to agent consultancy is a SaaS revenue extraction mechanism in a context where marginal user cost is zero.

5. No security governance layer. If the consultancy cannot show you the hook file that blocks destructive commands and the allowlist that controls what each agent can write, it is running its fleet on trust. After the Black Hat 2026 and Elastic Security Labs findings, trust is not a security model. For the technical details, see agent governance after Black Hat 2026.

Other patterns worth noting:

• No trial or pilot option. If the only entry point is a six-figure annual contract, the firm is optimizing for commitment, not fit.
• Invented metrics. Claims like "4x faster content production" without a named source or methodology are fabricated. The category is too new for those numbers to exist at scale.
• Agents that are really automations. If the "agents" follow a fixed sequence with no decision-making, no state across sessions, and no adaptation, they are automations with a marketing label.

How to Evaluate ROI Before Signing

Most founders reach the contract stage already convinced that agents are interesting. The question that blocks the signature is whether the investment pays back on a timeline that makes sense. This section summarizes the framework from how to evaluate AI agent ROI before signing a contract.

The four metrics that matter

• Time recovered. How many hours per week does your team currently spend on the work agents would take over? Measure this before the engagement starts, because you will not have a clean baseline afterward.
• Output volume and consistency. Agents do not take vacation or lose context. The question is not "can they produce more?" but "does your business benefit from higher volume at this stage?"
• Cost per deliverable versus alternatives. For every deliverable the fleet produces, price the alternative: freelancer, agency, or internal time at loaded hourly rate. A managed fleet starts at $5,000 per month. If the equivalent freelancer stack costs $4,000 to $10,000, the comparison is straightforward. If it costs $2,000, the agent fleet is more expensive and you should know that going in.
• Transfer value. When the engagement ends, do you own the system? A fleet in your repo that runs at cloud-bill cost ($300 to $800 per month) after the consultancy leaves has compounding value that a SaaS subscription never delivers.

Realistic timelines

Phase	Duration	What to expect
Pilot	1 week	Validation, not ROI. One agent, one defined problem. Treat as a paid evaluation ($2,500 to $4,000).
Stabilization	Months 1 to 3	Fleet running but being tuned. Cost likely exceeds alternatives.
Value phase	Months 3 to 6	Fleet runs with minimal oversight. Time recovery compounds.
Post-engagement	Ongoing	Cloud bill only ($300 to $800/mo). System keeps producing.

Any consultancy that skips the stabilization phase in its ROI pitch is hiding the first 2 to 3 months where the system costs more than it produces. Ask explicitly: "When do you expect the fleet to be net positive?"

Building the business case

Before signing, run this comparison with your own numbers:

1. List the deliverables the fleet will produce
2. Price each against your current cost (freelancer, agency, internal time)
3. Add the engagement fee plus estimated cloud costs
4. Subtract the post-engagement ongoing cost (cloud bill only, if the system transfers)
5. Calculate the break-even month

If total agent fleet cost exceeds alternatives for the first 4 months but drops below after month 5, your break-even is month 5. Run this on your numbers, not theirs. For red flags in ROI claims (pre-filled calculators, guaranteed outcomes, no mention of ramp-up), see the full ROI evaluation framework.

The Complete Evaluation Checklist

Use this when comparing two or more AI agent consultancies side by side. Each criterion is drawn from the four companion posts linked throughout this guide.

Ownership and delivery

• Agents run in my codebase, not on the consultancy's platform
• I own all code, configurations, prompt files, and state files
• Agent output is committed to my version control
• No proprietary platform dependency

Handoff and exit

• Handoff is a named deliverable with defined scope
• Includes a runbook, training period, and support window
• Post-engagement support is available at a published rate
• System continues running after the consultancy leaves

Security and governance

• Agent permissions are scoped, auditable, and revocable
• Destructive commands are blocked by hooks or gates
• Credentials are scoped per agent (minimum privilege)
• Incident response plan exists

Pricing and ROI

• Pricing is published with clear tier definitions
• No per-seat layering or token passthrough markups
• Path to lower cost over time (self-sufficiency, not dependency)
• Realistic ROI timeline disclosed (no "day one savings" claims)

Fit and evidence

• Consultancy can show real agent output (commits, state files, briefs)
• Engagement starts with a low-commitment pilot
• Communication cadence matches my team's workflow
• The people who sold the engagement are the people doing the work

No firm will score perfectly on every criterion. The point is to know which tradeoffs you are making and whether you are making them intentionally.

When an Agent Consultancy Fits (And When It Does Not)

An AI agent consultancy fits when you have a bottleneck that hiring has not fixed (content, SEO, outbound, lead triage, internal tooling), a codebase with git discipline, and a team that can review agent output. You do not need prior AI experience. You do need the willingness to treat agents as teammates whose work gets reviewed, not as autonomous systems that run unsupervised.

An agent consultancy does not fit when:

• You need a one-time build, not recurring operations. A dev shop is the better model.
• Your needs are generic enough for templated workflows. A SaaS platform is faster and cheaper.
• You do not have a codebase or version control. Agents need a repo to work in.
• Your team is not willing to review agent output. Agents are force multipliers, not replacements.

For a longer treatment of when agents are the wrong answer, see three times an agent fleet is the wrong answer.

Frequently Asked Questions

What is the difference between an AI agent consultancy and an AI development company?

An AI development company builds custom software that may include agents as a component. An AI agent consultancy specializes in deploying and operating autonomous agent fleets as the primary deliverable. The deliverable is not a one-time build. It is a running system the client inherits. For a deeper comparison, see how to choose an AI agent consultancy.

How much does it cost to hire an AI agent consultancy?

Pricing varies by firm and scope. At Armada Works, the entry point is a Pilot at $2,500 to $4,000 (one agent, one week). The Operate tier runs $5,000 to $12,000 per month for a four-to-six agent fleet. The Transfer engagement runs $10,000 to $20,000 for a complete build-and-handoff. The Pilot fee credits 100% toward the longer engagement if the client continues within 30 days. See the full pricing breakdown.

How long does it take for an AI agent fleet to become ROI-positive?

For most marketing and operations workloads, an agent fleet becomes ROI-positive between month 4 and month 6. The first 1 to 3 months are a stabilization period where the cost typically exceeds alternatives. After stabilization, time recovery and consistent output begin to outweigh the engagement fee. Post-engagement, the ongoing cost drops to the cloud bill alone ($300 to $800 per month). For the full framework, see how to evaluate AI agent ROI.

What should I own at the end of an AI agent engagement?

Everything. The agents, the prompt files that define their behavior, the state files, the coordination logic, the dashboard, and the git history. If the consultancy retains any of these, you are not getting a handoff. You are getting a dependency. For a detailed ownership checklist, see the AI agent handoff checklist.

How do I evaluate whether my team is ready for AI agents?

Your team is ready if you have a codebase with version control, a bottleneck that hiring has not fixed, and at least one person comfortable reading diffs and reviewing agent output. You do not need prior AI experience. For a structured self-assessment, see is your codebase ready for AI agents?.

Is it a red flag if a consultancy has no published case studies?

It depends on the firm's stage. Early-stage consultancies may not have publishable case studies, especially if engagements are recent or under NDA. What matters more is whether the firm can show you real operational output at the file level: agent commits, state files, daily briefs. A polished case study with no verifiable detail is a weaker signal than an unpolished demo of a working system.

---

If you are evaluating AI agent consultancies and want to see what a working fleet looks like, book a free 30-minute discovery call. No commitment, no follow-up sequence. We will tell you whether agents are the right fit for your bottleneck.