20 Questions to Ask an AI Development Agency Before You Sign

Before you sign with an AI development agency, ask 20 questions across five themes: track record, technical approach, data & security, pricing & scope, and post-launch support. A credible agency answers all 20 specifically in a single sales call, and its numbers land near market: roughly $15,000-50,000 for a single AI feature, $50,000-150,000 for a focused AI MVP shipped in 30-45 days, and a straight answer on what month two costs.

The sales call is the cheapest place you will ever discover a problem with a vendor – every question you skip there gets answered later as a change order, a missed deadline, or a product that never leaves staging. This checklist gives you all 20 questions, why each one matters, and what a green-flag answer sounds like, so you can run the call without being the most technical person in the room. It pairs with our full buyer's guide to choosing an AI development company, which covers vendor models, red flags, and IP in more depth.

Why does a question checklist beat a gut call?

Because the market is crowded with polished sellers and the money at stake keeps growing. According to McKinsey, 2025, 92% of companies plan to increase their AI investments over the next three years, yet only 1% of leaders describe their organizations as mature in how they deploy AI. That gap is exactly where bad engagements live: budgets are committed faster than buyers learn to evaluate the people spending them. A convincing AI demo is now cheaper to produce than ever, so polish no longer signals capability – specific answers to specific questions do.

The 20 questions at a glance

Each theme below has four questions. If you only have time for one per theme, ask the killer question and listen for the green flag.

What should you ask about track record?

Everything else in the call is theoretical if the agency has never carried an AI product all the way to production. These four questions separate shipped from staged.

1. "What have you shipped to production, and who uses it today?" Demos are cheap to fake; production is not. A good answer names live software, the real users behind it, and how it behaves when inputs get messy. A portfolio of prototypes and concept videos is a missing track record, not a private one.
2. "Can we speak to a client whose product is live right now?" References from dead projects tell you about the sales process, not the product. A confident agency offers a current client without hesitation; a scramble for names is itself an answer.
3. "Who exactly will build our product, and what have they shipped?" Agency portfolios often showcase work by people who have since left. You want the actual team named, with their own production history – not a bait-and-switch from the senior pitch team to junior delivery.
4. "Tell us about a project that went wrong. What changed?" Every real builder has one. A good answer is specific about the failure and the process fix that followed. An agency with no failure story is either new or not being straight with you.

Before you spend these questions on vendors, it is worth knowing whether your own side is ready to buy. Our free AI readiness scorecard scores your data, team, and use case in about two minutes – and an honest score changes which questions matter most in the call.

What should you ask about technical approach?

You do not need to be technical to expose technical depth. These questions have answers that are hard to fake, and the tell is always the same: specifics versus buzzwords.

5. "What is your model strategy for version one, and why?" The green flag is a foundation-model API plus retrieval by default, with custom training treated as something the project must earn. Unjustified fine-tuning or from-scratch training is the most common way a $50k-150k MVP quote quietly doubles.
6. "How do you measure whether the AI is good enough?" A real answer mentions evaluation sets, an agreed quality bar, and regression testing before releases. "We test it thoroughly" is not an answer – it is the absence of one.
7. "How do you stop wrong, unsafe, or off-brand output?" Listen for guardrails, constrained outputs, and a description of the failure path, not just the happy path. For anything customer-facing, no guardrails story is disqualifying.
8. "What happens when a model provider changes or retires a model?" Providers update models constantly, and output that passed yesterday can fail tomorrow. A good answer covers version pinning, re-running evals on upgrades, and an abstraction that makes switching providers a task rather than a rebuild.

What should you ask about data & security?

Your data will flow through the agency's hands, its cloud accounts, and its model providers. These questions establish where it goes and what it can never be used for.

9. "Where does our data go, and who can see it?" A good answer names the model providers and subprocessors involved, where data is stored and processed, and who on the team has access. Vagueness here means they have not thought about it – which is worse than a bad answer.
10. "Will our data ever train your models or anyone else's?" The only acceptable answer is no, in writing, including the API and data-processing terms of the underlying providers. This one sentence in a contract prevents your proprietary data from becoming someone else's capability.
11. "How do you handle regulated data?" If HIPAA, financial rules, or EU privacy law touches your product, the agency should describe architecture, audit trails, and documentation as budgeted line items – compliance work is real engineering, and a quote that ignores it is a quote for a different product.
12. "How are credentials and access managed during the build?" You want least-privilege access, secrets kept out of code, accounts in your name, and a defined revocation step when the engagement ends. Agencies that run everything through their own accounts are building you a landlord, not a product.

What should you ask about pricing & scope?

The headline number on a quote tells you almost nothing. These questions convert it into something you can actually compare across vendors.

13. "What is the total price to a working product we own?" Judge cost to the outcome, not the hourly rate. As 2026 ballparks: $15k-50k for a single AI feature, $50k-150k for a focused AI MVP, $150k and up for a production-grade product. A low rate attached to an open-ended scope is usually the most expensive option in the room.
14. "What exactly is out of scope?" A credible quote names what it is not building. All-in numbers with no exclusions list are how budgets balloon, because everything contested later becomes a change order.
15. "What does 'done' mean, in writing?" The answer you want is "live in production, evaluated against an agreed quality bar, documented, and handed over with credentials" – not "delivered." A 30-45 day MVP timeline only means something if it ends at that definition.
16. "Who owns the code, the data, the model artifacts, and the accounts?" The right answer is "you do," in the contract, before you sign. Renegotiating ownership after the product exists is slow, expensive, and entirely avoidable.

What should you ask about post-launch support?

AI products are not done at launch: models drift, providers change, and real usage surfaces edge cases no test suite predicted. These questions establish whether you are buying a product or an artifact.

17. "What does month two cost?" Inference, evals, observability, and maintenance are recurring. A reasonable planning figure is 20-40% of the original build cost per year for a moderately used product, and an agency should volunteer a number in that neighborhood rather than treating the question as a surprise.
18. "Who fixes it when it breaks, and how fast?" You want a named response path – a retainer with response times, or a documented handoff that lets your own team respond. A shrug here means outages will be negotiated instead of fixed.
19. "How do you handle model drift after launch?" Outputs degrade as providers update models and your data shifts, without anyone touching your code. The green flag is scheduled re-evaluation against the original quality bar, not a promise to "keep an eye on it."
20. "What does handoff look like if we part ways?" Ask for the exit story up front: repository transfer, credential rotation, documentation, and enough knowledge transfer that another team could operate the product. An agency confident in its work makes leaving easy – lock-in is a substitute for quality.

What do green-flag answers have in common?

Read back through the good answers and a pattern emerges: they are specific, they are written down, and they treat the product as something that must keep working after the invoice clears. Specificity is the whole test – a strong agency names users, numbers, dates, and exclusions, while a weak one offers enthusiasm and adjectives. You do not need all 20 answers to be perfect; you need zero themes where the answers went vague, because each theme protects you from a different failure mode. Track record protects you from demo-ware, technical approach from unjustified spend, data & security from quiet liabilities, pricing & scope from change-order creep, and post-launch from owning an orphan.

These are also the questions we believe every builder should be able to answer on the spot – it is how Game Changer Labs runs its own engagements, from a written definition of done to a month-two number in the first conversation. You can see how we scope and ship production AI end-to-end on our services page, and if you are still comparing vendor models – agency versus studio versus in-house – the full buyer's guide walks through that decision before you ever get to the call.