TL;DR: AI vendors often show reference customers running different model versions or providers. They also show different routing setups. Your team will inherit something different in production. This gap is the single biggest under-rated risk in enterprise AI buying. Standard reference calls fail to detect it. The fix is a due diligence process. It forces model clarity in the RFP. It also shapes reference calls around failure, not success.

Key Takeaways: - Reference customers often run different models than what the vendor shows you. This gap only shows up if your questions are built to find it. - "What model do you use?" is the wrong question. Vendors answer with vague labels. These labels hide the real architecture and routing logic. - A second-pass QA layer runs on a different model from the response model. This is the single strongest sign of serious enterprise AI design.

The CTO on the reference call told you their AI deployment was seamless. What they did not mention matters. Their production runs a different model than the one your team will inherit.

This is the reference call paradox. Vendors arrange reference calls with customers whose setup matches the story they want to sell.

Your team speaks to a CTO who reports smooth onboarding, high accuracy, and a happy governance team. You sign.

Then month three arrives. The model your vendor ships behaves differently from the one the reference described. The reference was upgraded last quarter, and you inherited the older tier.

Or per-workspace routing was reconfigured the week after the call.

For CTOs evaluating enterprise AI solutions, this gap is the largest hidden risk in buying. It is also the risk vendors are least likely to disclose on their own.

No sales engineer will say this on their own. Their top reference runs Claude. Your deployment uses a different provider. Not unless you ask the right question the right way.

Most teams respond by asking the obvious question, and get the obvious non-answer.

The obvious question is: "Which model powers your platform?" Vendors answer with vague labels. "Our latest model." "The enterprise tier." "Our proprietary foundation model."

These answers sound specific. They reveal nothing.

A vendor can run three different model setups across three reference customers. They can still answer "yes" to every model question on your checklist. Customer A runs Claude for general queries and GPT for summarization. Customer B runs Gemini for everything. Customer C uses a fine-tuned open-source model the vendor never names in sales. All three are accurate references for the same AI platform.

The deeper problem is changing model selection. Per-intent routing sends different requests to different models based on cost, latency, or quality thresholds. Per-workspace overrides let individual teams pick their own provider.

A vendor that supports this flexibility looks more capable than one that does not. But no single answer to "what model do you use?" can be honest.

Standard AI vendor review checklists treat model as a fixed attribute. In practice, model is a moving target that requires probing specific design details.

A checklist with 40 yes/no questions cannot catch a routing layer. This layer changes behavior based on which workspace sent the request.

So the question is not whether to ask about models. It is which questions produce signal.

Standard reference questions produce standard reference answers. "How was your onboarding?" gets you "smooth." "Are you happy with accuracy?" gets you "yes."

Every reference call sounds the same, because every vendor handpicks references who will say these things.

The questions that produce signal are different. Ask: "Tell me about a time the AI got something wrong in production." Then ask how the vendor responded.

This forces the reference to narrate an incident, not a sales pitch. The quality of their answer tells you more than any demo.

Follow up with this: "What surprised you in month three or month six?" Add: "What did the sales process not prepare you for?"

This is where model swaps show up. It is also where routing changes show up. It is also where unexpected retraining schedules show up.

Vendors will not warn you about these. References will, if you ask the right way.

Ask directly: "Has the vendor changed the underlying model since you went live?" Then ask: "Were you notified?" Finally ask: "Did your evaluation metrics shift?" A long pause here tells you everything.

A vendor that has never changed models is either very young or not telling the truth. A reference that hesitates is telling you something important about the AI deployment you are about to inherit.

This is AI vendor due diligence that works. It forces the reference to narrate the messy middle, not the polished opening.

The design itself is where the deeper signals live.

Strong vendors support multiple frontier providers (Claude, GPT, Gemini) with per-workspace selection and optional per-intent routing. This is table stakes for serious production AI in 2026.

If your vendor cannot name which providers they support natively, that is a problem. You are looking at a wrapper, not a platform.

The critical signal is a second-pass QA layer. This layer runs on a different model from the response model.

A response model generates an answer. A QA model reviews that answer against the conversation, retrieved sources, tool calls, and applicable policy. Then it outputs approve, rewrite, or escalate.

This separation is what separates serious enterprise AI systems from consumer wrappers. It also explains why a reference might run a different model from yours.

The QA layer does the actual quality work. The response model is a cost-based choice. The response model can change without affecting output quality.

What the QA layer should see: the full conversation, retrieved sources, tool calls, and applicable policy. Its outputs should be discrete (approve, rewrite, escalate) and visible in the product UI.

If your governance team cannot see why the AI made a given decision, they will reject the deployment.

Ask for reasoning logs. If the vendor cannot show why the AI made a given decision, walk away.

An AI reference design that keeps model choice apart from app code is the only defense. It protects you from the model-swap problem.

When model selection is tied to your app code, a model upgrade breaks your workflows. When it is separate, you swap providers without rebuilding anything.

Knowing the design is one thing. Getting a vendor to admit it on paper is another.

Questions on paper are needed. These five convert AI vendor due diligence from a checklist task into a design audit. - Q1: Which model providers are supported natively, and which require custom work? List them by name. - Q2: Can we select different models per workspace, per use case, and per intent? Describe the routing logic. - Q3: Does your QA layer run on a different model from the response model by default? What does it see? What are its output actions? - Q4: When you change the underlying model for a customer, what is the notification process? What evaluation metrics do you re-run? - Q5: Provide the reasoning log format. Can our governance team audit it without engineering support?

The first question exposes whether you are talking to a platform or a wrapper. The second exposes whether model selection is a setup choice or a rebuild. The third is the strongest signal of design maturity. The fourth exposes whether silent model swaps are part of the vendor playbook. The fifth exposes whether your governance team can do their job.

These questions matter. The difference between a vendor whose enterprise AI survives model changes and one that breaks on every upgrade is design-based. It is not about operations.

Vendors who can answer all five in writing have thought about this problem. Vendors who deflect will change models under you and hope you do not notice.

The difference between a smooth deployment and a rework comes down to answers. These five questions must be asked before the contract is signed.

Questions on paper are needed. But the real test happens when you call the references.

Open with: "Walk me through the model you signed a contract with." Then ask: "What model are you running today?" Silence after this question is itself an answer.

A reference whose model has never changed is either the vendor first customer or a made-up story. A reference who answers honestly is a reference worth talking to.

Probe with: "Has your vendor ever silently changed models?" Then ask: "How did you find out?" Finally ask: "What was the business impact?" This is the question that separates mature vendors from the rest.

The reference who says "yes, and they told us six weeks in advance" describes a vendor worth buying from. The reference who says "we noticed when accuracy dropped" describes a vendor you should avoid.

Test QA maturity: "Show me an example where the QA layer caught an error before it reached a user." Then ask how long it took to investigate. If the reference cannot answer, the QA layer is either not running or not visible.

Both are red flags for any deployment that touches regulated AI systems.

Probe lock-in: "If you wanted to swap the response model to a different provider tomorrow, what would break?" Then ask: "What would you need to rebuild?" The answer tells you a lot. It shows whether model choice is truly separate from app logic. It also shows whether the vendor locked you in by design.

Close with: "What would you ask the vendor now that you wish you had asked before signing?" This single question surfaces more than the rest combined.

It turns a reference call from a sales proxy into a real intelligence source.

For enterprise AI buying, this script replaces generic "how was your experience" with structural questioning. Vendors cannot coach references for questions that surface what went wrong.

When you run this process, the picture changes.

You stop trusting demos as proxies for production behavior. You start evaluating design commitment instead.

The vendor demo is a performance; the design is the contract. When you evaluate design, you stop being surprised by model swaps.

The vendor answer to Q4 tells you a lot. It tells you when they will change models. It tells you how they will change them. It also tells you what they re-run first.

Your enterprise AI buying process catches lock-in risk before you sign. It does not wait for the first upgrade to break a workflow.

Your governance team gets the audit trail and reasoning logs they need. They can approve deployment. They do not need to rebuild internal review from scratch.

You select vendors whose genai design survives model changes without requiring a rebuild.

This is why some enterprise AI systems stay in production for years while others get replaced within months. The difference is not the model.

The difference is whether the vendor built a platform that hides model choice from app logic. Or a wrapper that hard-codes today provider into tomorrow deployment.

The vendors worth trusting can answer the five RFP questions and the six reference call questions without flinching.

If you have to choose between polished demos with vague answers, and rough demos with design clarity, choose the latter.

For teams building this due diligence process from scratch, the Levitation team production experience helps. It shows what survives model changes. It also shows what breaks.

How do you evaluate an AI vendor model flexibility?

Ask whether the vendor supports multiple frontier model providers natively. Ask if you can select models per workspace or per intent. Ask if the QA layer runs on a different model from the response model.

Request the reasoning log format and the model-change notification process in writing. Flexibility is a design property, not a marketing claim.

What questions should I ask AI vendor reference customers?

Skip "how was your onboarding." Ask: "Has the vendor changed the underlying model since you went live?" Also ask: "Tell me about a time the AI got something wrong in production and how the vendor responded." Finally ask: "What surprised you in month three or six that sales did not prepare you for?"

These surface model swaps, routing changes, and the messy middle that polished reference calls never reveal.

How do you avoid vendor lock-in with AI platforms?

Demand a design that keeps model choice apart from app logic. It should have per-workspace model selection. It should have per-intent routing. It should also have a QA layer that is separate from the response model.

Ask the vendor to demo a model swap in a sandbox. If they cannot, the platform is locked in by design, not by accident.

What is a multi-model AI architecture?

A multi-model design routes different requests to different models based on intent, cost, latency, or quality needs. It often runs a separate QA model to review outputs before delivery.

It is the standard for production AI systems that need to balance performance, cost, and governance across use cases.

How long does enterprise AI deployment take?

The timeline depends on whether due diligence happened before or after vendor selection.

Fast deployments and slow ones differ for a reason. It comes down to model review. It also comes down to governance integration. It also comes down to reference design choices made before or after vendor selection.