There is a sentence I keep hearing in AI engineering that sounds innocent, practical, and mature: “Just add a fallback provider.”
Clean. Elegant. Wonderful. The kind of sentence that usually survives only until production starts touching it. Because in a demo, fallback means: Provider A fails, call Provider B. In production, fallback means something very different.
Will Provider B interpret the prompt in the same way? Will it serialize the tool schema in the same way? Will it respect cache directives in the same way? Will it stream tokens in the same way? Will it expose errors in the same way? Will it count tokens in the same way? Will it respect timeouts in the same way? Will it fail in a way your system can actually understand?
Most of the time, the answer is no.
And this is where the current AI industry keeps doing its favorite magic trick. It takes something deeply unstable, wraps it in a familiar API shape, gives it a shiny compatibility label, and suddenly everyone behaves as if we have a standard. We do not have a standard. We have a costume!
Most of famous “OpenAI compatible" APIs are laying, and hiding the lack of standards behind a known name. In reality the is compatible only on the shallowest path. You can send a basic chat request and get text back. Fantastic. The demo works. The slide looks good. The architecture diagram has fewer boxes. But the moment you move beyond “hello model, summarize this paragraph”, things start to fracture.
Tool calling. Structured output. JSON enforcement. Prompt caching. Streaming. Retry behavior. Usage accounting. Model aliases. Safety overlays. Regional routing. Timeout semantics. Error objects. Response envelopes. Context handling. Provider-specific parameters. All the boring parts. In other words, all the parts that decide whether your AI system survives production.
As we know, the demo works because the demo is NOT the system
The demo is usually a happy path. One user. One model. One provider. One prompt. One task. Maybe no cache. Maybe no concurrency. Maybe no structured output. Maybe no audit trail. Maybe no fallback. Maybe no customer-specific version pinning. Maybe no compliance requirement. Maybe no cost pressure. Maybe no incident where several parallel streams connect and then produce absolutely nothing for two minutes.
In that world, AI feels magical. In production, AI feels like distributed systems decided to have a child with legal ambiguity and probabilistic behavior.
You are not only integrating a model. You are integrating a runtime. And that runtime is usually not specified clearly enough. This is the part people keep missing.
The model is not the full product. The provider’s serving stack is part of the product. The SDK is part of the product. The serialization layer is part of the product. The cache implementation is part of the product. The safety wrapper is part of the product. The regional routing strategy is part of the product.
So when someone says, “it is the same model”, I increasingly hear: “we did not measure the parts around the model.”
Same weights do not mean same behavior. Same model family does not mean same production contract. Same endpoint shape does not mean same system.
Same model, different reality
One of the strongest examples I have seen came from a direct comparison between Provider A and Provider B using the same model family on real production-like workflows. The headline looked simple: same model family, different provider path. The result was not simple.
On Workflow 1, the quality regression was statistically significant. Provider A had a mean score of 0.716. Provider B had a mean score of 0.497. The p-value was below 0.0001, with a medium effect size. That is not “a bit of noise.” That is the kind of difference that should stop a migration.
The interesting part is that not every workflow regressed. On Workflow 2 and Workflow 3, the result was basically fine.
Good. That makes the result more credible, not less. Because real provider migrations do not fail everywhere. They fail in specific workflows, specific prompts, specific schema paths, specific flows, specific edge cases. The average can look acceptable while one critical workflow quietly gets worse.
This is exactly why “we tested a few prompts manually and it looked okay” is not engineering. It is theater with curl commands.
If you want to switch provider, you need replay traces. You need evals. You need per-workflow scores. You need statistical comparison. You need to know where the behavior changed, not just whether the model still speaks fluent corporate English.
Cost is not only list price
The same comparison showed around a 2x cost premium on a high-volume workflow. At first glance, you might blame provider pricing. But the back-calculation pointed somewhere more boring and more dangerous: prompt caching.
On Provider A, implied token volume was 60 to 67 percent below reported tokens. That is the cache signature. You are still sending the structure, but you are not paying the full input cost every time because the provider is reusing cached prompt blocks.
On Provider B, one high-volume path showed exactly 0 percent gap. Cache was either off or always missing. Other paths showed partial cache behavior, around 14 to 21 percent in one case and around 33 percent in another.
Same model family. Different cache reality. Different bill. This is where the “just switch provider” crowd usually becomes very quiet.
Because caching is not decoration. In high-volume AI systems, caching is part of the economic architecture. If cache semantics change, your unit economics change. If regional routing causes cache misses, your cost model changes. If one provider respects cache directives differently from another, your production bill changes while every individual request still “works.”
That is the worst kind of failure. The successful one. No exception. No stack trace. No screaming service. Just a quiet invoice telling you the abstraction was fake.
Cross-region cache is a beautiful little trap
Cross-region inference sounds robust. More regions. More availability. More resilience. Then you look at the cache behavior.
A request served in Region A writes a cache in Region A. The next request may route to Region B. Region B does not have that cache. So it misses and writes again. Then another call may route back to Region A, or somewhere else, depending on capacity and routing.
This is not a clean “double pay” situation. It is worse conceptually. You keep paying the cache write premium without reliably amortizing it through cheap cache reads. That is how you can end up with a measured 0 percent hit rate while thinking you configured caching correctly.
Again, from the outside everything looks compatible. The API accepts your request. The model responds. The integration works. Except the economics are different because the serving layer changed.
This is why AI production work is becoming less about prompts and more about contracts. What exactly is guaranteed? What is pinned? What is regional? What is cached? What is counted? What is replayable? What is stable?
If the answer is “trust us, it is compatible”, my engineering translation is simple: no contract found.
Reliability is not portable either
Another production-style incident: under concurrency, multiple parallel streams connected and then produced nothing for roughly two minutes. No tokens. No useful error. Just waiting.
The likely reading was capacity or throttle queueing. The provider may have been holding the request instead of returning a clean throttling response. Depending on the endpoint, one path may queue in-flight work while another may throw a clear rate-limit error.
That distinction matters. A clear rate-limit error is ugly but useful. You can react to it. You can retry with backoff. You can trigger fallback. You can protect the system. A connected stream producing nothing for two minutes is a different species of failure. Your system is alive enough to wait and dead enough to be useless.
There was also a competing hypothesis: maybe the network layer was involved. Gateway behavior, private endpoints, load balancers, idle timeouts, streaming connection drops, or capacity errors could all produce overlapping symptoms.
So the correct response was not “the provider is bad.” The correct response was: inspect runtime metrics during the hang windows. Check throttle counters. Check server error counters. Check network timeouts. Check connection lifetime. Check whether the request reached the model runtime at all.
This is what production AI looks like. Not prompt magic. Not demo videos. Not “look, I built an agent in 20 minutes.” It looks like debugging whether a zero-token two-minute hang is caused by model capacity, runtime queueing, network infrastructure, streaming semantics, retry policy, or your own concurrency design.
Very glamorous. Someone should put that in the launch video.
Structured output is not standard output
Then there is the SDK serialization problem. Same model. Same app-level input. Different token count.
One comparison showed Provider B using around 10,473 input tokens while Provider A used around 10,019. That is a 454-token delta, roughly 4.5 percent.
The clue was structured output. On one provider path, structured output was implemented by injecting a tool schema into the prompt. On the other path, it was handled differently. Even after making payloads byte-identical at the application level, the remaining structural difference came from provider-specific cache directive serialization.
This is a perfect example of why API compatibility is not enough. Your prompt may be identical. Your provider prompt is not.
The actual thing seen by the model may include hidden scaffolding, injected schemas, translated parameters, safety wrappers, tool definitions, response constraints, or provider-specific envelopes. Then we compare eval scores and pretend we tested the same thing.
Did we? Maybe. Maybe not. And if we cannot answer that confidently, then we are not measuring model quality. We are measuring a mix of model behavior, SDK translation, provider scaffolding, and our own assumptions.
Very scientific. Very enterprise. Very “move fast and accidentally compare different systems.”
Fallback can become the outage
Cross-provider fallback sounds responsible. It can be responsible. But it is not free.
One concrete incident involved a preview model on Provider C. The model had intermittent hangs, produced retry-exhaustion errors after repeated timeouts, and reported zero input tokens and zero output tokens. So the model did not even really start.
The retry budget burned for several minutes. Then a failure-rate guard aborted the whole job. The fix was to add a fallback model. Good fix. But the lesson is bigger.
The fallback path needs its own engineering. It needs its own timeout budget. It needs its own cost assumption. It needs its own quality expectation. It needs its own reason to exist.
A useful rule: scale up on fallback by default. If fallback runs rarely, a usable answer matters more than saving a few cents. Scale down only when the primary failed because the request exceeded model limits.
But if your fallback inherits the same exhausted timeout budget from the primary, congratulations, you did not build fallback. You built a decorative second failure.
Fallback is not a backup model. Fallback is a second production path.
Even parameter names are not portable
Small example, but very revealing: a “compatible” API for a model behaved differently around a reasoning-related parameter. The workaround was to force a safe default.
That is reasonable. But the real portability risk is not only the value. It is the parameter contract itself.
Another provider may call it something else. Another may ignore it. Another may reject it. Another may apply a different default. Another may support it only on some models. Another may support it in preview and remove it later with very little warning.
This is where “compatible API” starts to feel like saying every car is steering-wheel-compatible. Technically true. Please do not use that as your safety case.
Preview is not production
A lot of AI teams are building production workflows on preview models, preview parameters, preview endpoints, preview SDK behavior, and preview pricing assumptions. Then they act surprised when preview behaves like preview.
Preview can mean weaker guarantees. It can mean limited support. It can mean behavior changes. It can mean short deprecation windows. It can mean different rate limits. It can mean hidden routing changes. It can mean features that work today and become “not recommended” tomorrow.
That is fine for exploration. That is not fine when your production system depends on it and nobody wrote down the risk.
Again, the issue is not that preview exists. Preview is useful. The issue is pretending preview is stable because the demo worked.
We need stable interfaces, not demo optimism
I am increasingly convinced that production AI needs something closer to long-term-support thinking.
Not because models should stop improving. They will improve. The field moves fast. Fine. But production systems cannot keep pretending that every model upgrade, provider switch, SDK change, cache behavior update, or model alias movement is harmless.
When a system is performing fine, switching the model or serving path can create more issues than benefits.
The defensible version is conditional: long-term support becomes inevitable when capability growth slows enough that stability outweighs the next incremental benchmark gain. At that point, many companies will not want the newest model. They will want the model-runtime contract that keeps working.
But the deeper point is that the thing needing long-term support is not only the model weights. It is the interface.
The stable surface must include serving stack, quantization, SDK behavior, tool serialization, cache semantics, timeout behavior, safety overlays, error formats, and versioned model aliases.
Maybe the real answer is not long-term-support models. Maybe it is long-term-support interfaces with deterministic check layers.
Swappable models behind a stable contract. Replayable traces. Eval gates. Schema normalization. Provider-specific adapters. Explicit cache tests. Timeout isolation. Failure-mode classification. Version-pinned prompts. Known fallback policy.
That sounds boring. Good. Production should be boring.
The problem is not that AI is useless
This is usually where someone misunderstands the argument. The point is not that AI is useless. The point is not that demos are bad. The point is not that teams should stop experimenting.
The point is that demos and production systems are different organisms. A demo proves possibility. Production requires repeatability. A demo proves that the model can answer. Production requires knowing what happens when it does not answer, answers differently, answers slowly, answers with a hidden schema injection, misses cache across regions, changes token accounting, streams forever, returns a provider-specific error, or silently regresses one workflow while improving another.
AI interfaces today are still too fragmented for the amount of confidence people are placing in them. We are building production systems on unstable runtime surfaces and pretending the abstraction is mature because the JSON shape looks familiar.
That is not engineering maturity. That is hope with headers.
So maybe not every demo belongs in prod yet
Maybe it is not yet time to bring every AI demo to production. Or more precisely: maybe it is not time to bring demos to production without first building the missing runtime layer around them.
Not another wrapper. Not another “universal SDK” that hides provider differences until they explode. A real layer. One that treats each provider as a different runtime with different semantics.
One that records traces. Replays production samples. Compares quality. Measures cost after caching. Tracks token deltas. Normalizes errors. Separates timeout budgets. Tests fallback paths. Pins model versions. Detects serialization drift. Audits structured output behavior. Makes provider migration observable before it becomes an outage.
Because changing provider is not changing a base URL. It is migrating the runtime contract of your AI system.
And if your system does not know what that contract is, then the provider switch is not a migration. It is an experiment in production.
Very innovative, yes. Also known in some older engineering traditions as a bad idea.
