Compatibility is the shape; the contract is the behavior
An application benefits from a common request and response format. It can change models or providers with less client work, and a team can put access behind one interface. That is real value. Yet the same JSON fields do not tell an operator whether a request was accepted, which provider actually ran it, when the application may retry, how a partial answer is classified or how final usage will be reconciled. Those questions become part of the product contract as soon as people depend on the system.
A gateway therefore needs defined boundaries: input validation, eligibility and policy checks, route selection, execution, outcome classification and usage evidence. These are design responsibilities, not claims that every gateway implements them identically. A particularly important distinction is between a successful HTTP exchange and successful work. A response can be syntactically valid while the requested generation is incomplete or unusable. Conversely, a caller can lose the response after the provider has already performed work. Treating both cases as a generic error hides the decision the operator actually needs to make.
A stream is a sequence of evidence, not a single answer
Streaming lets an application present output while a model is still generating it. OpenAI's documented HTTP stream, for example, uses server-sent events and distinguishes incremental output from a completion event. The application can display a useful prefix before it knows whether the response has completed. That changes the failure boundary: a client may receive text, then see its connection time out without receiving a terminal event. The visible text is not proof of a completed operation. [2]
In that situation, the gateway should preserve what it knows: request identity, selected route, whether upstream accepted the call, the events it observed and whether completion was confirmed. It should also separate the user-visible partial answer from a finalized result. This is an architecture recommendation, not a promise that the upstream provider can resume every stream. A system may expose an incomplete result, let the caller decide whether to start a new attempt or consult a provider-side response record when the provider supports one. The important behavior is to name the uncertainty instead of silently converting it into success or failure.
The retry decision must account for duplicate work
Consider an explicitly illustrative drafting request. An application sends a generation request; the provider begins streaming; the application receives two paragraphs; then its connection times out. The provider may continue generating and may charge for the work already done. If the application immediately sends the same prompt again, it may create a second generation and a second cost, even if it later shows only one answer. The gateway cannot infer from the timeout alone that the first attempt never happened. OpenAI's Chat Completions documentation also notes that an interrupted stream may omit the final usage chunk. [3]
HTTP's semantics make the caution precise: a client should not automatically retry a non-idempotent request unless it knows the operation is effectively idempotent or knows the first request was never applied; a proxy must not automatically retry such requests. Model generation is commonly invoked by POST, so the safe choice cannot be based on a familiar endpoint shape. [1] A practical policy may allow a retry before dispatch, use a provider-supported idempotency mechanism when available, or require an explicit new attempt after an ambiguous stream. The tradeoff is latency and convenience versus duplicate generation and uncertain cost. No universal retry rule can remove that tradeoff.
Decide what is allowed before choosing where to run it
Routing is the question of where an eligible request should execute. Policy asks whether the request may execute at all and under what constraints. Those decisions need different evidence. A workload may specify a modality, context size, region, approved provider set, tool permissions or spending ceiling. If a preferred provider is unavailable, a fallback is useful only when it still satisfies those constraints. Quietly switching to an ineligible route is not reliability; it changes the agreement with the application.
A legible flow is: validate the request → evaluate policy and budget → select an eligible route → execute with a deadline → classify the outcome → record observed usage → issue any billing event through the relevant product's own system. That flow is illustrative architecture, not a description of shipped RouterShift internals. It makes the point that route choice, failure handling and accounting should be separately explainable. A deterministic rule is easier to audit; adaptive selection may help when reliable feedback exists, but it adds a burden to show why a particular option was chosen.
Measure observed usage before turning it into money
A gateway can observe a request, an attempt and sometimes a provider's final usage record. Those are different facts. An interrupted stream can leave the final usage unknown, so a partial token count inferred from visible text should not be presented as a settled charge. Keep the original provider record, the attempt identity and the basis for any estimate. Reconciliation can happen when authoritative usage arrives; corrections should be traceable rather than silently rewriting a prior figure. [3]
OpenTelemetry's current GenAI conventions name input and output token attributes, model and provider context, and error classification. They are a useful vocabulary for telemetry, not a billing system or a guarantee that all providers expose identical units. The conventions also warn that captured message content may contain sensitive information. [4] Pricing and balance movement belong downstream of measurement, with their own currency, rate version and audit trail. For image, audio or video work, a token-only assumption may be inadequate; the system should preserve the provider's actual unit and modality rather than forcing unlike work into one counter.
Make the contract inspectable in ordinary operations
The gateway becomes useful when a team can answer ordinary incident questions: Which request was affected? Which route was selected and why? Did upstream accept it? Was the output complete? What usage was reported, estimated or still unknown? Which policy and price version applied? Request and attempt identifiers, outcome states and trace links make those answers possible without turning prompt content into a default log. The resulting interface should be understandable to people who operate the product, not only to its implementers.
That also gives a practical test for a gateway's claims. If a provider fails before dispatch, a different eligible route may be straightforward. If a stream breaks after generation began, the system should say that the outcome is incomplete or uncertain and explain the available next action. If usage is missing, it should say so until reconciled. API compatibility lowers the entry cost; the production contract is what makes behavior predictable when the happy path ends. RouterShift is SlateMoth's live model-access product. The framework here explains the design problem, not a claim that every mechanism described has shipped in RouterShift.