Erik Caldwell
Connect on LinkedIn
Menu

The Topic Catalog · 11

Enterprise AI Gateway & Control Plane

Last updated · 8 min read

11.1 AI Gateways & Model Gateways

Priority: Must Understand

Executive Definition: An AI gateway is a control-plane layer that sits between applications and the LLMs they call (hosted or third-party), giving the organization one place to enforce policy, meter usage, and standardize integration. Instead of every application team wiring its own SDK directly to a model provider, all traffic is proxied through a gateway that can authenticate callers, apply guardrails, log every request/response, and route to whichever underlying model backend is appropriate: functionally analogous to an API gateway in front of microservices, but for model calls instead of service calls.

Why It Matters: Without a gateway, every application team makes its own decisions about keys, logging, rate limits, and content safety: producing inconsistent security posture and no enterprise-wide visibility into what models are being called, by whom, and at what cost. A gateway is also the only practical place to retrofit policy and cost controls across dozens of applications without touching each one's code. Both AWS and Microsoft now ship this as first-party infrastructure (Bedrock's built-in security/observability layer, Azure's API Management (APIM) AI Gateway capabilities), signaling this has become a standard architectural layer, not an optional add-on.

What I Need to Understand:

  • The gateway is a proxy/control plane, not a model: it does not change model quality, only how calls are authenticated, routed, observed, and constrained.
  • Core gateway functions: auth/identity passthrough, request/response logging, rate limiting, retries/failover across providers, semantic caching, and unified API translation (so app code targets one interface regardless of underlying model).
  • Cloud-native versions (Azure API Management's GenAI gateway capabilities, Bedrock) integrate natively with existing Identity and Access Management (IAM) and network controls; independent gateways (LiteLLM, Kong AI Gateway, Portkey) trade that native integration for multi-cloud/multi-provider flexibility.
  • A gateway is the natural enforcement point for centralized policy controls and the telemetry source for observability work: it should not be evaluated in isolation from those.
  • Gateway choice interacts directly with build-vs-buy and vendor lock-in decisions (see Multi-Model Strategy topic).

Questions I Should Be Able to Ask My Team:

  1. Is every production LLM call in the organization routed through a single gateway, or can teams still call provider APIs directly and bypass it?
  2. What happens to in-flight requests and cost/policy enforcement if the gateway itself has an outage: is there a documented failure mode?
  3. Does the gateway log full prompts/responses by default, and who can access those logs?

Technologies / Standards / Companies to Know: AWS Bedrock (native gateway + guardrails), Azure AI Foundry / Azure API Management GenAI gateway capabilities, Google Cloud API Gateway model routing / Apigee, LiteLLM, Kong AI Gateway, Portkey.

Recommended Learning:

Time Investment: 1 hour

11.2 Centralized Policy Enforcement (authZ, rate limits, Data Loss Prevention (DLP), content filtering, cost controls)

Priority: Must Understand

Executive Definition: Centralized policy enforcement means authorization, rate limiting, data-loss-prevention (DLP), content filtering, and cost controls are applied consistently at the gateway layer rather than left to individual application teams to implement (or forget to implement). This covers who is allowed to call which model with what data (authZ), how much traffic and spend any caller can generate (rate/cost limits), what sensitive data is blocked from leaving or entering the model (DLP), and what categories of content the model is disallowed from producing or acting on (content filtering).

Why It Matters: Decentralized enforcement produces the same outcome every time: some team's integration has weaker controls than the others, and that becomes the incident. OWASP's Top 10 for LLM Applications names excessive agency, sensitive information disclosure, and improper output handling among the leading LLM application risks specifically because these are usually per-application gaps rather than platform-wide policy failures. Centralizing enforcement turns a governance requirement into a configuration change in one place instead of a code change in fifty places.

What I Need to Understand:

  • AuthZ at the gateway should be scoped beyond "can this caller reach the model" to "can this caller's identity/role access this specific model, dataset, or tool": this is the same excessive-agency risk OWASP flags for agentic systems.
  • DLP for LLM traffic means inspecting both inbound prompts (for secrets, customer Personally Identifiable Information (PII) being pasted into prompts) and outbound completions (for PII/regulated data being generated): this is materially different from traditional network DLP, which does not understand model context.
  • Content filtering (both AWS Bedrock Guardrails and Vertex AI's safety filters implement this as configurable severity thresholds per harm category, not a single on/off switch) needs to be tuned per use case: a legal-research assistant and a customer support bot need different filter thresholds.
  • Rate limits and cost controls at this layer are the first line of defense against runaway-cost scenarios, and should be enforced per identity/application, not just globally.
  • Policy enforcement needs its own audit trail: decisions to block, redact, or allow need to be logged for incident-response and audit-replay purposes.

Questions I Should Be Able to Ask My Team:

  1. Are our DLP rules inspecting both prompts and completions, or only one direction?
  2. Which OWASP LLM Top 10 categories does our current guardrail configuration actually cover, and which are we accepting as unmitigated risk?
  3. Can any single application or identity exceed its rate/cost limit by routing around the gateway, and how would we detect that?

Technologies / Standards / Companies to Know: OWASP Top 10 for LLM Applications, NIST AI RMF (AI 600-1 Generative AI Profile), Amazon Bedrock Guardrails, Azure AI Content Safety, Google Model Armor / Vertex AI safety filters.

Recommended Learning:

Time Investment: 2-3 hours

11.3 Gateway Architecture Choices (AWS Bedrock, Azure AI, Google Vertex AI, direct model APIs, build vs buy)

Priority: Monitor

Executive Definition: Enterprises choosing a gateway approach are really choosing among four architecture patterns: a cloud provider's native gateway tied to that provider's models and IAM (AWS Bedrock, Azure AI Foundry), a cloud-native but multi-model gateway (Vertex AI plus Google Cloud API Gateway's model routing, or Azure API Management sitting in front of both Azure and non-Azure models), an independent multi-cloud gateway (LiteLLM, Kong, Portkey) that is provider-agnostic but adds an operational component you now own, or direct model API calls with no gateway at all. Each trades integration depth against lock-in and flexibility.

Why It Matters: This decision determines how hard it is to add a second model provider later, how much of your existing cloud IAM/networking investment you can reuse, and who is on the hook when the gateway itself has an incident. It is an architecture decision with multi-year switching costs, not a procurement checkbox: treating it as the latter is how organizations end up locked into a single model provider's roadmap for reasons that were never actually evaluated.

What I Need to Understand:

  • Native cloud gateways (Bedrock, Azure AI Foundry's model router and APIM AI Gateway) give you tighter IAM/VPC/logging integration but couple your model strategy to that cloud's model catalog and release cadence.
  • Vertex AI plus Google Cloud API Gateway follows the same native pattern on GCP, including model routing across Gemini and third-party models hosted there.
  • Independent gateways decouple model choice from cloud provider but add a new piece of infrastructure your team must operate, secure, and scale: the gateway becomes a new single point of failure and a new attack surface.
  • "Direct API" (no gateway) is viable only at very small scale or for a single, tightly scoped application; it does not survive contact with more than a handful of applications or any centralized policy requirement.
  • Build-vs-buy here is really "adopt the cloud-native layer we're already paying for" vs. "add an independent control plane for multi-cloud flexibility": there is rarely a case for building this from scratch given how much this space has matured.

Questions I Should Be Able to Ask My Team:

  1. If we standardize on one cloud provider's native gateway, what is our actual plan and cost to add a second model provider later?
  2. What is the operational owner and SLA for our gateway layer: is it treated with the same rigor as any other tier-1 production dependency?
  3. Does our gateway choice let security apply the same authZ/DLP policy regardless of which underlying model is called, or does policy have to be re-implemented per provider?

Technologies / Standards / Companies to Know: AWS Bedrock, Azure AI Foundry / Azure API Management, Google Vertex AI / Google Cloud API Gateway, LiteLLM, Kong AI Gateway, Portkey.

Recommended Learning:

Time Investment: 2-3 hours