Erik Caldwell
Connect on LinkedIn
Menu

The Topic Catalog · 9

Foundation Models & AI Engineering

Last updated · 9 min read

9.1 Frontier, Reasoning & Multimodal Models

Priority: Must Understand

Executive Definition: Frontier models are the current highest-capability general-purpose models from the major labs (Anthropic Claude, OpenAI Generative Pre-trained Transformer (GPT), Google Gemini). "Reasoning" models are variants trained or configured to spend extra inference-time computation working through a problem step by step before answering, trading latency and cost for accuracy on harder tasks. Multimodal models accept and/or produce more than text (images, audio, video, and documents) within the same model.

Why It Matters: The three labs now ship closely competitive, frequently-updated models rather than one stable leader: Anthropic's Claude Opus 4.5, OpenAI's GPT-5.1, and Google's Gemini 3 all shipped within months of each other in late 2025, so architecture decisions should assume ongoing model churn rather than a fixed, permanent choice. Reasoning mode is a cost/latency dial, not a universal upgrade, and should be applied selectively rather than defaulted on everywhere.

What I Need to Understand:

  • Extended reasoning ("thinking") trades latency and token cost for accuracy on complex tasks: enable it selectively for tasks that need it, not as a blanket default.
  • Frontier leadership rotates every few months across labs; procurement and architecture should assume multi-model flexibility (see Model Routing topic) rather than long-term single-model commitment.
  • Multimodal input (documents, images, audio, video) opens real new use cases, but quality varies significantly by modality and provider: validate per use case rather than trusting headline benchmarks.
  • Benchmark leaderboards are frequently gamed or non-representative of your task mix: insist on evaluation against your own representative tasks before switching models.

Questions I Should Be Able to Ask My Team:

  1. Which of our workloads actually need reasoning/extended-thinking mode, versus using a faster non-reasoning model and saving the cost and latency?
  2. What's our process for re-evaluating model choice as new frontier releases ship, and how often do we actually revisit it?
  3. For multimodal use cases under consideration, has accuracy been validated against our own documents/images, not just published benchmarks?

Technologies / Standards / Companies to Know: Anthropic Claude Opus/Sonnet/Haiku, OpenAI GPT-5.x, Google Gemini 3, model/system cards.

Recommended Learning:

Time Investment: 1 hour, ongoing (expect quarterly refresh as models update)

9.2 Open-Weight vs Proprietary Models

Priority: Must Understand

Executive Definition: Proprietary models (Claude, GPT, Gemini) are accessed only via API: you never hold the weights, and the provider controls updates, hosting, and terms. Open-weight models (Llama, DeepSeek, Mistral, Qwen) publish downloadable weights you can run on your own infrastructure, but "open" refers to distribution, not necessarily a permissive open-source license: terms vary and materially affect what you may do with them.

Why It Matters: Meta's Llama license is not OSI-approved open source: it carries usage conditions that require legal review, not a "free to use" assumption, according to the Open Source Initiative. Open-weight models trade frontier-tier convenience and vendor-managed reliability for control (on-prem/air-gapped deployment, no data leaving your infrastructure, no vendor roadmap dependency), but you take on hosting, scaling, and patching responsibility: this is a deployment-control and cost-structure decision, not simply a leaderboard comparison.

What I Need to Understand:

  • "Open-weight" is not synonymous with "open source": read the actual license (Llama Community License, Mistral's various licenses, DeepSeek's license) before assuming unrestricted commercial use.
  • Self-hosting an open-weight model means owning hosting, scaling, security patching, and update cadence: real infrastructure and staffing cost that a proprietary API abstracts away.
  • Proprietary frontier models generally still lead on the hardest reasoning/multimodal tasks; open-weight models have closed much of the gap on everyday tasks at meaningfully lower inference cost.
  • Air-gapped/on-prem or strict residency requirements are often the actual deciding factor for open-weight adoption, more than raw capability comparisons.

Questions I Should Be Able to Ask My Team:

  1. For any open-weight model we deploy, has legal reviewed the license terms for our specific use case (commercial use, redistribution, data restrictions)?
  2. What is the fully-loaded cost (infrastructure, staffing, patching) of self-hosting versus the equivalent proprietary API cost at our expected volume?
  3. Which of our use cases have a genuine residency or air-gap requirement that only a self-hosted, open-weight model satisfies?

Technologies / Standards / Companies to Know: Meta Llama, DeepSeek, Mistral, Qwen, OpenAI gpt-oss, vLLM/TGI for self-hosted serving.

Recommended Learning:

Time Investment: 1 hour

9.3 Context Windows, Tokens & Inference Fundamentals

Priority: Should Understand

Executive Definition: Tokens are the units (roughly word-fragments) models process; a context window is the maximum number of tokens a model can consider at once, spanning the prompt, retrieved documents, conversation history, and the response together. Inference is running the model to produce output, with real cost and latency that scales with tokens processed: a larger context window doesn't make more tokens free, it just raises the ceiling.

Why It Matters: Context window size determines what fits in a single request, but a bigger window doesn't guarantee the model uses everything in it well: long, cluttered contexts measurably degrade output quality even within the stated limit, which is exactly why context engineering (final topic below) exists as a discipline. Cost and latency scale with tokens in both directions, so context window size is a real budget lever your team is choosing, not just a technical ceiling.

What I Need to Understand:

  • Context window size sets an upper bound on what can be given to the model in one call: retrieval (RAG topic) exists precisely because you can't put an entire knowledge base in context.
  • More context per request is not free: cost scales with tokens processed, and excessive or cluttered context can degrade accuracy even within the advertised limit.
  • Track, don't implement: cost-per-request and latency-per-request trends, and whether accuracy holds as more of the context window is filled on your actual tasks.
  • Delegate to your team: the exact tokenization scheme, embedding dimensionality, and low-level inference optimizations (batching, quantization, KV-caching): these are implementation detail, not executive-level decisions.
  • Prompt caching (reusing a previously-processed prefix of tokens across repeated calls) is a real, delegable cost-optimization technique: you need to know it exists and ask whether it's used.

Questions I Should Be Able to Ask My Team:

  1. What is our typical cost and latency per request today, and how does that change with the context sizes we actually use in production?
  2. Are we using prompt caching or equivalent techniques for repeated large contexts (e.g., a shared system prompt or document set across many calls)?
  3. Have we tested whether accuracy degrades as we fill more of the context window on our own tasks, rather than assuming the full advertised limit is usable at full quality?

Technologies / Standards / Companies to Know: Claude/GPT/Gemini context window tiers, prompt caching, tokenizers.

Recommended Learning:

Time Investment: 30 minutes

9.4 Model Routing, Fallback & Structured Output

Priority: Should Understand

Executive Definition: Model routing means automatically directing each request to the most appropriate model by cost, capability, or latency need, rather than hard-coding one model everywhere. Fallback means automatically retrying with an alternate model or region when the primary one is unavailable or degraded. Structured output means constraining a model's response to a defined schema (e.g., JSON) so downstream systems can reliably parse it without brittle text parsing.

Why It Matters: Committing an entire application to one model creates both a cost problem (using an expensive model for simple tasks) and a resilience problem (a single point of failure). Cloud providers now ship routing and failover as core infrastructure: AWS Bedrock's intelligent prompt routing and cross-region inference exist because production systems need this, not because it's a nice-to-have, and structured output has moved from "hope the model formats it right" to provider-enforced schema guarantees, a real reliability improvement over ad hoc text parsing.

What I Need to Understand:

  • Routing by task (cheap/fast model for simple requests, frontier model for hard ones) is a cost-control lever your team should actively use, not a one-time setup.
  • Fallback strategy (what happens when the primary provider is down or rate-limited) should be an explicit, tested design decision, not something discovered during an outage.
  • Structured output (schema-constrained generation) is now natively supported by major providers and is materially more reliable than asking a model to "output JSON" in free text: insist on it wherever output is parsed programmatically.
  • Delegate to your team: the specific routing algorithm, gateway product, and JSON-schema plumbing.
  • Track as an executive: cost per model tier used, fallback incident frequency, and structured-output parse-failure rate.

Questions I Should Be Able to Ask My Team:

  1. Do we route requests by task complexity/cost today, or does every request go to the same (likely most expensive) model regardless of need?
  2. What actually happens when our primary provider has an outage or rate-limits us: is there a tested fallback, or does the application just fail?
  3. For systems that parse model output automatically, are we using provider-enforced structured output, and what's our current parse-failure rate?

Technologies / Standards / Companies to Know: AWS Bedrock intelligent prompt routing / cross-region inference, OpenAI Structured Outputs, Anthropic tool use, LLM gateways (LiteLLM, Portkey).

Recommended Learning:

Time Investment: 1 hour

9.5 Fine-Tuning, Distillation, Small Language Models & Edge Inference

Priority: Monitor

Executive Definition: Fine-tuning further trains an existing model on your own examples so it performs a specific task better without repeating guidance in every prompt. Distillation trains a smaller "student" model to imitate a larger "teacher" model on a specific task, producing a cheaper, faster, narrower model. Small language models (SLMs) and edge inference mean running compact models locally (on a laptop, device, or private server) instead of calling a large model over the network.

Why It Matters: These techniques trade generality for cost, latency, and control: a fine-tuned or distilled small model can be dramatically cheaper for a narrow, stable, high-volume task than always calling a frontier model: OpenAI's own distillation tooling exists specifically to make "use the big model to teach a cheap model for one task" a supported production pattern, not a research exercise. This is the clearest "decide the strategy, delegate the mechanics" topic in this domain.

What I Need to Understand:

  • Fine-tuning/distillation is justified for tasks that are narrow, high-volume, and stable: not for general-purpose or rapidly-changing tasks, where prompting or RAG against a general model is more practical.
  • A fine-tuned model still needs the same data governance (see Data Classification topic) applied to its training data: it can memorize and leak sensitive examples.
  • Own as an executive: which use cases justify the fixed cost of fine-tuning/distillation, what the ongoing retraining/maintenance commitment is, and whether the resulting model gets the same governance and access controls as any other production system.
  • Delegate to your team: the actual training run, hyperparameters, and evaluation harness.
  • A fine-tuned or distilled model is a new asset with its own lifecycle (versioning, retraining triggers, deprecation): it does not inherit automatic updates the way a hosted frontier model does.

Questions I Should Be Able to Ask My Team:

  1. Which specific use cases have we identified as narrow and high-volume enough to justify fine-tuning or distillation instead of prompting a general model?
  2. What's the retraining/maintenance commitment once we fine-tune or distill a model, and who owns it long-term?
  3. Does the training data for any fine-tuned model meet the same classification and governance bar as our other data assets?

Technologies / Standards / Companies to Know: OpenAI supervised fine-tuning and distillation tooling, small Llama/Mistral/Qwen variants for edge, on-device runtimes (llama.cpp, ONNX Runtime).

Recommended Learning:

Time Investment: 1 hour