Erik Caldwell
Connect on LinkedIn
Menu

The Topic Catalog · 22

Machine Judgment & Decision Models

Last updated · 32 min read

Based on research by Erik Caldwell, "Beyond Generative AI: The Emerging Role of Machine Judgment in Enterprise and Government Operations" (Second Edition, October 2026). Evidence is current to 23 October 2026, and the products described here are weeks old in their current form. Figures reported by vendors are labeled as vendor claims; the cost and use-case examples are hypothetical illustrations, not measurements.

22.1 Machine Judgment & the Four-Stage Framework: Information, Judgment, Decision, Authorized Action

Priority: Must Understand

Executive Definition: Machine judgment is a model's estimate of what category, condition, level, or probability best describes a situation: "this ticket is a security incident, 0.86." A four-stage framework keeps it in its place. Stage 1 is information (requests, documents, records). Stage 2 is machine judgment (what best describes the situation). Stage 3 is the institutional decision (what policy, law, and risk tolerance require). Stage 4 is authorized action (who or what has permission to act). Models contribute evidence at Stage 2; Stages 3 and 4 belong to institutions.

Why It Matters: A technically correct machine judgment can still lead to a wrong, unfair, or unauthorized institutional outcome. Take a benefits agency that uses a model to flag applications as "likely incomplete," accurate 95% of the time. If its process turns the flag directly into a denial letter, the other 5% of applicants lose benefits without a human reading their file. The classification was accurate and the decision was improper, because the authority to deny benefits carries obligations of notice, reasons, and appeal that a probability cannot discharge. The same flag used to prioritize caseworker review would be valuable and lawful. Economists Ajay Agrawal, Joshua Gans, and Avi Goldfarb make the general point: AI lowers the cost of prediction, not the cost of judgment about which outcomes matter.

What I Need to Understand:

  • Most enterprise AI investment since 2023 has gone to generative systems that write, summarize, and converse, but much of the work inside a company or agency is narrow, repetitive judgment: is this ticket a security incident, which department owns this complaint, does this purchase need a second approver. These are bounded questions with a short list of permitted answers, asked millions of times a year.
  • The simple test for any deployment: identify which stage the model's output feeds, and confirm that Stages 3 and 4 are governed by policy and accountable people, not by the model's output alone.
  • Many decisions turn on more than prediction. Kleinberg, Ludwig, Mullainathan, and Obermeyer called decisions that turn mainly on a prediction "prediction policy problems" and cautioned that many policy decisions are not of that kind. In their bail-decision study, outcomes were observed only for defendants judges chose to release, and judges weigh objectives the algorithm cannot see.
  • Wang, Kapoor, Barocas, and Narayanan argue that systems optimized for predictive accuracy about individuals often lack legitimacy in consequential settings, because accurate predictions do not ensure good interventions and people cannot contest what they cannot see.
  • A more accurate model does not automatically produce a better decision. Outcomes depend on thresholds, the cost of different errors, downstream processes, and who is accountable. Evaluate the decision process, not only the model.
  • An output is an advisory signal when it informs a person or a rule, and an authorization when it directly causes something to happen (a ticket closed, a payment released, an application denied). The same model and the same probability carry very different risk in the two roles.

Questions I Should Be Able to Ask My Team:

  1. For each AI use in our organization, which of the four stages does its output feed?
  2. Where does a model's output become an action with no person or rule in between?
  3. For each use, who by name owns the Stage 3 decision, and who holds Stage 4 authority to act?

Technologies / Standards / Companies to Know: Advisory signal versus authorization, prediction policy problems, delegation of authority, decision logging.

Recommended Learning:

Time Investment: 1 hour

22.2 Decision Models: What They Are and What Is Actually New

Priority: Should Understand

Executive Definition: A decision model reads unstructured input (text, structured data, sometimes images) plus one or more bounded questions written in plain language, and returns a probability for every permitted answer, without task-specific training and without generating prose. Three question types recur across vendors: a binary judgment (TypeSafe calls it a "Noul," OpenAI a "predicate"), a choice among unordered options, and a score on an ordered scale.

Why It Matters: In September and October 2026 a new category went from one product to a crowded field in about five weeks. TypeSafe AI released JEV in early access on September 15 and calls it a "System One model." OpenAI's Decisions API, built on GPT-6 Luna, entered public beta on October 6 and became generally available on October 22. Microsoft (Microsoft-Decision-1 in Foundry), Cloudflare (Clef, with open weights under Apache 2.0), Perplexity, and several research groups released comparable models within weeks. Capabilities that once required a dedicated machine learning project are becoming inexpensive building blocks that a developer can add to an application in an afternoon.

What I Need to Understand:

  • Separate three layers of novelty. The science, probabilistic classification, is decades old. The capability, a single general-purpose model answering arbitrary plain-language questions with useful probabilities and no per-task training, is recent and real. The packaging (typed question primitives, many questions per request evaluated in parallel, input-only pricing, software kits built for applications rather than chat) is where much of the practical value lies.
  • The capability is not proprietary magic. A Harvard preprint (Geuter, Cheng, and Alvarez-Melis, October 2026) showed that presenting options to an existing open language model as labeled choices and reading the probability it assigns to each label in a single pass roughly matched JEV 1.13 zero-shot. The implication is portability, credible open alternatives, and the option to self-host.
  • Neither TypeSafe nor OpenAI has disclosed enough to show whether their approaches differ materially. OpenAI describes its Decisions API as a dedicated endpoint on GPT-6 Luna, not a new model. Similar interfaces do not imply similar internals.
  • "Type safe" is not "right." A decision model can choose only among the answers you supply, which removes malformed output but not wrong answers. A constrained output format is not determinism either: about 3% of JEV's verdicts changed at least once across five repeated passes on one benchmark.
  • "System One" is a metaphor, not a mechanism. The phrase borrows Daniel Kahneman's fast and slow thinking, but it has no established technical definition, and even in psychology researchers prefer "Type 1" and "Type 2" processing to avoid implying two physical systems.
  • Several providers have copied the interface, which suggests conventions are emerging, but no standards body governs them. Vendors differ in limits, input types, confidence definitions, data handling, and residency, and "compatible" is a vendor claim to test. An internal abstraction layer over the three question types preserves the option to switch.

Questions I Should Be Able to Ask My Team:

  1. Which of our AI calls ask for a bounded answer but pay for generated text?
  2. If we adopted a decision model, could we switch vendors or self-host through an abstraction layer, or would we be coding to one vendor's kit?
  3. Where does this vendor process our data, how long does it retain it, and is that documented?

Technologies / Standards / Companies to Know: JEV (TypeSafe AI), OpenAI Decisions API (GPT-6 Luna), Microsoft-Decision-1 (Microsoft Foundry), Cloudflare Clef, open-weight decision models, discriminative versus generative tasks.

Recommended Learning:

Time Investment: 1 hour

22.3 Choosing the Right Tool: Rules, Classifiers, LLMs, Decision Models & Hybrids

Priority: Must Understand

Executive Definition: Five approaches handle machine judgment, and each fits a different problem. Deterministic rules fit policy that can be written down completely. Traditional machine learning classifiers fit repetitive classification on stable categories with a labeled history. Decision models fit bounded semantic judgments on varied input with no training data to start. Large language models (LLMs) fit reasoning, synthesis, and generation. Hybrids combine them for workflows that mix needs.

Why It Matters: A new model is not a reason to replace a system that works. Probabilistic classification did not begin with decision models: spam filters, fraud detection, and ticket routing have run on classifiers for decades. What decision models change is flexibility, because you can ask a new question in plain English without collecting labeled data and training a separate model for each one. The right question is never "is this better than an LLM?" but "what is the simplest, most reliable, most economical technology for this specific task?"

What I Need to Understand:

  • When a decision can be expressed completely and reliably as rules, use rules: eligibility thresholds, approval limits, statutory deadlines, access permissions. Modern business rules engines manage thousands of interrelated, versioned rules, and they fail visibly, because when no rule matches the system knows it does not know.
  • A well-built classifier trained on a large, clean labeled history remains a strong, inexpensive, auditable baseline. Its limits are practical: it needs labeled data for each task and must be retrained when categories change.
  • LLMs classify well and handle phrasings and categories they were never trained on (a 2023 PNAS study found ChatGPT outperformed crowd workers on several annotation tasks), but they cost more, run slower, and vary. Answers can differ between identical calls and shift with the order in which options are listed. Their extra capability is justified when a task needs multi-step reasoning, synthesis across long documents, or an explanation a person will read.
  • Decision models are fast, inexpensive per item, and answer many questions at once with explicit uncertainty. They give no rationale, their probabilities need local validation, and they can be misled by adversarial text. TypeSafe's own documentation says JEV reads questions literally, does not count reliably, and is "not a calculator." Do arithmetic, dates, and filtering in code, and use the model only for semantic judgment.
  • Multi-issue requests need a multi-label design. A choice question returns exactly one answer, so a message that says a car has been abandoned on the street and recycling was missed needs one binary question per department, or an issue-separation step before routing. Make the safety question asymmetric: a false alarm costs minutes of staff time, a missed hazard involving children could be serious, so set a low bar for human review.
  • A hybrid for a constituent-services portal: rules handle structured form submissions, a decision model answers binary questions on free-text messages, any safety probability above a low threshold routes to a person, and an LLM separates and summarizes complex messages for staff.
  • Whatever technology produces the judgment, the consequence of an error determines whether a person must make the decision.

Questions I Should Be Able to Ask My Team:

  1. Could this decision be written as rules, and if so, why are we using a model?
  2. Do we already have an accurate classifier for this task, and what exactly would a new model add?
  3. Does this task need one answer or several, and have we designed for requests that belong in two categories?

Technologies / Standards / Companies to Know: Business rules engines, gradient-boosted tree and logistic regression classifiers, LLM structured output, decision models (JEV, OpenAI Decisions, Microsoft-Decision-1, Cloudflare Clef), multi-label classification, model routing.

Recommended Learning:

Time Investment: 2-3 hours

22.4 Vendor Landscape & Independent Evidence on Decision Models

Priority: Monitor

Executive Definition: The evidence base for decision models is weeks old and thin: vendor documentation and claims, three independent preprints that have not completed peer review, and one community benchmark. This topic is a snapshot of what is established and what is open, to be rechecked quarterly rather than mastered now.

Why It Matters: Vendor figures run well ahead of independent evidence. TypeSafe's "193.6x faster and 444.6x cheaper" comes from workflows TypeSafe built and from an LLM wrapper TypeSafe acknowledges is slower and costlier than ordinary LLM use, and TypeSafe itself describes the figures as likely on the high end of real-world gains. Independent studies found large speed and cost advantages over the LLM configurations they tested, but the size of the gap depended on the comparison model, pricing tier, and workload.

What I Need to Understand:

  • Rao and Callison-Burch (University of Pennsylvania): across JEV's 27 comparisons with LLM judges, 8 showed significant differences (4 leads, 4 deficits), 8 were equivalent within a 5-point margin chosen after seeing the results, and 11 were inconclusive. The LLM judges cost 16 to 325 times as much and took 28 to 350 times as long. They also repeated 96.0% of JEV's most confident errors, against 50.3% expected by chance.
  • Denney and DiGiuseppe (Leiden University): accuracy was broadly comparable and varied by task, JEV was faster (0.09 to 0.27 seconds median per decision against 0.75 to 1.04 for GPT-6 Luna), and batch-priced Luna cost less on six of seven tasks. At standard real-time prices JEV was cheaper on six of seven tasks. The authors recommend validating task by task, as with any LLM.
  • The absence of a statistically significant difference is not proof of equal performance. Comparisons that are inconclusive show neither a difference nor equivalence.
  • No peer-reviewed or preprint comparison of JEV and OpenAI Decisions existed as of 23 October 2026. The one reproducible comparison is a single-author community benchmark of 1,000 logic puzzles that code solves perfectly, and its author states it does not establish a general ranking. It should not drive selection for semantic tasks such as routing a complaint.
  • The products differ in maturity. JEV is a startup product in early access with an unpublished deprecation policy (60 days' notice for 1.13 and 30 for 1.14), documented SOC 2 Type II availability, and no documented data residency. OpenAI's endpoint is generally available, documents United States and European regional processing, zero data retention, and HIPAA eligibility for qualifying customers, but documents no snapshot or versioning policy. Neither documents a FedRAMP authorization for this endpoint.
  • Rankings vary by task in the independent literature, and a community leaderboard ranked JEV 1.13 third of 106 systems, statistically close to two open-weight models. Choose workload by workload, on your own data.
  • Open questions: stability across months and version upgrades, accuracy on IT, procurement, and government workloads, measured resistance to adversarial input, multilingual performance, price sustainability after early access (TypeSafe states it cannot prove its pricing is unsubsidized), and calibration on your data.

Questions I Should Be Able to Ask My Team:

  1. For each vendor claim we are relying on, is the evidence independent or from the vendor?
  2. Have we tested on our own data and workload, or are we relying on published benchmarks?
  3. What does our contract say about version changes, deprecation notice, and our right to publish evaluation results?

Technologies / Standards / Companies to Know: TypeSafe AI, OpenAI, Microsoft Foundry, Cloudflare Workers AI, JevBench, equivalence testing, preprints versus peer review.

Recommended Learning:

Time Investment: 1 hour

22.5 Confidence, Calibration & Uncertainty

Priority: Should Understand

Executive Definition: A model is calibrated when its stated probabilities match reality: of all the cases it labels "90% likely," about nine in ten turn out to be correct. A reported 95% does not mean the model is right 95% of the time unless calibration has been verified on data like yours.

Why It Matters: Both JEV and OpenAI Decisions return probabilities and a confidence value, and neither vendor guarantees that those probabilities are calibrated to your data. TypeSafe's confidence is a statistic computed from the returned probabilities (1 when all probability is on one answer, 0 when it is spread evenly), and its documentation tells developers to test thresholds on their own data. Escalation rules built on confidence are only as good as the calibration underneath them.

What I Need to Understand:

  • Three levels of assurance. Raw probabilities are what an API returns: useful for ranking cases by uncertainty, not guarantees. Calibrated probabilities have been adjusted using labeled examples from your own work. Coverage guarantees, produced by conformal prediction, give a set of possible answers (for example, "one of these two departments") that contains the correct answer at a chosen rate such as 95%, provided future cases resemble the labeled sample.
  • Calibration is fragile. Modern neural networks are often overconfident, LLM probabilities can shift with prompt format, and a model's internal probabilities can differ from the confidence it states in words.
  • Confident errors are the ones confidence-gated escalation misses. In every study reviewed, models made confident errors, and JEV's confidence ranked its own errors only modestly on graded tasks (a ranking score of 0.57 to 0.70, where 0.5 is chance).
  • Set thresholds on one portion of the labeled data and confirm them on a separate held-out portion, and recheck after every model or workload change, because guarantees do not survive large shifts in the input.
  • The formal basis for confidence-gated escalation is the classical theory of when a classifier should abstain and the research on deferring to human experts. Prediction-powered inference shows how a small number of human labels can be combined with many model predictions to produce valid estimates.
  • Thresholds should differ by action. A read-only lookup can proceed at modest confidence, while an irreversible or externally visible action should require high confidence, an independent policy check, and in many cases a person. Keep a low bar to escalate safety signals and a higher bar to auto-close.

Questions I Should Be Able to Ask My Team:

  1. Have we verified on our own labeled data that a stated 0.9 means roughly 90% correct?
  2. How were our escalation thresholds set, on what data, and when were they last rechecked?
  3. What share of high-confidence automated decisions does a person review anyway?

Technologies / Standards / Companies to Know: Calibration, conformal prediction, prediction-powered inference, confidence-gated escalation, held-out validation, version pinning.

Recommended Learning:

Time Investment: 2-3 hours

22.6 Hybrid Architecture: Advisory Signals Versus Authorization

Priority: Must Understand

Executive Definition: A hybrid architecture assigns each kind of logic to the component best suited to it, in six layers: systems of record, business rules and policy controls, classifiers and decision models, large language models, workflow orchestration, and human oversight with audit and monitoring. Decision models supply advisory signals; rules and workflow controls hold authority.

Why It Matters: The same model and the same probability carry very different risk as an advisory signal and as an authorization. The architecture should make authorization by the model alone structurally impossible, with thresholds set per action by reversibility and consequence. That keeps the value of fast, cheap judgment while keeping permissions, policy, and accountability outside the model.

What I Need to Understand:

  • Walk one request through the layers (illustrative). An employee asks for 40 design-tool licenses at $38,400. The decision model returns "software subscription" at 0.91, a 0.22 probability of a policy exception, and a 0.97 probability that a quote is attached. The rules engine takes the amount from the form and the category from the model, sees that the amount exceeds the manager's $25,000 limit and that software requires security review, and routes to the manager, IT security, and the controller. An LLM summarizes the quote, humans approve within delegated authority, and the procurement system executes and logs the model outputs and version, the rules applied, and each approval.
  • The critical design choice: the model's category affects which rules apply, but it cannot change an approval limit, skip a reviewer, or issue a purchase order. Credentials to act belong to the workflow layer. A requester who learns to phrase requests to look low-risk can influence the model but cannot bypass the rules.
  • A second model is not an independent control. Independence requires that the checker fail for different reasons, and models with overlapping data and methods often do not. LLM judges from three vendors repeated 96.0% of JEV's most confident errors, and JEV-first cascades improved accuracy over the best single judge by at most 1.5 points on average with thresholds fitted on held-out data, though they often cut cost by half or more.
  • Prompt injection reaches decision models too. TypeSafe documents that injected instructions can change answers, and a guardrail model reading the same text may be misled the same way. Treat input as untrusted and separate instructions from data.
  • Four practices manage the residual risk: sampling (a fixed share of automated decisions reviewed by people regardless of confidence, the only reliable way to find confident errors), monitoring of score distributions, override rates, and outcomes, independent validation on freshly labeled data by a team other than the builders, and escalation rules that send safety, financial, and irreversible actions to people irrespective of confidence.
  • Running a model in production is harder than evaluating one. Sculley and colleagues at Google showed the model is a small part of a production system, and vendor APIs sharpen the problem: if a vendor moves an alias such as jev-latest to a new version, thresholds tuned on the old one can silently become wrong. Pin versions and recalibrate on every change.

Questions I Should Be Able to Ask My Team:

  1. Can any model output directly cause an action in our workflow, with no rule or person in between?
  2. If a requester learned to phrase requests to look low-risk, which control would stop them, and would our sampling detect it?
  3. When the vendor changes a model version, what retests our thresholds before the change reaches production?

Technologies / Standards / Companies to Know: Workflow orchestration, business rules engines, decision logs, version pinning, OWASP Top 10 for LLM Applications, prompt injection.

Recommended Learning:

Time Investment: 2-3 hours

22.7 Governing Machine Judgment: Consequence, Fairness & Accountability

Priority: Must Understand

Executive Definition: Governance of machine judgment starts by separating two categories of use. Low-consequence administrative routing (assigning a ticket to a queue, tagging a document type, choosing which model handles a request) involves errors that are recoverable, usually internal, and detectable, so a model may act autonomously above validated thresholds with sampled review. High-consequence determinations (anything affecting rights, benefits, eligibility, employment, legal status, health, safety, or access to essential services) involve errors that may be irreversible and fall on people with limited recourse, so a probabilistic model should not make them unsupervised.

Why It Matters: For high-consequence uses, a qualified, accountable person must make and own the decision, with reasons the affected person can understand and contest. The line is not always obvious: a misrouted benefits inquiry can delay a payment a family depends on, so classify each use by its worst plausible consequence, not its typical one. No vendor documentation or independent study supports unsupervised use of decision models in high-consequence decisions.

What I Need to Understand:

  • Match transparency to consequence. Cynthia Rudin argues that high-stakes decisions should use inherently interpretable models, because after-the-fact explanations of complex models can be unfaithful to them. Decision models return probabilities and no rationale, which is acceptable for routing an IT ticket and hard to defend when a resident asks why a request was flagged.
  • Accurate models can still produce unfair outcomes. Models trained on historical data reproduce historical patterns, removing protected attributes does not remove proxies such as postal code, language, or name, and when groups differ in underlying rates, common fairness definitions cannot all be satisfied at once. Obermeyer and coauthors found a widely used health algorithm underserved Black patients because it predicted cost as a proxy for need.
  • A router that assigns lower urgency to messages in non-standard English creates unequal service with no protected attribute in sight. Measure accuracy by language and user group before and after deployment, and do not assume human review corrects model bias: Green and Chen found people using algorithmic risk scores did not reliably correct the algorithm's errors.
  • Frameworks and law differ in force. The NIST AI Risk Management Framework (Govern, Map, Measure, Manage) is voluntary, and ISO/IEC 42001 is a certifiable management-system standard. For US federal agencies, OMB Memorandum M-25-21 required minimum practices for high-impact AI with a September 2026 compliance date, but its White House page became unavailable without a formal rescission, so confirm current requirements with OMB and your Chief AI Officer. The EU AI Act treats AI used to evaluate eligibility for essential public benefits as high-risk, and according to legal commentary the Digital Omnibus postponed those obligations to 2 December 2027, which should be confirmed against the Official Journal. Canada's Directive on Automated Decision-Making requires algorithmic impact assessments, and GDPR Article 22 restricts decisions based solely on automated processing with legal or similarly significant effects.
  • Controls that matter: log the input, questions, options, model version, probabilities, threshold, and action for every decision, use zero-retention terms and redact personal data before model calls, set contractual residency terms (or self-host open-weight models where data cannot leave), keep an abstraction layer, a second source, and an exit plan, disclose public uses of AI with an easy way to reach a person, and name decision owners, train reviewers, and track override rates.
  • Public buyers should require the right to evaluate on their own data before commitment, advance notice of model version changes, documented data handling, retention, and residency, security attestations matched to the data classification, exportable decision logs, and freedom to publish independent evaluation results. Contracts should be short, and solicitations should describe the capability (binary, choice, and score questions with probabilities) rather than one vendor's product.

Questions I Should Be Able to Ask My Team:

  1. Which of our uses touch rights, benefits, eligibility, employment, legal status, health, or safety, and who is the named human decision-maker for each?
  2. Do we measure accuracy by language and user group, and what do the numbers show?
  3. If a resident or employee asks why the system flagged their request, who answers, and with what reasons?

Technologies / Standards / Companies to Know: NIST AI RMF and its Generative AI Profile, ISO/IEC 42001, OMB M-25-21, EU AI Act, GDPR Article 22, Canada's Directive on Automated Decision-Making, algorithmic impact assessments.

Recommended Learning:

Time Investment: 2-3 hours

22.8 The Economics of Machine Judgment

Priority: Should Understand

Executive Definition: The relevant question is not what a model call costs but what the whole process costs per correct outcome. A practical frame is Net Annual Value, which equals labor savings plus avoided error costs plus other operational benefits, minus annualized implementation and operating costs (integration, inference, monitoring, labeling, version management, vendor management, and governance).

Why It Matters: In a hypothetical service operation handling one million requests a year, with every figure an assumption chosen for illustration, the annual inference bill at list prices current on 23 October 2026 comes to about $34 for JEV ($0.042 per million input tokens), about $80 for OpenAI Decisions ($0.10), and about $230 for GPT-6 Luna through a generative API. Whole-process totals for the three are about $1.69 million each, against $3.08 million for manual handling. The inference line is less than two-hundredths of one percent of the total, so the choice of vendor is not where the money is.

What I Need to Understand:

  • Coverage and accuracy dominate. Human handling and error costs are about four orders of magnitude larger than inference, so safely automating 80% of requests instead of 75%, or cutting misroutes by one percentage point, outweighs any price difference among vendors.
  • In the example, a hybrid (rules resolving 40% of requests at 97% accuracy, a decision model resolving 42% more at 94%, and people with LLM summaries handling the rest) produced a net annual value of about $1.82 million, against about $1.39 million for a single decision model. That result depends on the assumptions and is not universal.
  • The ranking changes quickly. When errors are expensive, a model less accurate than the people it replaces (90% against 92%) destroys most of the value and can destroy all of it: at $60 per error and 250,000 requests a year it produced a loss of about $97,000. At 250,000 requests a year a single well-performing decision model can beat the hybrid, because the hybrid's higher fixed costs are spread over less work, and at 50,000 requests fixed costs of integration, monitoring, and governance can exceed the savings for every approach. Compute your own crossover points.
  • Estimate labor savings from time studies and workflow timestamps, not vendor estimates, net of the human time that remains for escalated cases and quality sampling. Estimate avoided error costs separately for each error type, using incident records and rework data.
  • Do not deploy a decision model, even if inference is cheap, when the decision can be fully specified as rules, fixed costs exceed achievable savings, a stable well-labeled classifier already performs well, errors are severe and hard to detect, no labeled data exists to set thresholds, affected people are entitled to reasons, or the work can be batch processed (batch-priced LLMs may cost the same or less).
  • Inexpensive inference does not mean inexpensive operations. Budget for labeling, evaluation, monitoring, version management, and incident response for at least three years.

Questions I Should Be Able to Ask My Team:

  1. What are our measured, not vendor-estimated, handling time, error rate, and cost per error type today?
  2. At our volume, do the fixed costs of integration, monitoring, and governance exceed what we can save?
  3. Is this work latency-sensitive, or can it run as an overnight batch at lower prices?

Technologies / Standards / Companies to Know: Net Annual Value, total cost of ownership, token-based pricing (input versus output tokens), batch pricing, model routing.

Recommended Learning:

Time Investment: 1 hour

22.9 Where Decision Models Fit: Six Use Patterns

Priority: Should Understand

Executive Definition: Decision models fit best where the input is messy, the answer is bounded, the volume is high, and mistakes can be caught and corrected. In each of six hypothetical patterns below, the model's job is to describe the situation and the workflow does the work.

Why It Matters: These are application patterns, not verified deployments. The research behind this domain found no independently documented production deployment of JEV or OpenAI Decisions in these settings as of 23 October 2026, so treat each as a design to test.

What I Need to Understand:

  • IT service management: answer several questions per ticket at once (assignment group, urgency, possible security incident, duplicate of an open major incident). Use asymmetric thresholds, a low bar to escalate security signals and a higher bar to auto-close, and measure reassignment rate, time to correct assignment, and missed-priority rate.
  • Procurement and financial approvals: classify spend category, flag likely policy exceptions, and check that required documents are present. Approval limits, delegation of authority, sanctioned-vendor checks, and segregation of duties stay deterministic, and unflagged requests are sampled to catch requesters who learn to phrase requests to avoid scrutiny.
  • Government constituent services: use one binary question per department, binary safety and vulnerability questions, and an urgency score. Any safety or vulnerability signal goes to a person regardless of confidence, non-English requests are reviewed until subgroup accuracy is demonstrated, and the public is told AI is used and can reach a person easily. Evidence on accuracy across languages and dialects is thin.
  • Document and case intake: ask which document type this is and whether the signature page is present, without retraining. Image input is generally available in OpenAI's API and only in preview for JEV. Anything that feeds an eligibility or enforcement determination gets human review, and decision logs need a retention schedule.
  • Enterprise AI model routing: ask whether a request is answerable from the FAQ, needs multi-step reasoning, or contains personal data, and use the answers to pick deterministic logic, a small model, or a frontier model. The risk is invisible quality loss, a plausible but poor answer from a weak model that no one notices, so enforce data-classification rules before any model sees the request and sample quality by route. Both vendors emphasize this use.
  • AI agent oversight: a fast second opinion on a proposed action (does it match the user's request, does it contain personal data, was it induced by injected instructions). Permissions, scopes, spending limits, and allow-lists define what an agent may do and cannot be delegated to a model, and irreversible, financial, or externally visible actions need human approval. Keep model-based guardrails for agents experimental until red-team results support them.

Questions I Should Be Able to Ask My Team:

  1. Which of our high-volume workflows have messy input, bounded answers, and mistakes we can catch and correct?
  2. For each candidate, what is the worst plausible consequence of a wrong answer?
  3. Which of our controls, such as permissions, limits, and deadlines, must stay deterministic regardless of what any model says?

Technologies / Standards / Companies to Know: Multi-label classification, document intake, AI gateways and model routing, agent guardrails, Microsoft-Decision-1, Cloudflare Clef.

Recommended Learning:

Time Investment: 1 hour

22.10 Running a 90-Day Evaluation: Gates, Evidence & Executive Decisions

Priority: Must Understand

Executive Definition: A 90-day comparative evaluation tests decision models against the alternatives on one or two high-volume, bounded, low-consequence, measurable workflows, measuring the whole process rather than the model alone, and ends in explicit decision gates.

Why It Matters: The evidence is early, the products are weeks old, and vendor claims outpace independent findings. The only dependable way to know whether a decision model helps is to test it against rules, your existing classifier, and an LLM on your own data, then shadow it against live work before it acts on anything.

What I Need to Understand:

  • Days 1 to 30, identify and baseline: choose one or two use cases such as ticket routing or document-type classification, and exclude eligibility, enforcement, and employment decisions. Classify each use's consequence level, document the current process (handling times, error types, the cost of each error type, who holds decision authority), and assemble 1,000 to 2,000 historical cases with verified answers. Where labels are unreliable, have experts relabel a sample and measure how often they agree, which sets a realistic accuracy ceiling. Write the questions with observable criteria, distinct options, an "other" option, one concern per question, and separate binary questions for multi-label cases.
  • Days 31 to 60, compare technologies: benchmark rules, a traditional classifier where labeled data permits, JEV, OpenAI Decisions, and prompted LLM classification on identical data, adding an open-weight decision model where self-hosting matters, and pin and record every model version. Measure accuracy, errors by type, calibration, stability across repeated runs, sensitivity to option order and wording, latency at expected concurrency, and cost. Stress-test with other languages, unusual writing styles, long inputs, multi-issue cases, and adversarial input including prompt injection. Set thresholds on one portion of the data and confirm them on a held-out portion, and test error overlap if any design relies on one model checking another.
  • Days 61 to 90, assess production readiness: run in shadow mode alongside the live process without acting on outputs, and compare recommendations with human decisions on live work. Build authentication, logging, retries, rate limits, and fallback to rules or people when the service is down. Put decision logs, monitoring, an incident playbook, records retention, privacy review, and contract terms in place, and recompute Net Annual Value from measured figures.
  • Day-90 gates, each with a proceed and a stop condition: business value (positive measured Net Annual Value with a margin), accuracy on automated cases at or above current human accuracy with high-cost error types meeting agreed floors, calibration (observed accuracy close to stated confidence), security and privacy (injection testing within tolerance, data handling approved), governance (logging, monitoring, rollback, named owners, acceptable subgroup accuracy), vendor readiness (version pinning, change notice, an exit path), human oversight (escalation staffed, reviewers trained, override rates tracked), and a funded three-year total cost of ownership.
  • Executive posture. Approve now: the 90-day evaluation, an internal abstraction layer over the common question types, logging and monitoring standards for model-driven routing, and contract requirements for version notice, data handling, and the right to publish evaluations. Keep experimental: model-based guardrails for AI agents, cascades that rely on one model to check another, image-based decision tasks, multilingual routing until subgroup accuracy is demonstrated, and any commitment longer than one year to a single decision-model vendor. Do not automate without additional safeguards: decisions affecting rights, benefits, eligibility, employment, legal status, health, or safety, irreversible or externally visible actions, and any use where a model's output would directly authorize action.
  • The organizations that benefit most will not be those that pick the "best" decision model. They will know which of their decisions are rules, which are predictions, which are judgments, and which belong to people, and build systems honest about the difference.

Questions I Should Be Able to Ask My Team:

  1. How many verified historical cases do we have for this workflow, and how often do our own experts agree on the labels?
  2. Will the candidate run in shadow mode against live work before it is allowed to act?
  3. Which day-90 gate is a hard stop if it fails, and who has the authority to call it?

Technologies / Standards / Companies to Know: Shadow testing, held-out validation, version pinning, abstraction layer over question types, NIST AI RMF (Map function), inter-rater agreement.

Recommended Learning:

Time Investment: Half day