Foundations
MDX
The Role of Models in Agent Systems
What the language model contributes—and what must be handled by the surrounding application.
Editorial review: clarity, operational relevance, safety boundaries, and source quality.
A flexible decision engine
The model interprets goals, works with ambiguous language, selects among actions, and synthesizes results. It is best treated as a probabilistic decision component rather than the entire system.
Models are particularly useful at semantic boundaries: translating a vague request into a task, mapping messy text to a known category, and selecting an action when rules would be brittle. They are poor substitutes for authoritative storage, exact calculations, access control, or guarantees about whether an external change occurred.
The application owns control
Ordinary software should enforce permissions, schemas, budgets, retries, timeouts, and irreversible-action checks. These controls should not depend on the model remembering a sentence in its prompt.
A robust runtime assumes every model output can be wrong. It parses structured output, rejects invalid arguments, rechecks resource access, applies idempotency controls, and verifies postconditions after a write. Prompts can explain these rules, but enforcement belongs at the point where an action becomes real.
Match the model to the step
A single agent may use a capable model for planning and a smaller model for extraction or classification. Routing work by difficulty can reduce cost and latency without weakening the steps that need stronger reasoning.
Routing should follow measured task difficulty rather than a broad belief that larger is always better. Build an evaluation set for each step, establish quality and latency targets, and route only when a cheaper model meets them. Include handoffs and retries in the comparison because routing overhead can erase per-call savings.
Practical example
Different models in one claims workflow
A small model extracts policy numbers and document types. A stronger model compares contradictory descriptions and proposes what evidence is missing. Deterministic code calculates coverage amounts and verifies customer access. The final response is assembled from typed results. Each component does the kind of work it can be tested against, and no model is trusted to invent financial state.
Field checklist
Apply it in practice
- Identify which steps require semantic judgment.
- Move exact rules and authoritative state into code.
- Evaluate models per step, including retries and handoffs.
- Verify external effects independently of model narration.
Decision framework
Questions to answer before you build
Use models as flexible semantic decision engines, not as databases, permission systems, calculators, or proof that an external effect occurred.
Does this step require semantic judgment?
Models add value when interpreting ambiguous intent, comparing messy evidence, or choosing among safe actions. Exact transformations belong in code.
What must be independently checked?
Revalidate identifiers, ownership, numerical results, policy rules, and postconditions outside the model before they influence real state.
Which model meets this step's target?
Evaluate each step with representative cases. Route to a cheaper model only when its errors do not dominate the completed task.
Common failure signals
Watch for these warning signs
- Using model confidence as a substitute for evidence or authorization.
- Sending every step to the strongest model without measuring marginal value.
- Assuming structured output makes the contents factually or semantically valid.
Selected primary references
Continue with the source material
These sources inform the wider editorial perspective for this topic. They are not presented as line-by-line citations for every statement.