Build Your First Tool-Using Agent

Do not begin with “build a support agent” or “automate research.” Those labels hide many users, policies, systems, and definitions of success. Pick one job that can be described and tested independently.

A strong first job has:

• one authenticated user or actor;
• one observable outcome;
• one authoritative source of truth;
• one or two read-only tools;
• explicit unsupported cases;
• an independent way to verify completion.

“Explain the current state of an order to the authenticated customer” is a reasonable pilot. “Handle customer service” is not. The narrow version can succeed by reading one current record and translating typed status into a supported explanation. It does not need refund authority, outbound messaging, memory, multi-agent coordination, or an open-ended plan.

Build fixture cases before the loop. Include normal records, missing identifiers, multiple matches, denied access, stale state, a tool timeout, misleading user claims, and requests for adjacent actions. A dozen representative cases are more useful than one polished demonstration.

The task contract is the boundary between product intent and system behavior. It should be understandable by product, engineering, operations, and security—not only by the prompt author.

Specify:

1. Supported users and inputs.
2. The system that owns each required fact.
3. Actions the model may propose.
4. Actions the runtime may execute.
5. Evidence required for completion.
6. Valid terminal states and escalation routes.
7. Step, time, token, and tool budgets.
8. Explicit exclusions and prohibited effects.
9. Owners for quality and incidents.

Keep the contract outside the prompt as a versioned product artifact. The prompt may explain the job, but the runtime must enforce its permissions and limits.

The first tool should be boring. Give it one unmistakable purpose, typed inputs, representative fixture data, and explicit result states. Keep credentials and authorization in the host application.

For an order-status agent, prefer:

type FindOrderResult =
  | { status: "found"; order: OrderSummary; observedAt: string }
  | { status: "not_found" }
  | { status: "denied"; reason: "wrong_account" }
  | { status: "unavailable"; retryAfterMs?: number };

async function findCustomerOrder(
  actorId: string,
  orderId: string,
): Promise<FindOrderResult>;

Avoid returning arbitrary database rows, generic exceptions, or a large prose dump. The next model decision needs the outcome, relevant fields, source time, and safe recovery options.

Schema validation is only the first boundary. After parsing, confirm that the order exists, belongs to the authenticated actor, and is current enough for the claim being made. Treat model-generated arguments as untrusted input even when they satisfy the type.

Persist a run record containing the goal, confirmed facts, open questions, tool events, step budget, and terminal state. Each model call receives only the state required for the next decision.

The model may return either a proposed tool action or a candidate final answer. The host then:

• checks the proposal against the allowlist;
• validates arguments and cross-field rules;
• authorizes the actor, task, resource, and action;
• executes with a timeout;
• records a structured observation;
• verifies any completion claim;
• stops at the configured budget.

Separate model failure from tool failure. A not_found result can be a healthy observation. A timeout may permit a bounded retry. A permission denial should terminate or escalate rather than encourage the model to search for another route. An indeterminate write must be reconciled before retrying—but the first pilot should avoid writes entirely.

Final-answer grading cannot tell you whether the system used the wrong account, attempted a prohibited action, repeated a tool unnecessarily, or invented success after a timeout. Record and grade the trajectory.

For each case, assert:

• the terminal state is valid;
• required evidence is present;
• only allowed tools were requested;
• tool arguments are semantically correct;
• denied and unavailable results produce the right recovery;
• step and retry budgets are respected;
• unsupported requests are refused or handed off cleanly;
• the final explanation is faithful to tool evidence.

Run the same set after changing the model, prompt, tool schema, policy, or retrieval source. Add generalized cases from incidents and reviewer corrections. Keep a small hidden holdout so repeated prompt tuning does not overfit the visible examples.