Building Golden Datasets for Agent Evaluation

Collect anonymized examples that reflect common, difficult, ambiguous, and high-risk tasks. Synthetic cases are useful for coverage but can miss the messiness of actual users.

Sample the real task distribution, then deliberately oversample rare high-consequence cases. Preserve the user's language, missing context, tool conditions, and environmental state while removing sensitive information. A bare prompt without the surrounding records is not a reproducible agent test.

Many agent tasks have more than one valid path. Specify required facts, prohibited actions, evidence standards, and success conditions instead of one exact output string.

Represent acceptable behavior as constraints and outcome properties. Record required facts, allowed and forbidden actions, maximum attempts, evidence expectations, and valid escalation reasons. When multiple trajectories work, do not force one reference sequence; doing so rewards imitation instead of task success.

Add cases from production incidents and human corrections. Maintain a stable core for trend comparison while rotating a private holdout set to reduce overfitting.

Version each case with its fixtures and rubric. Review additions from incidents to ensure they test a general failure rather than one accidental wording. Maintain a hidden holdout, measure inter-reviewer agreement, and periodically retire cases whose product behavior or policies are no longer current.