Evaluation & Operations
Building Golden Datasets for Agent Evaluation
Create representative tasks, expected outcomes, and edge cases that reveal meaningful regressions.
Start from real work
Collect anonymized examples that reflect common, difficult, ambiguous, and high-risk tasks. Synthetic cases are useful for coverage but can miss the messiness of actual users.
Define acceptable outcomes
Many agent tasks have more than one valid path. Specify required facts, prohibited actions, evidence standards, and success conditions instead of one exact output string.
Keep the set alive
Add cases from production incidents and human corrections. Maintain a stable core for trend comparison while rotating a private holdout set to reduce overfitting.