Evaluation & Operations
MDX
Data Privacy for Agentic AI Systems
Minimize sensitive context, control retention, isolate tenants, and make agent data flows understandable.
Editorial review: clarity, operational relevance, safety boundaries, and source quality.
Map the data journey
Document what enters prompts, which tools receive it, what traces store, and which providers process it. Agent loops can copy data across more surfaces than a conventional request.
Create a data-flow map for prompts, retrieval indexes, tools, caches, traces, human review queues, and model providers. Mark purpose, lawful basis, geography, retention, and recipients. Agentic systems deserve special attention because a single task can copy one record across several otherwise separate services.
Minimize by default
Retrieve only the fields required for the current decision, redact secrets before model calls, and avoid placing durable identifiers in free-form context when scoped references will work.
Minimize at query time. Retrieve only required columns, use opaque references instead of durable identifiers, redact before model calls, and avoid placing secrets in prompts for convenience. Apply the same controls to test fixtures and evaluation datasets, which are often less protected than production while containing realistic data.
Give memory a lifecycle
Define why information is saved, who may retrieve it, how users correct it, and when it is deleted. Personalization should not become indefinite invisible retention.
Memory needs explicit consent and lifecycle controls. Distinguish user-stated facts from model inferences, expose saved items, support correction and deletion, and prevent retrieval across tenants or purposes. Deletion must cover derived indexes and caches, not only the primary row.
Practical example
A privacy-aware benefits assistant
The agent receives an employee-scoped token and retrieves only eligibility fields needed for the question. Medical attachments are processed in an isolated service that returns a narrow status rather than raw contents. Traces store redacted event codes. No health information becomes personalization memory, and deletion propagates to the retrieval index and review queue.
Field checklist
Apply it in practice
- Map every processor and retained copy in the task path.
- Retrieve and disclose only fields needed for the current decision.
- Apply privacy controls to traces and evaluation data.
- Support correction, expiry, and complete deletion of memory.
Decision framework
Questions to answer before you build
Agent privacy requires controlling an entire data journey through prompts, tools, retrieval, traces, evaluations, human review, memory, and external providers.
Why is each field needed?
Retrieve only the columns required for the current decision, use opaque references, and redact before data enters a model or trace.
Where are derived copies stored?
Include indexes, caches, test fixtures, review queues, logs, and provider retention in data maps and deletion procedures.
Should this become memory?
Require a defined personalization purpose, distinguish user statements from inferences, and support inspection, correction, expiry, and deletion.
Common failure signals
Watch for these warning signs
- Protecting the database while prompts and traces copy the same data elsewhere.
- Using realistic production data in evaluations without equivalent controls.
- Retaining inferred personal attributes indefinitely because they might be useful later.
Selected primary references
Continue with the source material
These sources inform the wider editorial perspective for this topic. They are not presented as line-by-line citations for every statement.
- NISTAI Risk Management Framework ↗A voluntary framework for governing, mapping, measuring, and managing AI risk.
- OWASPOWASP Top 10 for LLM Applications 2025 ↗A practical catalogue of common security risks in LLM-enabled applications.
- AnthropicDemystifying evals for AI agents ↗Evaluation strategies for multi-step, tool-using agent systems.