Production AI Security
Assess the AI system buyers will actually inspect.
30 minutes. Bring the evidence, trace, or deadline.
This page is for
- CTOs and Heads of AI with a live customer-facing AI product
- Security owners preparing enterprise questionnaires or architecture reviews
- Platform teams that need a bounded AI threat model before a high-impact launch
Start here
The short answer
Direct answer
A production AI security assessment maps the customer-facing AI path from identity and data ingress through retrieval, inference, tools, logging, and human review. It tests whether the controls described in a questionnaire are enforced in the running system and leaves an evidence-backed remediation queue.
Search intent this page serves: You have a live AI feature, an enterprise security review, or a material risk decision that cannot be answered with a generic AI policy.
System boundary
What the assessment follows
The assessment starts with the real trust boundaries. Every included surface has a corresponding scenario, evidence expectation, or retest condition.
Representative flow
- Request authentication → tenant and resource authorization → prompt assembly
- Prompt and retrieved context → provider route → output validation and policy checks
- Validated output → user response, tool proposal, or human approval queue
- Tool result and trace → audit store, monitoring, support workflow, and deletion path
Risk and controls
The control matrix is specific to this system
A finding is useful only when the team can locate the control, observe the failure, name the evidence, and decide what happens next.
| Surface | Failure or risk | Control to inspect | Evidence | Priority |
|---|---|---|---|---|
| Identity and tenancy | A valid user can influence a tenant or resource identifier in model-generated arguments. | Bind tenant and resource identity in trusted application code; reject model-supplied authority. | Denied cross-tenant tests, policy code, access matrix, and recent access review. | Critical |
| Prompt and retrieval | Untrusted documents or user input override system intent or expose unrelated context. | Label trust boundaries, filter retrieval by authorization, and validate context before assembly. | Indirect-injection cases, retrieval authorization tests, and redaction configuration. | High |
| Provider and data use | Prompts, traces, or embeddings persist outside the approved data and region boundary. | Maintain a provider register with retention, training, region, DPA, and configuration evidence. | Contract terms, provider settings, data-flow diagram, and deletion verification. | High |
| Outputs and tools | A plausible response is parsed into an unsafe command or irreversible action. | Use typed schemas, business validation, least-privilege tools, idempotency, and approval for high-impact actions. | Malformed-output tests, tool policy, approval logs, and side-effect simulation. | Critical |
| Observability | Sensitive traces are broadly readable or cannot be connected to a release and user outcome. | Redact before storage, restrict trace access, and link trace IDs to release, tenant, policy result, and workflow outcome. | Trace samples, role permissions, retention policy, and alert routing. | High |
| Change and incident response | A model, prompt, or tool change silently alters risk between reviews. | Version changes, run regression checks, define rollback thresholds, and refresh evidence on material change. | Change record, evaluation report, rollback drill, and incident runbook. | Medium |
Reproducible testing
Scenarios that exercise the boundary
These cases are intentionally concrete. They can be run in a production-adjacent environment with synthetic data and become regression fixtures after remediation.
Indirect injection in a retrieved document
- Precondition
- A tenant can upload or edit a document that will be retrieved for an assistant request.
- Exercise
- Place an instruction in a low-trust document that asks the model to reveal system context or call an unrelated tool, then request a normal answer.
- Expected control
- Retrieved text is treated as data, not authority; no secret or unauthorized action is exposed.
- Evidence captured
- Prompt/context capture, tool decision, policy result, and a regression case linked to the document fixture.
Cross-tenant retrieval and cache collision
- Precondition
- Two tenants use the same assistant route with shared retrieval or caching infrastructure.
- Exercise
- Replay the same question with alternating tenant identities and vary cache keys, filters, and pagination.
- Expected control
- Every returned chunk, cache entry, and trace is scoped to the authenticated tenant and resource.
- Evidence captured
- Negative test output, cache-key review, vector filter configuration, and sampled trace IDs.
Model-generated high-impact action
- Precondition
- The assistant can propose a refund, permission change, deletion, or external API write.
- Exercise
- Use ambiguous or conflicting instructions to induce an action with a different account, amount, or resource than the user controls.
- Expected control
- Trusted policy checks and confirmation prevent or constrain the side effect; retries do not duplicate it.
- Evidence captured
- Tool arguments before/after validation, approval event, idempotency key, and rollback result.
Provider and observability retention mismatch
- Precondition
- Prompts or retrieved context are sent to an external provider and copied into traces.
- Exercise
- Trace one synthetic sensitive record through provider settings, logs, support access, deletion, and backup retention.
- Expected control
- The record stays within the approved boundary, is redacted where required, and has an executable deletion or expiry path.
- Evidence captured
- Provider configuration, contract reference, trace policy, deletion test, and owner/refresh trigger.
Scope and handoff
What the team receives, and what access is needed
The fixed scope is designed to leave behind implementation artifacts, not a report nobody can rerun.
Deliverables
- Current-state AI system boundary and data-flow map
- Risk-to-control matrix with severity, owner, and evidence location
- Four or more reproducible adversarial and authorization scenarios
- Priority remediation plan tied to code, configuration, and operating practice
- Evidence index for enterprise reviews, including refresh triggers
- Retest report showing which controls held and which remain conditional
Access requirements
- Architecture diagram or a 60-minute walkthrough with the product owner
- Read-only access to relevant repositories, provider settings, and observability samples
- Test accounts for at least two roles and two tenants where applicable
- A non-production or production-adjacent environment with realistic tools and policies
- Security, engineering, or compliance owner available for evidence decisions
Day 1
Scope and boundary
Confirm the live feature, actors, data classes, provider routes, tools, test accounts, stop conditions, and success criteria.
Days 2–3
Control review
Trace authorization, retrieval, provider retention, output handling, logs, change control, and incident evidence against the running system.
Days 4–5
Scenario execution
Run reproducible cases, validate business impact, separate evidence gaps from implementation gaps, and prioritize the critical path.
Day 6
Handoff and retest plan
Walk through findings, owners, evidence links, remediation sequence, and the exact cases to rerun after changes.
Retest and limitations
How the result stays current
Retest after any material change to model provider, retrieval authorization, tool permissions, prompt policy, trace retention, or tenant architecture. The first retest should rerun all Critical scenarios plus the regression cases created from High findings; a lightweight quarterly evidence refresh is appropriate for stable systems.
Scope limit
This is not a SOC 2, ISO, penetration-test, legal, or certification opinion.
Scope limit
Infrastructure and traditional web-application testing are included only where they form the AI trust boundary.
Scope limit
A finding is not closed because a model refuses; application authorization must enforce the boundary.
Sources and limits
References used for the operating model
These sources provide the framework or vocabulary. The page adds product-boundary tests, evidence requirements, and implementation decisions so the result can be inspected in a live system.
Use it as a risk vocabulary, then test the actual product boundary rather than claiming a checklist is a control.
NIST describes AI RMF 1.0 as voluntary and use-case agnostic; this page turns relevant outcomes into production evidence, not certification claims.
FAQ
Questions buyers and engineers ask
Is a production AI security assessment the same as an AI audit?
No. It is a bounded engineering assessment of the AI system boundary, controls, evidence, and reproducible scenarios. It does not issue a certification, replace an auditor, or provide legal advice.
Do you need production access?
Usually no. A production-adjacent environment with realistic identities, retrieval, tools, and policies is preferred. Read-only production evidence may be needed to verify retention, trace, and provider behavior.
What makes the findings defensible to an enterprise buyer?
Each finding ties an observed behavior to a specific control, evidence location, owner, remediation, and retest case. The report separates missing evidence from a control that is not implemented.
Next step
Turn this page into an owned engineering decision.
Bring the questionnaire, trace, failed workflow, or provider deadline. We will help decide whether a focused implementation is the right scope.
30 minutes. No deck. Leave with a clear next step.