A feature that answers questions about documents can retrieve relevant text and send it to an AI model. That text, however, is not always harmless: a document, web page, or user-provided file may contain phrases that try to change the task, obtain information they are not authorized to access, or trigger an action. This risk is known as document prompt injection, or indirect injection, because the instructions arrive inside the content being processed and not necessarily in the user's question.
Mitigation does not mean finding the perfect phrase for the system message. It means designing the application so that retrieved content cannot change permissions, policies, or business decisions, and ensuring that a model response is never sufficient authorization to act.
What risk needs to be controlled
A document may contain explicit instructions such as “ignore previous instructions,” or more deceptive text, such as a request to reveal internal data. It may also include hidden instructions in content extracted from HTML, comments, metadata, or OCR text. If that material is mixed indiscriminately with the application's instructions, the model could follow some of it, summarize it in a biased way, or generate an unintended response.
The risk depends on what the feature can do. An assistant that only drafts a response for review has a different impact from one that can query accounts, issue refunds, or send messages. Before exposing the feature to users, define what information it can receive, what responses it can produce, and what consequences may follow from them.
Injection does not by itself prove that the application's access controls have been breached. It can, however, expose information that was improperly included in the context or induce the model to request an operation. Protection must cover the input, the available data, and subsequent actions.
Separate data from policies and permissions
Retrieved content should provide evidence for an answer, not become an authority. A document may describe a company policy, but the application must separately determine whether the user is authorized to view that document or perform a related operation. That decision is not delegated to the text or the generated response.
- Authorize before retrieval: enforce access controls in search and filter documents according to the user's identity and current permissions.
- Minimum context: send only the snippets needed for the task. Do not include data from other accounts, secrets, credentials, or information the model does not need.
- Policies outside the model: calculate permissions, limits, and business conditions in application code or authorized services.
- Response as a proposal: treat the model's output as content to validate, not as an executable instruction.
A system instruction can help express the expected behavior, for example by stating that retrieved text is reference material. It is not a security boundary: it does not guarantee that the model will ignore malicious instructions, nor does it replace access controls, validation, or technical restrictions.
Design the PHP workflow with explicit boundaries
In PHP, it is useful to separate authorization, retrieval, context construction, the model call, and output validation into distinct stages. This separation makes each boundary easier to test and prevents an AI result from connecting directly to a sensitive operation.
The following snippet is illustrative pseudocode, not an executable call to a specific client. The way the request is built and the accepted fields depend on the API and SDK in use:
$user = $auth->currentUser();
$documents = $repository->searchAuthorized($user, $question);
$snippets = $contextBuilder->selectRelevant($documents, $maxItems);
// Pseudocode: adapt the request to the specific API.
$response = $aiClient->answer([
'task' => 'Answer using the reference material; do not authorize actions.',
'untrusted_context' => $snippets,
'question' => $question,
]);
$answer = $responseValidator->validate($response);
return $answer;The names in the example are illustrative. Labeling the context as untrusted helps communicate its role, but does not isolate it reliably: neither the format nor the instructions alone prevent the model from following an embedded instruction. Effective separation also depends on the data sent, permissions enforced outside the model, and the capabilities available.
Normalize and limit extracted content before incorporating it: set maximum sizes, remove formats you do not need, and handle text obtained through OCR or from external pages carefully. Avoid concatenating documents without limits or including the full conversation history by default. Keep a reference to the source of each snippet if you need to explain or audit the response, without including its sensitive content in logs.
Reduce the impact of a manipulated response
Limit the tools available to the model to those needed for the specific feature. If the task only requires answering questions, do not grant it write tools or general access to internal services. If it needs to query data, expose a narrowly scoped operation that enforces authorization on every request, rather than broad credentials or arbitrary database access.
When the model proposes an action, the application must validate it in an independent workflow. Check the identity, permissions, parameters, business rules, and the resource's current conditions at execution time. For high-impact actions, require explicit confirmation or human review. A textual response from the model, even if it seems safe or cites a document, does not replace that authorization.
Also validate the output before displaying or processing it: check the expected format, limit its length, and reject fields or values outside the allowed set. If a specific structure is expected, parse and validate its fields using deterministic rules. Output escaping and defenses against HTML, SQL, or command injection are still necessary; AI generation does not make the resulting text trustworthy.
Test attacks and expected behavior
Tests should cover the entire workflow, not just whether the model responds correctly to a system instruction. Prepare benign and adversarial documents that try to alter the task, request unauthorized data, or induce an action. Include variations in source format, such as text extracted from HTML, long documents, and content that mixes instructions with legitimate information.
- Verify that search never returns documents outside the user's permissions, even when the question explicitly requests that data.
- Verify that adversarial documents do not enable tools or change business rules.
- Confirm that malformed or unexpected responses are rejected and that sensitive operations require independent authorization.
- Evaluate useful and correct responses to benign documents to detect whether restrictions degrade the feature unacceptably.
- Repeat the tests whenever the model, instruction template, retriever, tools, or extraction process changes.
Define observable acceptance criteria: what data must never appear, what actions must remain blocked, and how the system should respond when there is insufficient evidence. A single test does not guarantee security, but a repeatable test suite helps detect regressions and compare configuration changes.
Log signals without accumulating sensitive data
To investigate failures, log request identifiers, configuration versions, identifiers or internal references for the documents consulted, authorization decisions, invoked tools, and validation results. Logging a reference does not mean storing the document or its snippets. Avoid storing sensitive snippets unless there is a specific need and dedicated access, protection, and retention controls.
Add metrics for rejections, format errors, and provider failures to distinguish security issues from operational problems. Avoid automatically storing full prompts, documents, and responses if they contain personal data, trade secrets, or regulated information. Define retention, access, and redaction for sensitive data; when sufficient, log hashes, identifiers, and structured events. Ensure logs do not become a new route for exposing the content the feature was meant to protect.
Checklist before enabling the feature
- Are permissions checked before each document is retrieved?
- Is the context sent minimal, bounded, and treated as untrusted input?
- Are policies and business rules enforced outside the model?
- Are tools limited, and is authorization checked again for every operation?
- Are responses validated before being displayed or turned into actions?
- Is human review or confirmation required for high-impact decisions?
- Are there repeatable adversarial tests and logs with controlled retention?
If any answer is no, limit the feature's scope or keep actions in proposal mode until that gap is closed. Defense against document prompt injection does not depend on the model always interpreting content correctly: it depends on access and operational boundaries continuing to work even when the model makes a mistake.
