AI and LLM Security
Model Output Handling Security
We check if model output runs without validation in a browser, database query or other system. We recommend controls that treat the model as an untrusted user.
What it is
Model output is text that anyone who can influence the model can indirectly write. If shown on a web page, added to a query or run as a command without validation, classic vulnerabilities arise. Cross-site scripting (XSS) and SQL injection are examples. OWASP defines this risk as Improper Output Handling.
Scope
- Rendering answers as HTML or Markdown in the interface
- Output passed into database queries or API calls
- Environments where model-generated code or commands run
- Links and images in answers that send requests to external addresses
- Output forwarded to other systems as emails, documents or support tickets
The need
When organizations need it
- When the assistant's answers are displayed as rich text in a web interface.
- When user questions in natural language are converted to SQL queries by the model.
- When code or scripts written by the model are automatically run by an agent.
Method
How we work
-
Output paths
We list every interface, query and system that model output reaches.
-
Impact testing
We try to prompt the model to produce harmful content on these paths and check if it executes.
-
Controls
We recommend layers such as output encoding, parameterized queries, allowlists and Content Security Policy (CSP).
Outcome
What you have at the end
You see where model output can run as code, query or request. For each point, you have a validation control independent of the model.
Frequently asked questions
Is it enough to tell the model not to produce harmful output?
No. The model can be steered and does not follow instructions the same way every time. Output should be validated and encoded like user input.
How is this different from a classic application test?
The vulnerability types are the same, but the harmful input is produced by the model. The model must first be prompted to generate it. Both tests can be run together.
AI and LLM Security
Other services in this area
Let's talk about this service.
Tell us briefly what you need. We will agree on the scope and timing together in the first meeting.
Schedule a Meeting