AI and LLM Security
AI Threat Modeling
Before your AI system goes live, we identify potential abuse scenarios. We define permission, approval and monitoring controls with you at the design stage.
What it is
Threat modeling examines how a system could be abused before it goes live. Traditional methods assume data and instructions are separate, but any content an AI model reads may contain instructions. We set trust boundaries based on content sources and tools. Scenarios are built using the OWASP Top 10 for LLMs and MITRE ATLAS.
Scope
- Data flow diagram and trust boundaries
- List of content sources the model reads and tools it calls
- Abuse cases and possible impacts
- Requirements for permissions, human approval and output validation
- Requirements for monitoring, logging and incident response
- Security test plan before go-live
The need
When organizations need it
- When the AI project is still at the design stage and architectural changes are still feasible.
- When an ISO/IEC 42001 AI management system is being established and its risk assessment requires technical input.
- When the system will be used in an area that may be considered high-risk under the EU AI Act.
Method
How we work
-
Architecture review
We map the data flow from design documents and interviews with your team.
-
Scenarios
For each trust boundary, we document who could abuse what and how.
-
Prioritization
We rank scenarios by impact and likelihood.
-
Requirements
For each significant scenario, we define the control to implement and the test to verify it.
Outcome
What you have at the end
Before the system goes live, key abuse cases and the controls addressing them are defined. Developers know the requirements to implement, and testers know the scenarios to validate.
Frequently asked questions
Can threat modeling be done if the system is already live?
Yes. In this case, the threat model is derived from the existing architecture. The result also helps define the scope of a security test.
Who needs to take part?
The system architect, product owner and a developer are usually sufficient. If personal data is processed, including someone from legal or compliance is helpful.
AI and LLM Security
Other services in this area
Let's talk about this service.
Tell us briefly what you need. We will agree on the scope and timing together in the first meeting.
Schedule a Meeting