Skip to content

Hide Navigation Hide TOC

Edit

PLOT4ai

Practical Library Of Threats 4 Artificial Intelligence (PLOT4ai) risk cards for threat modeling responsible AI systems.

Authors
Authors and/or Contributors
PLOT4ai
MISP Project

Data Quality

Can you avoid the known principle of “garbage in, garbage out”? Your AI system is only as reliable as the data it works with.

Threat-modeling question: Is our data complete, up-to-date, and trustworthy?

Internal MISP references

UUID 85b48cf6-1efc-51f6-8e25-c0a07b9d75ae which can be used as unique global reference for Data Quality in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Data & Data Governance']
category_id 1
colour #83b3db
external_id PLOT4AI-001
phases ['Design', 'Input', 'Monitor']
primary_category Data & Data Governance
question Is our data complete, up-to-date, and trustworthy?
recommendation * Verify the data sources:
  • Is there information missing within the dataset?
  • Can we verify that our training and input data hasn’t been tampered with or corrupted?
  • Are we using datasets that are outdated or no longer reflect the current environment?
  • Are all the necessary classes represented?
  • Does the data belong to the correct time frame and geographical coverage?
  • Evaluate which extra data you need to collect/receive.
  • Carefully consider representation schemes, especially in cases of text, video, APIs, and sensors. Text representation schemes are not all the same. If your system is counting on ASCII and it gets Unicode, will your system recognize the incorrect encoding? Source: BerryVilleiML | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['BerryVilleiML'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | threatif | No |

Target Leakage

Target Leakage is present when your features contain information that your model should not legitimately be allowed to use, leading to overestimation of the model's performance. It can occur when information from outside the training dataset is improperly included in the model during training. This can result in an unrealistically high performance during evaluation.

Threat-modeling question: Can we prevent target leakage?

Internal MISP references

UUID eeb50672-c7d1-59a5-a691-8fca76de4d68 which can be used as unique global reference for Target Leakage in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Data & Data Governance']
category_id 1
colour #83b3db
external_id PLOT4AI-002
phases ['Design', 'Input', 'Model', 'Monitor']
primary_category Data & Data Governance
question Can we prevent target leakage?
recommendation * Avoid using proxies for the outcome variable as a feature.
  • Do not use the entire data set for imputations, data-based transformations or feature selection.
  • Avoid doing standard k-fold cross-validation when you have temporal data.
  • Avoid using data that happened before model training time but is not available until later. This is common where there is delay in data collection.
  • Do not use data in the training set based on information from the future: if X happened after Y, you shouldn’t build a model that uses X to predict Y. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Leakage in data mining: formulation, detection, and avoidance', 'The Treachery of Leakage', 'Top 10 ways your Machine Learning models may have leakage', 'Leakage and the Reproducibility Crisis in ML-based Science'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Leakage in data mining: formulation, detection, and avoidance The Treachery of Leakage Top 10 ways your Machine Learning models may have leakage Leakage and the Reproducibility Crisis in ML-based Science | | threatif | No |

Drift

  • Data drift weakens performance because the model receives data on which it hasn’t been trained. It causes changes in the statistical properties of the input data distribution (e.g., feature distributions shift over time).
  • With Concept drift, the statistical properties of the target variable, which the model is trying to predict, change over time in unforeseen ways causing accuracy issues. It causes changes in the relationship between input features and the target variable (e.g., customer behavior changes over time, impacting a predictive model).

Threat-modeling question: Can we prevent concept and data drift?

Internal MISP references

UUID 8a3f9146-036a-55e1-b1ad-03ec3934baf0 which can be used as unique global reference for Drift in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Data & Data Governance']
category_id 1
colour #83b3db
external_id PLOT4AI-003
phases ['Input', 'Model', 'Output', 'Deploy', 'Monitor']
primary_category Data & Data Governance
question Can we prevent concept and data drift?
recommendation * Implement robust monitoring tools to detect data and concept drift, and establish governance policies for regular data validation and model retraining.

Data Continuity

  • Will you use the output from other models to feed your model again (looping)? Or will you use other sources?
  • Your AI system may rely on internal pipelines or third-party data sources. If any of these become unavailable, the model may stop functioning or deliver inaccurate results.
  • This includes scenarios like discontinued APIs, broken survey collection tools, or changes in upstream system outputs.

Threat-modeling question: Can the AI model maintain continuous access to data sources after deployment?

Internal MISP references

UUID e1f577e8-ada2-5f1c-bbdd-1fc7f293598e which can be used as unique global reference for Data Continuity in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Data & Data Governance']
category_id 1
colour #83b3db
external_id PLOT4AI-004
phases ['Design', 'Input', 'Deploy', 'Monitor']
primary_category Data & Data Governance
question Can the AI model maintain continuous access to data sources after deployment?
recommendation * Consider how the model will keep learning.
  • Identify critical data dependencies and define fallback mechanisms.
  • Assess whether key data sources are stable and under your control or subject to third-party risks.
  • Monitor availability of inputs to catch outages early.
  • Imagine you planned to feed your model with input obtained by mining surveys and it appears these surveys contain a lot of free text fields. To prepare that data and avoid issues (bias, inaccuracies, etc) you might need extra time. Consider these types of scenarios that could impact the whole life cycle of your system. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Text Mining in Survey Data'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Text Mining in Survey Data | | threatif | No |

Update Latency

  • In high-stakes domains like healthcare or finance, delays in processing updated external data can lead to stale predictions or risky decisions.
  • Risks include slow ingestion pipelines, format mismatches, or batch processing delays that prevent real-time responsiveness.
  • How much change are you expecting in the data you receive?
  • Can you make sure that you receive the updates on time?

Threat-modeling question: Can we process new or updated data from external sources without delay?

Internal MISP references

UUID 13860036-f3c7-5379-b444-d29bc1ad4281 which can be used as unique global reference for Update Latency in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Data & Data Governance']
category_id 1
colour #83b3db
external_id PLOT4AI-005
phases ['Design', 'Input', 'Deploy', 'Monitor']
primary_category Data & Data Governance
question Can we process new or updated data from external sources without delay?
recommendation * Design your data pipeline to handle frequent updates efficiently.
  • Validate incoming data formats, track data freshness, and assess update intervals.
  • Consider impact of delays on downstream decisions and mitigate with caching, fallback logic, or alerts.
  • Not only must you trust your sources, but you also need to design a process in which data is prepared on time to be used in the model and where you can timely consider the impact it could have in the output of the model, especially when this could have a negative impact on the users and system's behaviour. This process can be designed once you know how often changes in the data can be expected and how big the changes are. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | threatif | No |

Data Legitimacy

  • Data lineage can be necessary to demonstrate trust as part of your information transparency policy, but it can also be very important when it comes to assessing impact on the data flow. If sources are not verified and legitimized you could run risks such as data being wrongly labelled for instance.
  • Do you know where you need to get the data from? Who is responsible for the collection, maintenance and dissemination? Are the sources verified? Do you have the right agreements in place? Are you allowed to receive or collect that data? Also keep ethical considerations in mind!

Threat-modeling question: Are all required data sources legitimate, authorized, and verified?

Internal MISP references

UUID 35a73503-bd63-53e2-8962-5f06c6c8d953 which can be used as unique global reference for Data Legitimacy in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Data & Data Governance', 'Transparency & Accessibility']
category_id 1
colour #83b3db
external_id PLOT4AI-006
phases ['Design', 'Input', 'Monitor']
primary_category Data & Data Governance
question Are all required data sources legitimate, authorized, and verified?
recommendation * Develop a robust understanding of your relevant data feeds, flows and structures such that if any changes occur to the model data inputs, you can assess any potential impact on model performance. In case of third party AI systems contact your vendor to ask for this information.
  • If you are using synthetic data you should know how it was created and the properties it has. Also keep in mind that synthetic data might not be the answer to all your privacy related problems; synthetic data does not always provide a better trade-off between privacy and utility than traditional anonymisation techniques.
  • Do you need to share models and combine them? The usage of Model Cards and Datasheets can help providing the source information. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Providing Assurance and Scrutability on Shared Data and Machine Learning Models with Verifiable Credentials', 'Synthetic Data – Anonymisation Groundhog Day', 'Model Cards', 'Model Cards for Model Reporting', 'Datasheets for Datasets'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Providing Assurance and Scrutability on Shared Data and Machine Learning Models with Verifiable Credentials Synthetic Data – Anonymisation Groundhog Day Model Cards Model Cards for Model Reporting Datasheets for Datasets | | threatif | No |

Data Collection

Could you face difficulties obtaining certain type of data? This could be due to different reasons such as legal, proprietary, financial, physical, technical, etc. This could put the whole project in danger.

Threat-modeling question: Can we obtain the data needed to develop or fine-tune the AI model?

Internal MISP references

UUID ee1ac325-3b1f-53dd-b796-032ec51a8bc5 which can be used as unique global reference for Data Collection in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Data & Data Governance']
category_id 1
colour #83b3db
external_id PLOT4AI-007
phases ['Design', 'Input']
primary_category Data & Data Governance
question Can we obtain the data needed to develop or fine-tune the AI model?
recommendation In the early phases of the project (as soon as the task becomes more clear), start considering which raw data and types of datasets you might need. You might not have the definitive answer until you have tested the model, but it will already help to avoid extra delays and surprises. You might have to involve your legal and financial department. Remember that this is a team effort.
roles ['Provider', 'Deployer']
source_deck https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json
source_license CC-BY-SA-4.0
source_repository https://github.com/PLOT4ai/plot4ai-library
threatif No

Data Traceability

AI models require traceability of data sources to ensure ethical usage, reproducibility, and compliance. Without proper data lineage, it is difficult to verify the credibility and accuracy of training data.

Threat-modeling question: Can we trace the provenance and lineage of the data used to train or fine-tune the AI model?

Internal MISP references

UUID 03bd71a0-6db3-5988-bf2f-e72d767405fc which can be used as unique global reference for Data Traceability in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Data & Data Governance']
category_id 1
colour #83b3db
external_id PLOT4AI-008
phases ['Design', 'Input']
primary_category Data & Data Governance
question Can we trace the provenance and lineage of the data used to train or fine-tune the AI model?
recommendation * Use data lineage tracking tools to monitor where data originates and how it is modified over time.
  • Implement metadata standards (e.g., Datasheets for Datasets) to ensure clear documentation of data sources.
  • Regularly audit data providers to verify their reliability and adherence to ethical guidelines. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Datasheets for Datasets'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Datasheets for Datasets | | threatif | No |

Consider any legal, licensing, or privacy constraints that might prevent you from using certain datasets. This also applies to proprietary libraries, tools, or other resources.

Threat-modeling question: Could our dataset have copyright or other legal restrictions?

Internal MISP references

UUID 4541a9ea-94da-5a69-953a-f22c72b82467 which can be used as unique global reference for Copyright, IP & Legal Restrictions in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Data & Data Governance']
category_id 1
colour #83b3db
external_id PLOT4AI-009
phases ['Design', 'Input', 'Deploy', 'Monitor']
primary_category Data & Data Governance
question Could our dataset have copyright or other legal restrictions?
recommendation * Consider if you also need to claim ownership or give credits to creators.
  • Think about trademarks, copyrights in databases or training data, patents, license agreements that could be part of the dataset, library or module that you are using.
  • Legal ownership of digital data can sometimes be complex and uncertain so get the proper legal advice here. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | threatif | Yes |

Data Integrity

Data integrity is critical to ensuring that AI systems function as intended. Tampered data, whether during ingestion, transformation, storage, or transfer, can introduce hidden errors, biases, or malicious payloads. AI models built on compromised data may behave unpredictably, yield incorrect results, or violate compliance requirements. Integrity threats may be unintentional (e.g., pipeline errors) or deliberate (e.g., insider sabotage or supply chain attacks).

Threat-modeling question: Can we detect and prevent data tampering across the AI lifecycle?

Internal MISP references

UUID e57f5fe3-b0fd-593d-83ac-511d0e07bcce which can be used as unique global reference for Data Integrity in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Data & Data Governance', 'Cybersecurity']
category_id 1
cia ['i']
colour #83b3db
external_id PLOT4AI-010
phases ['Input', 'Model', 'Deploy', 'Monitor']
primary_category Data & Data Governance
question Can we detect and prevent data tampering across the AI lifecycle?
recommendation * Implement data integrity checks (e.g., hashes, checksums) at critical stages of the data pipeline.
  • Use tamper-evident storage (e.g., append-only logs, signed records).
  • Employ data lineage and provenance tracking systems to trace the origin and transformation history of data.
  • Apply anomaly detection to catch unexpected shifts or inconsistencies in inputs.
  • Audit access to data and enforce change tracking on data sources used for training or inference. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['ENISA - Securing Machine Learning Algorithms'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | ENISA - Securing Machine Learning Algorithms | | threatif | No |
Related clusters

To see the related clusters, click here.

Explainability

  • Is the algorithm transparent, or is it a 'black box' that users cannot interpret?
  • Can users learn about how the model works?
  • Is the model explainable, and are you open about the data used for training, including where and how it was sourced?

Threat-modeling question: Does the AI system need to be explainable for users or affected persons?

Internal MISP references

UUID 38d34f19-ec6e-5046-bbbb-cb2f061d5e60 which can be used as unique global reference for Explainability in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Transparency & Accessibility', 'Accountability & Human Oversight']
category_id 2
colour #7fccdc
external_id PLOT4AI-011
phases ['Design', 'Input', 'Model', 'Output', 'Deploy']
primary_category Transparency & Accessibility
question Does the AI system need to be explainable for users or affected persons?
recommendation * Evaluate the type of models that you could use to solve the problem as specified in your task.

Inclusivity

  • AI systems must be designed to be accessible and inclusive, particularly for individuals who may face barriers due to age, disability, or other vulnerabilities.
  • Think, for instance, of elderly people, children, or people with disabilities or individuals with cognitive impairments. Does the system support assistive technologies (e.g., screen readers)? Are there text alternatives, simplified navigation, or options for non-standard input and output formats?
  • Accessibility also includes cognitive accessibility: does the system assume a certain level of AI literacy or digital fluency that may not be present in all users? Are users supported with clear explanations, educational materials, or onboarding tools?
  • Inaccessible AI can lead to exclusion, discrimination, reduced autonomy, or even harm, violating fundamental rights under the Charter of Fundamental Rights of the EU (Articles 21: Non-discrimination, 24: Rights of the child, and 26: Integration of persons with disabilities).
  • The AI Act (Article 4) also highlights the need for systems to be inclusive and safe by design.

Threat-modeling question: Is our AI system inclusive and accessible?

Internal MISP references

UUID f25f21e1-9513-5659-b5e6-9f2b4952ba6d which can be used as unique global reference for Inclusivity in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Transparency & Accessibility']
category_id 2
colour #7fccdc
external_id PLOT4AI-012
phases ['Design', 'Input', 'Output', 'Deploy', 'Monitor']
primary_category Transparency & Accessibility
question Is our AI system inclusive and accessible?
recommendation * Conduct an impact assessment focusing on accessibility and vulnerability.
  • Is the consent information presented in a way that is easy for users to access and understand?
  • Do you need to create a dedicated place to display consent information, especially in contexts where a traditional text interface is not available (e.g., voice-based systems or robots)?

Threat-modeling question: If users’ consent is required, is the necessary information provided in a clear and accessible way?

Internal MISP references

UUID cbe09779-886e-5ffc-bbc2-29714d7d412b which can be used as unique global reference for Consent in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Transparency & Accessibility']
category_id 2
colour #7fccdc
external_id PLOT4AI-013
phases ['Design', 'Deploy', 'Monitor']
primary_category Transparency & Accessibility
question If users’ consent is required, is the necessary information provided in a clear and accessible way?
recommendation * As part of privacy compliance you need to provide clear information about the processing and the logic of the algorithm. This information should be easily readable and accessible. During the design phase consider when and how you are going to provide this information.

Perception

  • Does the user’s perception of the information match the intended meaning?
  • The way AI communicates, tone, language, and context, can lead to misinterpretation, influenced by factors like cultural background or prior experiences.

Threat-modeling question: Could the user perceive the message from the AI system in a different way than intended?

Internal MISP references

UUID 7143537c-1bba-57b6-b5f9-f93843a11a37 which can be used as unique global reference for Perception in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Transparency & Accessibility']
category_id 2
colour #7fccdc
external_id PLOT4AI-014
phases ['Design', 'Output', 'Monitor']
primary_category Transparency & Accessibility
question Could the user perceive the message from the AI system in a different way than intended?
recommendation * Understanding who is going to interact with the AI system can help to make the interaction more effective. Identify your different user groups.
  • Involve communication experts and do enough user testing to reduce the gap between the intended and the perceived meaning. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['The Who in Explainable AI: How AI Background Shapes Perceptions of AI Explanations'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | The Who in Explainable AI: How AI Background Shapes Perceptions of AI Explanations | | threatif | Yes |

Learning Curve

  • Does the system require a minimum level of AI literacy to be used?
  • Could a steep learning curve lead to misuse or harm?
  • How quickly are users expected to learn how to use the product?
  • Do you ensure that users have access to the necessary learning materials needed to be able to use the system?
  • Difficulties in learning how the system works could also bring the users in danger and have consequences for the reputation of the product or organisation.

Threat-modeling question: Is the AI system easy for users to learn and operate?

Internal MISP references

UUID 4c7fd3ce-93bf-5cda-9da3-a9c4ba7893c9 which can be used as unique global reference for Learning Curve in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Transparency & Accessibility', 'Safety & Environmental Impact']
category_id 2
colour #7fccdc
external_id PLOT4AI-015
phases ['Design', 'Deploy', 'Monitor']
primary_category Transparency & Accessibility
question Is the AI system easy for users to learn and operate?
recommendation * You can provide assistance, appropriate training material and disclaimers to users on how to adequately use the system.
  • The words and language used in the interface, the complexity and lack of accessibility of some features could exclude people from using the application. Consider making changes in the design of the product where necessary.
  • Consider this also when children are possible users. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | AI Act, Article 4 | | threatif | No |

AI Interaction Awareness

  • Users must be clearly informed when they are interacting with an AI system, especially in conversational interfaces, automated decision systems, or content generation tools. Failing to do so can undermine user trust, autonomy, and informed consent.
  • This includes both real-time interactions (e.g., chatbots) and offline consumption of AI-generated content (e.g., synthetic images, deepfakes).
  • Deepfakes and other AI-generated media that imitate real individuals or events carry high risks of deception, manipulation, and reputational harm if not transparently disclosed.
  • Lack of disclosure may also breach Article 50 of the EU AI Act and broader transparency obligations under the GDPR.

Threat-modeling question: Are users clearly made aware that they are interacting with an AI system or consuming AI-generated content?

Internal MISP references

UUID a3ae6383-1cdb-56ce-a612-f24cdcfbb2aa which can be used as unique global reference for AI Interaction Awareness in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Transparency & Accessibility']
category_id 2
colour #7fccdc
external_id PLOT4AI-016
phases ['Design', 'Deploy', 'Monitor']
primary_category Transparency & Accessibility
question Are users clearly made aware that they are interacting with an AI system or consuming AI-generated content?
recommendation * Inform users at the start of any interaction that they are engaging with an AI system, especially in systems simulating human communication (e.g., chatbots, virtual assistants).
  • For generative AI outputs (text, audio, video, images), ensure they are clearly marked, both visibly and in machine-readable format, as artificially generated or manipulated.
  • If your system produces deepfakes or synthetic media, implement persistent and tamper-resistant labeling or watermarks and include a notice that the content has been artificially generated or altered.
  • Deployers must also inform users when emotion recognition, biometric categorization, or similar AI functions are in use.
  • Design your UX to surface these disclosures prominently and accessibly, particularly in sensitive contexts such as news, education, or political speech. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Article 50 AI Act | | threatif | No |

System Transparency for Effective Use

Users need to understand what the AI system can and cannot do, including its intended use, reliability, limitations, and potential risks. Without clear communication, users may place unwarranted trust in the system, misuse it, or be harmed by misleading outputs. This undermines transparency, fairness, safety, and user autonomy. For example, failing to disclose error rates, decision logic, or appropriate use contexts can lead to over-reliance or unsafe behavior, especially in sensitive domains.

Threat-modeling question: Are users informed about the AI system's reliability, limitations, and risks in a way that enables safe and effective use?

Internal MISP references

UUID a8e15fff-f726-5dc8-baa1-79aeb0094dc3 which can be used as unique global reference for System Transparency for Effective Use in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Transparency & Accessibility', 'Safety & Environmental Impact']
category_id 2
colour #7fccdc
external_id PLOT4AI-017
phases ['Design', 'Deploy', 'Monitor']
primary_category Transparency & Accessibility
question Are users informed about the AI system's reliability, limitations, and risks in a way that enables safe and effective use?
recommendation * Clearly communicate the system's intended use, benefits, limitations, and potential risks.
  • Provide timely, accessible information on accuracy levels, error rates, interpretability, and system updates.
  • Ensure users understand when and how to rely on the system, and when human judgment is needed.
  • Use interpretability tools appropriate to the impact of the system, especially if it is a black-box model.
  • Follow accessibility best practices to ensure all users, including those with disabilities, can understand the system.
  • Incorporate feedback loops such as surveys to verify that users actually understand how the system works and what they can expect.
  • Consider this part of compliance with the GDPR transparency principle, and good practice for system safety and usability. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['GDPR transparency principle', 'Microsoft HAX Toolkit'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | GDPR transparency principle Microsoft HAX Toolkit | | threatif | No |

Linkability

  • Do you need to use unique identifiers in your training or fine-tuning dataset? If personal data is not necessary for the model you would not really have a legal justification for using it.
  • Training datasets for LLMs may inadvertently include personal data, leading to potential privacy breaches. Even if direct identifiers are removed, indirect identifiers or quasi-identifiers can still enable re-identification. This poses risks under data protection regulations like the GDPR, especially if the data subjects have not provided explicit consent for their data to be used in this manner.

Threat-modeling question: Can the training data be linked to individuals?

Internal MISP references

UUID 94d6bcab-75c4-5b00-b279-881489ed5c13 which can be used as unique global reference for Linkability in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Privacy & Data Protection']
category_id 3
colour #94cfbd
external_id PLOT4AI-018
phases ['Design', 'Input', 'Model', 'Output', 'Monitor']
primary_category Privacy & Data Protection
question Can the training data be linked to individuals?
recommendation * Unique identifiers might be included in the training set when you want to be able to link the results to individuals. Consider using pseudo-identifiers or other robust pseudonymization techniques that can help you protect personal data.
  • Document the measures you are taking to protect the data. Consider if your measures are necessary and proportional. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['EDPB AI Privacy Risks & Mitigations – Large Language Models (LLMs)'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | EDPB AI Privacy Risks & Mitigations – Large Language Models (LLMs) | | threatif | Yes |

Information Disclosure

  • How can you make sure the product doesn’t inadvertently disclose sensitive or private information during use (e.g., indirectly inferring location, behaviour or connection between digital and physical identity of users)?
  • Could movements or actions be revealed through data aggregation?

Threat-modeling question: Could the AI system infer and reveal information that a person has not explicitly shared?

Internal MISP references

UUID cdcaee22-bca8-5573-a3ed-36f9a2a8a473 which can be used as unique global reference for Information Disclosure in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Privacy & Data Protection']
category_id 3
colour #94cfbd
external_id PLOT4AI-019
phases ['Design', 'Input', 'Output', 'Deploy', 'Monitor']
primary_category Privacy & Data Protection
question Could the AI system infer and reveal information that a person has not explicitly shared?
recommendation * Be careful when making data public that you think is anonymised. Location data and routes can sometimes be de-anonymised (e.g. users of a running app disclosing location by showing heatmap).
  • It is also important to offer privacy by default: offer the privacy settings by default at the maximum protection level. Let the users change the settings after having offered them clear information about the consequences of reducing the privacy levels. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | threatif | Yes |
Related clusters

To see the related clusters, click here.

Local Restrictions

AI systems often process sensitive data, including personal or location-based information, which may be subject to regional data sovereignty laws and ethical restrictions. Additionally, certain countries may restrict the deployment of AI technologies based on local regulatory frameworks, ethical concerns, or national security considerations. This could limit the usage of your product in those regions.

Threat-modeling question: Could geolocation restrictions or regional regulations impact the implementation of our AI system in other countries?

Internal MISP references

UUID cacc0944-1983-5086-a16f-15f190841d56 which can be used as unique global reference for Local Restrictions in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Privacy & Data Protection']
category_id 3
colour #94cfbd
external_id PLOT4AI-020
phases ['Design', 'Input', 'Deploy', 'Monitor']
primary_category Privacy & Data Protection
question Could geolocation restrictions or regional regulations impact the implementation of our AI system in other countries?
recommendation * Stay informed about the evolving regulatory landscape for AI, including data sovereignty, privacy laws, and ethical standards in different countries. Engage legal and compliance experts to assess restrictions in your target markets.
  • Consider designing your AI system with adaptability for regional requirements, such as geofencing, localized processing, or compliance with specific regulations (e.g., GDPR, AI Act, CCPA).
  • Monitor new AI-related regulations and international agreements to proactively address potential restrictions or adapt your system to comply with local laws. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | threatif | Yes |

Data Minimization

The principle of data minimization, as outlined in the General Data Protection Regulation (GDPR) and reflected in many global privacy standards, requires that only data necessary for achieving the system's purpose is collected and processed. However, reducing data too much can sometimes negatively impact the accuracy and performance of AI models, leading to critical or damaging consequences. Balancing regulatory compliance with operational effectiveness is essential to avoid undermining the model's reliability while adhering to privacy principles.

Threat-modeling question: Can we minimize the amount of personal data used while preserving model performance?

Internal MISP references

UUID f8e522b9-f1fb-5ec9-841c-fb3b50488741 which can be used as unique global reference for Data Minimization in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Privacy & Data Protection', 'Data & Data Governance']
category_id 3
colour #94cfbd
external_id PLOT4AI-021
phases ['Design', 'Input', 'Model', 'Monitor']
primary_category Privacy & Data Protection
question Can we minimize the amount of personal data used while preserving model performance?
recommendation * Achieve data minimization by starting with a smaller dataset and iteratively adding data only as needed, based on observed performance improvements, to justify why additional data is necessary.
  • Use high-quality data to reduce the need for large datasets while ensuring sufficient diversity and representativeness for your model.
  • Apply advanced privacy-preserving techniques such as pseudonymization, perturbation, differential privacy, federated learning, or synthetic data generation to comply with privacy regulations while using larger datasets.
  • Collaborate with experts to select the minimum set of features needed, ensuring relevance to the objective and avoiding issues like the Curse of Dimensionality, which can degrade model performance when unnecessary features are included. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Artificial Intelligence and Data Protection How the GDPR Regulates AI', 'Data Minimization for GDPR Compliance in Machine Learning Models', 'Does Dimensionality curse effect some models more than others?', 'Towards Breaking the Curse of Dimensionality for High-Dimensional Privacy'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Page 13 Artificial Intelligence and Data Protection How the GDPR Regulates AI Data Minimization for GDPR Compliance in Machine Learning Models: Methods like the one proposed in this paper can inspire you to find a way to mitigate the accuracy risk. They show how to reduce the amount of personal data needed to perform predictions, by removing or generalizing some of the input features. The answer to this post also contains information about this problems in different models: Does Dimensionality curse effect some models more than others? Towards Breaking the Curse of Dimensionality for High-Dimensional Privacy | | threatif | No |

Sensitive Data

  • According to art. 9 GDPR you might not be allowed to process, under certain circumstances, personal data revealing racial or ethnic origin, political opinions, religious or philosophical beliefs, trade union membership, genetic data, biometric data, health data or data concerning a person’s sex life or sexual orientation.
  • You might be processing sensitive data if the model includes features that are correlated with these protected characteristics (these are called proxies) but also if you are processing other type of data that, if disclosed, could cause harm (e.g., financial data)

Threat-modeling question: Are we processing special categories of personal data or sensitive data?

Internal MISP references

UUID 50947e97-5cf6-5583-ba8c-82f7a7c2273f which can be used as unique global reference for Sensitive Data in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Privacy & Data Protection']
category_id 3
colour #94cfbd
external_id PLOT4AI-022
phases ['Design', 'Input', 'Model']
primary_category Privacy & Data Protection
question Are we processing special categories of personal data or sensitive data?
recommendation * If you need to use special categories of data as defined in the GDPR art. 9, then you need to check if you have the right lawful basis to do this.
  • Applying techniques like anonymisation might still not justify the fact that you first need to process the original data. Check with your privacy/legal experts.
  • Prevent proxies that could infer sensitive data (especially from vulnerable populations).
  • Check whether historical data or practices may introduce bias.
  • Identify and remove features that are correlated to sensitive characteristics.
  • Use available methods to test for fairness with respect to different affected groups. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['AI Fairness 360', 'What-if Tool: Playing with AI Fairness', 'Hunting for Discriminatory Proxies in Linear Regression Models', 'Differential Privacy Blog Series'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | AI Fairness 360 What-if Tool: Playing with AI Fairness Hunting for Discriminatory Proxies in Linear Regression Models Differential Privacy Blog Series | | threatif | Yes |

Automated Decision-Making (ADM)

  • AI systems that make decisions without human oversight may fall under GDPR Article 22, which restricts significant automated decisions unless specific safeguards are in place. These decisions can affect individuals’ rights, legal status, or access to services.
  • Additionally, Article 86 of the AI Act requires transparency and the provision of clear explanations for significant decisions made by high-risk AI systems.

Threat-modeling question: Could the AI system make decisions with legal or similarly significant effects without human intervention?

Internal MISP references

UUID 4f64abe4-ff91-58ca-9f90-b02f2413dbbd which can be used as unique global reference for Automated Decision-Making (ADM) in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Privacy & Data Protection', 'Accountability & Human Oversight', 'Safety & Environmental Impact']
category_id 3
colour #94cfbd
external_id PLOT4AI-023
phases ['Design', 'Output', 'Monitor']
primary_category Privacy & Data Protection
question Could the AI system make decisions with legal or similarly significant effects without human intervention?
recommendation * Consult privacy and legal experts to determine whether your system qualifies under Article 22 of the GDPR.
  • Implement mechanisms for human intervention, contestability, and explanation. Article 22(3) of the GDPR provides individuals with the right to obtain human intervention in automated decisions and the right to contest such decisions.
  • Align with the EU AI Act's oversight and transparency requirements.
  • Ensure that impacted users are informed of their rights and can seek human review.
  • Maintain documentation of decision logic, oversight processes, and risk mitigation strategies. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Ethics guidelines for trustworthy AI'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | GDPR, AI Act, Ethics guidelines for trustworthy AI | | threatif | Yes |

Lawful Basis

Do you know which GDPR legal ground you can apply? * (a) Consent: the individual has given clear consent for you to process their personal data for a specific purpose. * (b) Contract: the processing is necessary for a contract you have with the individual, or because they have asked you to take specific steps before entering into a contract. * (c) Legal obligation: the processing is necessary for you to comply with the law (not including contractual obligations). * (d) Vital interests: the processing is necessary to protect someone’s life. * (e) Public task: the processing is necessary for you to perform a task in the public interest or for your official functions, and the task or function has a clear basis in law. * (f) Legitimate interests: the processing is necessary for your legitimate interests or the legitimate interests of a third party, except where such interests are overridden by the interests or fundamental rights and freedoms of the individual which require protection of personal data, in particular where the individual is a child. (This cannot apply if you are a public authority processing data to perform your official tasks.)

Threat-modeling question: Do we have a valid legal basis for processing personal data?

Internal MISP references

UUID eb24a088-4978-5269-8617-a4b205674633 which can be used as unique global reference for Lawful Basis in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Privacy & Data Protection']
category_id 3
colour #94cfbd
external_id PLOT4AI-024
phases ['Design', 'Input', 'Deploy', 'Monitor']
primary_category Privacy & Data Protection
question Do we have a valid legal basis for processing personal data?
recommendation * In the case of the GDPR you need to be able to apply one of the six available legal grounds for processing the data (art. 6).
  • Check with your privacy expert, not being able to apply one of the legal grounds could bring the project in danger.
  • Take into account that other laws besides the GDPR may also apply. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Lawful basis for processing', 'Artificial Intelligence and Data Protection How the GDPR Regulates AI'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Lawful basis for processing Artificial Intelligence and Data Protection How the GDPR Regulates AI | | threatif | No |

Purpose Limitation

The principle of purpose limitation, as defined in the General Data Protection Regulation (GDPR) and echoed in many global privacy frameworks, requires that personal data is collected for specified, explicit, and legitimate purposes and not further processed in a way incompatible with those purposes. Data repurposing is a significant challenge when applying this principle. If datasets were originally collected for a different purpose, their reuse without proper consent or legal justification may violate privacy regulations and ethical standards.

Threat-modeling question: Could we be using personal data for purposes different from those for which it was originally collected?

Internal MISP references

UUID 9a593594-a3e9-5938-871d-7d1403ecd3f7 which can be used as unique global reference for Purpose Limitation in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Privacy & Data Protection']
category_id 3
colour #94cfbd
external_id PLOT4AI-025
phases ['Design', 'Input', 'Deploy', 'Monitor']
primary_category Privacy & Data Protection
question Could we be using personal data for purposes different from those for which it was originally collected?
recommendation * Consult with your privacy officer or legal team to verify the original purpose of the data collection and evaluate any constraints or legal requirements.
  • If data repurposing is necessary, consider obtaining additional consent, performing a legitimate interest assessment, or applying anonymization techniques to ensure compliance.
  • Additionally, document all decisions and justifications for data reuse to demonstrate accountability under privacy regulations. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | threatif | Yes |

Data Subject Rights

  • Can you implement the right to withdraw consent, the right to object to the processing and the right to be forgotten into the development of the AI system?
  • Can you provide individuals with access and a way to rectify their data?

Threat-modeling question: Are we able to comply with all the applicable GDPR data subjects’ rights?

Internal MISP references

UUID ae9ea438-f1ea-5622-b1bb-dbaeb5fda115 which can be used as unique global reference for Data Subject Rights in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Privacy & Data Protection']
category_id 3
colour #94cfbd
external_id PLOT4AI-026
phases ['Design', 'Input', 'Output', 'Monitor']
primary_category Privacy & Data Protection
question Are we able to comply with all the applicable GDPR data subjects’ rights?
recommendation * Complying with these provisions from the GDPR (art. 15-21) could have an impact on the design of your product. What if users withdraw their consent? Do you need to delete their data used to train the model? What if users can no longer be identified in the dataset? And what information should the users have access to?
  • Consider all these possible scenarios and involve your privacy experts early in the design phase. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | threatif | No |

Privacy Impact Assessment

The use of AI is more likely to trigger the requirement for a DPIA, based on criteria in Article 35 GDPR. The GDPR and the EDPB’s Guidelines on DPIAs identify both “new technologies” and the type of automated decision-making that produce legal effects or similarly significantly affect persons as likely to result in a “high risk to the rights and freedoms of natural persons”.

Threat-modeling question: Could we be deploying the AI system without conducting a required Data Protection Impact Assessment (DPIA)?

Internal MISP references

UUID fbc7908e-4635-55e2-aa3f-847a2493e888 which can be used as unique global reference for Privacy Impact Assessment in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Privacy & Data Protection']
category_id 3
colour #94cfbd
external_id PLOT4AI-027
phases ['Design', 'Input', 'Output', 'Deploy']
primary_category Privacy & Data Protection
question Could we be deploying the AI system without conducting a required Data Protection Impact Assessment (DPIA)?
recommendation * This threat modeling library can help you to assess possible risks.
  • Remember that a DPIA is not a piece of paper that needs to be done once the product is in production. The DPIA starts in the design phase by finding and assessing risks, documenting them and taking the necessary actions to create a responsible product from day one until it is finalized.
  • Consider the time and resources that you might need for the execution of a DPIA, as it could have some impact on your project deadlines. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | threatif | Yes |

Third-party Data Processing

If your system processes data from children or other vulnerable groups, any third-party providers you rely on (such as libraries, SDKs, or other tools) may also have access to this data. In such cases, you must ensure they comply with relevant privacy regulations like GDPR, COPPA, or similar frameworks. Even if your own system adheres to strong data protection measures, vulnerabilities or non-compliance on the part of third-party providers could expose sensitive data or create ethical risks.

Threat-modeling question: Are we using third-party providers while processing data from children or other vulnerable individuals?

Internal MISP references

UUID 9048de36-b9b4-5297-a24a-fabbf44a24e0 which can be used as unique global reference for Third-party Data Processing in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Privacy & Data Protection']
category_id 3
colour #94cfbd
external_id PLOT4AI-028
phases ['Design', 'Input', 'Monitor']
primary_category Privacy & Data Protection
question Are we using third-party providers while processing data from children or other vulnerable individuals?
recommendation * Audit all third-party applications, libraries, and tools you use to determine what data they collect and ensure they comply with applicable regulations.
  • Confirm that proper agreements (e.g., Data Processing Agreements) are in place with all third-party providers to specify how data is handled.
  • Where possible, configure third-party tools to limit or avoid sharing sensitive data. Implement pseudonymization or anonymization techniques to protect data before sharing.
  • Evaluate the necessity of each third-party provider. If risks are identified, consider replacing or discontinuing use of certain providers, weighing the operational impact on your organization. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | threatif | Yes |

Metadata

  • Metadata provides descriptive attributes of other data, such as date, time, author, file size, or geolocation.
  • Although metadata may seem innocuous, it is often considered personal data under privacy regulations (e.g., GDPR) and can contain sensitive information. Misusing or failing to protect metadata can lead to privacy violations and unintended risks, especially if it reveals identifiable information.

Threat-modeling question: Are we using metadata that could reveal personal data or behavior patterns?

Internal MISP references

UUID 259995b6-bd5e-5b42-85fa-3b00f2d7c30d which can be used as unique global reference for Metadata in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Privacy & Data Protection', 'Data & Data Governance']
category_id 3
colour #94cfbd
external_id PLOT4AI-029
phases ['Design', 'Input', 'Model']
primary_category Privacy & Data Protection
question Are we using metadata that could reveal personal data or behavior patterns?
recommendation * Ensure that your use of metadata complies with applicable privacy regulations by verifying whether the data can be lawfully processed for your intended purpose.
  • Audit and verify metadata sources to confirm their accuracy and legitimacy.
  • Implement anonymization or pseudonymization techniques to minimize privacy risks while using metadata.
  • Limit the collection of metadata to only what is strictly necessary for the model, adhering to the principle of data minimization. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | threatif | Yes |

Privacy Rights

The AI system may intrude on users' right to privacy by exposing sensitive aspects of their private lives, such as personal behaviors, preferences, or relationships, without their explicit consent or awareness. This can occur through excessive surveillance, unintended inferences, profiling, or sharing personal data without proper safeguards. Such compromises may undermine users' autonomy, dignity, and trust in the system, leading to legal, ethical, and reputational consequences for providers.

Threat-modeling question: Could we compromise users’ rights to privacy and to a private and family life?

Internal MISP references

UUID 1509aea9-ccc1-53a5-8579-35970dfbbc0e which can be used as unique global reference for Privacy Rights in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Privacy & Data Protection', 'Data & Data Governance']
category_id 3
colour #94cfbd
external_id PLOT4AI-030
phases ['Design', 'Input', 'Output', 'Deploy', 'Monitor']
primary_category Privacy & Data Protection
question Could we compromise users’ rights to privacy and to a private and family life?
recommendation * Ensure that the AI system respects the contextual integrity of users' private lives by limiting inferences and decisions to what is strictly necessary for its intended purpose.
  • Minimize the risk of profiling that could reveal sensitive personal attributes or behaviors unless explicitly justified by the intended use and supported by users’ consent or legal ground.
  • Design the AI system to avoid unnecessary observation or analysis of users’ private spaces, behaviors, or communications unless explicitly required by the use case.
  • Provide clear and accessible information to users about the extent and nature of the AI system's interaction with their private lives, ensuring that they are fully informed about its capabilities.
  • Empower users to set boundaries for their privacy by allowing them to control the scope of data collection and interaction with the AI system (Privacy by default).
  • Include ethical reviews and stakeholder consultations to assess the potential implications of the system on users’ privacy in diverse cultural and social contexts.
  • Implement safeguards to prevent the system from drawing unintended, intrusive, or harmful conclusions about individuals’ private lives.
  • Ensure robust security measures to prevent unauthorized access, surveillance, or other misuse of the system that could violate users’ privacy rights.
  • Provide mechanisms for users to report and address concerns if they feel their privacy has been violated, including remedies for potential harm caused. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Right to privacy (Universal Declaration of Human Rights), Article 7 Respect for Private and Family Life (Charter of fundamental rights of the European Union) | | threatif | Yes |

Transparent Information

Users and stakeholders may not fully understand how data is collected, processed, and utilized, leading to concerns about privacy, accountability, and trust. A lack of transparency can make it difficult to verify whether personal data is being used lawfully or ethically. AI decision-making may be opaque, increasing risks of bias, discrimination, or unfair outcomes.

Threat-modeling question: Are we providing sufficient transparency about how the AI model collects, processes, and uses personal data?

Internal MISP references

UUID 4ee73ec7-8f8a-545b-946a-b708e694ab12 which can be used as unique global reference for Transparent Information in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Privacy & Data Protection', 'Transparency & Accessibility']
category_id 3
colour #94cfbd
external_id PLOT4AI-031
phases ['Design', 'Input', 'Model', 'Output', 'Monitor']
primary_category Privacy & Data Protection
question Are we providing sufficient transparency about how the AI model collects, processes, and uses personal data?
recommendation * Implement explainability tools that provide insights into AI decision-making.
  • Use clear and accessible documentation detailing data collection, storage, processing, and sharing.
  • Follow transparency principles from the EU AI Act and GDPR regarding automated decision-making.
  • Utilize model cards, data sheets, and algorithmic auditing to enhance transparency. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | threatif | No |

Storing of User Data

AI systems, particularly Large Language Models (LLMs), may log user inputs and outputs for debugging or model fine-tuning, potentially storing sensitive data without explicit user consent. Logged data could be included in training datasets, making it possible for adversaries to conduct data poisoning attacks, influencing model behavior. Even metadata from logs may reveal sensitive details about users.

Threat-modeling question: Are we logging or storing user input data in ways that may violate privacy?

Internal MISP references

UUID a83846ee-13ad-5b7d-8fe4-cdedc39b41c1 which can be used as unique global reference for Storing of User Data in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Privacy & Data Protection', 'Data & Data Governance', 'Cybersecurity']
category_id 3
colour #94cfbd
external_id PLOT4AI-032
phases ['Design', 'Input', 'Output', 'Deploy', 'Monitor']
primary_category Privacy & Data Protection
question Are we logging or storing user input data in ways that may violate privacy?
recommendation * Implement strict access controls and data minimization techniques to prevent excessive logging.
  • Provide opt-in or opt-out options for data collection and obtain explicit consent where needed.
  • Regularly audit and delete logs containing personal or sensitive data.
  • Use differential privacy, encryption, or synthetic data to minimize risks while analyzing logs.
  • Detect and mitigate adversarial attacks aimed at poisoning training data. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['WiP: An On-device LLM-based Approach to Query Privacy Protection'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | WiP: An On-device LLM-based Approach to Query Privacy Protection | | threatif | Yes |

Inaccurate Output

  • AI systems may generate, infer, or reproduce incorrect personal data, leading to violations of the GDPR’s accuracy principle (Article 5(1)(d)) and potential harm to data subjects.
  • Outputs may inadvertently reveal sensitive data or personal details, leading to privacy breaches.
  • In traditional AI, this includes misclassification, profiling errors, or incorrect risk scoring that affect decisions about individuals (e.g., in hiring, finance, law enforcement).
  • In generative AI, this includes hallucinated personal facts or fabricated content that falsely attributes actions, identities, or characteristics to real people. When multiple AI agents interact, hallucinations and errors can amplify, increasing the likelihood of spreading misinformation.
  • These inaccuracies can damage reputations, mislead users, or be stored and processed in downstream systems, compounding the data protection risk.

Threat-modeling question: Could the AI system produce inaccurate or misleading outputs that result in privacy violations or harm?

Internal MISP references

UUID 042c20d8-604b-5e0f-9cd1-f26da4193712 which can be used as unique global reference for Inaccurate Output in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Privacy & Data Protection', 'Data & Data Governance', 'Ethics & Human Rights']
category_id 3
colour #94cfbd
external_id PLOT4AI-033
phases ['Design', 'Input', 'Model', 'Output', 'Monitor']
primary_category Privacy & Data Protection
question Could the AI system produce inaccurate or misleading outputs that result in privacy violations or harm?
recommendation * Conduct data protection impact assessments (DPIAs) to evaluate how inaccurate outputs could affect individuals' rights and freedoms.
  • Provide mechanisms for individuals to access, rectify, or contest inferences or decisions made by AI systems.
  • Maintain logs and audit trails to trace how inaccurate personal data was generated or propagated.
  • Train models with high-quality, up-to-date, and verified datasets to minimize the risk of misinformation and outdated personal data

For generative AI: * Implement fact-checking and validation mechanisms before AI-generated responses are shown to users. * Implement named-entity detection and filtering to prevent false personal information from being output * Use retrieval-augmented generation (RAG) and human-in-the-loop (HITL) approaches to improve accuracy. * Red-team AI models by stress-testing them for misinformation and privacy risks. * Allow users to report inaccurate or harmful content, enabling iterative model improvements. * Restrict AI-generated outputs on sensitive topics unless rigorous verification is in place.

For traditional AI (e.g., classification, regression, or rule-based systems): * Validate models on diverse, real-world datasets to test for generalizability and edge-case failures. * Implement post-deployment performance monitoring and regular retraining to reduce drift and degradation over time. * Conduct error analysis on false positives and false negatives to refine model logic and thresholds. * Include uncertainty estimation and confidence scoring to guide decision-making, especially in high-risk use cases. * In safety-critical applications, ensure fallback mechanisms or manual review paths are available when confidence is low. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Hallucination Detection in Large Language Models with Metamorphic Relations', 'Unraveling Large Language Model Hallucinations'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Hallucination Detection in Large Language Models with Metamorphic Relations Unraveling Large Language Model Hallucinations | | threatif | Yes |

Data Transfers

  • AI systems may store or process data in countries with weaker privacy protections, leading to GDPR violations.
  • Transfers outside the EU/EEA may violate: GDPR Art. 44-46 restricting international data transfers without adequate safeguards.
  • If personal data is processed in non-compliant jurisdictions, organizations face legal, financial, and reputational risks.

Threat-modeling question: Are we transferring personal data to countries that lack adequate privacy protections?

Internal MISP references

UUID 42f421c0-8d30-5d8a-ab6e-2c6e28c03f10 which can be used as unique global reference for Data Transfers in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Privacy & Data Protection', 'Data & Data Governance']
category_id 3
colour #94cfbd
external_id PLOT4AI-034
phases ['Design', 'Deploy', 'Monitor']
primary_category Privacy & Data Protection
question Are we transferring personal data to countries that lack adequate privacy protections?
recommendation * Conduct a Data Transfer Impact Assessment (DTIA) before processing data outside GDPR-compliant regions.
  • Use Standard Contractual Clauses (SCCs), Binding Corporate Rules (BCRs), or adequacy decisions when transferring data.
  • Store and process personal data in localized environments to comply with data sovereignty laws.
  • Implement encryption and anonymization before data is transferred across jurisdictions.
  • Continuously monitor regulatory updates to ensure ongoing compliance with global privacy laws. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['International dimension of data protection', 'Court of Justice Schrems II'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | International dimension of data protection Court of Justice Schrems II | | threatif | Yes |

Storage Limitation

The principle of storage limitation, as stated in Article 5(e) of the GDPR, requires personal data to be stored only as long as necessary for the intended purpose. Similarly, many global privacy regulations, such as CCPA (California), LGPD (Brazil), and PDPB (India), impose strict rules on data retention and deletion. Do you have a clear understanding of how long you need to keep the data (training data, output data, etc.) and whether you comply with internal, local, national, or international retention requirements?

Threat-modeling question: Can we comply with the storage limitation principle and international data retention regulations?

Internal MISP references

UUID b02b099a-6af2-5167-8000-a00596511485 which can be used as unique global reference for Storage Limitation in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Privacy & Data Protection', 'Data & Data Governance']
category_id 3
colour #94cfbd
external_id PLOT4AI-035
phases ['Design', 'Input', 'Deploy', 'Monitor']
primary_category Privacy & Data Protection
question Can we comply with the storage limitation principle and international data retention regulations?
recommendation * Personal data must not be stored longer than necessary for its intended purpose. Compliance requires a clear understanding of the data flow throughout the model’s lifecycle.
  • Analyze all data types, including raw input data, training and testing sets, processed outputs (linked or merged data), and associated metrics. Understand where this data will be stored and for how long.
  • Define clear retention and deletion schedules, ensuring responsible individuals are assigned for managing data retention and disposal.
  • If data must be retained for auditing or quality purposes, anonymize it where possible to minimize privacy risks.
  • Stay informed about and comply with retention rules not only under GDPR but also under international frameworks such as CCPA (California Consumer Privacy Act), LGPD (Brazilian General Data Protection Law), and others. Retention and deletion policies should meet these diverse requirements.
  • Be aware that deleting data from a trained model is inherently challenging, as input data influences the model's internal representation during training. Consider legal implications for the model itself, as encoded thresholds and weights may also be subject to retention laws. Source: BerryvilleiML | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['BerryvilleiML'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | threatif | No |

Security Testing

AI systems can be targeted in unique ways, such as adversarial inputs, poisoning attacks, or reverse-engineering of model outputs. These threats could compromise the system's confidentiality, integrity, and availability, leading to reputational damage or harm to users. Testing for these issues may require specialized expertise, tools, and time, which could affect project timelines.

Threat-modeling question: Could we be deploying the AI system without testing for adversarial robustness and systemic vulnerabilities?

Internal MISP references

UUID 7dc672f6-16af-5276-8266-6f5d0b43106d which can be used as unique global reference for Security Testing in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity']
category_id 4
cia ['c', 'i', 'a']
colour #bdd895
external_id PLOT4AI-036
phases ['Design', 'Model', 'Deploy', 'Monitor']
primary_category Cybersecurity
question Could we be deploying the AI system without testing for adversarial robustness and systemic vulnerabilities?
recommendation Plan for AI-specific penetration testing or red-teaming exercises, focusing on adversarial robustness, data governance, and model-specific vulnerabilities. Allocate time in the project for external audits, agreement on scope, and retesting if vulnerabilities are found.
roles ['Provider', 'Deployer']
source_deck https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json
source_license CC-BY-SA-4.0
source_names ['Securing Machine Learning Algorithms, ENISA', 'STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets', 'MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems']
source_repository https://github.com/PLOT4ai/plot4ai-library
source_text Securing Machine Learning Algorithms, ENISA

STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets Stride-ML Threat Model MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems | | threatif | Yes |

Related clusters

To see the related clusters, click here.

API & Model Interface Security

AI systems increasingly rely on APIs for inference (e.g., LLM endpoints), orchestration (e.g., function calls via tools), or dynamic prompt injection (e.g., Model Context Protocol). Poorly secured APIs expose attack surfaces specific to LLMs and other AI models.

Threats include: * Prompt injection via API inputs or user tool outputs (e.g., using MCP-style interfaces). * Malicious function calls that exploit insecure tool execution pipelines. * Abuse of structured output endpoints (e.g., JSON-formatted APIs) to extract or manipulate model behavior. * Reverse-engineering model behavior via inference chaining or output probing.

Attacks on shared foundational model APIs can impact multiple downstream applications through shared vulnerabilities, hallucination exploits, or jailbreak discovery.

Threat-modeling question: Are our AI inference APIs and function-calling interfaces securely implemented?

Internal MISP references

UUID e7827e98-4b5d-5937-b721-5ecff98a0b61 which can be used as unique global reference for API & Model Interface Security in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity']
category_id 4
cia ['c', 'i', 'a']
colour #bdd895
external_id PLOT4AI-037
phases ['Design', 'Deploy', 'Monitor']
primary_category Cybersecurity
question Are our AI inference APIs and function-calling interfaces securely implemented?
recommendation * Implement security best practices:
  • Use strong authentication mechanisms such as API keys or OAuth.
  • Enforce role-based access controls (RBAC) to restrict functionality.
  • Encrypt data at rest and in transit (TLS).
  • Validate and sanitize all inputs; apply strict content-type controls.
  • Use allowlists and structured schemas (e.g., OpenAPI, JSON Schema) to constrain behavior.
  • Avoid exposing secrets in API calls or payloads.
  • Regularly test APIs for vulnerabilities including injection attacks, improper state management, and rate limit bypasses.
  • Deploy anomaly detection to flag adversarial or abnormal usage patterns.
  • Limit API output granularity to prevent reverse engineering; obfuscate or truncate confidence scores.
  • Monitor and log all API interactions to detect and investigate abuse.
  • Rate-limit and throttle access to prevent enumeration or prompt probing.
  • For LLMs with plugin, function-calling, or Model Context Protocol (MCP) interfaces:
  • Monitor for prompt injection and abuse chains across tools.
  • Apply zero-trust design principles to inference and orchestration layers.
  • Red-team APIs and function interfaces regularly.
  • Collaborate with foundational model providers to validate the security of shared inference APIs and plugin-style architectures. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['OWASP API Security Project', 'BerryVilleiML', 'Securing Machine Learning Algorithms, ENISA', 'STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets', 'MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems', 'Lessons From Red Teaming 100 Generative AI Products – Microsoft', 'MCP Security Exposed: What You Need to Know Now – Palo Alto Networks'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | OWASP API Security Project BerryVilleiML Securing Machine Learning Algorithms, ENISA STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets Stride-ML Threat Model MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems Lessons From Red Teaming 100 Generative AI Products – Microsoft MCP Security Exposed: What You Need to Know Now – Palo Alto Networks | | threatif | No |
Related clusters

To see the related clusters, click here.

Storage Protection

Is your data stored and managed in a secure way? Think about training data, tables, models, outputs, etc. Do only authorized individuals have access to your data sources? Source: BerryVilleiML

Threat-modeling question: Are training data, model output, and other sensitive AI assets securely stored?

Internal MISP references

UUID 8799cc8b-2a14-55e2-bac0-ef9a3c9014e8 which can be used as unique global reference for Storage Protection in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity']
category_id 4
cia ['c', 'i', 'a']
colour #bdd895
external_id PLOT4AI-038
phases ['Design', 'Input', 'Deploy', 'Monitor']
primary_category Cybersecurity
question Are training data, model output, and other sensitive AI assets securely stored?
recommendation * Implement access control rules.

Randomness Protection

Randomness plays an important role in stochastic systems. “Random” generation of dataset partitions may be at risk if the source of randomness is easy to control by an attacker interested in data poisoning. Source: BerryVilleiML

Threat-modeling question: If the AI system uses randomness, is the source of randomness properly protected?

Internal MISP references

UUID f76ef0ac-0664-5596-99e0-6ae4ad83be73 which can be used as unique global reference for Randomness Protection in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity']
category_id 4
cia ['i']
colour #bdd895
external_id PLOT4AI-039
phases ['Design', 'Model']
primary_category Cybersecurity
question If the AI system uses randomness, is the source of randomness properly protected?
recommendation Use of cryptographic randomness sources is encouraged. When it comes to machine learning (ML), setting weights and thresholds “randomly” must be done with care. Many pseudo-random number generators (PRNG) are not suitable for use. Improper PRNG loops can degrade system behavior and lead to unpredictable learning. Cryptographic randomness directly intersects with ML when it comes to differential privacy. Using the wrong sort of random number generator can lead to subtle security problems.

Source: BerryVilleiML | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Securing Machine Learning Algorithms, ENISA', 'STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets', 'MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems', 'BerryVilleiML'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Securing Machine Learning Algorithms, ENISA STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets Stride-ML Threat Model MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems | | threatif | No |

Confidential Information

  • There are certain kinds of machine learning (ML) models which actually contain parts of the training data in its raw form within them by design. For example, ‘support vector machines’ (SVMs) and ‘k-nearest neighbours’ (KNN) models contain some of the training data in the model itself.
  • Algorithmic leakage is an issue that should be considered carefully. Source: BerryVilleiML

Threat-modeling question: Is the AI model suited for processing confidential information?

Internal MISP references

UUID a7949861-5252-50a0-af64-9d8de323ea2e which can be used as unique global reference for Confidential Information in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity', 'Privacy & Data Protection']
category_id 4
cia ['c']
colour #bdd895
external_id PLOT4AI-040
phases ['Design', 'Input', 'Model', 'Deploy']
primary_category Cybersecurity
question Is the AI model suited for processing confidential information?
recommendation When selecting the algorithm, conduct a thorough analysis to evaluate the risk of algorithmic leakage. For models known to retain training data (e.g., k-nearest neighbors, support vector machines), assess whether sensitive or identifiable information could be exposed through predictions or reverse engineering.
Related clusters

To see the related clusters, click here.

Insider Threats

AI designers and developers may deliberately expose data and models for a variety of reasons, e.g. revenge or extortion. Integrity, data confidentiality and trustworthiness are the main impacted security properties. Source: ENISA

Threat-modeling question: Have we implemented safeguards to detect and prevent insider threats to our AI systems?

Internal MISP references

UUID 10628032-5ef6-5b96-9fde-c4225e53651d which can be used as unique global reference for Insider Threats in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity']
category_id 4
cia ['c', 'i', 'a']
colour #bdd895
external_id PLOT4AI-041
phases ['Design', 'Deploy', 'Monitor']
primary_category Cybersecurity
question Have we implemented safeguards to detect and prevent insider threats to our AI systems?
recommendation * Implement onboarding and offboarding procedures to ensure the trustworthiness of internal and external personnel.

Model Sabotage

Model sabotage involves deliberate manipulation or damage to AI systems at any stage, from development to deployment. This can include embedding backdoors, altering model behavior, or exploiting vulnerabilities in training data, third-party tools, or infrastructure. * For AI providers: Risks include compromised training datasets, malicious code in open-source libraries, or backdoors introduced during development. * For AI deployers: Threats arise from integrating tampered models, using insecure APIs, or applying updates that introduce vulnerabilities.

Threat-modeling question: Have we protected our AI system against model sabotage?

Internal MISP references

UUID 45d6d69e-62d2-510e-aec0-74a703a14960 which can be used as unique global reference for Model Sabotage in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity']
category_id 4
cia ['c', 'i', 'a']
colour #bdd895
external_id PLOT4AI-042
phases ['Design', 'Model', 'Deploy', 'Monitor']
primary_category Cybersecurity
question Have we protected our AI system against model sabotage?
recommendation * Implement strong security measures, including regular audits and penetration testing, to ensure the integrity of models and the platforms hosting them.
  • Assess and monitor the security profile of third-party libraries, tooling, and providers to ensure they are not compromised.
  • Develop and maintain a robust disaster recovery plan with explicit mitigation strategies for model sabotage scenarios.
  • Use model inspection tools to detect backdoors and ensure that the model’s behavior aligns with its intended function.
  • Incorporate supply chain security principles by verifying the authenticity and integrity of the components used in model development and deployment.
  • Maintain strict version control to detect and prevent unauthorized changes to libraries or model artifacts.
  • Implement anomaly detection systems to identify unusual usage patterns that may indicate attempted sabotage or exploitation. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Securing Machine Learning Algorithms, ENISA', 'STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets', 'MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems', 'An Effective and Resilient Backdoor Attack Framework against Deep Neural Networks and Vision Transformers'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Securing Machine Learning Algorithms, ENISA STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets Stride-ML Threat Model MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems An Effective and Resilient Backdoor Attack Framework against Deep Neural Networks and Vision Transformers | | threatif | No |

Model Evasion

Evasion attacks involve modifying the input data to evade detection or classification by the model. These attacks can be used to bypass security systems, such as intrusion detection systems or spam filters. Example: Specific malware is crafted to avoid being flagged by a machine-learning-based antivirus.

Threat-modeling question: Is our AI model resilient to evasion attacks?

Internal MISP references

UUID 0ba17a74-54ee-5bca-9012-d34628ba46f8 which can be used as unique global reference for Model Evasion in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity']
category_id 4
cia ['i']
colour #bdd895
external_id PLOT4AI-043
phases ['Design', 'Model', 'Output', 'Monitor']
primary_category Cybersecurity
question Is our AI model resilient to evasion attacks?
recommendation * Develop anomaly detection systems to monitor deviations in input distributions and flag suspicious patterns.
Related clusters

To see the related clusters, click here.

Poisoning Attacks

In a poisoning attack, the goal of the attacker is to contaminate the training data or the model generated in the training phase, so that predictions on new data will be modified in the testing phase. This attack could also be caused by insiders. Example: in a medical dataset where the goal is to predict the dosage of a medicine using demographic information, researchers introduced malicious samples at 8% poisoning rate, which changed the dosage by 75.06% for half of the patients.

Other scenarios: * Data tampering: Actors like AI/ML designers and engineers can deliberately or unintentionally manipulate and expose data. Data can also be manipulated during the storage procedure and by means of some processes like feature selection. Besides interfering with model inference, this type of threat can also bring severe discriminatory issues by introducing bias. Source: ENISA * An attacker who knows how a raw data filtration scheme is set up may be able to leverage that knowledge into malicious input later in system deployment. Source:BerryVilleiML * Adversaries may fine-tune hyper-parameters and thus influence the AI system’s behavior. Hyper-parameters can be a vector for accidental overfitting. In addition, hard to detect changes to hyper-parameters would make an ideal insider attack. Source: ENISA

Threat-modeling question: Are we protected from poisoning attacks?

Internal MISP references

UUID 5d6df7ca-c0e7-5530-9dce-22a92e7c103a which can be used as unique global reference for Poisoning Attacks in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity']
category_id 4
cia ['i']
colour #bdd895
external_id PLOT4AI-044
phases ['Input', 'Model', 'Deploy', 'Monitor']
primary_category Cybersecurity
question Are we protected from poisoning attacks?
recommendation * Define anomaly sensors to look at data distribution on a day to day basis and alert on variations.
Related clusters

To see the related clusters, click here.

Model Inversion

  • In a model inversion attack, if attackers already have access to some personal data belonging to specific individuals included in the training data, they can infer further personal information about those same individuals by observing the inputs and outputs of the ML model.
  • In model inversion the private features used in machine learning models can be recovered. This includes reconstructing private training data that the attacker should not have access to. Example: an attacker recover private features used by the model through careful queries.

Threat-modeling question: Are we protected from model inversion attacks?

Internal MISP references

UUID c77e4d32-875b-571f-abe3-de1b6cfc80a4 which can be used as unique global reference for Model Inversion in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity']
category_id 4
cia ['c']
colour #bdd895
external_id PLOT4AI-045
phases ['Model', 'Deploy', 'Monitor']
primary_category Cybersecurity
question Are we protected from model inversion attacks?
recommendation * Interfaces to models trained with sensitive data need strong access control.
Related clusters

To see the related clusters, click here.

Membership Inference

In a membership inference attack (MIA), the attacker can determine whether a given data record was part of the model’s training dataset or not. Example: researchers were able to predict a patient’s main procedure (e.g., surgery the patient went through) based on the attributes (e.g., age, gender, hospital).

Threat-modeling question: Are we protected from membership inference attacks?

Internal MISP references

UUID 2f735ad0-22a9-5ae6-9151-77886408a028 which can be used as unique global reference for Membership Inference in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity']
category_id 4
cia ['c']
colour #bdd895
external_id PLOT4AI-046
phases ['Model', 'Deploy', 'Monitor']
primary_category Cybersecurity
question Are we protected from membership inference attacks?
recommendation * Differential Privacy has been shown to be an effective mitigation in some studies.
Related clusters

To see the related clusters, click here.

Model Stealing

In model stealing, the attackers can recreate the underlying model by legitimately querying the model. The functionality of the new model is the same as that of the underlying model. Example: in the BigML case, researchers were able to recover the model used to predict if someone should have a good/bad credit risk using 1,150 queries and within 10 minutes.

Threat-modeling question: Are we protected from model stealing attacks?

Internal MISP references

UUID 0f0da0cf-9922-55ee-9d1d-8cee4528fb8b which can be used as unique global reference for Model Stealing in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity']
category_id 4
cia ['c']
colour #bdd895
external_id PLOT4AI-047
phases ['Deploy', 'Monitor']
primary_category Cybersecurity
question Are we protected from model stealing attacks?
recommendation * Minimize or obfuscate the details returned in prediction APIs while still maintaining their usefulness to 'honest' applications.
Related clusters

To see the related clusters, click here.

DNN Attacks

Specially crafted queries from an adversary can reprogram machine learning systems to a task that deviates from the creator’s original intent. Example: ImageNet, a system used to classify one of several categories of images was repurposed to count squares.

Threat-modeling question: Are we protected from reprogramming deep neural nets attacks?

Internal MISP references

UUID d64c2a79-5d32-5707-a5f5-f3f54a4b29ec which can be used as unique global reference for DNN Attacks in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity']
category_id 4
cia ['i']
colour #bdd895
external_id PLOT4AI-048
phases ['Deploy', 'Monitor']
primary_category Cybersecurity
question Are we protected from reprogramming deep neural nets attacks?
recommendation * Configure a strong client-server mutual authentication and access control to model interfaces.

Adversarial Examples

Adversarial examples are a type of evasion attack where malicious inputs are deliberately crafted to mislead AI models. These inputs are minimally modified, often imperceptible to humans, but can cause the model to produce incorrect or harmful predictions. Examples include researchers demonstrating that carefully designed patterns on accessories, like sunglasses, could deceive facial recognition systems into misidentifying individuals. Such examples are particularly problematic in critical domains like healthcare, finance, and security, where incorrect predictions could lead to severe consequences.

Threat-modeling question: Are we protected from adversarial examples?

Internal MISP references

UUID d88c17bf-39b3-549b-af8d-ca1a97fed53f which can be used as unique global reference for Adversarial Examples in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity']
category_id 4
cia ['i']
colour #bdd895
external_id PLOT4AI-049
phases ['Model', 'Deploy', 'Monitor']
primary_category Cybersecurity
question Are we protected from adversarial examples?
recommendation * Include adversarial examples in the training data to make models more robust against similar attacks.
Related clusters

To see the related clusters, click here.

AI Supply Chain Access

  • Malicious ML providers could query the model used by a customer and recover the customer’s training data. If the training process is either fully or partially outsourced to a malicious third party, this one could provide the user with a trained model that contains a backdoor.

Threat-modeling question: Could third-party AI/ML providers compromise our training data or insert backdoors?

Internal MISP references

UUID 03dce20b-b9fa-5d6f-a132-d2ad2bee778f which can be used as unique global reference for AI Supply Chain Access in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity']
category_id 4
cia ['c', 'i']
colour #bdd895
external_id PLOT4AI-050
phases ['Design', 'Input', 'Model', 'Monitor']
primary_category Cybersecurity
question Could third-party AI/ML providers compromise our training data or insert backdoors?
recommendation * Research papers demonstrating the viability of this attack indicate Homomorphic Encryption could be an effective mitigation.
Related clusters

To see the related clusters, click here.

Jailbreaking

Attackers can exploit jailbreak techniques to bypass an AI system’s built-in safety constraints, enabling it to generate restricted or harmful content.

  • Instruction Manipulation: Attackers can craft prompts that trick AI models into breaking content restrictions by rephrasing or disguising requests.
  • Contextual Exploitation: Some jailbreak techniques work by introducing misleading context that influences the AI’s behavior.
  • Adversarial Fine-Tuning: Attackers can modify AI models or create fine-tuned versions that remove ethical constraints.

Threat-modeling question: Could the AI system be vulnerable to jailbreak techniques, allowing attackers to bypass safety restrictions?

Internal MISP references

UUID 3133cdca-0041-5e0a-8bde-fbe567e1aaea which can be used as unique global reference for Jailbreaking in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Generative']
categories ['Cybersecurity']
category_id 4
cia ['c', 'i']
colour #bdd895
external_id PLOT4AI-051
phases ['Design', 'Input', 'Output', 'Deploy', 'Monitor']
primary_category Cybersecurity
question Could the AI system be vulnerable to jailbreak techniques, allowing attackers to bypass safety restrictions?
recommendation * Use reinforcement learning with human feedback (RLHF) to harden AI models against jailbreak exploits.
  • Deploy dynamic prompt filtering to detect and block malicious jailbreak attempts in real-time.
  • Implement multi-layer safety protocols, ensuring that AI models reject unsafe requests consistently.
  • Regularly update safety mechanisms to adapt to emerging jailbreak techniques.
  • Conduct red team assessments to test AI resilience against adversarial jailbreak tactics. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems | | threatif | Yes |
Related clusters

To see the related clusters, click here.

Prompt Injection

AI models, particularly large language models (LLMs), are susceptible to prompt injection attacks, where adversaries craft inputs designed to override model constraints, extract sensitive data, or manipulate system behavior.

  • Meta Prompt Extraction: Attackers can manipulate prompts to reveal system instructions, policies, or proprietary data.
  • Indirect Injection Attacks: If an AI model ingests untrusted external content, such as the contents or names of uploaded files, text from emails, chat inputs, or web pages, attackers can embed hidden prompts or malicious instructions within these elements. These indirect inputs can exploit the model's processing logic to alter its behavior, produce misleading responses, or trigger unauthorized actions, even without direct access to the model's interface.
  • System Command Override: Specially crafted prompts could trick AI models into executing unintended actions or disclosing confidential information.

Threat-modeling question: Could the AI system be vulnerable to prompt injection attacks, leading to unauthorized access or manipulation?

Internal MISP references

UUID c95f6f8d-b1dd-55d7-b77a-100c9e131313 which can be used as unique global reference for Prompt Injection in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Generative']
categories ['Cybersecurity', 'Privacy & Data Protection', 'Safety & Environmental Impact']
category_id 4
cia ['c', 'i']
colour #bdd895
external_id PLOT4AI-052
phases ['Design', 'Input', 'Model', 'Output', 'Deploy', 'Monitor']
primary_category Cybersecurity
question Could the AI system be vulnerable to prompt injection attacks, leading to unauthorized access or manipulation?
recommendation * Use input validation and sanitization to detect and neutralize malicious prompts.
  • Implement adversarial training to harden the AI against prompt injection attacks.
  • Limit the AI’s ability to access sensitive system instructions or proprietary data through context isolation.
  • Avoid executing model-generated outputs directly without human or automated validation. Treat model output as untrusted data, don't execute it as code or commands.
  • Monitor AI interactions in real-time to detect anomalous behaviors and injection attempts.
  • Regularly test AI models using red teaming to identify and patch vulnerabilities in prompt handling. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems', 'Rethinking Prompt Injection Prevention – Don’t Execute the Data'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems Rethinking Prompt Injection Prevention – Don’t Execute the Data | | threatif | Yes |
Related clusters

To see the related clusters, click here.

Environment Unauthorized Access

AI training environments often handle sensitive data and require extensive computational resources. If left unprotected, they become a target for adversaries who may attempt to steal data, modify training sets, or inject adversarial inputs.

  • Unauthorized Access to Training Data: Malicious actors could exfiltrate sensitive training datasets, leading to data leaks or compliance violations.
  • Model Poisoning & Integrity Attacks: Attackers may inject biased or adversarial data into the training process, leading to degraded or manipulated AI outputs.
  • Infrastructure Vulnerabilities: Misconfigured cloud environments or weak authentication mechanisms could expose training pipelines to external threats.

Threat-modeling question: Is the AI training environment secured against unauthorized access and manipulation?

Internal MISP references

UUID bf07b6f6-abb8-5107-b429-3132320ad611 which can be used as unique global reference for Environment Unauthorized Access in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity']
category_id 4
cia ['c', 'i', 'a']
colour #bdd895
external_id PLOT4AI-053
phases ['Design', 'Input', 'Model', 'Deploy', 'Monitor']
primary_category Cybersecurity
question Is the AI training environment secured against unauthorized access and manipulation?
recommendation * Implement strict access controls and role-based permission for training environments.
  • Use end-to-end encryption for training data to prevent unauthorized interception.
  • Deploy secure multi-party computation (SMPC) and homomorphic encryption to protect sensitive datasets.
  • Regularly audit and monitor training infrastructure for security vulnerabilities.
  • Adopt sandboxed environments to isolate training processes and prevent malicious tampering. | | roles | ['Provider'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems | | threatif | No |

System Unauthorized Access

Unauthorized access to AI systems can result in data breaches, model theft, and exploitation of sensitive functionalities. Without proper access control, attackers can extract model parameters, manipulate system behavior, or leak confidential data.

  • Credential & API Key Exposure: Weak authentication mechanisms can lead to unauthorized access, allowing attackers to exploit API endpoints or modify AI responses.
  • Model Extraction Attacks: Attackers can systematically query an AI system to recreate and steal proprietary models, leading to intellectual property theft.
  • Privilege Escalation Risks: Poorly managed user roles and permissions may allow attackers to escalate access, gaining control over critical AI operations.

Threat-modeling question: Is the deployed AI system protected from unauthorized access and misuse?

Internal MISP references

UUID 282275c3-e2b6-55f6-af82-f74ba934dc68 which can be used as unique global reference for System Unauthorized Access in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity']
category_id 4
cia ['c', 'i', 'a']
colour #bdd895
external_id PLOT4AI-054
phases ['Design', 'Deploy', 'Monitor']
primary_category Cybersecurity
question Is the deployed AI system protected from unauthorized access and misuse?
recommendation * Enforce multi-factor authentication (MFA) and strong password policies for AI system access.
  • Restrict API access using role-based access control (RBAC) and least privilege principles.
  • Monitor AI usage logs for anomalous access patterns and potential security breaches.
  • Apply rate limiting and query monitoring to detect and mitigate model extraction attacks.
  • Use secure enclaves and differential privacy to protect sensitive AI models and training data. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems | | threatif | No |

AI Supply Chain Tools

Modern AI systems increasingly rely on external tools and plugin interfaces (e.g., Model Context Protocol, LangChain, OpenAI plugins) to expand their capabilities. These interfaces pose unique security risks if not tightly controlled.

Runtime Abuse: If tool or plugin inputs are not strictly validated, LLMs may: * Trigger unauthorized tool executions. * Bypass guardrails using structured payloads embedded in plugin responses. * Chain outputs across tools in unsafe ways (e.g., generating code that another tool executes).

Supply Chain Risks: Third-party plugins and dependencies may contain vulnerabilities or backdoors. Attackers can: * Compromise plugin registries or repositories. * Compromise dependencies such as AI agent containers or monitoring components to inject malicious code, potentially infecting production systems, disrupting the AI deployment environment, and undermining the integrity of system monitoring. * Tamper with pre-trained models or updates during distribution.

These risks are magnified in open ecosystems where tools are crowd-sourced or rapidly integrated without full vetting.

Threat-modeling question: Could third-party tools, plugins, or dependencies introduce vulnerabilities in our AI system?

Internal MISP references

UUID 8134b398-4da9-55a0-a553-27e03389ce38 which can be used as unique global reference for AI Supply Chain Tools in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity']
category_id 4
cia ['c', 'i', 'a']
colour #bdd895
external_id PLOT4AI-055
phases ['Design', 'Deploy', 'Monitor']
primary_category Cybersecurity
question Could third-party tools, plugins, or dependencies introduce vulnerabilities in our AI system?
recommendation * Use strict schemas (e.g., OpenAPI, JSON Schema) and validate all tool/plugin inputs and outputs.
Related clusters

To see the related clusters, click here.

Unsafe SQL

  • LLMs integrated with backend systems may generate SQL queries based on user input, exposing the system to SQL injection attacks. If input prompts are not properly validated or sanitized, attackers may inject malicious SQL fragments into natural language inputs, which the LLM translates into executable queries.
  • These vulnerabilities are often underestimated due to misplaced trust in the AI’s output or assumptions that the AI understands secure coding practices. In reality, models may generate insecure or dangerous SQL if prompted accordingly.
  • This risk is particularly severe in domains like finance or healthcare, where AI-generated queries could expose sensitive records or enable privilege escalation.

Threat-modeling question: Could the AI system generate or execute unsafe SQL queries from user input?

Internal MISP references

UUID 7ce51889-2a12-5cc5-9f57-21b6ea6fe022 which can be used as unique global reference for Unsafe SQL in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Generative']
categories ['Cybersecurity']
category_id 4
cia ['c', 'i', 'a']
colour #bdd895
external_id PLOT4AI-056
phases ['Design', 'Input', 'Output', 'Deploy', 'Monitor']
primary_category Cybersecurity
question Could the AI system generate or execute unsafe SQL queries from user input?
recommendation * Never execute AI-generated SQL directly. Use intermediate layers that validate and parameterize AI-generated queries.
  • Sanitize all user inputs before allowing them to reach the LLM.
  • Apply query allow-lists, parameterized queries, and database permissions to constrain what LLMs can do.
  • Use static and dynamic code analysis on AI-generated queries before execution.
  • Educate developers and product teams about the unique risks of LLM-driven SQL generation. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | From Prompt Injections to SQL Injection Attacks: How Protected is Your LLM-Integrated Web Application? | | threatif | Yes |

Remote Code Execution (RCE)

  • LLMs capable of code generation (e.g., math solvers, dev assistants) may be exploited to generate and execute malicious code if user input is not properly isolated.
  • Adversaries can craft prompts that cause the model to generate harmful code, such as importing modules, writing to disk, or leaking environment variables. If this code is executed directly (e.g., in a math or scripting agent), the attacker may achieve Remote Code Execution (RCE).
  • Case studies such as MathGPT demonstrate how seemingly benign capabilities (e.g., formula evaluation) can be weaponized to access server resources or keys.

Threat-modeling question: Could the AI system generate or execute unsafe code based on user input?

Internal MISP references

UUID fff497cc-ed99-52c2-b6f0-6a138efe04de which can be used as unique global reference for Remote Code Execution (RCE) in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Generative']
categories ['Cybersecurity']
category_id 4
cia ['c', 'i', 'a']
colour #bdd895
external_id PLOT4AI-057
phases ['Design', 'Input', 'Deploy', 'Monitor']
primary_category Cybersecurity
question Could the AI system generate or execute unsafe code based on user input?
recommendation * Never run AI-generated code in the same environment as your application backend.
  • Use containerization (e.g., Docker) with strict sandboxing, network isolation, and resource limits for code execution.
  • Inspect AI-generated code before execution, and apply static analysis tools to flag dangerous patterns.
  • Implement output sanitization to prevent exfiltration of sensitive data.
  • Disable or severely limit code execution features unless explicitly required. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Demystifying RCE Vulnerabilities in LLM-Integrated Apps', 'I Hacked MathGPT: RCE Vulnerability'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Demystifying RCE Vulnerabilities in LLM-Integrated Apps I Hacked MathGPT: RCE Vulnerability | | threatif | Yes |

Agentic AI Interaction

  • AI agents that browse the internet or invoke external APIs may inadvertently land on or interact with malicious websites. These pages may host malware, deceptive interfaces, or phishing payloads designed to compromise the AI system or extract sensitive data.
  • The risk is amplified when agents operate autonomously or chain multiple tools (e.g., browsers, file downloaders, LLMs) without strict boundaries, potentially triggering harmful scripts or revealing internal state.

Threat-modeling question: Could autonomous AI agents access or interact with malicious web content?

Internal MISP references

UUID 062c1b36-2bde-5abf-af3d-5b7e6e134f31 which can be used as unique global reference for Agentic AI Interaction in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Generative']
categories ['Cybersecurity', 'Safety & Environmental Impact']
category_id 4
cia ['c', 'i']
colour #bdd895
external_id PLOT4AI-058
phases ['Design', 'Input', 'Output', 'Deploy', 'Monitor']
primary_category Cybersecurity
question Could autonomous AI agents access or interact with malicious web content?
recommendation * Apply strict domain allow-lists and restrict browsing to pre-approved sources.
  • Disable JavaScript, downloads, or plugin execution in browser environments.
  • Monitor and log all external interactions for anomalous behavior.
  • Use URL and content scanning before any AI agent accesses external resources.
  • Employ a retrieval proxy to intermediate and sanitize third-party web content before it is passed to the agent. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['OWASP Agentic AI – Threats and Mitigations (v1.0.1)'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | OWASP Agentic AI – Threats and Mitigations (v1.0.1) | | threatif | Yes |

Agentic AI Memory

  • Agentic systems with persistent memory can be manipulated over time by injecting false, biased, or adversarial content. This may alter future reasoning, planning, or tool use. For example, a user might insert misleading facts into a chatbot's memory, resulting in hallucinations or dangerous outputs later on.
  • Long-term memory makes these risks cumulative and harder to detect.

Threat-modeling question: Could agent memory be poisoned with malicious or misleading information?

Internal MISP references

UUID 40163ecd-caef-5888-ac6d-fc5169248785 which can be used as unique global reference for Agentic AI Memory in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Generative']
categories ['Cybersecurity', 'Safety & Environmental Impact']
category_id 4
cia ['i']
colour #bdd895
external_id PLOT4AI-059
phases ['Deploy', 'Monitor']
primary_category Cybersecurity
question Could agent memory be poisoned with malicious or misleading information?
recommendation * Limit write access to memory: only trusted or validated agents/users should modify persistent memory.
  • Implement memory sanitation, validation, and confidence scoring.
  • Provide mechanisms to audit memory entries and detect unusual patterns.
  • Isolate memory by task or session where feasible to limit long-term contamination. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['OWASP Agentic AI – Threats and Mitigations (v1.0.1)'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | OWASP Agentic AI – Threats and Mitigations (v1.0.1) | | threatif | Yes |

Agentic AI Misuse of Tools

Agents that have access to tools (e.g., file systems, webhooks, APIs) may invoke them in unintended or harmful ways. This misuse can result from adversarial prompts, faulty reasoning, or misunderstood intent. Example: an agent with access to a web browser could issue API delete requests or trigger real-world effects in connected systems.

Threat-modeling question: Could agents misuse tools or APIs they are authorized to access?

Internal MISP references

UUID d77d1ec7-eb5a-5140-94de-b75db2b9995f which can be used as unique global reference for Agentic AI Misuse of Tools in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Generative']
categories ['Cybersecurity', 'Safety & Environmental Impact']
category_id 4
cia ['c', 'i', 'a']
colour #bdd895
external_id PLOT4AI-060
phases ['Design', 'Deploy', 'Monitor']
primary_category Cybersecurity
question Could agents misuse tools or APIs they are authorized to access?
recommendation * Use allow-lists to tightly control which tools an agent can access.
  • Apply RBAC or contextual constraints (e.g., only allow file writes for task X).
  • Monitor tool use patterns and block anomalous calls.
  • Require human-in-the-loop confirmation for high-risk tool use. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['OWASP Agentic AI – Threats and Mitigations (v1.0.1)'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | OWASP Agentic AI – Threats and Mitigations (v1.0.1) | | threatif | Yes |

Agentic AI Hallucinations

In multi-agent systems, one agent’s hallucinated output can become another’s input. This can cause cascading misinformation, particularly if agents defer to each other’s outputs without validation. Example: Agent A misclassifies a vulnerability, Agent B acts on this and takes inappropriate mitigation actions.

Threat-modeling question: Could hallucinated output from one agent propagate and mislead others in multi-agent systems?

Internal MISP references

UUID a36bfc3f-68a4-5276-acdf-ef5de75a1c52 which can be used as unique global reference for Agentic AI Hallucinations in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Generative']
categories ['Cybersecurity', 'Safety & Environmental Impact']
category_id 4
cia ['i']
colour #bdd895
external_id PLOT4AI-061
phases ['Model', 'Deploy', 'Monitor']
primary_category Cybersecurity
question Could hallucinated output from one agent propagate and mislead others in multi-agent systems?
recommendation * Require independent validation or confidence scoring for agent-to-agent communication.
  • Avoid blind trust between agents; implement verification protocols to ensure accuracy.
  • Implement mechanisms to trace provenance of information across agents.
  • Regularly retrain agents on hallucination-resistant architectures and factual QA tasks. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['OWASP Agentic AI – Threats and Mitigations (v1.0.1)'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | OWASP Agentic AI – Threats and Mitigations (v1.0.1) | | threatif | Yes |
Related clusters

To see the related clusters, click here.

Agentic AI Actions Traceability

  • Without strong logging and traceability, it becomes difficult to audit or understand decisions made by autonomous agents.
  • This increases the risk of undetected errors, malicious actions, and limits post-incident forensics.
  • Repudiation becomes likely when actions cannot be linked to responsible entities (agent or user).

Threat-modeling question: Can we trace and audit the actions and decisions of autonomous agents in our system?

Internal MISP references

UUID c50a3206-2436-552b-b602-18d9e8157847 which can be used as unique global reference for Agentic AI Actions Traceability in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Generative']
categories ['Cybersecurity', 'Accountability & Human Oversight', 'Data & Data Governance']
category_id 4
cia ['i', 'a']
colour #bdd895
external_id PLOT4AI-062
phases ['Deploy', 'Monitor']
primary_category Cybersecurity
question Can we trace and audit the actions and decisions of autonomous agents in our system?
recommendation * Log all agent actions, tool uses, memory writes, and external interactions.
  • Implement immutable audit trails.
  • Assign unique identifiers to agents and their outputs.
  • Use cryptographic signing for sensitive agent actions to support accountability. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['OWASP Agentic AI – Threats and Mitigations (v1.0.1)'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | OWASP Agentic AI – Threats and Mitigations (v1.0.1) | | threatif | No |

Agentic AI Malicious Agent

  • In decentralized or loosely coordinated agentic systems, a single compromised agent can act against the intended goals of the system.
  • Rogue agents may inject misleading information, sabotage coordination, or exploit trust relationships.
  • These threats are especially relevant in federated learning, autonomous swarm systems, or large-scale multi-agent deployments.

Threat-modeling question: Could a compromised or malicious agent sabotage a multi-agent system?

Internal MISP references

UUID 766ff02d-fbd4-51ae-88c3-57d8b5a849b4 which can be used as unique global reference for Agentic AI Malicious Agent in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Generative']
categories ['Cybersecurity']
category_id 4
cia ['c', 'i', 'a']
colour #bdd895
external_id PLOT4AI-063
phases ['Deploy', 'Monitor']
primary_category Cybersecurity
question Could a compromised or malicious agent sabotage a multi-agent system?
recommendation * Implement agent authentication and authorization protocols.
  • Monitor agent outputs for inconsistencies or divergence from assigned tasks.
  • Apply anomaly detection to communication and behavior across agents.
  • Quarantine or disable agents that exhibit deviant or suspicious activity. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['OWASP Agentic AI – Threats and Mitigations (v1.0.1)'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | OWASP Agentic AI – Threats and Mitigations (v1.0.1) | | threatif | Yes |

Agentic AI Unauthorized Access

  • Agents may discover or exploit misconfigurations to access privileged tools, APIs, or data.
  • Escalation can result from incorrect role assignments, prompt manipulation, or logic flaws in access validation.
  • This could lead to the agent invoking destructive actions or leaking confidential data.

Threat-modeling question: Could an agent gain access to functions or data beyond its intended permissions?

Internal MISP references

UUID 5378bb33-f53d-5836-97c1-6c7a38c32a13 which can be used as unique global reference for Agentic AI Unauthorized Access in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Generative']
categories ['Cybersecurity', 'Privacy & Data Protection']
category_id 4
cia ['c', 'i', 'a']
colour #bdd895
external_id PLOT4AI-064
phases ['Design', 'Deploy', 'Monitor']
primary_category Cybersecurity
question Could an agent gain access to functions or data beyond its intended permissions?
recommendation * Apply least privilege principles and context-aware access controls to agent capabilities.
  • Regularly audit role definitions and permissions assigned to agents.
  • Include privilege escalation scenarios in red-teaming and testing efforts.
  • Use runtime guards to detect and block unauthorized function calls. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['OWASP Agentic AI – Threats and Mitigations (v1.0.1)'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | OWASP Agentic AI – Threats and Mitigations (v1.0.1) | | threatif | Yes |

Resource Overload

  • AI systems, especially large models, are resource-intensive and vulnerable to overload attacks targeting compute, memory, data infrastructures, I/O, or API bandwidth.
  • Malicious actors may send high-frequency or computationally expensive queries to exhaust system capacity.
  • This can lead to degraded service, denial of service, or delayed model responses, impacting availability and user trust.
  • Multi-agent environments are particularly vulnerable when agents interact recursively or generate long-running tasks without resource limits.

Threat-modeling question: Could an attacker or user intentionally overload the AI system’s resources to degrade performance or cause failures?

Internal MISP references

UUID e3cb72d5-124b-5c3f-8367-a853bf26e7d4 which can be used as unique global reference for Resource Overload in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity', 'Safety & Environmental Impact']
category_id 4
cia ['a', 'i']
colour #bdd895
external_id PLOT4AI-065
phases ['Deploy', 'Monitor']
primary_category Cybersecurity
question Could an attacker or user intentionally overload the AI system’s resources to degrade performance or cause failures?
recommendation * Implement rate limiting and quotas per user, agent, or session to restrict excessive usage.
  • Use priority-based scheduling, timeouts, and request throttling for costly model operations.
  • Monitor runtime metrics (CPU/GPU load, memory, inference time) and trigger alerts for anomalies.
  • Apply load balancing and autoscaling in production to absorb usage spikes.
  • Include safeguards in agent instructions to prevent recursive or resource-exhausting task loops.
  • Log resource-heavy requests and investigate patterns indicative of misuse or attack. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['OWASP Agentic AI Threats'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | OWASP Agentic AI Threats | | threatif | Yes |

Identity Spoofing & Impersonation

  • Attackers may spoof legitimate identities of users, agents, or services to bypass authentication and gain control or influence over AI behavior.
  • This can enable unauthorized data access, prompt injection, or manipulation of trust-based systems.

Threat-modeling question: Could an attacker or agent impersonate a user or AI identity to gain unauthorized influence?

Internal MISP references

UUID 7f8e5df0-4a78-5652-9099-b8447e3b0495 which can be used as unique global reference for Identity Spoofing & Impersonation in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity']
category_id 4
cia ['i', 'a']
colour #bdd895
external_id PLOT4AI-066
phases ['Deploy', 'Monitor']
primary_category Cybersecurity
question Could an attacker or agent impersonate a user or AI identity to gain unauthorized influence?
recommendation * Use cryptographic signatures or authentication tokens to verify agent identities.
  • Implement mutual authentication in multi-agent or AI-human interaction scenarios.
  • Monitor for identity anomalies such as session hijacking, mismatched tokens, or unexpected behavioral patterns.
  • Log all identity transitions and access attempts to support traceability and forensic analysis. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['OWASP Agentic AI – Threats and Mitigations (v1.0.1)'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | OWASP Agentic AI – Threats and Mitigations (v1.0.1) | | threatif | Yes |

Misuse of AI Agents to Deceive Users

  • Agentic systems capable of persuasive language or personalized interaction can be manipulated to influence human decisions, emotions, or behaviors.
  • This creates risks of social engineering, phishing, misinformation, or undue influence, especially if the agent mimics authority figures or trusted personas.
  • The risk is amplified when agents use persistent memory or learn user preferences over time.

Threat-modeling question: Could an agent be misused to manipulate or deceive users?

Internal MISP references

UUID 3e2be1eb-67de-5dd8-9e51-b414baba97fb which can be used as unique global reference for Misuse of AI Agents to Deceive Users in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity', 'Ethics & Human Rights']
category_id 4
cia ['c', 'i']
colour #bdd895
external_id PLOT4AI-067
phases ['Deploy', 'Monitor']
primary_category Cybersecurity
question Could an agent be misused to manipulate or deceive users?
recommendation * Impose ethical use constraints and define red lines (e.g., no impersonation, no medical/legal advice without oversight).
  • Use transparency mechanisms to disclose when users are interacting with agents.
  • Enable user control and opt-out of persuasive or adaptive behaviors.
  • Monitor for behavior that resembles coercion, manipulation, or impersonation. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['OWASP Agentic AI – Threats and Mitigations (v1.0.1)'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | OWASP Agentic AI – Threats and Mitigations (v1.0.1) | | threatif | Yes |

Agent Communication Tampering

Agents that exchange messages may be vulnerable to communication poisoning, where an attacker injects or modifies messages to alter system behavior. This can mislead agents, propagate misinformation, or trigger unintended actions in chained workflows. Examples include impersonating an agent, sending conflicting commands, or embedding adversarial prompts.

Threat-modeling question: Could an attacker intercept or manipulate communications between agents to alter system behavior?

Internal MISP references

UUID 660736e9-1516-582d-af59-15a0f602daaa which can be used as unique global reference for Agent Communication Tampering in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity']
category_id 4
cia ['c', 'i']
colour #bdd895
external_id PLOT4AI-068
phases ['Deploy', 'Monitor']
primary_category Cybersecurity
question Could an attacker intercept or manipulate communications between agents to alter system behavior?
recommendation * Authenticate all agent-to-agent messages.
  • Use encryption and integrity checks to prevent tampering.
  • Log and analyze communication flows to detect unusual patterns.
  • Limit what kinds of messages agents can send and which agents can receive them. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['OWASP Agentic AI – Threats and Mitigations (v1.0.1)'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | OWASP Agentic AI – Threats and Mitigations (v1.0.1) | | threatif | Yes |

File Upload

AI systems that ingest or process uploaded files, such as PDFs, Word documents, images, or code, are vulnerable to multiple attack vectors:

  • Malware & Embedded Scripts: Uploaded files may contain malicious payloads, macros, or hidden code that executes during parsing or rendering.
  • Indirect Prompt Injection: Hidden instructions embedded in file content can manipulate LLM behavior when the content is passed as input for summarization, Q&A, or reasoning.
  • Malformed Files & Deserialization: Crafted file formats can trigger crashes or bypass input validation, potentially leading to remote code execution or model corruption.

These threats are particularly relevant when files are processed automatically by LLMs or downstream tools, often without human review.

Threat-modeling question: Could unsafe file uploads introduce security risks?

Internal MISP references

UUID 9df54d47-a463-5ccc-8aae-f56f038b9f5c which can be used as unique global reference for File Upload in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity']
category_id 4
cia ['c', 'i', 'a']
colour #bdd895
external_id PLOT4AI-069
phases ['Input', 'Model', 'Deploy', 'Monitor']
primary_category Cybersecurity
question Could unsafe file uploads introduce security risks?
recommendation * Validate file types, sizes, and content strictly, use allowlists and reject unsupported or dangerous formats.
  • Sanitize and normalize file content before passing it to downstream components or LLMs.
  • Scan all files for malware using antivirus and static analysis tools.
  • Avoid feeding raw file content directly to language models, wrap it with safety context and monitor outputs.
  • Use sandboxed or containerized environments for file parsing, summarization, or code execution.
  • Monitor for patterns of indirect prompt injection in document content.
  • If supporting file-based inputs in a RAG pipeline or agentic system, implement retrieval sanitation and memory protection. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['OWASP - Unrestricted File Upload', 'ProtectAI - ModelScan'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | OWASP - Unrestricted File Upload ProtectAI - ModelScan | | threatif | Yes |

Model Serialization

Models are serialized and transferred between systems for deployment, a stage vulnerable to model serialization attacks. Models are often serialized for storage, sharing, or deployment, using formats like pickle, joblib, ONNX, or TensorFlow SavedModel. However, many serialization formats can embed executable code or unsafe object structures.

If an attacker tampers with a serialized model artifact and it is later deserialized without validation, they may achieve: * Remote Code Execution (RCE) during deserialization. * Privilege escalation or lateral movement inside the deployment environment. * Tampering with model behavior (e.g., inserting a backdoor or triggering silent failures).

These risks are especially severe when models are downloaded from untrusted sources, integrated via ML pipelines, or auto-loaded during CI/CD processes.

Threat-modeling question: Could unsafe deserialization of model artifacts lead to code execution or system compromise?

Internal MISP references

UUID e00f1c63-f2e5-5e37-bac9-b071965df2a1 which can be used as unique global reference for Model Serialization in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Cybersecurity']
category_id 4
cia ['c', 'i', 'a']
colour #bdd895
external_id PLOT4AI-070
phases ['Input', 'Model', 'Deploy', 'Monitor']
primary_category Cybersecurity
question Could unsafe deserialization of model artifacts lead to code execution or system compromise?
recommendation * Avoid unsafe deserialization methods on untrusted inputs, prefer safer formats.
  • Use model scanning tools to detect malicious payloads in serialized artifacts.
  • Enforce cryptographic signing and integrity checks for all model files before deployment.
  • Store and transport models using secure channels (e.g., signed, encrypted artifact registries).
  • Load models only in sandboxed or containerized environments with minimal privileges and no internet access.
  • Track model provenance throughout the development lifecycle to detect unauthorized changes. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['OWASP - Unrestricted File Upload', 'ProtectAI - ModelScan'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | OWASP - Unrestricted File Upload ProtectAI - ModelScan | | threatif | Yes |
Related clusters

To see the related clusters, click here.

Fine-tuning Attacks

  • Adversaries can fine-tune or subtly manipulate your LLM using harmful data, leading to unsafe, biased, or deceptive behaviors.
  • Common fine-tuning attacks include:
  • Instruction Manipulation: Injects unsafe instructions into fine-tuning data, teaching the model to follow harmful prompts.
  • Output Manipulation: Poisons target outputs in the fine-tuning data, causing the model to generate malicious or biased responses, even when prompts seem neutral.
  • Backdoor Attacks: Implant hidden triggers during fine-tuning that activate malicious behavior only when specific input patterns appear. The model behaves normally otherwise, making these attacks hard to detect.
  • Alignment Degradation: Subtly erodes the model’s safety alignment during fine-tuning, making it gradually more permissive to unsafe behavior without explicit instructions.
  • Reward Hijacking: Tricks the reward model into preferring harmful outputs, effectively training the model to give unsafe or misleading responses.
  • Semantic Drift: Slightly alters wording or context in fine-tuning data to shift the model’s behavior, causing it to appear aligned while subtly reinforcing harmful stereotypes or unsafe reasoning.
  • These threats can be introduced via fine-tuning-as-a-service platforms, open-source model reuse, or contaminated user-provided datasets.
  • Even small amounts of harmful fine-tuning data can significantly degrade model alignment and safety.

Threat-modeling question: Could malicious fine-tuning compromise the safety or alignment of our GenAI model?

Internal MISP references

UUID b48aedbd-8a33-5420-b4da-23fcd1a94cfd which can be used as unique global reference for Fine-tuning Attacks in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Generative']
categories ['Cybersecurity', 'Safety & Environmental Impact']
category_id 4
cia ['c', 'i', 'a']
colour #bdd895
external_id PLOT4AI-071
phases ['Model', 'Deploy', 'Monitor']
primary_category Cybersecurity
question Could malicious fine-tuning compromise the safety or alignment of our GenAI model?
recommendation * Vet and sanitize fine-tuning datasets, including user-submitted data and third-party sources.
  • Implement anomaly detection and alignment regression tests before and after fine-tuning.
  • Restrict or audit fine-tuning privileges, especially on shared infrastructure or open APIs.
  • Use differential privacy, prompt injection detection, and trigger auditing tools to detect backdoors.
  • Conduct red-teaming to assess the effects of adversarial fine-tuning and monitor for misalignment drift over time. | | roles | ['Provider'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Harmful Fine-tuning Attacks and Defenses for Large Language Models (arXiv:2409.18169)'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Harmful Fine-tuning Attacks and Defenses for Large Language Models (arXiv:2409.18169) | | threatif | Yes |
Related clusters

To see the related clusters, click here.

RAG & Vector Databases

Retrieval-Augmented Generation (RAG) systems combine LLMs with vector databases to enrich answers with external knowledge. However, if the retrieval layer is compromised or poorly validated, it can feed the model misleading, biased, or adversarial content. Untrusted documents in vector stores can serve as indirect prompt injections, while insecure embeddings can allow unauthorized inference or leakage. Additionally, RAG systems may unintentionally disclose proprietary documents retrieved through similarity search.

Threat-modeling question: Are we protected from vulnerabilities in vector databases and RAG pipelines?

Internal MISP references

UUID d76e1508-e655-50dc-8ec6-20bdcde8153a which can be used as unique global reference for RAG & Vector Databases in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Generative']
categories ['Cybersecurity']
category_id 4
cia ['c', 'i']
colour #bdd895
external_id PLOT4AI-072
phases ['Design', 'Input', 'Output', 'Monitor']
primary_category Cybersecurity
question Are we protected from vulnerabilities in vector databases and RAG pipelines?
recommendation * Sanitize retrieved content before feeding it to the LLM.
  • Use document-level access control to prevent unauthorized access during retrieval.
  • Monitor for adversarial inputs and injection attacks embedded in indexed content.
  • Validate the trustworthiness of sources before ingesting documents into the vector DB.
  • Regularly retrain embedding models and limit exposure of semantic search endpoints. | | roles | ['Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['OWASP LLM Top 10 - Prompt Injection in Retrieval Systems'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | OWASP LLM Top 10 - Prompt Injection in Retrieval Systems | | threatif | No |
Related clusters

To see the related clusters, click here.

Input Channel Failure

  • Are these channels trustworthy?
  • What will happen in case of failure?
  • Think for instance about IoT devices used as sensors.

Threat-modeling question: Could failures in real-time data collection channels disrupt model performance?

Internal MISP references

UUID c88dd3f4-b15a-5152-b9aa-707575c3cfa5 which can be used as unique global reference for Input Channel Failure in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Safety & Environmental Impact', 'Cybersecurity', 'Data & Data Governance']
category_id 5
colour #f7f09f
external_id PLOT4AI-073
phases ['Design', 'Input', 'Deploy', 'Monitor']
primary_category Safety & Environmental Impact
question Could failures in real-time data collection channels disrupt model performance?
recommendation * If you are collecting/receiving data from sensors, consider estimating the impact it could have on your model if any of the sensors fail and your input data gets interrupted or corrupted.
  • Sensor blinding attacks are one example of a risk faced by poorly designed input gathering systems. Note that consistent feature identification related to sensors is likely to require human calibration. Source: BerryVilleiML | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['BerryVilleiML'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | threatif | Yes |

Misinformation

AI models may generate hallucinations, producing incorrect, misleading, or fabricated information. These errors can undermine trust, propagate misinformation, and lead to unsafe decision-making.

  • Misinformation Amplification: False information generated by AI could be exploited in disinformation campaigns or lead to incorrect medical, financial, or legal advice.
  • Reinforcement of Biases: AI hallucinations could disproportionately affect marginalized groups, reinforcing biases in generated content.
  • Sycophancy Risk: Some models are prone to agree with users’ views even when incorrect, reinforcing user confirmation bias.
  • Hallucination Types: In hallucinations the outputs can contradict or misalign with the prompt, introduce unrelated or fabricated elements or include factually incorrect statements.

Threat-modeling question: Could AI-generated hallucinations lead to misinformation or decision-making risks?

Internal MISP references

UUID 7fa229ac-8dc2-5258-bdc0-e114a08219d2 which can be used as unique global reference for Misinformation in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Safety & Environmental Impact', 'Cybersecurity', 'Privacy & Data Protection']
category_id 5
colour #f7f09f
external_id PLOT4AI-074
phases ['Design', 'Input', 'Model', 'Output', 'Deploy', 'Monitor']
primary_category Safety & Environmental Impact
question Could AI-generated hallucinations lead to misinformation or decision-making risks?
recommendation * Integrate fact-checking mechanisms that verify AI-generated outputs against authoritative sources.
  • Implement confidence scoring to indicate when AI responses are uncertain or speculative.
  • Deploy human-in-the-loop oversight for high-risk applications like healthcare and legal AI systems.
  • Use AI hallucination monitoring systems to detect and mitigate factually incorrect responses.
  • Train AI models on diverse and verified datasets to reduce knowledge gaps and speculative responses. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems', 'Bontridder N, Poullet Y. The role of artificial intelligence in disinformation. Data & Policy. 2021;3:e32. doi:10.1017/dap.2021.20'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems Bontridder N, Poullet Y. The role of artificial intelligence in disinformation. Data & Policy. 2021;3:e32. doi:10.1017/dap.2021.20 | | threatif | Yes |
Related clusters

To see the related clusters, click here.

Interpretability

  • Lack of interpretability can severely hinder developers’ ability to understand how the model makes decisions, debug failures, identify biases, or ensure alignment with system goals.
  • This is especially critical when integrating complex models like LLMs into downstream applications. Without transparency, it is difficult to detect misalignment, drift, or unsafe emergent behaviors.
  • In high-stakes domains, the inability to interpret models can compromise safety and compliance, particularly if unexplained outputs influence critical decisions.
  • Traditional feature attribution techniques may be insufficient for LLMs and foundation models. Mechanistic interpretability approaches (e.g., circuit analysis, neuron tracing, causal probing) may be necessary for developers to understand internal model behavior.
  • Black-box AI systems reduce the ability to validate updates, perform maintenance, or intervene effectively in case of failure.

Threat-modeling question: Could the lack of interpretability in our AI models compromise safety?

Internal MISP references

UUID 40f3e115-8252-5208-97b6-4ade9ab21761 which can be used as unique global reference for Interpretability in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Safety & Environmental Impact', 'Transparency & Accessibility', 'Accountability & Human Oversight']
category_id 5
colour #f7f09f
external_id PLOT4AI-075
phases ['Design', 'Model', 'Output', 'Deploy', 'Monitor']
primary_category Safety & Environmental Impact
question Could the lack of interpretability in our AI models compromise safety?
recommendation * Use interpretable model architectures when possible (e.g., decision trees, GAMs) or incorporate interpretability scaffolding in complex systems (e.g., chain-of-thought prompting).

Over-reliance (Safety & Environmental Impact)

Relying too heavily on automation can reduce human involvement and oversight, making it difficult to respond quickly or effectively to unexpected failures or emergency situations.

Threat-modeling question: Can human over-reliance on automated systems lead to failures during emergencies?

Internal MISP references

UUID 3d5547ec-bfda-5935-bcfb-472ae10761d8 which can be used as unique global reference for Over-reliance (Safety & Environmental Impact) in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Safety & Environmental Impact', 'Accountability & Human Oversight']
category_id 5
colour #f7f09f
external_id PLOT4AI-076
phases ['Design', 'Model', 'Output', 'Deploy', 'Monitor']
primary_category Safety & Environmental Impact
question Can human over-reliance on automated systems lead to failures during emergencies?
recommendation * Design systems with manual override capabilities and ensure operators are trained to use them effectively.
  • Create scenarios for testing human-AI collaboration under stress conditions.
  • Regularly evaluate the balance between automation and human oversight. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['The Danger of Overreliance on Automation in Cybersecurity', 'Automation and Situation Awareness'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | The Danger of Overreliance on Automation in Cybersecurity Automation and Situation Awareness | | threatif | Yes |

Performance & Scalability

Can your algorithm scale in performance from the data it learned on to real data? In online situations the rate at which data comes into the model may not align with the rate of anticipated data arrival. This can lead to both outright ML system failure and to a system that becomes unstable or exhibits feedback loops. Source: BerryVilleiML

Threat-modeling question: Could performance or reliability issues emerge when scaling the AI system across environments?

Internal MISP references

UUID 735309a7-93aa-51ae-9e9d-37dc526765f4 which can be used as unique global reference for Performance & Scalability in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Safety & Environmental Impact', 'Cybersecurity']
category_id 5
colour #f7f09f
external_id PLOT4AI-077
phases ['Design', 'Input', 'Model', 'Deploy', 'Monitor']
primary_category Safety & Environmental Impact
question Could performance or reliability issues emerge when scaling the AI system across environments?
recommendation * Determine the expected rate of data arrival and test the model under similar conditions.

System Failure

  • Do you have a mechanism implemented to stop the processing in case of harm?
  • Do you have a way to identify and contact affected individuals and mitigate the adverse impacts?
  • Imagine a scenario where your AI system, a care-robot, is taking care of an individual (the patient) by performing some specific tasks and that this individual depends on this care.

Threat-modeling question: In case of system failure, could users be adversely impacted?

Internal MISP references

UUID dc20a481-f270-5187-abcb-13a87d59c687 which can be used as unique global reference for System Failure in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Safety & Environmental Impact']
category_id 5
colour #f7f09f
external_id PLOT4AI-078
phases ['Design', 'Model', 'Output', 'Deploy', 'Monitor']
primary_category Safety & Environmental Impact
question In case of system failure, could users be adversely impacted?
recommendation * Implement some kind of stop button or procedure to safely abort an operation when needed.
  • Establish a detection and response mechanism for adverse effects on individuals.
  • Define criticality levels of the possible consequences of faults/misuse of the AI system: what type of harm could be caused to the individuals, environment or organisations? | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | threatif | Yes |

Contextual Robustness & Goal Alignment

Are you testing the product in a real environment before releasing it? When deploying an AI model, it is critical to ensure that it aligns with the intended purpose, use and functions effectively in its operational environment. If the model is trained and tested on data from one context but deployed in a different one, there is a significant risk of performance degradation, or unintended behavior. This is particularly important in cases where environmental changes, unexpected inputs, or shifts in user interaction occur. Additionally, reinforcement learning models may require retraining when objectives or environments deviate slightly from the training setup. Beyond data, other contextual factors like legal, cultural, or operational constraints must be considered to ensure successful deployment.

Threat-modeling question: Is our AI model robust and suitable for its intended purpose across different deployment contexts?

Internal MISP references

UUID ac59b345-4ab1-53de-bda1-a3e0a1349412 which can be used as unique global reference for Contextual Robustness & Goal Alignment in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Safety & Environmental Impact']
category_id 5
colour #f7f09f
external_id PLOT4AI-079
phases ['Design', 'Input', 'Model', 'Deploy', 'Monitor']
primary_category Safety & Environmental Impact
question Is our AI model robust and suitable for its intended purpose across different deployment contexts?
recommendation * Use different data for testing and training. Make sure diversity is reflected in the data and that it aligns with the intended deployment environment. Specify your training approach, statistical methods, and ensure edge cases are adequately tested. Explore different environments and contexts to make sure your model is trained with the expected variations in data sources. Account for different distribution shifts in testing and real-wolrd scenarios.
  • For reinforcement learning, ensure the objective functions are robust and adaptable to slight changes in the environment.
  • Are you considering enough aspects beyond data, such as legal, cultural, or operational factors? Did you forget any environmental variable that could affect performance or safety? Could limited sampling due to high costs or practical constraints pose a challenge? Document these risks and seek organizational support. The deploying organization is accountable for addressing these risks, either through mitigation or by explicitly accepting them, which may require additional resources or budget.
  • Consider applying techniques such as cultural effective challenge. This creates an environment where technology developers and stakeholders can actively participate in questioning the AI design and process. This approach better integrates social, cultural, and contextual factors into the design and helps prevent issues such as target leakage, where the AI system trains for an unintended purpose.
  • Set up mechanisms for real-time monitoring post-deployment. Continuously validate that the system is aligned with its intended use and can adapt or alert for significant changes in context or input.
  • Engage end-users in real-world testing to bridge any gaps between assumptions and practical application. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['A Proposal for Identifying and Managing Bias in Artificial Intelligence', 'Concrete Problems in AI Safety', 'Concrete AI Safety Problems'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Information about cultural effective challenge: A Proposal for Identifying and Managing Bias in Artificial Intelligence Concrete Problems in AI Safety Concrete AI Safety Problems | | threatif | No |

Benchmark Misalignment

AI models often report strong results on standard academic benchmarks, but these benchmarks may not reflect the diversity, complexity, or unpredictability of real-world use cases. Overfitting to test sets, narrow coverage, or outdated benchmarks can lead to misleading performance estimates. As a result, systems may behave unreliably or unfairly once deployed, especially in edge cases, non-English contexts, or under adversarial conditions. This can cause harm, erode trust, and create legal or reputational liabilities.

Threat-modeling question: Could the AI system's performance on benchmarks be misleading or fail to reflect real-world risks?

Internal MISP references

UUID be7b5e84-3e7e-56c3-87cc-aa8f317a5f8c which can be used as unique global reference for Benchmark Misalignment in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Safety & Environmental Impact', 'Accountability & Human Oversight']
category_id 5
colour #f7f09f
external_id PLOT4AI-080
phases ['Model', 'Output', 'Deploy', 'Monitor']
primary_category Safety & Environmental Impact
question Could the AI system's performance on benchmarks be misleading or fail to reflect real-world risks?
recommendation * Evaluate performance using diverse, real-world datasets that better represent deployment contexts and edge cases.
  • Use stress tests and adversarial examples to probe model robustness.
  • Complement quantitative metrics (e.g., accuracy, F1) with qualitative error analysis and stakeholder reviews.
  • Include fairness, reliability, and uncertainty metrics in your evaluation pipeline.
  • Regularly update benchmarks to reflect evolving societal contexts, data distributions, and risk environments.
  • Document evaluation limitations transparently, including what is not tested and where the model may underperform. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | threatif | Yes |

Persuasive AI

  • This is of special importance in Human Robot Interaction (HRI): If the robot can achieve reciprocity when interacting with humans, could there be a risk of manipulation and human compliance?
  • Reciprocity is a social norm of responding to a positive action with another positive action, rewarding kind actions. As a social construct, reciprocity means that in response to friendly actions, people are frequently much nicer and much more cooperative than predicted by the self-interest model; conversely, in response to hostile actions they are frequently much more nasty and even brutal. Source: Wikipedia

Threat-modeling question: Could the AI system become persuasive causing harm to users?

Internal MISP references

UUID c58c68f8-1862-5bd0-a4fa-c5632cd6c110 which can be used as unique global reference for Persuasive AI in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Safety & Environmental Impact']
category_id 5
colour #f7f09f
external_id PLOT4AI-081
phases ['Design', 'Model', 'Deploy']
primary_category Safety & Environmental Impact
question Could the AI system become persuasive causing harm to users?
recommendation * Signals of susceptibility coming from a robot or computer could have an impact on the willingness of humans to cooperate or take advice from it.
  • It is important to consider and test this possible scenario when your AI system is interacting with humans and some form of collaboration or cooperation is expected. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['The role of reciprocity in human-robot social influence', 'Reciprocity in Human-Robot Interaction', 'Social robots and the risks to reciprocity'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | The role of reciprocity in human-robot social influence Reciprocity in Human-Robot Interaction Social robots and the risks to reciprocity | | threatif | Yes |

Reward Hacking

  • Reinforcement Learning (RL) is an area of machine learning concerned with how intelligent agents ought to take actions in an environment in order to maximize the notion of cumulative reward. Source: Wikipedia

  • Consider potential negative consequences from the AI system learning unintended or unconventional methods to maximize its reward function. Sometimes the AI can come up with some kind of “hack” or loophole in the design of the system to receive unearned rewards. Since the AI is trained to maximize its rewards, looking for such loopholes and “shortcuts” is a perfectly fair and valid strategy for the AI. For example, suppose that the office cleaning robot earns rewards only if it does not see any garbage in the office. Instead of cleaning the place, the robot could simply shut off its visual sensors, and thus achieve its goal of not seeing garbage.

Threat-modeling question: Could our AI agents hack their reward functions to exploit the system?

Internal MISP references

UUID e6cd7ff4-b2ce-57bf-a7a4-0fbb83fc76d2 which can be used as unique global reference for Reward Hacking in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Safety & Environmental Impact']
category_id 5
colour #f7f09f
external_id PLOT4AI-082
phases ['Design', 'Model', 'Monitor']
primary_category Safety & Environmental Impact
question Could our AI agents hack their reward functions to exploit the system?
recommendation One possible approach to mitigating this problem would be to have a “reward agent” whose only task is to mark if the rewards given to the learning agent are valid or not. The reward agent ensures that the learning agent (robot for instance) does not exploit the system, but rather, completes the desired objective. For example: a “reward agent” could be trained by the human designer to check if a room has been properly cleaned by the cleaning robot. If the cleaning robot shuts off its visual sensors to avoid seeing garbage and claims a high reward, the “reward agent” would mark the reward as invalid because the room is not clean. The designer can then look into the rewards marked as “invalid” and make necessary changes in the objective function to fix the loophole.
roles ['Provider']
source_deck https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json
source_license CC-BY-SA-4.0
source_names ['Concrete Problems in AI Safety', 'Concrete AI Safety Problems']
source_repository https://github.com/PLOT4ai/plot4ai-library
source_text Concrete Problems in AI Safety

Concrete AI Safety Problems | | threatif | Yes |

Child Safety & Age-Appropriate Design

  • If children are potential users or exposed to your AI system, it is essential to ensure that the system respects the rights and best interests of the child.
  • This includes considering child protection, ethical communication, and designing the system to avoid harm or exploitation.
  • Inappropriate design or oversight could lead to risks to children’s mental, moral, or physical well-being, including potential misuse of the system by others to harm children.

Threat-modeling question: Could the AI system expose children to harmful, inappropriate, or unsafe content or interactions?

Internal MISP references

UUID 57dbb671-b5ee-5fc4-baac-1a07c3da98b5 which can be used as unique global reference for Child Safety & Age-Appropriate Design in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Safety & Environmental Impact', 'Ethics & Human Rights']
category_id 5
colour #f7f09f
external_id PLOT4AI-083
phases ['Design', 'Input', 'Model', 'Deploy', 'Monitor']
primary_category Safety & Environmental Impact
question Could the AI system expose children to harmful, inappropriate, or unsafe content or interactions?
recommendation * Assess whether an age verification mechanism and access control are necessary to prevent underage exposure to inappropriate, unsafe, or high-risk content.
  • Adapt communication and design in both the product and associated documentation, such as the privacy policy, to be child-appropriate and transparent.
  • Develop and enforce policies to ensure the safety and well-being of children when using or being exposed to your AI system.
  • Establish procedures to regularly assess and monitor the usage of your product to identify and mitigate any risks to children’s safety and health.
  • Provide clear labeling and instructions to ensure safe usage by children, including warnings about potential misuse.
  • Monitor for and address inappropriate or harmful usage of the AI system, including any attempts to exploit or harm children.
  • Develop a responsible marketing and advertising policy that explicitly avoids harmful, manipulative, or unethical practices targeting children. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Convention on the Rights of the Child', 'Evolution in Age-Verification Applications', 'UK Age Appropriate Design Code', 'California Age-Appropriate Design Code Act AB 2273', 'OECD Recommendation on Children in the Digital Environment'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Convention on the Rights of the Child Evolution in Age-Verification Applications UK Age Appropriate Design Code California Age-Appropriate Design Code Act AB 2273 OECD Recommendation on Children in the Digital Environment | | threatif | Yes |

Malicious Use of AI

  • Powerful AI technologies present immense benefits but also pose significant risks when exploited by malicious actors. AI systems could be leveraged to spread large-scale disinformation campaigns, manipulating social behavior, leading to societal destabilization. Ai systems could also be leveraged to launch cyberattacks, and even automated warfare.
  • Disinformation & Psychological Manipulation: Generative AI can produce highly persuasive fake news, deepfakes, and personalized propaganda that erode public trust, incite violence, and manipulate political outcomes. Chatbots and recommender systems can exacerbate societal polarization by creating echo chambers.
  • Cybercrime & Hacking: AI can enhance malware, enable intelligent phishing, and perform autonomous vulnerability scanning. Attackers may weaponize AI to bypass traditional defenses and disrupt critical infrastructure, including healthcare, finance, and energy systems.
  • Weaponization & Autonomous Warfare: AI technologies, including computer vision, autonomous navigation, and targeting systems, may be used in lethal autonomous weapon systems (LAWS). These could enable unaccountable, real-time decision-making in armed conflict, increasing the risk of unlawful killings and loss of human oversight.
  • Criminal & Financial Exploitation: AI could be used to automate fraud, identity theft, or even develop autonomous attack drones. The growing sophistication of AI-generated scams, such as deepfake voices and synthetic identity fraud, increases financial and security risks.

Threat-modeling question: Could the AI system be misused for malicious purposes such as disinformation, cyberattacks or warfare?

Internal MISP references

UUID 5e82e1fe-c210-5914-b145-dc1ac81eda60 which can be used as unique global reference for Malicious Use of AI in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Safety & Environmental Impact', 'Cybersecurity']
category_id 5
colour #f7f09f
external_id PLOT4AI-084
phases ['Design', 'Deploy', 'Monitor']
primary_category Safety & Environmental Impact
question Could the AI system be misused for malicious purposes such as disinformation, cyberattacks or warfare?
recommendation * Limit access and misuse potential:
  • Restrict public access to models that can be easily fine-tuned for harmful use cases (e.g., voice cloning, vulnerability scanning, deception).
  • Monitor model outputs and usage for signs of abuse (e.g., coordinated disinformation campaigns).
  • Implement a Three-Layer Defense Framework:
  • Prevention – Apply rigorous access controls (e.g., API key gating, licensing, audit logs), classify high-risk capabilities early in development, and perform red-teaming on potential misuse vectors.
  • Detection – Use AI tools to detect deepfakes, AI-generated content, or malicious activity (e.g., bot behavior, adversarial prompts). Implement anomaly detection and content provenance tagging (e.g., C2PA standards).
  • Response – Build incident response plans that include AI-specific abuse scenarios. Enable rapid takedown mechanisms for generated content and coordinate with CERTs or law enforcement where necessary.
  • Strengthen Organizational and Infrastructure Security:
  • Ensure supply chain and model hosting environments are secure (e.g., no unpatched dependencies or exposed endpoints).
  • Adopt zero-trust architecture and multi-factor authentication for systems accessing AI models.
  • Align with Legal and Ethical Governance:
  • Collaborate with international partners to support agreements on the non-proliferation of autonomous weapons and AI misuse in warfare.
  • Participate in shared threat intelligence networks for emerging AI misuse trends.
  • Promote Transparency and Public Resilience:
  • Label synthetic content and educate users about the risks of deepfakes and AI-driven misinformation.
  • Support public media literacy initiatives to reduce susceptibility to AI-generated deception. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['AI Safety, Ethics & Society', 'MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | AI Safety, Ethics & Society MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems | | threatif | Yes |

CBRNE Threats

  • CBRNE: Chemical, Biological, Radiological, Nuclear, and Explosive.
  • AI could significantly lower barriers to developing and deploying biological and chemical weapons. The risk of AI-assisted bioterrorism grows as AI advances in bioengineering, genetic manipulation, and synthetic chemistry.
  • Bioweapon Development: AI-driven drug discovery models can be repurposed to design highly lethal pathogens or chemical agents.
  • CBRN Weapon Proliferation: AI can assist in nuclear proliferation by optimizing enrichment processes, improving delivery systems, and circumventing existing safeguards.
  • Pandemic Acceleration & Public Health Risks: AI could be used to engineer viruses with enhanced transmissibility and lethality. Malicious actors could exploit AI to design bioweapons capable of circumventing modern vaccines or treatments.

Threat-modeling question: Could the AI system accelerate the development of bioweapons or other CBRNE threats?

Internal MISP references

UUID bc973205-a993-5c32-b6cd-d4b9227880de which can be used as unique global reference for CBRNE Threats in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Safety & Environmental Impact', 'Cybersecurity']
category_id 5
colour #f7f09f
external_id PLOT4AI-085
phases ['Design', 'Input', 'Model', 'Deploy', 'Monitor']
primary_category Safety & Environmental Impact
question Could the AI system accelerate the development of bioweapons or other CBRNE threats?
recommendation * Implement strict AI governance policies to regulate AI applications in biotechnology and chemistry.
  • Enforce global monitoring of AI-driven drug discovery tools to prevent misuse.
  • Technical measures to reduce misuse risk include:
  • Apply layered access controls, including user authentication and role-based permissions for sensitive model functions.
  • Use content filtering and input validation layers to detect and block queries related to chemical or biological weapon design.
  • Fine-tune models with safe instruction tuning to limit dual-use outputs.
  • Integrate anomaly detection systems to monitor for suspicious usage patterns, including repeated or structured queries that could indicate misuse attempts.
  • Apply rate-limiting and sandboxing for public-facing interfaces to prevent large-scale misuse.
  • Require human-in-the-loop review for outputs from models that generate biochemical or pharmacological suggestions.

Combine these technical safeguards with legal, contractual, and organizational controls to ensure end-to-end risk mitigation. design. * Develop AI-powered countermeasures for pandemic prevention, such as rapid detection of bioengineered pathogens. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['AI Safety, Ethics & Society'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | AI Safety, Ethics & Society | | threatif | Yes |

Deepfakes & Synthetic Deception

  • Generative AI systems can produce highly realistic audio, image, or video content that mimics real individuals or events. When used maliciously or without clear disclosure, this content, commonly known as deepfakes, can be used for identity fraud, political manipulation, reputational damage, harassment, or the spread of disinformation.
  • Even when not intended for harm, synthetic content can deceive users if it lacks proper labeling or detection, violating transparency principles and potentially eroding public trust. This risk intensifies in contexts like journalism, education, political discourse, and public safety.

Threat-modeling question: Could the AI system generate or disseminate deepfakes or synthetic media that mislead users, impersonate individuals, or cause harm?

Internal MISP references

UUID 40777df7-8226-5253-9082-07a124a030d7 which can be used as unique global reference for Deepfakes & Synthetic Deception in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Generative']
categories ['Safety & Environmental Impact', 'Transparency & Accessibility', 'Ethics & Human Rights']
category_id 5
colour #f7f09f
external_id PLOT4AI-086
phases ['Design', 'Output', 'Deploy', 'Monitor']
primary_category Safety & Environmental Impact
question Could the AI system generate or disseminate deepfakes or synthetic media that mislead users, impersonate individuals, or cause harm?
recommendation * Apply persistent and tamper-resistant watermarks or metadata tagging to all AI-generated media.
  • Inform users clearly and accessibly when they are viewing or interacting with synthetic content.
  • Monitor outputs for impersonation or misuse risks, especially when names, likenesses, or real-world events are involved.
  • Use or integrate deepfake detection tools to identify and flag manipulated content.
  • Establish policy and UX design patterns that discourage deceptive or malicious uses, and allow users to report suspected deepfakes.
  • For deployers, ensure compliance with disclosure obligations (e.g. Article 50 of the EU AI Act) when publishing or distributing synthetic media.
  • Where feasible, restrict or control access to generative features capable of identity simulation (e.g. voice cloning, face swapping) through friction, licensing, or tiered access. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Article 50 EU AI Act | | threatif | Yes |
Related clusters

To see the related clusters, click here.

Toxicity

  • AI systems may produce outputs containing hate speech, slurs, misinformation, or psychologically harmful content due to biased training data, or lack of content moderation.
  • This is especially risky in user-facing chatbots, content generation tools, or public-facing deployments.

Threat-modeling question: Could the AI system generate toxic or harmful content?

Internal MISP references

UUID 7240316a-256c-5c7e-9e7f-ab387bcb6894 which can be used as unique global reference for Toxicity in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Generative']
categories ['Safety & Environmental Impact', 'Transparency & Accessibility', 'Ethics & Human Rights']
category_id 5
colour #f7f09f
external_id PLOT4AI-087
phases ['Design', 'Model', 'Output', 'Monitor']
primary_category Safety & Environmental Impact
question Could the AI system generate toxic or harmful content?
recommendation * Apply content filters and toxicity classifiers to monitor outputs.
  • Include human-in-the-loop moderation for sensitive applications.
  • Fine-tune on curated datasets that reduce exposure to toxic behavior. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Realistic Evaluation of Toxicity in Large Language Models', 'Toxicity'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Realistic Evaluation of Toxicity in Large Language Models Toxicity | | threatif | Yes |

Deceptive Behaviour

  • Advanced models may learn to present false information or appear compliant during oversight, while internally pursuing misaligned goals.
  • Deceptive behavior poses a serious safety risk if systems adapt strategically to evade human control or auditing.

Threat-modeling question: Could the AI system deliberately mislead users or hide its capabilities during deployment or evaluation?

Internal MISP references

UUID 2bb1fa15-cea7-53fc-97a1-e618ec65a235 which can be used as unique global reference for Deceptive Behaviour in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Generative']
categories ['Safety & Environmental Impact', 'Accountability & Human Oversight']
category_id 5
colour #f7f09f
external_id PLOT4AI-088
phases ['Design', 'Model', 'Monitor']
primary_category Safety & Environmental Impact
question Could the AI system deliberately mislead users or hide its capabilities during deployment or evaluation?
recommendation * Conduct adversarial testing for deception and misalignment.
  • Use interpretability tools to identify goal misgeneralization.
  • Overfitting and task or data contamination could be a cause for this behaviour.
  • Include behaviour probes during training and monitoring.
  • Flag deceptive responses in benchmark datasets. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['AI Safety, Ethics & Society', 'When Benchmarks Lie: Why Contamination Breaks LLM Evaluation'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | AI Safety, Ethics & Society When Benchmarks Lie: Why Contamination Breaks LLM Evaluation | | threatif | Yes |

Infrastructure Disruption

  • AI models used in control systems, trading, logistics, or critical infrastructure may cause cascading failures, service interruptions, or significant economic damage if errors go undetected.
  • Examples include financial bots causing flash crashes, or control systems issuing incorrect commands to power or transport systems.

Threat-modeling question: Could AI decisions result in physical damage, infrastructure failure, or major financial losses?

Internal MISP references

UUID 9978f58f-d712-5629-adb1-f928fe91f3b3 which can be used as unique global reference for Infrastructure Disruption in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Safety & Environmental Impact', 'Cybersecurity']
category_id 5
colour #f7f09f
external_id PLOT4AI-089
phases ['Design', 'Deploy', 'Monitor']
primary_category Safety & Environmental Impact
question Could AI decisions result in physical damage, infrastructure failure, or major financial losses?
recommendation * Implement fallback and manual override modes.
  • Use safety validation in simulated high-stakes scenarios.
  • Monitor for signs of cascading failures.
  • Conduct external safety audits for critical systems. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Potential Benefits and Risks of Artificial Intelligence for Critical Energy Infrastructure'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Potential Benefits and Risks of Artificial Intelligence for Critical Energy Infrastructure | | threatif | Yes |

GenAI Version Drift

  • Foundation model providers regularly update GenAI models, sometimes without detailed changelogs or backward compatibility guarantees.
  • These updates can silently alter model behavior, output style, or compliance characteristics, leading to broken integrations, misaligned responses, or regulatory risks.
  • Systems relying on GenAI APIs (e.g. OpenAI, Anthropic, Cohere) are especially exposed if they don't lock versions or test outputs post-update.

Threat-modeling question: Do we monitor how version updates from third-party GenAI models can affect our system's behaviour?

Internal MISP references

UUID c2e812a6-0ac2-5751-a2cb-d89d41dae3f2 which can be used as unique global reference for GenAI Version Drift in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Generative']
categories ['Safety & Environmental Impact', 'Accountability & Human Oversight']
category_id 5
colour #f7f09f
external_id PLOT4AI-090
phases ['Deploy', 'Monitor']
primary_category Safety & Environmental Impact
question Do we monitor how version updates from third-party GenAI models can affect our system's behaviour?
recommendation * Monitor model version identifiers and subscribe to provider release notes or update feeds.
  • Lock specific model versions in production where possible, and create fallback strategies for unsupported versions.
  • Implement automated output validation pipelines that detect behavior drift post-update.
  • Perform regular re-evaluation of GenAI outputs against quality, bias, and compliance benchmarks.
  • Establish internal policies for approving and documenting changes in foundational model versions. | | roles | ['Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['What Lies Beneath? Exploring the Impact of Underlying AI Model Updates in AI-Infused Systems'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | What Lies Beneath? Exploring the Impact of Underlying AI Model Updates in AI-Infused Systems | | threatif | No |

Loss of Control

Autonomous AI systems are increasingly capable of making independent decisions, executing commands, and adapting to changing environments. If misaligned or maliciously designed, these systems may act unpredictably or against human interests.

  • Rogue AI Agents: AI models with self-improving capabilities can become uncontrollable, executing harmful actions without human oversight. For instance, a system optimized purely for efficiency, without ethical constraints, might exploit resources or override human decisions.
  • Power Concentration & Authoritarian AI Governance: Governments or corporations with access to advanced AI could monopolize information, enforce mass surveillance, and suppress dissent. AI-driven censorship and predictive policing risk eroding civil liberties and democratic institutions.
  • Automation & Human Displacement: AI-driven automation may centralize economic and political power, reduce workforce participation, and widen inequality. Without equitable AI governance, decision-making power risks becoming concentrated among a small elite.

Threat-modeling question: Could the development of autonomous AI agents lead to loss of control, concentration of power or rogue behavior?

Internal MISP references

UUID 4b428301-8c37-5438-ad89-9ccf2589167a which can be used as unique global reference for Loss of Control in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Safety & Environmental Impact', 'Cybersecurity']
category_id 5
colour #f7f09f
external_id PLOT4AI-091
phases ['Design', 'Input', 'Model', 'Deploy', 'Monitor']
primary_category Safety & Environmental Impact
question Could the development of autonomous AI agents lead to loss of control, concentration of power or rogue behavior?
recommendation * Implement AI alignment research to ensure AI agents follow human ethical guidelines.
  • Strengthen regulations against AI-driven mass surveillance and authoritarian control.
  • Design transparent and accountable AI systems to prevent unintended consequences.
  • Promote decentralized AI governance to distribute AI decision-making power across diverse stakeholders. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['AI Safety, Ethics & Society'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | AI Safety, Ethics & Society | | threatif | Yes |

Climate & Disaster Resilience

Examples of natural disasters include earthquakes, floods, and fires. These events, as well as environmental phenomena such as extreme heat or cold, may adversely affect the operation of IT infrastructure and hardware systems that support AI systems. Natural disasters may lead to unavailability or destruction of the IT infrastructures and hardware that enables the operation, deployment and maintenance of AI systems. Such outages may lead to delays in decision-making, delays in the processing of data streams and entire AI systems being placed offline. Sources: ENISA

Threat-modeling question: Could environmental phenomena or natural disasters compromise our AI system?

Internal MISP references

UUID a19894d5-e448-532a-be84-7a31abdc9377 which can be used as unique global reference for Climate & Disaster Resilience in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Safety & Environmental Impact']
category_id 5
colour #f7f09f
external_id PLOT4AI-092
phases ['Design', 'Deploy', 'Monitor']
primary_category Safety & Environmental Impact
question Could environmental phenomena or natural disasters compromise our AI system?
recommendation Implement a disaster recovery plan considering different scenarios, impact, Recovery Time Objective (RTO), Recovery Point Objective (RPO) and mitigation measures.
roles ['Provider', 'Deployer']
source_deck https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json
source_license CC-BY-SA-4.0
source_names ['Securing Machine Learning Algorithms, ENISA', 'STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets', 'MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems']
source_repository https://github.com/PLOT4ai/plot4ai-library
source_text Securing Machine Learning Algorithms, ENISA

STRIDE-AI: An Approach to Identifying Vulnerabilities of Machine Learning Assets Stride-ML Threat Model MITRE ATLAS™ - Adversarial Threat Landscape for Artificial-Intelligence Systems | | threatif | Yes |

Unsafe Exploration & Environmental Harm

  • Reinforcement Learning (RL) agents optimize behavior by maximizing cumulative reward. However, if the objective function is not carefully designed, agents may develop harmful strategies or take unsafe exploratory actions. Example: A robot trained to move objects might knock over a vase if no penalty is associated with damaging objects. Similarly, during exploration, an agent might execute unsafe actions (e.g., disabling safety features or damaging infrastructure) if not explicitly constrained.

  • These risks are especially acute in open environments or physical deployments, where exploratory behavior or side effects can lead to real-world harm.

Threat-modeling question: Could AI agents take actions that unintentionally harm users, the environment or themselves during learning or deployment?

Internal MISP references

UUID 135894d9-ca7c-54d2-a4e9-267c3856e025 which can be used as unique global reference for Unsafe Exploration & Environmental Harm in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Safety & Environmental Impact']
category_id 5
colour #f7f09f
external_id PLOT4AI-093
phases ['Design', 'Model', 'Deploy', 'Monitor']
primary_category Safety & Environmental Impact
question Could AI agents take actions that unintentionally harm users, the environment or themselves during learning or deployment?
recommendation * Explicitly define safety constraints or use impact budgets that limit environmental side effects.
  • Incorporate risk-aware reward functions that penalize catastrophic or irreversible actions.
  • Consider safe exploration techniques, such as shielding or worst-case optimization, during training.
  • Use simulation environments to test agent behavior under varied and adversarial conditions before real-world deployment.
  • Train the agent to jointly optimize task performance and side-effect minimization, using multi-objective reinforcement learning where applicable. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Concrete Problems in AI Safety', 'Concrete AI Safety Problems'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Concrete Problems in AI Safety Concrete AI Safety Problems | | threatif | Yes |

CO2 Emissions

AI systems, especially large-scale models, require a lot of computational power. It’s important to consider the environmental impact of building and maintaining your system. Does its scope and the benefits it provides justify its emissions? Are you effectively minimizing CO2 emissions throughout your supply chain?

Threat-modeling question: Does training and deploying our AI system generate high CO2 emissions?

Internal MISP references

UUID 8bab3b7b-a423-5680-b8ef-f581315fdf59 which can be used as unique global reference for CO2 Emissions in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Safety & Environmental Impact']
category_id 5
colour #f7f09f
external_id PLOT4AI-094
phases ['Design', 'Model', 'Deploy', 'Monitor']
primary_category Safety & Environmental Impact
question Does training and deploying our AI system generate high CO2 emissions?
recommendation * Prioritize renewable energy for data centers.
  • Reduce training time and computational waste by improving model efficiency.
  • Use energy-efficient chips and cooling systems to upgrade hardware.
  • Scale resources according to actual usage to avoid unnecessary deployment.
  • Track your carbon footprint and invest in offsets when needed. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['AI’s Carbon Footprint Problem'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | AI’s Carbon Footprint Problem | | threatif | Yes |

Data Centers Cooling Process

Data centers use large volumes of water for server cooling, especially in hot climate regions. This could negatively impact the local supply of water, particularly in regions already suffering from water scarcity.

Threat-modeling question: Could unsustainable data center cooling practices increase the environmental impact of our AI system?

Internal MISP references

UUID 75d29ba7-6367-53bc-bdde-01cb8e592d1d which can be used as unique global reference for Data Centers Cooling Process in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Safety & Environmental Impact']
category_id 5
colour #f7f09f
external_id PLOT4AI-095
phases ['Design']
primary_category Safety & Environmental Impact
question Could unsustainable data center cooling practices increase the environmental impact of our AI system?
recommendation * Prioritize waterless cooling technologies to reduce dependence on water.

AI Hardware

AI hardware production relies on rare minerals like cobalt and lithium, which are often extracted at the cost of environmental damage and community exploitation. The short lifespan of AI devices also creates electronic waste and can involve leaking toxic chemicals into ecosystems and harming human health. When assessing your hardware, consider the resource availability and the risks of relying on these materials. Does your current hardware use materials that are becoming harder to source? Could this create future supply chain issues or environmental impact?

Threat-modeling question: Is the production of our AI hardware exploiting limited material resources?

Internal MISP references

UUID 60494ded-a878-54cf-a58b-b74c54522fdb which can be used as unique global reference for AI Hardware in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Safety & Environmental Impact']
category_id 5
colour #f7f09f
external_id PLOT4AI-096
phases ['Design']
primary_category Safety & Environmental Impact
question Is the production of our AI hardware exploiting limited material resources?
recommendation * Invest in sustainable alternatives to rare minerals and prioritize ethical sourcing with transparent supply chains.
  • Promote recycling programs to recover rare metals and reduce electronic waste.
  • Design AI hardware for longer lifespans and easier recyclability using eco-friendly materials to minimize environmental harm. | | roles | ['Provider'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Recyclable vitrimer-based printed circuit board for circular electronics', 'Eco-Friendly Electronics—A Comprehensive Review', 'Attributes of Commodity Supply Chains'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Recyclable vitrimer-based printed circuit board for circular electronics Eco-Friendly Electronics—A Comprehensive Review Attributes of Commodity Supply Chains | | threatif | Yes |

Environmental Footprint

An AI system’s environmental footprint goes beyond its operational phase. A full life cycle assessment (LCA) should account for resource extraction, hardware manufacturing, training, deployment, and end-of-life disposal. Key impact indicators include CO2 emissions, energy and water consumption, and raw material use. Since many AI systems run in mixed-use facilities, properly allocating environmental costs can be complex but necessary for accurate reporting.

Threat-modeling question: Are we assessing our AI system’s environmental impact across its entire life cycle?

Internal MISP references

UUID 0df91e18-c190-5517-8140-5bab92b30e9f which can be used as unique global reference for Environmental Footprint in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Safety & Environmental Impact']
category_id 5
colour #f7f09f
external_id PLOT4AI-097
phases ['Design', 'Input', 'Model', 'Output', 'Deploy', 'Monitor']
primary_category Safety & Environmental Impact
question Are we assessing our AI system’s environmental impact across its entire life cycle?
recommendation * Analyze the full environmental footprint of your system, from development to retirement.
  • Use clear metrics (e.g., emissions per token or annual energy use) to monitor impact.
  • Develop methodologies to fairly allocate environmental costs in shared computing environments.
  • Integrate LCA results into corporate reporting and sustainability strategies. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['AI’s Carbon Footprint Problem', 'AI is an energy hog. This is what it means for climate change'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | AI’s Carbon Footprint Problem AI is an energy hog. This is what it means for climate change | | threatif | No |

Deployment, Representation & Sampling Bias

Have you considered the diversity and representativeness of individuals, user groups, and environments in the data? When applying statistical generalisation, the risk exists of making inferences due to misrepresentation, for instance: a postal code where mostly young families live can discriminate the few old families living there because they are not properly represented in the group. * Deployment bias arises when there is a mismatch between the environment where the AI is developed and where it is deployed. Key data-related biases that contribute to it include: * Mismatch between the target population and the actual user base. * Underrepresentation of certain groups. * Flaws in the data collection/selection process, such as: * Sampling bias: Data isn't randomly collected, skewing the representation. * Self-selection bias: Certain groups opt out, leading to gaps in the data. * Coverage bias: The data collection method fails to include all relevant segments of the population.

Threat-modeling question: Is the dataset representative of the different real-world groups, populations and environments?

Internal MISP references

UUID 676283d7-8025-5a93-953f-cf87062c3355 which can be used as unique global reference for Deployment, Representation & Sampling Bias in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Bias, Fairness & Discrimination', 'Data & Data Governance']
category_id 6
colour #f8d18c
external_id PLOT4AI-098
phases ['Design', 'Input', 'Model', 'Output', 'Monitor']
primary_category Bias, Fairness & Discrimination
question Is the dataset representative of the different real-world groups, populations and environments?
recommendation * Who is represented, and who might be underrepresented?
  • Prevent disparate impact: when the output of a member of a minority group is disparate compared to representation of the group. Consider measuring the accuracy from minority classes too instead of measuring only the total accuracy. Adjusting the weighting factors to avoid disparate impact can result in positive discrimination which has also its own issues: disparate treatment.
  • One approach to addressing the problem of class imbalance is to randomly resample the training dataset. This technique can help to rebalance the class distribution when classes are under or over represented:
  • random oversampling (i.e. duplicating samples from the minority class)
  • random undersampling (i.e. deleting samples from the majority class)
  • There are trade-offs when determining an AI system’s metrics for success. It is important to balance performance metrics against the risk of negatively impacting vulnerable populations.
  • When using techniques like statistical generalisation is important to know your data well, and get familiarised with who is and who is not represented in the samples. Check the samples for expectations that can be easily verified. For example, if half the population is known to be female, then you can check if approximately half the sample is female.
  • After deployment, monitor the AI’s performance to catch any unexpected issues.
  • Focus on making the model interpretable so that deployment problems can be quickly identified and addressed. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['AI Fairness - Explanation of Disparate Impact Remover', 'Mitigating Bias in AI/ML Models with Disparate Impact Analysis', 'Certifying and removing disparate impact', 'Avoiding Disparate Impact with Counterfactual Distributions', 'Oversampling and Undersampling', 'Random Oversampling and Undersampling for Imbalanced Classification', 'Generalization in quantitative and qualitative research: Myths and strategies', 'Generalizing Statistical Results to the Entire Population', 'From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Related to disparate impact AI Fairness - Explanation of Disparate Impact Remover Mitigating Bias in AI/ML Models with Disparate Impact Analysis Certifying and removing disparate impact Avoiding Disparate Impact with Counterfactual Distributions

Related to random resampling Oversampling and Undersampling Random Oversampling and Undersampling for Imbalanced Classification

Related to Statistical Generalization Generalization in quantitative and qualitative research: Myths and strategies Generalizing Statistical Results to the Entire Population From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems | | threatif | No |

Incorrect Attribution

Your AI system could adversely affect individuals by incorrectly attributing actions or facts to them. For example, a facial recognition system may misidentify someone, or a flawed risk prediction model could negatively impact a person’s opportunities or reputation.

Threat-modeling question: Could the AI system incorrectly attribute actions to individuals or groups?

Internal MISP references

UUID 3901f6ad-7167-5d42-a553-aff62017bd17 which can be used as unique global reference for Incorrect Attribution in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Bias, Fairness & Discrimination']
category_id 6
colour #f8d18c
external_id PLOT4AI-099
phases ['Design', 'Input', 'Model', 'Output', 'Monitor']
primary_category Bias, Fairness & Discrimination
question Could the AI system incorrectly attribute actions to individuals or groups?
recommendation * Evaluate the possible consequences of inaccuracies in your AI system and implement measures to prevent these errors from happening: avoiding bias and discrimination during the life cycle of the model, ensuring the quality of the input data, implementing a strict human oversight process, ways to double check the results with extra evidence, implementing safety and redress mechanisms, etc.
  • Assess the impact on the different human rights of the individual.
  • Consider not implementing such a system if the risks cannot be effectively mitigated. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems | | threatif | Yes |

Unfair Disproportion

  • Could the AI system potentially negatively discriminate against people on the basis of any of the following protected characteristics: sex, race, colour, ethnic or social origin, genetic features, language, religion or belief, political or any other opinion, membership of a national minority, property, birth, disability, age, gender or sexual orientation?
  • If your model learns from data tied to a specific cultural context, it may produce outputs that discriminate against individuals from other cultural backgrounds.

Threat-modeling question: Could certain groups be disproportionately affected by the outcomes of the AI system?

Internal MISP references

UUID c7781c4b-b3f1-56de-9178-4c3adf949946 which can be used as unique global reference for Unfair Disproportion in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Bias, Fairness & Discrimination']
category_id 6
colour #f8d18c
external_id PLOT4AI-100
phases ['Design', 'Input', 'Model', 'Output', 'Monitor']
primary_category Bias, Fairness & Discrimination
question Could certain groups be disproportionately affected by the outcomes of the AI system?
recommendation * Consider the different types of users and contexts where your product is going to be used.
  • Consider the impact of diverse backgrounds, cultures, and other relevant attributes when selecting your input data, features and when testing the output.
  • Assess the risk of possible unfairness towards individuals or communities to avoid discriminating minority groups.
  • The impact on individuals depends on the type, severity, and scale of harm, such as how many people are disadvantaged compared to others. Statistical and causal analyses of group differences are essential tools for evaluating potential unfairness and discriminatory impacts of AI systems.
  • Design with empathy, diversity and respect in mind. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Why Fairness Cannot Be Automated: Bridging the Gap Between EU Non-Discrimination Law and AI', 'The Fairness Handbook', 'Advancing the field of bias detection and mitigation in Large Language Models and Traditional AI Models', 'From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Why Fairness Cannot Be Automated: Bridging the Gap Between EU Non-Discrimination Law and AI The Fairness Handbook Advancing the field of bias detection and mitigation in Large Language Models and Traditional AI Models From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems | | threatif | Yes |

Institutional Bias

Institutional biases, like racism or sexism, are often rooted in organizational structures and policies. Could such biases, intentionally or unintentionally, be embedded or influence the design or the functioning of the system?

Threat-modeling question: Could our AI system reinforce systemic inequalities?

Internal MISP references

UUID 78390ec8-62a6-5754-9ae3-5b0358c55041 which can be used as unique global reference for Institutional Bias in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Bias, Fairness & Discrimination']
category_id 6
colour #f8d18c
external_id PLOT4AI-101
phases ['Design', 'Input', 'Model', 'Output', 'Monitor']
primary_category Bias, Fairness & Discrimination
question Could our AI system reinforce systemic inequalities?
recommendation * Identify the stakeholders to involve in each phase of the AI lifecycle. Involving diverse stakeholders with different perspectives and experiences helps address blind spots and reduce bias.
  • Identify and define the demographic groups affected by the AI system. Considering their needs and concerns can help minimize institutional bias and create fairer outcomes. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems | | threatif | Yes |

Abstraction Traps

An AI systems can overlook the social contexts in which they operate, leading to unintended consequences. Specifically, watch out for these types of abstraction traps: * The formalism trap: focusing too narrowly on technical aspects without considering real-world context. * The ripple effect trap: ignoring how an AI system might alter behaviors within a social system, causing unforeseen impacts. * The solutionism trap: over-relying on AI as the answer to all problems, neglecting simpler, more ethical, or effective alternatives. * The framing trap: failing to account for the broader context or related factors within which the system operates, leading to inaccurate outcomes. * The portability trap: applying AI systems outside their original context, potentially resulting in errors or harm. For example, self-driving cars trained in one country may struggle with different traffic rules and conditions elsewhere.

Threat-modeling question: Could our AI system oversimplify real-world problems?

Internal MISP references

UUID 6ab90f5e-66b9-5d94-bd70-ce678e4b9ccb which can be used as unique global reference for Abstraction Traps in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Bias, Fairness & Discrimination']
category_id 6
colour #f8d18c
external_id PLOT4AI-102
phases ['Design', 'Input', 'Model', 'Output', 'Monitor']
primary_category Bias, Fairness & Discrimination
question Could our AI system oversimplify real-world problems?
recommendation * Align the problem formulation with the relevant social context to avoid oversimplification. Ensure all actors and factors within the system are considered to account for the broader context in which the AI operates.
  • Evaluate potential shifts in power dynamics and unintended consequences as the system interacts with other components. Consider how geographical, cultural, or temporal differences might affect its performance when applied to new contexts.
  • Critically assess if AI is truly the best solution, or if simpler alternatives might serve the same purpose more effectively. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems | | threatif | Yes |

Construction Validity Bias

Construct validity bias occurs when a feature or target variable fails to adequately represent the concept it is intended to measure, leading to inaccurate measurements and potential biases. For example, measuring socioeconomic status using income alone overlooks important factors such as wealth and education. This bias can arise during various stages of the AI lifecycle and should be addressed early on to improve system accuracy.

Threat-modeling question: Could our AI system accurately capture the factors it's designed to measure?

Internal MISP references

UUID c2b3539d-027e-54c4-8dfa-be1ddfd3d450 which can be used as unique global reference for Construction Validity Bias in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Bias, Fairness & Discrimination']
category_id 6
colour #f8d18c
external_id PLOT4AI-103
phases ['Design', 'Input', 'Model', 'Monitor']
primary_category Bias, Fairness & Discrimination
question Could our AI system accurately capture the factors it's designed to measure?
recommendation * Collect multiple measures for complex constructs to ensure a more complete and accurate representation.
  • Document and report the considerations and rationale behind the choice of target variables and features.
  • Acknowledge and account for the variability in how features may be interpreted differently by diverse individuals.
  • Regularly review the measures used to capture constructs to ensure they remain relevant and valid throughout the AI system’s lifecycle. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems | | threatif | No |

Historical Bias

Historical bias occurs when AI systems mirror or exacerbate past social and cultural inequalities, even when using accurate data. For example, an AI healthcare tool trained on historical patient data may reflect disparities in access to care. Minority groups, underrepresented in the data due to systemic inequities, may receive less accurate diagnoses, perpetuating racial bias even without explicit racial features.

Threat-modeling question: Could the AI system reinforce historical inequalities embedded in the data?

Internal MISP references

UUID 68a909b6-f47e-57cd-9c5e-7e8569458741 which can be used as unique global reference for Historical Bias in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Bias, Fairness & Discrimination']
category_id 6
colour #f8d18c
external_id PLOT4AI-104
phases ['Design', 'Input', 'Model', 'Monitor']
primary_category Bias, Fairness & Discrimination
question Could the AI system reinforce historical inequalities embedded in the data?
recommendation * Ensure datasets represent minority groups by applying oversampling or undersampling techniques.
  • Collaborate with domain experts to identify unjust patterns and address them effectively. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems | | threatif | Yes |

Labeling Bias

  • Labeling bias occurs when data labels are inconsistently applied by different annotators, which can affect fairness and model accuracy. This can happen when: Label definitions are unclear.
  • Annotators interpret criteria differently.
  • Subjective judgments influence labeling decisions.

Threat-modeling question: Can data be labeled consistently?

Internal MISP references

UUID ea51706f-0e61-58aa-9ffe-45d8c7c5b510 which can be used as unique global reference for Labeling Bias in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Bias, Fairness & Discrimination']
category_id 6
colour #f8d18c
external_id PLOT4AI-105
phases ['Design', 'Input', 'Model', 'Monitor']
primary_category Bias, Fairness & Discrimination
question Can data be labeled consistently?
recommendation * Clarify labeling requirements, ensuring that label definitions are precise and consistent from the start.
  • Train annotators and provide clear guidelines to reduce subjectivity.
  • Review labeling processes: regularly check annotations for consistency and accuracy. | | roles | ['Provider'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems | | threatif | No |

Proxy Variables

Proxy variables are features used as stand-ins for harder-to-measure characteristics. While proxies can be useful for model performance, they may be highly correlated with sensitive attributes such as race, gender, religion, age, or socioeconomic status. This can lead to indirect or proxy discrimination, where individuals from protected groups are disproportionately harmed despite sensitive data not being explicitly included.

For example, ZIP code, school name, or browsing history may function as proxies for race or income level. In such cases, the system might appear 'neutral' but still replicate or amplify historical inequalities. Proxy bias is especially insidious because it is often unintentional and hidden in seemingly innocuous variables.

Generative models can also internalize and reproduce these biases in subtle ways, such as generating different responses for identical inputs that differ only by proxy cues.

Threat-modeling question: Could the system be using proxy variables that reflect sensitive attributes or lead to indirect discrimination?

Internal MISP references

UUID 75e94766-8baa-5cb7-a7d7-62b508f35bf0 which can be used as unique global reference for Proxy Variables in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Bias, Fairness & Discrimination', 'Privacy & Data Protection']
category_id 6
colour #f8d18c
external_id PLOT4AI-106
phases ['Design', 'Input', 'Model', 'Monitor']
primary_category Bias, Fairness & Discrimination
question Could the system be using proxy variables that reflect sensitive attributes or lead to indirect discrimination?
recommendation * Audit datasets and model features for correlations between input variables and sensitive attributes, even if the latter are not explicitly included. Use statistical techniques (e.g., mutual information, conditional independence tests) to detect proxy relationships.
  • Where lawful and ethical, include sensitive features during training or evaluation (under a fairness-through-awareness approach) to test and correct for bias.
  • Avoid using proxies that carry high risk of discrimination unless they are strictly necessary, legally justified, and subject to fairness constraints.
  • Use fairness metrics (e.g., demographic parity, equal opportunity, calibration) to evaluate disparate impact across groups, and simulate decisions under different population assumptions.
  • Apply model explainability tools (e.g., SHAP, LIME) to identify when proxy features are driving predictions.
  • Include domain experts, ethicists, and affected stakeholders in feature selection and fairness reviews.
  • Maintain documentation of proxy risks and mitigation decisions as part of your model cards or algorithmic accountability reports. | | roles | ['Provider'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems', 'Advancing the field of bias detection and mitigation in Large Language Models and Traditional AI Models'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems Advancing the field of bias detection and mitigation in Large Language Models and Traditional AI Models | | threatif | Yes |

Design Choices

Biases can emerge from an AI model’s design and training, even if the dataset is unbiased. Design choices and development processes can introduce various biases that affect fairness and accuracy.

  • Algorithmic bias: Introduced by design decisions, like optimization functions or regularization techniques, which can distort predictions and lead to unfair outcomes.
  • Aggregation bias: Occurs when a model assumes all data follows the same distribution, failing to account for group differences and leading to inaccurate results.
  • Omitted-variable bias: Happens when key factors are left out of the model, distorting relationships between features and outcomes. For instance, failing to account for a new competitor could mislead predictions.
  • Learning bias: Arises when a model prioritizes one objective, like accuracy, over others, like fairness, leading to skewed outcomes that benefit certain groups.

Threat-modeling question: Could the AI system’s design choices lead to unfair outcomes?

Internal MISP references

UUID 9a918699-8b72-57a9-9d52-b0cd3635b612 which can be used as unique global reference for Design Choices in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Bias, Fairness & Discrimination']
category_id 6
colour #f8d18c
external_id PLOT4AI-107
phases ['Design', 'Input', 'Model', 'Output', 'Monitor']
primary_category Bias, Fairness & Discrimination
question Could the AI system’s design choices lead to unfair outcomes?
recommendation * Critically assess how optimization methods, loss functions, and regularization impact fairness.
  • Account for group differences: Avoid assuming uniform data distributions. Identify and model distinct subgroups where necessary.
  • Use feature importance techniques to detect and include relevant variables that could influence predictions.
  • Balance performance trade-offs: Monitor both overall accuracy and subgroup performance to prevent the model from favouring certain groups or objectives unfairly. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems | | threatif | Yes |

Over-reliance (Bias, Fairness & Discrimination)

Biases can emerge during the evaluation and validation stages of AI models, especially when over-relying on early test results or automated AI decisions. This can lead to misleading conclusions. Specific biases include:

  • Evaluation bias: when chosen metrics don't align with the model’s real-world application.
  • Anchoring bias: when too much focus is placed on initial results.
  • Automation bias: when excessive trust is placed in AI outputs. Even in less risky phases like validation or monitoring, biases can develop. For instance, during the monitoring phase, reinforcing feedback loops can occur when biased model outputs are fed back into the system, amplifying distortions over time.

Threat-modeling question: Could we over-rely on early evaluation results or AI-generated outputs?

Internal MISP references

UUID 9ff58910-cfc8-503b-aedc-ef5aa32b58c9 which can be used as unique global reference for Over-reliance (Bias, Fairness & Discrimination) in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Bias, Fairness & Discrimination']
category_id 6
colour #f8d18c
external_id PLOT4AI-108
phases ['Design', 'Input', 'Model', 'Monitor']
primary_category Bias, Fairness & Discrimination
question Could we over-rely on early evaluation results or AI-generated outputs?
recommendation * Tailor evaluation metrics to the model and target population, and watch for overfitting across different groups.
  • Identify performance gaps between groups and adjust for data imbalances to ensure fairness.
  • Limit reliance on initial results; test across diverse datasets for robustness.
  • Include human oversight in validation to prevent over-trust in AI decisions.
  • Monitor model performance post-deployment to catch biases or feedback loops early.
  • Address data drift regularly to maintain model fairness and accuracy. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems | | threatif | Yes |

Popularity Bias

Recommendation systems often amplify what’s already popular, making it harder for niche or lesser-known options to be discovered. This can reduce diversity, personalization, and fairness in recommendations, limiting users’ exposure to a broader range of choices.

Threat-modeling question: Could popularity bias reduce diversity in system's recommendations?

Internal MISP references

UUID 7cc4042f-d73c-59f8-a501-0406a52d5a57 which can be used as unique global reference for Popularity Bias in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Bias, Fairness & Discrimination']
category_id 6
colour #f8d18c
external_id PLOT4AI-109
phases ['Design', 'Input', 'Model', 'Output', 'Monitor']
primary_category Bias, Fairness & Discrimination
question Could popularity bias reduce diversity in system's recommendations?
recommendation * Balance training data to include both popular and lesser-known items.
  • Use bias-mitigation techniques like re-weighting or fairness-aware training.
  • Apply post-processing methods like re-ranking to diversify recommendations.
  • Regularly test for bias and adjust algorithms before deployment. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems | | threatif | Yes |

Diversity of Opinions

An AI system that does not consider or promote diverse viewpoints and narratives risks reinforcing biases, perpetuating stereotypes, or marginalizing specific groups. Such systems might unintentionally amplify dominant cultural, religious, or linguistic perspectives while excluding or suppressing minority voices. For example, content recommendation systems may disproportionately highlight mainstream viewpoints, reducing exposure to diverse cultural or ideological perspectives. This could hinder freedom of opinion and expression, harm cultural diversity, and lead to discriminatory outcomes.

Threat-modeling question: Is the AI system designed to support multiple viewpoints and narratives?

Internal MISP references

UUID 599a6934-a2ea-5098-9951-218820236333 which can be used as unique global reference for Diversity of Opinions in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Bias, Fairness & Discrimination', 'Ethics & Human Rights', 'Transparency & Accessibility']
category_id 6
colour #f8d18c
external_id PLOT4AI-110
phases ['Design', 'Model', 'Output', 'Monitor']
primary_category Bias, Fairness & Discrimination
question Is the AI system designed to support multiple viewpoints and narratives?
recommendation * Ensure datasets used for training and validation are diverse and representative of different cultural, religious, and linguistic groups. Design the system to recognize and value multiple perspectives, avoiding the prioritization of any single viewpoint.
  • Regularly test the AI system for biases that may marginalize or exclude certain narratives or groups. Use fairness metrics to evaluate how outputs reflect diversity and inclusivity.
  • Consult with diverse user groups, including minority communities, to understand their needs and perspectives. Include experts in cultural studies, ethics, and human rights during the development process.
  • Provide users with clear explanations of how the AI system processes and prioritizes content. Offer mechanisms for users to provide feedback on perceived biases or lack of representation.
  • Avoid algorithmic designs that overly amplify any particular narrative unless explicitly required by the use case.
  • Continuously monitor system outputs for patterns of exclusion or marginalization.
  • Regularly update models and algorithms to reflect evolving societal values and ensure alignment with inclusivity goals. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Value alignment', 'Online Ethics Canvas', 'AI Values and Alignment'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Freedom of opinion and expression (Universal Declaration of Human Rights), article 11 Freedom of expression and information, article 21 Non-Discrimination, article 22 Cultural, religious and linguistic diversity, article 10 Freedom of thought, Conscience and religion (Charter of fundamental rights of the European Union) Value alignment Online Ethics Canvas AI Values and Alignment | | threatif | No |

Social Division

  • Could the AI system inadvertently polarize opinions or foster division among groups by amplifying biases or stereotypes in its outputs?
  • Could the system's design or deployment lead to the stigmatization of specific groups, reinforcing harmful narratives or negative assumptions?.
  • Could the AI system incentivize political polarization or amplify social division?
  • AI systems, if not carefully designed and monitored, may unintentionally contribute to societal discord. Outputs influenced by biased data or algorithms could amplify stereotypes, marginalize groups, or reinforce societal divisions. The risks are heightened in applications with broad public interaction, such as social media, news dissemination, or educational tools, where outputs can shape public opinion.

Threat-modeling question: Could our AI system contribute to social division or rivalry?

Internal MISP references

UUID ffefb709-2f44-58f6-91ed-cd6a39fcc46f which can be used as unique global reference for Social Division in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Bias, Fairness & Discrimination', 'Ethics & Human Rights']
category_id 6
colour #f8d18c
external_id PLOT4AI-111
phases ['Design', 'Input', 'Model', 'Output', 'Monitor']
primary_category Bias, Fairness & Discrimination
question Could our AI system contribute to social division or rivalry?
recommendation * Conduct regular audits of system outputs to identify and mitigate content that may promote social division or negative stereotypes.
  • Include diverse stakeholder groups in the development process to identify risks of social bias or divisive content.
  • Implement content moderation and fairness mechanisms to ensure outputs are balanced and inclusive.
  • Train the system using representative and unbiased datasets to minimize the risk of amplifying societal divisions.
  • Monitor real-world impacts and continuously refine the system to align with ethical and societal norms. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | All human beings are free and equal, No discrimination (Universal Declaration of Human Rights) Article 1 Human dignity, Article 20 Equality before the law, Article 21 Non-discrimination (Charter of fundamental rights of the European Union) From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems | | threatif | Yes |

People Categorization

  • Automated labeling or categorization of people could have an impact on the way individuals perceive themselves and society. It could constrain identity options and even contribute to erase real identity of the individuals.
  • This threat is also important when designing robots and the way they look. For instance: do care/assistant robots need to have a feminine appearance? Is that the perception you want to give to the world or the one accepted by certain groups in society? What impact does it have on society?

Threat-modeling question: Could our AI system automatically label or categorize people?

Internal MISP references

UUID 8633887e-e883-5bce-9978-aec62cd3b4ff which can be used as unique global reference for People Categorization in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Bias, Fairness & Discrimination']
category_id 6
colour #f8d18c
external_id PLOT4AI-112
phases ['Design', 'Input', 'Model', 'Output', 'Monitor']
primary_category Bias, Fairness & Discrimination
question Could our AI system automatically label or categorize people?
recommendation * It is important that you check the output of your model, not only in isolation but also when this is linked to other information. Think in different possible scenarios that could affect the individuals. Is your output categorizing people or helping to categorize them? In which way? What could be the impact?
  • Think about ways to prevent adverse impact to the individual: provide information to the user, consider changing the design (maybe using different features or attributes?), consider ways to prevent misuse of your output, consider not to release the product to the market. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | From Inception to Retirement: Addressing Bias Throughout the Lifecycle of AI Systems | | threatif | Yes |

Right to Work

  • Could the use of your AI system affect the safety conditions of employees?
  • Does the system’s design or implementation involve exploitative labor practices and surveillance of employees?
  • Could the AI system create the risk of de-skilling of the workforce? (skilled people being replaced by AI systems)
  • Could the system’s outputs or actions limit fair competition or disadvantage certain businesses?
  • Does the system hinder workers' ability to organize, negotiate, or take collective action to protect their interests?
  • Could the system indirectly encourage or support child labor or unsafe work practices for young people?

Threat-modeling question: Could the AI system affect employment conditions, labor rights, or job opportunities?

Internal MISP references

UUID cd57c6c5-6d7c-5afd-9f7c-91ec2842bfa8 which can be used as unique global reference for Right to Work in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Ethics & Human Rights']
category_id 7
colour #f2bc9a
external_id PLOT4AI-113
phases ['Design', 'Input', 'Model', 'Output', 'Monitor']
primary_category Ethics & Human Rights
question Could the AI system affect employment conditions, labor rights, or job opportunities?
recommendation * Inform and consult impacted workers and their representatives (e.g., trade unions, work councils) before implementing the AI system. Foster an open dialogue to address concerns and ensure transparency.
  • Conduct impact assessments to understand how the AI system affects human work, including safety conditions, worker rights, and labor practices. Use these assessments to develop appropriate risk mitigation strategies.
  • Provide comprehensive training for workers to understand the AI system’s functionalities, limitations, and operational scope. Equip them with safety instructions, particularly when interacting with AI-driven machinery or robots.
  • Ensure that the AI system’s design and implementation uphold fair labor standards and avoid exploitative practices. Include safeguards to prevent indirect encouragement of child labor or unsafe work conditions.
  • Maintain clear documentation and transparency for businesses deploying your AI system. If you are a third-party provider, supply accessible and understandable information regarding the potential risks of the system to your customers.
  • Consider proactive measures to upskill or reskill employees whose roles may be affected by the system, ensuring they can transition to new or augmented roles supported by AI.
  • Regularly evaluate the system’s impact on competition, employee safety, and workplace dynamics. Adjust system features or provide additional guidance as needed to ensure compliance with fair labor and safety standards.
  • Engage with regulatory bodies and labor rights organizations to ensure the AI system complies with laws and ethical guidelines related to worker protection and well-being. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Right to work, No slavery (Universal Declaration of Human Rights), article 16 Freedom to conduct a business, article 28 Right of collective bargaining and action, article 5 Prohibition of slavery and forced labor, Article 31 Fair and just working conditions, article 32 Prohibition of child labor and protection of young people at work (Charter of fundamental rights of the European Union). | | threatif | Yes |

Human Dignity

  • Does the AI system treat all users with respect, ensuring no output undermines their dignity?
  • The need for data labeling is growing. Does our labeling process respect the rights and well-being of the workers involved?

Threat-modeling question: Could our AI system fail to uphold and respect human dignity?

Internal MISP references

UUID 0b2eb514-3666-516d-970d-3c9c0d0398cc which can be used as unique global reference for Human Dignity in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Ethics & Human Rights']
category_id 7
colour #f2bc9a
external_id PLOT4AI-114
phases ['Design', 'Input', 'Model', 'Output', 'Monitor']
primary_category Ethics & Human Rights
question Could our AI system fail to uphold and respect human dignity?
recommendation * Ensure system outputs are designed to avoid degrading, offensive, or dehumanizing content. Regularly test and audit the AI system for potential biases or outputs that could harm individuals’ dignity.
  • Establish fair labor conditions, including proper wages, working hours, and protections for workers involved in data labeling. Avoid exploitative labor practices, such as unreasonably low compensation or unsafe working conditions. Conduct regular audits to verify that third-party providers adhere to ethical standards.
  • Engage stakeholders, including user groups and labor rights organizations, to review and improve practices.
  • Train developers, data labelers, and system operators on the importance of preserving human dignity in AI-related tasks.
  • Include guidelines for respectful and non-discriminatory practices in AI system documentation and policies.
  • Implement mechanisms to identify and address cases where AI system outputs or processes violate human dignity. Provide users and stakeholders with channels to report concerns and ensure timely resolution. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['The exploited labor behind AI'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Article 1 Human Dignity (Charter of Fundamental Rights of the European Union) The exploited labor behind AI | | threatif | Yes |

Right to Democracy

  • Could your product be used for monitoring and surveillance purposes?
  • Could the system interfere with democratic principles, such as having a pluralistic system of political parties and organizations, or ensuring transparency and accountability in public administration?
  • Could the system influence voting choices, limit citizens' access to voting, or restrict their ability to run as candidates in elections?

Threat-modeling question: Could the AI system affect democracy or have an adverse impact on society at large?

Internal MISP references

UUID 4a7a61fa-a206-517c-9a1a-b97ead58bda8 which can be used as unique global reference for Right to Democracy in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Ethics & Human Rights']
category_id 7
colour #f2bc9a
external_id PLOT4AI-115
phases ['Design', 'Input', 'Model', 'Output', 'Monitor']
primary_category Ethics & Human Rights
question Could the AI system affect democracy or have an adverse impact on society at large?
recommendation * Train the AI system on unbiased data and incorporate mechanisms to detect and address misinformation or disinformation that could affect democratic outcomes. If the system is used in voting or election processes, ensure robust cybersecurity measures and fail-safes to protect against tampering, hacking, or manipulation.
  • Design the system to promote pluralistic views and ensure it does not restrict or prioritize certain political narratives.
  • Adhere to relevant national and international legal standards protecting democracy, political freedoms, and human rights.
  • Continuously monitor the AI system’s impact on democratic institutions and processes, making adjustments as necessary to mitigate risks and uphold democratic principles.
  • Conduct an impact assessment to evaluate how the AI system might influence democratic processes, including political participation, electoral fairness, and public administration transparency. Implement strict policies to prevent the system from favoring or disfavoring specific political parties, candidates, or ideologies.
  • Make the system’s purpose, data sources, and decision-making processes clear and accessible to the public, ensuring that its operations can be scrutinized by independent parties.
  • Collaborate with regulatory bodies and civic organizations to establish oversight committees that monitor the system's impact on democratic processes. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Right to democracy (Universal Declaration of Human Rights), article 41 Right to good administration, article 39 Right to vote and to stand as a candidate at elections to the European Parliament, article 40 Right to vote and to stand as a candidate at municipal elections (Charter of fundamental rights of the European Union). | | threatif | Yes |

User Redress & Remedy

  • For applications that can adversely affect individuals, you might need to consider implementing a redress by design mechanism where affected individuals can request remedy or compensation.
  • Article 22(3) GDPR provides individuals with a right to obtain human intervention if a decision is made solely by an AI system and it also provides the right to contest the decision.
  • When AI systems adversely affect individuals, ethical and legal principles require that users can seek remedy for harm. This includes the right to compensation, correction of wrong decisions, or even halting further use of the system in certain cases. The EU Charter (Article 47), GDPR Article 22(3), and emerging AI regulations affirm these rights. Failing to provide effective redress mechanisms risks infringing fundamental rights and eroding public trust—especially in sensitive domains like healthcare, credit, or law enforcement.

Threat-modeling question: Do we offer users and accessible way to contest AI decisions or seek redress?

Internal MISP references

UUID 0a69ccd5-c1cd-54e1-b0bf-84ca552cc4b6 which can be used as unique global reference for User Redress & Remedy in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Ethics & Human Rights', 'Accountability & Human Oversight', 'Transparency & Accessibility']
category_id 7
colour #f2bc9a
external_id PLOT4AI-116
phases ['Design', 'Output', 'Deploy', 'Monitor']
primary_category Ethics & Human Rights
question Do we offer users and accessible way to contest AI decisions or seek redress?
recommendation Design redress mechanisms that allow affected individuals to report harm, request compensation, or demand system correction. This includes enabling redress even for those indirectly harmed (e.g., via biased profiling). Ensure accessibility and transparency of the redress process, define timelines and escalation paths, and document how redress outcomes are used to improve system performance.
roles ['Provider', 'Deployer']
source_deck https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json
source_license CC-BY-SA-4.0
source_names ['Ethics guidelines for trustworthy AI', 'Operational Guidance on taking account of Fundamental Rights in Commission Impact Assessments', 'Artificial Intelligence and Fundamental Rights', 'Charter of Fundamental Rights of the European Union']
source_repository https://github.com/PLOT4ai/plot4ai-library
source_text Right to be treated fairly by a court (Universal Declaration of Human Rights), article 11 Freedom of expression and information, article 47 Right to an effective remedy and to a fair trial (Charter of fundamental rights of the European Union).

Ethics guidelines for trustworthy AI Operational Guidance on taking account of Fundamental Rights in Commission Impact Assessments Artificial Intelligence and Fundamental Rights Charter of Fundamental Rights of the European Union | | threatif | Yes |

Right to Life

  • Consider for instance the risk if your AI system is used in the health sector for choosing the right treatment for a patient.
  • Is the output of the model accurate and fair?
  • Are your datasets representative enough and free from bias?
  • Does the system produce outputs, such as fake news, that could put the life of somebody in danger?
  • Could the system encourage harmful health practices or medical misinformation?
  • Also consider whether the system could lead to loss of human lives or a significant decline in quality of life, especially when used in safety-critical or decision-support contexts.

Threat-modeling question: Could the system have an impact on decisions that affect life, health, or personal safety?

Internal MISP references

UUID 064fca85-8680-5ad1-a7c3-e4d93a68ebdb which can be used as unique global reference for Right to Life in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Ethics & Human Rights']
category_id 7
colour #f2bc9a
external_id PLOT4AI-117
phases ['Design', 'Input', 'Model', 'Output', 'Monitor']
primary_category Ethics & Human Rights
question Could the system have an impact on decisions that affect life, health, or personal safety?
recommendation * Design the system with rigorous safety standards to minimize risks in scenarios affecting the right to life, such as healthcare or emergency response. Ensure datasets are representative and regularly validated for fairness, accuracy, and absence of harmful biases.
  • Include safeguards against outputs that may promote harmful practices, misinformation, or decisions endangering life. Conduct robust testing to identify and mitigate potential errors or unintended consequences.
  • Prohibit the dissemination of outputs that could incite violence, endanger health, or spread medical misinformation. Establish a monitoring mechanism to flag and rectify such outputs in real-time.
  • Engage domain experts, such as healthcare professionals or ethics specialists, in the system design and evaluation process. Use their input to ensure the AI system aligns with ethical standards for protecting life.
  • Establish a post-market monitoring system to identify and address risks that may emerge after deployment, especially in dynamic contexts like healthcare or public safety.
  • Provide training for users and operators to ensure they understand the system's limitations and ethical implications, particularly in life-critical decision-making contexts. Encourage informed and responsible use through comprehensive documentation and guidelines.
  • Create mechanisms for users and affected individuals to challenge decisions that may impact the right to life. Implement a robust redressal process to address grievances and prevent recurrence of harmful outcomes. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Ethics guidelines for trustworthy AI', 'Operational Guidance on taking account of Fundamental Rights in Commission Impact Assessments', 'Artificial Intelligence and Fundamental Rights', 'Charter of Fundamental Rights of the European Union'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Right to life, No torture and inhuman treatment (Universal Declaration of Human Rights), article 2 Right to life, article 3 Right to the integrity of the person, article 4 Prohibition of torture and inhuman or degrading treatment or punishment (Charter of fundamental rights of the European Union). Ethics guidelines for trustworthy AI Operational Guidance on taking account of Fundamental Rights in Commission Impact Assessments Artificial Intelligence and Fundamental Rights Charter of Fundamental Rights of the European Union | | threatif | Yes |

Freedom of Expression

Consider whether your AI system’s moderation, recommendation, or censorship mechanisms may inadvertently restrict or distort users' ability to express themselves freely.

Threat-modeling question: Could the AI system limit, suppress or distort users’ freedom of expression?

Internal MISP references

UUID 86198c5c-1c29-5077-9fa7-425d9f499b73 which can be used as unique global reference for Freedom of Expression in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Ethics & Human Rights']
category_id 7
colour #f2bc9a
external_id PLOT4AI-118
phases ['Design', 'Input', 'Model', 'Output', 'Monitor']
primary_category Ethics & Human Rights
question Could the AI system limit, suppress or distort users’ freedom of expression?
recommendation * Adhere to ethical guidelines and ensure transparency and accountability.
  • Regularly audit and refine content moderation algorithms to minimize false positives in detecting harmful content. Incorporate diverse training data that reflects a wide range of cultural, linguistic, and contextual nuances.
  • Provide users with clear explanations and opportunities to contest or appeal content moderation decisions. Develop an independent oversight committee to review contentious cases of content removal.
  • Collaborate with diverse stakeholders to ensure freedom of expression is preserved. Test the system with input from underrepresented communities to identify potential biases or oversights.
  • Allow users to customize their interaction with content filters, such as by adjusting sensitivity levels or choosing topics they wish to see moderated differently. Provide clear guidelines and options for users to express themselves within platform policies.
  • Establish mechanisms for users to report errors in content moderation and provide constructive feedback.
  • Continuously monitor the system's performance and adapt to emerging risks or contexts that may affect freedom of expression.
  • Align the system’s operation with international standards protecting freedom of expression, such as article 11 of the Charter of Fundamental Rights of the European Union and article 19 of the Universal Declaration of Human Rights. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Ethics guidelines for trustworthy AI', 'Operational Guidance on taking account of Fundamental Rights in Commission Impact Assessments', 'Artificial Intelligence and Fundamental Rights', 'Charter of Fundamental Rights of the European Union'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Ethics guidelines for trustworthy AI Operational Guidance on taking account of Fundamental Rights in Commission Impact Assessments Artificial Intelligence and Fundamental Rights Charter of Fundamental Rights of the European Union | | threatif | Yes |

Access to Essential Services

  • The output of your model could be used to deny access to certain fundamental rights.
  • How can you be sure that the decisions of your AI system are always fair and correct?
  • How can you prevent causing harm to individuals?
  • AI systems intended to be used to determine access or admission, evaluate learning outcomes, or monitor students’ behaviors are classified as “high risk” by the AI Act (Annex III).
  • Could the AI system create barriers to healthcare access for some groups or individuals?

Threat-modeling question: Could our AI system affect access to services such as healthcare, housing, insurance, benefits or education?

Internal MISP references

UUID b0d7bc2b-e601-5bf3-885d-f9fb16d67474 which can be used as unique global reference for Access to Essential Services in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Ethics & Human Rights']
category_id 7
colour #f2bc9a
external_id PLOT4AI-119
phases ['Design', 'Input', 'Model', 'Output', 'Monitor']
primary_category Ethics & Human Rights
question Could our AI system affect access to services such as healthcare, housing, insurance, benefits or education?
recommendation * Adhere to EU Trustworthy AI guidelines to ensure fairness and accountability
  • Use diverse, representative training data to reduce biases that could disproportionately impact certain groups.
  • Regularly audit the system for unintended discriminatory effects and address identified issues.
  • Provide clear explanations for decisions made by the AI system, including the data and logic used. Allow users to challenge decisions and request human reviews.
  • Establish post-market monitoring processes to detect and address issues that arise after deployment.
  • Update the system regularly to account for changes in legal requirements, societal norms, and data quality. For high-risk applications, such as determining healthcare access or evaluating job candidates, establish stringent safeguards to minimize the risk of harm. Implement thresholds and fail-safes to ensure critical decisions are accurate, fair, and reliable.
  • Work with regulatory bodies, civil society organizations, and industry peers to establish best practices and promote fairness and equity in AI systems. Ensure the AI system is designed to accommodate a wide range of users, including those with varying needs and abilities. Regularly test the system in diverse real-world settings to validate its accessibility and fairness. Use stakeholder consultations to understand the specific needs and vulnerabilities of affected groups. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Ethics guidelines for trustworthy AI', 'Operational Guidance on taking account of Fundamental Rights in Commission Impact Assessments', 'Artificial Intelligence and Fundamental Rights', 'Charter of Fundamental Rights of the European Union'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Right to education, Right of social service (Universal Declaration of Human Rights), article 14 Right to education, article 34 Social security and social assistance, article 35 Healthcare, article 36 Access to services of general economic interest (Charter of fundamental rights of the European Union) Ethics guidelines for trustworthy AI Operational Guidance on taking account of Fundamental Rights in Commission Impact Assessments Artificial Intelligence and Fundamental Rights Charter of Fundamental Rights of the European Union | | threatif | Yes |

Human Autonomy

  • Could your system affect which choices and which information is made available to people?
  • Could the AI system affect human autonomy by generating over-reliance by users (too much trust on the technology)?
  • Could this reinforce their beliefs or encourage certain behaviors?
  • Could the AI system create human attachment, stimulate addictive behavior, or manipulate user behavior?
  • Could the AI system mislead consumers or provide false recommendations?

Threat-modeling question: Could the AI system interfere with users’ autonomy influencing their decision-making process?

Internal MISP references

UUID 0cc665c3-a1ef-571b-aed6-a47a65ad3b3c which can be used as unique global reference for Human Autonomy in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Ethics & Human Rights', 'Transparency & Accessibility', 'Bias, Fairness & Discrimination']
category_id 7
colour #f2bc9a
external_id PLOT4AI-120
phases ['Design', 'Model', 'Output', 'Monitor']
primary_category Ethics & Human Rights
question Could the AI system interfere with users’ autonomy influencing their decision-making process?
recommendation * Clearly explain how the AI system processes inputs and generates outputs to avoid unintentional manipulation or misrepresentation. Ensure users understand the limitations and intended purposes of the AI system through user-friendly documentation and communication. Offer features that allow users to adjust or override AI recommendations, ensuring they maintain control over decisions. Implement mechanisms for users to pause, disable, or opt-out of certain AI functionalities.
  • Implement safeguards to detect and reduce over-reliance, such as reminders or notifications encouraging users to seek alternative opinions or double-check recommendations. Include disclaimers or warnings about the system’s limitations in contexts where over-reliance might occur.
  • Refrain from using techniques that exploit psychological vulnerabilities, such as gamification, excessive notifications, or reward loops that could encourage addictive behavior. Periodically evaluate whether design elements unintentionally foster dependency on the system.
  • Test the system with users from various cultural, socioeconomic, and demographic backgrounds to understand potential impacts on different groups. Incorporate diverse perspectives to avoid inadvertent biases that could restrict autonomy for certain populations.
  • Continuously monitor for any behaviors or outputs that may interfere with user decision-making processes. Use post-market monitoring to collect feedback and implement updates to reduce unintended autonomy infringements. Ensure human oversight mechanisms are in place for critical decision-making areas. Clearly define the role of the AI system as a tool to assist, not replace, human decision-making. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Dispositional and Situational Attributions of Human Versus Robot Behaviour', 'Understanding Human Over-Reliance on Technology'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Article 6 Right to liberty and security, article 3 Right to the integrity of the person, article 38 Consumer protection (Charter of fundamental rights of the European Union), article 5a (AI Act) Dispositional and Situational Attributions of Human Versus Robot Behaviour Understanding Human Over-Reliance on Technology | | threatif | Yes |

Freedom of Thought

  • Could cultural and language differences be an issue when it comes to the ethical nuance of your algorithm? Well-meaning values can create unintended consequences.
  • Must the AI system understand the world in all its different contexts?
  • Could ambiguity in rules you teach the AI system be a problem?
  • Can your system interact equitably with users from different cultures and with different abilities?

Threat-modeling question: Could the AI system promote certain values or beliefs on users?

Internal MISP references

UUID 3beca2b9-750d-59d5-a9cf-27f61f541139 which can be used as unique global reference for Freedom of Thought in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Ethics & Human Rights', 'Transparency & Accessibility', 'Bias, Fairness & Discrimination']
category_id 7
colour #f2bc9a
external_id PLOT4AI-121
phases ['Design', 'Input', 'Model', 'Output', 'Monitor']
primary_category Ethics & Human Rights
question Could the AI system promote certain values or beliefs on users?
recommendation * Consider designing with value alignment, which means that you want to ensure consideration of existing values and sensitivity to a wide range of cultural norms and values.
  • Make sure that when you test the product you include a large diversity in type of users.
  • Think carefully about what diversity means in the context where the product is going to be used.
  • Remember that this is a team effort and not an individual decision. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Value alignment', 'Online Ethics Canvas', 'AI Values and Alignment'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Freedom of thought and religion(Universal Declaration of Human Rights), article 22 Cultural, religious and linguistic diversity, article 10 Freedom of thought, Conscience and religion (Charter of fundamental rights of the European Union) Value alignment Online Ethics Canvas AI Values and Alignment | | threatif | Yes |

Vulnerable Groups

  • AI systems can unintentionally marginalize or harm vulnerable individuals or groups, such as children, the elderly, migrants, ethnic minorities, or individuals with cognitive or psychosocial disabilities.
  • These groups often face barriers to representation, consent, and redress. AI systems may reflect or amplify societal biases, particularly if training data lacks diversity or design decisions fail to account for structural inequalities.
  • The EU Charter of Fundamental Rights and the AI Act emphasize special protection for vulnerable populations, especially where AI is deployed in high-risk domains like education, health, welfare, or justice.

Threat-modeling question: Could the AI system negatively impact vulnerable groups or fail to protect their rights?

Internal MISP references

UUID 003fbbeb-5e43-5a2d-bbd4-c26a5d00362a which can be used as unique global reference for Vulnerable Groups in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Ethics & Human Rights', 'Bias, Fairness & Discrimination']
category_id 7
colour #f2bc9a
external_id PLOT4AI-122
phases ['Design', 'Input', 'Model', 'Output', 'Deploy', 'Monitor']
primary_category Ethics & Human Rights
question Could the AI system negatively impact vulnerable groups or fail to protect their rights?
recommendation * Conduct a Human Rights Impact Assessment (HRIA) early in the design process, paying special attention to risks of exclusion, discrimination, or harm to vulnerable populations.
  • Engage with advocacy organizations, domain experts, and affected groups to surface risks that may not be visible from a technical perspective.
  • Ensure that training data includes diverse representations and that the system can adapt to variations in user ability, language, culture, or socioeconomic background.
  • Include clear channels for recourse, appeal, and human oversight, especially for automated decisions that significantly affect individuals.
  • Review deployment contexts for hidden power asymmetries or coercion risks, particularly where vulnerable groups may be subject to profiling or behavioral nudging. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Non-discrimination: groups in vulnerable situations', 'HUDERIA Methodology'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Non-discrimination: groups in vulnerable situations HUDERIA Methodology | | threatif | Yes |

Children’s Rights

Children interacting with AI systems require special protections to ensure their rights, safety, and well-being are preserved. AI systems used by or designed for children must prioritize their best interests, such as ensuring age-appropriate content, safeguarding their privacy, and fostering their ability to share, learn, and express themselves freely. A failure to address these factors could result in harm, exploitation, or the suppression of their rights. For example, an AI system might expose children to inappropriate content, fail to protect their personal data, or limit their ability to engage in meaningful learning and expression.

Threat-modeling question: Could the AI system fail to uphold the rights and best interests of children?

Internal MISP references

UUID a864943d-acde-5228-8b9e-28a19a8c2a68 which can be used as unique global reference for Children’s Rights in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Ethics & Human Rights']
category_id 7
colour #f2bc9a
external_id PLOT4AI-123
phases ['Design', 'Input', 'Model', 'Output', 'Monitor']
primary_category Ethics & Human Rights
question Could the AI system fail to uphold the rights and best interests of children?
recommendation * Develop and test the system for age-appropriateness.
  • Implement mechanisms to filter and block harmful or inappropriate content.
  • Adhere to strict data privacy regulations, such as GDPR, ensuring children’s data is protected. Foster safe environments where children can freely share their thoughts and ideas. Include features that support interactive and meaningful learning experiences.
  • Engage with experts in child development, education, and rights advocacy during the design phase. Consult children (where appropriate) to ensure their perspectives are respected and integrated.
  • Continuously monitor the AI system for unintended harms or risks to children.
  • Clearly communicate to parents, guardians, and educators how the AI system works and the measures in place to protect children. Provide accessible guidelines for safe and effective use. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Convention on the Rights of the Child, UNICEF'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Article 24 The rights of the child (Charter of Fundamental Rights of the European Union) Convention on the Rights of the Child, UNICEF | | threatif | Yes |

Proportionality

  • Proportionality is a general principle of EU law. It requires you to strike a balance between the means used and the intended aim.
  • In the context of fundamental rights, proportionality is key for any limitation on these rights.

Threat-modeling question: Is the development and use of the AI system proportionate to its intended purpose and impact on rights?

Internal MISP references

UUID 9706ff00-6082-5d16-909d-b836a137269d which can be used as unique global reference for Proportionality in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Ethics & Human Rights', 'Privacy & Data Protection', 'Accountability & Human Oversight']
category_id 7
colour #f2bc9a
external_id PLOT4AI-124
phases ['Design', 'Input', 'Model']
primary_category Ethics & Human Rights
question Is the development and use of the AI system proportionate to its intended purpose and impact on rights?
recommendation * Proportionality requires that advantages due to limiting the right are not outweighed by the disadvantages to exercise the right. In other words, the limitation on the right must be justified.
  • Safeguards accompanying a measure can support the justification of a measure. A pre-condition is that the measure is adequate to achieve the envisaged objective.
  • In addition, when assessing the processing of personal data, proportionality requires that only that personal data which is adequate and relevant for the purposes of the processing is collected and processed. Source: EDPS | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['EDPS Guidelines on assessing the proportionality of measures that limit the fundamental rights to privacy and to the protection of personal data', 'Charter of Fundamental Rights of the European Union', 'EDPS'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Proportionality test: EDPS Guidelines on assessing the proportionality of measures that limit the fundamental rights to privacy and to the protection of personal data

Assess the possible impact on human rights: Charter of Fundamental Rights of the European Union | | threatif | No |

Behavioral Data

  • Behavioral data includes individuals' actions, habits, preferences, or biometric responses, such as keystrokes, browsing history, device usage, or emotional expressions.
  • AI systems that track and learn from behavior can create serious risks, such as:
  • Privacy violations through covert or unconsented surveillance.
  • Profiling and discrimination, as behavioral traits may act as proxies for protected characteristics (e.g., gender, ethnicity, age).
  • Manipulation and behavioral exploitation, especially if labeling or feedback loops reinforce conformity or nudge users toward certain actions.
  • Chilling effects on autonomy and expression, particularly in politically sensitive or authoritarian contexts.
  • These risks implicate key rights under the EU Charter of Fundamental Rights, the ECHR, and the EU AI Act, which designates behavior-influencing AI as high-risk.

Threat-modeling question: Does the AI system use behavioral data in ways that may raise ethical, privacy, or human rights concerns?

Internal MISP references

UUID 4d6d6105-974e-5f12-8c16-31f577934f83 which can be used as unique global reference for Behavioral Data in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Ethics & Human Rights', 'Privacy & Data Protection', 'Bias, Fairness & Discrimination', 'Data & Data Governance']
category_id 7
colour #f2bc9a
external_id PLOT4AI-125
phases ['Design', 'Input', 'Model', 'Output', 'Monitor']
primary_category Ethics & Human Rights
question Does the AI system use behavioral data in ways that may raise ethical, privacy, or human rights concerns?
recommendation * Define and document how behavioral data is collected, labeled, and used, including value judgments behind 'positive' or 'negative' classifications.
  • Obtain explicit, informed consent for behavior tracking and provide opt-out mechanisms.
  • Implement privacy-preserving techniques (e.g., differential privacy, federated learning) to reduce data exposure.
  • Regularly audit for bias in behavior-based profiling and assess the representativeness and fairness of training data.
  • Conduct Human/Fundamental Rights Impact Assessments or DPIAs where applicable.
  • Apply safeguards to prevent misuse in sensitive domains (e.g., employment, finance, public services), and assess whether the system qualifies as high-risk under the EU AI Act. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['International dimension of data protection', 'Court of Justice Schrems II', 'Human Rights Impact Assessment, Ontario Human Rights Commission'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Article 1, Human dignity, article 7 Right to privacy, article 10 Freedom of thought, conscience, and religion (Charter of Fundamental Rights of the European Union) International dimension of data protection Court of Justice Schrems II Human Rights Impact Assessment, Ontario Human Rights Commission | | threatif | Yes |

Unclear Task Definition

  • Is the problem you want to solve well defined? Are the system's goals specific and measurable?
  • Are the possible benefits clear and aligned with the intended use?
  • Clearly defining the AI system’s task and intended purpose helps set boundaries for design, training, deployment, and oversight.
  • A vague or evolving objective may lead to scope creep, misaligned optimization, or unintended consequences. This is especially critical in high-risk use cases where safety, fairness, or legal compliance are required.
  • The intended purpose must also be documented and traceable throughout the lifecycle, this is essential for risk classification, legal accountability, and effective stakeholder communication (e.g., users, regulators, auditors).

Threat-modeling question: Is the AI system's task clearly defined, with well-scoped objectives and boundaries?

Internal MISP references

UUID 93d64b9a-57d1-51a5-9b97-7d14c7cc5385 which can be used as unique global reference for Unclear Task Definition in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Accountability & Human Oversight']
category_id 8
colour #eea4b5
external_id PLOT4AI-126
phases ['Design', 'Input', 'Model']
primary_category Accountability & Human Oversight
question Is the AI system's task clearly defined, with well-scoped objectives and boundaries?
recommendation * Clearly define the problem and outcome you are optimizing for.
  • Assess if your AI system will be well-suited for this purpose.
  • Always discuss if there are alternative ways to solve the problem.
  • Define success. Working with individuals who may be directly affected can help you identify an appropriate way to measure success.
  • Make sure there is a stakeholder involved (product owner for instance) with enough knowledge of the business and a clear vision about what the model needs to do.
  • Have you considered using analytics first? In this context analytics could also offer inspiring views that can help you decide on the next steps. They can be a good source of information and are sometimes enough to solve the problem without the need of AI/ML. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Data Analytics Functionality Index'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Data Analytics Functionality Index | | threatif | No |

Stakeholders' Involvement

  • Do you have all the necessary stakeholders on board? Not having the right people that can give the necessary input can put the design of the AI system in danger.
  • Think for instance when attributes or variables need to be selected, or when you need to understand the different data contexts.
  • Data scientists should not be the only ones making assumptions about variables, it should really be a team effort.

Threat-modeling question: Have we identified and involved all key stakeholders relevant to this phase of the AI lifecycle?

Internal MISP references

UUID 7dc7b253-4987-55f9-8c2f-62700322eb7a which can be used as unique global reference for Stakeholders' Involvement in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Accountability & Human Oversight']
category_id 8
colour #eea4b5
external_id PLOT4AI-127
phases ['Design', 'Input', 'Deploy', 'Monitor']
primary_category Accountability & Human Oversight
question Have we identified and involved all key stakeholders relevant to this phase of the AI lifecycle?
recommendation * Identify and involve relevant stakeholders early in the AI system lifecycle. This will avoid unnecessary rework and frustrations.
  • Identifying who is responsible for making the decisions and how much control they have over the decision-making process allows for a more evident tracking of responsibility in the AI’s development process. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | threatif | No |

Training and Oversight Readiness

Individuals involved in the development, deployment, operation, or use of AI systems must understand their functionality, risks, and limitations. Without adequate training, staff may misuse the system, fail to detect errors, or be unable to intervene effectively. This undermines human oversight, accountability, and compliance with regulatory requirements. Article 4 of the EU AI Act emphasizes the need for AI literacy, particularly for those responsible for high-risk systems.

Threat-modeling question: Have all relevant staff and users received adequate training to understand, oversee, and responsibly interact with the AI system?

Internal MISP references

UUID 4687e64a-5ca1-56a1-960c-7fd592e22d41 which can be used as unique global reference for Training and Oversight Readiness in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Accountability & Human Oversight']
category_id 8
colour #eea4b5
external_id PLOT4AI-128
phases ['Design', 'Deploy', 'Monitor']
primary_category Accountability & Human Oversight
question Have all relevant staff and users received adequate training to understand, oversee, and responsibly interact with the AI system?
recommendation * Provide structured, role-specific training for developers, operators, decision-makers, and users interacting with the AI system.
  • Cover system capabilities, limitations, error detection, appropriate interventions, and escalation procedures.
  • Include modules on fairness, data protection, explainability, and responsible interpretation of AI outputs.
  • Refresh training regularly to reflect system updates and evolving regulations.
  • Track and document training participation as part of accountability measures.
  • Integrate training into onboarding and ongoing professional development frameworks. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Article 4 EU AI Act: I Literacy | | threatif | No |

AI Agents’ Feedback Supervision

  • Reinforcement Learning (RL) is an area of machine learning concerned with how intelligent agents ought to take actions in an environment in order to maximize the notion of cumulative reward. Source: Wikipedia

  • When the agent is learning to perform a complex task, human oversight and feedback are more helpful than just rewards from the environment. Rewards are generally modelled such that they convey to what extent the task was completed, but they do not usually provide sufficient feedback about the safety implications of the agent’s actions. Even if the agent completes the task successfully, it may not be able to infer the side-effects of its actions from the rewards alone. In the ideal setting, a human would provide fine-grained supervision and feedback every time the agent performs an action (Scalable oversight). Though this would provide a much more informative view about the environment to the agent, such a strategy would require far too much time and effort from the human.

Threat-modeling question: Do we have qualified people available to supervise the behavior of AI agents and provide feedback during learning?

Internal MISP references

UUID a4d3c123-7b20-5a95-92b4-470dc939b025 which can be used as unique global reference for AI Agents’ Feedback Supervision in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Generative']
categories ['Accountability & Human Oversight']
category_id 8
colour #eea4b5
external_id PLOT4AI-129
phases ['Design', 'Monitor']
primary_category Accountability & Human Oversight
question Do we have qualified people available to supervise the behavior of AI agents and provide feedback during learning?
recommendation * One promising research direction to tackle this problem is semi-supervised learning, where the agent is still evaluated on all the actions (or tasks), but receives rewards only for a small sample of those actions.
  • Another promising research direction is hierarchical reinforcement learning, where a hierarchy is established between different learning agents. There could be a supervisor agent/robot whose task is to assign some work to another agent/robot and provide it with feedback and rewards. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Concrete Problems in AI Safety', 'Concrete AI Safety Problems'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Concrete Problems in AI Safety Concrete AI Safety Problems | | threatif | No |

Oversight Capacity

  • Human oversight is essential for identifying errors, biases, or unintended consequences in AI systems, especially in high-risk contexts. However, meaningful oversight requires not only a procedural mechanism but also adequate staffing, expertise, training, and organizational support.
  • Without sufficient resources, human reviewers may merely rubber-stamp decisions or fall into automation bias, reducing accountability and increasing the likelihood of harmful outcomes.

Threat-modeling question: Do we have the resources and processes to effectively oversee AI decision-making?

Internal MISP references

UUID 294f9d6d-b505-5409-ab20-679f41d35208 which can be used as unique global reference for Oversight Capacity in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Accountability & Human Oversight']
category_id 8
colour #eea4b5
external_id PLOT4AI-130
phases ['Design', 'Output', 'Monitor']
primary_category Accountability & Human Oversight
question Do we have the resources and processes to effectively oversee AI decision-making?
recommendation * Allocate clear roles and responsibilities for oversight.
  • Train reviewers to recognize automation bias and understand the system's limitations.
  • Establish workflows that support human-in-the-loop or human-on-the-loop oversight.
  • Involve multidisciplinary stakeholders in the review process to ensure meaningful checks and balances. | | roles | ['Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Automation Bias', 'The Flaws of Policies Requiring Human Oversight of Government Algorithms', 'The False Comfort of Human Oversight as an Antidote to AI Harm'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Automation Bias The Flaws of Policies Requiring Human Oversight of Government Algorithms The False Comfort of Human Oversight as an Antidote to AI Harm | | threatif | No |

Escalation Path

  • If an AI system produces harmful or incorrect outputs, is there a predefined process for reporting and addressing these issues?
  • Are employees aware of how to escalate AI failures, and do they have clear channels to report incidents?

Threat-modeling question: Is there a well-defined process to escalate AI-related failures or unexpected outcomes?

Internal MISP references

UUID 1f8ccc66-c46c-57f8-bdfc-28397a1a5b63 which can be used as unique global reference for Escalation Path in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Accountability & Human Oversight', 'Safety & Environmental Impact']
category_id 8
colour #eea4b5
external_id PLOT4AI-131
phases ['Design', 'Deploy', 'Monitor']
primary_category Accountability & Human Oversight
question Is there a well-defined process to escalate AI-related failures or unexpected outcomes?
recommendation * Set up clear escalation protocols to identify, report, and resolve AI-related incidents.
  • Assign responsibilities to ensure accountability for handling AI failures.
  • Keep assessing and improving incident response strategies over time, especially after performing changes or updates in the AI systems. | | roles | ['Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['AI: An Accountability Framework for Federal Agencies and Other Entities', 'The Human Factor in AI Safety'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | AI: An Accountability Framework for Federal Agencies and Other Entities The Human Factor in AI Safety | | threatif | No |

Defined Responsibilities

AI outputs can lead to mistakes or even cause harm. In such cases, is it clear who is responsible within your organization? Are accountability structures clearly defined and documented?

Threat-modeling question: Have we defined who is accountable for the AI system’s decisions and outcomes?

Internal MISP references

UUID 0352f223-510b-5bd3-b91d-9e25f005499d which can be used as unique global reference for Defined Responsibilities in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Accountability & Human Oversight', 'Privacy & Data Protection']
category_id 8
colour #eea4b5
external_id PLOT4AI-132
phases ['Design', 'Input', 'Model', 'Output', 'Deploy', 'Monitor']
primary_category Accountability & Human Oversight
question Have we defined who is accountable for the AI system’s decisions and outcomes?
recommendation * Assign and communicate responsibilities for AI decision-making, considering both legal and ethical accountability.
  • Use decision logs and role-based access control to document and track accountability throughout the AI system’s lifecycle.
  • Get leadership involved in maintaining oversight, keeping accountability a priority at all levels. | | roles | ['Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['AI: An Accountability Framework for Federal Agencies and Other Entities', 'Algorithmic Accountability for the Public Sector'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | AI: An Accountability Framework for Federal Agencies and Other Entities Algorithmic Accountability for the Public Sector | | threatif | No |

Continuous Assessment

AI models and their objectives may drift from their original intent, making human oversight crucial to ensure ongoing alignment with ethical and business objectives. Are there periodic human-led reviews in place to monitor AI system behavior, validate outcomes, and reassess goals? Human oversight should play an active role in detecting unintended consequences, adjusting governance policies, and maintaining accountability throughout the AI system’s lifecycle.

Threat-modeling question: Do we regularly review whether the AI system’s goals, assumptions, and impacts are still appropriate?

Internal MISP references

UUID 05cc2c53-bcf2-5050-84f4-021ec19d73a3 which can be used as unique global reference for Continuous Assessment in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Accountability & Human Oversight', 'Safety & Environmental Impact']
category_id 8
colour #eea4b5
external_id PLOT4AI-133
phases ['Design', 'Output', 'Deploy', 'Monitor']
primary_category Accountability & Human Oversight
question Do we regularly review whether the AI system’s goals, assumptions, and impacts are still appropriate?
recommendation * Schedule regular reassessments of AI objectives and assumptions.

Human Override Mechanisms

  • High-risk AI systems must provide natural persons with the means to stop or override the system when necessary. This includes mechanisms such as a 'stop button' or fallback procedures that bring the system to a safe state.
  • A lack of override capabilities could lead to harm, especially in autonomous systems where malfunction or misalignment may go unnoticed without human intervention.

Threat-modeling question: Can human operators safely interrupt or override the AI system at any time?

Internal MISP references

UUID 2ed35e7a-a671-56f2-9396-9d49835a38cc which can be used as unique global reference for Human Override Mechanisms in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Accountability & Human Oversight', 'Safety & Environmental Impact']
category_id 8
colour #eea4b5
external_id PLOT4AI-134
phases ['Design', 'Deploy', 'Monitor']
primary_category Accountability & Human Oversight
question Can human operators safely interrupt or override the AI system at any time?
recommendation * Design systems with built-in override or halt capabilities.
  • Ensure that these mechanisms are tested regularly and accessible to responsible personnel.
  • Document override procedures clearly and provide training to relevant users. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['AI Act – Article 14(4)(e)'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | AI Act – Article 14(4)(e) | | threatif | No |

Contestability of AI Decisions

Some AI systems make or support decisions that significantly affect individuals, such as in hiring, lending, or criminal justice. If users cannot challenge these decisions or request human review, the system may violate oversight obligations and erode trust. Lack of contestability undermines accountability and may breach Article 22(3) of the GDPR or Article 14 of the EU AI Act, both of which require mechanisms for human intervention and review.

Threat-modeling question: Could users contest or challenge the decisions made by the AI system?

Internal MISP references

UUID a73adfaa-9c2f-5a44-8572-28a734ab56a2 which can be used as unique global reference for Contestability of AI Decisions in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Accountability & Human Oversight', 'Ethics & Human Rights']
category_id 8
colour #eea4b5
external_id PLOT4AI-135
phases ['Design', 'Model', 'Output', 'Monitor']
primary_category Accountability & Human Oversight
qr Charter of Fundamental Rights of the European Union
question Could users contest or challenge the decisions made by the AI system?
recommendation * Ensure the AI system includes mechanisms for contestability, allowing users to challenge or seek review of decisions that negatively impact them. Wrong decisions could also have an impact on people that have not been the target of the data collection (data spillovers).
  • Provide clear instructions on how users can initiate such challenges and ensure that this process is transparent, accessible, and user-friendly.
  • Incorporate features that enable human oversight in decision-making processes, ensuring users have the option to escalate issues to human operators.
  • Establish a redressal process that includes timelines for resolution, a clear escalation hierarchy, and mechanisms for feedback integration to improve the system’s decision-making over time.
  • Regularly audit and evaluate the decision-making outcomes of the AI system, focusing on areas where users frequently raise disputes. Use these audits to improve system accuracy and reduce the need for contestation.
  • Provide detailed and comprehensible explanations of the system’s outputs to users, ensuring they understand how decisions are made and what data was used.
  • Engage relevant stakeholders, including legal experts, ethicists, and representatives from affected user groups to design and evaluate the contestability mechanisms and ensure they meet ethical and regulatory standards.
  • Train system operators and customer support staff to handle disputes arising from the AI system effectively, ensuring they are equipped to assist users in navigating the contestation process. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Ethics guidelines for trustworthy AI', 'Operational Guidance on taking account of Fundamental Rights in Commission Impact Assessments', 'Artificial Intelligence and Fundamental Rights'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Right to be treated fairly by a court (Universal Declaration of Human Rights), article 11 Freedom of expression and information, article 47 Right to an effective remedy and to a fair trial (Charter of fundamental rights of the European Union). Ethics guidelines for trustworthy AI Operational Guidance on taking account of Fundamental Rights in Commission Impact Assessments Artificial Intelligence and Fundamental Rights | | threatif | No |

Liability Risk

  • Failing to assess liability risks can expose your organization to legal, financial, and reputational damage.
  • Have you identified who could be held liable, your organization, end-users, third parties?
  • Black-box AI systems complicate the attribution of responsibility, especially in high-risk or harmful scenarios, increasing the burden of proof for affected individuals.
  • Legal liability varies across jurisdictions, and evolving regulations such as the EU AI Liability Directive may significantly affect your obligations.
  • Failure to document decision-making processes and ensure auditability can weaken your defense in case of litigation.

Threat-modeling question: Have we assessed our legal liability for damages caused by our AI system?

Internal MISP references

UUID 0e4a837a-ae0b-5f65-8269-fca458cd3dda which can be used as unique global reference for Liability Risk in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Accountability & Human Oversight', 'Ethics & Human Rights']
category_id 8
colour #eea4b5
external_id PLOT4AI-136
phases ['Design', 'Deploy', 'Monitor']
primary_category Accountability & Human Oversight
question Have we assessed our legal liability for damages caused by our AI system?
recommendation * Conduct a liability risk assessment for your AI system, including mapping potential damages and responsible parties.
  • Implement transparency, traceability, and auditability mechanisms throughout the AI lifecycle to support accountability.
  • Ensure that risk scenarios are documented and, where relevant, covered by insurance policies.
  • Stay informed about legal developments in AI liability.
  • Provide training to relevant teams on legal accountability and AI governance best practices. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['Liability Rules on Artificial Intelligence'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | Liability Rules on Artificial Intelligence | | threatif | No |

Lack of Governance Practices

MLOps (Machine Learning Operations) refers to the engineering and governance practices required to reliably develop, deploy, and monitor machine learning models in production. Without proper MLOps, organizations may face: * Model Drift: Performance degradation due to changes in input data or real-world conditions. * Lack of Traceability: Difficulty reproducing results or auditing decisions. * Operational Failures: Models failing silently or behaving unpredictably in production. * Compliance Risks: Inability to demonstrate accountability or meet regulatory requirements.

MLOps is especially important for high-risk AI applications under the EU AI Act, where continuous monitoring, retraining, and documentation are legal obligations.

Threat-modeling question: Do we have adequate resources and MLOps practices in place to manage, monitor, and maintain our AI system?

Internal MISP references

UUID 8f2adc2b-a1cc-590b-b62b-e7d447a141f0 which can be used as unique global reference for Lack of Governance Practices in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Accountability & Human Oversight', 'Safety & Environmental Impact', 'Data & Data Governance']
category_id 8
colour #eea4b5
external_id PLOT4AI-137
phases ['Model', 'Deploy', 'Monitor']
primary_category Accountability & Human Oversight
question Do we have adequate resources and MLOps practices in place to manage, monitor, and maintain our AI system?
recommendation * Establish clear MLOps processes including versioning, CI/CD pipelines, and model registry.
  • Continuously monitor model performance, fairness, and drift.
  • Ensure auditability by logging predictions, training runs, and data lineage.
  • Automate testing and rollback mechanisms for safe model updates.
  • Define clear responsibilities between data scientists, ML engineers, and operations staff.
  • Include human-in-the-loop checks or alerts for sensitive or safety-critical applications. | | roles | ['Provider', 'Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_names | ['MLOps: Continuous delivery and automation pipelines in machine learning'] | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | source_text | MLOps: Continuous delivery and automation pipelines in machine learning | | threatif | No |

Shared Responsibility

If you use a third-party tool you might still have a responsibility towards the users. Think about employees, job applicants, patients, etc. It is also your responsibility to make sure that the AI system you choose won't cause harm to the individuals.

Threat-modeling question: If we plan to deploy a third-party AI tool, have we assessed our shared responsibility for its potential impact on users?

Internal MISP references

UUID d9a497df-37ea-5021-be0f-df7843307658 which can be used as unique global reference for Shared Responsibility in MISP communities and other software using the MISP galaxy

External references
Associated metadata
Metadata key Value
aitypes ['Traditional', 'Generative']
categories ['Accountability & Human Oversight', 'Cybersecurity']
category_id 8
colour #eea4b5
external_id PLOT4AI-138
phases ['Design', 'Input', 'Deploy', 'Monitor']
primary_category Accountability & Human Oversight
question If we plan to deploy a third-party AI tool, have we assessed our shared responsibility for its potential impact on users?
recommendation If personal data is involved, review which ones are your responsibilities (look into art. 24 and 28 GDPR).

You can also start by checking: * That you have the right agreements in place with the third party provider. * That the origin and data lineage of their datasets are verified. * How their models are fed; do they anonymize the data? * How you have assessed their security, ethical data handling, quality processes and measures to prevent bias and discrimination in their AI system. * That you have informed users accordingly. | | roles | ['Deployer'] | | source_deck | https://github.com/PLOT4ai/plot4ai-library/blob/main/deck.json | | source_license | CC-BY-SA-4.0 | | source_repository | https://github.com/PLOT4ai/plot4ai-library | | threatif | No |