As AI/ML applications handle sensitive data and influence critical decisions, security breaches can have severe consequences. Penetration testing specifically designed for AI/ML applications is essential to identify and address potential vulnerabilities and threats before they can be exploited.
Why AI/ML Penetration Testing is Crucial:
Benefits
This list emphasizes the critical vulnerabilities found in Machine Learning (ML) models, the core of Artificial Intelligence (AI), and their potential business impact. By conducting penetration testing specifically for ML models, organizations can proactively address these vulnerabilities and:
Prevent Model Evasion
Ensure attackers can’t manipulate inputs to trick the model into making incorrect predictions.
Safeguard Model Integrity
Mitigate the risk of attackers altering the model itself, potentially leading to biased or flawed outputs
Protect ML Intellectual Property
Prevent unauthorized access to the models themselves, which can be valuable assets for businesses
Secure Connected Systems
Identify vulnerabilities in systems that rely on ML models to function, preventing attackers from exploiting these weaknesses.
Prevent Denial-of-Service (DoS) Attacks on ML Models
Ensure attackers can’t overload or disrupt the model, rendering it unavailable for legitimate use
Process
1
Reconnaissance
1. Identify Publicly Accessible Research on Target Models.
2. Locate Publicly Known Adversarial Research.
3. Scan Victim-Controlled Websites.
4. Explore Public Application Code Repositories.
5. Active Reconnaissance and Scanning.
2. Locate Publicly Known Adversarial Research.
3. Scan Victim-Controlled Websites.
4. Explore Public Application Code Repositories.
5. Active Reconnaissance and Scanning.
2
Resource and development
1. Gather Public Machine Learning Resources
2. Obtain Capabilities for
   2.1 Adversarial Machine Learning Attack Methods
  2.2 Relevant Software and Tools
3. Develop Adversarial Machine Learning Attack Techniques
4. Secure Resources for:
  4.1 Machine Learning Development Environments:
  4.2 Computational Resources
5. Introduce Manipulated Datasets
6. Compromise Training Data.
7. Create Accounts (on Targeted Platforms)
2. Obtain Capabilities for
   2.1 Adversarial Machine Learning Attack Methods
  2.2 Relevant Software and Tools
3. Develop Adversarial Machine Learning Attack Techniques
4. Secure Resources for:
  4.1 Machine Learning Development Environments:
  4.2 Computational Resources
5. Introduce Manipulated Datasets
6. Compromise Training Data.
7. Create Accounts (on Targeted Platforms)
3
Initial Access
1. Supply Chain Compromise
  a. Hardware.
  b. Software.
  c. Data.
  d. Model.
2. Unauthorized Access.
3. Model Evasion.
4. Public Application Exploits
  a. Hardware.
  b. Software.
  c. Data.
  d. Model.
2. Unauthorized Access.
3. Model Evasion.
4. Public Application Exploits
4
ML Model Access
1. Compromising ML Model Inference Access.
2. Exploiting an ML-Powered Product/Service.
3. Physical Access to Machine Learning Environment.
4. Complete Compromise of the Machine Learning Model.
2. Exploiting an ML-Powered Product/Service.
3. Physical Access to Machine Learning Environment.
4. Complete Compromise of the Machine Learning Model.
5
Execution
1. User Execution
  a. Deploying Untrusted Machine Learning Resources.
2. Script Execution Environment.
  a. Deploying Untrusted Machine Learning Resources.
2. Script Execution Environment.
6
Persistence
1. Data Manipulation
2. Model Tampering
  a. Compromised Machine Learning Model
  b. Embed Malicious Functionality.
2. Model Tampering
  a. Compromised Machine Learning Model
  b. Embed Malicious Functionality.
7
Defence Evasion
Bypass Machine Learning Model
8
Discovery
1. Identify Machine Learning Model Details
2. Identify Similar Machine Learning Models.
3. Locate Machine Learning Resources.
2. Identify Similar Machine Learning Models.
3. Locate Machine Learning Resources.
9
Collection
1. Machine Learning Resource Collection.
2. Data Acquisition from External Sources.
3. Data Acquisition from Local Systems.
2. Data Acquisition from External Sources.
3. Data Acquisition from Local Systems.
