ACADIA: Efficient and Robust Adversarial Attacks Against Deep
Reinforcement Learning

Haider Ali

Artificial Intelligence (AI) techniques such as Deep Neural Networks (DNN) and Deep Reinforcement Learning (DRL) are prone to adversarial attacks. For example, a perturbed stop sign can force a self-driving car’s AI algorithm to increase the speed rather than stop the vehicle.

There has been little work developing attacks and defenses against DRL. In DRL, a DNN based policy decides to take an action based on the observation into the environment and gets the reward in feedback for its improvements.

We perturb that observation to attack the DRL agent. There are two main aspects to developing an attack on DRL. One aspect is to identify the optimal time to attack (when-to-attack?). Second aspect is to identify an efficient method to attack (how-to- attack?).

To answer the second aspect, we propose a suite of novel DRL adversarial attacks, called ACADIA, representing Attack’s Against Deep reinforcement learning. We consider two well- known DRL algorithms, Deep-Q Learning Network (DQN) and Proximal Policy Optimization (PPO), under DRL environments of Atari games and MuJoCo where both targeted and non-targeted attacks are considered with or without the state-of-the-art defenses.

Our results demonstrate that the proposed ACADIA outperforms state-of-the-art perturbation methods under a wide range of experimental settings. ACADIA is nine times faster than the state-of-the-art Carlini & Wagner (CW) method with better performance under defenses of DRL.

1.1 Motivation

Deep Reinforcement Learning (DRL) algorithms learn policies to guide a DRL agent to take optimal actions based on the state of the environment. These algorithms have successfully achieved high performance on various complex as well as critical tasks, such as robotics [3], autonomous vehicles [22], and cybersecurity [5].

A policy, a probabilistic distribution of actions by the DRL agent, is learned by Deep Neural Networks (DNN) to approximate the action-value function. The vulnerabilities of DNNs to adversarial attacks have been significantly studied [16, 38, 46] to mitigate the impact of the adversarial attacks when the DNNs are exploited by the adversaries.

Common adversarial examples include adversarial perturbations imperceptible to humans but fooling DNNs easily in the testing or deployment stage [46]. A self-driving can be fooled by a small perturbation in traffic signals to cause accidents as shown in Figure 1.1. For example, a perturbed stop sign can be perceived as 45 speed limit and perturbed 35 speed limit traffic signals can be perceived as 85 speed limits.

Researchers have explored various attacks and defenses for supervised DNN applications, such as image classification [16] or natural language processing [2]. However, adversarial attacks and defenses are largely unexplored in DRL environments. DRL also has numerous critical safety and security applications and accordingly drew our attention to the need for robust DRL. For robust DRL, there is a prerequisite of developing efficient, effective,

Figure 1.1: Motivation of this dissertation: Perturbations in traffic signals can turn them into hazardous traffic signals. For example, Stop Sign can be perceived as 45 Speed, or 35 speeds can be perceived as 85 speed limit using small perturbations.

And robust adversarial attacks which can be used to evaluate the robustness of defense mechanisms.

In the adversarial machine learning research community, researchers have developed adversarial attacks in DRL by answering the following two questions:

(1) How to attack? and (2) When to attack?

The first how-to-attack question is related to what perturbation method should be used for disrupting the state during an episode. The second when-to-attack question is associated with identifying an optimal time to attack during an episode.

In this work, we aim to answer how-to-attack by proposing ACADIA, a set of novel adversarial Attacks Against Deep reinforcement learning. To be specific, the goal of this work is to develop robust and fast attacks by generating effective and efficient adversarial states in DRL settings.

Unlike DNN settings, there are non-trivial challenges in developing efficient and effective adversarial states under various DRL settings as shown in Figure 1.2: First, there is no stationary dataset and correct action available in DRL settings.

Figure 1.2: Differences between DRL (left) and DNN (right) processes. Unlike single classification in DNN, DRL is a continuous learning process driven by reward as feedback given by the environment.

Instead, there is a dynamic series of steps where the DRL agent is continuously learning through a reward upon taking a series of actions. This means that the DRL agent is continuously tackling multiple situations in an episode.

Second, unlike DNN environments, there can be discrete as well as continuous action spaces in the DRL depending upon the environment. Third, defenses in DRL work on different principles as compared to the defenses in DNNs. Previous attack variants are not comprehensive enough to be used in various DRL settings.

1.2 Key Contributions

In this work, we propose ACADIA that integrates momentum, ADAM optimizer, random initialization and Fast Gradient Sign Method (FGSM) to effectively and efficiently solve the challenges faced in DRL settings. The existing state perturbation attacks are not comprehensive enough to be used in various DRL settings.

Our proposed ACADIA is the first novel effective and efficient perturbation attacks under the DRL. We compare the performance of our proposed ACADIA with those of state-of-the-art adversarial attacks in DRL application environments under various settings with or without defenses when attackers may perform targeted or non-targeted attacks.

Via extensive comparative performance analyses, we validated the performance of ACADIA in terms of attack success rate metrics (ASR), average attack execution time per perturbation (AET), and average reward (AR) to the DRL agent.

ACADIA is a research work that focuses on efficient and robust adversarial attacks against deep reinforcement learning (DRL) agents. Let me provide you with more details about it:

Description:
- ACADIA provides a set of efficient and robust perturbation-based adversarial attacks.
- These attacks aim to disturb the decision-making process of DRL agents.
- The attacks are based on novel combinations of techniques, including:
  - Utilizing momentum.
  - Leveraging the ADAM optimizer (specifically, Root Mean Square Propagation or RMSProp).
  - Incorporating initial randomization ¹.
Availability:
- You can find more information about ACADIA in the following places:

Feel free to explore these resources to learn more about ACADIA and its contributions to adversarial attacks in deep reinforcement learning! 😊

Don’t Stop Now👀

“ACADIA: Efficient and Robust Adversarial Attacks Against Deep Reinforcement Learning”

Click on the Blue Button Below for Instant Access!

Our 100% Money Back Guarantee:

If for any reason you decided within 30 days that “ACADIA: Efficient and Robust Adversarial Attacks Against Deep Reinforcement Learning” isn’t for you, simply notify us by email and we’ll gladly refund your money – no questions asked. That’s our Ironclad Guarantee! The risk is entirely ours! You absolutely have nothing to lose!

Confirm Subscription

Just use your name and valid email address – I will never sell or share your email address with anyone. Never. You may unsubscribe anytime. I hate spam just as much as you do.

Warmest Regards, Coyalita

Behavioral Health Rehabilitative Specialist & Addiction Counselor

Share on Social Media