Adversarial Deep Reinforcement Learning for Improving the Robustness of Multi-agent Autonomous Driving Policies

Aizaz Sharif1, Dusica Marijan1,
1Simula Research Laboratory,
Paper arXiv Code
Paris

Abstract

Autonomous cars are well known for being vulnerable to adversarial attacks that can compromise the safety of the car and pose danger to other road users. To effectively defend against adversaries, it is required to not only test autonomous cars for finding driving errors but to improve the robustness of the cars to these errors. To this end, in this paper, we propose a two-step methodology for autonomous cars that consists of (i) finding failure states in autonomous cars by training the adversarial driving agent, and (ii) improving the robustness of autonomous cars by retraining them with effective adversarial inputs. Our methodology supports testing autonomous cars in a multi-agent environment, where we train and compare adversarial car policy on two custom reward functions to test the driving control decision of autonomous cars. We run experiments in a vision-based high-fidelity urban driving simulated environment. Our results show that adversarial testing can be used for finding erroneous autonomous driving behavior, followed by adversarial training for improving the robustness of deep reinforcement learning-based autonomous driving policies. We demonstrate that the autonomous cars retrained using the effective adversarial inputs noticeably increase the performance of their driving policies in terms of reduced collision and offroad steering errors.

In this work, we propose a framework named MAD-ARL which is a multi-agent driving environment designed for improving the robustness of autonomous cars using adversarial driving models. Adversarial reinforcement learning (ARL) is trained against a victim player in order to find unwanted driving decisions of autonomous cars that are also trained on a DRL-based policy. By exposing the same adversarial car against the victim agents for retraining, the agents show improvements in their end-to-end decision driving controls, mainly in terms of fewer collisions and offroad steering errors compared to their originally trained (adversary-free) policies.

BibTeX

@INPROCEEDINGS{10043282,
  author={Sharif, Aizaz and Marijan, Dusica},
  booktitle={2022 29th Asia-Pacific Software Engineering Conference (APSEC)}, 
  title={Adversarial Deep Reinforcement Learning for Improving the Robustness of Multi-agent Autonomous Driving Policies}, 
  year={2022},
  volume={},
  number={},
  pages={61-70},
  doi={10.1109/APSEC57359.2022.00018}}
}