Testing Self-Healing Cyber-Physical Systems Under Uncertainty with Reinforcement Learning


Self-healing behaviors are essential for the dependability of Cyber-Physical Systems (CPSs) in the presence of faults and uncertainties. Therefore, it is critical to test if such behaviors can correctly heal faults under uncertainties. By testing such behaviors in various conditions and learning from testing results, reinforcement learning algorithms can gradually optimize a testing policy and apply it to detect faults. However, there is insufficient evidence to know which reinforcement learning algorithms perform the best in terms of testing self-healing behaviors of CPSs under uncertainties. To this end, we conducted an empirical study to evaluate fault detection abilities of 14 combinations of reinforcement learning algorithms, with two value function learning-based methods for operation invocations and seven policy optimization-based algorithms for introducing uncertainties.

A modeling and testing framework, MOSH , is developed to specify test models and test Self-Healing Cyber-Physical Systems (SH-CPSs) under uncertainty. Two value function learning based algorithms (Q-learning and SARSA) and seven policy optimization based algorithms (ACER, A3C, ACKTR, TRPO, DDPG, PPO, and UPO) are implemented based on the OpenAI Baselines

Test Models

The test models used in the empirical study can be downloaded from the table.

# Case Study Download
1 ArduCopter download
2 ArduPlane download
3 ArduRover download
4 Adaptive Production Cell download
5 RailCab download
6 Mobile Robot download