Recently, the application of machine learning (ML) in the field of cybersecurity, particularly in the detection and prevention of malware, has received significant attention and interest. Numerous research works on malware analysis have been proposed, showing promising results for practical applications. In such works, the use of Generative Adversarial Networks (GANs) or Reinforcement Learning (RL) can help adversaries create mutated malware to evade detection. In this study, we propose a method for generating mutated Windows malware against malware detection based on ensemble learning by combining GANs and RL to overcome the limitations of the MalGAN model. Specifically, we develop the FeaGAN model, an extension of MalGAN, by incorporating the model with the Deep Q-network anti-malware Engines Attacking Framework (DQEAF) RL model. Furthermore, the FeaGAN model employs ensemble learning for malware detection to enhance the evasion capabilities of the generated adversarial samples. Experimental results show that 100% of the selected mutation samples maintain their format integrity. Additionally, the ability to preserve the executable functionality of the malware variants achieves promising results with a stable success rate.
