Overview

This study introduces a deep learning surrogate model-based reinforcement learning (DL-MBRL) for active control of two-dimensional (2D) wake flow past a square cylinder confined between parallel walls using antiphase jets. In the training of this framework, a proximal policy optimisation (PPO) reinforcement learning agent alternates its interaction between a deep learning-based surrogate model (DL-SM) and a computational fluid dynamics (CFD) simulation to suppress wake vortex shedding, thereby significantly reducing computational costs. The DL-SM, built with a Transformer for temporal dynamics and a multiscale enhanced super-resolution generative adversarial network (MS-ESRGAN) for spatial reconstruction, is trained on 2D direct numerical simulation wake flow data to effectively and accurately emulate complex nonlinear flow behaviours. Compared to standard model-free reinforcement learning, the DL-MBRL approach reduces training time by about 50% while maintaining or improving wake stabilisation. Specifically, it achieves approximately a 98% reduction in shedding energy and a 95% reduction in the standard deviation of the lift coefficient, demonstrating strong suppression of vortex shedding. By leveraging the inherent stochasticity of DL-SM, DL-MBRL also addresses the nonzero mean lift coefficient issue observed in model-free methods, promoting more robust exploration. These results highlight the potential of the framework for extension to practical and industrial flow control problems.

Dependencies

Python 3.6

Tensorflow 1.13.1

Tensorforce 0.4.2

CFD numerical simulation: open-source code Nek5000.

Training

-Transformer training

Use the training data in folder ‘DL-SM/Transformer’ and Transformer-tf1.py to train the transformer model.

-MS-ESRGAN training

Use the training data in folder ‘DL-SM/ESRGAN’ and ESRGAN-tf1.py to train MS-ESRGAN.

-DL-MBRL training

Use the training.py in the folder ‘DL-MBRL’ to train the DL-MBRL framework.