An adaptive firewall framework using deep reinforcement learning for threat-aware cyber threat detection and mitigation policy learning

Abstract

<p>Rapid digital infrastructure growth has enabled sophisticated multi-vector cyberattacks that bypass traditional signature-based intrusion detection and static firewall configurations. Current defenses rely on fixed rule-based engines, signature-dependent classifiers, or threshold-based anomaly detectors, all of which struggle with con current threat patterns, zero-day attacks, and high-dimensional decision spaces. Machine and deep learning approaches for intrusion detection also face limitations, including slow adaptation to novel threats, high false positive rates, and limited support for dynamic firewall policy adjustment. To address these limitations, we propose an Adaptive Firewall Framework using Deep Reinforcement Learning (DRL) for threat-aware cyber threat detection and mitigation policy learning. The framework incorporates a policy-optimization DRL agent with a novel Threat-Aware Prioritized Experience Replay (TA-PER) mechanism, which prioritizes threat expe riences according to operational severity, CVSS criticality, and buffer rarity. Through state-space modeling, threat-sensitive reward shaping, and incremental threat scoring, the DRL-TA-PER agent learns firewall response policies, including traffic blocking, isolation, adaptive rerouting, throttling, and benign traffic allowance. The framework is evaluated in an offline sequential setting using public benchmark datasets, where network-flow records are processed as ordered decision steps. Evaluation on CIC-IDS-2018, UNSW-NB15, CIC-DDoS2019, and CIC-IoT-2023 shows that the proposed framework outperforms K-Nearest Neighbor, Random Forest, Lo gistic Regression, Support Vector Machine, Convolutional Neural Network, Long Short-Term Memory, standard DRL with uniform experience replay, Snort, and Suricata. The model achieved 96.8% accuracy, 97.2% recall, and an AUC-ROC of 0.981 across five trials, indicating that threat-aware replay prioritization can improve DRL-based firewall policy learning under offline benchmark evaluation conditions.<br></p>