Artificial Intelligence for Network Security: A Review of Machine Learning-Based Intrusion Detection, Anomaly Detection, and Threat Intelligence

Abstract

The increasing sophistication of cyberattacks and the growing scale of network infrastructures have created significant challenges for conventional network security mechanisms. Artificial Intelligence (AI), particularly Machine Learning (ML) and Deep Learning (DL), has become an important technology for detecting malicious activities and identifying previously unknown network threats. This review examines the application of AI techniques in network security, focusing on intrusion detection systems, anomaly detection, malware traffic classification, botnet detection, denial-of-service detection, and threat intelligence. Supervised, unsupervised, semi-supervised, and deep learning approaches are systematically compared according to their detection capabilities, datasets, feature representations, evaluation metrics, and computational requirements. The paper additionally investigates emerging techniques such as federated learning, graph neural networks, explainable AI, and large language models for cybersecurity applications. Major challenges—including adversarial attacks, false positives, imbalanced datasets, concept drift, privacy concerns, and the lack of representative datasets—are critically analyzed. The review concludes with research opportunities for developing robust, explainable, privacy-preserving, and adaptive AI-based network security systems.