AI-Driven Automation of SOC Workflows Using Hybrid Models for Trojan and DDoS Attack Identification

This research systematically addresses the increasing operational burden experienced by Security Operations Center (SOC) analysts by introducing an intelligent automation framework for cyber threat detection. Modern SOC environments are overwhelmed by high-volume alert streams, heterogeneous data sources, and sophisticated attack vectors, often leading to alert fatigue, delayed incident response, and increased false-positive rates. To mitigate these challenges, we propose a dual-model machine learning architecture specifically engineered to automate the detection of two high-impact and widely prevalent cyber threats: Trojan malware intrusions and Distributed Denial of Service (DDoS) attacks. The proposed framework adopts a modular design in which each attack category is managed by a dedicated, independently optimized detection engine. This separation enhances model specialization, interpretability, and scalability while minimizing computational overhead. By offloading repetitive analytical tasks to intelligent models, the framework significantly reduces manual triage efforts and enables SOC analysts to focus on high-priority incidents and strategic threat analysis. The first detection engine targets Trojan-based intrusions using a hybrid deep learning methodology that combines unsupervised and supervised learning paradigms. A multi-layer stacked autoencoder is employed for hierarchical feature extraction and dimensionality reduction. From an initial set of 86 network traffic attributes—including flow duration, packet statistics, header metadata, and protocol behavior—the autoencoder compresses the input space into a compact 16-dimensional latent representation. This latent embedding effectively captures nonlinear correlations and subtle behavioral signatures associated with Trojan communication patterns, including command-and-control (C2) activities and covert data exfiltration. The reduced feature set is subsequently processed by a Random Forest classifier, which performs robust ensemble-based supervised classification. The integration of ensemble decision trees enhances generalization capability, improves resilience against overfitting, and significantly lowers false-positive rates compared to traditional signature-based or shallow learning approaches. The second detection engine focuses on identifying DDoS attacks through a multi-stage analytical pipeline designed to capture both volumetric anomalies and temporal attack evolution. Initially, an autoencoder-based anomaly detection module establishes a baseline model of legitimate traffic behavior and flags statistically significant deviations. These anomalous traffic instances are then evaluated using a Logistic Regression classifier, which provides probabilistic discrimination between benign bursts and malicious flooding activities.