AI RESEARCH PAPER ANALYSIS

ML/DL Baselines for Intrusion Detection

Summary: Traditional ML and DL models are well-established for IDS, offering robust performance and efficiency, especially with structured data like Zeek logs. They serve as strong benchmarks for evaluating newer LLM approaches.

Key Findings:

• ML models (XGBoost, RF, DT) demonstrate dominant performance and efficiency. XGBoost reached a 96.96% F1-score with only 4% CPU usage.
• ML models have the smallest resource footprint (<1MB disk, <0.6GB RAM, <2% CPU, no GPU requirement).
• DL models (MLP, GRU, LeNet-5) show mixed results; GRU benefited from larger datasets, but MLP and LeNet-5 struggled in multiclass classification (<0.85 F1-score).
• DL models require more resources than ML (up to 0.5MB disk, 1.75GB RAM, 25% CPU for MLP, 4.8GB GPU).
• XGBoost exhibits near-perfect classification in multiclass scenarios, with very minimal confusion, particularly in distinguishing Mirai from other attack types.

Implications: ML models, especially tree-based methods like XGBoost, remain the optimal choice for large-scale, real-time IDS deployments due to their superior accuracy, rapid inference speed, and minimal resource requirements. Their established performance and efficiency make them a reliable foundation for robust cybersecurity systems.

Comparative Performance & Efficiency

Summary: This section provides a direct comparison of classification effectiveness, inference speed, and resource consumption across LLM, ML, and DL models, revealing clear trade-offs essential for deployment decisions.

Key Findings:

• Effectiveness (F1-score): ML models (XGBoost >0.96) generally outperformed LLMs (GPT-Neo >0.96), which, in turn, surpassed DL models (GRU/MLP/LeNet-5 <0.85 in multiclass).
• Inference Speed (flows per second): ML models (DT 6.25M fps, XGBoost 1.5M fps) were orders of magnitude faster than DL (MLP 200k fps, GRU 12k fps) and LLMs (GPT-Neo 517 fps, LLaMA 105 fps). LLMs were approximately 10,000 times slower than the top ML performers.
• Resource Consumption:
  • Disk: ML (low <1MB) < DL (low <0.5MB) < LLM (high, 2GB+ for pretrained models with adapters).
  • RAM: ML (<0.6GB) < DL (up to 1.75GB) < LLM (1.74-2.74GB).
  • CPU: ML (<2%) < DL (up to 25% for MLP) < LLM (3-12% for GPT-Neo/GPT-2).
  • GPU: ML (none) << LLM (667MB-3.8GB, 12-96% utilization) ≈ DL (4.8GB).
• Training Strategy Impact: Fine-tuning pretrained LLMs (Experiment 1) showed marginal performance improvement with more data. Training from scratch (Experiment 2) achieved near-optimal performance with fewer samples, indicating architecture's intrinsic suitability.

Implications: A clear trade-off exists: ML models are ideal for large-scale, real-time, resource-constrained IDS. LLMs, despite competitive performance, incur higher inference costs and resource demands, suggesting their role in specialized settings where their semantic richness and interpretability could be advantageous, potentially in layered or filtered deployment strategies.

Explainability & Future Research Directions

Summary: The study utilized t-SNE and SHAP to analyze ML model decision-making processes, laying a groundwork for future efforts to enhance LLM explainability in IDS. It also highlights limitations and promising avenues for future research.

Key Findings:

• t-SNE Analysis: Revealed distinct clusters for DDoS, Mirai, and Benign traffic. Scanning flows significantly overlapped with Benign in 2D projections but showed a nuanced separation in 3D, explaining frequent misclassifications.
• SHAP Analysis (XGBoost): Identified `tunnel_parents`, `orig_bytes`, and `proto` as highly influential features. `tunnel_parents` was strongly associated with Mirai traffic, confirming that a small subset of features drives predictive signals for robust ML performance.
• Limitations of the Study: SHAP-based interpretation was limited to tree-based models, leaving LLM decision pathways opaque. Only three relatively small LLMs were evaluated. The dataset focused on known attack types, limiting generalizability to novel threats. Single-run results for models.

Future Work:

• Explore advanced prompt engineering for LLMs to improve precision in ambiguous traffic classes.
• Investigate continual and online learning frameworks for LLMs to adapt to evolving threat landscapes.
• Evaluate larger LLM architectures to understand the full spectrum of trade-offs.
• Develop hybrid detection pipelines where ML performs fast triage, and LLMs serve as secondary classifiers or explainability engines.
• Assess LLM robustness through cross-dataset generalization and zero-shot transfer to unseen environments.
• Integrate attention visualization or token-level attribution methods to bring LLM explainability closer to SHAP-level transparency.

Implications: ML models offer immediate interpretability. Future hybrid approaches could effectively leverage LLM semantic depth for complex contextual understanding and explainability, complementing ML's efficiency in a layered IDS framework.