Authors

Reginald Ekene Ogu

Department of Computer Engineering, Federal University of Technology, Owerri, Imo State, Nigeria

Abstract

The TON_IoT telemetry files, covering seven IoT and IIoT devices, are widely used to train and compare attack detectors, and many studies report near-perfect accuracy on these files. This audit asks what those scores measure. Each device file was examined for class timing, sampling behaviour, value encoding and information content. Classifiers were then trained on sensor values, timestamp features and acquisition features under random and within-class chronological splits. Every attack class was recorded in its own time window, and all normal data predate the attacks by three weeks. Timestamp features alone reached a macro-F1 of 1.00 on six devices. Normal readings were burst-sampled, so reading intervals alone separated normal from attack traffic with binary F1 of 0.99 or higher on three devices. In the garage-door file, one field is encoded as true/false for every normal row and 0/1 for every attack row. With sensor values only, macro-F1 under random splitting ranged from 0.07 to 0.98, but under chronological blocking the best device fell from 0.98 to 0.46. Three devices carry so little information that no classifier can exceed 38% accuracy. Recommendations for evaluation and dataset maintenance are given.

Keywords

IoT telemetry shortcut learning data leakage temporal confounding intrusion detection dataset audit

Citation of this Article

Reginald Ekene Ogu. (2026). An Audit of Temporal and Acquisition Confounds in the TON_IoT Telemetry Datasets. Current Journal of Engineering and Science Research. 3(11), 26-34. Article DOI: https://doi.org/10.47001/CJESR/2026.311004 

Licence Copyright (c) 2026 Current Journal of Engineering and Science Research. This work is licensed under a Creative Commons Attribution Non Commercial 4.0 International Licence.

References

  1. A. Alsaedi, N. Moustafa, Z. Tari, A. Mahmood, and A. Anwar, “TON_IoT telemetry dataset: A new generation dataset of IoT and IIoT for data-driven intrusion detection systems,” IEEE Access, vol. 8, pp. 165130–165150, 2020, doi: 10.1109/ACCESS.2020.3022862.
  2. M. A. Umer, K. N. Junejo, M. T. Jilani, and A. P. Mathur, “Machine learning for intrusion detection in industrial control systems: Applications, challenges, and recommendations,” International Journal of Critical Infrastructure Protection, vol. 38, Art. no. 100516, 2022, doi: 10.1016/j.ijcip.2022.100516.
  3. A. M. Y. Koay, R. K. L. Ko, H. Hettema, and K. Radke, “Machine learning in industrial control system (ICS) security: Current landscape, opportunities and challenges,” Journal of Intelligent Information Systems, vol. 60, no. 2, pp. 377–405, 2023, doi: 10.1007/s10844-022-00753-1.
  4. N. Moustafa, “A new distributed architecture for evaluating AI-based security systems at the edge: Network TON_IoT datasets,” Sustainable Cities and Society, vol. 72, Art. no. 102994, 2021, doi: 10.1016/j.scs.2021.102994.
  5. N. Naz, M. A. Khan, S. A. Alsuhibany, M. Diyan, Z. Tan, M. A. Khan, and J. Ahmad, “Ensemble learning-based IDS for sensors telemetry data in IoT networks,” Mathematical Biosciences and Engineering, vol. 19, no. 10, pp. 10550–10580, 2022, doi: 10.3934/mbe.2022493.
  6. N. Naz, M. A. Khan, M. A. Khan, M. A. Khan, S. U. Jan, S. A. Shah, Arshad, Q. H. Abbasi, and J. Ahmad, “A comparison of ensemble learning for intrusion detection in telemetry data,” in Advances on Intelligent Computing and Data Science, Lecture Notes on Data Engineering and Communications Technologies, vol. 179, Cham: Springer, 2023, pp. 451–462, doi: 10.1007/978-3-031-36258-3_40.
  7. S. Poonkuzhal, M. Shobana, and J. Jeyalakshmi, “A deep transfer learning approach for IoT/IIoTcyber attack detection using telemetry data,” Neural Network World, vol. 33, no. 4, pp. 225–244, 2023, doi: 10.14311/NNW.2023.33.014.
  8. R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann, “Shortcut learning in deep neural networks,” Nature Machine Intelligence, vol. 2, no. 11, pp. 665–673, 2020, doi: 10.1038/s42256-020-00257-z.
  9. S. Kapoor and A. Narayanan, “Leakage and the reproducibility crisis in machine-learning-based science,” Patterns, vol. 4, no. 9, Art. no. 100804, 2023, doi: 10.1016/j.patter.2023.100804.
  10. D. Arp, E. Quiring, F. Pendlebury, A. Warnecke, F. Pierazzi, C. Wressnegger, L. Cavallaro, and K. Rieck, “Pitfalls in machine learning for computer security,” Communications of the ACM, vol. 67, no. 11, pp. 104–112, 2024, doi: 10.1145/3643456.
  11. L. D’hooge, M. Verkerken, B. Volckaert, T. Wauters, and F. De Turck, “Establishing the contaminating effect of metadata feature inclusion in machine-learned network intrusion detection models,” in Detection of Intrusions and Malware, and Vulnerability Assessment (DIMVA 2022), LNCS, vol. 13358, Cham: Springer, 2022, pp. 23–41, doi: 10.1007/978-3-031-09484-2_2.
  12. G. Engelen, V. Rimmer, and W. Joosen, “Troubleshooting an intrusion detection dataset: The CICIDS2017 case study,” in Proc. 2021 IEEE Security and Privacy Workshops (SPW), 2021, pp. 7–12, doi: 10.1109/SPW53761.2021.00009.
  13. M. Lanvin, P.-F. Gimenez, Y. Han, F. Majorczyk, L. Mé, and É. Totel, “Errors in the CICIDS2017 dataset and the significant differences in detection performances it makes,” in Risks and Security of Internet and Systems (CRiSIS 2022), LNCS, vol. 13857, Cham: Springer, 2023, pp. 18–33, doi: 10.1007/978-3-031-31108-6_2.
  14. T. M. Booij, I. Chiscop, E. Meeuwissen, N. Moustafa, and F. T. H. den Hartog, “ToN_IoT: The role of heterogeneity and the need for standardization of features and attack types in IoT network intrusion data sets,” IEEE Internet of Things Journal, vol. 9, no. 1, pp. 485–496, 2022, doi: 10.1109/JIOT.2021.3085194.
  15. S. Haque, F. El-Moussa, N. Komninos, and M. Rajarajan, “A systematic review of data-driven attack detection trends in IoT,” Sensors, vol. 23, no. 16, Art. no. 7191, 2023, doi: 10.3390/s23167191.
  16. A. Heidari and M. A. Jabraeil Jamali, “Internet of Things intrusion detection systems: A comprehensive review and future directions,” Cluster Computing, vol. 26, no. 6, pp. 3753–3780, 2023, doi: 10.1007/s10586-022-03776-z.
  17. E. Gyamfi and A. Jurcut, “Intrusion detection in Internet of Things systems: A review on design approaches leveraging multi-access edge computing, machine learning, and datasets,” Sensors, vol. 22, no. 10, Art. no. 3744, 2022, doi: 10.3390/s22103744.
  18. M. A. Ferrag, O. Friha, D. Hamouda, L. Maglaras, and H. Janicke, “Edge-IIoTset: A new comprehensive realistic cyber security dataset of IoT and IIoT applications for centralized and federated learning,” IEEE Access, vol. 10, pp. 40281–40306, 2022, doi: 10.1109/ACCESS.2022.3165809.
  19. M. Al-Hawawreh, E. Sitnikova, and N. Aboutorab, “X-IIoTID: A connectivity-agnostic and device-agnostic intrusion data set for industrial Internet of Things,” IEEE Internet of Things Journal, vol. 9, no. 5, pp. 3962–3977, 2022, doi: 10.1109/JIOT.2021.3102056.
  20. E. C. P. Neto, S. Dadkhah, R. Ferreira, A. Zohourian, R. Lu, and A. A. Ghorbani, “CICIoT2023: A real-time dataset and benchmark for large-scale attacks in IoT environment,” Sensors, vol. 23, no. 13, Art. no. 5941, 2023, doi: 10.3390/s23135941.
  21. S. Kaufman, S. Rosset, C. Perlich, and O. Stitelman, “Leakage in data mining: Formulation, detection, and avoidance,” ACM Transactions on Knowledge Discovery from Data, vol. 6, no. 4, pp. 1–21, 2012, doi: 10.1145/2382577.2382579.
  22. M. Rosenblatt, L. Tejavibulya, R. Jiang, S. Noble, and D. Scheinost, “Data leakage inflates prediction performance in connectome-based machine learning models,” Nature Communications, vol. 15, no. 1, Art. no. 1829, 2024, doi: 10.1038/s41467-024-46150-w.
  23. S. Whalen, J. Schreiber, W. S. Noble, and K. S. Pollard, “Navigating the pitfalls of applying machine learning in genomics,” Nature Reviews Genetics, vol. 23, no. 3, pp. 169–181, 2022, doi: 10.1038/s41576-021-00434-9.
  24. L. Liu, G. Engelen, T. Lynar, D. Essam, and W. Joosen, “Error prevalence in NIDS datasets: A case study on CIC-IDS-2017 and CSE-CIC-IDS-2018,” in Proc. 2022 IEEE Conference on Communications and Network Security (CNS), 2022, pp. 254–262, doi: 10.1109/CNS56114.2022.9947235.
  25. J. L. Guerra, C. Catania, and E. Veas, “Datasets are not enough: Challenges in labeling network traffic,” Computers & Security, vol. 120, Art. no. 102810, 2022, doi: 10.1016/j.cose.2022.102810.
  26. M. Sarhan, S. Layeghy, and M. Portmann, “Evaluating standard feature sets towards increased generalisability and explainability of ML-based network intrusion detection,” Big Data Research, vol. 30, Art. no. 100359, 2022, doi: 10.1016/j.bdr.2022.100359.
  27. S. Layeghy and M. Portmann, “Explainable cross-domain evaluation of ML-based network intrusion detection systems,” Computers and Electrical Engineering, vol. 108, Art. no. 108692, 2023, doi: 10.1016/j.compeleceng.2023.108692.
  28. G. Apruzzese, P. Laskov, and J. Schneider, “SoK: Pragmatic assessment of machine learning for network intrusion detection,” in Proc. 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P), 2023, pp. 592–614, doi: 10.1109/EuroSP57164.2023.00042.
  29. G. Apruzzese, P. Laskov, E. Montes de Oca, W. Mallouli, L. Brdalo Rapa, A. V. Grammatopoulos, and F. Di Franco, “The role of machine learning in cybersecurity,” Digital Threats: Research and Practice, vol. 4, no. 1, pp. 1–38, 2023, doi: 10.1145/3545574.
  30. D. R. Roberts, V. Bahn, S. Ciuti, M. S. Boyce, J. Elith, G. Guillera-Arroita, et al., “Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure,” Ecography, vol. 40, no. 8, pp. 913–929, 2017, doi: 10.1111/ecog.02881.
  31. H. Hewamalage, K. Ackermann, and C. Bergmeir, “Forecast evaluation for data scientists: Common pitfalls and best practices,” Data Mining and Knowledge Discovery, vol. 37, no. 2, pp. 788–832, 2023, doi: 10.1007/s10618-022-00894-5.
  32. L. Yang and A. Shami, “IoT data analytics in dynamic environments: From an automated machine learning perspective,” Engineering Applications of Artificial Intelligence, vol. 116, Art. no. 105366, 2022, doi: 10.1016/j.engappai.2022.105366.
  33. K. DeMedeiros, A. Hendawi, and M. Alvarez, “A survey of AI-based anomaly detection in IoT and sensor networks,” Sensors, vol. 23, no. 3, Art. no. 1352, 2023, doi: 10.3390/s23031352.
  34. A. Chatterjee and B. S. Ahmed, “IoT anomaly detection methods and applications: A survey,” Internet of Things, vol. 19, Art. no. 100568, 2022, doi: 10.1016/j.iot.2022.100568.
  35. N. Moustafa, “ToN_IoT datasets,” IEEE Dataport, Oct. 16, 2019. [Online]. Available: https://ieee-dataport.org/documents/toniot-datasets, doi: 10.21227/fesz-dm97.
  36. R. Shwartz-Ziv and A. Armon, “Tabular data: Deep learning is not all you need,” Information Fusion, vol. 81, pp. 84–90, 2022, doi: 10.1016/j.inffus.2021.11.011.