Security for Machine Learning-based Software Systems: A Survey of Threats, Practices, and Challenges
Open Access
- 23 February 2024
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Computing Surveys
- Vol. 56 (6) , 1-38
Abstract
The rapid development of Machine Learning (ML) has demonstrated superior performance in many areas, such as computer vision and video and speech recognition. It has now been increasingly leveraged in software systems to automate the core tasks. However, how to securely develop the machine learning-based modern software systems (MLBSS) remains a big challenge, for which the insufficient consideration will largely limit its application in safety-critical domains. One concern is that the present MLBSS development tends to be rushed, and the latent vulnerabilities and privacy issues exposed to external users and attackers will be largely neglected and hard to be identified. Additionally, machine learning-based software systems exhibit different liabilities towards novel vulnerabilities at different development stages from requirement analysis to system maintenance, due to its inherent limitations from the model and data and the external adversary capabilities. The successful generation of such intelligent systems will thus solicit dedicated efforts jointly from different research areas, i.e., software engineering, system security, and machine learning. Most of the recent works regarding the security issues for ML have a strong focus on the data and models, which has brought adversarial attacks into consideration. In this work, we consider that security for machine learning-based software systems may arise from inherent system defects or external adversarial attacks, and the secure development practices should be taken throughout the whole lifecycle. While machine learning has become a new threat domain for existing software engineering practices, there is no such review work covering the topic. Overall, we present a holistic review regarding the security for MLBSS, which covers a systematic understanding from a structure review of three distinct aspects in terms of security threats. Moreover, it provides a thorough state-of-the-practice for MLBSS secure development. Finally, we summarize the literature for system security assurance and motivate the future research directions with open challenges. We anticipate this work provides sufficient discussion and novel insights to incorporate system security engineering for future exploration.Keywords
All Related Versions
Funding Information
- Cyber Security Research Centre Limited
- Australian Government’s Cooperative Research Centres Programme
This publication has 140 references indexed in Scilit:
- Wild patterns: Ten years after the rise of adversarial machine learningPattern Recognition, 2018
- Mobile Agents System SecurityACM Computing Surveys, 2017
- Cloud security engineering: Early stages of SDLCFuture Generation Computer Systems, 2017
- Security knowledge representation artifacts for creating secure IT systemsComputers & Security, 2017
- Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiersInternational Journal of Security and Networks, 2015
- An exploration of technical debtJournal of Systems and Software, 2013
- A systematic review of security requirements engineeringComputer Standards & Interfaces, 2010
- Security Requirements Engineering: A Framework for Representation and AnalysisIEEE Transactions on Software Engineering, 2008
- Software securityIEEE Security & Privacy, 2004
- Integrating Software Lifecycle Process Standards with Security EngineeringComputers & Security, 2002