A Large Language Model Approach to Code and Privacy Policy Alignment
Authors:
- 12 March 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 79-90
Abstract
As mobile technology has advanced, individuals have started relying on their smartphones to conduct more of their everyday tasks. From playing games or streaming media to social networking and banking, apps on a user's device may have access to the most sensitive information on the device. Privacy policies are designed to inform users of such data practices so that they can make reasonable decisions when using the app. However, an app's true behavior may not always align with the statements in a privacy policy. In this work, we divide our study into two components and compare the viability of various large language models (LLMs): methods for extracting and summarizing privacy policy data practices, or information-type extraction and action-verb extraction; and methods for measuring whether the policy acknowledges the interaction with certain information (sensitive data) compared to identified methods within its app's source code. Fine-tuning GPT-3.5 Turbo delivers a higher average F1-score for both action verb extraction (0.50) and information-type extraction (0.84) compared to other LLMs. ChatGPT outperforms other language models in traditional semantic similarity, providing a consistently high performance, including the highest F1-score (0.52) for this task. Our approaches demonstrate that these LLMs are viable in performing such tasks and additionally that pre-trained instruction-based LLMs are capable of identifying the complex relationships between policies and source code.Keywords
Funding Information
- NSF (#1948243,#2318915)
This publication has 48 references indexed in Scilit:
- A Question Answering Tool for Website Privacy Policy Comprehension2023
- ChatGPT: Jack of all trades, master of noneInformation Fusion, 2023
- Large language models for oncological applicationsJournal of Cancer Research and Clinical Oncology, 2023
- The artificially intelligent entrepreneur: ChatGPT, prompt engineering, and entrepreneurial rhetoric creationJournal of Business Venturing Insights, 2023
- Artificial intelligence-based text generators in hepatology: ChatGPT is just the beginningHepatology Communications, 2023
- Privacy explanations – A means to end-user trustJournal of Systems and Software, 2022
- Examining the role of consumer impulsiveness in multiple app usage behavior among mobile shoppersJournal of Business Research, 2022
- What drives m-banking clients to continue using m-banking services?Journal of Business Research, 2022
- Inferring Semantic Mapping Between Policies and Code: The Clue is in the Language2016
- Measuring nominal scale agreement among many raters.Psychological Bulletin, 1971