Aysu Betin-Can
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Aysu Betin-Can has a total of 46 co-authors. Here are the top co-authors by number of shared publications:
- Elif Surer
Graduate School of Informatics, Middle East Technical University Ankara Türkiye
7 shared publications Tevfik Bultan
University of California at Santa Barbara, Santa Barbara
7 shared publications- Mohammed Gh. AL Zamil
Yarmouk University
3 shared publications - Demet Demir
Graduate School of Informatics, Middle East Technical University, Ankara, Türkiye
2 shared publications - Sinan Ariyurek
Graduate School of Informatics, Middle East Technical University, Ankara, Turkey
2 shared publications - Sinan Ariyurek
Graduate School of Informatics Middle East Technical University 06800 Ankara Turkey
2 shared publications
Latest Publications
- Open Access
- 11 September 2024
- Published by Association for Computing Machinery (ACM)
- p. 678-690
Test Selection for Deep Neural Networks using Meta-Models with Uncertainty Metrics
With the use of Deep Learning (DL) in safety-critical domains, the systematic testing of these systems has become a critical issue for human life. Due to the data-driven nature of Deep Neural Networks (DNNs), the effectiveness of tests is closely related to the adequacy of test datasets. Test data need to be labeled, which requires manual human effort and sometimes expert knowledge. DL system testers aim to select the test data that will be most helpful in identifying the weaknesses of the DNN model by using resources efficiently. To help achieve this goal, we propose a test data prioritization approach based on using a meta-model that gets uncertainty metrics as input, which are derived from outputs of other base models. Integrating different uncertainty metrics helps overcome individual limitations of these metrics and be effective in a wider range of scenarios. We train the meta-models with the objective of predicting whether a test input will lead the tested model to make an incorrect prediction or not. We conducted an experimental evaluation with popular image classification datasets and DNN models to evaluate the proposed approach. The results of the experiments demonstrate that our approach effectively prioritizes the test datasets and outperforms existing state-of-the-art test prioritization methods used in comparison. In the experiments, we evaluated the test prioritization approach from a distribution-aware perspective by generating test datasets with and without out-of-distribution data.
- 11 September 2024
- Published by Association for Computing Machinery (ACM)
- p. 678-690
Open AccessTest Selection for Deep Neural Networks using Meta-Models with Uncertainty Metrics
With the use of Deep Learning (DL) in safety-critical domains, the systematic testing of these systems has become a critical issue for human life. Due to the data-driven nature of Deep Neural Networks (DNNs), the effectiveness of tests is closely related to the adequacy of test datasets. Test data need to be labeled, which requires manual human effort and sometimes expert knowledge. DL system testers aim to select the test data that will be most helpful in identifying the weaknesses of the DNN model by using resources efficiently. To help achieve this goal, we propose a test data prioritization approach based on using a meta-model that gets uncertainty metrics as input, which are derived from outputs of other base models. Integrating different uncertainty metrics helps overcome individual limitations of these metrics and be effective in a wider range of scenarios. We train the meta-models with the objective of predicting whether a test input will lead the tested model to make an incorrect prediction or not. We conducted an experimental evaluation with popular image classification datasets and DNN models to evaluate the proposed approach. The results of the experiments demonstrate that our approach effectively prioritizes the test datasets and outperforms existing state-of-the-art test prioritization methods used in comparison. In the experiments, we evaluated the test prioritization approach from a distribution-aware perspective by generating test datasets with and without out-of-distribution data.
- Open Access
- 26 October 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 11, 119481-119505
Distribution Aware Testing Framework for Deep Neural Networks
The increasing use of deep learning (DL) in safety-critical applications highlights the critical need for systematic and effective testing to ensure system reliability and quality. In this context, researchers have conducted various DL testing studies to identify weaknesses in Deep Neural Network (DNN) models, including exploring test coverage, generating challenging test inputs, and test selection. In this study, we propose a generic DNN testing framework that takes into consideration the distribution of test data and prioritizes them based on their potential to cause incorrect predictions by the tested DNN model. We evaluated the proposed framework using the image classification as a use case. We conducted empirical evaluations by implementing each phase with carefully chosen methods. We employed Variational Autoencoders to identify and eliminate out-of-distribution data from the test datasets. Additionally, we prioritize test data that increase uncertainty in the model, as these cases are more likely to reveal potential faults. The elimination of out-of-distribution data enables a more focused analysis to uncover the sources of DNN failures while using prioritized test data reduces the cost of test data labeling. Furthermore, we explored the use of post-hoc explainability methods to identify the cause of incorrect predictions, a process similar to debugging. This study can be a prelude to incorporating explainability methods into the model development process after testing.
- 26 October 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 11, 119481-119505
Open AccessDistribution Aware Testing Framework for Deep Neural Networks
The increasing use of deep learning (DL) in safety-critical applications highlights the critical need for systematic and effective testing to ensure system reliability and quality. In this context, researchers have conducted various DL testing studies to identify weaknesses in Deep Neural Network (DNN) models, including exploring test coverage, generating challenging test inputs, and test selection. In this study, we propose a generic DNN testing framework that takes into consideration the distribution of test data and prioritizes them based on their potential to cause incorrect predictions by the tested DNN model. We evaluated the proposed framework using the image classification as a use case. We conducted empirical evaluations by implementing each phase with carefully chosen methods. We employed Variational Autoencoders to identify and eliminate out-of-distribution data from the test datasets. Additionally, we prioritize test data that increase uncertainty in the model, as these cases are more likely to reveal potential faults. The elimination of out-of-distribution data enables a more focused analysis to uncover the sources of DNN failures while using prioritized test data reduces the cost of test data labeling. Furthermore, we explored the use of post-hoc explainability methods to identify the cause of incorrect predictions, a process similar to debugging. This study can be a prelude to incorporating explainability methods into the model development process after testing.
- 30 August 2022
- Published by Springer Nature in The Journal of Supercomputing
- Vol. 79 (3) , 3112-3156
Blockchain technology applications in the health domain: a multivocal literature review
Blockchain technology has been changing the nature of several businesses, from supply chain management to electronic record management systems and copyright management to healthcare applications. It provides a resilient and secure platform for modifications due to its distributed and shared nature and cryptographic functions. Each new technology, however, comes with its challenges alongside its opportunities. Previously, we performed a systematic literature review (SLR) to explore how blockchain technology potentially benefits health domain applications. The previous SLR included 27 formal literature papers from 2016 to 2020. Noticing that blockchain technology is rapidly growing, we extended the previous SLR with a multivocal literature review (MLR) approach to present the state of the art in this study. We focused on understanding to what degree blockchain could answer the challenges inherited in the health domain and whether blockchain technology may bring new challenges to health applications. The MLR consists of 78 sources of formal literature and 23 sources of gray literature from 2016 to 2021. As a result of this study, we specified 17 health domain challenges that can be categorized into four groups: (i) meeting regulatory requirements and public health surveillance, (ii) ensuring security and privacy, (iii) ensuring interoperability, and (iv) preventing waste of resources. The analysis shows that blockchain makes significant contributions to the solutions of these challenges. However, 10 new pitfalls come with adopting the technology in the health domain: the inability to delete sensitive data once it is added to a chain, limited ability to keep large-scale data in a blockchain, and performance issues. The data we extracted during the MLR is available in a publicly accessible online repository.
- 30 August 2022
- Published by Springer Nature in The Journal of Supercomputing
- Vol. 79 (3) , 3112-3156
Blockchain technology applications in the health domain: a multivocal literature review
Blockchain technology has been changing the nature of several businesses, from supply chain management to electronic record management systems and copyright management to healthcare applications. It provides a resilient and secure platform for modifications due to its distributed and shared nature and cryptographic functions. Each new technology, however, comes with its challenges alongside its opportunities. Previously, we performed a systematic literature review (SLR) to explore how blockchain technology potentially benefits health domain applications. The previous SLR included 27 formal literature papers from 2016 to 2020. Noticing that blockchain technology is rapidly growing, we extended the previous SLR with a multivocal literature review (MLR) approach to present the state of the art in this study. We focused on understanding to what degree blockchain could answer the challenges inherited in the health domain and whether blockchain technology may bring new challenges to health applications. The MLR consists of 78 sources of formal literature and 23 sources of gray literature from 2016 to 2021. As a result of this study, we specified 17 health domain challenges that can be categorized into four groups: (i) meeting regulatory requirements and public health surveillance, (ii) ensuring security and privacy, (iii) ensuring interoperability, and (iv) preventing waste of resources. The analysis shows that blockchain makes significant contributions to the solutions of these challenges. However, 10 new pitfalls come with adopting the technology in the health domain: the inability to delete sensitive data once it is added to a chain, limited ability to keep large-scale data in a blockchain, and performance issues. The data we extracted during the MLR is available in a publicly accessible online repository.
- Open Access
- 21 June 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 10, 66492-66510
Verifying Maze-Like Game Levels With Model Checker SPIN
This study presents a framework that procedurally generates maze-like levels and leverages an automated verification technique called model checking to verify and produce a winning action sequence for that level. By leveraging the counterexample generation feature of the SPIN model checker, one or more solutions to the level-in-test are found, and the solutions are animated using a video game description language, PyVGDL. The framework contains four behavioral templates developed to model the logic of maze-like puzzle games in the modeling language of SPIN. These models automatically are tailored according to the level-in-test. To show the proposed methodology’s effectiveness, we conducted five different experiments. These experiments include performance comparisons in level-solving between the proposed and existing methodologies —A* Search and Monte Carlo Tree Search— and demonstrations of the use of the proposed approach to check a game level with respect to requirements. This study also proposes a pipeline to generate maze-like puzzle levels with two levels of cellular automata.
- 21 June 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 10, 66492-66510
Open AccessVerifying Maze-Like Game Levels With Model Checker SPIN
This study presents a framework that procedurally generates maze-like levels and leverages an automated verification technique called model checking to verify and produce a winning action sequence for that level. By leveraging the counterexample generation feature of the SPIN model checker, one or more solutions to the level-in-test are found, and the solutions are animated using a video game description language, PyVGDL. The framework contains four behavioral templates developed to model the logic of maze-like puzzle games in the modeling language of SPIN. These models automatically are tailored according to the level-in-test. To show the proposed methodology’s effectiveness, we conducted five different experiments. These experiments include performance comparisons in level-solving between the proposed and existing methodologies —A* Search and Monte Carlo Tree Search— and demonstrations of the use of the proposed approach to check a game level with respect to requirements. This study also proposes a pipeline to generate maze-like puzzle levels with two levels of cellular automata.
- 11 April 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Games
- Vol. 15 (3) , 348-359
Playtesting: What is Beyond Personas
We present two approaches to improve automated playtesting. First, we propose developing persona, which allows a persona to progress to different goals. In contrast, the procedural persona is fixed to a single goal. Second, a human playtester knows which paths she has tested before, and during the consequent tests, she may test different paths. However, Reinforcement Learning (RL) agents disregard these previous paths. We propose a novel methodology that we refer to as Alternative Path Finder (APF). We train APF with previous paths and employ APF during the training of a RL agent. APF modulates the reward structure of the environment while preserving the agent's goal. When evaluated, the agent generates a different trajectory that achieves the same goal. We use the General Video Game Artificial Intelligence and VizDoom frameworks to test our proposed methodologies. We use Proximal Policy Optimization RL agent during experiments. First, we compare the playtest data generated by developing and procedural persona. Our experiments show that developing persona provides better insight into the game and how different players would play. Second, we present the alternative paths found using APF and argue why traditional RL agents cannot learn those paths.
- 11 April 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Games
- Vol. 15 (3) , 348-359
Playtesting: What is Beyond Personas
We present two approaches to improve automated playtesting. First, we propose developing persona, which allows a persona to progress to different goals. In contrast, the procedural persona is fixed to a single goal. Second, a human playtester knows which paths she has tested before, and during the consequent tests, she may test different paths. However, Reinforcement Learning (RL) agents disregard these previous paths. We propose a novel methodology that we refer to as Alternative Path Finder (APF). We train APF with previous paths and employ APF during the training of a RL agent. APF modulates the reward structure of the environment while preserving the agent's goal. When evaluated, the agent generates a different trajectory that achieves the same goal. We use the General Video Game Artificial Intelligence and VizDoom frameworks to test our proposed methodologies. We use Proximal Policy Optimization RL agent during experiments. First, we compare the playtest data generated by developing and procedural persona. Our experiments show that developing persona provides better insight into the game and how different players would play. Second, we present the alternative paths found using APF and argue why traditional RL agents cannot learn those paths.
- Open Access
Playtesting: What is Beyond Personas
Playtesting is an essential step in the game design process. Game designers use the feedback from playtests to refine their designs. Game designers may employ procedural personas to automate the playtesting process. In this paper, we present two approaches to improve automated playtesting. First, we propose developing persona, which allows a persona to progress to different goals. In contrast, the procedural persona is fixed to a single goal. Second, a human playtester knows which paths she has tested before, and during the consequent tests, she may test different paths. However, Reinforcement Learning (RL) agents disregard these previous paths. We propose a novel methodology that we refer to as Alternative Path Finder (APF). We train APF with previous paths and employ APF during the training of an RL agent. APF modulates the reward structure of the environment while preserving the agent's goal. When evaluated, the agent generates a different trajectory that achieves the same goal. We use the General Video Game Artificial Intelligence (GVG-AI) and VizDoom frameworks to test our proposed methodologies. We use Proximal Policy Optimization (PPO) RL agent during experiments. First, we compare the playtest data generated by developing and procedural persona. Our experiments show that developing persona provides better insight into the game and how different players would play. Second, we present the alternative paths found using APF and argue why traditional RL agents cannot learn those paths.
Open AccessPlaytesting: What is Beyond Personas
Playtesting is an essential step in the game design process. Game designers use the feedback from playtests to refine their designs. Game designers may employ procedural personas to automate the playtesting process. In this paper, we present two approaches to improve automated playtesting. First, we propose developing persona, which allows a persona to progress to different goals. In contrast, the procedural persona is fixed to a single goal. Second, a human playtester knows which paths she has tested before, and during the consequent tests, she may test different paths. However, Reinforcement Learning (RL) agents disregard these previous paths. We propose a novel methodology that we refer to as Alternative Path Finder (APF). We train APF with previous paths and employ APF during the training of an RL agent. APF modulates the reward structure of the environment while preserving the agent's goal. When evaluated, the agent generates a different trajectory that achieves the same goal. We use the General Video Game Artificial Intelligence (GVG-AI) and VizDoom frameworks to test our proposed methodologies. We use Proximal Policy Optimization (PPO) RL agent during experiments. First, we compare the playtest data generated by developing and procedural persona. Our experiments show that developing persona provides better insight into the game and how different players would play. Second, we present the alternative paths found using APF and argue why traditional RL agents cannot learn those paths.
- Open Access
Enhancing the Monte Carlo Tree Search Algorithm for Video Game Testing
In this paper, we study the effects of several Monte Carlo Tree Search (MCTS) modifications for video game testing. Although MCTS modifications are highly studied in game playing, their impacts on finding bugs are blank. We focused on bug finding in our previous study where we introduced synthetic and human-like test goals and we used these test goals in Sarsa and MCTS agents to find bugs. In this study, we extend the MCTS agent with several modifications for game testing purposes. Furthermore, we present a novel tree reuse strategy. We experiment with these modifications by testing them on three testbed games, four levels each, that contain 45 bugs in total. We use the General Video Game Artificial Intelligence (GVG-AI) framework to create the testbed games and collect 427 human tester trajectories using the GVG-AI framework. We analyze the proposed modifications in three parts: we evaluate their effects on bug finding performances of agents, we measure their success under two different computational budgets, and we assess their effects on human-likeness of the human-like agent. Our results show that MCTS modifications improve the bug finding performance of the agents.
Open AccessEnhancing the Monte Carlo Tree Search Algorithm for Video Game Testing
In this paper, we study the effects of several Monte Carlo Tree Search (MCTS) modifications for video game testing. Although MCTS modifications are highly studied in game playing, their impacts on finding bugs are blank. We focused on bug finding in our previous study where we introduced synthetic and human-like test goals and we used these test goals in Sarsa and MCTS agents to find bugs. In this study, we extend the MCTS agent with several modifications for game testing purposes. Furthermore, we present a novel tree reuse strategy. We experiment with these modifications by testing them on three testbed games, four levels each, that contain 45 bugs in total. We use the General Video Game Artificial Intelligence (GVG-AI) framework to create the testbed games and collect 427 human tester trajectories using the GVG-AI framework. We analyze the proposed modifications in three parts: we evaluate their effects on bug finding performances of agents, we measure their success under two different computational budgets, and we assess their effects on human-likeness of the human-like agent. Our results show that MCTS modifications improve the bug finding performance of the agents.
- 15 October 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Games
- Vol. 13 (1) , 50-67
Automated Video Game Testing Using Synthetic and Humanlike Agents
- 15 October 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Games
- Vol. 13 (1) , 50-67
Automated Video Game Testing Using Synthetic and Humanlike Agents
- Open Access
A Survey of Software Engineering Practices in Turkey (extended version)
Context: Understanding the types of software engineering practices and techniques used in the industry is important. There is a wide spectrum in terms of the types and maturity of software engineering practices conducted in each software team and company. To characterize the type of software engineering practices conducted in software firms, a variety of surveys have been conducted in different countries and regions. Turkey has a vibrant software industry and it is important to characterize and understand the state of software engineering practices in this industry. Objective: Our objective is to characterize and grasp a high-level view on type of software engineering practices in the Turkish software industry. Among the software engineering practices that we have surveyed in this study are the followings: software requirements, design, development, testing, maintenance, configuration management, release planning and support practices. The current survey is the most comprehensive of its type ever conducted in the context of Turkish software industry. Method: To achieve the above objective, we systematically designed an online survey with 46 questions based on our past experience in the Canadian and Turkish contexts and using the Software Engineering Body of Knowledge (SWEBOK). 202 practicing software engineers from the Turkish software industry participated in the survey. We analyze and report in this paper the results of the questions. Whenever possible, we also compare the trends and results of our survey with the results of a similar 2010 survey conducted in the Canadian software industry.
Open AccessA Survey of Software Engineering Practices in Turkey (extended version)
Context: Understanding the types of software engineering practices and techniques used in the industry is important. There is a wide spectrum in terms of the types and maturity of software engineering practices conducted in each software team and company. To characterize the type of software engineering practices conducted in software firms, a variety of surveys have been conducted in different countries and regions. Turkey has a vibrant software industry and it is important to characterize and understand the state of software engineering practices in this industry. Objective: Our objective is to characterize and grasp a high-level view on type of software engineering practices in the Turkish software industry. Among the software engineering practices that we have surveyed in this study are the followings: software requirements, design, development, testing, maintenance, configuration management, release planning and support practices. The current survey is the most comprehensive of its type ever conducted in the context of Turkish software industry. Method: To achieve the above objective, we systematically designed an online survey with 46 questions based on our past experience in the Canadian and Turkish contexts and using the Software Engineering Body of Knowledge (SWEBOK). 202 practicing software engineers from the Turkish software industry participated in the survey. We analyze and report in this paper the results of the questions. Whenever possible, we also compare the trends and results of our survey with the results of a similar 2010 survey conducted in the Canadian software industry.
- 1 May 2014
- Published by Elsevier in Journal of Systems and Software
- Vol. 91, 174-201
Web application testing: A systematic literature review
- 1 May 2014
- Published by Elsevier in Journal of Systems and Software
- Vol. 91, 174-201
Web application testing: A systematic literature review