Kenneth Kotovsky
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Kenneth Kotovsky has a total of 18 co-authors. Here are the top co-authors by number of shared publications:
- Jonathan Cagan
Department of Mechanical Engineering, Carnegie Mellon University, Pittsburgh, USA
18 shared publications - Christopher McComb
Carnegie Mellon University
5 shared publications - Joshua T. Gyory
Carnegie Mellon University Department of Mechanical Engineering Pittsburgh, PA 15213
5 shared publications - Ethan Brownell
Carnegie Mellon University Department of Mechanical Engineering Pittsburgh, PA 15213
2 shared publications - Kosa Goucher-Lambert
Department of Mechanical Engineering, University of California, Berkeley, California, USA
2 shared publications - Ayush Raina
Carnegie Mellon University Department of Mechanical Engineering Pittsburgh, PA 15213
1 shared publication
Latest Publications
- 5 March 2024
- Published by Wiley in Journal of Mechanical Design
- Vol. 146 (8) , 1-34
Human Designers' Dynamic Confidence and Decision-Making When Working With More Than One Artificial Intelligence
As artificial intelligence (AI) systems become increasingly capable of performing design tasks, they are expected to be deployed to assist human designers' decision-making in a greater variety of ways. For complex design problems such as those with multiple objectives, one AI may not always perform its expected accuracy due to the complexity of decision-making, and therefore, multiple AIs may be implemented to provide design suggestions. For such assistance to be productive, human designers must develop appropriate confidence in each AI and in themselves and accept or reject AI inputs accordingly. This work conducts a human subjects experiment to examine the development of a human designer's confidence in each AI and self-confidence throughout decision-making assisted by two AIs and how these confidences influence the decision to accept AI inputs. Major findings demonstrate severe decreases in a human designer's confidence especially when working with one or more low-performing AI teammates and/or receiving negative feedback. Additionally, a human designer's decision to accept AI suggestions depends on their self-confidence and confidence in one of the two AIs. Finally, an additional AI does not increase a human designer's likelihood of conforming to AI suggestions. Therefore, in comparison to a scenario with one AI, the results in this work caution against the implementation of an additional AI to AI-assisted decision-making scenarios. The insights also inform the design and management of human–AI teams to improve the outcome of AI-assisted decision-making.
- 5 March 2024
- Published by Wiley in Journal of Mechanical Design
- Vol. 146 (8) , 1-34
Human Designers' Dynamic Confidence and Decision-Making When Working With More Than One Artificial Intelligence
As artificial intelligence (AI) systems become increasingly capable of performing design tasks, they are expected to be deployed to assist human designers' decision-making in a greater variety of ways. For complex design problems such as those with multiple objectives, one AI may not always perform its expected accuracy due to the complexity of decision-making, and therefore, multiple AIs may be implemented to provide design suggestions. For such assistance to be productive, human designers must develop appropriate confidence in each AI and in themselves and accept or reject AI inputs accordingly. This work conducts a human subjects experiment to examine the development of a human designer's confidence in each AI and self-confidence throughout decision-making assisted by two AIs and how these confidences influence the decision to accept AI inputs. Major findings demonstrate severe decreases in a human designer's confidence especially when working with one or more low-performing AI teammates and/or receiving negative feedback. Additionally, a human designer's decision to accept AI suggestions depends on their self-confidence and confidence in one of the two AIs. Finally, an additional AI does not increase a human designer's likelihood of conforming to AI suggestions. Therefore, in comparison to a scenario with one AI, the results in this work caution against the implementation of an additional AI to AI-assisted decision-making scenarios. The insights also inform the design and management of human–AI teams to improve the outcome of AI-assisted decision-making.
- 19 July 2023
- Published by Wiley in Journal of Mechanical Design
- Vol. 145 (10) , 1-17
A Computational Model of Human Proficiency in Engineering Configuration Design
This work introduces the Proficient Simulated Annealing Design Agent Model (PSADA), a cognitively inspired, agent-based model of engineering configuration design. PSADA models different proficiency agents using move selection heuristics and problem space search strategies, both of which are identified and extracted from prior human subject studies. The model is validated with two design problems. Agents are compared to human designers and show the accurate simulation of the behaviors of the different proficiency designers. These behavior differences lead to significantly different performance levels, matching the human performance levels with just one exception. These validated heterogeneous agents are placed into teams and confirmed previous findings that the most proficient member of a configuration design team has the largest impact (positive or negative) on team performance. The PSADA model is introduced as a scalable platform to further explore configuration design proficiency’s role in design team performance and organizational behavior.
- 19 July 2023
- Published by Wiley in Journal of Mechanical Design
- Vol. 145 (10) , 1-17
A Computational Model of Human Proficiency in Engineering Configuration Design
This work introduces the Proficient Simulated Annealing Design Agent Model (PSADA), a cognitively inspired, agent-based model of engineering configuration design. PSADA models different proficiency agents using move selection heuristics and problem space search strategies, both of which are identified and extracted from prior human subject studies. The model is validated with two design problems. Agents are compared to human designers and show the accurate simulation of the behaviors of the different proficiency designers. These behavior differences lead to significantly different performance levels, matching the human performance levels with just one exception. These validated heterogeneous agents are placed into teams and confirmed previous findings that the most proficient member of a configuration design team has the largest impact (positive or negative) on team performance. The PSADA model is introduced as a scalable platform to further explore configuration design proficiency’s role in design team performance and organizational behavior.
- 31 October 2022
- Published by Wiley in Journal of Mechanical Design
- Vol. 145 (3) , 1-37
The Evolution and Impact of Human Confidence in Artificial Intelligence and in Themselves on AI-Assisted Decision-Making in Design
Decision-making assistance by artificial intelligence (AI) during design is only effective when human designers properly utilize the AI input. However, designers often misjudge the AI’s and/or their own ability, leading to erroneous reliance on AI and therefore bad designs occur. To avoid such outcomes, it is crucial to understand the evolution of designers’ confidence in both their AI teammate(s) and themselves during AI-assisted decision-making. Therefore, this work conducts a cognitive study to explore how to experience various and changing (without notice) AI performance levels and feedback affects these confidences and consequently the decisions to accept or reject AI suggestions. The results first reveal that designers’ confidence in an AI agent changes with poor, but not with good, AI performance in this work. Interestingly, designers’ self-confidence initially remains unaffected by AI accuracy; however, when the accuracy changes, self-confidence decreases regardless of the direction of the change. Moreover, this work finds that designers tend to infer flawed information from feedback, resulting in inappropriate levels of confidence in both the AI and themselves. Confidence in AI and self-confidence are also shown to affect designers’ probability of accepting AI input in opposite directions in this study. Finally, results that are uniquely applicable to design are identified by comparing the findings from this work to those from a similar study conducted with a non-design task. Overall, this work offers valuable insights that may enable the detection of designers’ dynamic confidence and their consequent misuse of AI input in the design.
- 31 October 2022
- Published by Wiley in Journal of Mechanical Design
- Vol. 145 (3) , 1-37
The Evolution and Impact of Human Confidence in Artificial Intelligence and in Themselves on AI-Assisted Decision-Making in Design
Decision-making assistance by artificial intelligence (AI) during design is only effective when human designers properly utilize the AI input. However, designers often misjudge the AI’s and/or their own ability, leading to erroneous reliance on AI and therefore bad designs occur. To avoid such outcomes, it is crucial to understand the evolution of designers’ confidence in both their AI teammate(s) and themselves during AI-assisted decision-making. Therefore, this work conducts a cognitive study to explore how to experience various and changing (without notice) AI performance levels and feedback affects these confidences and consequently the decisions to accept or reject AI suggestions. The results first reveal that designers’ confidence in an AI agent changes with poor, but not with good, AI performance in this work. Interestingly, designers’ self-confidence initially remains unaffected by AI accuracy; however, when the accuracy changes, self-confidence decreases regardless of the direction of the change. Moreover, this work finds that designers tend to infer flawed information from feedback, resulting in inappropriate levels of confidence in both the AI and themselves. Confidence in AI and self-confidence are also shown to affect designers’ probability of accepting AI input in opposite directions in this study. Finally, results that are uniquely applicable to design are identified by comparing the findings from this work to those from a similar study conducted with a non-design task. Overall, this work offers valuable insights that may enable the detection of designers’ dynamic confidence and their consequent misuse of AI input in the design.
- 30 June 2022
- Published by Wiley in Journal of Mechanical Design
- Vol. 144 (10)
Comparing the Impacts on Team Behaviors Between Artificial Intelligence and Human Process Management in Interdisciplinary Design Teams
This brief extends prior research by the authors on studying the impacts of interventions provided by either a human or an artificial intelligence (AI) process manager on team behaviors. Our earlier research found that a created AI process manager matched the capabilities of human process management. Here, these data are studied further to identify the impact of different types of interventions on team behaviors and outcomes. This deeper dive is done via two unique perspectives: comparing teams’ problem-solving processes before and after interventions are provided, and through a regression analysis between intervention counts and performance. Results show overall mixed adherence to the provided interventions, and that this adherence also depends on the intervention type. The most significant impact on the team process arises from the communication frequency interventions. Furthermore, a regression analysis identifies the interventions with the greatest correlation with team performance, indicating a better selection of interventions from the AI process manager. Paired together, the results show the feasibility of automated process management via AI and shed light on the effective implementation of intervention strategies for future development and deployment.
- 30 June 2022
- Published by Wiley in Journal of Mechanical Design
- Vol. 144 (10)
Comparing the Impacts on Team Behaviors Between Artificial Intelligence and Human Process Management in Interdisciplinary Design Teams
This brief extends prior research by the authors on studying the impacts of interventions provided by either a human or an artificial intelligence (AI) process manager on team behaviors. Our earlier research found that a created AI process manager matched the capabilities of human process management. Here, these data are studied further to identify the impact of different types of interventions on team behaviors and outcomes. This deeper dive is done via two unique perspectives: comparing teams’ problem-solving processes before and after interventions are provided, and through a regression analysis between intervention counts and performance. Results show overall mixed adherence to the provided interventions, and that this adherence also depends on the intervention type. The most significant impact on the team process arises from the communication frequency interventions. Furthermore, a regression analysis identifies the interventions with the greatest correlation with team performance, indicating a better selection of interventions from the AI process manager. Paired together, the results show the feasibility of automated process management via AI and shed light on the effective implementation of intervention strategies for future development and deployment.
- Open Access
- 14 October 2021
- Published by Wiley in Journal of Mechanical Design
- Vol. 144 (2)
Human Versus Artificial Intelligence: A Data-Driven Approach to Real-Time Process Management During Complex Engineering Design
Managing the design process of teams has been shown to considerably improve problem-solving behaviors and resulting final outcomes. Automating this activity presents significant opportunities in delivering interventions that dynamically adapt to the state of a team in order to reap the most impact. In this work, an artificial intelligence (AI) agent is created to manage the design process of engineering teams in real time, tracking features of teams’ actions and communications during a complex design and path-planning task in multidisciplinary teams. Teams are also placed under the guidance of human process managers for comparison. Regarding outcomes, teams perform equally as well under both types of management, with trends toward even superior performance from the AI-managed teams. The managers’ intervention strategies and team perceptions of those strategies are also explored, illuminating some intriguing similarities. Both the AI and human process managers focus largely on communication-based interventions, though differences start to emerge in the distribution of interventions across team roles. Furthermore, team members perceive the interventions from both the AI and human manager as equally relevant and helpful, and believe the AI agent to be just as sensitive to the needs of the team. Thus, the overall results show that the AI manager agent introduced in this work is able to match the capabilities of humans, showing potential in automating the management of a complex design process.
- 14 October 2021
- Published by Wiley in Journal of Mechanical Design
- Vol. 144 (2)
Open AccessHuman Versus Artificial Intelligence: A Data-Driven Approach to Real-Time Process Management During Complex Engineering Design
Managing the design process of teams has been shown to considerably improve problem-solving behaviors and resulting final outcomes. Automating this activity presents significant opportunities in delivering interventions that dynamically adapt to the state of a team in order to reap the most impact. In this work, an artificial intelligence (AI) agent is created to manage the design process of engineering teams in real time, tracking features of teams’ actions and communications during a complex design and path-planning task in multidisciplinary teams. Teams are also placed under the guidance of human process managers for comparison. Regarding outcomes, teams perform equally as well under both types of management, with trends toward even superior performance from the AI-managed teams. The managers’ intervention strategies and team perceptions of those strategies are also explored, illuminating some intriguing similarities. Both the AI and human process managers focus largely on communication-based interventions, though differences start to emerge in the distribution of interventions across team roles. Furthermore, team members perceive the interventions from both the AI and human manager as equally relevant and helpful, and believe the AI agent to be just as sensitive to the needs of the team. Thus, the overall results show that the AI manager agent introduced in this work is able to match the capabilities of humans, showing potential in automating the management of a complex design process.
- Open Access
- 16 July 2021
- Published by ASME International
A Topic Modeling Approach to Study the Impact of Manager Interventions on Design Team Cognition
This work utilizes a topic modeling approach to analyze design team discourse during problem-solving, in order to computationally study design cognition under design process management. The particular experimental design, from previous work by the authors, places one of the design team conditions under the guidance of a human process manager. In that work, teams under this guidance
- 16 July 2021
- Published by ASME International
Open AccessA Topic Modeling Approach to Study the Impact of Manager Interventions on Design Team Cognition
This work utilizes a topic modeling approach to analyze design team discourse during problem-solving, in order to computationally study design cognition under design process management. The particular experimental design, from previous work by the authors, places one of the design team conditions under the guidance of a human process manager. In that work, teams under this guidance
- Open Access
- 16 July 2021
- Published by ASME International
Only As Strong As the Strongest Link: The Impact of Individual Team Member\x0bProficiency in Configuration Design
Prior research has demonstrated how the average characteristics of a team impact team performance. Individual characteristics of team members and individual team member behavior have been largely ignored, especially in the context of engineering design. In this work, a behavioral study was conducted to uncover whether the most or least proficient member of a configuration design
- 16 July 2021
- Published by ASME International
Open AccessOnly As Strong As the Strongest Link: The Impact of Individual Team Member\x0bProficiency in Configuration Design
Prior research has demonstrated how the average characteristics of a team impact team performance. Individual characteristics of team members and individual team member behavior have been largely ignored, especially in the context of engineering design. In this work, a behavioral study was conducted to uncover whether the most or least proficient member of a configuration design
- Open Access
- 21 May 2021
- Published by Wiley in Journal of Mechanical Design
- Vol. 143 (11)
The Influence of Process Management: Uncovering the Impact of Real-Time Managerial Interventions via a Topic Modeling Approach
Computationally studying team discourse can provide valuable, real-time insights into the state of design teams and design cognition during problem-solving. The particular experimental design, adopted from previous work by the authors, places one of the design team conditions under the guidance of a human process manager. In that work, teams under this process management outperformed the unmanaged teams in terms of their design performance. This opens the opportunity to not only model design discourse during problem-solving, but more critically, to explore process manager interventions and their impact on design cognition. Utilizing this experimental framework, a topic model is trained on the discourse of human designers of both managed and unmanaged teams collaboratively solving a conceptual engineering design task. Results show that the two team conditions significantly differ in a number of the extracted topics and, in particular, those topics that most pertain to the manager interventions. A dynamic look during the design process reveals that the largest differences between the managed and unmanaged teams occur during the latter half of problem-solving. Furthermore, a before and after analysis of the topic-motivated interventions reveals that the process manager interventions significantly shift the topic mixture of the team members’ discourse immediately after intervening. Taken together, these results from this work not only corroborate the effect of the process manager interventions on design team discourse and cognition but provide promise for the computational detection and facilitation of design interventions based on real-time, discourse data.
- 21 May 2021
- Published by Wiley in Journal of Mechanical Design
- Vol. 143 (11)
Open AccessThe Influence of Process Management: Uncovering the Impact of Real-Time Managerial Interventions via a Topic Modeling Approach
Computationally studying team discourse can provide valuable, real-time insights into the state of design teams and design cognition during problem-solving. The particular experimental design, adopted from previous work by the authors, places one of the design team conditions under the guidance of a human process manager. In that work, teams under this process management outperformed the unmanaged teams in terms of their design performance. This opens the opportunity to not only model design discourse during problem-solving, but more critically, to explore process manager interventions and their impact on design cognition. Utilizing this experimental framework, a topic model is trained on the discourse of human designers of both managed and unmanaged teams collaboratively solving a conceptual engineering design task. Results show that the two team conditions significantly differ in a number of the extracted topics and, in particular, those topics that most pertain to the manager interventions. A dynamic look during the design process reveals that the largest differences between the managed and unmanaged teams occur during the latter half of problem-solving. Furthermore, a before and after analysis of the topic-motivated interventions reveals that the process manager interventions significantly shift the topic mixture of the team members’ discourse immediately after intervening. Taken together, these results from this work not only corroborate the effect of the process manager interventions on design team discourse and cognition but provide promise for the computational detection and facilitation of design interventions based on real-time, discourse data.
- 11 February 2021
- Published by Wiley in Journal of Mechanical Design
- Vol. 143 (8) , 1-16
Only as Strong as the Strongest Link: The Relative Contribution of Individual Team Member Proficiency in Configuration Design
Prior research has demonstrated how the average characteristics of a team impact team performance. The relative contribution of team members has been largely ignored, especially in the context of engineering design. In this work, a behavioral study was conducted with 78 participants to uncover whether the most or least proficient member of a configuration design team had a larger impact on overall performance. Proficiency is an individual's ability to deal with a specific range of problem. It was found that a configuration design team is most dependent on the proficiency of its most proficient member. The most proficient member had a significant positive effect on how quickly the team reached performance thresholds and the other members of the team were not found to have the same positive impact throughout the design study. Behavioral heuristics were found using hidden Markov modeling to capture the differences in behavior and design strategy between different proficiency members. Results show that high proficiency and low proficiency team members exhibit different behavior, with the most proficient member's behavior leading to topologically simpler designs and other members adopting their designs, leading to the most proficient member driving the team design and thus the team performance. These results underscore the value of the relative contribution model in constructing engineering teams by demonstrating that different team members had unequal effects on team performance. It is shown that enhancing the most proficient member of a team is more likely to contribute to increased team performance than enhancing the least proficient member.
- 11 February 2021
- Published by Wiley in Journal of Mechanical Design
- Vol. 143 (8) , 1-16
Only as Strong as the Strongest Link: The Relative Contribution of Individual Team Member Proficiency in Configuration Design
Prior research has demonstrated how the average characteristics of a team impact team performance. The relative contribution of team members has been largely ignored, especially in the context of engineering design. In this work, a behavioral study was conducted with 78 participants to uncover whether the most or least proficient member of a configuration design team had a larger impact on overall performance. Proficiency is an individual's ability to deal with a specific range of problem. It was found that a configuration design team is most dependent on the proficiency of its most proficient member. The most proficient member had a significant positive effect on how quickly the team reached performance thresholds and the other members of the team were not found to have the same positive impact throughout the design study. Behavioral heuristics were found using hidden Markov modeling to capture the differences in behavior and design strategy between different proficiency members. Results show that high proficiency and low proficiency team members exhibit different behavior, with the most proficient member's behavior leading to topologically simpler designs and other members adopting their designs, leading to the most proficient member driving the team design and thus the team performance. These results underscore the value of the relative contribution model in constructing engineering teams by demonstrating that different team members had unequal effects on team performance. It is shown that enhancing the most proficient member of a team is more likely to contribute to increased team performance than enhancing the least proficient member.
- Open Access
- 9 March 2020
- Published by Wiley in Journal of Mechanical Design
- Vol. 142 (9)
Adaptive Inspirational Design Stimuli: Using Design Output to Computationally Search for Stimuli That Impact Concept Generation
Design activity can be supported using inspirational stimuli (e.g., analogies, patents) by helping designers overcome impasses or in generating solutions with more positive characteristics during ideation. Design researchers typically generate inspirational stimuli a priori in order to investigate their impact. However, for a chosen stimulus to possess maximal utility, it should automatically reflect the current and ongoing progress of the designer. In this work, designers receive computationally selected inspirational stimuli midway through an ideation session in response to the contents of their current solution. Sourced from a broad database of related example solutions, the semantic similarity between the content of the current design and concepts within the database determines which potential stimulus is received. Designers receive a particular stimulus based on three experimental conditions: a semantically near stimulus, a semantically far stimulus, or no stimulus (control). Results indicate that adaptive inspirational stimuli can be determined using latent semantic analysis (LSA) and that semantic similarity measures are a promising approach for real-time monitoring of the design process. The ability to achieve differentiable near versus far stimuli was validated using both semantic cosine similarity values and participant self-response ratings. As a further contribution, this work also explores the impact of different types of adaptive inspirational stimuli on design outcomes using a newly introduced “design innovation” measure. The design innovation measure mathematically captures the overall goodness of a design concept by uniquely combining expert ratings across easier to evaluate subdimensions of feasibility, usefulness, and novelty. While results demonstrate that near inspirational stimuli increase the feasibility of design solutions, they also show the significant impact of the overall inspirational stimulus innovativeness on final design outcomes. In fact, participants are more likely to generate innovative final design solutions when given innovative inspirational stimuli, regardless of their experimental condition.
- 9 March 2020
- Published by Wiley in Journal of Mechanical Design
- Vol. 142 (9)
Open AccessAdaptive Inspirational Design Stimuli: Using Design Output to Computationally Search for Stimuli That Impact Concept Generation
Design activity can be supported using inspirational stimuli (e.g., analogies, patents) by helping designers overcome impasses or in generating solutions with more positive characteristics during ideation. Design researchers typically generate inspirational stimuli a priori in order to investigate their impact. However, for a chosen stimulus to possess maximal utility, it should automatically reflect the current and ongoing progress of the designer. In this work, designers receive computationally selected inspirational stimuli midway through an ideation session in response to the contents of their current solution. Sourced from a broad database of related example solutions, the semantic similarity between the content of the current design and concepts within the database determines which potential stimulus is received. Designers receive a particular stimulus based on three experimental conditions: a semantically near stimulus, a semantically far stimulus, or no stimulus (control). Results indicate that adaptive inspirational stimuli can be determined using latent semantic analysis (LSA) and that semantic similarity measures are a promising approach for real-time monitoring of the design process. The ability to achieve differentiable near versus far stimuli was validated using both semantic cosine similarity values and participant self-response ratings. As a further contribution, this work also explores the impact of different types of adaptive inspirational stimuli on design outcomes using a newly introduced “design innovation” measure. The design innovation measure mathematically captures the overall goodness of a design concept by uniquely combining expert ratings across easier to evaluate subdimensions of feasibility, usefulness, and novelty. While results demonstrate that near inspirational stimuli increase the feasibility of design solutions, they also show the significant impact of the overall inspirational stimulus innovativeness on final design outcomes. In fact, participants are more likely to generate innovative final design solutions when given innovative inspirational stimuli, regardless of their experimental condition.