James Smith
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
James Smith has a total of 18 co-authors. Here are the top co-authors by number of shared publications:
- Bjoern Hartmann
University of California, Berkeley, Berkeley, CA, USA
5 shared publications Bjoern Hartmann
University of California, Berkeley, Berkeley, CA, USA
2 shared publications- Timothy J. Aveni
University of California, Berkeley, Berkeley, California, USA
2 shared publications - Adolfo G. Ramirez-Aristizabal
Accenture, San Francisco, USA
1 shared publication - Anya Agarwal
University of California, Berkeley, Berkeley, California, USA
1 shared publication - Armando Fox
University of California, Berkeley, Berkeley, USA
1 shared publication
Latest Publications
- Open Access
- 2 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-15
Predicting Tool Extension in VR Interactions Using Gaze Features
- 2 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-15
Open AccessPredicting Tool Extension in VR Interactions Using Gaze Features
- Open Access
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-20
Noise Pilot: Enabling Artistic Workflow Composition with Diffusion-Based Image Generation
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-20
Open AccessNoise Pilot: Enabling Artistic Workflow Composition with Diffusion-Based Image Generation
- Open Access
- 13 June 2025
- Published by Association for Computing Machinery (ACM)
- p. 65-71
Supporting Students in Prototyping AI-backed Software with Hosted Prompt Template APIs
- 13 June 2025
- Published by Association for Computing Machinery (ACM)
- p. 65-71
Open AccessSupporting Students in Prototyping AI-backed Software with Hosted Prompt Template APIs
- Open Access
- 11 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-13
What's the Game, then? Opportunities and Challenges for Runtime Behavior Generation
Procedural content generation (PCG), the process of algorithmically creating game components instead of manually, has been a common tool of game development for decades. Recent advances in large language models (LLMs) enable the generation of game behaviors based on player input at runtime. Such code generation brings with it the possibility of entirely new gameplay interactions that may be difficult to integrate with typical game development workflows. We explore these implications through GROMIT, a novel LLM-based runtime behavior generation system for Unity. When triggered by a player action, GROMIT generates a relevant behavior which is compiled without developer intervention and incorporated into the game. We create three demonstration scenarios with GROMIT to investigate how such a technology might be used in game development. In a system evaluation we find that our implementation is able to produce behaviors that result in significant downstream impacts to gameplay. We then conduct an interview study with n=13 game developers using GROMIT as a probe to elicit their current opinion on runtime behavior generation tools, and enumerate the specific themes curtailing the wider use of such tools. We find that the main themes of concern are quality considerations, community expectations, and fit with developer workflows, and that several of the subthemes are unique to runtime behavior generation specifically. We outline a future work agenda to address these concerns, including the need for additional guardrail systems for behavior generation.
- 11 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-13
Open AccessWhat's the Game, then? Opportunities and Challenges for Runtime Behavior Generation
Procedural content generation (PCG), the process of algorithmically creating game components instead of manually, has been a common tool of game development for decades. Recent advances in large language models (LLMs) enable the generation of game behaviors based on player input at runtime. Such code generation brings with it the possibility of entirely new gameplay interactions that may be difficult to integrate with typical game development workflows. We explore these implications through GROMIT, a novel LLM-based runtime behavior generation system for Unity. When triggered by a player action, GROMIT generates a relevant behavior which is compiled without developer intervention and incorporated into the game. We create three demonstration scenarios with GROMIT to investigate how such a technology might be used in game development. In a system evaluation we find that our implementation is able to produce behaviors that result in significant downstream impacts to gameplay. We then conduct an interview study with n=13 game developers using GROMIT as a probe to elicit their current opinion on runtime behavior generation tools, and enumerate the specific themes curtailing the wider use of such tools. We find that the main themes of concern are quality considerations, community expectations, and fit with developer workflows, and that several of the subthemes are unique to runtime behavior generation specifically. We outline a future work agenda to address these concerns, including the need for additional guardrail systems for behavior generation.
- Open Access
- 10 July 2023
- Published by Association for Computing Machinery (ACM)
- p. 230-243
Dual Body Bimanual Coordination in Immersive Environments
A common way to enable immersion in VR is to render a virtual body that mirrors the user’s physical movements. VR allows us to design interaction schemes that go beyond direct avatar embodiments. In particular, there is a growing body of literature investigating the simultaneous control of multiple bodies in VR. We contribute to this literature by investigating the important case where multiple bodies perform a coordinated interaction with each other. Such actions directly question what kind of embodiment users experience. Concretely, we investigate people’s abilities to perform coordinated bimanual selection and handoff tasks between a first-person and third-person body through a user study with 19 participants. Results provide quantitative & qualitative evidence for people’s ability to perform complex coordinated tasks through two bodies. Furthermore we characterize participant performance in different task and interaction configurations, summarize the strategies they employed, and discuss qualities of user’s proprioception.
- 10 July 2023
- Published by Association for Computing Machinery (ACM)
- p. 230-243
Open AccessDual Body Bimanual Coordination in Immersive Environments
A common way to enable immersion in VR is to render a virtual body that mirrors the user’s physical movements. VR allows us to design interaction schemes that go beyond direct avatar embodiments. In particular, there is a growing body of literature investigating the simultaneous control of multiple bodies in VR. We contribute to this literature by investigating the important case where multiple bodies perform a coordinated interaction with each other. Such actions directly question what kind of embodiment users experience. Concretely, we investigate people’s abilities to perform coordinated bimanual selection and handoff tasks between a first-person and third-person body through a user study with 19 participants. Results provide quantitative & qualitative evidence for people’s ability to perform complex coordinated tasks through two bodies. Furthermore we characterize participant performance in different task and interaction configurations, summarize the strategies they employed, and discuss qualities of user’s proprioception.
- Open Access
- 17 October 2019
- Published by Association for Computing Machinery (ACM)
- p. 987-998
LabelAR
Computer vision is applied in an ever expanding range of applications, many of which require custom training data to perform well. We present a novel interface for rapid collection of labeled training images to improve CV-based object detectors. LabelAR leverages the spatial tracking capabilities of an AR-enabled camera, allowing users to place persistent bounding volumes that stay centered on real-world objects. The interface then guides the user to move the camera to cover a wide variety of viewpoints. We eliminate the need for post hoc labeling of images by automatically projecting 2D bounding boxes around objects in the images as they are captured from AR-marked viewpoints. In a user study with 12 participants, LabelAR significantly outperforms existing approaches in terms of the trade-off between detection performance and collection time.
- 17 October 2019
- Published by Association for Computing Machinery (ACM)
- p. 987-998
Open AccessLabelAR
Computer vision is applied in an ever expanding range of applications, many of which require custom training data to perform well. We present a novel interface for rapid collection of labeled training images to improve CV-based object detectors. LabelAR leverages the spatial tracking capabilities of an AR-enabled camera, allowing users to place persistent bounding volumes that stay centered on real-world objects. The interface then guides the user to move the camera to cover a wide variety of viewpoints. We eliminate the need for post hoc labeling of images by automatically projecting 2D bounding boxes around objects in the images as they are captured from AR-marked viewpoints. In a user study with 12 participants, LabelAR significantly outperforms existing approaches in terms of the trade-off between detection performance and collection time.