Zhenjie Zhao

Generating reliable grasp configurations in cluttered scenes is an important guarantee for robots to successfully grasp objects. Existing learning-based methods encode the voxelized scene into a latent space to predict graspable points, which overlook both the reliability of grasp selection and the explicit geometric constraint imposed by the object’s surface occupancy. In this article, we propose a 3-D sparse grasp network (3-D SPGNet), which maps voxel features of object surfaces to grasp candidates to enforce explicit constraints, while leveraging the dense truncated signed distance function (TSDF) reconstruction to enhance grasp reliability through implicit synergy. Specifically, we first use a 3-D convolutional neural network to extract multiscale features at three different resolutions from the voxelized input. Then, we apply the proposed 3-D surface constraint block to aggregate features and perform explicit geometric mapping on the encoded nonempty voxels. Finally, we reconstruct the scene from sparse to dense and generate the grasp configuration accordingly. Moreover, we propose a novel loss function for estimating reliable grasp scores and poses. Simulation experiments demonstrate that our method improves the grasp success rates and declutter rates by approximately 9% compared to state-of-the-art baselines. In addition, we deploy the model on a real robot arm, and the real-world experimental results show that our method achieves over a 10% improvement in performance, which further verifies the effectiveness of the proposed method.

Developing social skills like elevator pitch requires being situated within authentic activities and contexts, which is difficult to achieve on a daily basis. In this work, we explore whether an artificial agent with embodied feedback in virtual reality (VR) can foster a situated learning experience. Previous works on computer-mediated feedback have shown that VR can foster oral presentation competence for preuniversity and junior undergraduates through delivering feedback. However, it is unclear how well the learning experiences are and how well students perceive an artificial coach in VR, especially for senior undergraduates and postgraduates seeking job and research opportunities. To inform the design of such a system, we conducted interviews with experts and observed real elevator pitches. We then designed a proof-of-concept VR coaching system with three embodied feedback strategies: immediate, after-action, and the combination of both. Through a between-subject experiment with 40 participants, we studied learners’ perceptions under the embodied feedback. We found that receiving embodied feedback can create a stronger sense of cognitive apprenticeship, i.e., coaching and helping from experts, and help improve the perception of the artificial agent and the effect of learning. We further investigated the pros and cons of different strategies and discussed room for improvement.

An immersive experience brought about by virtual reality can potentially enhance the appreciation of classical Chinese poetry, which is difficult to describe clearly in everyday language or ordinary media. However, making 3-dimensional illustrations for a 360-degree display in virtual reality is usually a labor-intensive and time-consuming procedure and hard to master for non-professional media creators, such as teachers. Motivated by the homology theory of classical Chinese poetry and painting, we propose an image-based approach of building 2.5-dimensional immersive stories to visualize classical Chinese poetry. Specifically, using Chinese shadow play as a metaphor, we have designed and implemented ShadowPlay2.5D , a sketch-based authoring tool to help novices create 360-degree videos of classical Chinese poetry easily. To ensure coverage of the diverse themes in Chinese poetry and preserve the sense of culture, we build a Chinese ink-painting style image repository of essential poetic elements identified via crowdsourcing. To facilitate construction of 2.5-dimensional scenes, we design features that support puppet-like animation, instancing, and camera organization in a 3-dimensional environment. Through two user studies, we show that ShadowPlay2.5D can help novices make a short 360-degree video in about 10--15 minutes, and the 2.5D stylized illustrations created can bring about a better immersive experience for poetry appreciation.

We introduce a novel cartoon-style hybrid emotion embodiment model for live storytelling in virtual reality (VR). It contains an avatar with six basic emotions and an auxiliary multimodal display to enhance the expression of emotions. We further design and implement a system to teleoperate the embodiment model in VR for live storytelling. Specifically, 1) we design a novel visual programming tool that allows users to customize emotional effects based on the emotion embodiment model; 2) we design a novel face tracking module to map presenters' emotional states to the avatar in VR. Our web-based implementation makes the application easy to use. This is an accompanying paper extracted from [1] for the demo session in IEEE AIVR 2019.