Jingbo Zhao

Reliable estimation of torso direction is essential for virtual locomotion as it controls the locomotion direction in virtual environments. Existing approaches rely on external sensors to track torso direction or use the poses and/or movements from the head, hands, or feet to control the direction of travel. In addition, deep learning methods based on a head‐mounted display (HMD) and its controllers were proposed for torso direction estimation but are unable to produce continuous estimates or that the estimation errors were not systematically evaluated. On the other hand, 3D full‐body human motion generation methods based on an HMD and its controllers are capable of estimating torso direction but this capability was largely ignored for its use in virtual steering. We proposed a new method called TCNPoser that can perform torso direction estimation and generate 3D full‐body human motion. Through offline evaluation and a user study, we show that the proposed method achieves the state‐of‐the‐art (SOTA) performance for real‐time torso direction estimation.

Jujube defects typically occur during the growth, harvesting, packaging and transportation of jujube fruits. Reliable and rapid identification of jujube fruits defects and control measures are crucial for defect identification. However, traditional jujube fruits defects identification mostly relies on expert experience. It requires lots of labor and is subjective. In our study, we improved the ECA attention module and designed the Efficient Channel Attention2 (ECAA) attention module. Then we added ECAA to the MobileNet V2 and ConvNeXt base modules. Finally, for the output of each sub-model, we used weighted sum to achieve sub-models fusion to construct the ECAA MobileNet ConvNeXt(EMCNet) model. We also established a jujube fruits defects identification dataset that contained 4203 images of healthy and defected jujube fruits and their category labels. For EMC-Net model, we tested the performance of EMC-Net on the jujube fruits defects identification dataset. Experimental results show that the proposed model achieve an accuracy of 98.2% on the jujube fruits dataset. These results suggest that the proposed EMC-Net network can effectively identify jujube fruits defects.

Scrap steel grading is a vital stage of steel production. Thickness is an important indicator to determine the grade of scrap steel. Existing methods based on 2D vision cannot accurately estimate the thickness of scrap steel due to the lack of 3D information. Consequently, estimation results from 2D methods cannot satisfy the requirements of the steel industry. To solve this problem, we propose a 3D vision-based scrap steel grading system. We introduce a novel thickness feature descriptor that can perform edge detection on organized point clouds. It can detect the pairs of points that satisfy the thickness feature from edge points. Once the detection of all thickness features is completed, invalid thickness features are filtered and correct thickness features are merged based on the contextual information. Finally, a thickness feature histogram (TFH) is calculated and TFH is used for grading scrap steel. Our proposed method is integrated into a scrap steel grading system, which has been used in an actual production environment. Comprehensive experiments have been conducted to validate the proposed method. Moreover, A specialized scrap steel image dataset was created to train deep learning-based methods in comparison experiment. Experimental results show that our method can extract thickness features and estimate the distribution of thickness of scrap steel effectively. The grading system has been successfully used in actual task scenarios and satisfies the industrial grading requirements. The scrap steel dataset is publicly available at https://github.com/zichengzichengzi/Scrap Steel Image Dataset.

Body actions and head gestures are natural interfaces for interaction in virtual environments. Existing methods for in-place body action recognition often require hardware more than a head-mounted display (HMD), making body action interfaces difficult to be introduced to ordinary virtual reality (VR) users as they usually only possess an HMD. In addition, there lacks a unified solution to recognize in-place body actions and head gestures. This potentially hinders the exploration of the use of in-place body actions and head gestures for novel interaction experiences in virtual environments. We present a unified two-stream 1-D convolutional neural network (CNN) for recognition of body actions when a user performs walking-in-place (WIP) and for recognition of head gestures when a user stands still wearing only an HMD. Compared to previous approaches, our method does not require specialized hardware and/or additional tracking devices other than an HMD and can recognize a significantly larger number of body actions and head gestures than other existing methods. In total, ten in-place body actions and eight head gestures can be recognized with the proposed method, which makes this method a readily available body action interface (head gestures included) for interaction with virtual environments. We demonstrate one utility of the interface through a virtual locomotion task. Results show that the present body action interface is reliable in detecting body actions for the VR locomotion task but is physically demanding compared to a touch controller interface. The present body action interface is promising for new VR experiences and applications, especially for VR fitness applications where workouts are intended.

Generating forward-backward self-representation leg animation in virtual environments for walking-in-place (WIP) techniques is an underexplored research topic. A challenging aspect of the problem is to find an appropriate mapping from tracked vertical foot motion to natural cyclical movements of real walking. In this work, we present a kinematic approach based on animation rigging to generating real-time leg animation. Our method works by tracking vertical in-place foot movements of a user with a Kinect v2 sensor and mapping tracked foot height to inverse kinematics (IK) targets. These IK targets were aligned with an avatar's feet to guide the virtual feet to perform cyclic walking motions. We conducted a user study to evaluate our approach. Results showed that the proposed method produced compelling forward-backward leg animation during walking. We show that the proposed technique can be easily integrated into existing WIP techniques.

Generating forward-backward self-representation leg animation in virtual environments for walking-in-place (WIP) techniques is an underexplored research topic. A challenging aspect of the problem is to find an appropriate mapping from tracked vertical foot motion to natural cyclical movements of real walking. In this work, we present a kinematic approach based on animation rigging to generating real-time leg animation. Our method works by tracking vertical in-place foot movements of a user with a Kinect v2 sensor and mapping tracked foot height to inverse kinematics (IK) targets. These IK targets were aligned with an avatar's feet to guide the virtual feet to perform cyclic walking motions. We conducted a user study to evaluate our approach. Results showed that the proposed method produced compelling forward-backward leg animation during walking. We show that the proposed technique can be easily integrated into existing WIP techniques.