Juntao Ye
Research Subjects
Juntao Ye has a total of 31 co-authors. Here are the top co-authors by number of shared publications:
- Dong-Ming Yan
Chinese Academy of Sciences
2 shared publications - Xiaopeng Zhang
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China
2 shared publications - Caigui Jiang
Max Planck Institute for Informatics Germany
1 shared publication - Chengchen Lyu
Institute of Software, Chinese Academy of Sciences, Beijing, China
1 shared publication - Frank Ding
Renderring Dept., Chase Laser Animation Studio, Beijing, Beijing China
1 shared publication - Gang Xiong
State Key Laboratory of Multimodal Artificial Intelligence Systems, Beijing Engineering Research Center of Intelligent Systems and Technology, Institute of Automation, Chinese Academy of Sciences, Beijing, China
1 shared publication
Latest Publications
- Open Access
- 4 June 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-7
CSA-CNN: A Contrastive Self-Attention Neural Network for Pupil Segmentation in Eye Gaze Tracking
This paper presents a novel Contrastive Self-Attention Convolutional Neural Network (CSA-CNN) model with enhanced Difficulty Aware (DA) loss function to improve the segmentation of pupils in eye images. The incorporation of transformer-style self-attention and Difficulty-Aware loss in a UNET-style architecture allows for robust feature representation and promotes shape alignment. The novel model was trained on two public databases (LPW and RIT-Eyes) and evaluated on two other public datasets (ExCuSe and ElSe). When compared with seven state-of-the-art pupil center detection methods, the CSA-CNN showed improvement of over 6% in pupil center detection accuracy (detection within 5 pixels of the labeled center) and more than 9% in Intersection Over Union (IOU) accuracy, compared to the best performer among the other seven methods. Furthermore, when the CSA-CNN model was integrated into a glint-based eye tracking system that uses learning based methods to detect pupil-center, we saw a 25% improvement in gaze accuracy.
- 4 June 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-7
Open AccessCSA-CNN: A Contrastive Self-Attention Neural Network for Pupil Segmentation in Eye Gaze Tracking
This paper presents a novel Contrastive Self-Attention Convolutional Neural Network (CSA-CNN) model with enhanced Difficulty Aware (DA) loss function to improve the segmentation of pupils in eye images. The incorporation of transformer-style self-attention and Difficulty-Aware loss in a UNET-style architecture allows for robust feature representation and promotes shape alignment. The novel model was trained on two public databases (LPW and RIT-Eyes) and evaluated on two other public datasets (ExCuSe and ElSe). When compared with seven state-of-the-art pupil center detection methods, the CSA-CNN showed improvement of over 6% in pupil center detection accuracy (detection within 5 pixels of the labeled center) and more than 9% in Intersection Over Union (IOU) accuracy, compared to the best performer among the other seven methods. Furthermore, when the CSA-CNN model was integrated into a glint-based eye tracking system that uses learning based methods to detect pupil-center, we saw a 25% improvement in gaze accuracy.
- 10 August 2023
- Published by Wiley in Computer Graphics Forum
- Vol. 42 (5)
Deep Deformation Detail Synthesis for Thin Shell Models
In physics‐based cloth animation, rich folds and detailed wrinkles are achieved at the cost of expensive computational resources and huge labor tuning. Data‐driven techniques make efforts to reduce the computation significantly by utilizing a preprocessed database. One type of methods relies on human poses to synthesize fitted garments, but these methods cannot be applied to general cloth animations. Another type of methods adds details to the coarse meshes obtained through simulation, which does not have such restrictions. However, existing works usually utilize coordinate‐based representations which cannot cope with large‐scale deformation, and requires dense vertex correspondences between coarse and fine meshes. Moreover, as such methods only add details, they require coarse meshes to be sufficiently close to fine meshes, which can be either impossible, or require unrealistic constraints to be applied when generating fine meshes. To address these challenges, we develop a temporally and spatially as‐consistent‐as‐possible deformation representation (named TS‐ACAP) and design a DeformTransformer network to learn the mapping from low‐resolution meshes to ones with fine details. This TS‐ACAP representation is designed to ensure both spatial and temporal consistency for sequential large‐scale deformations from cloth animations. With this TS‐ACAP representation, our DeformTransformer network first utilizes two mesh‐based encoders to extract the coarse and fine features using shared convolutional kernels, respectively. To transduct the coarse features to the fine ones, we leverage the spatial and temporal Transformer network that consists of vertex‐level and frame‐level attention mechanisms to ensure detail enhancement and temporal coherence of the prediction. Experimental results show that our method is able to produce reliable and realistic animations in various datasets at high frame rates with superior detail synthesis abilities compared to existing methods.
- 10 August 2023
- Published by Wiley in Computer Graphics Forum
- Vol. 42 (5)
Deep Deformation Detail Synthesis for Thin Shell Models
In physics‐based cloth animation, rich folds and detailed wrinkles are achieved at the cost of expensive computational resources and huge labor tuning. Data‐driven techniques make efforts to reduce the computation significantly by utilizing a preprocessed database. One type of methods relies on human poses to synthesize fitted garments, but these methods cannot be applied to general cloth animations. Another type of methods adds details to the coarse meshes obtained through simulation, which does not have such restrictions. However, existing works usually utilize coordinate‐based representations which cannot cope with large‐scale deformation, and requires dense vertex correspondences between coarse and fine meshes. Moreover, as such methods only add details, they require coarse meshes to be sufficiently close to fine meshes, which can be either impossible, or require unrealistic constraints to be applied when generating fine meshes. To address these challenges, we develop a temporally and spatially as‐consistent‐as‐possible deformation representation (named TS‐ACAP) and design a DeformTransformer network to learn the mapping from low‐resolution meshes to ones with fine details. This TS‐ACAP representation is designed to ensure both spatial and temporal consistency for sequential large‐scale deformations from cloth animations. With this TS‐ACAP representation, our DeformTransformer network first utilizes two mesh‐based encoders to extract the coarse and fine features using shared convolutional kernels, respectively. To transduct the coarse features to the fine ones, we leverage the spatial and temporal Transformer network that consists of vertex‐level and frame‐level attention mechanisms to ensure detail enhancement and temporal coherence of the prediction. Experimental results show that our method is able to produce reliable and realistic animations in various datasets at high frame rates with superior detail synthesis abilities compared to existing methods.
- 7 May 2023
- Published by Taylor & Francis in International Journal of Human–Computer Interaction
- Vol. 40 (15) , 3952-3971
Optical Character Recognition (OCR)-Based and Gaussian Mixture Modeling-OCR-Based Slide-Level “With-Me-Ness”: Automated Measurement and Feedback of Learners’ Attention State during Video Lectures
As video lectures are gaining more popularity, determining their effectiveness and obtaining valuable feedback have become necessary. To measure the learners’ attention state during video lectures, we specified the conceptual “with-me-ness” (WMN) as slide-level WMN (SL-WMN). The content domain on each slide was automatically extracted via an optical character recognition (OCR)-based method, while the eye gazing behaviors were analyzed through a Gaussian mixture modeling (GMM) fixation clustering method. Both domain-specific WMN and behavior-enriched WMN were then computed via OCR- and GMM-OCR-based methods to measure the learners’ attention levels. We conducted an experiment to collect in-lecture eye-tracking data, video recordings, and post-lecture test scores from 50 Grade 8 students. The results demonstrated that both OCR- and GMM-OCR-based SL-WMNs are reliable and compatible automatic measurements of learners’ attention states during video lectures. A survey from participating learners and lecturers also revealed highly favorable feedback for the developed SL-WMNs.
- 7 May 2023
- Published by Taylor & Francis in International Journal of Human–Computer Interaction
- Vol. 40 (15) , 3952-3971
Optical Character Recognition (OCR)-Based and Gaussian Mixture Modeling-OCR-Based Slide-Level “With-Me-Ness”: Automated Measurement and Feedback of Learners’ Attention State during Video Lectures
As video lectures are gaining more popularity, determining their effectiveness and obtaining valuable feedback have become necessary. To measure the learners’ attention state during video lectures, we specified the conceptual “with-me-ness” (WMN) as slide-level WMN (SL-WMN). The content domain on each slide was automatically extracted via an optical character recognition (OCR)-based method, while the eye gazing behaviors were analyzed through a Gaussian mixture modeling (GMM) fixation clustering method. Both domain-specific WMN and behavior-enriched WMN were then computed via OCR- and GMM-OCR-based methods to measure the learners’ attention levels. We conducted an experiment to collect in-lecture eye-tracking data, video recordings, and post-lecture test scores from 50 Grade 8 students. The results demonstrated that both OCR- and GMM-OCR-based SL-WMNs are reliable and compatible automatic measurements of learners’ attention states during video lectures. A survey from participating learners and lecturers also revealed highly favorable feedback for the developed SL-WMNs.
- 7 August 2018
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 25 (10) , 2873-2885
A Semi-Explicit Surface Tracking Mechanism for Multi-Phase Immiscible Liquids
We introduce a new method to efficiently track complex interfaces among multi-phase immiscible fluids. Unlike existing techniques, we use a mesh-based representation for global liquid surfaces, while selectively modeling some local surficial regions with regional level sets (RLS) to handle complex geometry that is difficult to resolve with explicit topology operations. Such a semi-explicit surface mechanism can preserve volume, fine features and foam-like thin films under a relatively low computational expenditure. Our method processes the surface evolution by sampling the fluid domain onto a spectrally refined grid (SRG) and performs efficient grid scanning, generalized interpolations and topology operations on the basis of this grid structure. For the RLS surface part, we propose an accurate advection scheme targeted at SRG. For the explicit mesh part, we develop a fast grid scanning technique to voxelize the meshes and introduce novel strategies to detect grid regions that contain inconsistent mesh components. A robust algorithm is proposed to construct consistent local meshes to resolve mesh penetrations, and handle the coupling between explicit mesh and RLS surficial regions. We also provide further improvement on handling complicated topological variations, as well as strategies for remeshing mesh/RLS interconversions.
- 7 August 2018
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 25 (10) , 2873-2885
A Semi-Explicit Surface Tracking Mechanism for Multi-Phase Immiscible Liquids
We introduce a new method to efficiently track complex interfaces among multi-phase immiscible fluids. Unlike existing techniques, we use a mesh-based representation for global liquid surfaces, while selectively modeling some local surficial regions with regional level sets (RLS) to handle complex geometry that is difficult to resolve with explicit topology operations. Such a semi-explicit surface mechanism can preserve volume, fine features and foam-like thin films under a relatively low computational expenditure. Our method processes the surface evolution by sampling the fluid domain onto a spectrally refined grid (SRG) and performs efficient grid scanning, generalized interpolations and topology operations on the basis of this grid structure. For the RLS surface part, we propose an accurate advection scheme targeted at SRG. For the explicit mesh part, we develop a fast grid scanning technique to voxelize the meshes and introduce novel strategies to detect grid regions that contain inconsistent mesh components. A robust algorithm is proposed to construct consistent local meshes to resolve mesh penetrations, and handle the coupling between explicit mesh and RLS surficial regions. We also provide further improvement on handling complicated topological variations, as well as strategies for remeshing mesh/RLS interconversions.
- 28 February 2018
- Published by Wiley in Computer Graphics Forum
- Vol. 37 (6) , 343-357
Field‐Aligned Isotropic Surface Remeshing
We present a novel isotropic surface remeshing algorithm that automatically aligns the mesh edges with an underlying directional field. The alignment is achieved by minimizing an energy function that combines both centroidal Voronoi tessellation (CVT) and the penalty enforced by a six‐way rotational symmetry field. The CVT term ensures uniform distribution of the vertices and high remeshing quality, and the field constraint enforces the directional alignment of the edges. Experimental results show that the proposed approach has the advantages of isotropic and field‐aligned remeshing. Our algorithm is superior to the representative state‐of‐the‐art approaches in various aspects.
- 28 February 2018
- Published by Wiley in Computer Graphics Forum
- Vol. 37 (6) , 343-357
Field‐Aligned Isotropic Surface Remeshing
We present a novel isotropic surface remeshing algorithm that automatically aligns the mesh edges with an underlying directional field. The alignment is achieved by minimizing an energy function that combines both centroidal Voronoi tessellation (CVT) and the penalty enforced by a six‐way rotational symmetry field. The CVT term ensures uniform distribution of the vertices and high remeshing quality, and the field constraint enforces the directional alignment of the edges. Experimental results show that the proposed approach has the advantages of isotropic and field‐aligned remeshing. Our algorithm is superior to the representative state‐of‐the‐art approaches in various aspects.
- 13 October 2017
- Published by Wiley in Computer Graphics Forum
- Vol. 36 (7) , 217-228
A Unified Cloth Untangling Framework Through Discrete Collision Detection
We present an efficient and stable framework, called Unified Intersection Resolver (UIR), for cloth simulation systems where not only impending collisions but also pre‐existing penetrations often arise. These two types of collisions are handled in a unified manner, by detecting edge‐face intersections first and then forming penetration stencils to be resolved iteratively. A stencil is a quadruple of vertices and it reveals either a vertex‐face or an edge‐edge collision event happened. Each quadruple also implicitly defines a collision normal, through which the four stencil vertices can be relocated, so that the corresponding edge‐face intersection disappear. We deduce three different ways, i.e., from predefined surface orientation, from history data and from global intersection analysis, to determine the collision normals of these stencils robustly. Multiple stencils that constitute a penetration region are processed simultaneously to eliminate penetrations. Cloth trapped in pinched environmental objects can be handled easily within our framework. We highlight its robustness by a number of challenging experiments involving collisions.
- 13 October 2017
- Published by Wiley in Computer Graphics Forum
- Vol. 36 (7) , 217-228
A Unified Cloth Untangling Framework Through Discrete Collision Detection
We present an efficient and stable framework, called Unified Intersection Resolver (UIR), for cloth simulation systems where not only impending collisions but also pre‐existing penetrations often arise. These two types of collisions are handled in a unified manner, by detecting edge‐face intersections first and then forming penetration stencils to be resolved iteratively. A stencil is a quadruple of vertices and it reveals either a vertex‐face or an edge‐edge collision event happened. Each quadruple also implicitly defines a collision normal, through which the four stencil vertices can be relocated, so that the corresponding edge‐face intersection disappear. We deduce three different ways, i.e., from predefined surface orientation, from history data and from global intersection analysis, to determine the collision normals of these stencils robustly. Multiple stencils that constitute a penetration region are processed simultaneously to eliminate penetrations. Cloth trapped in pinched environmental objects can be handled easily within our framework. We highlight its robustness by a number of challenging experiments involving collisions.
- 3 August 2015
- Published by Wiley in Computer Graphics Forum
- Vol. 35 (1) , 89-99
Anisotropic Strain Limiting for Quadrilateral and Triangular Cloth Meshes
The cloth simulation systems often suffer from excessive extension on the polygonal mesh, so an additional strain‐limiting process is typically used as a remedy in the simulation pipeline. A cloth model can be discretized as either a quadrilateral mesh or a triangular mesh, and their strains are measured differently. The edge‐based strain‐limiting method for a quadrilateral mesh creates anisotropic behaviour by nature, as discretization usually aligns the edges along the warp and weft directions. We improve this anisotropic technique by replacing the traditionally used equality constraints with inequality ones in the mathematical optimization, and achieve faster convergence. For a triangular mesh, the state‐of‐the‐art technique measures and constrains the strains along the two principal (and constantly changing) directions in a triangle, resulting in an isotropic behaviour which prohibits shearing. Based on the framework of inequality‐constrained optimization, we propose a warp and weft strain‐limiting formulation. This anisotropic model is more appropriate for textile materials that do not exhibit isotropic strain behaviour.
- 3 August 2015
- Published by Wiley in Computer Graphics Forum
- Vol. 35 (1) , 89-99
Anisotropic Strain Limiting for Quadrilateral and Triangular Cloth Meshes
The cloth simulation systems often suffer from excessive extension on the polygonal mesh, so an additional strain‐limiting process is typically used as a remedy in the simulation pipeline. A cloth model can be discretized as either a quadrilateral mesh or a triangular mesh, and their strains are measured differently. The edge‐based strain‐limiting method for a quadrilateral mesh creates anisotropic behaviour by nature, as discretization usually aligns the edges along the warp and weft directions. We improve this anisotropic technique by replacing the traditionally used equality constraints with inequality ones in the mathematical optimization, and achieve faster convergence. For a triangular mesh, the state‐of‐the‐art technique measures and constrains the strains along the two principal (and constantly changing) directions in a triangle, resulting in an isotropic behaviour which prohibits shearing. Based on the framework of inequality‐constrained optimization, we propose a warp and weft strain‐limiting formulation. This anisotropic model is more appropriate for textile materials that do not exhibit isotropic strain behaviour.
- Open Access
History-free Collision Response for Deformable Surfaces
Continuous collision detection (CCD) and response methods are widely adopted in dynamics simulation of deformable models. They are history-based, as their success is strictly based on an assumption of a collision-free state at the start of each time interval. On the other hand, in many applications surfaces have normals defined to designate their orientation (i.e. front- and back-face), yet CCD methods are totally blind to such orientation identification (thus are orientation-free). We notice that if such information is utilized, many penetrations can be untangled. In this paper we present a history-free method for separation of two penetrating meshes, where at least one of them has clarified surface orientation. This method first computes all edge-face (E-F) intersections with discrete collision detection (DCD), and then builds a number of penetration stencils. On response, the stencil vertices are relocated into a penetration-free state, via a global displacement minimizer. Our method is very effective for handling penetration between two meshes, being it an initial configuration or in the middle of physics simulation. The major limitation is that it is not applicable to self-collision within one mesh at the time being.
Open AccessHistory-free Collision Response for Deformable Surfaces
Continuous collision detection (CCD) and response methods are widely adopted in dynamics simulation of deformable models. They are history-based, as their success is strictly based on an assumption of a collision-free state at the start of each time interval. On the other hand, in many applications surfaces have normals defined to designate their orientation (i.e. front- and back-face), yet CCD methods are totally blind to such orientation identification (thus are orientation-free). We notice that if such information is utilized, many penetrations can be untangled. In this paper we present a history-free method for separation of two penetrating meshes, where at least one of them has clarified surface orientation. This method first computes all edge-face (E-F) intersections with discrete collision detection (DCD), and then builds a number of penetration stencils. On response, the stencil vertices are relocated into a penetration-free state, via a global displacement minimizer. Our method is very effective for handling penetration between two meshes, being it an initial configuration or in the middle of physics simulation. The major limitation is that it is not applicable to self-collision within one mesh at the time being.
- 30 July 2012
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 21 (12) , 4735-4745
Geometric Flow Approach for Region-Based Image Segmentation
Geometric flows have been successfully used for surface modeling and designing, largely because they are inherently good at controlling geometric shape evolution. Variational image segmentation approaches, on the other hand, detect objects of interest by deforming certain given shapes. This motivates us to revisit the minimal partition problem for segmentation of images, and propose a new geometric flow-based formulation and solution to it. Our model intends to derive a mapping that will evolve given contours or piecewise-constant regions toward objects in the image. The mapping is approximated by B-spline basis functions, and the positions of the control points are to be determined. Starting with the energy functional based on intensity averaging, we derive a Euler-Lagrange equation and then a geometric evolution equation. The linearized system of equations is efficiently solved via a special matrix-vector multiplication technique. Furthermore, we extend the piecewise-constant model to a piecewise-smooth model which effectively handles images with intensity inhomogeneity.
- 30 July 2012
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 21 (12) , 4735-4745
Geometric Flow Approach for Region-Based Image Segmentation
Geometric flows have been successfully used for surface modeling and designing, largely because they are inherently good at controlling geometric shape evolution. Variational image segmentation approaches, on the other hand, detect objects of interest by deforming certain given shapes. This motivates us to revisit the minimal partition problem for segmentation of images, and propose a new geometric flow-based formulation and solution to it. Our model intends to derive a mapping that will evolve given contours or piecewise-constant regions toward objects in the image. The mapping is approximated by B-spline basis functions, and the positions of the control points are to be determined. Starting with the energy functional based on intensity averaging, we derive a Euler-Lagrange equation and then a geometric evolution equation. The linearized system of equations is efficiently solved via a special matrix-vector multiplication technique. Furthermore, we extend the piecewise-constant model to a piecewise-smooth model which effectively handles images with intensity inhomogeneity.