Sunjin Jung
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Sunjin Jung has a total of 13 co-authors. Here are the top co-authors by number of shared publications:
- Junyong Noh
Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
5 shared publications Byungkuk Choi
Weta Digital Wellington New Zealand
2 shared publications- Kwanggyoon Seo
Graduate School of Culture Technology, KAIST
2 shared publications Kyungmin Cho
Visual Media Lab KAIST Daejeon South Korea
2 shared publicationsSeokpyo Hong
Visual Media Lab KAIST Daejeon South Korea
2 shared publications- Seonghyeon Kim
KAIST, Visual Media Lab, Seoul, Republic of Korea
2 shared publications
Latest Publications
- Open Access
- 1 October 2024
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 44 (1) , 1-14
Speed-Aware Audio-Driven Speech Animation using Adaptive Windows
We present a novel method that can generate realistic speech animations of a 3D face from audio using multiple adaptive windows. In contrast to previous studies that use a fixed size audio window, our method accepts an adaptive audio window as input, reflecting the audio speaking rate to use consistent phonemic information. Our system consists of three parts. First, the speaking rate is estimated from the input audio using a neural network trained in a self-supervised manner. Second, the appropriate window size that encloses the audio features is predicted adaptively based on the estimated speaking rate. Another key element lies in the use of multiple audio windows of different sizes as input to the animation generator: a small window to concentrate on detailed information and a large window to consider broad phonemic information near the center frame. Finally, the speech animation is generated from the multiple adaptive audio windows. Our method can generate realistic speech animations from in-the-wild audios at any speaking rate, i.e., fast raps, slow songs, as well as normal speech. We demonstrate via extensive quantitative and qualitative evaluations including a user study that our method outperforms state-of-the-art approaches.
- 1 October 2024
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 44 (1) , 1-14
Open AccessSpeed-Aware Audio-Driven Speech Animation using Adaptive Windows
We present a novel method that can generate realistic speech animations of a 3D face from audio using multiple adaptive windows. In contrast to previous studies that use a fixed size audio window, our method accepts an adaptive audio window as input, reflecting the audio speaking rate to use consistent phonemic information. Our system consists of three parts. First, the speaking rate is estimated from the input audio using a neural network trained in a self-supervised manner. Second, the appropriate window size that encloses the audio features is predicted adaptively based on the estimated speaking rate. Another key element lies in the use of multiple audio windows of different sizes as input to the animation generator: a small window to concentrate on detailed information and a large window to consider broad phonemic information near the center frame. Finally, the speech animation is generated from the multiple adaptive audio windows. Our method can generate realistic speech animations from in-the-wild audios at any speaking rate, i.e., fast raps, slow songs, as well as normal speech. We demonstrate via extensive quantitative and qualitative evaluations including a user study that our method outperforms state-of-the-art approaches.
- 1 October 2021
- Published by Wiley in Computer Graphics Forum
- Vol. 40 (7) , 45-55
Deep Learning‐Based Unsupervised Human Facial Retargeting
Traditional approaches to retarget existing facial blendshape animations to other characters rely heavily on manually paired data including corresponding anchors, expressions, or semantic parametrizations to preserve the characteristics of the original performance. In this paper, inspired by recent developments in face swapping and reenactment, we propose a novel unsupervised learning method that reformulates the retargeting of 3D facial blendshape‐based animations in the image domain. The expressions of a source model is transferred to a target model via the rendered images of the source animation. For this purpose, a reenactment network is trained with the rendered images of various expressions created by the source and target models in a shared latent space. The use of shared latent space enable an automatic cross‐mapping obviating the need for manual pairing. Next, a blendshape prediction network is used to extract the blendshape weights from the translated image to complete the retargeting of the animation onto a 3D target model. Our method allows for fully unsupervised retargeting of facial expressions between models of different configurations, and once trained, is suitable for automatic real‐time applications.
- 1 October 2021
- Published by Wiley in Computer Graphics Forum
- Vol. 40 (7) , 45-55
Deep Learning‐Based Unsupervised Human Facial Retargeting
Traditional approaches to retarget existing facial blendshape animations to other characters rely heavily on manually paired data including corresponding anchors, expressions, or semantic parametrizations to preserve the characteristics of the original performance. In this paper, inspired by recent developments in face swapping and reenactment, we propose a novel unsupervised learning method that reformulates the retargeting of 3D facial blendshape‐based animations in the image domain. The expressions of a source model is transferred to a target model via the rendered images of the source animation. For this purpose, a reenactment network is trained with the rendered images of various expressions created by the source and target models in a shared latent space. The use of shared latent space enable an automatic cross‐mapping obviating the need for manual pairing. Next, a blendshape prediction network is used to extract the blendshape weights from the translated image to complete the retargeting of the animation onto a 3D target model. Our method allows for fully unsupervised retargeting of facial expressions between models of different configurations, and once trained, is suitable for automatic real‐time applications.
- Open Access
- 1 July 2020
- Published by Korea Computer Graphics Society in Journal of the Korea Computer Graphics Society
- Vol. 26 (3) , 49-59
Speech Animation Synthesis based on a Korean Co-articulation Model
사단법인 한국컴퓨터그래픽스학회의 논문지로서 그래픽스 분야와 그에 연관된 연구영역의 새로운 발견과 최첨단 연구결과를 발표하는 토론의 장입니다. 학회를 대표하는 학술지인 논문지는 1995년에 논문지 발간을 시작으로 2000년부터 3월, 6월, 9월, 12월 총 4회 발간되었고, 2015년부터는 KCGS 학술대회 특별호를 7월에 발간하면서 총 5회 발간되고 있습니다.
- 1 July 2020
- Published by Korea Computer Graphics Society in Journal of the Korea Computer Graphics Society
- Vol. 26 (3) , 49-59
Open AccessSpeech Animation Synthesis based on a Korean Co-articulation Model
사단법인 한국컴퓨터그래픽스학회의 논문지로서 그래픽스 분야와 그에 연관된 연구영역의 새로운 발견과 최첨단 연구결과를 발표하는 토론의 장입니다. 학회를 대표하는 학술지인 논문지는 1995년에 논문지 발간을 시작으로 2000년부터 3월, 6월, 9월, 12월 총 4회 발간되었고, 2015년부터는 KCGS 학술대회 특별호를 7월에 발간하면서 총 5회 발간되고 있습니다.
- 29 November 2019
- Published by Wiley in Computer Graphics Forum
- Vol. 39 (1) , 595-606
Synthesizing Character Animation with Smoothly Decomposed Motion Layers
The processing of captured motion is an essential task for undertaking the synthesis of high‐quality character animation. The motion decomposition techniques investigated in prior work extract meaningful motion primitives that help to facilitate this process. Carefully selected motion primitives can play a major role in various motion‐synthesis tasks, such as interpolation, blending, warping, editing or the generation of new motions. Unfortunately, for a complex character motion, finding generic motion primitives by decomposition is an intractable problem due to the compound nature of the behaviours of such characters. Additionally, decomposed motion primitives tend to be too limited for the chosen model to cover a broad range of motion‐synthesis tasks. To address these challenges, we propose a generative motion decomposition framework in which the decomposed motion primitives are applicable to a wide range of motion‐synthesis tasks. Technically, the input motion is smoothly decomposed into three motion layers. These are base‐level motion, a layer with controllable motion displacements and a layer with high‐frequency residuals. The final motion can easily be synthesized simply by changing a single user parameter that is linked to the layer of controllable motion displacements or by imposing suitable temporal correspondences to the decomposition framework. Our experiments show that this decomposition provides a great deal of flexibility in several motion synthesis scenarios: denoising, style modulation, upsampling and time warping.
- 29 November 2019
- Published by Wiley in Computer Graphics Forum
- Vol. 39 (1) , 595-606
Synthesizing Character Animation with Smoothly Decomposed Motion Layers
The processing of captured motion is an essential task for undertaking the synthesis of high‐quality character animation. The motion decomposition techniques investigated in prior work extract meaningful motion primitives that help to facilitate this process. Carefully selected motion primitives can play a major role in various motion‐synthesis tasks, such as interpolation, blending, warping, editing or the generation of new motions. Unfortunately, for a complex character motion, finding generic motion primitives by decomposition is an intractable problem due to the compound nature of the behaviours of such characters. Additionally, decomposed motion primitives tend to be too limited for the chosen model to cover a broad range of motion‐synthesis tasks. To address these challenges, we propose a generative motion decomposition framework in which the decomposed motion primitives are applicable to a wide range of motion‐synthesis tasks. Technically, the input motion is smoothly decomposed into three motion layers. These are base‐level motion, a layer with controllable motion displacements and a layer with high‐frequency residuals. The final motion can easily be synthesized simply by changing a single user parameter that is linked to the layer of controllable motion displacements or by imposing suitable temporal correspondences to the decomposition framework. Our experiments show that this decomposition provides a great deal of flexibility in several motion synthesis scenarios: denoising, style modulation, upsampling and time warping.
- 12 May 2017
- Published by Wiley in Computer Animation and Virtual Worlds
- Vol. 28 (3-4)
Age‐related gait motion transformation based on biomechanical observations
We present a novel approach for synthesizing human gait motions according to a range of input ages by transforming a given motion based on biomechanical observations. Given an original motion, our method first extracts gait cycles that are periodically defined by foot contact on the ground and then transforms the original motion to achieve a desirable posture and motions that respectively correspond to the input age. Among many biomechanical features that gradually change with aging, we mainly focus on spatiotemporal and kinematic features as well as postural changes. Exploiting these features, we formulate the biomechanical changes as continuous functions that reflect visually significant features corresponding to the input age. Finally, we demonstrate that our system can automatically generate plausible gait motions given a wide range of input ages.
- 12 May 2017
- Published by Wiley in Computer Animation and Virtual Worlds
- Vol. 28 (3-4)
Age‐related gait motion transformation based on biomechanical observations
We present a novel approach for synthesizing human gait motions according to a range of input ages by transforming a given motion based on biomechanical observations. Given an original motion, our method first extracts gait cycles that are periodically defined by foot contact on the ground and then transforms the original motion to achieve a desirable posture and motions that respectively correspond to the input age. Among many biomechanical features that gradually change with aging, we mainly focus on spatiotemporal and kinematic features as well as postural changes. Exploiting these features, we formulate the biomechanical changes as continuous functions that reflect visually significant features corresponding to the input age. Finally, we demonstrate that our system can automatically generate plausible gait motions given a wide range of input ages.