JZJinsong Zhang

Jinsong Zhang

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Jinsong Zhang has a total of 29 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Multi-Scale Optimal Transport Transformer for Efficient Exemplar-Based Image Translation

    by Jinsong Zhang, Xiongzheng Li, Yuqin Lin

    Exemplar-based image translation generates an output image by transferring appearance from a reference exemplar to a content image. Existing works only consider the local correspondences between two modalities, and ignore the global distributions in each modality, struggling to obtain fine-grained details with efficient computation. In this paper, we propose OTFormer, a multi-scale Optimal Transport transformer for exemplarbased image translation. We formulate cross-modal alignment as a multi-scale optimal transport problem, which progressively provides a globally coherent matching. In addition, we design a lightweight multi-scale fusion block to extract and fuse features efficiently. Experiments on CelebA-HQ and DeepFashion demonstrate that OTFormer improves both image fidelity and style adherence, while reducing model parameters by 62% and achieving faster inference compared with strong baselines. These results highlight OTguided global alignment as an effective and deployable solution for high-fidelity exemplarbased image translation.

  • Open Access

    H2Avatar: Expressive Whole-Body Avatars from Monocular Video via Hierarchical Geometry and Hybrid Rendering

    by Jinsong Zhang, Cheng Guan, Zhihua Lin, Yuqin Lin

    Reconstructing photorealistic and animatable whole-body avatars from monocular videos is a hot topic in computer vision and computer graphics. However, existing methods still face challenges due to the limited frequency response of single-scale geometry encodings and the instability of appearance modeling without an explicit surface anchor. In this paper, we present H2Avatar, a real-time framework that builds on a mesh-embedded 3D Gaussian representation guided by SMPL-X and disentangles geometry and appearance into hierarchical and hybrid components. For geometry, we propose a semantic-aware hierarchical encoding based on a multi-scale tri-plane pyramid, where features at different resolutions capture both global structure and high-frequency surface details such as clothing wrinkles. For appearance, we introduce a hybrid rendering strategy that anchors canonical colors using a learnable UV texture map, and complements it with a neural residual color branch conditioned on tri-plane features, pose embedding, and surface normals to model pose- and view-dependent shading variations. This design improves temporal stability and preserves identity details while enhancing photorealism under complex motions. Experiments on the NeuMan dataset demonstrate that H2Avatar consistently outperforms representative baselines across multiple sequences, outperforming ExAvatar by up to 0.66 dB in PSNR and reducing LPIPS by up to 16.3%. These results validate the effectiveness of hierarchical geometry encoding and texture-anchored hybrid appearance modeling.


  • Open Access

    HOIMamba: Bidirectional State-Space Modeling for Monocular 3D Human–Object Interaction Reconstruction

    by Jinsong Zhang, Yuqin Lin

    Monocular 3D human–object interaction (HOI) reconstruction requires jointly recovering articulated human geometry, object pose, and physically plausible contact from a single RGB image. While recent token-based methods commonly employ dense self-attention to capture global dependencies, isotropic all-to-all mixing tends to entangle spatial-geometric cues (e.g., contact locality) with channel-wise semantic cues (e.g., action/affordance), and provides limited control for representing directional and asymmetric physical influence between humans and objects. This paper presents HOIMamba, a state-space sequence modeling framework that reformulates HOI reconstruction as bidirectional, multi-scale interaction state inference. Instead of relying on symmetric correlation aggregation, HOIMamba uses structured state evolution to propagate interaction evidence. We introduce a multi-scale state-space module (MSSM) to capture interaction dependencies spanning local contact details and global body–object coordination. Building on MSSM, we propose a spatial-channel grouped SSM (SCSSM) block that factorizes interaction modeling into a spatial pathway for geometric/contact dependencies and a channel pathway for semantic/functional correlations, followed by gated fusion. HOIMamba further performs explicit bidirectional propagation between human and object states to better reflect asymmetric reciprocity in physical interactions. We evaluate HOIMamba on two public benchmarks, BEHAVE and InterCap, using Chamfer distance for human/object meshes and contact precision/recall induced by reconstructed geometry. HOIMamba achieves consistent improvements over representative prior methods. On the BEHAVE dataset, it reduces human Chamfer distance by 8.6% and improves contact recall by 13.5% compared to the strongest Transformer-based baseline, with similar gains observed on the InterCap dataset. Ablation studies on BEHAVE verify the contributions of state-space modeling, multi-scale inference, spatial-channel factorization, and bidirectional interaction reasoning.

  • SpeechAct: Towards Generating Whole-Body Motion From Speech

    by Jinsong Zhang, Minjie Zhu, Yuxiang Zhang, Zerong Zheng, Yebin Liu, Kun Li

    Whole-body motion generation from speech audio is crucial for computer graphics and immersive VR/AR. Prior methods struggle to produce natural and diverse whole-body motions from speech. In this paper, we introduce a novel method, named SpeechAct, based on a hybrid point representation and contrastive motion learning to boost realism and diversity in motion generation. Our hybrid point representation leverages the advantages of keypoint representation and surface points of 3D body model, which is easy to learn and helps to achieve smooth and natural motion generation from speech audio. We design a VQ-VAE to learn a motion codebook using our hybrid presentation, and then regress the motion from the input audio using a translation model. To boost diversity in motion generation, we propose a contrastive motion learning method according to the intuitive idea that the generated motion should be different from the motions of other audios and other speakers. We collect negative samples from other audio inputs and other speakers using our translation model. With these negative samples, we pull the current motion away from them using a contrastive loss to produce more distinctive representations. In addition, we compose a face generator to generate deterministic face motion due to the strong connection between the face movements and the speech audio. Experimental results validate the superior performance of our model. The code is available at http://cic.tju.edu.cn/faculty/likun/projects/SpeechAct/index.html.


  • PISE-V: person image and video synthesis with decoupled GAN

    by Jinsong Zhang, Yu-Kun Lai, Jingyu Yang, Kun Li


  • Open Access

    DualAvatar: Robust Gaussian Splatting Avatar with Dual Representation

    by Jinsong Zhang, I-Chao Shen, Jotaro Sakamiya, Yu-Kun Lai, Takeo Igarashi, Kun Li


  • Open Access

    STATE: Learning structure and texture representations for novel view synthesis

    by Xinyi Jing, Qiao Feng, Yu-Kun Lai, Jinsong Zhang, Yuanqiang Yu, Kun Li

    Novel viewpoint image synthesis is very challenging, especially from sparse views, due to large changes in viewpoint and occlusion. Existing image-based methods fail to generate reasonable results for invisible regions, while geometry-based methods have difficulties in synthesizing detailed textures. In this paper, we propose STATE, an end-to-end deep neural network, for sparse view synthesis by learning structure and texture representations. Structure is encoded as a hybrid feature field to predict reasonable structures for invisible regions while maintaining original structures for visible regions, and texture is encoded as a deformed feature map to preserve detailed textures. We propose a hierarchical fusion scheme with intra-branch and inter-branch aggregation, in which spatio-view attention allows multi-view fusion at the feature level to adaptively select important information by regressing pixel-wise or voxel-wise confidence maps. By decoding the aggregated features, STATE is able to generate realistic images with reasonable structures and detailed textures. Experimental results demonstrate that our method achieves qualitatively and quantitatively better results than state-of-the-art methods. Our method also enables texture and structure editing applications benefiting from implicit disentanglement of structure and texture. Our code is available at http://cic.tju.edu.cn/faculty/likun/projects/STATE.


  • High-Quality Animatable Dynamic Garment Reconstruction From Monocular Videos

    by Xiongzheng Li, Jinsong Zhang, Yu-Kun Lai, Jingyu Yang, Kun Li

    Much progress has been made in reconstructing garments from an image or a video. However, none of existing works meet the expectations of digitizing high-quality animatable dynamic garments that can be adjusted to various unseen poses. In this paper, we propose the first method to recover high-quality animatable dynamic garments from monocular videos without depending on scanned data. To generate reasonable deformations for various unseen poses, we propose a learnable garment deformation network that formulates the garment reconstruction task as a pose-driven deformation problem. To alleviate the ambiguity estimating 3D garments from monocular videos, we design a multi-hypothesis deformation module that learns spatial representations of multiple plausible deformations. Experimental results on several public datasets demonstrate that our method can reconstruct high-quality dynamic garments with coherent surface details, which can be easily animated under unseen poses. The code will be provided for research purposes.


  • Toward Grouping in Large Scenes With Occlusion-Aware Spatio–Temporal Transformers

    by Jinsong Zhang, Lingfeng Gu, Yu-Kun Lai, Xueyang Wang, Kun Li

    Group detection, especially for large-scale scenes, has many potential applications for public safety and smart cities. Existing methods fail to cope with frequent occlusions in large-scale scenes with multiple people, and are difficult to effectively utilize spatio-temporal information. In this paper, we propose an end-to-end framework, GroupTransformer , for group detection in large-scale scenes. To deal with the frequent occlusions caused by multiple people, we design an occlusion encoder to detect and suppress severely occluded person crops. To explore the potential spatio-temporal relationship, we propose spatio-temporal transformers to simultaneously extract trajectory information and fuse inter-person features in a hierarchical manner. Experimental results on both large-scale and small-scale scenes demonstrate that our method achieves better performance compared with state-of-the-art methods. On large-scale scenes, our method significantly boosts the performance in terms of precision and F1 score by more than 10%. On small-scale scenes, our method still improves the performance of F1 score by more than 5%. We will release the code for research purposes .