Niloy Jyoti Mitra
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Niloy Jyoti Mitra has a total of 150 co-authors. Here are the top co-authors by number of shared publications:
- Paul Guerrero
Adobe Research, London, United Kingdom
11 shared publications - Andrea Vedaldi
University of Oxford
7 shared publications - Niladri Shekhar Dutt
University College London (UCL), London, United Kingdom
7 shared publications - Peter Wonka
KAUST
5 shared publications - Vladimir G. Kim
Adobe Research, Seattle, USA
5 shared publications - Noam Aigerman
Université de Montréal, Montréal, Canada
4 shared publications
Latest Publications
- 13 November 2026
- Published by Wiley in Computer Graphics Forum
A Study of the Design Space of Motion and Disocclusion Control in Video Generation
Disocclusion occurs when object movement reveals previously hidden content. Existing image‐to‐video methods provide different forms of motion and disocclusion control, yet the design space they span remains poorly understood. In this work, we systematically explore an algorithmic design space spanning motion specification (text‐based versus tracking‐based) and dis‐occlusion specification (text‐based versus image‐based). To facilitate this study, we construct a benchmark consisting of both synthetic scenes and captured scenes, covering rigid and deformable motions, as well as static and dynamic disoccluded objects. We compare existing and adapted methods across this design space to analyze the strengths and limitations of different control strategies for handling disocclusions during motion. Our study reveals that tracking‐based guidance improves motion accuracy, while image‐based guidance improves the fidelity of newly revealed content. However, jointly controlling the appearance and dynamics of disoccluded objects remains a challenging open problem, particularly in scenarios where newly revealed objects undergo their own motion after becoming visible.
- 13 November 2026
- Published by Wiley in Computer Graphics Forum
A Study of the Design Space of Motion and Disocclusion Control in Video Generation
Disocclusion occurs when object movement reveals previously hidden content. Existing image‐to‐video methods provide different forms of motion and disocclusion control, yet the design space they span remains poorly understood. In this work, we systematically explore an algorithmic design space spanning motion specification (text‐based versus tracking‐based) and dis‐occlusion specification (text‐based versus image‐based). To facilitate this study, we construct a benchmark consisting of both synthetic scenes and captured scenes, covering rigid and deformable motions, as well as static and dynamic disoccluded objects. We compare existing and adapted methods across this design space to analyze the strengths and limitations of different control strategies for handling disocclusions during motion. Our study reveals that tracking‐based guidance improves motion accuracy, while image‐based guidance improves the fidelity of newly revealed content. However, jointly controlling the appearance and dynamics of disoccluded objects remains a challenging open problem, particularly in scenarios where newly revealed objects undergo their own motion after becoming visible.
- Open Access
- 13 November 2026
- Published by Wiley in Computer Graphics Forum
Blended Chart Surfaces: A Seamless Explicit Representation for Smooth Surface Fitting
A surface representation suitable for geometry processing should be compact and explicit , provide global smoothness guarantees , support a wide range of surface topologies , and offer reliable access to differential quantities such as normals and surface energies, while remaining compatible with modern differentiable optimization. Yet existing neural representations typically sacrifice one or more of these properties: implicit fields typically require iso‐surfacing for downstream use, while explicit neural maps are constrained by canonical‐domain parametrizations and/or exhibit seam artifacts between local charts. We introduce Blended Chart Surfaces (BCS) , a compact, network‐free, explicit representation that is smooth by construction and anchored to user‐provided topology. Given a coarse proxy mesh encoding the intended surface topology and approximate geometry, Blended Chart Surfaces jointly optimize for a polynomial map at each proxy vertex using an off‐the‐shelf optimizer to fit to an implicit target shape, avoiding the need for an input parametrization. Neighboring maps are fused using a smooth ‘one‐ring coordinate’ blending scheme, decoupling surface topology and coarse geometry (carried by the proxy) from geometric details (carried by the smooth local patches). The resulting surface is globally smooth, fully differentiable, and enables stable evaluation of positions and derivatives, making differential quantities and surface energies directly accessible. Additionally, our construction is equivariant to rigid motions and scaling of the proxy mesh. We evaluate Blended Chart Surfaces on surfaces spanning varying topology and geometric complexity, and compare against explicit alternatives including interpolating‐function baselines and mesh‐displacement MLPs. Across these, Blended Chart Surfaces achieves a favorable trade‐off among compactness, simplicity, access to differential quantities, and expressivity while remaining smooth across patch boundaries. Link to project webpage.
- 13 November 2026
- Published by Wiley in Computer Graphics Forum
Open AccessBlended Chart Surfaces: A Seamless Explicit Representation for Smooth Surface Fitting
A surface representation suitable for geometry processing should be compact and explicit , provide global smoothness guarantees , support a wide range of surface topologies , and offer reliable access to differential quantities such as normals and surface energies, while remaining compatible with modern differentiable optimization. Yet existing neural representations typically sacrifice one or more of these properties: implicit fields typically require iso‐surfacing for downstream use, while explicit neural maps are constrained by canonical‐domain parametrizations and/or exhibit seam artifacts between local charts. We introduce Blended Chart Surfaces (BCS) , a compact, network‐free, explicit representation that is smooth by construction and anchored to user‐provided topology. Given a coarse proxy mesh encoding the intended surface topology and approximate geometry, Blended Chart Surfaces jointly optimize for a polynomial map at each proxy vertex using an off‐the‐shelf optimizer to fit to an implicit target shape, avoiding the need for an input parametrization. Neighboring maps are fused using a smooth ‘one‐ring coordinate’ blending scheme, decoupling surface topology and coarse geometry (carried by the proxy) from geometric details (carried by the smooth local patches). The resulting surface is globally smooth, fully differentiable, and enables stable evaluation of positions and derivatives, making differential quantities and surface energies directly accessible. Additionally, our construction is equivariant to rigid motions and scaling of the proxy mesh. We evaluate Blended Chart Surfaces on surfaces spanning varying topology and geometric complexity, and compare against explicit alternatives including interpolating‐function baselines and mesh‐displacement MLPs. Across these, Blended Chart Surfaces achieves a favorable trade‐off among compactness, simplicity, access to differential quantities, and expressivity while remaining smooth across patch boundaries. Link to project webpage.
- Open Access
- 9 November 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 45 (6) , 1-21
ViPS: Video-informed Pose Spaces for Auto-Rigged Meshes
Kinematic rigs provide a structured interface for articulating 3D meshes but lack any associated pose space , i.e., an explicit representation of the plausible manifold of joint configurations for a given mesh. Without such a pose space, stochastic sampling or manual manipulation of raw rig parameters easily results in semantic and/or geometric violations, such as anatomical hyperextension and non-physical self-intersections. We propose Video-informed Pose Spaces (ViPS), a feed-forward framework that discovers the latent distribution of valid articulations for (auto-)rigged meshes by distilling motion priors from a pretrained video diffusion model. Unlike existing methods that rely on scarce, artist-authored 4D datasets, or focus on reconstructing instances of individual motions, ViPS transfers generative video model priors into a universal distribution over the given rig parameterization. Differentiable geometric constraints on the skinned mesh naturally preserve shape-specific geometric consistency without requiring additional (hand-coded) mesh regularization. Our feed-forward model reveals a smooth, compact, and controllable pose space. This, in turn, supports sampling for diverse shape variations, manifold projection for inverse kinematics, and temporally coherent trajectories for animation and keyframing. Further, the distilled 3D pose samples serve as semantic proxies to guide video diffusion, effectively closing the loop between generative 2D priors and structured 3D kinematic control. Our evaluations show that ViPS, trained solely using video priors, matches the performance of state-of-the-art models trained on synthetic artist-created 4D data in both plausibility and diversity. Additionally, as a universal model, ViPS exhibits robust zero-shot generalization to out-of-distribution species and unseen skeletal topologies. Project Page: https://honglin-c.github.io/vips/
- 9 November 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 45 (6) , 1-21
Open AccessViPS: Video-informed Pose Spaces for Auto-Rigged Meshes
Kinematic rigs provide a structured interface for articulating 3D meshes but lack any associated pose space , i.e., an explicit representation of the plausible manifold of joint configurations for a given mesh. Without such a pose space, stochastic sampling or manual manipulation of raw rig parameters easily results in semantic and/or geometric violations, such as anatomical hyperextension and non-physical self-intersections. We propose Video-informed Pose Spaces (ViPS), a feed-forward framework that discovers the latent distribution of valid articulations for (auto-)rigged meshes by distilling motion priors from a pretrained video diffusion model. Unlike existing methods that rely on scarce, artist-authored 4D datasets, or focus on reconstructing instances of individual motions, ViPS transfers generative video model priors into a universal distribution over the given rig parameterization. Differentiable geometric constraints on the skinned mesh naturally preserve shape-specific geometric consistency without requiring additional (hand-coded) mesh regularization. Our feed-forward model reveals a smooth, compact, and controllable pose space. This, in turn, supports sampling for diverse shape variations, manifold projection for inverse kinematics, and temporally coherent trajectories for animation and keyframing. Further, the distilled 3D pose samples serve as semantic proxies to guide video diffusion, effectively closing the loop between generative 2D priors and structured 3D kinematic control. Our evaluations show that ViPS, trained solely using video priors, matches the performance of state-of-the-art models trained on synthetic artist-created 4D data in both plausibility and diversity. Additionally, as a universal model, ViPS exhibits robust zero-shot generalization to out-of-distribution species and unseen skeletal topologies. Project Page: https://honglin-c.github.io/vips/
- Open Access
- 9 November 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 45 (6) , 1-12
Open-CHOIR: Open-World Contact-Aware 4D Hand-Object Interaction Reconstruction
We ask whether everyday open-world monocular videos can be turned into reusable 4D interaction primitives: articulated hand motion, object shape with 6D pose over time, and the when/where of contact. Such a capability would enable scalable mining of real interactions and, beyond reconstruction, support scene-aware synthesis and planning. However, reconstructing hand-object interaction (HOI) from challenging monocular videos remains difficult: methods often assume known objects or curated scenes, and separately estimated hands and objects easily become misaligned under clutter, occlusion, and unseen object geometries. Targeting this setting, we present Open-CHOIR, an Open-world Contact-aware HOI Reconstruction framework for a monocular camera, using contact as an explicit coupling signal between hands and objects. Open-CHOIR first initializes a coarse, contact-agnostic 4D HOI sequence from open-world visual priors. It then introduces a generative HOI spatial rectification module to predict ray-depth corrections and rectify hand-object relative placement, then derive initial per-frame contact correspondences on the rectified geometry. Last, a contact-aware joint optimization with dynamically updated contact constraints enforces geometric, temporal, and contact consistency. Experiments on controlled and challenging videos show that Open-CHOIR improves object reconstruction, physical plausibility, and temporal consistency over state-of-the-art methods. Code, data, and pretrained weights are available at https://github.com/hxwork/CHOIR.
- 9 November 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 45 (6) , 1-12
Open AccessOpen-CHOIR: Open-World Contact-Aware 4D Hand-Object Interaction Reconstruction
We ask whether everyday open-world monocular videos can be turned into reusable 4D interaction primitives: articulated hand motion, object shape with 6D pose over time, and the when/where of contact. Such a capability would enable scalable mining of real interactions and, beyond reconstruction, support scene-aware synthesis and planning. However, reconstructing hand-object interaction (HOI) from challenging monocular videos remains difficult: methods often assume known objects or curated scenes, and separately estimated hands and objects easily become misaligned under clutter, occlusion, and unseen object geometries. Targeting this setting, we present Open-CHOIR, an Open-world Contact-aware HOI Reconstruction framework for a monocular camera, using contact as an explicit coupling signal between hands and objects. Open-CHOIR first initializes a coarse, contact-agnostic 4D HOI sequence from open-world visual priors. It then introduces a generative HOI spatial rectification module to predict ray-depth corrections and rectify hand-object relative placement, then derive initial per-frame contact correspondences on the rectified geometry. Last, a contact-aware joint optimization with dynamically updated contact constraints enforces geometric, temporal, and contact consistency. Experiments on controlled and challenging videos show that Open-CHOIR improves object reconstruction, physical plausibility, and temporal consistency over state-of-the-art methods. Code, data, and pretrained weights are available at https://github.com/hxwork/CHOIR.
- Open Access
- 9 October 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 46 (2) , 1-29
ShapeLib: Designing a library of programmatic 3D shape abstractions with Large Language Models
We present ShapeLib , the first method that uses the priors of Large Language Models (LLMs) to design libraries of programmatic 3D shape abstractions. Our system accepts two forms of user-provided design intent: high-level text descriptions of functions to include in the output library and a small seed set of exemplar shapes. We discover a library of abstractions that matches this design intent with a guided LLM workflow that first proposes different ways of applying and implementing functions, and then validates these functions are helpful in representing seed set shapes. To extend beyond the seed set, we develop library-specific recognition networks that map shapes (represented as primitives, voxels, or point clouds) to programs that use these newly discovered abstractions. Across multiple modeling domains (split by shape category), we find that LLMs, when thoughtfully combined with geometric reasoning, can be guided to author libraries of abstraction functions that generalize across shape distributions. Our framework takes a step towards realizing the long-standing shape analysis aspiration of discovering reusable, programmatic shape abstractions while exposing interpretable, semantically aligned interfaces. Our extensive evaluation demonstrates that ShapeLib provides distinct advantages over prior alternative abstraction discovery works in terms of generalization, usability, and maintaining plausibility under manipulation. Finally, we demonstrate that ShapeLib’s abstraction functions unlock a number of downstream applications, combining LLM reasoning over shape programs with geometry processing tools to support shape editing and generation workflows.
- 9 October 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 46 (2) , 1-29
Open AccessShapeLib: Designing a library of programmatic 3D shape abstractions with Large Language Models
We present ShapeLib , the first method that uses the priors of Large Language Models (LLMs) to design libraries of programmatic 3D shape abstractions. Our system accepts two forms of user-provided design intent: high-level text descriptions of functions to include in the output library and a small seed set of exemplar shapes. We discover a library of abstractions that matches this design intent with a guided LLM workflow that first proposes different ways of applying and implementing functions, and then validates these functions are helpful in representing seed set shapes. To extend beyond the seed set, we develop library-specific recognition networks that map shapes (represented as primitives, voxels, or point clouds) to programs that use these newly discovered abstractions. Across multiple modeling domains (split by shape category), we find that LLMs, when thoughtfully combined with geometric reasoning, can be guided to author libraries of abstraction functions that generalize across shape distributions. Our framework takes a step towards realizing the long-standing shape analysis aspiration of discovering reusable, programmatic shape abstractions while exposing interpretable, semantically aligned interfaces. Our extensive evaluation demonstrates that ShapeLib provides distinct advantages over prior alternative abstraction discovery works in terms of generalization, usability, and maintaining plausibility under manipulation. Finally, we demonstrate that ShapeLib’s abstraction functions unlock a number of downstream applications, combining LLM reasoning over shape programs with geometry processing tools to support shape editing and generation workflows.
- 17 September 2026
- Published by Springer Nature
- p. 73-90
LooseControlVideo: Directorial Video Control Using Spatial Blocking
- 17 September 2026
- Published by Springer Nature
- p. 73-90
LooseControlVideo: Directorial Video Control Using Spatial Blocking
- Open Access
- 19 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-12
B-repLer: Language-guided Editing of CAD Models
Computer-Aided Design (CAD) models, given their compactness and precision, remain the industry standard for designing and fabricating engineering objects. However, language-guided CAD editing is still in its infancy, largely due to missing semantic connection between user commands and underlying shape geometry, a problem exacerbated by the shortage of paired text-and-edit CAD datasets. While recent Multimodal Large Language Models (mLLMs) have attempted to bridge this gap, their reliance on CAD construction history –often an expensive and hard to obtain input– severely limits their expressiveness and restricts their usage. We present B-repLer, a novel framework that directly connects natural language with editing CAD models by operating in a learned latent space. Importantly, our approach bypasses the need for construction history, enabling semantic edits on a wide range of geometries, from simple prismatic parts to complex freeform shapes defined by B-Spline surfaces. To facilitate this research, we introduce BrepEDIT-240K, the first large-scale dataset for this task. We demonstrate how this paired dataset can be automatically generated, (user) validated, and scaled by leveraging existing CAD tools, in conjunction with mLLMs, to create the required paired data without relying on any external annotations. Our results demonstrate that B-repLer can accurately perform complex edits on complex CAD shapes, even when the input edit specifications are high-level and ambiguous to interpret, consistently producing valid, high-quality CAD outputs enabling a class of text-guided edits not previously possible. Project page is at https://yilinliu77.github.io/brepler.github.io/.
- 19 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-12
Open AccessB-repLer: Language-guided Editing of CAD Models
Computer-Aided Design (CAD) models, given their compactness and precision, remain the industry standard for designing and fabricating engineering objects. However, language-guided CAD editing is still in its infancy, largely due to missing semantic connection between user commands and underlying shape geometry, a problem exacerbated by the shortage of paired text-and-edit CAD datasets. While recent Multimodal Large Language Models (mLLMs) have attempted to bridge this gap, their reliance on CAD construction history –often an expensive and hard to obtain input– severely limits their expressiveness and restricts their usage. We present B-repLer, a novel framework that directly connects natural language with editing CAD models by operating in a learned latent space. Importantly, our approach bypasses the need for construction history, enabling semantic edits on a wide range of geometries, from simple prismatic parts to complex freeform shapes defined by B-Spline surfaces. To facilitate this research, we introduce BrepEDIT-240K, the first large-scale dataset for this task. We demonstrate how this paired dataset can be automatically generated, (user) validated, and scaled by leveraging existing CAD tools, in conjunction with mLLMs, to create the required paired data without relying on any external annotations. Our results demonstrate that B-repLer can accurately perform complex edits on complex CAD shapes, even when the input edit specifications are high-level and ambiguous to interpret, consistently producing valid, high-quality CAD outputs enabling a class of text-guided edits not previously possible. Project page is at https://yilinliu77.github.io/brepler.github.io/.
- Open Access
- 30 April 2026
- Published by Wiley in Computer Graphics Forum
SAGE: Structure‐Aware Generative Video Transitions between Diverse Clips
Video transitions aim to synthesize intermediate frames between two clips, but naïve approaches such as linear blending introduce artifacts that limit professional use or break temporal coherence. Traditional techniques (cross‐fades, morphing, frame interpolation) and recent generative inbetweening methods can produce high‐quality plausible intermediates, but they struggle with bridging diverse clips involving large temporal gaps or significant semantic differences, leaving a gap for content‐aware and visually coherent transitions. We address this challenge by drawing on artistic workflows, distilling strategies such as aligning silhouettes and interpolating salient features to preserve structure and perceptual continuity. Building on these strategies, we propose SAGE (Structure‐Aware Generative vidEo transitions) as a simple yet effective zeroshot approach that combines structural guidance, provided via line maps and motion flow, with generative synthesis, enabling smooth, motion‐consistent transitions without fine‐tuning. Extensive experiments and comparison with current alternatives, namely [RKT*22, ZCL*24, ZZX*24, JHM*25, ZRW*25], demonstrate that SAGE outperforms both classical and the latest generative baselines on quantitative metrics and user studies for producing transitions between diverse clips. The simple method effectively bypasses the need to acquire suitable training data, which is particularly difficult in our creative setting involving diverse clips. Code is available via the project page at https://kan32501.github.io/sage.github.io/ .
- 30 April 2026
- Published by Wiley in Computer Graphics Forum
Open AccessSAGE: Structure‐Aware Generative Video Transitions between Diverse Clips
Video transitions aim to synthesize intermediate frames between two clips, but naïve approaches such as linear blending introduce artifacts that limit professional use or break temporal coherence. Traditional techniques (cross‐fades, morphing, frame interpolation) and recent generative inbetweening methods can produce high‐quality plausible intermediates, but they struggle with bridging diverse clips involving large temporal gaps or significant semantic differences, leaving a gap for content‐aware and visually coherent transitions. We address this challenge by drawing on artistic workflows, distilling strategies such as aligning silhouettes and interpolating salient features to preserve structure and perceptual continuity. Building on these strategies, we propose SAGE (Structure‐Aware Generative vidEo transitions) as a simple yet effective zeroshot approach that combines structural guidance, provided via line maps and motion flow, with generative synthesis, enabling smooth, motion‐consistent transitions without fine‐tuning. Extensive experiments and comparison with current alternatives, namely [RKT*22, ZCL*24, ZZX*24, JHM*25, ZRW*25], demonstrate that SAGE outperforms both classical and the latest generative baselines on quantitative metrics and user studies for producing transitions between diverse clips. The simple method effectively bypasses the need to acquire suitable training data, which is particularly difficult in our creative setting involving diverse clips. Code is available via the project page at https://kan32501.github.io/sage.github.io/ .
- Open Access
- 14 December 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-11
Capturing Non-Linear Human Perspective in Line Drawings
- 14 December 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-11
Open AccessCapturing Non-Linear Human Perspective in Line Drawings
- Open Access
- 4 December 2025
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 44 (6) , 1-11
SMF: Template-free and Rig-free Animation Transfer using Kinetic Codes
Animation retargetting applies sparse motion description (e.g., keypoint sequences) to a character mesh to produce a semantically plausible and temporally coherent full-body mesh sequence. Existing approaches come with restrictions - they require access to template-based shape priors or artist-designed deformation rigs, suffer from limited generalization to unseen motion and/or shapes, or exhibit motion jitter. We propose Self-supervised Motion Fields (SMF), a self-supervised framework that is trained with only sparse motion representations, without requiring dataset-specific annotations, templates, or rigs. At the heart of our method are Kinetic Codes, a novel autoencoder-based sparse motion encoding, that exposes a semantically rich latent space, simplifying large-scale training. Our architecture comprises dedicated spatial and temporal gradient predictors, which are jointly trained in an end-to-end fashion. The combined network, regularized by the Kinetic Codes' latent space, has good generalization across both unseen shapes and new motions. We evaluated our method on unseen motion sampled from AMASS, D4D, Mixamo, and raw monocular video for animation transfer on various characters with varying shapes and topology. We report a new SoTA on the AMASS dataset in the context of generalization to unseen motion.
- 4 December 2025
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 44 (6) , 1-11
Open AccessSMF: Template-free and Rig-free Animation Transfer using Kinetic Codes
Animation retargetting applies sparse motion description (e.g., keypoint sequences) to a character mesh to produce a semantically plausible and temporally coherent full-body mesh sequence. Existing approaches come with restrictions - they require access to template-based shape priors or artist-designed deformation rigs, suffer from limited generalization to unseen motion and/or shapes, or exhibit motion jitter. We propose Self-supervised Motion Fields (SMF), a self-supervised framework that is trained with only sparse motion representations, without requiring dataset-specific annotations, templates, or rigs. At the heart of our method are Kinetic Codes, a novel autoencoder-based sparse motion encoding, that exposes a semantically rich latent space, simplifying large-scale training. Our architecture comprises dedicated spatial and temporal gradient predictors, which are jointly trained in an end-to-end fashion. The combined network, regularized by the Kinetic Codes' latent space, has good generalization across both unseen shapes and new motions. We evaluated our method on unseen motion sampled from AMASS, D4D, Mixamo, and raw monocular video for animation transfer on various characters with varying shapes and topology. We report a new SoTA on the AMASS dataset in the context of generalization to unseen motion.