Research

Visual perception and representation learning build the fundamental capability of machines, extracting meaningful structure from complex visual signals and learning representations that support understanding and reasoning. The Perception & Representation Learning team studies learning principles and model architectures that capture objects, actions, scenes, and their relationships across images and videos.

Building on this foundation, we focus on detailed video understanding, which requires reasoning about how objects, actions, and events unfold over time. We develop methods for video summarization, referring video object segmentation, highlight detection, video retrieval, and related tasks. We also explore multimodal information beyond vision, such as language and audio, and leverage multimodal large language models for richer perception and reasoning. We are further interested in extending this line of research toward world models and intelligent agents that model dynamic environments, anticipate future events, and connect perception with reasoning and action.

Recent publications

• Domain Generalization via Text-Anchored Information Bottleneck (ECCV'26)
• TripleSumm: Adaptive Triple-Modality Fusion for Video Summarization (ICLR'26)
• Towards Motion-aware Referring Image Segmentation (AISTATS'26)
• SummDiff: Generative Modeling of Video Summarization with Diffusion (ICCV'25)

5-min presentations

• TripleSumm: Adaptive Triple-Modality Fusion for Video Summarization (Yoori Oh)
• Do Not Stop at Frame Scoring: Enabling End-to-end Training for Video Summarization (Sumin Kim)
• From Matching to Reasoning: Query-Aware Long Video Summarization (Mingu Kang)
• Domain Generalization via Text-Anchored Information Bottleneck (Eunyi Lyou)
• Motion-guided Referring Video Object Segmentation (Hyunjin Lee)

Generative modeling aims to enable the machines to create artifcats that human can consume. With recent adances, diffusion and flow-based models have become the standard for visual generative modeling. Overcoming their drawbacks on inference latency, diffusion is being brought out into the real world, expanding toward distillation for single- (or few-) step generation, steering for human preference alignment, and new generative frameworks designed for one-step generation. On top of that, with world models emerging as a major trend, few-step video diffusion models are now where most of the attention goes.

In step with this trend, we are focusing on reward fine-tuning of diffusion models for better human preference alignment, efficient generative modeling for faster generation, few-step posterior sampling, and a wide range of world models, including long-horizon, 3D-consistent, and interactive video generation as well as world action models.

Recent publications

• Efficient Generative Modeling beyond Memoryless Diffusion via Adjoint Schrödinger Bridge Matching (ICML'26)
• Equivariant Latent Alignment via Flow Matching under Group Symmetries (ICML'26)
• Geometry-Aware Image Flow Matching (ICML'26)
• QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning (ICML'26)

5-min presentations

• Geometry-Aware Image Flow Matching (Junho Lee)
• Preserving Compositionality for Robust Multi-Subject Personalization in Text-to-Image Generation (Eugene Baek)
• Efficient Generative Modeling beyond Memoryless Diffusion via Adjoint Schrödinger Bridge Matching (Jeongwoo Shin)
• No More Hit-and-Hope: Optimal Initial Noise Forging via Noise Shuffling (Suho Ryu)
• Diverse Motion Customization via Control-based Dynamic Optimization (Youngyoon Choi)
• Efficient Adjoint Matching for Fine-tuning Diffusion Models (Dongsoo Shin)
• Explicit Negative Guidance for Object Removal (Eugene Baek)

Multimodal large language models aim to build AI systems that can understand, connect, and reason over information from diverse modalities, such as vision, language, audio, and other sensory signals. By integrating modality-specific perception with the reasoning and generation capabilities of LLMs, they provide a unified framework for interacting with and understanding complex real world information. Key challenges include effectively representing heterogeneous inputs, aligning and integrating information across modalities, and enabling reliable and efficient multimodal reasoning.

Our research spans the full MLLM pipeline, with a focus on multimodal evaluation, perception and visual representation, and efficient model architectures and data strategies for training and inference. We develop diagnostic benchmarks for model reliability, investigate effective ways to encode and represent visual information, and explore computationally efficient approaches to multimodal learning. Building on this foundation, we are expanding the scope of multimodality toward embodied action, including vision-language-action models, and omnimodal intelligence that integrates a broader range of sensory modalities.

Recent publications

• Fine-Grained Multi-Image Object Hallucination Benchmark (CVPR'26)
• A More Word-like Image Tokenization for MLLMs (CVPR'26)

5-min presentations

• A More Word-like Image Tokenization for MLLMs (Hyun Lee)
• Fine-Grained Multi Image Object Hallucination Benchmark (Joonki Min)
• Target-Aware Data Selection Along the Training Path for Multimodal Instruction Tuning (Hyungwook Choi)
• Sparse Visual Contextualization via Input-Dependent Layer Routing (Hyemin Jeong)

Recent publications

• GaussianVideo: Efficient Video Representation and Compression by Gaussian Splatting (CVPRW'25)
• Towards Robust and Smooth 3D Multi-Person Pose Estimation from Monocular Videos in the Wild (ICCV'23)
• Towards Efficient Neural Scene Graphs by Learning Consistency Fields (BMVC'22)

The world around us is inherently three-dimensional, and enabling machines to perceive, understand, and interact with this 3D world is a fundamental goal of artificial intelligence. Our research focuses on developing robust and efficient 3D representations that bridge visual observations and spatial understanding, with applications in robotics, healthcare, and embodied AI.

Our recent research focuses on advancing 3D understanding through efficient scene representations and geometric learning. We develop methods for articulated object reconstruction from sparse observations, 3D CT reconstruction from X-ray images, and integrating 3D representations into world models to enable spatially consistent prediction and reasoning for embodied intelligence. We also study novel 3D representations that improve the scalability, robustness, and efficiency of 3D perception systems.

5-min presentations

• ArtSplat: Feed-Forward Articulated 3D Gaussian Splatting from Sparse Multi-State Uncalibrated Views (Inseo Lee)
• Feed-forward 3D CT Reconstruction (Jiwoong Lee)

Modern online platforms operate on a massive scale of users and items, making efficient recommender systems crucial for optimizing both user experience and business profitability. Our lab has established a strong foundational focus on Graph Representation Learning (GRL) to model complex user-item interactions. Recently, our research has expanded to integrate Large Language Models (LLMs) and the noise-to-sample generative paradigms of Diffusion Models into recommendation systems.

Also, having experience both on computer vision and recommendation systems, VIP Lab is one of the best places to work on content-based video recommendation. Taking advantage of this, we extensively work on content-based recommendation systems, particularly multimedia features on movie and music domain.

Recent publications

• A Scalable Inter-edge Correlation Modeling in CopulaGNN for Link Sign Prediction (ICLR'26)
• Local Large Language Models for Recommendation (CIKM'25)
• Graph Spectral Filtering with Chebyshev Interpolation for Recommendation (SIGIR'25)
• Content-based Graph Reconstruction for Cold-start item recommendation (SIGIR'24)

5-min presentations

• A Scalable Inter-edge Correlation Modeling in CopulaGNN for Link Sign Prediction (Jinkyu Sung)
• When Co-occurrence is Not Enough: Principled Signal Construction and Regularization for Distilled Recommendation (Junhyeong Park)
• Fatigue-aware Micro-Video Recommendation (Myunggeum Jee)
• One-step Diffusion for Recommendation (Yebonn Han)

Machine learning is widely applied not just to commercial applications but to scientific problems. Recently, machine learning plays a key role in scientific discoveries beyond the scale that human brain can handle. Climate science is a good example where machine learning and computer vision technology helps natural science.

We work on molecular structure estimation and brain imaging, with collaborators both from academia and industry, developing machine learning models that aim to understand general physical rules of our world.

Recent publications

• Channel-wise Latent Space Decorrelation for 3D Brain MRI Generation (MIUA'26)
• Robust and Interpretable Adaptation of Equivariant Materials Foundation Models via Sparsity-promoting Fine-tuning (ICLR'26)
• Towards Physically Reliable Molecular Representation Learning (UAI'23)

Presentations

• Optimal Crystal Flow for Fast and Reliable Crystal Generation (Seunghoon Yi)
• Channel-Wise Latent Space Decorrelation for 3D Brain MRI Generation (Wonyoung Jang)