From perception to understanding.

I’m Jeonghwan Kim, a Ph.D. candidate at UIUC’s BLENDER Lab, advised by Professor Heng Ji. I am a recipient of the Capital One Ph.D. Fellowship (2026–2027). My research is also supported by HERE Technologies. I worked with Shane Moon and Xin Luna Dong at Meta Reality Labs and Emre Eftelioglu at Amazon.

I study multimodal foundation models that perceive, ground, and reason about fine-grained spatial and temporal information. My work connects visual grounding, cross-modal alignment, and knowledge integration to make these systems more accurate and interpretable. Recently, I’ve been extending this work to visually grounded action policies for embodied AI and robotics.

I’m on the industry and academic job market for 2027. Get in touch if you would like to connect.

Research keywords: multimodal LLMs, multimodal post-training, visual tokenization & memory, vision-language-action.

Google Scholar CV Email LinkedIn X

01 / Updates

News

Latest first · scroll for more

2026

The project page for Mixture of Layers is now live. I’ll be in Atlanta for NeurIPS 2026!
Explore Mixture of Layers
Our PARTONOMY project page is now available.
Explore PARTONOMY
Our paper Mixture of Layers: Dynamic Layer Routing for Visual Reasoning was accepted to NeurIPS 2026.
Project
Our paper Pixel-Grounded Retrieval for Knowledgeable Large Multimodal Models (PixSearch) was accepted to NeurIPS 2026 @ VLM4RWD. Oral Presentation
Paper
I received the Capital One Ph.D. Fellowship (2026–2027) at UIUC.
I joined Meta in May as a Research Scientist Intern, working on language-guided world action models for smart glasses.
Our paper MemGuard: Preventing Memory Contamination in Long-Term Memory-Augmented Large Language Models was accepted to EMNLP 2026 Findings.
Our paper PEARL: Self-Evolving Assistant for Time Management with Reinforcement Learning was accepted to ACL 2026.
Paper
Our paper ProMac: Language-Guided World Action Modeling for Single-View Egocentric Navigation is a preprint under review.

2025

Our paper PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding was accepted to NeurIPS 2025. Spotlight
Project
I joined Meta in May as a Research Scientist Intern (Part-time Student Researcher), working on pixel-level retrieval-augmented generation for multimodal models.
Our paper Infogent: An Agent-Based Framework for Web Information Aggregation was accepted to NAACL 2025 Findings.
Paper

2024

Our paper Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models was accepted to EMNLP 2024.
Paper
Our paper ARMADA: Attribute-Based Multimodal Data Augmentation was accepted to WikiNLP at EMNLP 2024.
Paper
I joined Amazon in May as an Applied Scientist Intern, developing a generalizable multimodal encoder for geospatial applications.
Our paper Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise was accepted to NAACL 2024 Findings.
Paper

2023

Our paper FinePrompt: Unveiling the Role of Finetuned Inductive Bias on Compositional Reasoning in GPT-4 was accepted to EMNLP 2023 Findings.

02 / Research

Selected publications

All publications on Scholar

* indicates equal contribution.

NeurIPS 2026

Mixture of Layers: Dynamic Layer Routing for Visual Reasoning

Jeonghwan Kim*, Sofia Stoica*, Jiwan Chung, Ansel Blume, Hyeonjeong Ha, Zhenhailong Wang, Xin Luna Dong, Heng Ji

Project

NeurIPS 2026

Advancing Affordance-Grounded Creative Tool Use in Large Multimodal Models

Cheng Qian, Hyeonjeong Ha, Jiayu Liu, Jeonghwan Kim, Emre Can Acikgoz, Bingxuan Li, Kunlun Zhu, Jiateng Liu, Aditi Tiwari, Zhenhailong Wang, Xiusi Chen, Heng Ji

Paper

NeurIPS 2026 @ VLM4RWD Oral Presentation