From perception to understanding.

I’m Jeonghwan Kim, a Ph.D. candidate at UIUC’s BLENDER Lab, advised by Professor Heng Ji. I am a recipient of the Capital One Ph.D. Fellowship (2026–2027).

I study multimodal foundation models that perceive, ground, and reason over fine-grained visual information. My work connects visual grounding, cross-modal alignment, and knowledge integration to make these systems more accurate and interpretable. Recently, I’ve been extending this work to visually grounded action policies for embodied AI and robotics.

Google Scholar CV Email

01 / Research

Selected publications

All publications on Scholar

* indicates equal contribution.

Research overview for Mixture of Layers: Dynamic Layer Routing for Visual Reasoning

NeurIPS 2026

Mixture of Layers: Dynamic Layer Routing for Visual Reasoning

Jeonghwan Kim*, Sofia Stoica*, Jiwan Chung, Ansel Blume, Hyeonjeong Ha, Zhenhailong Wang, Xin Luna Dong, Heng Ji

Research overview for Advancing Affordance-Grounded Creative Tool Use in Large Multimodal Models

NeurIPS 2026

Advancing Affordance-Grounded Creative Tool Use in Large Multimodal Models

Cheng Qian, Hyeonjeong Ha, Jiayu Liu, Jeonghwan Kim, Emre Can Acikgoz, Bingxuan Li, Kunlun Zhu, Jiateng Liu, Aditi Tiwari, Zhenhailong Wang, Xiusi Chen, Heng Ji

Research overview for Pixel-Grounded Retrieval for Knowledgeable Large Multimodal Models

Preprint, 2026

Pixel-Grounded Retrieval for Knowledgeable Large Multimodal Models

Jeonghwan Kim, Renjie Tao, Sanat Sharma, Jiaqi Wang, Kai Sun, Zhaojiang Lin, Seungwhan Moon, Lambert Mathias, Anuj Kumar, Heng Ji, Xin Luna Dong

Paper

Research overview for Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignment

Preprint, 2026

Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignment

Dwip Dalal, Shivansh Patel, Chahit Jain, Jeonghwan Kim, Utkarsh Mishra, Alex Baratian, Hyeonjeong Ha, Heng Ji, Svetlana Lazebnik, Unnat Jain

Paper

Research overview for Constructive Distortion: Improving MLLMs with Attention-Guided Image Warping

ICLR 2026

Constructive Distortion: Improving MLLMs with Attention-Guided Image Warping

Dwip Dalal, Gautam Vashishtha, Utkarsh Mishra, Jeonghwan Kim, Madhav Kanda, Hyeonjeong Ha, Svetlana Lazebnik, Heng Ji, Unnat Jain

Paper

Research overview for PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding

NeurIPS 2025 Spotlight

PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding

Ansel Blume*, Jeonghwan Kim*, Hyeonjeong Ha, Elen Chatikyan, Xiaomeng Jin, Khanh Duy Nguyen, Nanyun Peng, Kai-Wei Chang, Derek Hoiem and Heng Ji

Paper

Research overview for Infogent: An Agent-Based Framework for Web Information Aggregation

NAACL 2025, Findings

Infogent: An Agent-Based Framework for Web Information Aggregation

Revanth Gangi Reddy*, Sagnik Mukherjee*, Jeonghwan Kim*, Zhenhailong Wang*, Dilek Hakkani-Tur, Heng Ji

Paper

Research overview for Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models

EMNLP 2024

Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models

Jeonghwan Kim, Heng Ji

Paper

Research overview for Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise

NAACL 2024, Findings

Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise

Giwon Hong*, Jeonghwan Kim*, Junmo Kang*, Sung-Hyon Myaeng, Joyce Jiyoung Whang

Paper

Research overview for Exploiting Numerical-Contextual Knowledge to Improve Numerical Reasoning in Question Answering

NAACL 2022, Findings

Exploiting Numerical-Contextual Knowledge to Improve Numerical Reasoning in Question Answering

Jeonghwan Kim, Junmo Kang, Giwon Hong, Kyung-min Kim, Sung-Hyon Myaeng

Paper Poster

Research overview for FinePrompt: Unveiling the Role of Finetuned Inductive Bias on Compositional Reasoning in GPT-4

EMNLP 2023, Findings

FinePrompt: Unveiling the Role of Finetuned Inductive Bias on Compositional Reasoning in GPT-4

Jeonghwan Kim*, Giwon Hong*, Sung-Hyon Myaeng, Joyce Jiyoung Whang

Paper Poster

Research overview for Graph-Induced Transformers for Efficient Multi-Hop Question Answering

EMNLP 2022

Graph-Induced Transformers for Efficient Multi-Hop Question Answering

Giwon Hong, Jeonghwan Kim, Junmo Kang, Sung-Hyon Myaeng

Paper Poster

Research overview for Have You Seen That Number? Investigating Extrapolation in Question Answering Models

EMNLP 2021

Have You Seen That Number? Investigating Extrapolation in Question Answering Models

Jeonghwan Kim, Giwon Hong, Kyung-min Kim, Junmo Kang, Sung-Hyon Myaeng

Paper Poster

02 / Experience

Professional Experience

Meta

Research Scientist Intern

World-Action Models for Wearables · Redmond, WA

May 2026 – Present

Meta

Research Scientist Intern (Part-time Student Researcher)

Multimodal RAG + MLLM · Redmond, WA

May – Oct 2025

Amazon

Applied Scientist Intern

Multimodal Representation Learning · Bellevue, WA

May – Aug 2024

UIUC

Graduate Research Assistant · Ph.D.

Advisor: Heng Ji

Aug 2023 – Present

KAIST

Research Associate

IR&NLP Lab · Advisor: Sung-Hyon Myaeng

Mar 2022 – Jul 2023

KAIST

Graduate Research Assistant · M.S.

IR&NLP Lab · Advisor: Sung-Hyon Myaeng

Feb 2020 – Feb 2022

03 / Background

Education & service

UIUC

Ph.D. in Computer Science

BLENDER Lab

Aug. 2023 - Present

KAIST

M.Sc. in School of Computing

IR&NLP Lab

Feb. 2020 - Feb. 2022

Republic of Korea Marine Corps

Honorably Discharged

Mandatory Military Service

Mar. 2015 - Dec. 2016

Handong Global University

B.Sc. in Computer Science & Electrical Engineering

Magna Cum Laude

Mar. 2014 - Feb. 2020