Jeonghwan Kim

About

Ph.D. candidate at BLENDER Lab at University of Illinois, Urbana-Champaign (UIUC), currently advised by Professor Heng Ji.

I work on advancing multimodal foundation models, with a focus on enabling systems to understand and reason over fine-grained visual information and aligning it with non-parametric & parametric knowledge of the model. My research aims to bridge the gap between perception and reasoning in vision-language models, pushing toward more reliable and context-aware AI systems.

Specifically, I design and develop models that:

Perform granular visual understanding and grounding beyond coarse image-level representations
Integrate general knowledge and domain-specific knowledge within multimodal reasoning pipelines
Improve alignment between visual and linguistic representations for more accurate generation and retrieval

My work lies at the intersection of:

Multimodal Large Language Models (MLLMs) & Vision-Language Models
Retrieval-Augmented Generation (RAG) and knowledge integration
Visual grounding and fine-grained perception

Research Experience

MetaRedmond, WA, USA

Research Scientist Intern May. 2026 - Present

May. 2025 - Oct. 2025

AmazonBellevue, WA, USA

Applied Scientist Intern May. 2024 - Aug. 2024

University of Illinois Urbana-Champaign (UIUC) Champaign, IL, USA

Graduate Research Assistant (Ph.D.) Aug. 2023 - Present

Advisor: Heng Ji

KAIST IR&NLP Lab Daejeon, Republic of Korea

Research Associate Mar. 2022 - July. 2023

Graduate Research Assistant (M.S.) Feb. 2020 - Feb. 2022

Advisor: Sung-Hyon Myaeng

Selected Publications

For more information, check out my Google Scholar.
^* indicates equal contribution.

Pixel-Grounded Retrieval for Knowledgeable Large Multimodal Models

Jeonghwan Kim, Renjie Tao, Sanat Sharma, Jiaqi Wang, Kai Sun, Zhaojiang Lin, Seungwhan Moon, Lambert Mathias, Anuj Kumar, Heng Ji, Xin Luna Dong

Preprint, 2026

Paper

Alignment-Aware Training for Generalizable VLAs

Dwip Dalal, Shivansh Patel, Jeonghwan Kim, Utkarsh Mishra, Alex Baratian, Hyeonjeong Ha, Heng Ji, Svetlana Lazebnik, Unnat Jain

Preprint, 2026

Paper

Constructive Distortion: Improving MLLMs with Attention-Guided Image Warping

Dwip Dalal, Gautam Vashishtha, Utkarsh Mishra, Jeonghwan Kim, Madhav Kanda, Hyeonjeong Ha, Svetlana Lazebnik, Heng Ji, Unnat Jain

ICLR 2026

Paper

PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding

Ansel Blume*, Jeonghwan Kim*, Hyeonjeong Ha, Elen Chatikyan, Xiaomeng Jin, Khanh Duy Nguyen, Nanyun Peng, Kai-Wei Chang, Derek Hoiem and Heng Ji

NeurIPS 2025 (Spotlight)

Paper

Infogent: An Agent-Based Framework for Web Information Aggregation

Revanth Gangi Reddy*, Sagnik Mukherjee*, Jeonghwan Kim*, Zhenhailong Wang*, Dilek Hakkani-Tur, Heng Ji

NAACL 2025, Findings

Paper

Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models

Jeonghwan Kim, Heng Ji

EMNLP 2024

Paper

Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise

Giwon Hong*, Jeonghwan Kim*, Junmo Kang*, Sung-Hyon Myaeng, Joyce Jiyoung Whang

NAACL 2024, Findings

Paper

Exploiting Numerical-Contextual Knowledge to Improve Numerical Reasoning in Question Answering

Jeonghwan Kim, Junmo Kang, Giwon Hong, Kyung-min Kim, Sung-Hyon Myaeng

NAACL 2022, Findings

Paper Poster

FinePrompt: Unveiling the Role of Finetuned Inductive Bias on Compositional Reasoning in GPT-4

Jeonghwan Kim*, Giwon Hong*, Sung-Hyon Myaeng, Joyce Jiyoung Whang

EMNLP 2023, Findings

Paper Poster

Graph-Induced Transformers for Efficient Multi-Hop Question Answering

Giwon Hong, Jeonghwan Kim, Junmo Kang, Sung-Hyon Myaeng

EMNLP 2022

Paper Poster

Have You Seen That Number? Investigating Extrapolation in Question Answering Models

Jeonghwan Kim, Giwon Hong, Kyung-min Kim, Junmo Kang, Sung-Hyon Myaeng

EMNLP 2021

Paper Poster

Vitæ

Full CV in PDF.

Meta May 2026 - Present

Research Scientist (intern)
Multimodal LLM
Medicas AI Jan 2026 - Mar 2026

Founding Engineer
Multimodal Representation Learning
Meta May 2025 - Oct 2025

Research Scientist (intern)
Multimodal RAG + MLLM
Amazon May 2024 - Aug. 2024

Applied Scientist (intern)
Multimodal Representation Learning
UIUC Aug. 2023 - Present

Ph.D. in Computer Science
BLENDER Lab
IR&NLP Lab, KAIST Mar. 2022 - Jul. 2023

Researcher
Working on QA and Graph Networks
KAIST Feb. 2020 - Feb. 2022

M.Sc. in School of Computing
IR&NLP Lab
Republic of Korea Marine Corps Mar. 2015 - Dec. 2016

Honorably Discharged
Mandatory Military Service
Handong Global University Mar. 2014 - Feb. 2020

B.Sc. in Computer Science & Electrical Engineering
Magna Cum Laude