From perception to understanding.
I’m Jeonghwan Kim, a Ph.D. candidate at UIUC’s BLENDER Lab, advised by Professor Heng Ji. I am a recipient of the Capital One Ph.D. Fellowship (2026–2027).
I study multimodal foundation models that perceive, ground, and reason over fine-grained visual information. My work connects visual grounding, cross-modal alignment, and knowledge integration to make these systems more accurate and interpretable. Recently, I’ve been extending this work to visually grounded action policies for embodied AI and robotics.
I’m on the industry and academic job market for 2027. Please get in touch if you know of opportunities or would like to connect.
01 / Updates
News
Latest first · scroll for more
-
The project page for Mixture of Layers is now live. I’ll be in Atlanta for NeurIPS 2026!
Explore Mixture of Layers -
Our PARTONOMY project page is now available.
Explore PARTONOMY -
Our paper Mixture of Layers: Dynamic Layer Routing for Visual Reasoning was accepted to NeurIPS 2026.
Project -
Our paper Pixel-Grounded Retrieval for Knowledgeable Large Multimodal Models (PixSearch) was accepted to NeurIPS 2026 @ VLM4RWD. Oral Presentation
Paper -
I received the Capital One Ph.D. Fellowship (2026–2027) at UIUC.
-
I joined Meta in May as a Research Scientist Intern, working on language-guided world action models for smart glasses.
-
Our paper MemGuard: Preventing Memory Contamination in Long-Term Memory-Augmented Large Language Models was accepted to EMNLP 2026 Findings.
-
Our paper PEARL: Self-Evolving Assistant for Time Management with Reinforcement Learning was accepted to ACL 2026.
Paper -
Our paper ProMac: Language-Guided World Action Modeling for Single-View Egocentric Navigation is a preprint under review.
-
Our paper PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding was accepted to NeurIPS 2025. Spotlight
Project -
I joined Meta in May as a Research Scientist Intern (Part-time Student Researcher), working on pixel-level retrieval-augmented generation for multimodal models.
-
Our paper Infogent: An Agent-Based Framework for Web Information Aggregation was accepted to NAACL 2025 Findings.
Paper
-
Our paper Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models was accepted to EMNLP 2024.
Paper -
Our paper ARMADA: Attribute-Based Multimodal Data Augmentation was accepted to WikiNLP at EMNLP 2024.
Paper -
I joined Amazon in May as an Applied Scientist Intern, developing a generalizable multimodal encoder for geospatial applications.
-
Our paper Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise was accepted to NAACL 2024 Findings.
Paper
-
Our paper FinePrompt: Unveiling the Role of Finetuned Inductive Bias on Compositional Reasoning in GPT-4 was accepted to EMNLP 2023 Findings.
02 / Research
Selected publications
* indicates equal contribution.
03 / Experience
Professional Experience
Meta
Research Scientist Intern
World-Action Models for Wearables · Redmond, WA
May 2026 – Present
Meta
Research Scientist Intern (Part-time Student Researcher)
Multimodal RAG + MLLM · Redmond, WA
May – Oct 2025
Amazon
Applied Scientist Intern
Multimodal Representation Learning · Bellevue, WA
May – Aug 2024
UIUC
Graduate Research Assistant · Ph.D.
Advisor: Heng Ji
Aug 2023 – Present
KAIST
Research Associate
IR&NLP Lab · Advisor: Sung-Hyon Myaeng
Mar 2022 – Jul 2023
KAIST
Graduate Research Assistant · M.S.
IR&NLP Lab · Advisor: Sung-Hyon Myaeng
Feb 2020 – Feb 2022
04 / Background
Education & service
UIUC
Ph.D. in Computer Science
BLENDER Lab
Aug. 2023 - Present
KAIST
M.Sc. in School of Computing
IR&NLP Lab
Feb. 2020 - Feb. 2022
Republic of Korea Marine Corps
Honorably Discharged
Mandatory Military Service
Mar. 2015 - Dec. 2016
Handong Global University
B.Sc. in Computer Science & Electrical Engineering
Magna Cum Laude
Mar. 2014 - Feb. 2020